Production-operations runbook

This page is for the SRE or sysadmin who has to stand CDC up and keep it running: what the platform depends on, what to configure before it will start, what to watch, and what to do when something goes wrong. It stays at operational altitude: it names the dependencies and the operational levers without descending into application internals. Read it alongside the deployment topology and scaling page, which explains the shapes this runbook operates.

What CDC depends on

A production installation of the Master needs:

A relational database, PostgreSQL or Microsoft SQL Server. The Master owns all durable state here and runs its schema migrations at startup, so the database user must be able to create and alter schemas on first boot. This is the one stateful dependency that must be backed up (see below). PostgreSQL is the default; the backend is selected with one configuration key, and SQL Server enables a fully PostgreSQL-free deployment. See database backend.
An OpenID Connect identity provider. The Master authenticates every caller as a resource server; it will not start without authentication configured. The provider is on the request path, so treat it as a first-class availability dependency (see the identity and authentication page).
A secret store. Plugin credentials and other secrets live here, never in configuration files. The store is pluggable: a local encrypted store for simple deployments, or an external vault (see the secrets posture).
A message broker (RabbitMQ or Azure Service Bus). This is required as soon as you run Engine Workers, the governed ingest/egress boundaries, or agents, since all of those ride the broker. A pure standalone Master with no workers and no governed boundaries can run without one, but any real governed deployment needs it.

Engine Workers need only the broker and the platform’s own configuration: no database and no direct identity-provider dependency. Everything a worker needs arrives over the broker, which is what makes workers disposable.

Bringing an installation up

The dependency-ordered sequence:

Provision the database (PostgreSQL or SQL Server) and grant the Master’s database user schema-management rights. See database backend for choosing between the two and for the extra SQL Server provisioning steps.
Configure the identity provider: register CDC’s API audience, the single-page-app client, and any machine clients; the Master validates issuer and audience on every request.
Configure the secret store and seed the secrets the plugins in this environment need.
Configure the broker if this environment runs workers or governed boundaries.
Start the Master. On boot it runs migrations, self-provisions its identity seed, and begins serving the API, the single-page app, and the health probes. It registers its embedded engine automatically.
Start Engine Workers (if any), pointing each at the broker with its region and its tags configured. Each worker registers itself and begins receiving the definitions the Master routes to it.

A healthy installation is one where the Master’s health probe is green, every expected engine has registered and is heartbeating, and the broker shows the platform’s queues bound with consumers attached.

Configuration levers that matter operationally

Engine region. Each engine declares the compute region it runs in. This is enforced for data sovereignty and fails closed: an engine is never sent data for a region it is not authorised for, checked both by the Master’s distribution gate and by the engine itself. Set it deliberately; an unset or wrong region is a containment fault, not a cosmetic one.
Engine tags. Labels that decide which definitions an engine is eligible to run (see deployment topology). Getting these wrong surfaces as “no compatible engine” errors when a tagged definition tries to start.
Execution concurrency. Each engine caps how many instances it runs at once, which bounds its resource footprint and makes a host’s throughput predictable.
Broker provider and claim-check. Which broker, and where large payloads are offloaded (a blob store or local files) so they don’t travel inline on the bus.
Consumer concurrency. How many messages one subscription hands to its handler at the same time, set under Messaging:AzureServiceBus (MaxConcurrentCalls) or Messaging:RabbitMQ (ConsumerDispatchConcurrency). Both default to 8, because a consumer that handles one message at a time makes the commit side, not the engine, the ceiling on governed ingest throughput. Raise it to push more governed writes per hour, but raise the database connection pool with it, and on RabbitMQ keep PrefetchCount at or above the dispatch concurrency or the extra dispatchers sit idle. A value below one is corrected to one rather than failing the host at startup. Note that the two settings differ in scope: the Service Bus value is a budget per subscription, while the RabbitMQ value is one budget shared by every subscription in the host. Above one the broker no longer guarantees that messages are handled in the order they arrived: writes to the same existing record still resolve, because the losing writer is retried after a concurrency conflict, but two messages that both introduce the same brand new business identity can each create a record, and only the first stays mapped to that identity. Set it to 1 if a producer may emit the same new identity twice and that outcome is unacceptable. Two things to watch once concurrency is raised on a workload with heavy same-record contention: a retried write records a failed ingest event followed by a successful one for the same message, so a single failure event is not on its own proof that nothing was committed; and retries carry no backoff, so on Azure Service Bus a heavily contended message can exhaust its delivery count into the subscription’s own dead-letter queue rather than the platform dead-letter topic. Monitor both.
Bounded delivery retry (RabbitMQ). A message whose handler keeps throwing is retried a fixed number of times and then set aside, so a message the consumer can never process stops occupying it. Messaging:RabbitMQ:MaxDeliveryAttempts defaults to 5; a message that spends all of them is moved to the durable queue named by Messaging:RabbitMQ:DeadLetterQueueName (default dc-dead-letter), which nothing consumes and which an operator inspects, drains or replays from the RabbitMQ management UI. The attempt count travels in the x-dc-delivery-attempt header on the message, so it survives a consumer restart and is not lost when a second replica picks the message up; a message that succeeds on a later attempt is simply acknowledged, and the count goes away with it. Between attempts the message waits Messaging:RabbitMQ:RetryDelay (default 00:00:05) in a per-queue -retry queue rather than in the consumer, so the consumer keeps working and a brief database failover does not burn the whole budget in milliseconds. The two defaults together let a handler ride out roughly 20 seconds of a dependency being unavailable. Raise the bound or the delay where dependencies are slow to come back, lower them to give up sooner on a large payload. Attempts below 1 are read as 1; a delay of zero retries immediately. Note that a retried message rejoins the tail of its queue, so it can be overtaken by a later message for the same record. Azure Service Bus needs no equivalent setting: its broker-side MaxDeliveryCount already bounds redelivery and moves the message to the subscription’s own dead-letter queue.
Egress audit confirmation (RabbitMQ). With governance on, an engine releases no row of a governed read before the broker confirms that it stored the read’s audit on the governance.egress.audit queue, which Master consumes. It waits at most Messaging:RabbitMQ:PublisherConfirmTimeout (default 00:00:05); a read whose audit is not confirmed in time, is refused by the broker or has no queue to go to is refused and returns nothing. The queue is declared when Master starts, so start Master before the first governed read on a new broker. Azure Service Bus needs no setting: a send completes only when the broker holds the message.
Bulk ingest batch cap and claim-check. A process element that ingests a collection publishes at most Execution:MaxIngestBatchSize records per message (default 500, roughly 700 KB of message body). A wider collection is split across several messages in order rather than refused or truncated, so raising the cap trades fewer, larger messages against the broker’s body limit and the claim-check threshold; lower it if either is tighter here. Because every bulk message is several times the claim-check threshold, a bulk ingest refuses to run at all when Messaging:ClaimCheck:Provider is unset: the step fails at the element naming the missing setting, which is a far better place to learn it than a publish-time failure at the broker. Single-record ingest is unaffected by both.
Claim-check blob retention. Messaging:ClaimCheck:DeleteAfterRetrieval (default true) deletes an offloaded payload as soon as it has been retrieved once, which keeps the store free of orphans with no lifecycle policy. It also means a redelivery finds nothing: if a consumer resolves a payload and then dies before acknowledging the broker, the redelivered message resolves to a missing blob, spends its delivery attempts on it and dead-letters. For a bulk ingest that costs up to one whole batch (500 records by default) per crash in that window, recoverable only by re-running the process instance. Set it to false to keep the blob so a redelivery resolves again, and then give the store its own expiry policy (a blob-container lifecycle rule, or a scheduled sweep of the local-file directory), because with retention on nothing else deletes it.
Entry-point reconcile interval. Each engine host runs a slow-timer backstop that periodically re-derives which published process versions it should be starting, so a host converges on the newest published version without a restart even if a publish-time signal was missed or a broker reconnect dropped it. The interval defaults to 5 minutes and is set under the Execution:EntryPointReconcile configuration section (the Interval value). It bounds the worst case window in which a just-superseded version could still start: shorten it to narrow that window at the cost of more no-op reconcile passes, lengthen it to reduce load where publishes are rare. A non-positive value falls back to the default.
Call chain depth limit. Execution:CallChain:MaxDepth (default 100) is the deepest Call Activity frame a call chain may open: the child of a top-level instance opens frame 1, its own child frame 2, and so on. A call that would open a deeper frame fails its step with call_chain_depth_exceeded, is not retried, and starts no child. It is one value for the whole deployment and covers every Call Activity; no process definition can override it. Set the same value on every host that executes steps.
Stranded instance reconciler. Master fails a running instance that no host advances any more, for example after an engine host died mid-step, and writes a step log that says why. An instance that waits (for a message, an egress reply, an agent or a child process) is never failed, however long it waits; terminate it by hand if the wait must end. Each host that executes steps marks the instances it still works on, so a long plugin call or a retry delay keeps its instance alive. The settings sit under Execution:ProcessReconciler: Enabled (default true) switches the reconciler off, ScanInterval (default 1 minute) sets how often Master checks and each host marks its instances, and StaleThreshold (default 10 minutes) sets how long an instance can go without any sign of work before Master fails it. Keep StaleThreshold several times longer than ScanInterval.
Audit retention period. Governance:AuditRetention (default P5Y, five years) is how long Master keeps the governance audit. It is an ISO 8601 date duration such as P5Y, P18M or P90D, and it is one period for the whole deployment. Once a day the Audit expiry deletes every governed operation decided before the period, oldest first, and writes one Expiry entry that names the period it used. A shorter period applies to every row at the next run, so moving from five years to two deletes three years of operations the next day. Master refuses to start when the value is zero, negative or not a date duration. An operator sets this value, not a data steward, because a shorter period deletes evidence. The audit tables grow to roughly the period times the rate of governed operations; a read-heavy deployment can shorten the period to keep them smaller.

What to monitor

CDC is instrumented with OpenTelemetry: traces, metrics, and structured logs. Export to a collector is enabled by pointing the standard OTLP endpoint environment variable at one; when unset, the platform is still instrumented but does not ship telemetry off-box. Point it at your existing observability stack rather than standing up a bespoke one. The observability posture page covers what is emitted; operationally, watch:

Health probes. The Master exposes an anonymous liveness probe and a broader readiness/health probe; engines expose their own health endpoint. Wire these into your orchestrator’s liveness and readiness checks. They are deliberately anonymous so a probe never needs a credential.
Engine registration and heartbeats. Every engine you expect should be registered and heartbeating. A missing or stale engine is the Master’s cue to redistribute work, and it should be yours to investigate the host.
Broker depth. Growth in the governed-ingest queue is the clearest signal that the Master is down or falling behind: ingest queues durably rather than failing, so a rising queue is the symptom to alert on.
The dead-letter path. Ingest messages that fail validation or commit land in a dead-letter path with a reason and category. A non-empty dead-letter store is a data or configuration problem to triage, not a transient blip.
The transport poison queue. Distinct from the store above: a message the consumer could not process at all, because it is malformed or fails for a reason the handler never resolves, ends up on the broker’s poison queue once its delivery attempts are spent (dc-dead-letter on RabbitMQ, the subscription dead-letter queue on Service Bus). Messages there carry x-dc-dead-letter-queue, x-dc-dead-letter-routing-key, x-dc-dead-letter-at, x-dc-dead-letter-attempts and x-dc-dead-letter-reason headers naming where they came from, what they cost and why they stopped. The body is byte for byte the original, and the retry counter is stripped, so a corrected copy republished onto the real exchange gets a full budget again. Anything arriving here is a defect to investigate, not a retry to wait out.

Backup and recovery

Back up the database. It is the single source of truth: process and model definitions, running instances and their step history, engine assignments, the governed event-store data, the secret-store index, and the audit trail all live there. Standard point-in-time backup practice for the chosen backend (PostgreSQL or SQL Server) applies; the recovery objective for the whole platform is the recovery objective of this database.
The secret store is backed up on its own terms. For a local store, the encrypting master key must be backed up and protected separately; without it the stored secrets cannot be decrypted. An external vault is backed up per that vault’s own practice.
Engine Workers need no backup. They hold no durable state; recovery of a worker is redeployment, and its in-flight instances are recovered from the Master automatically on restart.
The broker holds transient state. Durable queues buffer in-flight governed messages while the Master is down; they drain when it returns. The broker is not a system of record, but losing its durable queues while the Master is down would drop queued-but-unprocessed governed writes, so run it durably.

Common operational scenarios

An engine goes offline. Expected and self-healing: within roughly a minute of its last heartbeat the Master redistributes its definitions to compatible engines, and its instances resume elsewhere or on its own restart. Investigate the host; no manual failover step is required.
The Master is down (planned or unplanned). New governed ingest queues on the broker and new egress reads park on their instances; both resume when the Master returns. Callers to the management API receive errors while it is down. Plan maintenance windows around the tolerance of your consumers and callers for that pause. There is no data loss, only a pause. See the availability discussion in deployment topology and scaling.
A governed read fails. Where to look depends on how the read was resolved. On a self-contained Master the read is answered in the same process, so a failure surfaces on the process instance: the instance fails and the error is on its step, and there is no dead letter to find because nothing was published. On an Engine Worker the read crosses the broker, so a failed read can appear as a dead letter as well as a stalled instance. Check the instance first in both cases. Setting Egress:ForceBrokerPath to true on the Master restores the broker path there, which is a diagnostic aid for comparing the two, not a steady-state setting.
The identity provider is unreachable. The management API cannot validate tokens, so it refuses access rather than serving it, the platform’s fail-closed stance. Treat the identity provider’s availability as part of CDC’s.
Ingest is being rejected. Check the dead-letter store’s reasons: schema validation failures point at a source-data or model mismatch; identity conflicts point at ambiguous cross-system keys. Fix the cause, then replay the dead-lettered messages. A replay runs the ingest governance check again for the source that sent the record, so a replay that the Rules now refuse stays in the store. The replay records the operator as its repairer, and says whether the operator edited the payload, on the dead letter, on the governance decision and on the stored event. The record keeps its source as the actor, and the dead letter keeps the link from the original rejection to the replay. A dead letter whose model definition is missing is resolved by an edit that supplies a valid model, or by discarding it: an as-is replay of such a row leaves it in the store for one of those two actions.
Master will not start in strict governance mode. The startup error names the models that are still Ungoverned. Activate or retire each of them in the Governance area while Master runs in permissive mode, then switch to strict. A Master in permissive mode writes a warning to its startup log instead.
An engine refuses every governed request after a mode change. Its governance mode differs from Master’s, so Master sends it no governance cache and the engine fails closed. The engine’s startup log says that Master refused its mode. Set the same Governance:DefaultGovernanceMode on every host and restart the engine.
The upgrade stops on model names that differ only in case. A model name is unique among the live models, ignoring case, and the database itself keeps that rule. The upgrade that moves the model name onto the model lineage first lists every set of live models whose names differ only in case, such as Customer and customer, with each lineage id, and stops before it changes anything. CDC never renames a model on its own. Rename all but one model of each set, or archive the ones you no longer need, then run the upgrade again.
A tagged process will not start. No engine carries a matching tag. Either add the tag to an existing engine or start a worker that advertises it.
A step fails naming a control character. Not transient, and not retried by design. The value a step brought in from outside carries a character CDC does not accept in governed context; the error names the element, the loop iteration and the context path. This is an upstream data defect, so the fix is in the source feed, not in the platform. See context value integrity.

Diagnostic: a stored null character in a JSON column

This is an optional, run-it-by-hand diagnostic, not a maintenance task. The platform ships no scanner, no endpoint and no scheduled sweep for this, and none is planned. One upgrade runs the same check itself: the migration that turns the record and event documents into PostgreSQL’s native JSON type (see below).

A text-typed column is not a safe harbour. Two JSON columns are stored as text rather than as a native JSON type: value_json on execution.instance_context_snapshots (one row per context key since ADR-0053 lever 3; before that, one context_json document per instance step), and state_json on model.domain_model_records. A text column accepts the six-character escape sequence for a null character at write time, and then every later read of that row fails, because those reads interpret the column as JSON and the JSON type cannot hold a null character. Such a row is already broken today, before the character rule ships. What the rule changes is that no new one can be created, so the affected set is closed rather than growing.

An operator who suspects such a row exists can find it directly. The query below is for PostgreSQL:

-- Rows carrying an escaped null character in a text-typed JSON column. Such a
-- row was accepted at write time and fails every later read that interprets the
-- column as JSON (PostgreSQL 22P05, unsupported Unicode escape sequence).
SELECT 'instance_context_snapshots' AS source,
       process_instance_id::text    AS id,
       context_key || ' @ step ' || step_number::text AS detail
FROM   execution.instance_context_snapshots
WHERE  value_json LIKE E'%\\u0000%'
UNION ALL
SELECT 'domain_model_records' AS source,
       aggregate_id::text     AS id,
       NULL                   AS detail
FROM   model.domain_model_records
WHERE  state_json LIKE E'%\\u0000%'
UNION ALL
SELECT 'domain_model_events'  AS source,
       id::text               AS id,
       'aggregate ' || aggregate_id::text AS detail
FROM   model.domain_model_events
WHERE  payload_json LIKE E'%\\u0000%';

The pattern matches the literal six-character escape sequence, which is what the column actually holds. It does not match a raw zero byte, which a PostgreSQL text value cannot contain at all. Every other accepted or rejected character is valid, queryable and out of scope for this query by design: a row returned here is one that cannot be read, not merely one that is untidy.

The upgrade to native JSON storage checks for you. A later release stores state_json on model.domain_model_records and payload_json on model.domain_model_events as PostgreSQL jsonb. Its migration runs this check on both columns before it converts them. When any row holds the escape, the migration stops before it changes anything and reports the table, the number of rows and up to 50 of them. Repair those rows as described below and run the upgrade again. Once the upgrade has run, neither column can hold the escape, and only the value_json half of the query still applies.

Remediation is a deployment decision, not a platform one. The row cannot be repaired through the platform’s own read paths, since those are the paths that fail; it is corrected or removed directly in the database, and the governed record is then re-ingested from its source.

Upgrade: agent names that differ only in case

Two live agents may not share a name, ignoring case: “Order desk” and “order desk” cannot both exist. A deleted agent holds no name. The database keeps this rule with a unique index on agents.agents, on PostgreSQL and on SQL Server.

The upgrade that adds the rule copies each agent’s name from its current revision onto the agent. Before it creates the index, it looks for live agents whose names differ only in case. When it finds any, the migration stops before it changes anything and lists each set of names with the agent ids. CDC never renames an agent itself. Rename or delete all but one agent of each set, with the old release still running, then run the upgrade again.

An operator can find the clashes before the upgrade. The query below is for PostgreSQL:

-- Live agents whose current names differ only in case.
SELECT lower(btrim(r.snapshot->>'name')) AS name,
       string_agg(a.id::text, ', ')       AS agents
FROM   agents.agents a
JOIN   agents.agent_revisions r ON r.id = a.current_revision_id
WHERE  a.deleted_at IS NULL
GROUP  BY lower(btrim(r.snapshot->>'name'))
HAVING count(*) > 1;

Upgrade: process names that differ only in case

Two live processes may not share a name, ignoring case: “Order intake” and “order intake” cannot both exist. A process is live while it has a Draft or a Published version. An archived process holds no name. The database keeps this rule with a unique index on process.process_lineages, on PostgreSQL and on SQL Server.

The upgrade that adds the rule moves the name of each process off its versions onto the process itself, taking the name of the newest version. Before it creates the index, it looks for live processes whose names differ only in case. When it finds any, the migration stops before it changes anything and lists each set of names with the lineage ids. CDC never renames a process itself. Rename or archive all but one process of each set, with the old release still running, then run the upgrade again.

An operator can find the clashes before the upgrade. The query below is for PostgreSQL:

-- Live processes whose newest names differ only in case.
WITH newest AS (
    SELECT DISTINCT ON (lineage_id) lineage_id, name
    FROM   process.process_definitions
    ORDER  BY lineage_id, version DESC
)
SELECT lower(btrim(n.name))                 AS name,
       string_agg(n.lineage_id::text, ', ') AS lineages
FROM   newest n
WHERE  EXISTS (SELECT 1 FROM process.process_definitions d
               WHERE d.lineage_id = n.lineage_id AND d.status IN ('Draft', 'Published'))
GROUP  BY lower(btrim(n.name))
HAVING count(*) > 1;

Upgrade: a model or process with more than one Draft

A model or a process has at most one Draft at a time. Every later version starts as a copy of the Published version, and while a Draft is open, a request for a new version returns that Draft. The database keeps this rule with a unique index on the lineage id of the Draft versions, on model.model_definitions and on process.process_definitions, on PostgreSQL and on SQL Server. For processes it replaces the older index on source_definition_id, which only kept apart two Drafts copied from the same version.

Older releases could leave a model or a process with two Drafts. Before the upgrade creates the indexes, it looks for every model and process lineage with more than one Draft. When it finds any, the migration stops before it changes anything and lists each lineage with its kind, the number of Drafts and the ids of those Drafts. CDC never deletes a Draft itself, because only you know which one holds the work to keep. With the old release still running, open each listed model or process, keep the Draft you want and delete the others, then run the upgrade again.

An operator can find these lineages before the upgrade. The query below is for PostgreSQL:

-- Model and process lineages with more than one Draft.
SELECT 'model' AS kind, lineage_id, string_agg(id::text, ', ' ORDER BY version) AS drafts
FROM   model.model_definitions
WHERE  status = 'Draft'
GROUP  BY lineage_id
HAVING count(*) > 1
UNION ALL
SELECT 'process', lineage_id, string_agg(id::text, ', ' ORDER BY version)
FROM   process.process_definitions
WHERE  status = 'Draft'
GROUP  BY lineage_id
HAVING count(*) > 1;

Upgrade: references move to lineage ids

A reference from one definition to another, such as the process a Call Activity calls, names the lineage of its target and runs the target’s Published version at each use. Older releases stored the id of one version instead. The upgrade rewrites each stored version id to the id of that version’s lineage. It rewrites every version of every definition, whatever its status, and changes no status and no version number. Of the running instances it touches only the message waits, described below. A stored id that names no version stays as it is: the Dependencies view shows it as a dangling dependency, and a run that uses it fails.

Message events follow their Event model the same way, and a message now travels on the topic named by the Event model’s lineage id rather than by its name. The upgrade therefore also rewrites each instance that waits at a message catch event from the model name it waits for to that model’s lineage id. A name that no Event model holds any more, because the model was renamed before the upgrade, stays as it is and is listed as unmapped: that instance waits for a message nothing sends, so end it or resume it by hand. Before the upgrade:

Drain the broker queues of message catch and message start events. A message still queued on a name is not delivered after the upgrade.
Switch every system outside CDC that publishes to a CDC message event from the model name to the model’s lineage id, the lineageId the model definitions API returns.
Restart every Engine Worker after the upgrade, so it drops the message waits it holds in memory.

Run the read-only dry run before the upgrade. Start the new release’s Master once with the --lineage-reference-dry-run argument and the usual database configuration. It prints the report and exits. It does not migrate the database and writes nothing:

dotnet Combine.DataController.Master.Host.dll --lineage-reference-dry-run

The same start also prints the report of the connection move, see Upgrade: connections move to their plugin.

The report has a summary line, then one line per field. A rewrite line names the column, the row, the element and field, and the id before and after. For an agent, the row is the agent revision and the field is tools[<tool id>].processId or tools[<tool id>].modelId; every revision is rewritten, not only the current one. A form that still waits for approval is rewritten too: the row is the conversation message and the field is <tool call id>.processId. A form that was already answered keeps the id it had. An unmapped line names an id that maps to no version, which the upgrade leaves as it is. Ids are printed in lower case on both PostgreSQL and SQL Server. At the upgrade, the migration writes the same report to the Master’s log and then rewrites, with no stop.

In the same start, after the migrations, the Master fills the Dependency graph again from the rewritten definitions. Each reference then names the lineage it follows and no version. After the upgrade, open the Dependencies view of each definition that had an unmapped line. The id shows there as a dangling dependency. Point the reference at a definition that exists, or remove it, and publish.

Upgrade: each uploaded version joins its plugin

The upgrade gives every uploaded plugin version a plugin id: all uploads whose assembly name is the same, ignoring case, share one plugin. A plugin holds each version once. When two uploads have the same name, ignoring case, and the same version, the upgrade stops before it changes anything and names each pair, for example acme.erp 1.0.0 (2 uploads). Find them before the upgrade:

SELECT LOWER(name) AS name, version, COUNT(*) AS uploads
FROM   process.plugin_assemblies
GROUP  BY LOWER(name), version
HAVING COUNT(*) > 1;

Delete all but one upload of each pair in the plugin admin page, then upgrade again.

From this release the database also holds each plugin name, ignoring case, on one plugin, so two first uploads of one name at the same moment land on one plugin. If a name already spans two plugins, for example after such a race on a release candidate, the upgrade stops and names it, as acme.erp (2 lineages). Delete the uploads of all but one plugin of that name, then upgrade again.

Upgrade: connections move to their plugin

Before this release each uploaded version of a plugin held its own connections, and each upload created its own default. The upgrade moves every connection onto its plugin, so all versions of a plugin share one set of connections, and a connection name is unique per plugin. Connections of one plugin that share a name are merged: when their settings, literal values, curated models and secrets are equal, the oldest one stays and the others are deleted. An agent that named a deleted one names the one that stays. Two connections that each have a secret set never count as equal, because the upgrade has no access to the secret store and so cannot compare the secret values. Such a pair is a conflict like any other, even when the secret values are in fact the same. On an installation where each upload made its own default with a secret, expect a conflict for each plugin that has more than one upload.

When connections of one plugin share a name but differ, the upgrade stops before it changes anything and names each pair. CDC never chooses one, because the tasks would then run against another endpoint or other credentials. The dry run lists the same pairs, as conflict lines, together with a merge line for each connection that will be merged:

Connection merge: 1 Connection(s) merge into an equal one, 1 same-name pair(s) differ.
  merge     'default' <id> (<plugin> 2.0.0) into 'default' <id> (<plugin> 1.0.0)
  conflict  'prod' <id> (<plugin> 1.0.0) and 'prod' <id> (<plugin> 2.0.0) differ in options (settle them first)

Settle each pair before the upgrade: rename one of the two, delete the one no task needs, or give both the same settings. For a pair that differs only because both have secrets, keep one and delete the other. Both have the same name, so after the upgrade every task that named either one finds the one you kept. Then run the dry run again until it lists no conflict line.

Upgrade: plugin tasks name their plugin and connection

Before this release a plugin task named the method of one uploaded version of its plugin, and its connection by name. The upgrade rewrites every plugin task of every process version, whatever its status, so that it names its plugin, the method inside the plugin, and its connection by id. A task that named no connection now names the default connection of its plugin. A later upload of the plugin then keeps the task working, and a rename of the connection keeps the task bound to it.

The rewrite runs after the connections moved to their plugin, and it writes its report to the Master’s log. The report uses the plugin task lines of the lineage reference report:

  rewrite   process.process_definitions.element_configs_json row <id> Task_1.PluginMethodId: <method id> -> <plugin id>/<type>.ExecuteAsync
  rewrite   process.process_definitions.element_configs_json row <id> Task_1.PluginInstanceName: prod -> <connection id>
  unmapped  process.process_definitions.element_configs_json row <id> Task_2.PluginInstanceName: staging (left as it is)

An unmapped line names a method that no uploaded plugin holds, or a connection name that the plugin has no connection for. The upgrade leaves that task as it is. After the upgrade, open the Dependencies view of that process: a method that no plugin holds shows as a dangling plugin dependency. Publish refuses the process, naming the task and the method id or the connection name, until you bind the task again in the designer, and a run of an older Published version fails that task with the same message rather than run it without its method or connection.

The dry run lists these lines on an older database too. The plugin ids are created during the upgrade, so on a database whose plugins have no id yet the dry run makes up one per plugin name: the <plugin id> in its lines differs from the one the upgrade writes, and the method, the connection and every unmapped line are the same. The rewrite also clears the Dependency graph, and the Master fills it again at its next start.

Declared indexes

A model author can declare an index on a Domain Model. On PostgreSQL the Master builds it itself, at runtime, after the publish that declares it: the Master’s database role runs CREATE INDEX CONCURRENTLY and DROP INDEX CONCURRENTLY on model.domain_model_records. The role already holds these rights. On SQL Server it runs ALTER TABLE … ADD / DROP COLUMN (a computed column named cdc_dixc_ plus 16 hex characters, not persisted) and CREATE INDEX / DROP INDEX on the same table, so the Master’s login needs ALTER on that table. Publish never runs DDL and never fails because of a build.

When it runs. At startup (in the background), after each successful publish, after each successful empty-lineage delete, and every 15 minutes. One replica at a time: a pass takes a PostgreSQL advisory lock (SQL Server: sp_getapplock), and a replica that cannot take it skips the pass. The lock is held by a session, so the Master’s connection to PostgreSQL must not go through a pooler in transaction mode (such as PgBouncer transaction pooling).
What it costs. A concurrent build reads the model’s records once and does not block ingest or reads. It waits at most 5 seconds for its table lock (lock_timeout), then fails and is tried again by a later pass. Reads return the same rows with or without the index.
When a build fails. The row turns Failed, with the error in last_error and the count in attempts. The Master tries again 15 minutes later, then after 30 minutes, 1 hour and 2 hours (next_attempt_at). After 5 failed attempts it stops. A publish of the model, or Retry now in the model’s Declared indexes drawer, starts the count again.
SQL Server budget. On SQL Server every model shares one records table, so the Declared indexes of the installation share a budget of 900 computed columns and 900 indexes (the table’s own ceilings are 1,024 and 999). It counts what the head versions declare, Archived models and Failed builds included. Publish refuses a version above the budget with declared_index_budget_exceeded and the used and free amounts. The Operations page’s Declared indexes panel, and budget on GET /api-internal/operations/declared-indexes, show how much is used and free. To free room, remove Declared indexes from a model and publish it, or delete an empty archived lineage. A build that hits the budget anyway (two publishes at once) is Failed with The Declared index installation budget is full and is retried like any failed build. PostgreSQL has no budget.
SQL Server edition. The index is built WITH (ONLINE = ON) on Enterprise, Developer, Azure SQL Database and Azure SQL Managed Instance. Standard and Express do not support online index builds: there the Master builds offline and logs Declared index … is built offline as a warning. An offline build holds a table lock for its whole length, so ingest into every model waits for it; on those editions, publish a model that declares an index on a large lineage in a quiet period.
How to see it. Every index is named cdc_dix_ plus 16 hex characters. The table model.declared_indexes holds one row per index with its status (Building, Built or Failed), attempts, last_error, next_attempt_at and updated_at. The Master logs Declared index … is built at information level, and each failed attempt as a warning: Declared index … failed to build, attempt n of 5: …, with the lineage, the paths, the error, and the next retry time or “no automatic retry left”.
/health is Degraded. The declared-indexes health check on /health (not /alive) reports Degraded while an index is Failed after 5 of 5 attempts. Degraded answers 200, so probes do not restart the Master. Reads still return the same records, only slower. To clear it:
1. Find the index: the Declared indexes panel on the Operations page lists every index with its model, status and error (it needs models:read), as does GET /api-internal/operations/declared-indexes. Or query the table: SELECT index_name, lineage_id, paths, last_error FROM model.declared_indexes WHERE status = 'Failed' AND attempts >= 5.
2. Fix the cause that the error names. A lock timeout means that a long transaction held a lock on model.domain_model_records. A disk or memory error needs room for one more index.
3. Ask the model author to select Retry now in the model’s Declared indexes drawer, or call POST /api-internal/model-definitions/{id}/declared-indexes/retry with models:write. A publish of the model also retries. Do not edit the row by hand.
When an index goes. A publish that removes a declaration drops its index. A changed key type builds a new index before it drops the old one, so for a moment both exist and the table needs room for both. Archive and unarchive keep the indexes; deleting an empty archived lineage drops them.
What not to do. Do not create or drop a cdc_dix_ index or a cdc_dixc_ column by hand. The Master owns every index and column with those prefixes and drops any that no model declares.

What to take into an evaluation

CDC’s stateful surface is small and standard: a relational database (PostgreSQL or SQL Server), an OIDC provider, a secret store, and a broker. The Master owns the state; workers are stateless and disposable.
It is secure-by-default at the operational layer too: it refuses to start without authentication, and its health probes are the only anonymous surface.
Day-two operations lean on your existing stack: OpenTelemetry to your collector, PostgreSQL backup to your backup regime, health probes to your orchestrator.
The failure modes are well-defined: engine loss self-heals, Master loss queues and parks without data loss, and rejected data is dead-lettered for replay rather than dropped.