Deployment topology and scaling

This page describes the shapes CDC can be deployed in, how work is placed and scaled across them, and where the availability boundaries sit, at the altitude an architect or capacity planner needs to size an installation and reason about its failure modes. It builds on the runtime roles introduced in the architecture overview; the production-operations runbook covers standing an installation up and keeping it running.

The two roles, recapped

CDC runs as two cooperating roles: a single Master (the control plane that serves the API, owns the durable stores, and orchestrates execution) and one or more Engines (workers that run process instances). The Master always embeds a local engine, so the smallest possible deployment is a single Master process that both governs and executes. Everything below is a decision about how far to grow from that starting point.

Deployment shapes

Standalone Master

One Master process serves the API, owns the data, and executes every process through its built-in engine. There is no message broker to run and no separate worker to manage. This is the natural starting point and the default development shape, and it handles moderate workloads on its own. Its ceiling is the ceiling of a single machine: execution and the control plane share the same host, so heavy execution load competes with API responsiveness.

Master with Engine Workers

One or more standalone Engine Workers run alongside the Master. Each connects to the shared message broker, registers itself, and receives the work the Master routes to it. The Master keeps serving the API and owning the data; engines carry the execution load. The built-in engine and remote engines coexist (the local engine can handle some processes while remote engines handle others), so this is not an either/or migration but an incremental one: add the first worker when a single host stops being enough, add more as load grows.

The broker is the enabling piece. A deployment with no broker configured is a pure standalone Master; configuring a broker (RabbitMQ or Azure Service Bus) turns on the channel that lets workers join. There is no separate “enable clustering” switch: a worker simply starts, connects to the same broker, and registers for work.

How work is placed

Sharding: one owning engine per published definition

Placement is at the granularity of a process definition, not an individual instance. When a definition is published, the Master assigns it to exactly one engine, the least-loaded engine whose tags are compatible (see below), and pushes the definition and its plugin code to that engine. Every instance of that definition then runs on its owning engine. This shard-per-definition model keeps placement decisions simple and cache locality high: an engine holds the definitions and plugin assemblies it actually runs, and load is spread by distributing definitions across engines rather than by contending over a shared queue per instance.

Sub-flows invoked by a call activity run on the parent’s engine, so a composed process is not split across hosts mid-flow. A definition explicitly scoped to run on the Master executes only on the embedded engine, the mechanism that keeps Master-pinned work off the general worker pool.

Tags: keeping work on infrastructure that is allowed to run it

Placement is constrained by tags, plain labels (for example Cloud, On-Prem, GPU, EU) attached to both process definitions and engines. The rules are uniform for the local and remote engines alike:

• An untagged definition runs on any engine.
• An untagged engine runs only untagged definitions.
• A tagged definition runs only on an engine that shares at least one of its tags (case-insensitive).

If a tagged definition has no compatible engine, it cannot start and the API says so explicitly: a misconfiguration surfaces as a clear, actionable error rather than as silently stalled work. Tags are how an operator pins region-restricted, licence-restricted, or hardware-specialised work (GPU inference, an on-prem data zone) to the infrastructure that is permitted to run it. They are a placement control, not a security control: sovereignty containment is enforced separately and fails closed (see the governance and audit model).

Scaling

• Vertical. A standalone Master scales by running on a larger host. Each engine also caps its own instance concurrency, so a single host’s throughput is bounded and predictable rather than unbounded.
• Horizontal. Add Engine Workers. Because work is routed by tags to the least-loaded compatible engine, adding a worker adds capacity for the tag classes it advertises. Scaling a specific workload (a region, a GPU class) is a matter of adding engines that carry the matching tag, not of resizing the whole installation.

The control plane does not scale out horizontally: there is a single Master by design (see availability below). Scaling is therefore asymmetric on purpose: the execution tier grows by adding engines, while the control tier grows vertically. For most landscapes the control plane is not the bottleneck; execution and integration throughput are, and those are exactly what the engine tier absorbs.

Placement, health, and recovery

Remote engines heartbeat to the Master on a short interval (a low tens of seconds by default, configurable). The Master checks for stale engines periodically and declares an engine offline once it has been silent past a timeout. With the defaults, an engine is detected offline within roughly a minute of its last heartbeat. When an engine is declared offline, the Master reassigns its definitions to the remaining compatible engines and re-pushes them, and each engine, on start or restart, recovers its own running instances, reloading them from the Master’s durable record and resuming them where they left off. Because the Master is the system of record for all instance and step state, a worker crash loses no committed history, and recovery is automatic: it does not depend on an operator intervening.

This detection-and-reassignment window is the number to plan around: a failed engine’s work is redistributed within roughly a minute of its last heartbeat under the default settings, and that is the window during which its in-flight instances pause before another engine (or the restarted engine) resumes them. Tightening the intervals shortens the window at the cost of more heartbeat chatter, a normal availability-versus-overhead trade-off.

Persistence

All durable state is owned by the Master and lives in PostgreSQL, organised into schemas by concern (process definitions, models, execution instances and step logs, engine assignments, the governed ingest data, the secret-store index, agents, governance assets, and identity audit). The Master runs its schema migrations at startup, so a deployment converges to the expected schema on boot. Engine Workers hold no database of their own: every fact an engine needs (definitions, plugin code, governance lookups) arrives over the broker, and every result it produces is persisted back through the Master. This is what lets a worker be disposable: it can be destroyed and recreated at will, because it holds no durable state that isn’t already the Master’s.

Availability boundaries

The single Master is the deliberate centre of gravity, and its availability characteristics are the ones to plan maintenance windows around:

• Master downtime stops new governed ingest and egress cluster-wide. All governed writes and reads flow through consumers that run only on the Master. While it is down, ingest messages queue durably on the broker (they are not lost, they are processed when the Master returns) and egress reads issued by an Engine Worker park on the process instance that issued them, resuming when the reply arrives after the Master is back. Engines continue executing non-governed steps of already-running instances, but any step that crosses a governed boundary waits.
• A reference followed on an Engine Worker asks the Master. A step that calls another process or uses an Event model runs the version that is published at that moment, and the Master holds that answer. If the Master does not answer within a few seconds, the step fails with a clear error instead of running an older version or waiting without end. The same holds when the worker reads the process or Event model it resolved and was never assigned: it asks the Master for it, and a called process read this way never makes the worker listen for that process’s start events. A plugin the worker was never given, such as one a called process uses, and the Connection a plugin task runs with are read from the Master the same way, with the same time limit, and such a plugin is not added to what the worker is assigned.
• Governance fails closed under uncertainty. If an engine cannot reach the Master to resolve an unknown asset, the operation is denied rather than allowed: an availability incident never becomes a governance bypass (see the governance and audit model).
• Rejected ingest is not dropped. Messages that fail validation or commit are routed to a dead-letter path and can be replayed once the underlying cause is fixed, preserving their audit correlation.

The practical consequence for an evaluator: CDC is highly resilient to engine loss (automatic redistribution and recovery) and treats Master loss as a planned-maintenance concern: work queues and parks rather than failing, but new governed movement pauses until the control plane returns. Sizing a maintenance window is therefore about how long consumers and callers can tolerate that pause, not about data loss.

How a governed read is resolved

The broker is the transport between processes, not inside one. A governed read therefore resolves in one of two ways, decided by deployment, and identical in everything an auditor or a process author can observe:

• In-process. On a host that runs both the engine and the egress consumer (the self-contained Master), the read is answered in that same process. The step costs about what the query costs, instead of paying a broker round trip plus a park on every page load.
• Across the broker. An Engine Worker has no local egress consumer, so it publishes the read, parks the step, and resumes when the Master replies. This is unchanged.

Governance does not vary with the choice. The pre-hook still decides, per principal, on the host running the process, and the post-hook still audits, so the authorisation and audit records of one read are the same pair, joined the same way, in either topology. The rows are the same rows: both paths run the same read, with the same lineage scope and the same projection.

Answering a read without fetching at all

An Engine Worker pays that broker round trip on every read, including reads of data that did not change. A process author can mark one egress read as cacheable, and the engine then keeps its answer in memory and serves a repeat of the same read from there. Today only a time-travel read qualifies: a read that asks for state at a point in the past has an answer that can never change, so holding it needs no invalidation and risks no staleness. A live read is never held, even on an element that opted in.

What the cache replaces is the fetch and nothing else. The pre-hook still decides, per principal, on every read, so a principal who loses access stops receiving rows at once rather than when an entry expires; and the post-hook still writes its audit row on every read, with the same returned count it would have written without the cache. The audit pair is therefore never broken by the optimisation.

Four operational properties are worth knowing:

• It is per engine, and in memory only. No configuration can turn it into a shared or distributed cache, so two engines in different regions can never serve each other’s rows. An engine restart empties it, which costs latency and nothing else.
• It holds one query’s answer, never a model’s rows. Entries are keyed on the whole query, including the point in time it asks about, so two different queries never share one.
• It is off unless an author turns it on, per element, with a lifetime that bounds how long an entry occupies memory. Another read of the same model on another element stays uncached.
• It disables itself if governance starts shaping rows. The day a policy asks for a per-caller transform of the data, the engine stops using the cache for that read and logs why, rather than serving rows shaped for someone else.

Two further operational consequences follow, and they matter when reading a timeline:

• A short-circuited read writes no park record. A park record means the token really stopped, so measurements derived from park records (waiting time, queue depth) stay correct. The step log for the element is still written with its timings.
• The step says which of them happened. Every governed-read step records how its read was resolved: in process, via the broker, or from the egress cache with no read at all. The process monitor shows it on the step under Read resolved. That is what tells the two topologies apart when reading a timeline, and it is why a host that should answer reads itself but is not doing so is a visible fact rather than unexplained slowness: its steps say via the broker. The value rides on the step payload, so a definition whose snapshot policy is Never records no step payloads and no resolution mode with them.
• A short-circuited read cannot dead-letter. A read that never reaches the broker has no message to dead-letter. A failure surfaces as an error on the process instance, which is where to look: the instance fails and the failure is visible in the monitor. This is the better failure mode (a poison message would otherwise dead-letter while its instance stayed parked indefinitely), but it does mean the place to investigate a failed governed read depends on the topology: the instance for an in-process read, the dead-letter store for a read that crossed the broker.

The in-process path can be turned off per deployment by setting Egress:ForceBrokerPath to true, which restores park-and-publish on a host that would otherwise short-circuit. It exists to compare the two paths while diagnosing a problem, not as a steady-state setting.

What to take into an evaluation

• CDC scales from a single self-contained Master up to a Master plus a fleet of tag-routed Engine Workers, incrementally and without re-architecture.
• Work is sharded per published definition to the least-loaded compatible engine; tags keep specialised or restricted work on the infrastructure allowed to run it.
• Engine loss is handled automatically: reassignment within ~a minute, no lost history, self-recovering instances.
• The single Master is an intentional design point: plan for its downtime as a maintenance window during which governed movement queues and parks rather than fails.