Architecture overview

This page describes how Combine Data Controller (CDC) is put together, at the altitude an architect needs to judge whether it fits an existing landscape. It covers the runtime shape (Master and Engines), the execution model (process-driven, plugin-bound), and the data model (a governed catalogue), without descending into implementation detail. Read it alongside the governance and audit model and the security posture, which cover the control and trust dimensions this page only names.

What CDC is

CDC is a process-driven execution platform that sits between source systems (ERP, CRM, line-of-business, SaaS) and data consumers (analytics, integration flows, AI applications, new digital products). Rather than encoding data-sharing and governance logic as scattered scripts or point-to-point integrations, CDC turns it into explicit, versioned processes that run as monitored, auditable instances.

The unit of design is a BPMN 2.0 process. An author lays out a flow in a web-based designer (tasks, gateways, events, sub-processes) and binds each element to a typed operation. When the process runs, every step is a recorded transition with a correlation identity, state, retries, timeouts, and compensation. Governance is not a layer bolted on top: it materialises inside the flow, at the boundaries where data enters and leaves the platform.

Three properties make this an architecture rather than a workflow tool:

• Execution is standardised. Every governed read and write crosses the same control points, so a policy or ownership rule is enforced once and applies everywhere, independent of the source or consumer system.
• Behaviour is auditable by construction. A running instance yields lineage (what happened, when, triggered by whom, against which model version) because the runtime emits it, not because someone remembered to log.
• The landscape can evolve underneath. Because consumers depend on CDC’s governed contracts rather than on the shape of any particular source system, source systems can be replaced or re-platformed without breaking the dependencies that hang off them.

The runtime: Master and Engines

CDC runs as two cooperating roles: a single Master and one or more Engines.

The Master is the control plane. It serves the API that the web designer and external callers use, owns the durable stores (process definitions, plugin assemblies, running instances, step logs, the governed data), and orchestrates the engines, deciding which engine runs which process and tracking engine health. The Master always embeds a local engine, so a minimal deployment is a single process that both governs and executes.

An Engine is a worker that executes process instances. It receives process definitions and plugin code from the Master, runs instances locally, and reports results back. Engines exist to scale out (spread load across machines), isolate workloads (pin sensitive or resource-heavy work to dedicated hardware or a specific network zone), and provide resilience (if an engine falls silent, the Master reassigns its work).

Where processes run: tags

Placement is controlled by tags: plain labels (e.g. Cloud, On-Prem, GPU, EU) attached to both process definitions and engines. The rules are uniform for the local and remote engines alike: an untagged process runs anywhere; an untagged engine runs only untagged processes; a tagged process runs only on an engine sharing at least one of its tags. If nothing matches, the process cannot start and the API says so. Tags are how an operator keeps region-restricted or specialised work on the infrastructure that is allowed to run it.

The Master-to-Engine channel

Master and remote engines communicate over a message broker (RabbitMQ or Azure Service Bus) on a dedicated internal channel, separate from any business messages a process itself sends, so platform traffic and business traffic never interfere. This is the only transport between an engine and the Master: definition and plugin distribution, status updates, step logs, heartbeats, and even the engine’s governance lookups all ride the same bus. Standardising on one transport means one channel to secure, one to monitor, and one firewall rule. This is an evaluation point for private-network and on-prem deployments, where an additional engine-to-Master HTTP path would be an extra surface to manage.

Engines heartbeat regularly; if the Master stops hearing from one, it declares that engine offline, reassigns its definitions to other available engines, and re-enqueues its in-flight instances so another engine resumes them. Recovery is automatic and does not depend on operator intervention.

Deployment shapes

• Standalone Master. One process serves the API, owns the data, and executes via its built-in engine. This is the natural starting point and handles moderate workloads on its own.
• Master with Engine Workers. One or more standalone engines run alongside the Master, each connecting to the broker and registering for work. The Master keeps serving the API and owning the data; engines carry the execution load, routed by tags. The two can be mixed: the built-in engine handles some processes while remote engines handle others.

The deployment-topology material covers sizing, high availability, and sharding at adoption-planning depth.

The execution model

Process definitions and instances

A process definition is a versioned blueprint authored in the designer. It moves through a lifecycle: Draft (editable), Published (locked and runnable), and Superseded (a published version a newer one replaced), with Archived as the human-retired terminal state. A published definition is immutable: anything depending on it sees a stable structure, and changing it means publishing a new version. Immutability is what makes an instance’s lineage trustworthy, because the definition it ran against cannot have been edited after the fact.

A process instance is a single run. It carries a context (a set of named variables) that steps read from and write to as the flow advances. Each instance has an observable status (Running, Completed, Failed) and a full step history.

Elements bind to typed operations

Each active element in a process binds to a typed operation with declared input and output. Two kinds of binding matter for evaluation:

• Plugin tasks call a method from an uploaded plugin (see the plugin framework below), reading values from the context via input mappings and writing results back via output mappings.
• Governed boundary elements are the points where data crosses into or out of CDC under policy, the subject of the data-control value and governance pages. These are not ordinary tasks; they are where ownership, sovereignty, masking, and audit materialise.

Gateways are decision points that evaluate conditions against the context and route the flow; the first matching condition wins. Events start a process (manually via the API, on a timer, or on an inbound message), finish it (silently or by emitting a message), or pause it mid-flow to send or await a message. Sub-processes group steps behind a disciplined boundary, with one entry and a returned output (see below); call activities invoke a separate reusable process, either blocking on the child or firing and forgetting.

Because the binding is typed and the flow is explicit, an evaluator can reason about a process the way they reason about a program: inputs, outputs, branches, and the exact points where external effects happen.

The sub-process boundary

A sub-process is a black box with one input seam and one output seam, and the execution model enforces that shape rather than leaving it to convention. On entry the engine always pushes a context frame for the sub-process, so the body works against an isolated overlay instead of writing into the parent’s context. On exit the engine captures that overlay, pops the frame back to the enclosing scope, and sets the captured context under the sub-process id. The parent’s first step after the sub-process therefore sees the sub-process output at a single, predictable key, and the sub-process cannot silently leak or lose work across its boundary.

Two publish-time constraints keep the boundary well defined: a sub-process has exactly one internal start (a manual start) and every internal end is a plain end, at any nesting depth. One entry and a uniform exit kind are what let the return contract be uniform in turn.

The return rides the same durable transition the engine already records when it leaves a sub-process, so it is atomic with that step and needs no separate write. This required no change to the stored schema and no data migration: the boundary discipline reuses the existing per-token context substrate and adds only an in-memory containment map used at publish time. The full rationale is recorded in the platform’s architecture decision records.

The plugin and adapter framework

CDC integrates with heterogeneous systems through plugins: compiled .NET libraries containing typed tasks (units of work such as “read rows from system X”, “validate against a rule set”, “call an external API”). Plugins are uploaded to the platform and their tasks wired into process steps. This is the seam where standardised connectors and domain operations (read/write, validation, enrichment, mapping, policy checks) live, so integration is encapsulated behind stable contracts rather than expressed as brittle point-to-point coupling.

The significance for an architect is decoupling: a process depends on a task’s typed contract, not on the internals of the system behind it. Swapping the system behind a connector, or adding a new source, is a plugin concern that does not ripple into the processes (or the consumers) that depend on the governed data. Plugins are versioned and instance-configured (an uploaded assembly can be instantiated with per-instance settings and secrets), so the same connector code serves many environments.

The data model: a governed catalogue

CDC maintains a central catalogue of domain models: named, typed data structures (a Customer with a name, an email, an account balance; an Order that references a Customer). Models give the data flowing through processes a shared vocabulary that is independent of any source system’s schema, and they are the unit that governance attaches to. Models are versioned and follow the same Draft/Published lifecycle as processes, so a consumer reads against a stable contract even as the model evolves.

Models carry more than shape. Properties can be marked sensitive, encrypted, read-only, or deprecated; a property can be designated as the record’s identity; and a model declares how the platform decides whether an incoming record is a new entity or an update to an existing one. Governed data is stored as an event log: the first write of an entity is a Created event and subsequent writes are Updated events that merge into current state, with the full history preserved for audit and replay. The governed domain-model page covers what this means for control and traceability.

How the pieces reinforce each other

The architecture is coherent because each choice supports the others:

• Process-driven execution makes every governed effect an explicit, recorded step, which is what makes audit by construction possible.
• Master/Engine separation lets execution scale and isolate without moving the control plane, so governance and placement stay centralised while capacity distributes.
• Typed plugin contracts and a governed model catalogue decouple consumers from source systems, which is the entire point of a data-control layer: the landscape can change without breaking the dependencies on top of it.

An evaluator can therefore treat CDC as a single control point for enterprise data exchange, rather than as one more integration tool that adds surface without adding accountability.