Observability posture

This page describes what CDC lets an operator see: its telemetry, its health signals, and how a governed operation can be traced end to end, at the altitude a technical evaluator needs to judge whether the platform will fit an existing monitoring practice and answer operational questions in production. It complements the production-operations runbook, which covers what to watch day to day, and draws on the runtime shape described in the architecture overview.

Standards-based, not a bespoke stack

CDC is instrumented with OpenTelemetry, the vendor-neutral standard for traces, metrics, and logs. It does not ship its own monitoring UI or lock telemetry into a proprietary format; it emits over OTLP to whatever collector you already run. Export is opt-in by configuration: point the standard OTLP endpoint at your collector and telemetry flows to it; leave it unset and the platform is still fully instrumented but does not ship telemetry off-box. The significance for an evaluator is that observability is a matter of pointing CDC at an existing stack (an OTLP-compatible backend, an Aspire dashboard, a cloud-native collector), not of adopting a new one.

Out of the box the instrumentation covers the standard surfaces (HTTP server and client calls, the .NET runtime, and the platform’s own activity sources), so request latency, error rates, and resource pressure are visible without bespoke wiring.

The three signals

Traces

CDC emits distributed traces so a unit of work can be followed across host boundaries. HTTP requests into the management API are traced automatically, and the platform defines its own activity sources for the operations that matter. For example, an agent’s tool calls are emitted as spans tagged with the conversation and turn they belong to and the tool that ran, so an AI-driven operation is traceable rather than opaque. Traces are the tool for answering “where did this request spend its time, and what did it touch.”

Metrics

Alongside the standard HTTP and runtime metrics, CDC instruments its own hot paths. The governance and execution cache (the per-engine cache that keeps governance decisions off the central database on every operation) is metered for hits, misses, stores, invalidations, and lookup latency. Because cache behaviour is exactly what governs both performance and the fail-closed lookup path, this is the signal that tells an operator whether the platform is warm and healthy or thrashing.

EgressCache is metered on the same meter, and separately, because it is a different cache with a different failure mode. It holds Governed Read results per engine and in memory, so it starts cold after every restart and warms independently on each host. Its counters are therefore dimensioned by engine: a warm engine and a cold one must not add into one number, or “is the cache warm?” has no answer. A hit and a miss are counted apart from a bypass, which is a read that never consulted the cache at all because it carries no asOf or names Expanded references. A governance Demand does not bypass it: the Principal transform runs after the cache answers. That separation matters during an incident: without it, a workload that is simply not cacheable would read as a cache that is permanently failing.

Logs

Logging is structured and flows through the same OpenTelemetry pipeline as traces and metrics, carrying scope so a log line can be tied to the operation that produced it. Fields a model marks sensitive are masked in logs (and in the UI), so classification is honoured in telemetry as well as in storage: a log export does not become a data-leak channel.

Health and readiness

Both roles expose anonymous health probes suitable for wiring directly into an orchestrator:

• The Master exposes a liveness probe (is the process up?) and a broader readiness/health probe (are its dependencies satisfied?). Both are deliberately unauthenticated (a probe never needs a credential), and health traffic is filtered out of traces so it does not drown the signal.
• Each Engine Worker exposes its own health endpoint.

These are the anonymous surfaces the security posture carves out explicitly: the health probe is a known, bounded exception to the default-deny stance, not a gap in it.

Tracing a governed operation end to end

Observability and governance reinforce each other. Every governed operation carries a correlation identity issued at decision time and threaded through everything that follows: the access decision, the commit outcome, the audit records, and the telemetry spans. That single identity is what lets an operator (or an auditor) reconstruct the whole story of one operation: the decision that authorised it, whether it committed, which model version it touched, and at whose request. The governance and audit model covers the audit side of this in depth; the point here is that the operational trace and the governance trail share a spine, so a performance investigation and a compliance investigation start from the same identity.

The platform is also the system of record for execution state: every process instance has an observable status (Running, Completed, Failed) and a full, persisted step history. This is queryable state rather than metered counters, the authoritative record of what ran and how far it got, and it is what an operator consults to see the health of the work itself, as distinct from the health of the hosts running it.

Monitoring the cluster

For a multi-engine deployment, the operationally important signals are the ones that describe the fleet:

• Engine registration and heartbeats. Every engine that should be present is registered and heartbeating; a stale engine is both the Master’s trigger to redistribute work and the operator’s cue to investigate a host.
• Broker queue depth. Growth in the governed-ingest queue is the clearest early signal that the Master is down or falling behind: governed writes queue durably rather than failing, so a rising queue is the thing to alert on.
• The dead-letter path. A non-empty ingest dead-letter store is a data- or configuration-quality signal to triage, carrying the reason and category of each rejection.

Service-level objectives

CDC does not impose a prescriptive set of SLOs (targets belong to the organisation running it), but it emits the service-level indicators an operator needs to define and measure them, and it carries a few internal reliability targets worth naming in an evaluation:

• Availability. The liveness and readiness probes are the natural indicators for a platform-availability objective, per role (Master and each engine). Because the single Master is the control-plane centre of gravity, an availability SLO is really two objectives: the Master’s uptime, and the fleet’s, the latter degrading gracefully as engines come and go.
• Latency. HTTP server spans give management-API request latency, and the governance-cache lookup histogram gives the latency of the hot path that gates every governed operation, the two indicators a responsiveness objective is built on.
• Governed throughput and freshness. Broker queue depth on the ingest path is the indicator for a data-freshness objective: how far behind real time governed writes are allowed to fall before it is a breach. It rises when the Master is down or saturated, so it doubles as the early-warning signal.
• Recovery. The platform’s own recovery behaviour sets an implicit objective for engine loss: a failed engine’s work is redistributed and resumed within roughly the heartbeat-plus-timeout window (see deployment topology and scaling), with no lost history. That window is the recovery-time expectation to build an objective around.

The point for an evaluator is that the raw indicators are already emitted over a standard protocol, so SLOs are defined and tracked in your existing SLO tooling, against CDC’s telemetry, rather than requiring anything bespoke from the platform.

What to take into an evaluation

• Observability is standards-based OpenTelemetry over OTLP: you point CDC at your existing collector rather than adopting a new monitoring stack.
• All three signals are covered: traces (including the platform’s own operations), metrics (standard plus the governance cache hot path), and structured logs with sensitive-field masking.
• Anonymous liveness and readiness probes wire straight into an orchestrator, and are a bounded, deliberate exception to default-deny.
• The indicators for availability, latency, freshness, and recovery SLOs are emitted over a standard protocol, so objectives live in your existing SLO tooling rather than in anything platform-specific.
• Operational and governance investigations share a correlation identity, so one operation can be followed from its authorising decision through to its telemetry.