Context value integrity: the character rule

A data control layer earns its name by refusing to pass on data it can see is broken. CDC applies that stance to the text itself. When a value enters a process instance’s governed context from outside the platform, CDC inspects every string in it.

CDC then does one of two things. A value it cannot store is refused: the run stops at the element that produced the value, and the message names where the value came from. A value it can store but judges is not text is carried through and recorded: the run continues, and the character, the element, the iteration and the path are written to the log so the feed can be audited.

That split is deliberate. CDC is usually the consumer of a master system it does not own, so it cannot make cleaning that source a precondition for the feed running at all. Refusing to pass broken data on silently is the promise. Stopping a 1,400-product feed over one character in one product’s description is not what keeps that promise.

This page states the rule and what each of the two responses looks like operationally.

What is checked, and where

Governed process context is the state a running process instance carries. Every step reads from it and writes to it, and what a step writes is what downstream consumers eventually see: governed records, integration flows, analytics, AI applications.

The check sits at one boundary: the point where a value CDC did not itself produce enters that context. In practice that is a plugin task’s output, a script task result, an agent result, an inbound message payload, a governed read result, and the input supplied when an instance is started. Values CDC produced itself, and values it is re-reading from its own storage, are not re-checked: they were checked when they arrived.

Both the values and the keys of an object are inspected, at any depth. A character hidden three levels down inside an array element is found.

The character scheme

The axis is is this legitimate text, not “does the JSON format permit it”. The JSON format permits an escaped backspace, and a backspace in the middle of a Danish product description is garbage in exactly the way a printer control code is.

Tier	Characters	Behaviour
Legitimate text	Tab (0x09), line feed (0x0A), carriage return (0x0D)	Accepted. Never rejected.
Not text, but storable	0x01 to 0x08, 0x0B, 0x0C, 0x0E to 0x1F, 0x7F (DEL)	Carried through and recorded. The run continues.
Mis-decoded text	The range U+0080 to U+009F	Rejected.
Not representable	0x00 (the null character) and half of a surrogate pair	Rejected.
No rule	Everything else, including the byte-order mark U+FEFF, the line and paragraph separators U+2028 and U+2029, and every valid surrogate pair	Accepted.

Three properties of that table are load-bearing.

Nothing is rejected by implication. The rejected set is enumerated, and a character that is not in it is accepted until a decision record says otherwise. An invisible but harmless mark such as a byte-order mark mid-string is untidy, not invalid, and CDC accepts it. The rule exists to catch what cannot be stored and what is not text, never what is merely unwanted.

A valid surrogate pair is never caught. Emoji and every other character outside the basic plane round-trip unchanged. What is rejected is half of such a pair on its own, which is what code that truncates a string mid-character produces. Left alone, that half character is silently replaced by a replacement marker somewhere downstream, which is an unrecorded mutation of a governed value.

The null character is not a judgement. No relational store CDC supports can hold it inside a JSON document, so there is no configuration under which it could be accepted.

What a rejection looks like

The step fails, and nothing is written: the context is left as it was before the step ran. The failure is an ordinary step failure in every other respect. The element’s configured failure behaviour still routes the flow, and the failed step is recorded in the run log like any other failure.

The failure message names:

• the element that produced the value,
• the loop iteration, when the element sits inside a looped scope,
• the context path of every offending string, capped at ten paths followed by a count of the rest,
• the offending character, in hexadecimal.

It deliberately does not carry two things. There is no byte offset: an offset into a serialised document is not something a person can navigate, and a context path is, which makes the offset redundant. And the message never quotes the offending value, because governed context routinely carries customer data and the message is written to logs and to the run log, which have a wider audience than the data itself.

The failure is never retried, whatever retry policy the element carries. The character is the same on every attempt, so retrying only delays the message somebody has to read.

There is no opt-out

There is no switch at process-definition level and none at element level. A process cannot be configured to accept these characters, and neither can a single task inside it, agent output included. An invariant with a switch is not an invariant: the moment one feed is exempted, the guarantee CDC offers downstream consumers stops being a guarantee.

The remedy for a feed carrying such text is to fix it at the source system. Where the producing element is a script task, whose body the author writes, it can also be cleaned there.

Note the limit honestly: a plugin task has no output mapping, so there is no author-facing seam between the plugin returning a value and the value entering context. For a plugin task the only remedy is at the source. That is one reason a storable character is now carried and recorded rather than refused: for the most common producer of external data, refusing left the author with nowhere to go.

The rule applies on the way in, and only on the way in

Nothing already stored is re-validated. There is no sweep, no report, no migration, and no check on read. A stored context or governed record that carries one of these characters keeps loading exactly as it does today, and replaying or resuming an instance is unaffected. A value that would be refused if it were written now, but which was written before, stays where it is.

That is not an oversight to be closed later. Re-validating on read would break an installation whose stored data round-trips perfectly well, in order to enforce a rule retroactively that buys nothing at that point: the data is already inside, and the producing element is long gone.

One consequence is worth knowing for operations. A value that a text-typed column accepted at write time can still fail every later read, because those reads interpret the column as JSON. Such a row is already broken today, independently of this rule, and the rule is what stops any new one from being created. See the production-operations runbook for the diagnostic an operator can run.

The accepted regression

One part of this is knowingly disruptive, and it is better read here than discovered in production.

The range U+0080 to U+009F passes through CDC unharmed today, on every hop. It is rejected not because anything breaks on it but because of what its presence means. That range is where the Windows-1252 code page keeps the curly quotes, the en dash, the em dash and the ellipsis. A character from that range inside UTF-8 text therefore means one of those punctuation marks was decoded with the wrong code page: the text is already visibly corrupt, and the original character is already lost.

So the day this rule ships, a feed carrying one mis-decoded apostrophe starts failing a step that succeeded the day before. That is the intended trade, not a side effect. The alternative is publishing visibly corrupt text to every downstream consumer, silently, for as long as the feed runs. Rejecting it does not recover the lost character; it stops the run and names the element, the iteration and the path, so a newly failing feed is diagnosable in a single read of the error.

Where the rule is enforced today

The boundary covers the paths a value normally travels: task, script and agent output, inbound message payloads, governed read results, and the start input of an instance started through the management API.

Three start paths do not currently pass through that seam and are therefore unchecked: a start triggered by a webhook route, a start triggered by a timer, and the start of a child process invoked by a calling process. A value entering through one of those reaches the context unvalidated, and is then subject to the rule only at the next external write inside the instance. Closing those paths is outstanding work, not a decision.

One further rough edge is worth naming: a start request rejected through the management API currently surfaces as a server error rather than as a client error, even though the cause is entirely in the request body. The message itself is correct and names the offending path.