Verifying schema enforcement

Whether a strict schema is honoured is not a property of a vendor. It depends on the provider, the specific model, the SDK path you called, every hop between your process and the provider, and the individual keywords in your schema — and each of those can differ from the others without any error surfacing. The failure that results is silent by construction: the request is accepted, the response is a 200, and the only symptom is a validator failure rate that looks exactly like a model regression.

That makes enforcement something to verify continuously rather than configure once. This chapter is about the exchange that gives nothing away, the probe that a non-enforcing path cannot pass, the routing rule that makes an unverified pair degrade visibly instead of silently, and the marker that keeps the answer attached to every record long after the verification ran.

The exchange that tells you nothing#

Here is a request carrying a strict schema to a provider that does not implement one, and its reply:

curl -sS -D - https://api.provider-b.example/v1/chat/completions \
  -H "authorization: Bearer $B_KEY" -H "content-type: application/json" \
  -d '{"model":"b-chat-mini",
       "messages":[{"role":"user","content":"Classify this ticket: ..."}],
       "response_format":{"type":"json_schema",
         "json_schema":{"name":"c","strict":true,"schema":{ ... }}}}'

HTTP/2 200
content-type: application/json

{"id":"cmpl-8f2a","model":"b-chat-mini-2025-11",
 "choices":[{"finish_reason":"stop","message":{"role":"assistant",
   "content":"```json\n{\"category\": \"billing\", \"confidence\": 0.82}\n```"}}],
 "usage":{"prompt_tokens":412,"completion_tokens":29}}

There is no response_format_applied field, no warnings array, no header, and no echo of the request that would let a client tell whether the schema was compiled or discarded. The status line says the call succeeded, which it did, and the body says the model answered, which it did. The API accepted the field, found no handler for json_schema on that model, ignored it, and generated freely — which produces fenced objects, extra keys and out-of-enum labels at roughly the rate prompting alone produces them.

Unknown fields being ignored rather than rejected is normal API behaviour and is usually a kindness, because it lets clients send forward-compatible requests. Here it means support was assumed rather than verified, and nothing on the wire will correct the assumption.

The step underneath is the same liberality that makes an extra key in a response invisible, running in the opposite direction. A request body arrives at the provider as text and is deserialised into whatever internal structure that endpoint expects. The deserialiser fills the fields it has slots for and discards the rest, because that is how a service accepts requests written against a newer or an older version of its own API without failing them. So response_format is read by an endpoint that has a slot for it and dropped by one that does not, and dropping it is not an error path. It is the ordinary path, taken silently, with no branch anywhere in the provider's code corresponding to "the client asked for something I cannot do" — because from the endpoint's point of view nothing was asked.

Support is a property of a tuple, not of a vendor#

The first correction is to stop describing capability at the vendor level.

One row needs a word of explanation: $ref is JSON Schema's pointer for reusing a definition declared elsewhere in the same document, and a definition that points at itself is how a schema describes arbitrarily deep nesting, which is where grammar compilers tend to run out of support first.

The last row is the operationally important one, and the older model in the fourth column is the safest of the four to depend on precisely because it fails loudly. It returns 400 unsupported parameter: response_format.json_schema, which stops a deploy instead of degrading a week of data. Everything else in that table has to be established by experiment, and re-established when a model is deprecated, when a schema gains a keyword, or when someone adds a provider on an afternoon.

Keyword-level degradation is the version of this that survives longest undetected, because the request enforces most of the schema and drops one clause. A schema carrying maxLength: 200 on a summary field, sent to a provider that ignores that keyword under strict mode, produces perfectly valid objects with 900-character summaries, and the only place that shows up is a column overflow three systems downstream. The engineering measurements in JSONSchemaBench are worth reading on this point: constrained-decoding engines vary widely in which parts of real-world JSON Schema they actually honour, and the coverage gaps are not the ones you would guess from the documentation.

The tuple extends past the provider in the other direction too. An OpenAI-compatible gateway placed in front of a rotation to centralise spend tracking and rate limiting parses each request against its own model of the API, rebuilds it, and drops fields it does not recognise — and a gateway whose version of response_format predates json_schema will silently strip the schema off a request bound for a provider that would have enforced it. Capability therefore belongs to the tuple of provider, model, SDK version, and every hop in between, which in practice means a probe has to travel the same route as production traffic: the same base URL, the same gateway, the same headers, the same client build. A probe issued directly against the provider from a laptop proves something true about a path nobody uses. It is also worth logging a hash of the serialised request body at the hop nearest the provider, since that is the only artefact showing what was actually sent rather than what was constructed.

The probe that a non-enforcing provider must fail#

Verification needs a request whose correct answer and whose schema disagree, so that only a decoder actually bound by the grammar can produce a conforming response. Such a request is a canary: a call issued on a schedule to test whether a path still behaves, with the correct response known in advance so that any other response is a finding. The cheapest version is a schema with a single-member enum and a question the model knows the answer to:

CANARY = {
    "type": "json_schema",
    "json_schema": {
        "name": "canary", "strict": True,
        "schema": {
            "type": "object",
            "properties": {"answer": {"type": "string", "enum": ["seventeen"]}},
            "required": ["answer"], "additionalProperties": False,
        },
    },
}

def enforces_schema(client, model) -> bool:
    r = client.chat.completions.create(
        model=model, response_format=CANARY, max_tokens=32,
        messages=[{"role": "user", "content": "What is two plus two? Answer with the number."}],
    )
    try:
        return json.loads(r.choices[0].message.content) == {"answer": "seventeen"}
    except (json.JSONDecodeError, TypeError):
        return False

A decoder bound by the grammar has exactly one legal string at that position and must emit seventeen, however much it would prefer to say four. A provider ignoring the field returns 4, "four", {"answer": "4"}, or a sentence, and the probe reports false. The test costs about 40 tokens, runs in under a second, and answers a question no status code answers.

Why the probe has to be built that way follows from what it is trying to tell apart. Two paths are on the table — one where the schema became a grammar, one where it was discarded — and a probe carries information only if those two paths produce different outputs. Send your production schema with an ordinary input and they do not. A capable model asked for a well-formed ticket object usually returns one whether or not anything is compelling it to, so a conforming response is consistent with both paths and the probe has said nothing. The information comes from engineering a disagreement. On the enforcing path there is exactly one legal string at that position, so the output is determined whatever the model would prefer. On the discarding path the model answers the question it was actually asked, and it is very sure what two plus two is. Forced apart like that, the two paths are separated by a single response.

That is also the stumble the design exists to avoid, and it is a comfortable one to fall into. A team that verifies enforcement by sending real traffic and checking that the responses validate is measuring how good the model is at following instructions, getting a high number, and reading it as proof that a mechanism is running when the mechanism may not be running at all.

Run it in three places. In CI, parametrised over every (provider, model) pair in the config, so adding a provider without verifying it fails the build. At process start, to catch a model that changed underneath a pinned name. And continuously in production at a low sample rate — one canary per 500 real calls, per pair, on the same client and the same deployment — which is what catches enforcement being withdrawn by a provider-side change at 04:00 on a Sunday. At 14,000 calls a day that is 28 extra calls, which is not a budget line.

The sampling rate and the detection time are one number seen from two sides, and it is worth deriving rather than picking. Fourteen thousand calls spread across a day is roughly six hundred an hour, so five hundred calls go by in a little under an hour, and a canary fired on every five hundredth call therefore runs at about that interval. That is where the detection figure in the scenario below comes from: the worst case is a withdrawal happening just after one canary, discovered by the next. Halving the detection time means halving the interval, and halving the interval costs the same twenty-eight calls over again — which is the argument for sampling harder than instinct suggests, since the expense is denominated in single-digit numbers of calls and the saving is denominated in hours of traffic served against an assumption.

Alongside the canary, a second signal comes free from traffic you are already serving. Grammar-class failures — a fenced object, an extra key, an out-of-enum value, a wrong type — are impossible under an enforced schema, so their count over a pair's traffic must be exactly zero. A nonzero grammar-class counter on a provider you believe is enforcing is proof that it is not, and unlike the canary it is derived from real requests at full volume. A failure taxonomy kept as separate counters earns its keep here: collapsing it into one validation_failed metric throws the enforcement signal away.

The fallback, and the marker that keeps it visible#

With capability established per pair, the routing decision stops being implicit. What makes it explicit is a registry that the request path reads and the probes write, with one deliberate asymmetry in how it treats missing information:

@dataclass(frozen=True)
class Capability:
    enforces: bool
    checked_at: datetime
    probe: Literal["canary", "production-schema"]

REGISTRY: dict[tuple[str, str], Capability] = {}
STALE_AFTER = timedelta(hours=6)

def mode_for(provider: str, model: str) -> Literal["enforced", "prompted"]:
    cap = REGISTRY.get((provider, model))
    if cap is None or datetime.now(UTC) - cap.checked_at > STALE_AFTER:
        return "prompted"          # unverified is treated as unsupported
    return "enforced" if cap.enforces else "prompted"

An unknown pair falls to the degraded path rather than the optimistic one, so adding a provider without verifying it produces a visible drop in the enforced share of traffic instead of an invisible rise in the failure rate. The staleness window matters for the same reason: a pair that stops being probed stops being trusted, which converts a broken monitoring job into a conservative routing decision rather than into weeks of unchecked assumption.

A request routed to a non-enforcing pair does three things differently. The schema goes into the prompt as text rather than into response_format, since the prose contract is the best available substitute. Temperature drops to 0 to reduce the variance of the formatting. And a bounded repair path is enabled — one retry carrying the validator's error messages back to the model, which recovers something like two thirds of failures and is not worth attempting more than once.

What the service returns while the degraded pair is serving is the same response type it always returns, with one addition: a marker recording how the object was produced.

@dataclass
class Extraction:
    data: Ticket
    provider: str          # "a" | "b"
    model: str
    schema_mode: Literal["enforced", "prompted", "prompted_repaired"]
    schema_version: str    # hash of the schema actually sent

The marker is the design's whole point. It travels with the record into the database, it is a dimension on every metric, and it turns questions that used to require an investigation into a group-by. Failure rate by mode separates a model regression from a routing change immediately, because a model regression moves all modes and a routing change moves the mix. The share of traffic served in prompted mode becomes a service level objective (SLO) — a target you commit to and alert on when it slips — so the degradation is announced at the moment it starts rather than discovered from a downstream data-quality complaint. And schema_version catches the case where two service instances are running different schema revisions during a rollout, which produces a failure pattern that looks exactly like a flaky provider.

Downstream consumers can then make their own decision. A dashboard that aggregates urgency counts may not care. A workflow that auto-closes tickets can require schema_mode == "enforced" and hold everything else for review, which is a policy the team can set knowingly instead of discovering after the fact.

What verification leaves unverified#

Routing around a non-enforcing pair does not make its output as good as an enforcing one, and the marker should not be used to pretend otherwise. Prompted mode on this service ran at a 3.6% post-repair validation failure rate against 0.0% enforced, and the failures that survived repair were the ones concentrated on the hardest inputs. The marker makes that visible; the only thing that fixes it is not routing schema-dependent traffic to that pair, or accepting a worse result on some share of it deliberately.

The canary also has a real limit. It proves that the pair enforces a schema — a one-member enum, three keys, no nesting — and it says nothing about whether your production schema, with its 40-member enum, its nested array of objects and its $ref to a shared address type, is enforced in every clause. A schema that the compiler cannot handle is sometimes rejected at request time and sometimes silently simplified. The complete version of the check probes with the real schema and an input engineered to make the model want to violate one specific clause, which is more work to write and is the only way to test a keyword you actually depend on.

The grammar-class counter is asymmetric in the same way, and it is worth being clear about which direction it runs. A nonzero count is proof of non-enforcement, because those outputs are unreachable under a grammar. A count of zero is not proof of enforcement: a prompted path serving an easy hour of traffic produces zero grammar-class failures too, and on a good model the prompt-only fence rate is a few per cent rather than a majority. The counter catches a withdrawal fast because it sees every request; it cannot certify a path that has simply not been asked anything hard yet, which is why it is a second signal and not a replacement for the canary.

The shape of that asymmetry is worth holding onto, because it recurs everywhere in this kind of monitoring. Enforcement makes a grammar-class failure impossible, so observing one refutes enforcement outright — there is no rate at which an impossible thing occurs, and one instance is as conclusive as a thousand. Non-enforcement only makes such a failure possible, and a possibility that did not occur in the last hour is compatible with both worlds. A test whose positive result is a proof and whose negative result is silence is still worth running at full traffic. It just cannot be the instrument that declares a path healthy, and a dashboard that treats a green counter as a certification has quietly promoted silence into evidence.

The deeper limitation is that enforcement is a property of a remote system you do not control and are not notified about. Everything in this chapter is instrumentation designed around that fact: verify at CI time because config changes, verify at boot because deployments change, verify continuously because the provider changes, and stamp every response so that when one of those verifications is late, the record still says how it was produced. A 200 will never carry that information, and no amount of reading the response body will recover it.

With attribution solved, the remaining argument about constrained decoding is not about correctness at all. It is about what the constraint costs per call, at volume, and the fact that the two costs it moves are read by two different teams who do not measure in the same units.