Constrained decoding versus prompting

Schema-constrained generation — which providers sell under labels like structured outputs or strict mode — means sending a machine-readable description of the object you want back alongside the prompt, and having the provider hold the model to that description as it writes. The description is a JSON Schema: an ordinary JSON document (JavaScript Object Notation, the same format the answer comes back in) stating which keys the object has, what type each value is, which keys are mandatory, and which strings may only be drawn from a fixed list. Holding the model to it is called constrained decoding, and it runs inside the provider's sampler rather than in your code.

It exists to close the gap between what a language model produces and what a program can consume. The model emits text one token at a time, each token chosen by likelihood, while the service on the other end of the call needs a record with named fields to hand to a database. Topic 5.1, on getting JSON out of a model, closed that gap by asking — describing the shape in the prompt and parsing whatever came back — which works most of the time and leaves a failure rate that grows with traffic. Once model output started feeding code paths rather than being read by a person, providers moved the contract out of the prompt and into the request, where it can be enforced mechanically. You meet it as a field on a call you already make: response_format on a chat-completions request, the argument schema attached to a tool definition, or a .parse() helper that takes a Pydantic or Zod class and converts it for you. In every case the schema travels beside the messages and is read before the first token is generated.

Two ways to specify the same shape#

The two approaches differ by about fifteen lines. Putting the contract in English, in the system message, is where most services start because it costs nothing and works everywhere:

SYSTEM = """You extract structured data from support emails.
Reply with JSON matching this shape and nothing else:
{"ticket_id": string, "urgency": "low"|"medium"|"high",
 "product": string, "assignee": string|null,
 "summary": string, "next_action": string}
Do not wrap the JSON in a code fence. Do not add commentary."""

resp = client.chat.completions.create(
    model="gpt-4.1",
    messages=[{"role": "system", "content": SYSTEM},
              {"role": "user", "content": email_body}],
)
raw = resp.choices[0].message.content

The strict version moves the same contract onto the wire as a schema, in the field the provider reads before it starts decoding:

resp = client.chat.completions.create(
    model="gpt-4.1",
    messages=[{"role": "system", "content": "You extract structured data from support emails."},
              {"role": "user", "content": email_body}],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "ticket_extraction",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "ticket_id":   {"type": "string"},
                    "urgency":     {"type": "string", "enum": ["low", "medium", "high"]},
                    "product":     {"type": "string"},
                    "assignee":    {"type": ["string", "null"]},
                    "summary":     {"type": "string"},
                    "next_action": {"type": "string"},
                },
                "required": ["ticket_id", "urgency", "product",
                             "assignee", "summary", "next_action"],
                "additionalProperties": False,
            },
        },
    },
)

The English version is a request; the schema version is a constraint on the sampler. At each decoding step the model produces a score for every token in its vocabulary — on the order of 100,000 to 200,000 candidates for a current model — and normally the sampler picks among all of them. With a schema attached, the provider compiles that schema into a grammar, tracks where in the grammar the partial output sits, and sets the probability of every token that cannot legally continue it to zero before sampling. After {"urgency": has been emitted, the only legal continuations are the opening quote of one of three enum members; after {"urgency": "hi, the only continuation is gh. The model still chooses on the same underlying preferences, from a filtered set.

That second example is the whole mechanism in miniature, and it is worth walking one character at a time. The grammar for urgency admits three strings and nothing else. Before any of the value has been written, every token that could begin one of the three is legal, so the mask permits the opening quote and then the starts of low, medium and high. Once "h is on the wire, two of the three members no longer have a path to them, because neither low nor medium begins with that letter, and the parser state has narrowed to the branch only high occupies. Once "hi is on the wire that branch is the only one left, so every token other than gh is masked — including the tokens that would spell hi, hint or high-priority, all of which are perfectly ordinary English and none of which can complete a legal member. Nothing in that sequence inspects meaning. A member was eliminated the moment a character ruled it out, which is the same narrowing a prefix search performs over a sorted word list.

The step it is easy to read past is what "masked" does numerically, and that step is the difference between this mechanism and every prompting technique. The mask does not lower a token's score. It sets that token's probability to zero before the sampler runs, and a sampler draws from the distribution it is handed, so a token holding zero mass cannot be drawn at any temperature, under any top-p cutoff, with any seed, on any retry. That is why the right word for a fenced response under a strict schema is unreachable rather than rare. A rate driven down by better wording is a small number, and small numbers grow again; a zeroed token is not a small number, and no input is adversarial enough to make an impossible transition legal.

Getting from a schema document to that filtered set takes a compilation step the request never mentions. The provider parses the JSON Schema, builds a state machine that accepts exactly the documents the schema admits, then relabels its transitions from characters onto tokens from that model's tokenizer, since the decoder emits tokens rather than characters and one token routinely straddles several characters. What comes out is a lookup that answers one question quickly: given the parser state so far, which token ids may legally come next.

The loop runs until the model emits an end-of-sequence token — the reserved token a model produces to signal that it has finished — or until the request hits a limit, and the grammar's entire contribution is the arrow from the parser state into the mask. The weights are untouched, the context is untouched, and the sampler behaves as it always did, which explains both what constrained decoding fixes and what it has no way to reach. Compilation happens once per distinct schema and is cached provider-side, at a cost the last chapter of this topic puts numbers on.

That is why one class of failure goes to zero rather than merely down. A response cannot come back wrapped in a code fence, because the first token of a fence has no legal position in the grammar and is masked at step one. It cannot come back with an unquoted key, an unbalanced brace, a trailing comma, a smart quote, or the sentence "Here is the JSON you requested" in front of the object, because none of those are reachable states. That class is not made less likely; it is made unreachable by construction, and no amount of adversarial input brings it back.

Which failures went to zero and which did not#

Sorting a fixed-set run by failure type makes the boundary visible in a way a headline number does not. The counts below come from two branches over an identical set of 500 real support emails, with every response passed through json.loads first and then through a JSON Schema validator:

Those totals are worth assembling out of the rows rather than taking on trust, because which row belongs to which total is the entire finding. A response fails to parse when json.loads raises on it, so the parse-failure total is the fenced objects, the preambles, the truncations and the bare keys added together: twenty-two, nine, six and four. A response parses and is still unusable when the document is well-formed but disagrees with the contract, so the second total is the extra keys, the missing required fields, the out-of-enum urgencies and the wrong scalar types: twenty-seven, eighteen, eleven and seven. The two groups are added separately rather than into one failure count because they are caught by different code at different times — the first by the parser, the second by a validator that only runs if the parser succeeded, and on most services by nothing at all.

In total the prompt-only branch produced 41 responses that would not parse and another 63 that parsed into something the service could not use. Reading the strict column the same way gives four of the first kind, every one of them a truncation, and fifteen of the second. That split — the grammar-governed rows going to zero and the rest going down but not away — is the shape of what a grammar buys, and it is worth being precise about which failures live in which column before deciding what strict mode has bought you. Anything that is a property of the token sequence is gone. Anything that is a property of the world the tokens describe is not. One row belongs to neither group, because a truncation is a property of how many tokens the request allowed rather than of the schema or the world.

The four truncations that survived are a different failure with a different cause, and the chapter on truncation is about them. The fifteen in the last row are the reason this topic does not end here. Every one of them is a response any validator will accept: "urgency": "high" on a routine password reset, an assignee naming somebody who does not work at the company, a summary describing the wrong incident. Constrained decoding governs the sequence of tokens, so it can guarantee syntax and vocabulary and it can guarantee nothing about whether the values are true.

The middle rows are the interesting ones politically, because extra keys, missing fields and out-of-enum values are the failures people do not count when arguing that prompting is good enough. None of the three raises anywhere. A missing next_action becomes an attribute error in whichever consumer reads it first, usually days later and usually in a different service. An extra key is discarded silently by most deserialisers, so the only trace is that the model has started answering a slightly different question from the one the schema asks. And a value outside the enum reaches a switch that has no case for it, which is the failure that costs the most to find, because the default branch was written for something else and does its job perfectly.

The silence in the middle case is worth one more sentence, because it is deliberate rather than a bug. A deserialiser that is liberal about what it receives, absorbing an unknown key instead of raising on it, is what lets a client keep working when a server adds a field it has never heard of. That is the behaviour every serialisation library ships with by default and the behaviour that makes an evolving API survivable, and it is the same behaviour that makes a contract you were relying on unenforceable. One decision in a library you did not write is doing both jobs, which is why the extra-key row cannot be closed by adding a check where the object is read.

The middle rung between prose and a grammar#

Most providers expose a third mode between those two, and running it over the same fixtures separates the guarantees that come from having any grammar at all from those that come from having your grammar. JSON mode — response_format={"type": "json_object"}, the mechanism topic 5.1 leaned on to get a parseable document back — compiles a grammar for the JSON language itself and makes no reference to your schema, because it has never seen it. As a third branch over the same 500 emails it produced 5 responses that would not parse, every one of them cut off mid-object, and 63 that parsed into something the service could not use.

The middle column is the shape of the confusion this topic exists to clear up. JSON mode guarantees that the response is a document; it has no opinion about which document, so the four contract rows carry exactly the counts the prompt-only branch produced. A team that moves from prose to JSON mode watches JSONDecodeError disappear from the logs, concludes the problem is solved, and has removed its loudest failures while keeping all of its quiet ones.

Why those four rows cannot move follows from what json_object has to compile. The grammar of the JSON language says that an object is an opening brace, then any number of quoted-string-then-colon-then-value pairs separated by commas, then a closing brace. Every part of that is generic on purpose. The string in the key position is any string, the value is any of JSON's value forms, and the number of pairs is unbounded, because those are the rules of the format rather than the rules of your record. A decoder held to that grammar can therefore rule out a fence, an unquoted key and a trailing comma, since each of those breaks the production itself. It cannot rule out a key your schema never declared, a required key that never appears, a string outside an enum, or an integer where a string was wanted, because to the JSON grammar each of those is an ordinary well-formed object. The guarantee is exactly as wide as the language and not one clause wider.

The belief that JSON mode is a weaker version of the same thing is tempting for a reason worth naming, and it is not carelessness. Almost every instrument a service has pointed at this problem is an exception counter, and json_object removes the class that raises exceptions. The JSONDecodeError rate goes to zero, the dead-letter queue empties, the alarm stops firing, and every signal the team owns agrees the problem is fixed. What no signal reports is the class that never raised in the first place, which is precisely the class json_object leaves untouched. The failure rate did not fall by the share the graph implies; the loud failures were removed and the quiet ones were left where they were, and the graph went flat because the graph was only ever plotting the loud ones.

The parser is not the contract#

json.loads answers one question — is this string syntactically valid JSON — and a service that depends on named fields depends on a second question that a parse never asks. {"urgency": "critical", "ticket_id": 88214, "confidence": 0.9} parses cleanly. It is missing three required fields, carries a key the schema does not define, uses an enum member that does not exist, and types ticket_id as an integer. A downstream router switching on three known urgency values will fall through to its default on critical, silently, forever.

Pushing the same responses through the real contract rather than through the parser is a dozen lines:

import json
from jsonschema import Draft202012Validator

validator = Draft202012Validator(TICKET_SCHEMA)

loads_ok = schema_ok = 0
for raw in saved_responses:            # the same 50 he sampled
    try:
        obj = json.loads(raw)
    except json.JSONDecodeError:
        continue
    loads_ok += 1
    errors = sorted(validator.iter_errors(obj), key=str)
    if not errors:
        schema_ok += 1
    else:
        for e in errors:
            print(f"{'.'.join(map(str, e.absolute_path)) or '<root>'}: {e.message}")

print(loads_ok, schema_ok)             # 50 31

The lines the validator prints name the clause of the contract that was broken, and where:

<root>: Additional properties are not allowed ('confidence' was unexpected)
<root>: 'next_action' is a required property
urgency: 'critical' is not one of ['low', 'medium', 'high']
ticket_id: 88214 is not of type 'string'

The general form of the mistake is using the parser as the contract. A parser accepts a superset of what your code can handle, and that gap is where silent corruption lives. Any before-and-after comparison that uses json.loads as its bar will find a small improvement, because it measures the one column where prompting does reasonably well and ignores the columns where it does not.

A prompt-only failure rate has a shelf life#

The second problem with a sample of fifty shows up over months rather than in one afternoon. A prompt-only failure rate is a fact about how a particular model behaves on a particular day, and it moves without anyone touching the prompt. On this service the fence rate — responses wrapped in a fenced JSON block despite the explicit instruction not to — sat between 4.2% and 4.6% for two months, rose to 9.1% over a weekend when the provider rolled a minor revision behind the same pinned model name, and came back down three weeks later. No commit in the repository corresponds to either move. Fifty responses sampled inside one of those regimes describe that regime and nothing else: the instruction that produced fifty passes is a request the model may weigh differently after an update, while a grammar's guarantee has no shelf life.

Three limits are worth being clear about now, because each of them is a chapter later in this topic and each gets mistaken for an argument against constraining the decoder.

Strict mode does not make the model correct. It restricts the output to the set of strings your schema admits and then the model picks the most likely member of that set, which on an ambiguous input can be a confident, well-typed, entirely invented value. That mechanism can make a service quietly worse in ways the validator will never report.

Strict mode does not remove the need to validate. The provider enforces the schema it was given, which is not always the schema you think you sent, and covers only the subset of JSON Schema that provider implements. Keeping a validator in front of the parsed object costs microseconds and is the only thing standing between a silently degraded upstream and your database.

Strict mode is not free. The mask has to be computed at every decoding step and the grammar compiled the first time a schema is seen, and both show up in the tail rather than the median. There is also a quality cost when the constraint bites on genuinely ambiguous inputs. Those numbers are small enough to be worth paying in most services and large enough to measure rather than assume.

The difference between the two mechanisms is therefore not one of degree. Asking buys a rate that drifts with the provider's release schedule; constraining buys a set of outcomes that are unreachable, and leaves you a much smaller and much more specific list of things still to worry about.

What all of that leaves open is a more immediate question. Both branches above hand-wrote the schema as a Python dictionary, which is fine for six fields and unsustainable the moment the same shape also has to exist as a type in the application code. Most teams solve that by declaring the shape once, as a Pydantic model or a Zod object, and generating the wire schema from it. That single declaration then does two jobs at two different moments in the request, and it does not carry the same information into both.