Decoding guarantees and firm instructions

Strict schema-constrained decoding is a property of the sampler rather than of the prompt: at each step, the tokens that would make the partial output impossible to complete into a valid instance of the schema have their logits set to negative infinity before sampling. A response that is not valid JSON, that omits a required field, or that puts a string where the schema says number, is not unlikely under that regime; it is unreachable. An instruction asking for the same shape produces valid output with some probability, and that probability is very close to one, which is a different thing from one.

Ports run into this constantly, because format enforcement is a capability providers differ on and price differences track it loosely at best. A migration that moves from a model with strict schema decoding to one offering only a JSON mode — parseable output with no control over its shape — has changed the class of guarantee the system rests on, and an eval comparing field-level accuracy will report the two as equivalent. This chapter is about what that swap actually costs, what has to be rebuilt in your own code, and why the same distinction decides whether a rule stated as "never" can live in a prompt at all.

Why 300 cases cannot tell you which you have#

Suppose an instruction-only arm fails to produce parseable, schema-conforming output on 0.5% of documents. On a 300-case eval, the chance of seeing zero failures is 0.995^300 ≈ 0.22, so roughly one run in five reports a perfect record for a system that will break on one document in two hundred in production.

That exponent carries the whole argument, so it is worth building rather than accepting. If one document fails with probability half a per cent, it passes with probability 0.995, which is one minus the failure rate. Two documents both pass with probability 0.995 times itself. Three hundred all pass with 0.995 multiplied by itself 300 times, which is what the exponent notation says. Multiplying is the right operation only because the outcomes are independent — one document failing tells you nothing about whether the next one will — and that assumption deserves a moment, because a batch of scans off the same broken feeder is not independent, and where it fails the real chance of a clean run is worse than this rather than better. The product works out at about 0.22, and a probability a little over a fifth is roughly one run in five.

Reading down the middle column shows that 300 is the wrong instrument rather than merely a small one. The chance of a clean run does fall off steeply, but it falls off in the wrong place: it is already down near a twentieth by the time the failure rate reaches one per cent, and still around three-quarters at a tenth of a per cent. The right-hand column converts the same rates into consequences at production volume, which is the conversion that makes them arguable at all — a tenth of a per cent of 100k documents is 100 documents, because a rate multiplied by a population is a count, and a hundred broken extractions is a staffing decision rather than a rounding error.

The 300-case eval reliably detects a two-per-cent problem and is close to blind to a half-per-cent one. That would matter less if the failures were uniformly distributed, and they are not. The documents that make a model hesitate, add a caveat, or wrap its answer in explanation are the unusual ones — a scanned invoice in a second language, a credit note with a negative total, a page where the OCR pass — optical character recognition, the step that turns the scan into text before the model sees it — produced two plausible invoice numbers. Those are also the documents least likely to be in an eval set sampled from ordinary traffic, and most likely to be the ones a customer escalates. The failure rate and the eval's blind spot are correlated, which is what makes the tail argument stronger than the arithmetic alone.

What each guarantee actually covers#

Three mechanisms get discussed as if they were interchangeable, and separating them is most of the work.

The first row's guarantee is worth taking apart, because "unreachable" is a strong word and it is meant literally. Sampling turns logits into probabilities by exponentiating them and rescaling so they sum to one, and the exponential of negative infinity is zero. A token whose probability is exactly zero cannot be drawn at any temperature, under any sampling scheme, on any retry. That is the entire mechanism, and it is why this guarantee does not weaken with conversation depth, with load, or with how the surrounding sentence is worded — the things that weaken every control in the previous two chapters.

What makes it workable rather than merely strong is what the mask is computed against. It is not the finished output, which the sampler cannot see yet. At each step the constraint asks which next tokens would leave the partial output still completable into some valid instance of the schema, and blocks everything else. A closing brace becomes available only once every required field is present. A letter is unavailable in a position where the schema declared a number. The question being answered is local, so it costs one mask per token, and the invariant it maintains is global.

The row that surprises people is the first one's third column. Strict decoding is a guarantee about shape, and a well-formed lie validates perfectly: {"invoice_number": "INV-2024-8831", "total": 184.25} is schema-valid whether or not the total on the page is 1842.50. It also does not protect you against truncation, because a response cut off at max_tokens is a prefix of a valid instance rather than a valid instance, and a provider that streams tokens under a mask will happily stop mid-object when the budget runs out. Setting max_tokens with real headroom is part of the guarantee, not separate from it.

That column catches people because one word is doing two jobs. "Valid" in the schema sense means the bytes conform to a declared shape. "Valid" in the sense anybody downstream cares about means the numbers are the ones that were on the page. Strict decoding delivers the first with certainty and says nothing at all about the second. Because the first is the one that stops the pipeline complaining, the guarantee gets remembered as the stronger of the two, and a schema-valid wrong total is in fact worse than an unparseable response — the unparseable one raises, and the wrong total is written to a database and believed.

Strict decoding also restricts the schemas you are allowed to write, because providers implement it by compiling the schema into a grammar over the token vocabulary and features that do not compile cheaply are refused. Regular-expression pattern constraints, top-level anyOf, recursive $ref and unbounded additionalProperties are the usual casualties, and several implementations require every property to appear in required and additionalProperties to be false. The first request against a new schema pays compilation, on the order of 100–400 ms, after which it is cached on a key derived from the schema itself, so a service that builds schemas dynamically per request pays that latency on every call while one with a fixed set pays it once after a deploy.

The refused features follow from the per-token budget, not from anyone's taste. A mask has to be produced before every single token, so the compiled grammar must answer "which tokens are legal here" in a time that disappears next to the forward pass. A regular-expression pattern over a string field, a top-level anyOf that keeps several candidate schemas alive at once, and a recursive $ref whose depth is not known in advance each turn that question into work that grows as the output grows rather than staying constant. Providers do not refuse them for being unusual. They refuse them because a guarantee that cannot be computed inside one token's budget is not a guarantee, and the honest options are to refuse the feature or to silently stop enforcing it.

There is a quality effect too, running in both directions. Masking removes the model's ability to hedge in a field with no room for hedging, so a model that cannot read an invoice number on a bad scan emits its best guess where the same model unconstrained would have written a sentence saying the page was illegible. That is an improvement for your parser and a loss of information for you. The remedy is to give the schema somewhere to put uncertainty — a nullable field, an explicit "illegible" enum member, a separate confidence number — so abstention stays expressible inside the shape you are enforcing. Those are decisions about the schema rather than about the prompt, and they are part of what a cheaper model takes away: with JSON mode you can still ask for the field, and nothing prevents its omission.

Changing the prompt for a model with no schema#

The prompt work for a model without strict decoding is not about firmness. It is about removing the ways the output can vary and giving the model less to get wrong.

The output contract moves out of prose and into a literal example, because a demonstration of the exact bytes carries more authority than a description of them on most families. Optional fields come out: a schema with six required fields and no nulls produces more consistent output than one with three required and five nullable, so the prompt asks for every field on every document and uses the sentinel "unknown" rather than allowing omission. Enums shrink and get listed inline at the point of use rather than in a separate section. Nested objects flatten where the downstream code does not need the nesting, since each level of nesting is another place a model can improvise.

Then the format constraint moves off the instruction channel entirely, using whatever the provider offers — the assistant prefill from the previous chapter if it exists, response_format: {"type": "json_object"} if that is all there is, and a stop sequence to cut anything after the closing brace:

resp = client.chat.completions.create(
    model="cheap-model-v2",
    response_format={"type": "json_object"},
    max_tokens=800,               # headroom: p99 valid response is 310 tokens
    temperature=0,
    stop=["\n\n"],          # anything after the object ends at a blank line
    messages=[
        {"role": "system", "content": EXTRACTION_CORE},
        {"role": "user", "content": document_text},
    ],
)

That combination gets first-pass validity high. It does not get it to one, so the rest of the guarantee has to be rebuilt in your code.

Rebuilding the guarantee outside the model#

What strict decoding gave you was an invariant at the boundary: nothing invalid crosses it. Without it, you enforce the invariant yourself, and the enforcement has three parts — validate, repair once, then fail loudly.

from pydantic import ValidationError

def extract(document_text: str, *, max_repairs: int = 1) -> Invoice:
    messages = build_messages(document_text)
    for attempt in range(max_repairs + 1):
        raw = call_model(messages)
        metrics.incr("extract.calls")
        try:
            invoice = Invoice.model_validate_json(strip_fences(raw))
        except (ValidationError, ValueError) as err:
            metrics.incr("extract.invalid", tags={"attempt": attempt})
            if attempt == max_repairs:
                raise ExtractionFailed(document_id, raw) from err
            messages += [
                {"role": "assistant", "content": raw},
                {"role": "user", "content":
                    f"That did not validate: {err}. Reply with the corrected "
                    f"JSON object only."},
            ]
            continue
        metrics.incr("extract.valid", tags={"attempt": attempt})
        return invoice

Four details in that function are the actual migration work. strip_fences exists because fenced output is a known, recoverable failure mode and stripping it locally is cheaper than a repair round trip. The repair message includes the validation error text, which raises single-shot repair success from roughly 60% to above 90% in most implementations, since the model is being told what specifically was wrong rather than asked to try again. max_repairs is one, because a second repair on the same document almost never succeeds and the latency is real. And the failure path raises rather than returning a partial object, so the document lands in a queue a human works through instead of a database row with a plausible wrong total in it.

The metrics matter as much as the control flow. extract.invalid broken out by attempt gives you the first-pass validity rate and the repair rate as standing production numbers, which is the only way you will ever know the true rate that a 300-case eval could not measure. The latency cost is quantifiable too: if 3% of documents need a repair and a repair adds 1.4 seconds, the mean cost is 42 ms and the p99 moves by the full 1.4 s, which is the number that matters if the extraction sits in a request path with a 3-second budget.

Both halves of that are one step, and they disagree on purpose. The mean is the cost of a repair multiplied by the share of documents that take one — 1.4 seconds against 3% of documents is 42 ms — because the documents that never repair contribute nothing to the average and only dilute it. A percentile asks a different question: what does the slowest one per cent of requests experience. If 3% of documents repair, the repairing population is three times the size of that tail, so the request sitting at the ninety-ninth percentile is itself a repairing request and carries the whole 1.4 seconds rather than an averaged fraction of it. Had the repair rate been comfortably below one per cent, the mean would still have moved and the p99 would not have moved at all. Which of the two you quote is decided by how the budget was written, and a budget written as a percentile is not satisfied by an average.

A rule stated as "never" is a specification for a check#

The same distinction reaches beyond output shape. A sales agent held at 0.33% quoted prices after the best prompt arrangement anyone could find, in the previous chapter, is at that rate for the same reason an instruction-only extractor is at 0.5%: wording sets a probability. Two moves get to zero, and neither is a prompt technique.

The first is a deterministic check on the response before it leaves your process. A currency regex over the assistant turn is a decidable function of the output, so it either fires or it does not, with no probability attached:

PRICE = re.compile(r"(?:[$€£]\s?\d[\d,]*(?:\.\d{2})?|\b\d[\d,]*\s?(?:USD|EUR|GBP)\b)")

def emit(reply: str) -> str:
    if PRICE.search(reply):
        metrics.incr("agent.price_blocked")
        return PRICING_PAGE_FALLBACK          # fixed text, no model involved
    return reply

The guarantee is now as strong as the regex, which is a thing you can review, test against 2,000 historical replies, and reason about. It has a false-positive cost — an order number formatted as 1,299.00 would trip it — and that cost is visible and bounded rather than probabilistic.

Bounded is the honest word there and it does not mean small. The fallback replaces a correct reply with fixed text, so every false positive is a user who asked about an order number and received a pricing page. That is a behaviour with a rate of its own, and it belongs on the same dashboard as agent.price_blocked rather than being filed as the closing cost of a solved problem. What the check buys is not the absence of a failure. It is the exchange of a failure that is probabilistic, silent and unbounded for one that is deterministic, logged and countable — and the second kind can be argued about with numbers, which is the only reason to prefer it.

The second is removing the capability that makes the violation expressible. If the agent has no pricing tool, no price fields in its retrieval index, and no prices in its system message, then quoting a correct price requires the model to recall one from pre-training, and quoting a wrong one is a different failure that the same regex catches. Cutting the information off at the source is usually stronger than filtering at the exit, and on that agent it was also simpler, since the pricing data was in the index only because someone had indexed the whole marketing site.

Where the analogy stops#

Deterministic checks are not free of judgement, and they do not scale to every rule. "Never quote a price" is checkable because prices have a surface form. "Never give medical advice" has no regex, and an output classifier standing in for one is another probabilistic control with its own error rates, which means some rules genuinely cannot be made absolute at the output boundary and have to be handled by removing the capability or by keeping a human in the loop.

What the distinction does settle is which class of control a requirement needs. A requirement about shape can be met by a decoder where one exists and by validation where it does not. A requirement stated as "never" is a specification for a check outside the model, and stating it more firmly inside the model is a category error that no model on any provider will fix. Both of those are decisions available before the migration runs, and both are cheaper to make then than after a paper has been written.

The controls in this chapter and the last are all about the form of what the model emits and the constraints around it. A different class of portability problem starts one step earlier, in the part of the prompt that tells the model how to arrive at the answer — instructions that were tuned on a model which did no thinking of its own, and that now land on one which does.