Truncation under constrained decoding

Turning strict schemas on removes fenced objects, preambles, bare keys and unbalanced braces, and it leaves one parse error standing. When the output runs into the request's max_tokens ceiling, the response stops wherever the counter ran out, which is usually in the middle of a string:

{"ticket_id":"91443","urgency":"high","assignee":null,"summary":"Customer reports that the nightly export to their warehouse has been failing since the 14th; three separate runs

Every character of that is legal JSON so far. The keys are quoted, the enum member is one of the three, the null is a real null, and the string that stopped mid-word was on its way to being closed. The response is a valid prefix of a valid document, which is precisely what a grammar-constrained decoder guarantees and precisely what a parser rejects. This chapter is about why that failure is structurally immune to the mechanism that removed the others, how to read it off the envelope rather than the exception, and why it appears without anybody deploying anything.

A prefix of valid JSON is not valid JSON#

The grammar tells the decoder which tokens may come next given everything emitted so far. It does not tell the decoder to keep going until the document is complete, because it has no authority over when generation stops — that is decided by the model emitting an end-of-sequence token, by a stop sequence — a string you nominate that halts generation the moment it appears — or by the request hitting its max_tokens ceiling and being cut off server-side. The first of those three happens at an accepting state of the grammar. The third happens wherever the counter runs out.

Only Done is an accepting state — a position at which the document is complete and generation may legally stop — and only a response that reaches it parses. Every failure class that went to zero under a strict schema went to zero because those outputs required an illegal transition; truncation requires no illegal transition at all. A team that has just watched malformed-grammar errors disappear will read a truncation as the same bug coming back, and will go looking at the schema, which is the wrong place.

The asymmetry behind that is worth stating as a rule, because it predicts the next case as well as this one. A grammar is a set of permissions about transitions: from where the output currently sits, these tokens may follow. Nothing in that form can express an obligation to continue, because continuing is not a transition — it is a decision about whether to take one at all. That decision is made outside the grammar by three parties, none of which consults it. The model may emit the end-of-sequence token. The client may have nominated a stop sequence. The server is counting tokens against max_tokens and will cut the stream mid-word without asking anything about the parser state. Because the cut is made by a counter that has never seen the schema, an unfinished object is not a schema violation, and no amount of tightening the schema prevents one.

A parser is the other kind of machine, and the difference is where each one is checked. The decoder is checked at every step and never at the end. The parser reads the whole string and asks its question only once the input has run out: am I in an accepting state now. Legal transitions all the way through are necessary for that and not sufficient for it. The gap between those two checks is the entire hole the failure lives in, and a truncated response is the one document that passes the first and fails the second.

The diagnostic is in the response envelope rather than in the body, and it is unambiguous:

resp = client.chat.completions.create(..., max_tokens=512, response_format=STRICT)

choice = resp.choices[0]
if choice.finish_reason == "length":
    log.warning("truncated: completion_tokens=%s ceiling=%s",
                resp.usage.completion_tokens, 512)
    raise TruncatedOutput(choice.message.content)   # do not attempt to parse

obj = Ticket.model_validate_json(choice.message.content)

finish_reason == "length" and usage.completion_tokens sitting exactly on the ceiling are the two facts that end the investigation. The reason this check is worth writing before you need it is that without it the failure arrives as a JSONDecodeError from model_validate_json, which is the same exception the service used to throw for fenced output and preambles, and it will be triaged as the same problem.

Where the failure rate actually comes from#

A truncation rate is not a property of the model or the schema. It is the tail mass of the output-length distribution that lies above the ceiling, which means it moves whenever either of those two things moves and stays put when neither does.

Completion length for a call of this kind is roughly log-normal — most responses cluster short, with a thin tail of much longer ones stretching to the right. Here is one such distribution before and after a shift in the inputs feeding it, on a service whose median completion was around 180 tokens:

The ceiling was 512. Before the change, 512 sat just above p99, so the truncation rate was a few calls in ten thousand and nobody noticed. After, 512 sits between p95 and p98, and the share of the distribution above the ceiling is a little over 2%. A modest shift in the median moves that mass a lot, because the tail is where the ceiling is.

Two things in that table are worth reading off rather than glancing at. The first is the shape of one column. In the before column the median response is 178 tokens and the slowest one per cent run to 497, so the longest responses are getting on for three times the length of a typical one — the percentiles bunch together at the short end and stretch apart at the long end, which is what a right-skewed distribution looks like when it is written out as a list. The second is what happened between the columns. Every percentile rose by roughly the same proportion, about a third, rather than the top one or two moving while the rest held still. That is the signature of a whole distribution being scaled, which is what a longer input does to a summary written in proportion to it, and it is a different diagnosis from a handful of pathological documents having entered the traffic.

Put those two together and the behaviour of a fixed ceiling follows. max_tokens is a count that scales with nothing. The distribution scaled by roughly a third, and because the scaling is proportional the parts already furthest out moved furthest in absolute terms — the median gained about fifty tokens while the ninety-ninth percentile gained more than two hundred. The ceiling sits out in the region where the large absolute movements happen, so it is overtaken by a shift that sounds trivial when it is quoted as a change in the median. A truncation rate is therefore not a property of the model and not a property of the schema. It is one minus the percentile at which your ceiling happens to sit, and that percentile is being moved by whatever is moving your inputs.

This is why the first question is the token ceiling and the second is how long an output the schema is actually asking for. A schema is a bill of materials for the response. Six scalar fields with short values ask for perhaps 60 tokens; the same six fields with one unbounded free-text field ask for however much the model wants to write; and an unbounded array of objects asks for a length proportional to the input, which means the tail of your output distribution is the tail of your input distribution with a multiplier on it. Long field names cost too — estimated_resolution_timestamp is five or six tokens spent on every response, and the keys are emitted by the decoder like any other token.

That estimate is one anyone can redo for their own schema, and redoing it is more useful than trusting the figure. Every field costs its key in tokens, plus the punctuation wrapped around it — two quotes, a colon, a comma — plus whatever the value takes. A short key is a token or two and a short scalar value another one or two, so a flat six-field object works out in the region of ten tokens a field once the braces are counted, which is where the estimate above comes from. The estimate stops being computable the moment a field's value is not bounded by the schema, because the cost of that field is then a property of the input rather than of the declaration. That is the line worth watching in a schema review: every bounded field adds a known number to the bill, and every unbounded one replaces the bill with a distribution.

What to do about it, and what it costs#

The remedies pull in different directions, and which one is right depends on whether the long outputs are legitimate.

Raising the ceiling is usually the right first move and it is not a fix on its own, because a ceiling set above today's p99 is a ceiling that will be hit again the next time inputs shift. What makes it durable is an alert on the distribution rather than on the failures: track completion_tokens as a histogram, alert when p99 exceeds 70% of the ceiling, and you get warned days before the parse errors start.

Two responses to truncation are actively harmful and both are tempting. The first is retrying the identical request. The output was cut because it was long, the input has not changed, and the length distribution for that input is roughly where it was, so a retry truncates again with high probability — on the forwarded threads below, the retry success rate was 12%, and the service was paying for two long completions to get one failure. Retry with a raised ceiling and it works; retry blind and you have doubled the cost of a known-bad call.

The twelve per cent deserves an explanation rather than being filed as noise, because it is the reason a blind retry survives review. One input does not have one output length; it has a distribution of them, since decoding samples rather than computing a fixed answer. A document whose lengths straddle the ceiling will therefore land under it now and then, and a recovery of roughly one attempt in eight is what that looks like on a graph. The retry was doing something real. What it was doing was buying a second draw from a distribution that mostly sits above the line, at the price of a full long completion every time, so the honest reading of a low double-digit recovery rate is not "it sometimes works" but "it is a weighted coin, and every flip is charged at full rate".

The second of the two harmful responses is repairing the JSON by appending closing tokens. There are libraries that will take the fragment above, close the string, close the object, and hand you a parseable document — and what you get is "summary": "Customer reports that the nightly export to their warehouse has been failing since the 14th; three separate runs", a sentence that ends mid-clause, plus whatever fields came after summary in the schema filled with nulls or omitted entirely. It parses. It validates, if your model is permissive. It is a truncated fact presented as a complete one, and it will sit in your database indistinguishable from a good extraction. If you do close fragments, mark the record, and never do it for fields that feed a decision.

The pull towards closing the fragment is strong, and it comes from an observation that is true: nearly all of the information is already there. The keys are right, the enum member is right, and the summary is a sentence that stops one clause short of ending. Discarding that to fetch it again feels like waste, and a library that closes the braces for you presents itself as a recovery rather than as a fabrication. What the observation misses is that the repair changes the record rather than the string. Before it, the failure is loud, typed and countable — an exception, a dead-letter row, a number on a graph. After it, the failure is a well-formed object no consumer can tell apart from a real extraction, in the same table as the real ones, with the only surviving evidence being a token count nobody kept. A repair that turns a caught error into an uncaught one has not recovered anything. It has moved the error to a place with no instrumentation on it.

Truncation, schema violations and stop sequences#

Truncation is not a schema violation, and conflating them makes the metrics useless. A schema violation says the contract and the output disagree; a truncation says the output never finished, and the contract is not implicated. Keep them as separate counters with separate alerts, because their causes and their fixes have nothing in common — one points at the model or the provider, the other points at a number in your request builder.

It is also not the same as a stop-sequence collision, which produces an identically-shaped symptom from a different cause. If you have configured a stop sequence that can occur inside a string value — "}", "\n\n", and "###" are all real examples from production code — generation halts mid-document with finish_reason reported as stop rather than length, and the token count sits nowhere near the ceiling. That distinction in the envelope is the whole diagnosis, which is another reason to log finish_reason alongside the failure rather than only the exception.

The reason a stop sequence can fire inside a string at all is that it is matched outside the grammar. The provider scans the emitted text for the literal you nominated and halts the moment it matches, and the machinery doing the scanning has no idea whether the position it matched is inside a string value, inside a key, or between two fields. So "\n\n" halts on a summary that contains a blank line and "}" halts on the first closing brace of a nested object rather than the last. Every transition the decoder took was legal, the model had not finished, and something that was never parsing JSON ended the generation on its behalf. A client attempting a parse on what comes back fails in exactly the same way it does on a truncation, which is why the envelope rather than the exception is where the two cases separate.

Nor is truncation evidence that constrained decoding is unreliable. A prompt-only branch and a strict branch truncate at roughly the same rate, because the ceiling applies to both and neither the prompt nor the grammar has any influence over it. What the strict branch changes is that truncation becomes the only remaining parse failure, which makes it far easier to see; a 2% truncation rate hiding inside a 4% general parse-failure rate is a needle, and a 2% truncation rate on a service that has been at 0% for eleven weeks is a wall.

The rule to carry out of this is that the ceiling is a parameter of your request and the output length is a property of your schema and your inputs, and a truncation rate is what happens where those two meet. Fixing it means changing one of the three, and the envelope tells you which — which is a much shorter investigation than the one that starts from a JSONDecodeError and goes looking at the schema.

Every failure in this chapter announces itself. The response does not parse, an exception is raised, a counter increments, and somebody gets paged. The remaining class does none of that: the object is complete, the grammar was satisfied, the validator passes, and the value is wrong in a way that only shows up in the business metric weeks later. The schema itself turns out to be one of the causes.