Assistant prefill and forced formats

The Messages API takes an array of turns, and a request normally ends with a user turn, leaving the model to begin the assistant turn from nothing. A prefill ends the request with an assistant turn that is already partly written — text the caller supplies as the opening of the model's own reply — and the model continues from where that content stops, exactly as it would continue its own generation. Everything in this topic so far has been about moving a constraint closer to the point of generation; the prefill is the limit of that move, sitting at position zero of the output rather than anywhere in the prompt.

The reason it works is worth stating before the hazards. A final-line instruction changes the probability distribution over the first token, and the model is still free to sample Here; at 400,000 attempts a day it will. The prefill removes the choice, because there is no first token to sample — generation begins from a state in which { has already been written, and the tokens most likely to follow an opening brace in an assistant turn are a quoted key.

The step under that is what the turn array becomes before the model sees it. The turns are rendered into a single sequence, and the model's only operation is to continue a sequence. When the request ends with a user turn, the first token the model produces is the first token of the reply, and every instruction about that token is a request competing with everything else in the window. When the request ends with an assistant turn carrying {, that brace is part of the sequence being continued rather than part of the output being produced. The first token the model generates is the one after the brace, and no distribution over the brace's position was ever consulted, because that position is not a decision the model is making.

The visible half of the failure it prevents is worth naming, since the string is the same one every time. Under the instruction-only layout the model would sometimes open with Here is the extracted data: and then produce a perfectly correct object underneath it. Nothing about the object is wrong. What fails is the parse, because a JSON document is a single value and a parser handed a document beginning with a word rather than a brace rejects it before reaching anything useful. The retry path exists to absorb exactly that, at the cost of a second call on every occurrence. The measurements below come from an invoice extraction pipeline processing about 400,000 scanned pages a day, scored on a frozen 5,000-page set with the replication discipline from chapter 2.

Writing the first tokens yourself#

resp = client.messages.create(
    model="claude-haiku-4-5",
    max_tokens=1024,
    system=EXTRACTION_RULES,
    messages=[
        {"role": "user", "content": f"{page_text}\n\nExtract the invoice fields."},
        {"role": "assistant", "content": "{"},      # no trailing whitespace
    ],
)
obj = json.loads("{" + resp.content[0].text)        # the prefix is NOT echoed back

Two mechanical details in four lines. The prefix is not returned in the response — resp.content[0].text begins with whatever the model generated after the brace — so the caller has to concatenate it back, and forgetting to do so produces a parse error that looks like a model failure and is not. And the prefill string cannot end in trailing whitespace; a request whose final assistant content is { is rejected, which is a surprisingly common way to lose an afternoon when the prefill is built by string interpolation.

On this pipeline the change took the parse rate from 97.8% to 99.7%. That is the honest headline and it is a good result: retries fell from 8,800 a day to about 1,200, and the p99 came down with them.

Those two retry counts are the parse rates read against the volume, and doing the multiplication is what turns a decimal place into a decision. A parse rate of 97.8% means a failure rate a little over two in a hundred, and two in a hundred of 400,000 pages is the 8,800. A parse rate of 99.7% means three failures in a thousand, and three in a thousand of the same volume is the 1,200. The p99 moves with them because a retry is a whole second call appended to a request that had already finished once, so the slowest one page in a hundred is mostly made of pages that were retried at all.

The failure that got quieter, not rarer#

About a tenth of the original failures were not preamble or code fences at all. They were prose explaining that the page appeared to be a fax cover sheet rather than an invoice. Under the instruction-only layout those responses failed to parse, hit the retry path, failed again, and landed in a dead-letter queue that a human reviewed each morning — about 900 pages a day, most of them genuinely not invoices, a handful of them badly scanned invoices worth rescanning.

Under the prefill, they parse. The model, having already committed to writing an object, writes one:

{"invoice_number": "N/A", "invoice_date": null, "vendor": "Unknown",
 "line_items": [], "total": 0.0, "currency": "USD"}

That object is schema-valid. It passes every downstream check, enters the ledger as a zero-total invoice from an unknown vendor, and is indistinguishable from a real invoice for zero dollars. The dead-letter queue empties out over the following week and a team can easily record that as a second benefit of the change. It is not a benefit; the 900 daily failures have become 900 daily silent bad rows, and the class of answer that would have caught them — "this page does not contain an invoice" — has been made unreachable by the shape of the first token.

Where the queue's volume comes from is the same multiplication run over a subset: about a tenth of the 8,800 daily parse failures were of that kind. It is worth holding those two numbers next to each other, because the change did not touch that tenth in any way that helped. The failures the prefill removed were preambles and code fences, which are failures of syntax — a correct answer in the wrong wrapper. The fax cover sheets were never a syntax problem. The model was answering the question it had been asked, correctly, in prose, and the only reason that answer failed to parse is that prose is not JSON. Forcing the syntax made those answers parse without making any of them true.

This is the hazard that matters most, and it generalises past JSON. A prefill commits the response's shape before the model has weighed the input, so any input for which the committed shape is wrong produces a confidently wrong response instead of a visibly odd one. It is the same trade as any strict output format, made at the strongest possible position and therefore made most completely.

"Before the model has weighed the input" is a loose way of saying something worth being exact about, because the page is in the request and the model has read it. The commitment is not the model's. It is the caller's, made once when the request builder was written, for every page that would ever arrive. That is what makes it stronger than any instruction and it is also what makes it blind. An instruction is evidence the model weighs against the page in front of it; a prefill is a fact about the sequence that no page can outweigh. Stated as a general rule: the stronger the mechanism forcing a format, the less the input is permitted to say about whether that format fits.

Fixing it does not mean abandoning the prefill. It means making the abstention reachable inside the committed shape: add a required document_type field with an enum that includes not_an_invoice, and a confidence field, and let the schema express the answer the model needs to be able to give. The general form is that a forced shape must be wide enough to contain every answer that is correct, and enumerating those answers is design work that the instruction-only version let you skip — badly, by routing the overflow into a queue.

Everything after the brace is ordinary generation#

The word that slips into conversations about prefill is "guaranteed", and it is wrong in a specific way. The prefill constrains the first tokens; from the second token onward, the model is sampling as it always does. A valid opening does not imply a valid object, which is why the remaining 0.3% exists at all. Sorted by cause, that residue was: about 60% truncation, where a page with 40 line items ran past max_tokens and produced a syntactically incomplete object; about 30% a complete object followed by a sentence of commentary, which is invalid JSON as a whole document; and about 10% a valid object with a hallucinated field name not in the schema, which parses and then fails validation downstream.

None of those are fixed by a longer prefix. A prefill of {"invoice_number": " narrows the second field too and creates a new problem, since it asserts that the invoice number is the first thing to extract and that it is a string, on every page, before the model has read any of them. On documents whose schema is a discriminated union — one of several possible shapes, selected by the value of a tag field — the assertion is worse: prefilling {"type": "invoice" selects the branch, and the classification the model was supposed to perform has been performed by the caller.

Three ways to make a format happen#

The support row is the one that decides the question on current models, and it is a genuine deprecation rather than a caveat. A request whose final turn is an assistant prefill returns a 400 on Claude Opus 4.6, 4.7 and 4.8, Claude Opus 5, Claude Sonnet 4.6 and 5, and Claude Fable 5. Assistant turns elsewhere in the array are unaffected — few-shot examples still work — it is specifically the trailing partial turn that is gone. Prefill remains available on models from the 4.5 generation and earlier, Claude Haiku 4.5 among them, which is why the invoice pipeline above could run it at all.

The replacement is the third column, and it is a better tool for this job rather than a consolation prize. output_config.format with a JSON Schema — the machine-readable description of which fields an object must carry and of what type — constrains the entire generation instead of its first tokens, so the truncation-and-commentary residue disappears as a category:

resp = client.messages.parse(
    model="claude-opus-5",
    max_tokens=2048,
    system=EXTRACTION_RULES,
    messages=[{"role": "user", "content": f"{page_text}\n\nExtract the invoice fields."}],
    output_config={"format": {"type": "json_schema", "schema": INVOICE_SCHEMA}},
)
invoice = resp.parsed_output          # None if the model refused; check before using

That last claim needs the qualification the comparison table already carries, and the two should be read together. Constrained decoding does remove the commentary case outright: a token that would begin a sentence after the closing brace is not a token the decoder is allowed to emit, so a complete object followed by a remark cannot be produced at all. The hallucinated field name goes the same way, since a key absent from the schema is not decodable either. Truncation does not go away, because a schema governs which tokens are legal and not how many are needed, and a page with forty line items still runs into max_tokens. What changes for truncation is the shape of the failure rather than its existence. Under a prefill it arrives as a syntactically incomplete object that dies at the parser and looks like every other parse error; under constrained decoding it arrives as a stop_reason of max_tokens, which names its own cause in the response.

The schema still has to contain not_an_invoice. Constraining decoding removes the syntax problem and leaves the semantic one exactly where it was: a schema with no way to express "this input does not fit" will produce a well-formed misrepresentation on every input that does not fit, whichever of the three mechanisms put it there.