Assistant prefill

Every request so far has ended with a user turn, and the model has produced the next entry in the conversation: an assistant turn replying to it. This chapter is about what happens when the array already ends with an assistant turn that is obviously unfinished — an opening brace, a label with nothing after it, half a sentence. On providers that support it, the model does not treat that entry as something to respond to. It carries on writing it, from exactly where the text stops, which means you get to choose the first tokens of the reply. Placing a partial assistant message at the end of the array is the whole of the technique. Prefilling the assistant turn steers output wherever the provider allows it, and does nothing at all where it does not.

That is a small mechanism with a large effect, because the opening of a generation constrains everything after it, and it is worth learning for two reasons even though newer facilities have overtaken its main use. It is frequently the fastest fix available when a model's output has the right content in the wrong wrapper. And it is a hard error on several current models, so code that depends on it needs to know which behaviour it is running against — including the middle case, where the entry is accepted, ignored, and silently does nothing. The word itself is overloaded: prefill elsewhere names the input-reading phase of generation, before any output token is produced, which is a different thing entirely. Here it means the opening of the reply that you write yourself.

Continuing versus replying#

The request is an ordinary body whose last entry is an assistant turn:

{
  "model": "some-model-that-supports-this",
  "max_tokens": 1024,
  "messages": [
    {"role": "user", "content": "Extract the invoice fields as JSON."},
    {"role": "assistant", "content": "{"}
  ]
}

Generation resumes from immediately after the {, so the very next token is produced in a context where an object has already been opened, and the natural continuation is a field name rather than a preamble. The preamble becomes not merely discouraged but implausible: Here is the extracted data is not something that follows an opening brace in any document the model was built from, so the scores for those tokens collapse. Nothing forbids them. The route to them has simply become one the sampler is very unlikely to take, which is a weaker statement than "cannot" and the one the rest of this chapter holds to.

Why that works at all is worth deriving rather than accepting, because once it is derived there is nothing left to memorise. The array is rendered into one flat token sequence with markers around each speaker's span. On an ordinary request the sequence ends with the user's span closed and an assistant span opened, and the model's job is to write tokens until it produces the marker that closes it. That is the whole task, on every request, prefill or not: continue a sequence that currently ends inside an open assistant span.

A prefill changes exactly one thing about that sequence. It puts some bytes between the assistant span's opening marker and the position the model writes from. The task is unchanged, the machinery is unchanged, and there is no "prefill mode" anywhere in the model — there is only a span that already has some text in it. Which is also why the feature needs no new field in the request and why it cannot be detected from the response: it is the ordinary mechanism with extra bytes in a place the format already allowed them.

The subtlety that catches people is what comes back. The response contains only the tokens the model generated, and the model did not generate the { — you did. A reply that reads as the complete object {"vendor": "Acme", "total": 41.20} arrives in content[0].text as "vendor": "Acme", "total": 41.20}, which fails to parse with an error about an unexpected character at position 0. The fix is to put back what you sent:

PREFILL = "{"

body["messages"].append({"role": "assistant", "content": PREFILL})
resp = post(body)
raw = resp["content"][0]["text"]
data = json.loads(PREFILL + raw)      # re-prepend before parsing

Two details bite here. The concatenation must be exact, including whitespace — a prefill of "{" and a prefill of "{\n " are different strings and the second one must be re-prepended in full. And a prefill with trailing whitespace is rejected by some providers' validators, so { (brace, space) may fail where { succeeds; keep the prefill tight and put any formatting inside the generated part.

The technique generalises beyond JSON. A prefill of Classification: forces a bare label instead of a paragraph of reasoning. A prefill of <result> opens an XML span the model will close. And when a long answer is cut off by max_tokens, prefilling with the tail of the truncated text asks the model to carry on from exactly where it stopped, rather than restarting and producing a different answer that has to be stitched.

Testing which behaviour you have#

The same body is a hard error elsewhere:

{
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "message": "messages: final assistant content cannot end with trailing whitespace"
  }
}

or, on models where the feature has been withdrawn entirely, a flat rejection of any array whose last entry is an assistant turn. This is not a difference between vendors so much as a difference between model generations: several current models that accept assistant turns anywhere else in the array return a 400 for one in final position. The reason is that a prefill puts words in the assistant's mouth, and a model whose safety behaviour is trained on what it has "already said" can be steered by a prefill that begins a sentence it would not have begun. Removing the affordance closes that vector, at the cost of a technique a great deal of production code depended on.

The word "mouth" is doing more work there than it looks, and the mechanism is the one from the first chapter of this topic. By the time the sequence reaches the model there is no field recording who wrote which bytes. Text inside the assistant markers is the assistant's prior output, full stop. The model is not choosing to trust it; those markers have no other meaning available to them. A refusal is a continuation like any other, and a model that has been trained to refuse is trained to refuse from the start of a turn; a turn whose first words already commit to complying is a much weaker place to refuse from. So a prefill is not fooling a check. It is supplying the evidence the check reads. That is a real vector, and it is why the affordance is being withdrawn rather than documented more carefully.

You cannot infer which behaviour you have from documentation alone, because it varies per model within one provider. Probe it, once, per model you deploy:

curl -s https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{
    "model": "'"$MODEL"'",
    "max_tokens": 32,
    "messages": [
      {"role": "user", "content": "Reply with a JSON object containing one key, ok, set to true."},
      {"role": "assistant", "content": "{"}
    ]
  }' | jq '{status: (.type), text: (.content[0].text? // .error.message)}'

Three outcomes, and each tells you what to build:

The middle row is the dangerous one, because nothing fails. A pipeline that prepends the prefill on a provider that ignored it produces {{"ok": true} — valid-looking, invalid JSON, and the bug reads as a model quality problem rather than a request-shape problem. Assert the probe result in a startup check rather than trusting a comment in the code.

The doubled brace is not a bug in the provider, and following the rendering explains why it is the exact symptom you get. A provider that does not implement continuation still has to render your array into a sequence, and the only rendering it knows for an assistant entry is a complete turn: opening marker, your text, closing marker. Your { therefore lands inside a turn that has been sealed. The model then finds itself at the start of a fresh assistant span with an empty slate, reads a previous turn consisting of a single brace, and does the sensible thing — writes the object it was asked for, from the beginning, brace included. Your code prepends the brace it sent, and the two braces meet. Every character of that outcome is correct behaviour by a component that was never told a prefill was intended, because nothing in the request says so.

The token accounting has a wrinkle worth knowing before it confuses a dashboard. The prefill is part of the array you send, so its tokens are billed as input, and the generated tokens that continue it are billed as output — which means usage.output_tokens undercounts the length of the object you end up parsing by exactly the size of the prefill. For a one-character { that is noise. For a prefill like <analysis>\n <summary> it is enough to make a length-based budget wrong in the unsafe direction, because the reply you assemble is longer than the number the response reported. If you size max_tokens against an expected output length, size it against the generated part and add the prefill separately.

The same constraint expressed in three roles#

A prefill is the third place you can put a constraint, after the system field that carries an operator's standing instructions and the user turn itself. Running the same request three ways is the fastest way to feel how differently the channels behave. Ask for one sentence of French with the constraint stated each way:

REQ = {"model": MODEL, "max_tokens": 200}
Q = "Describe the Douro valley."

variants = {
  "system":     {**REQ, "system": "Always answer in French.",
                 "messages": [{"role": "user", "content": Q}]},
  "user":       {**REQ,
                 "messages": [{"role": "user", "content": f"Always answer in French.\n\n{Q}"}]},
  "prefill":    {**REQ,
                 "messages": [{"role": "user", "content": Q},
                              {"role": "assistant", "content": "La vallée du Douro"}]},
}

All three usually produce French. They fail differently, and the failure modes are the point of the comparison. The system variant holds across a long conversation and resists a later user turn asking for English. The user variant holds for a turn or two and then drifts, because it is one instruction competing with everything the user says afterwards. The prefill variant is the most reliable for exactly one turn and has no influence at all on the next one — it constrains the first tokens of this generation and nothing else. If the model would have said something contradictory, the prefill does not prevent it; it merely forces a French-sounding start, after which the generation can wander.

That is the honest limitation of prefill in general. It shapes the beginning of an output, and the beginning of an output has outsized influence over the rest, which is why it works as well as it does. It is not a guarantee about the whole output. A prefilled { does not stop the model from emitting malformed JSON in the middle, or from running past max_tokens and truncating the object, and code that treats a prefill as a schema is trusting a nudge.

Constrained decoding, and what it replaces#

The reason several current models can withdraw prefill without leaving a hole is that the strongest use case now has a first-class mechanism. Structured output constraints let you attach a JSON Schema to the request and have the decoder itself restricted to tokens that keep the output valid against it:

resp = client.messages.create(
    model="claude-opus-5",
    max_tokens=1024,
    output_config={"format": {"type": "json_schema", "schema": INVOICE_SCHEMA}},
    messages=[{"role": "user", "content": f"Extract the invoice fields.\n\n{invoice_text}"}],
)

This is a different kind of thing from a prefill. A prefill makes an unwanted output improbable; a constrained decoder makes it unrepresentable, because a token that would break the schema is not among the tokens that can be sampled. There is no preamble to strip, nothing to re-prepend, and no 4% tail — and there is no is_error path where the model politely explains that the invoice was unreadable, because that explanation is also not in the grammar, which is a trade to make deliberately.

The step from "improbable" to "unrepresentable" happens outside the model, in the same picking code that max_tokens lives in, and it is small enough to state exactly. The decoder keeps track of where it is in the schema — inside an object, having just written a key, expecting a colon. At each step it works out which of the model's candidate tokens would keep the document parseable from that position, and the scores of all the others are driven to a value that cannot be sampled before the draw is made. The model still ranks every candidate, including Here; the sampler is no longer permitted to choose it. So the guarantee is not that the model became more obedient. It is that a whole class of outputs was deleted from the ballot after the model had voted.

Which fixes exactly one problem and it is worth being precise about which, because the sentence a reader would confidently write here — "with a schema attached, the fields come back correct" — is false. A schema constrains the shape of the reply and says nothing about whether the values in it are right. A required total field will be filled, on every invoice, including the one whose total was smudged; the decoder's job is to ensure a number appears there, and a number will. The failure mode has moved rather than gone: before, a malformed reply was loud and your parser caught it; now, a confidently wrong value is well formed and passes every check you have. That is a good trade for most extraction work and a bad one wherever "I could not read this" is an answer you needed, which is why the grammar should have somewhere to say so.

The other prefill uses have replacements too, and they are worth knowing as a table rather than discovering one at a time when a model upgrade breaks a service:

The first four are strictly better than what they replace. The fifth deserves a note: prefill was widely used to talk models out of over-cautious refusals, and current models refuse far less indiscriminately, so the need has largely evaporated. Where it has not, reaching for a prefill is treating a symptom.

Removing a prefill is rarely a one-line change, and budgeting for that is the difference between an afternoon and a week. A codebase that relied on one has usually grown a small apparatus around it: a stop sequence to cut the model off at the closing brace, a regex or a brace-counter to find the object inside whatever came back, a retry loop that catches the parse error and tries again, a metric counting how often that loop fires, and a set of tests asserting the exact shape of the request body including the trailing assistant entry. All of it exists to compensate for the model being free to write prose, and all of it becomes dead weight once the decoder is constrained. Delete it in one change rather than leaving the retry loop in place "just in case" — a retry path that can no longer fire is a path nobody maintains and everybody trusts.

A prefill, then, is a lever on the first few tokens of a reply, exercised by writing them yourself and letting the model continue. It costs one array entry, it is billed as input, it must be re-prepended before parsing, and its availability is a per-model fact you establish with a probe rather than a per-vendor fact you read in a document. Where a constrained decoder is available, use that instead; where it is not, a prefill remains the shortest distance between a model that answers correctly and a program that can read the answer.

Between the system channel, the user turns, and a prefilled assistant turn, you can now shape both what the model is told and how its reply begins. All of it is still text: three roles, one ordered list, everything the model reads arriving as a string somebody wrote. The array has a fourth shape in it — one where an entry is not prose at all but a structured request from the model, answered by another entry bound to it by an identifier the provider issued and validates. That binding is what makes certain cuts in a conversation unsafe, and it is next.