Chat templates and the rendered request

A chat template is a formatting rule shipped alongside a model's tokenizer that turns a list of role-tagged messages into the one string of token ids the model is fed. It sits between the object your code builds and the input the weights consume, and it is where a family of single-trace failures lives: instructions that appear to be ignored, responses that run past the length the prompt asked for, and a quality gap between two code paths calling the same model.

This chapter is about establishing, from a frozen trace, what the model received at that layer. The check belongs early in any investigation because it is cheap, its result is binary, and every conclusion drawn from a trace whose rendering was wrong is a conclusion about the wrong document.

Instruction tuning teaches a string format, not an API#

There is a habit that transfers directly from ordinary backend work. When an ORM query returns rows nobody expects, the first move is to log the generated SQL. The ORM's object graph is a convenience for the person writing the query, and the database only ever receives a string, so the object graph is evidence about intent while the string is evidence about what happened. Chat models have the same two layers. The messages array is a convenience provided by a client library; the model sees a sequence of token ids, and the chat template is the mapping between the two.

The belief that gets in the way here is small and almost never said out loud. Writing {"role": "system", "content": ...} feels like addressing a channel: the content travels one way, the role travels beside it as metadata, and something at the far end reads the role to decide how to treat what arrived. No such channel exists at the point where the weights run, because the weights take one flat sequence of ids and nothing else. The role value is a rendering instruction — a string that selects which marker tokens get written around that message's content when the list is flattened — and once the flattening has happened, the role survives only as those markers.

The flattening happens in a place you can name for any given request, which is what makes this checkable rather than a matter of opinion. Against a hosted chat API it happens inside the provider, before the ids reach the model, and you never see the result. Against vLLM's /v1/chat/completions it happens in the server, which reads the chat_template shipped in the model's tokenizer_config.json and applies it to whatever list you posted. In a pipeline you built yourself it happens wherever your code calls apply_chat_template — or it does not happen at all, and the completions endpoint is the one place that accepts a string, asks no questions about its shape, and reports nothing.

That template is not decoration. During instruction tuning, the model was shown hundreds of thousands of examples in one specific format, with reserved tokens marking role boundaries, and it learned that text appearing between the system header and the first end-of-turn marker is the standing instruction, that text after the user header is the thing to respond to, and that generation should stop at the end-of-turn token.

What makes those boundaries learnable is that the markers are not text. A model scores over a fixed inventory of fragments — its vocabulary — and a handful of entries in that inventory are reserved for structure, meaning that no sequence of characters anybody can type will ever tokenise to them. An id that can only arrive by having been deliberately placed there is an unambiguous signal, and an unambiguous signal is the kind of thing a training run can attach a behaviour to. Every system instruction in the tuning data sat between the same two ids, so what the model learned — treat what follows this as the standing instruction — is attached to those ids rather than to any English word. Here is a rendered request that skipped them, taken from a frozen trace of a self-hosted Llama-3.1-8B-Instruct behind a completion endpoint:

System: You are the internal documentation assistant. Answer in the format
the user requests. Do not volunteer information beyond what was asked.

User: Summarise the on-call rotation policy in two bullets.

Assistant:

Nothing in that string is a system prompt. It is 61 tokens of ordinary English, and the word System at the start carries exactly as much authority as the word Yesterday would. System and : are ordinary entries in the inventory, the same ids that turn up in a pasted chat log, a quoted email or a screenplay, and nothing distinguishes this occurrence from those because there is nothing left to distinguish it with. The model was given a document that happens to contain a colon-delimited transcript and asked to continue it, and continuing a transcript by writing more of the transcript is a correct completion of that document.

The request therefore arrives as one undifferentiated turn. There is no system prompt inside it for the model to obey and none for it to ignore, which makes the sentence everybody writes on the ticket — the model ignored the system prompt — a description of an object that was not in the request. For Llama 3.1 the correct rendering of the same messages is this:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are the internal documentation assistant. Answer in the format the user
requests. Do not volunteer information beyond what was asked.<|eot_id|><|start_header_id|>user<|end_header_id|>

Summarise the on-call rotation policy in two bullets.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

The tokens in angle brackets are single ids in the vocabulary, reserved and never produced by ordinary text. <|start_header_id|> is one token, not seven characters. Their absence explains two symptoms at once: the instruction was never marked as an instruction, and the model was never given the boundary at which it has learned to emit <|eot_id|>, so it keeps going until the max-token cap or a stop string catches it.

The second of those is worth following one step further, because it explains the particular shape the bad output takes. An assistant turn in the tuning data always ended at <|eot_id|>, so emitting that token is behaviour attached to being inside a marked assistant turn. The document in front of the model here contains no marked turn, and therefore contains no turn that could end. What it does contain is a transcript that has run out, and the most plausible continuation of a transcript is more transcript — another question, another answer — which is why an output of this kind so often contains a question the user never asked, answered at length.

The concatenation is tempting for reasons that are not stupid. The completions endpoint is simpler, it takes a plain string, it makes prefix caching and few-shot construction obvious, and it is what the same team used two years ago against base models where the transcript format genuinely was the interface. Base models have no template, so writing System: and User: was a reasonable convention for steering one, and code written in that era keeps working in the sense that it returns text. What changed underneath is that the weights being served are now tuned on a format the code does not produce. The failure is graded rather than binary — most questions still get answered, because the prose is still coherent instructions in English — and a graded failure is one that survives a staging rollout and surfaces as a single strange trace weeks later.

Both branches start from the same well-formed messages array, and they have already diverged by the second box. Everything downstream of that fork is the same model with the same weights, which is why nothing in the response, and nothing in an application log that records the array, can tell you which branch the request took.

Diff the ids, not the strings#

The check that settles it takes about fifteen lines, and it runs against the frozen trace rather than against a fresh request. An escalated trace is usually days old before anybody opens it — the record frozen in chapter 1 was a Saturday conversation reported the following Tuesday — and in the interval the prompt template, the nightly-rebuilt index and the resolved model snapshot have all had the opportunity to move. Re-rendering today's messages through today's template answers a question about today. The point is to establish what happened on Saturday and not what happens now.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")

expected = tok.apply_chat_template(
    trace["messages"], add_generation_prompt=True, tokenize=True
)
actual = tok(trace["rendered_request"], add_special_tokens=False).input_ids


def first_divergence(a: list[int], b: list[int]) -> int | None:
    for i, (x, y) in enumerate(zip(a, b)):
        if x != y:
            return i
    return None if len(a) == len(b) else min(len(a), len(b))


i = first_divergence(expected, actual)
print(f"expected {len(expected)} ids, sent {len(actual)} ids, diverge at {i}")
print("expected:", tok.convert_ids_to_tokens(expected[i : i + 6]))
print("sent    :", tok.convert_ids_to_tokens(actual[i : i + 6]))

Three arguments in that snippet carry the whole method. add_generation_prompt=True appends the trailing assistant header, which is the empty header block at the foot of the correct rendering above and the thing that tells the model the next turn is its own; without it, expected would be short by exactly that scaffolding and the diff would blame the tail of the sequence for the wrong reason. tokenize=True asks for ids rather than for the rendered string, which is the comparison being made rather than an optimisation. And add_special_tokens=False on the actual side matters more than it looks: it tells the tokenizer not to prepend a beginning-of-text token of its own to trace["rendered_request"]. Left at its default, the tokenizer would insert the very token whose absence is under test, position 0 would match, and the diff would come back clean on a request that never carried one.

first_divergence walks the two id lists in step and returns the first index at which they disagree. zip stops at the shorter list, so when one sequence is a prefix of the other the loop never finds a disagreement at all, and the function falls through to min(len(a), len(b)) — the position at which the shorter one ran out. That case reads differently from a mismatch and is worth holding apart from one. The two sequences agree for the whole of their shared length and then one of them simply stops, which is how a missing generation prompt at the tail, or a block of scaffolding present on one side only, presents from here.

On the trace above it prints a divergence at index 0, with ['<|begin_of_text|>', '<|start_header_id|>', 'system', '<|end_header_id|>', 'ĊĊ', 'You'] against ['System', ':', 'ĠYou', 'Ġare', 'Ġthe', 'Ġinternal']. The debug view spells whitespace out rather than printing it: Ċ is a newline and Ġ is a leading space, so ĊĊ is the blank line the template writes after the system header and ĠYou is the word You with the space in front of it folded into the same entry. Position 0 with a missing <|begin_of_text|> is the unambiguous version of this bug and it takes one run to find.

The index is doing more work than a yes or no would. Everything before it matched id for id, so the number is a bound on the fault: whatever went wrong went wrong at or after that position, and nothing before it is worth looking at. Index 0 is the strongest reading the check can return, because the first id of a correctly rendered Llama 3.1 request is <|begin_of_text|> whatever the messages happen to say. It is the one position whose expected value can be predicted without knowing anything at all about the content, and a mismatch there therefore cannot be explained away by a difference in the content, since no message body has been reached yet. The scaffolding is missing from the first token onward, which is the same statement as no template having run.

A divergence deep in the sequence carries the opposite news and is a good deal harder to act on. If several hundred ids matched before the split, the header tokens were among them, so a template did run and the envelope is well formed; what differs from that point is content — a retrieved passage re-fetched from an index rebuilt overnight, a date rendered at request time, a trailing newline in one field. None of that is a rendering bug, and the six tokens printed either side are what tell you which kind of difference you are holding. Depth is therefore a rough classifier in its own right: a divergence at zero indicts the wire format, and a divergence deep in the sequence indicts the contents. The third outcome is the clean one, where first_divergence returns None, there is nothing to print, and rendering has been eliminated as a suspect for this trace.

Comparing token ids rather than decoded strings is deliberate, and the reason is that equality of the strings is neither necessary nor sufficient for the property being tested. It is not sufficient because detokenised output frequently renders special tokens either invisibly or as their literal text. Two strings that print identically in a terminal can therefore be different sequences: the literal characters <|eot_id|> sitting inside a retrieved passage and the single reserved id are indistinguishable on screen and behave nothing alike. That difference is precisely the part you are investigating, and a string comparison is the one instrument guaranteed to hide it. It is not necessary because two strings can differ in whitespace that the tokenizer normalises away, in which case the ids are identical and the string diff has raised a difference the model was never in a position to see. Only one of the two artifacts was consumed by the weights, and it is the one whose equality means what you want it to mean.

Ids also make the length comparison meaningful: a template that adds 24 tokens of scaffolding to a 5,000-token prompt is a 0.5% difference in the string and a categorical difference in behaviour. Read as a fraction of the document that scaffolding rounds to nothing, and read as structure it is the entire difference between an instruction and a sentence.

Where the same class of bug hides on a hosted endpoint#

Being on a hosted chat endpoint removes the crude version of this and leaves several quieter ones. The rendering still happens; you have just delegated it, which means the question shifts from "was a template applied" to "which template, over what, and did anything I control get folded into the wrong place."

Even where the provider does all the rendering, the mapping from your fields to the final string is a product decision that changes between models and between API versions. Multiple system messages may be concatenated or may be rejected. A developer role may exist and outrank system, or may be an alias for it. Tool definitions are rendered into a block whose position relative to your system prompt is chosen by the provider, so an instruction that says "never call a tool without confirming" can end up textually before or after the schemas it refers to. Where the provider exposes the fully rendered prompt or an input-token breakdown, reading it once for your own request shape is cheap and occasionally surprising.

The id diff itself is not available in that setting, since the ids are produced on the far side of an HTTP boundary and never returned. What is often available is a count. Where the provider publishes a tokenizer for the family or a token-counting endpoint, tokenise your message contents yourself and compare the total against the input-token count in the response; the gap is the scaffolding the provider added, and its rough size tells you whether a system block and a tool section were rendered at all. That is a presence check rather than a correctness check, and it will not tell you where your instruction landed relative to the tool schemas. Where a rendered-prompt debug view exists, that document settles in one reading what a count can only bound.

Read the envelope before the prose#

The same discipline resolves a failure that looks nothing like a templating problem. A trace where the answer trails off into a partial sentence, or ends after two of the three requested bullets, gets read as the model being careless with instructions, and the team goes off to rewrite the formatting instruction. Topic 1.8 established why the text itself will not tell you: a length-capped completion is fluent right up to its last token. The operational rule that follows for single-trace debugging is that the envelope gets read before the content, every time.

The mechanism under that rule is what makes it necessary rather than merely tidy. Generation ends in one of two places. Either the model emits its end-of-turn token, which is a decision inside the sequence and visible in it, or the serving loop counts tokens and stops when the cap is reached. The second decision is taken outside the model and after the last token has already been produced, so there is nothing the model could have written to mark it, and the text simply stops in the middle of whatever it was doing. finish_reason is recorded by the serving stack rather than generated by the model, which is why it can report a cut that the prose structurally cannot, and why a finish reason of length masquerading as a bad answer is caught by the same habit that catches a skipped template.

Concretely, that means three fields in the frozen record, in this order: finish_reason, output token count against max_tokens, and total input tokens against the model's context window. If finish_reason is length and the output is 512 tokens against a max_tokens of 512, the investigation is over and the answer is a cap, not a prompt. If total input is within a few hundred tokens of the context window, you are in the territory of the next chapter, because something was dropped to make it fit. Two of the three checks are integer comparisons and cost nothing, and they close a meaningful fraction of single-trace investigations before any experiment is designed.

A verified template is a floor#

What verifying the template answers is exactly one question, and being precise about how little else it answers is what stops the check from becoming a false clearance. Having established that the model received a well-formed system block, you have ruled out one class of cause and gained no information about any other. The instruction can be correctly delimited and still be the fortieth line of a 900-token system prompt whose earlier lines contradict it; the retrieved passages can be correctly framed and contain nothing relevant; the model can be given a perfectly rendered request and still produce a wrong number. None of those are template problems, and none of them will be fixed by anything in this chapter.

Double application is the version that survives longest in a codebase, because it produces no visible artifact. A framework helper renders a ChatPromptTemplate to a string, that string is passed to a client that renders it again, and the model receives two begin-of-text tokens and a nested set of headers. It answers, mostly correctly, and slightly worse than it should. The reason the damage has no signature is that every document in training carried a beginning-of-text marker in exactly one position, so a second one part-way through puts the model in a configuration its training never covered — which is not an error it makes so much as behaviour nothing constrains. No single trace will ever prove it, and it shows up as a quality gap against the same model on a different code path.

The framework-boundary version is worth naming separately: prompt objects in orchestration libraries stringify, and stringification is lossy, so a three-message chat prompt handed to a component expecting a single string collapses into one user turn with the roles rendered as prose. That is the original bug reintroduced by a type coercion rather than by a design decision, and the rendered request in a frozen trace is the only place it is visible.

Rendering is also the wrong hypothesis for any failure that is intermittent at fixed settings. It is deterministic given the same messages and template, so a bug at this layer fires on every request of that shape, which is a useful diagnostic in itself: if the same route succeeds nine times out of ten, the template is not the variable, and time spent here is time not spent on the thing that is. Conversely, a route that fails constantly and looks like the model "ignoring instructions" is worth checking here first, because the fix is a one-line change to how the request is built rather than an open-ended rewrite of the prompt.

The rule this leaves is mechanical. Before reading a word of a model's output as evidence about the model, establish what the model was handed: the ids, the special tokens, the finish reason and the two token counts. None of those requires an opinion, and between them they either eliminate the wire format or end the investigation.

With the wire format established, the remaining suspects are the contents of the request. In a retrieval system the largest and least examined part of that content is the block of retrieved passages, and there is a specific and very common way of being wrong about them: assuming that because retrieval returned confident scores, it returned the answer.