Server-side and client-side templating

A chat template turns a list of message objects into one flat string of tokens, and somebody has to run it. A chat endpoint runs it server-side from the messages array you post. A raw completion endpoint runs nothing and forwards your string as-is, which makes rendering your responsibility. A client library, a framework wrapper or a serving proxy sitting between the two may run it as well, on text that already carries markers. None of those behaviours is wrong on its own, and any two of them combined produce a request the model reads differently from the one you meant to send.

This chapter is about establishing which layer owns the rendering for each path an application calls, and about recognising the three arrival states from evidence rather than from assumption. Doubling is the interesting one, because it does not raise an error, costs almost nothing in tokens, and produces output that reads as competent English while answering a question nobody asked.

Who owns the rendering on each path#

The split exists for a historical reason that is still visible in every API. Completion endpoints came first, when the interface to a language model was a string in and a string out, and the caller was responsible for everything that string contained. Chat endpoints were added on top, moving the serialisation into the server so that callers could stop tracking per-family markers. Both survived, most stacks expose both, and the wrappers people reach for were written across the transition, which is why a library can plausibly do either.

Other stacks draw the same line in their own vocabulary. Ollama exposes /api/chat, which templates, and /api/generate, which templates unless you pass "raw": true, so one server has three behaviours across two endpoints and one flag. Text Generation Inference distinguishes /generate, which sends your string untouched, from its messages API, which renders server-side. llama.cpp's server offers both a completion route and an OpenAI-compatible chat route, with a --chat-template flag that overrides whatever the GGUF metadata declares, GGUF being the single-file weight format llama.cpp reads. The question to ask of a new stack is always the same: which of these routes runs the tokenizer's template, and what happens when the checkpoint does not supply one.

The wrapper row costs the most time, because the wrapper is usually right. A framework that emits ChatML — the <|im_start|> / <|im_end|> format OpenAI published and much of the community copied — will work acceptably against a Qwen checkpoint and produce nonsense against a Mistral one, so the same code passes in staging against one model and fails in production against another, and the diff between the two environments is a model string in a config file.

The instinct that keeps the doubled state alive is that server-side templating is the careful option, so handing the server something already rendered can only be redundant. Redundancy would be harmless. What happens instead is not redundancy, because the second rendering does not wrap the first in an outer layer the model can see through. It produces one sequence in which your carefully assembled conversation is content, and content is the material a model treats as the thing being discussed rather than as the thing it is taking part in. A layer applied twice is not twice as safe. It is a different structure.

What twice looks like#

Nesting is worth reading token by token, because the failure it produces is specific rather than general. Take a four-message support conversation, render it, then hand the resulting string to the same template again as user content.

once = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
twice = tok.apply_chat_template(
    [{"role": "user", "content": once}], tokenize=False, add_generation_prompt=True
)
print(len(tok(once).input_ids), len(tok(twice).input_ids))
# 96 106
print(twice.count("<|begin_of_text|>"), twice.count("<|start_header_id|>assistant<|end_header_id|>"))
# 2 3

Ten extra tokens is nothing, which is part of why this survives casual inspection — a cost regression would have been noticed on the first invoice. The structural damage is what matters. The outer template opens a user turn, and the first thing inside that turn is a BOS marker, the beginning-of-sequence token the model has only ever seen at position zero of a document. Then, part way through what is supposedly one user message, an <|eot_id|> appears from the inner rendering, and as far as the model's learned structure is concerned that ends the outer user turn early. Everything after it — the inner user turns, the inner assistant turns, the inner generation prompt — is read as further turns of the outer conversation rather than as quoted text, and the outer template's own closing markers arrive after a conversation that has already moved on, leaving two assistant headers with an empty turn between them.

Two properties of the pipeline are what let all of that happen without a complaint from anything. The first is that rendering is string concatenation: the template walks the message list appending text, and it has no way to notice that the text it is appending already carries markers, because by then everything involved is characters. The second is that no layer downstream holds a model of what a well-formed conversation looks like. The tokenizer maps bytes to integers. The engine scores the next integer given the ones before it. There is no parser, no schema and no validation pass, so there is no component that could have refused the request — which is why a structurally impossible conversation produces an error message nowhere and about ten extra tokens on the invoice.

The word people reach for at this point is quoting, and it is the belief worth correcting. Quoting needs a distinction between using a marker and mentioning one, and a flat sequence of integers cannot carry that distinction. An end-of-turn ID is an end-of-turn ID whether the template emitted it or a caller pasted it into a message body. No field records where it came from and no bit marks it inert. So the outer template did not fail to quote the inner conversation. Quoting was never an operation available to it.

The behaviour that comes out is characteristic. Asked "My order hasn't arrived," a doubly-templated request returned:

This appears to be a support conversation between a customer and an agent.
The customer is asking about a missing order, and the agent has requested
the order number. Would you like me to summarise the exchange?

The model is not broken and is not ignoring instructions. It has been handed a document whose outer frame says "a user has pasted a conversation into a message", and commenting on a pasted conversation is a reasonable response to that document. Quality on a task like classification falls by ten or twenty points while every individual output still reads as competent English, which is precisely the failure that survives a subjective review.

Absent and doubled are not the only options#

A third state hides between them, where the template is applied exactly once and is the wrong one. Serving stacks need a template to render a chat request, and they get it from the tokenizer's chat_template field. When that field is missing — which is normal for a base checkpoint, and happens with community fine-tunes repackaged without their tokenizer config — the stack does one of three things depending on version and flags: refuses the request, falls back to a built-in default, or uses whatever you passed to --chat-template. A fallback default is usually ChatML, because it is the closest thing to a lingua franca, and ChatML markers sent to a Llama checkpoint are not control tokens at all.

That last clause carries the whole of the damage and is easy to read past. <|im_start|> is not registered in a Llama tokenizer, so it never becomes a reserved ID; it becomes a handful of ordinary vocabulary entries spelling those characters out. The request is then structurally identical to the hand-rolled Assistant: serialisation from the previous chapter, wearing markers that look official. A stack rendering with a borrowed template has not applied the wrong convention so much as applied none at all, as far as the checkpoint is concerned, and the output it produces is the fluent-continuation failure rather than anything that announces a mismatch.

# what the tokenizer config actually contains, before you trust anything downstream
python - <<'PY'
from transformers import AutoTokenizer
t = AutoTokenizer.from_pretrained("some-org/some-finetune")
print(bool(t.chat_template), t.eos_token, t.eos_token_id)
PY

If that prints False, every chat request you send to that checkpoint is being rendered by somebody's guess. The fix is to supply the template explicitly at serve time rather than to hope, and to pin it next to the model string so the two cannot drift apart when someone bumps a version.

The related trap is add_special_tokens. Tokenizer calls default it to True, so tokenising an already-rendered string prepends a second BOS. One duplicated marker is milder than a full nesting and correspondingly harder to find, since output stays on task and quality slips by a couple of points. The rule that avoids both is to decide, per code path, whether you are handing over structure or handing over bytes, and never to mix the two in one call.

A duplicated beginning-of-sequence marker deserves one paragraph of mechanism, because "one extra token" sounds as though it should cost nothing at all. That token appeared at position zero of every document the model was ever trained on, and nowhere else. Two of them puts the second at position one, a place it has never occupied, and shifts every position after it by one against the pattern the weights encode. The damage is small because the rest of the sequence is still coherent English in a still-recognisable frame. It is not zero, because the sequence now opens in a way no training document did, and the model's scores for the first few tokens are conditioned on an opening it has no experience of.

When rendering it yourself is the right call#

Server-side templating is the better default for most application code, and there are three situations where it is the wrong one.

The first is training data. A fine-tuning corpus has to be rendered with the exact template the serving path will use, including the generation prompt convention and the loss mask boundaries, because any difference between the two becomes a systematic distribution shift the model has no way to see. Generating that corpus by posting to a chat endpoint and hoping the server's version matches introduces a mismatch you cannot inspect afterwards.

A mismatch there is worse than a mismatch at inference for a reason worth following through. Fine-tuning nudges the model toward producing its examples' assistant spans given its examples' prefixes, so whatever prefix shape the corpus happened to use becomes the shape the new behaviour is conditioned on. Serve a slightly different prefix afterwards and you are asking for behaviour at a sequence state the training never visited. None of that shows up in a loss curve, in an evaluation that renders the same wrong way as the corpus, or in a diff of application code, which is the argument for pinning the rendering next to the corpus it produced.

The second is prefix caching. Serving stacks cache the attention keys and values for a prompt prefix and reuse them across requests that share it, which on a 2,000-token system block plus few-shot examples cuts time-to-first-token by around 80% and removes the prefill cost — prefill being the one pass over the entire prompt that has to happen before any output token can be sampled — from the bill on providers that price cached input separately. The cache key is the token sequence, so the prefix has to be byte-identical across requests, and a template that injects today's date, as Llama 3.1's does, breaks the match once a day at midnight unless you render it yourself and pass a fixed date_string. Teams have chased that as a nightly latency spike more than once.

The byte-identical requirement is not fussiness on the cache's part. What the server stored was the attention keys and values computed for those exact tokens at those exact positions, and each of those values was computed from every token ahead of it. Change one token near the front and every stored value after it describes a sequence that no longer exists, so nothing past the first difference can be salvaged. A prefix cache can therefore only ever match a prefix. A date that rolls over at midnight sits inside the system block, near the front of every prompt, which puts the change ahead of almost everything that was worth reusing.

The third is anything that reads or manipulates positions in the sequence: computing logprobs — the log-probabilities the model assigned to the token at a chosen position — at a specific index, constraining decoding with a grammar that starts after a known marker, or measuring how many tokens of a long context a retrieved document occupies. If you do not know the exact prefix, you do not know the index.

prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True,
                                 date_string="01 Jan 2026")   # pinned, so the cache key is stable
r = client.completions.create(model=MODEL, prompt=prompt, max_tokens=256,
                              extra_body={"add_special_tokens": False})   # BOS is already in the string

The add_special_tokens=False in that call is the part people leave out. Having rendered a string that begins with <|begin_of_text|>, you are handing over bytes rather than structure, and the tokenizer's default behaviour of adding a BOS on your behalf is now a duplication rather than a service. Most OpenAI-compatible servers accept the flag as an extra body field; the ones that do not are a reason to render without the leading BOS instead.

One application, three states at once#

An application that calls three endpoints can easily be in three different states simultaneously. A batch job hitting /v1/completions with its own rendered prompts is templated once and correct. A streaming chat handler posting messages to /v1/chat/completions is templated once and correct. An evaluation script that reuses the batch job's prompt builder and sends the result to the chat endpoint is templated twice, and its scores are measuring the wrong thing — usually downward, which makes a good model look bad, and occasionally upward on tasks where summarising the input happens to score well, which is worse because nobody investigates a number that improved.

None of this is a claim that the layers are badly designed. Server-side templating is the right default for a hosted API, since it removes an entire class of per-family formatting bugs from every caller. Client-side templating is the right choice when you need to control the exact bytes, which is the case for training data, for prefix caching where the cache key is the rendered prefix, and for anything that inspects logits at particular positions. What causes trouble is holding both conventions in the same codebase without writing down which one each path uses.

Three states, then, and the way to know which one a request is in is to trace the layers rather than to reason about the application as a whole. Untemplated looks like a fluent continuation that never stops. Templated once is what you wanted. Templated twice looks like a competent answer to a question about your conversation instead of an answer inside it, and it costs about ten tokens, which is why nothing in the bill or the error rate will point at it. Writing the ownership down next to each call site is cheap; discovering it from behaviour a week later is not.

There is one more source of markers that this accounting misses entirely. Every state described here comes from a layer of your own stack deciding to render something. The markers can also arrive from outside, carried in the content of a message, written by someone who is not on your team.