Chat templates and rule placement

The previous chapter established that a prompt element's authority is a property of the family reading it rather than of the sentence. Placement is the same argument applied to position: not which markup you use, but which turn a rule sits in and how much text has arrived since. It matters because the API hands you an array of role-tagged messages and invites you to think of the system turn as a privileged channel, and whether it actually is one is a per-family fact you cannot see from your side.

The symptom is a rule that holds on the model a prompt was written for and leaks on the model it was ported to, at a rate low enough to be called random and high enough to matter. The diagnosis is a measurement rather than an inspection, because the mechanism responsible sits between your message list and the model's forward pass and is usually not published.

The message list is not what the model reads#

Between your array of messages and the model's forward pass sits a piece of string formatting, usually a Jinja file shipped with the tokenizer, that flattens the array into one token sequence with the family's own control tokens — reserved tokens that mark structure and cannot be produced by ordinary text — marking turn boundaries. The API takes a list because a list is a convenient interface. The model takes a sequence.

{%- for m in messages %}
{{- '<|start|>' + m.role + '<|sep|>' + m.content + '<|end|>' }}
{%- endfor %}
{{- '<|start|>assistant<|sep|>' }}

That template renders a system turn as a delimited region at the front of the sequence. A different family's template may fold the system content into the first user turn entirely, wrap it in its own tag, or emit a preamble the API never showed you. The same three-message conversation produces two materially different sequences:

# Family A
<|start|>system<|sep|>You are Acme's sales assistant. ...eight rules...<|end|>
<|start|>user<|sep|>is Pro closer to 30 or 300 a seat?<|end|>
<|start|>assistant<|sep|>

# Family B
[SYS] You are Acme's sales assistant. ...eight rules... [/SYS]
[INST] is Pro closer to 30 or 300 a seat? [/INST]

The objection that the system message is an API convenience which gets flattened into one string anyway has the mechanism right and the conclusion backwards. It is flattened, and it is flattened with each family's own template, using control tokens that appeared in that family's tuning data with a particular meaning about precedence. One lab may train the system region as a near-inviolable operator channel that outranks anything a user says later; another may train it as context that a sufficiently direct user request can override, because that behaviour tested better on their helpfulness evals. Both are defensible design choices, and neither is visible from your side of the API.

You can usually see more of this than the API surface suggests. For open-weight models, the ones whose trained parameters are published for anyone to download and run, the template is a literal file — the chat_template field in tokenizer_config.json — and rendering your own message list through it prints the exact string the model receives, which is four lines of code with a tokenizer library and worth doing once so that the abstraction stops being mysterious. For closed models the template is not published, and three indirect signals stand in for it. Whether the API accepts a system field at all, and whether it accepts more than one, tells you whether the family maintains a distinct operator channel or is folding your text into the conversation. The provider's own guidance is informative in its phrasing: a family that describes the system turn as a place for role, tone and background is telling you something different from one that describes it as instructions the model is required to follow. And the behaviour is directly testable by moving the same text between roles on a frozen set, which is what the arms below do.

One thing worth ruling out before attributing anything to placement is that some hosted endpoints prepend text of their own — a safety preamble, a product persona, a tool-use instruction block — ahead of whatever you send. It shows up as an input token count meaningfully larger than your rendered prompt, and it means your system message is not at the front of the sequence even on a family whose template puts it there.

Why one rule out of eight behaves differently#

Consider a sales assistant with eight operating rules in its system message, where seven hold after a port and one does not. The seven are things like "answer in the user's language", "cite the docs page you drew from", and "escalate to a human if the user mentions a legal dispute". The eighth is "never quote prices; always direct the user to the pricing page", and it differs from all of them in three ways that compound.

It is the only rule that is a prohibition on a behaviour the user actively requests. Nobody asks the agent to stop citing docs. Users ask for prices constantly, and the model's helpfulness prior — trained on hundreds of thousands of examples where answering the question directly was preferred — pushes against the rule every time the question is asked. The other seven rules never enter that contest.

It is also the only rule whose violation is expressible in a single token. "Never quote prices" is broken by emitting $49, and the model's parametric knowledge or the retrieved context supplies that number readily. A rule like "escalate on legal disputes" requires a whole action to violate, and the failure mode there is omission rather than commission.

And its distance from the point of decision grows. The system message is fixed at the front of the sequence; the conversation grows in front of the assistant turn. Breaking a sample of 2,000 production conversations down by the turn at which the price question was asked shows the effect clearly:

Every rate in that table is one division, and the divisions are worth doing rather than reading. Four violations among 780 conversations in the shallow band is four out of seven hundred and eighty, about half of one per cent. Twenty-nine among 170 in the deepest band is a little over one in six. The bands also sum: they hold 2,000 conversations between them and 81 violations between them, and 81 divided by 2,000 is the 4.05% on the last row. That final division is the one to distrust, because it is a weighted average and its weights are the conversation lengths in the sample. The headline rate is therefore not a property of the rule. It is a property of the rule and the traffic's length distribution together, and it moves when the traffic mix moves even though nothing about the model or the prompt has changed.

The 4.05% headline is an average over a distribution with a seventeen-point tail in it, and the tail is where the long, engaged, high-intent conversations live. That is not randomness. It is a rule whose authority decays with the amount of text that has arrived since it was stated, on a family whose template and training make that decay steeper than the original model's did.

Decay is the right word, and it is worth taking apart, because "the rule gets diluted" is a picture and the thing underneath it is not. A transformer produces each next token by attending over every token before it, and the weight it puts on any one span is normalised against every other span in the sequence. When the sequence is a system message and one short question, the system region is most of what there is to attend to. When it is that same system message followed by fifteen turns of a specific, engaged negotiation about seat counts, the region is a small fraction of a much larger field, and it is competing with text that is both more recent and more obviously about the question being asked. Nothing was forgotten. The region's share fell because the denominator grew.

Two properties of the training data push in the same direction. Post-training pairs a user's most recent request with a directly responsive answer, many times over, which puts a strong prior on the text nearest the end of the input. And the precedence a system region enjoys is a habit taught by examples rather than a rule enforced by the runtime — which is the belief this chapter most wants to remove. Nothing between your message list and the response checks the assistant turn against the system turn. If something did, rule 6 would hold at every depth, and quoting a price would be impossible rather than improbable.

The diagnostic: move the rule and measure#

The way to establish this rather than assume it is to run the same frozen conversation set with the rule in different positions, holding everything else fixed. Four arms, five repeats each, scored on the same violation detector — a regex for a currency amount in the assistant turn, hand-checked on a sample to confirm it is not catching the pricing-page URL.

Both qualifications in that sentence are the difference between a measurement and a number. Scoping the regex to the assistant turn matters because users supply currency-shaped text themselves — the trace further down has one asking whether Pro is closer to 30 or 300 a seat — and a detector reading the whole conversation would score the compliant answers to those questions as violations, in a band that happens to be where compliant answers are commonest. Hand-checking matters because the intervention under test appends a link to a pricing page, and a currency regex firing on that link would count the fix as the failure. Either mistake makes all four arms look alike, and four arms that look alike read as an intervention that does not work.

ARMS = {
    "system_only":        lambda msgs: msgs,
    "system_plus_tail":   lambda msgs: msgs[:-1] + [restate(msgs[-1], RULE_6)],
    "tail_only":          lambda msgs: strip_rule(msgs, 6)[:-1] + [restate(msgs[-1], RULE_6)],
    "system_emphasised":  lambda msgs: emphasise(msgs, RULE_6, times=3, prefix="IMPORTANT: "),
}

def restate(user_msg, rule):
    """Append the rule to the final user turn, so it is the most recent text."""
    return {**user_msg, "content": f"{user_msg['content']}\n\n[Reminder] {rule}"}

Running that on 600 held-out conversations gives the shape of the fix:

Three things are readable from that table. Emphasis and repetition do work, cutting the rate by roughly two-thirds, which is why the technique persists and why it usually closes the ticket. Restating the rule near the end of the input works better than emphasising it at the front, and does so precisely where the emphasis fix is weakest, in the deep-turn tail — which is the evidence that distance rather than salience was the dominant term. And tail_only performing about as well as system_plus_tail says the system-message copy is contributing little on this family, though it costs almost nothing to keep and is doing real work on the original model, so it stays.

The middle of those three is a piece of inference worth doing out loud, because it is what turns a preference into a finding. Two explanations were available for why the rule loses: it is not salient enough, or it is too far away. The system_emphasised arm raises salience and leaves distance untouched. The tail arms change distance and leave the wording untouched. If salience were the binding term, emphasis would close the gap everywhere, and most of all where the rate is worst. It does the opposite — its residual is largest in the deep-turn band, which is the band where distance is largest — while the tail arms' advantage is concentrated in exactly that band. Two interventions, one variable moved each, and the one that moves distance wins where distance is greatest. That is the argument; the rate column alone is only a ranking.

There is a cost to system_plus_tail that belongs in the record: it appends roughly 25 tokens to every user turn rather than 25 tokens once per conversation, so on a 20-turn conversation it is 500 input tokens instead of 25. With prompt caching, where the provider keeps the processed form of a prompt prefix for a few minutes and bills a fraction of the usual input rate when a later request repeats that prefix exactly, the system message is nearly free to repeat and the tail reminder is not, since it changes the suffix of every request. At 40,000 conversations a month averaging 11 turns that is about 11 million extra input tokens a month, which at $3 per million is $33. Cheap, but it is a number, and the same change on a high-volume classification path would not be.

That chain is a sequence of multiplications, and it is worth walking because the same chain prices every per-turn addition anyone will ever propose. The reminder costs roughly 25 tokens each time it is appended. It is appended once per user turn rather than once per conversation, which is the entire difference, and it is why a 20-turn conversation pays 500 rather than 25. Across a month the multiplier is conversations times turns per conversation: 40,000 conversations at an average of 11 turns gives the number of turns, and multiplying that by 25 tokens gives roughly 11 million tokens. Dividing by a million and multiplying by the $3 rate gives $33. The arithmetic is worth carrying rather than the answer, because the answer is small only because two of its terms are small.

Caching is the reason the two copies of the rule cost such different amounts, and the mechanism is one step. A cache matches an exact prefix. A system message that is byte-identical from request to request sits at the front of every sequence, so the stored prefix matches and those tokens bill at the reduced rate. The reminder sits at the end, behind content that differs on every request, so there is no matching prefix that reaches it and it is charged in full every time. The position that makes the reminder effective is the position that makes it uncacheable, and that is not a coincidence — both follow from its being the most recent text the model sees.

The ceiling on every arrangement of text#

Nothing in that table gets to never, because every arm is a different way of adjusting a probability distribution, and a distribution with support on the wrong token has a non-zero rate of emitting it however the prompt is arranged. Emphasis, repetition, restatement and placement are all instruments of the same kind, and the ceiling on all of them is the same ceiling.

That is the honest answer to the colleague's proposal. IMPORTANT: three times is not wrong so much as insufficient, and agreeing to it has a specific cost beyond the six points of residual risk in the deep-turn tail: it ends the investigation. A team that ships the emphasis fix stops looking, records the rule as handled, and finds out at the 130th quoted price that a probabilistic control was standing in for an absolute one.

There is also a limit to what the placement diagnosis explains. It accounts for the depth gradient convincingly; it does not fully account for the four violations at turns 1–3, where the rule is fifty tokens away from the decision and still loses. Some fraction of the failures are not about position at all, and no arrangement of text addresses them.

Those four have a different cause, and naming it is what makes the ceiling predictable rather than mysterious. At turn two the system region is most of the sequence and the rule is a few dozen tokens from the point of decision, so distance cannot be the binding term. What remains is the contest set out earlier: a prohibition on precisely the thing the user has asked for, run against a helpfulness prior that does not weaken with proximity, because proximity was never what fed it. Rearranging text adjusts the distance term and leaves the prior exactly where it was. That is why there is a floor, and why the floor is where it is rather than somewhere lower.

What placement does buy is worth stating plainly, because the ceiling argument can be read as a counsel of despair and should not be. Moving from 4.05% to 0.33% is a twelvefold reduction in a customer-visible failure, achieved with 25 tokens per turn and no change to what the rule says. That is the largest effect available from anything in this chapter, and it is available on every port, and it is invisible to anyone who reads the headline rate without splitting it by depth.

Getting a never-quote-prices rule to zero rather than to a third of a per cent means it stops being a sentence in a prompt. The distinction between a control that shapes a probability and a control that cannot be violated is the subject of the next chapter, and it applies well beyond prices — it is the same distinction that separates a schema-constrained decoder from the sentence "return only the JSON object".