Defending a prompt layout decision

Two mechanisms point in opposite directions when a variable block is moved below a static one. Chapter 5's cache argument says the static policy has to come first or a shared prefix is impossible, and the arithmetic is not close. Chapter 1's attention argument says material moved further from the request binds less strongly, and that is equally mechanical. Both claims are correct, they are about different quantities, and neither says by how much — which is the part that decides.

What makes such a disagreement tractable is that all three quantities come off the same requests. Compliance comes from per-rule predicates on a frozen case set; cache hit rate and time to first token, the delay between sending a request and the first token of the answer arriving, come from the usage and timing fields every response already carries. Chapter 2's harness is the same harness; the only addition is a few lines in the collector. The measurements below are from a conversational assistant with 8,600 tokens of static policy, a 400-token per-user context block, and roughly 3 million calls a day.

Four layouts, three axes#

The eval set was 1,200 conversation steps sampled across tenants and step types, frozen, with three predicates: does the assistant address the user by the correct name when it addresses them at all, does the output use UK spellings where the user's preference says so, does it avoid emoji when the preference says so. Each layout ran five times. TTFT is reported at p50 and at p95, the slowest one call in twenty.

Layout B is the one the attention argument predicts and it loses compliance. Name accuracy fell 2.7 points and UK spelling fell 2.5, both several times the 0.6-point run-to-run spread on this eval, and the specific failure was the one predicted: the assistant occasionally addressing the user by a name that appeared in the retrieved conversation history rather than the one in the personalisation block.

Two of those columns move for reasons the previous chapters already supply, and it is worth saying which so the table reads as a consequence rather than as a set of measurements that happen to differ. The cost column separates into two groups, not four: layout A and everything else. A puts variable bytes at position zero, so nothing behind them can be reused and the whole prompt is charged at base rate on every call; B, C and D put the static policy first, so its 8,600 tokens are served as cache reads on all but the first call in a window. The 22-token preference line that distinguishes C from B is small enough that the column barely notices it. Time to first token separates the same way and for the same reason: the interval before the first token is mostly prefill, a hit removes the prefill work for the cached span, and the span is most of the prompt.

The compliance column is the one that does not follow from caching at all, which is why it had to be measured. The failure it records has a shape chapter 1 predicts. A name is a token the model has to choose, and several candidate names are in the window: the correct one in the personalisation block, others in the retrieved conversation history. In layout A the correct name is the first thing in the whole prompt, where the pull of the opening works for it. In layout B it is nominally last, and nominally is the operative word — it sits somewhere inside four hundred tokens whose bulk is a rolling summary, so a good part of that block stands between the name and the question. The rule did not get less clear. Its position stopped being the strong one it looked like.

The reason B loses compliance while sitting in the strongest position is chapter 4's tail budget arriving from an unexpected direction. "Last" is not a position when the thing you place there is 400 tokens long; that block has a beginning, an end and a middle of its own, and the three preference rules were scattered through a middle made mostly of a rolling summary of the user's recent actions. Layout C leaves the 400-token block where B put it and hoists the three rules out of it into a single 22-token line placed after everything, immediately before the user's turn. The block still supplies context; the rules stop competing with it.

The belief that layout B refutes is worth naming, because it is the one a reader of the first four chapters is most likely to be holding. It is that last is a place you can put things. It is tempting because the tail block in chapters 1 and 4 behaves like one — sixty tokens of rules, placed after the documents, reliably outperform the same rules placed before them. What that generalises to is not "the end is strong" but "a span is strong in proportion to how little stands between it and the first generated token", and a 400-token block placed at the end fails that test for everything except its own final line. Layout C is what the corrected version of the belief produces: keep the block where it costs nothing, and lift out of it the twenty-two tokens that actually have to bind.

Layout D adds what chapter 4 would require of the one rule here whose violation is visible to the user and embarrassing: the correct name is known to the caller, so a response that addresses the user by any other capitalised first name from the retrieved history is a mechanically detectable defect, and a substitution fixes it in code. That is what takes the column to 100%, and it costs about 3 ms of post-processing.

Naming the property the layout favours#

A review that reports the table and stops has not made a decision, because the table contains a real trade and the trade has to be chosen rather than discovered. Layout D favours prefix stability and latency: it accepts that the personalisation context sits in a weaker position than production had it, and buys the compliance back on the three rules that were measured, which leaves everything in that block that was not measured — the rolling action summary, the plan tier — in the weaker position with no evidence either way.

That is the compromise, and it belongs in writing next to the numbers, in a decision record that the next person to touch the layout will find before they touch it.

decision: prompt-layout-2024-11 (assistant, all tenants)
favours: prefix stability, then latency
layout:
  1. tool definitions + static policy   # 8,600 tok, cache breakpoint at end
  2. per-user context block             # 400 tok, uncached by construction
  3. preference line                    # 22 tok: name, spelling, emoji
  4. user turn
accepted_compromise: >
  The per-user context block sits in a weaker position than in layout A.
  The three rules we measure were hoisted out of it and are unaffected; the
  plan tier and the recent-action summary are not covered by any predicate,
  so their influence on responses is unmeasured and assumed to have fallen.
guarantees:
  correct_name: deterministic substitution in code (not the prompt)
  uk_spelling: 97.9% measured, no guarantee
  no_emoji:    99.0% measured, no guarantee
monitors:
  - cache_read_input_tokens == 0 rate, alerting above 15%
  - name_substitution_fired rate, alerting above 2%
  - weekly per-rule compliance on the frozen 1,200-step set
revisit_when: >
  model string changes; static block changes by more than 500 tokens;
  anything is added to the per-user context block.

The revisit_when clause is the part that makes the document worth writing rather than a formality. Every number in the table is a fact about one model version, one template and one traffic distribution, and all three move. The first monitor is the cheapest early warning available: a rising rate of responses with zero cache reads means something has started varying in the prefix, and it is a lot easier to find within a day of the deploy that introduced it than a month later.