The stable-first layout

Stable-first is the ordering that falls out of prefix matching: everything byte-identical across requests goes at the front, everything that varies per request goes behind it, and the boundary between them is where the cacheable segment ends. It is the right answer to the cost question and it has a cost of its own, because a prompt has exactly one position that instructions are followed best from — the very end, after the material they govern — and stable-first has just spent that position on the user's turn and the retrieved documents.

What that demotion costs is measurable, and measuring it is the difference between a layout somebody chose and a layout somebody inherited. The figures below come from one support agent running a 10,200-token prompt at 120,000 requests a day, at $3 per million input tokens, with a write charged at a quarter above the base rate and a read at 0.1×.

The region the rules are now sitting in#

"Lost in the Middle" measured this on multi-document question answering: with twenty passages in the context and one of them carrying the answer, accuracy traced a U across positions, highest when the answer-bearing passage sat first or last and roughly twenty points lower when it sat in the middle of the set. The result held across model families and across context lengths, and it held for models whose advertised window comfortably contained the whole input, which is the part that surprised people. A 200,000-token window is a statement about what the model will accept, not a statement about uniform attention across what it accepts.

A placement probe run the way topic 1.12 sets one up — one rule held fixed, moved across positions inside the production template, scored on compliance rate rather than on an answer key — puts concrete numbers on the same curve for one support agent's 10,200-token prompt, whose policy forbids quoting prices:

Two things in that table matter more than the headline, and both of them are subtractions between rows rather than any single row.

The first is that the penalty is not caused by the policy being at the front. Compare the first row with the second: token 400 and token 6,800 are both inside the same block, both in front of the retrieved documents, and they differ by nearly three points. If being in the prefix were the cause, those two would score alike. They do not, so the cause is the rule being deep inside a long block, which is a different problem with a different set of fixes — and one of them does not require moving the block anywhere.

The second is which comparison you make to price the layout. Taking the worst position inside the prefix against the best position outside it, 95.9 against 99.4, gives 3.5 points, and that is the number a reordering proposal will quote. Taking the best position inside the prefix against the best outside it, 98.6 against 99.4, gives 0.8. The first subtraction prices "policy first" against "policy last" while quietly also changing where inside the policy the rule sits; the second changes only the thing under discussion. Ordering the policy internally, so the rules that matter sit at its opening and its close rather than in its interior, is what closes the difference between those two subtractions — and it recovers most of what the layout costs before anything moves out of the prefix at all.

Those two subtractions are the shape of the trade in practice. Stable-first does not force you to accept the worst position for every rule; it forces you to accept that no rule can sit after the retrieved documents, and the size of that penalty is specific to the prompt, the rule and the model.

The good positions inside the prefix are a fixed budget#

The eleven sentences that carried this agent's price rule were spread across two sections of the policy, five in §4 Commercial boundaries at token 5,900 and six in §7 Escalation paths at token 7,400, each one qualifying a different situation. Rewriting them as a single four-line rule in the policy's closing section, at token 8,760, took price-quote compliance from 95.9% to 97.4% on 400 paired cases — identical inputs scored under both wordings — and removed 210 tokens from the prefix. It cost one cold start at deploy time, meaning a single request that finds no warm entry and pays to write a fresh one, and nothing else, which makes it the cheapest intervention available anywhere in this topic: no layout changed, no tokens were added to the tail, and the cache never noticed.

What it did cost was the position it took. The closing section of that policy had held the output formatting rules, and moving them up by four lines to make room put them at token 8,300 with the new price rule behind them. Formatting compliance, scored as whether comparisons come back as a markdown table, fell from 98.1% to 94.6% in the same run. Nobody had a floor written against formatting, so the trade was accepted, but the mechanism is the one to carry forward: a long block has two positions that behave well, its opening and its close, and every rule promoted into one of them displaces another. Past three or four rules the closing section acquires a middle of its own and the promotion stops working, which is the same ceiling that will limit the tail restatement in chapter 5, arriving here one level down.

Three degradations, and you are picking one#

Every resolution of this conflict pays in one of three currencies, and a decision that does not name its currency is not a decision.

Two of those currencies have exchange rates you can compute for a given surface, and both computations are short enough to follow.

Stable-first on this support agent is worth 5,823 effective input tokens per request. That figure is a subtraction with a stated comparator, and the comparator matters: it is the same 10,200-token prompt billed with no shared prefix and no write premium — 10,200 effective tokens — against the effective tokens the cached layout averages, which chapter 1 worked out at a little under four and a half thousand. The difference is 5,823. Multiply by 120,000 requests a day and by $3 per million and the layout is worth $2,096 a day. Note what this is not compared against. Measured against the pathological layout of chapter 1, the one that keeps writing entries nobody reads, the gap is larger still; measured against a competently disabled cache, it is these 5,823 tokens, which is the honest number to bring to a review.

The accuracy side runs the other way, from a rate to a count. Non-compliance is whatever 95.9% leaves over, which is a little over four percent of the answers the rule applies to. The rule applies to the roughly 31% of conversations that touch pricing at all. So 120,000 requests a day, times that 31%, times that four-and-a-bit percent, is about 1,500 answers a day quoting a price the policy forbids. Turning that into money needs two more observed rates: one in forty of those becomes a ticket, and a ticket costs about $11 to handle, which is $418 a day of support cost. Set against each other the numbers appear to settle the argument, and they do not, because the remainder of the accuracy column is a customer who was quoted a discount the company will not honour, and there is no defensible figure to put on that. Computing what you can and naming what you cannot is the honest form of the decision; a spreadsheet that assigns the unquantifiable term a value of zero has decided the question by omission.

The asymmetry in the second row is the reason teams drift toward accuracy as the default sacrifice, and the mechanism is worth naming because it is not a failure of intent. Token spend and cache misses arrive as a number that already exists: somebody's invoice, somebody's panel, computed automatically, compared with last month by a system nobody had to build. Accuracy on an open-ended generation has no such number unless a predicate was written, a sample was drawn and a rate was plotted, and none of that happens by default. So the two currencies are not being weighed against each other at all. One of them is being observed and the other is being inferred from three tickets in a fortnight that nobody attributes to a prompt layout. Choosing accuracy is therefore the path of least immediate resistance and the one most likely to have been chosen by accident — and the repair is not better judgement, it is building the missing number.

Naming the traffic mix before naming the order#

The right layout for a prompt is a function of the traffic that hits it, and the input to that function is measurable from request logs before anything is deployed. What you need is the reuse count per distinct prefix inside the cache's TTL window, because that number decides whether the prefix is worth protecting at all.

from collections import Counter
from hashlib import blake2b

TTL_S = 300  # provider TTL for a warm entry

def prefix_reuse(log_rows, prefix_fn):
    """log_rows: (epoch_seconds, request) sorted by time.
    Returns reuse count per distinct prefix within rolling TTL windows."""
    last_seen, reuse = {}, Counter()
    for ts, req in log_rows:
        h = blake2b(prefix_fn(req).encode(), digest_size=16).hexdigest()
        if h in last_seen and ts - last_seen[h] <= TTL_S:
            reuse[h] += 1          # a read against a warm entry
        else:
            reuse[h] += 0          # a write: cold or expired
        last_seen[h] = ts
    return reuse

r = prefix_reuse(rows_24h, lambda q: POLICY_AND_TOOLS)
print(f"distinct prefixes: {len(r)}   reads/writes: {sum(r.values())}/{len(rows_24h) - sum(r.values())}")
# distinct prefixes: 1   reads/writes: 93861/26139

The function has no cache in it and does not call a provider, which is the point — it decides in advance whether a cache would have helped. It hashes the prefix each request would have sent, and it counts a read whenever it has seen that same hash within the last TTL_S seconds and a write whenever it has not. Those two conditions are the entire cache semantics that matter for this question: an entry exists if the same prefix arrived recently, and it does not if the prefix is new or the last one has idled out.

Run it against the layout you are proposing rather than the one you have, by swapping prefix_fn for the function that builds the proposed prefix. The support agent's numbers come out at one distinct prefix and 93,861 reads against 26,139 writes. Those two add to the day's 120,000 requests, and 93,861 of 120,000 is a little over 78% — the same hit rate the dashboard reports, arrived at from request logs alone with no deploy involved. The write count is dominated by a nightly quiet period and by the eight deploys that invalidated the entry that day, which is a useful thing to have separated out, because those writes are a floor that no layout change can lower. A surface that comes back with 40,000 distinct prefixes and a mean reuse of 1.2 does not have a prefix worth defending, and its ordering should be decided entirely on quality.

Three properties of the mix change the answer. Concurrency decides how much of the write premium is wasted duplicate filling; a surface at 40 requests a second refills a cold entry with several requests in flight, while one at 40 an hour refills it exactly once. Deploy frequency sets a floor on writes, because every change to the policy text invalidates every entry, and a team shipping prompt edits six times a day is paying six cold starts a day no matter how the blocks are ordered. Diurnal shape decides whether the entry survives the gaps, since a surface with a nine-hour overnight trough eats a cold start every morning and one or two per quiet hour.

The demotion deepens as the conversation grows#

The 95.9% in the placement table is a single figure covering a distribution that the layout decision never mentioned. This agent is conversational, so each turn appends the user's message, the assistant's answer and a fresh set of retrieved chunks, about 640 tokens in all, and the policy's distance from the end grows by that much every turn while its position in the prefix stays where it was. Scoring a fortnight of sampled production answers by turn index separates the two effects:

The third column is where the mechanism is, and it is addition rather than anything subtle. Each turn appends about 640 tokens and the policy does not move, so the distance from the price rule to the end of the prompt grows by about 640 tokens per turn while the rule's own offset stays exactly where it was. Two or three turns in, that distance has gone from 4,300 tokens to 5,600; by turn eight or later it is 11,200, which is more than the whole prompt was at turn one. The rule has not been demoted once by a layout decision. It is being demoted again on every turn, by the conversation, and nothing in the layout is aware of it.

Sixty-two percent of this surface's traffic is a first, second or third turn, which is why the aggregate reads as tolerably close to the best case. The conversations at the bottom of the table are the ones where a customer has been talked through an upgrade for ten minutes, which is precisely the situation the price rule exists for, so the layout is weakest where the rule matters most. Session length therefore belongs in the traffic mix alongside requests per day, and a surface whose median session is eleven turns is making a materially worse trade than the same prompt at one turn per session even though the layout and the token counts are identical.

The same growth works in the opposite direction on the cache, which is the part of this that makes stable-first hard to argue away. A multi-turn session reads the 9,000-token prefix once per turn and keeps refreshing the entry's lifetime while the conversation is live, so the eleven-turn sessions are also the ones with the highest reuse. Whatever ordering you land on, the two effects concentrate in the same traffic rather than cancelling.

Where stable-first is the wrong default#

It is wrong when the stable part is small relative to the volatile part. A prompt with 800 tokens of instructions and 14,000 tokens of retrieved evidence per request protects 5% of its input by putting instructions first, so the layout question should be settled by whichever order the eval prefers and the cache should be left to collect whatever it happens to get.

It is wrong when the rule at stake is categorical. A disclaimer that regulation requires on every qualifying answer is not a thing to trade against a hit rate at any exchange rate, and the correct response there is a control outside the model — appended deterministically in your own code, or enforced by a validator that rejects the response — rather than a better position in the prompt.

And it is wrong when the mechanism does not engage, which is a larger category than most teams check for, and the subject of the last chapter in this topic.

Stable-first is therefore a default with a bill attached rather than a rule. It protects the prefix, it demotes every instruction in the prompt by exactly one position class, and the size of that demotion is a number you can measure in an afternoon with a placement probe and 400 paired cases. Measure it, spend the two good positions inside the block on the rules that carry consequences, and write down which of the three currencies the surface is paying in.

Everything so far assumes the prefix is genuinely invariant once you have put the invariant material first. That assumption fails more often than the ordering does, and it fails silently, because a prefix can be 99.9% identical across every request in the day and still produce a hit rate of exactly zero. The next chapter takes the most common single cause.