Evaluating a layout change

Layout changes get filed as infrastructure work. They move blocks rather than words, they are argued for with a cost graph, and they land in a different tracker from the one that records prompt edits. The premise underneath that filing is that the tokens are the same tokens, so cost optimisations do not change behaviour — and it is false in exactly the way this topic keeps insisting.

It is worth being precise about why, because the premise is not stupid. A model does not receive a set of tokens. It receives an ordered sequence, and every position's representation is computed from the positions before it, so the same tokens in a different order produce different representations at every position downstream of the first move. A permutation of a sequence is a different sequence. That is not a claim about attention or about a particular model family; it is what "conditions on the preceding context" means, and it holds before any empirical result about position is brought in. The empirical result adds the direction and the size: a rule's compliance rate is a function of where it sits, and moving a block moves every rule inside it.

The consequence is not only that reorderings regress things. It is that they regress things invisibly, because the change log a team consults during an investigation records edits to instruction text, and a permutation is not an edit to instruction text. The measurements below come from a research assistant serving about 85,000 requests a day, at $3 per million input tokens, whose citation rule requires a source link on every factual claim.

A cost win with a compliance column#

Both layouts, scored on one case list of 400 questions against one pinned model version on the same afternoon, produce a single sheet:

Three of those rows are one multiplication and the reader should be able to redo them. Effective tokens per request times requests per day times the price per token is the daily spend: 14,272 tokens across 85,000 requests at $3 per million is $3,640, and 8,265 across the same 85,000 is $2,108. The delta of $1,532 is the difference between those two, and the −42% beside the token row is 8,265 divided by 14,272 subtracted from one. One number, expressed three ways, because a review will ask for it in whichever of the three it is fluent in.

The row that decides this is none of them. It is the citation rate, and the reason it decides it is that the citation rate had a floor written against it and the judge score did not. Answer quality, scored here by a second model rating each answer one to five against a written rubric, moved by 0.03 points on a five-point scale, which is noise on 400 cases and would have been the row everyone looked at.

The reason the quality score cannot see the regression is arithmetic rather than judgement. A judge score is a mean over every case. A citation failure is a binary event on the subset of answers that make a factual claim, and when it happens the judge still has a fluent, correct, well-structured answer in front of it and marks it down by a little or not at all. So a seven-point move in a rate on a subset arrives at the mean already divided by the subset's share and already multiplied by however much a missing citation costs on a five-point rubric, and what is left is smaller than the noise. That is why compliance predicates are scored separately rather than folded into a quality score: an average is the wrong instrument for a property that is either present or absent.

The mechanism behind the citation row is position, and it is arithmetic rather than mystery. The citation rule had been sitting 900 tokens before the end of the prompt, immediately ahead of the user's question, in the position the placement probes in chapter 2 scored highest. After the reordering it sat at token 7,300 of a 16,000-token prompt with six retrieved documents between it and the answer the model was about to write. Nothing about the rule changed. Its position changed, and its compliance rate followed the curve.

Why an investigation starts three weeks downstream#

When a citation regression of that size surfaced in production, three hypotheses were opened before anyone reconsidered the layout, and each of them was a defensible place to look.

Two of the three are acquitted by dates that were available on the first afternoon, and the reason nobody sorted the timeline first is that the layout change was not in the timeline. It had been filed in the infrastructure tracker as cost work. The model does not distinguish the two, so the change log has to stop distinguishing them as well; a reordering, a retrieval cap and a schema serialization change are all prompt changes with the same standing as a rewritten sentence.

The cost of assuming otherwise is not just the three weeks. It is that three weeks of unrelated changes accumulate behind the real cause and give it somewhere to hide, so that by the time anyone reconsiders the layout there are four confounded variables and no clean baseline to compare against. An investigation that starts on the day of the reordering is a bisection with one candidate.

The gate this change needed#

The gate is cheap because both arms of it run the same cases against the same model on the same afternoon. What makes it a gate rather than a ritual is that the compliance floors are written down before the run, each floor attached to a predicate — a deterministic check that reads an answer and returns pass or fail, such as whether a citation block is present and well formed — so the decision is a comparison against a number rather than a judgement about whether 96% feels acceptable.

FLOORS = {"citation": 0.985, "no_price_quote": 0.995, "confirm_destructive": 0.999}

def score_layout(cases, build_prompt):
    hits = {k: 0 for k in FLOORS}
    tokens = {"read": 0, "write": 0, "fresh": 0}
    for c in cases:
        req = build_prompt(c)
        r = client.messages.create(model=PINNED_MODEL, temperature=1.0, **req)
        text = r.content[0].text
        for name, predicate in PREDICATES.items():
            hits[name] += predicate(text, c)
        tokens["read"] += r.usage.cache_read_input_tokens
        tokens["write"] += r.usage.cache_creation_input_tokens
        tokens["fresh"] += r.usage.input_tokens
    n = len(cases)
    return {k: v / n for k, v in hits.items()} | {
        "hit_rate": tokens["read"] / max(tokens["read"] + tokens["write"], 1),
        "eff_tokens": (0.1 * tokens["read"] + 1.25 * tokens["write"] + tokens["fresh"]) / n,
    }

before = score_layout(CASES, build_docs_first)
after  = score_layout(CASES, build_stable_first)

The eff_tokens line is the effective-token conversion from chapter 1 written as code: reads counted at a tenth, writes at a quarter above base, anything uncached at face value, divided by the number of cases to give a per-request figure. It is in the same function as the compliance counters on purpose, because the whole argument of this chapter is that those two numbers have to come out of one run.

Three details in that harness are load-bearing. The model version is pinned rather than aliased, because an eval run against a moving target measures two changes at once. The temperature is production temperature, since compliance is a rate over a stochastic process and a run at temperature 0 answers a question nobody asked — it reports what the model does at its most likely, and nothing that ships runs there. And both arms run against the identical case list, which matters more than the absolute numbers. A paired comparison holds the inputs fixed and varies only the layout, so any difference has exactly one candidate cause. That is why the cases do not need to be representative of production to answer the question: representativeness is what you need to estimate the production rate, and this run is estimating a difference.

How many cases the citation check needed#

A 400-case eval detects the drop in that scorecard easily, since going from 99.2% to 92.0% on 400 paired cases is far outside sampling noise. That comfort is specific to the size of the regression, and the relationship between the two is the thing to carry away rather than any particular case count. The number of cases a test needs rises roughly with the inverse square of the difference it has to detect, so halving the effect you care about multiplies the cases you need by about four. To detect a drop from 99% to 97% at 80% power — the chance the test flags a real drop of that size — and a 5% two-sided significance level, a two-proportion test needs roughly 770 cases per arm. To detect 99% to 98.5%, which is a quarter of the effect, it needs several thousand.

A fixed eval of 200 cases can therefore clear a layout change that quietly costs a point of compliance, and the trap is what "clear" means. A test that fails to find a difference has not found that there is no difference; it has found that the difference, if there is one, is smaller than the test could see. Those two readings license opposite actions, and only the second one is available.

This is where the floors earn their keep a second time. If the floor for a rule is 98.5% and the eval cannot resolve a difference smaller than two points at the case count you can afford, then the eval is not the instrument that enforces that floor, and something else has to. The production canary in the gate diagram — a standing sample of live answers scored against the same predicates every day after the change has shipped — is that instrument.

Sizing it is counting, not modelling. A daily sample of 200 answers at a baseline citation rate of 99.1% fails on whatever that rate leaves over, which is nine tenths of a percent, and nine tenths of a percent of 200 is an expected 1.8 failures a day. At the regressed rate of 91.6% the failure rate is a little over eight percent, and eight-and-a-bit percent of 200 is 16.8. Those two expectations are what a threshold has to separate, and they are nearly ten times apart, which is why a threshold works at all here.

Alarming on six or more failures in a single day sits well above the first and well below the second. Under the baseline a day that bad happens roughly one time in a hundred, so the alert cries wolf about once a quarter; at the regressed rate a day that good is so unlikely that the alert fires with probability above 0.999. A regression that took twenty-three days to notice would have raised an alert on day one.

The sensitivity scales the way the eval's did, and for the same reason. A regression from 99.1% to 98.0% doubles the failure rate rather than multiplying it by ten, so the daily expectation moves only from 1.8 to 4.0 — two distributions that overlap heavily, and no single-day threshold separates them cleanly. What separates them is more observations, because the spread of a count grows with the square root of the sample while the gap between the two rates grows with the sample itself. Seven days of 200 is 1,400 observations, which put 98.0% about three standard errors below baseline and make it visible within a fortnight. Choosing the window is choosing how long you are willing to serve a regression, and writing that number down next to the floor is what makes the floor enforceable rather than aspirational.

What a fixed set is fixed against#

The fixed set is fixed, which is its virtue for comparing two layouts and its limitation for everything else. It cannot tell you that retrieval started returning longer chunks last Tuesday and pushed the citation rule from token 7,300 to token 9,100, because the eval's documents are the ones captured when the set was built. A layout that passes the gate is a layout that was safe against the input distribution of the day it was measured, and prompts drift by having their inputs change underneath them.

Replaying real traffic is what covers that gap, and the capability is worth building before the week you need it, because the alternative is what the team did for the first two weeks, which is to reason about production from aggregate rates. With the three fields in the log, any production incident becomes a paired eval against a case list drawn from real traffic, and that is stronger evidence than the fixed set, because the inputs are the ones that failed rather than the ones somebody chose in advance.

It also says nothing about the failure mode that shows up only in multi-turn sessions, since the case list is single-turn, and nothing about how the layout behaves at the 99th percentile of context length, unless somebody deliberately put those cases in. Those gaps are what the canary is for, and the division of labour between the two instruments is worth stating: the eval clears a change against a distribution you chose, and the canary watches the distribution you actually get.

A reordering is a prompt change, then, and the practical content of that sentence is a scorecard rather than a principle. Score both layouts on one case list at production temperature, against predicates with floors written down first, and report hit rate and effective tokens on the same sheet as compliance — because the pull request that moves a block is asking to trade one for the other whether or not it says so.

The research assistant's team now has a scorecard that says the layout they are running costs 7.2 points of citation compliance and saves $1,532 a day, and reverting gives back the compliance and the money. That is a genuine dilemma only if the two orderings are the only options available, and they are not. The next chapter is the repair that keeps the prefix and puts the rules back where the model reads them, for a few dozen tokens per request.