Proving a long-context design is at least as good, before you switch

Two teams presented results for the same migration in the same week. The platform team reported that long context beat retrieval by 4.1 points and recommended the switch. The search team reported that retrieval beat long context by 2.3 points and recommended against it. Both had run 240 questions, both had graded with a model — a grader, meaning a second model prompted to score each answer against a written rubric of what counts as correct, since scoring eight hundred answers by hand is not something anyone does twice — and both were confident. Each had left one item unpinned, and neither experiment measured what its author believed it measured.

The two designs are worth naming plainly. A retrieval arm selects a small number of passages that look relevant to the question and puts only those in the prompt. A long context arm puts the documents themselves in the prompt and lets the model do the selecting. Cost, latency, correctness and what happens when a document changes all follow from that one difference in how the prompt gets assembled, which is why an experiment has to make it the only difference there is.

What has to be identical for the comparison to mean anything#

A comparison of two prompt-assembly strategies is only a comparison of two prompt-assembly strategies if every other input is pinned. Writing the list down before the run is cheap; discovering an unpinned variable afterwards costs the whole experiment, because there is no way to subtract it out.

The reason it cannot be subtracted out is arithmetic rather than a warning. The number a comparison reports is the true difference between the two designs plus one term for every input that moved along with the arm label. If the long-context arm ran on a newer model and the retrieval arm did not, "newer model" and "long context" are the same column in the data: no run has one without the other, so the model term cannot be estimated and cannot be removed. An unpinned variable does not add noise that averages away with more questions. It adds a fixed, unmeasurable offset that more questions estimate more precisely.

The last row catches more people than the rest combined. Running arm A for an hour and then arm B for an hour makes the latency comparison a comparison of two hours of provider load, and provider load is not a constant. Interleaving — question one to both arms, question two to both arms — removes it entirely at no extra cost.

Removes it entirely is a strong claim, and worth seeing why it holds. Every request's latency carries whatever the provider's load was when it ran. Run the arms in blocks and each carries a different average load, so the reported difference is the design difference plus the difference between two hours of somebody else's traffic. Interleave them and consecutive requests sample the load at almost the same instant, so the load term is identical on both sides of the subtraction and cancels. What cancels is the difference, and only the difference: neither arm's absolute p95 becomes a trustworthy prediction of production, which will run at whatever load it runs at. Interleaving licenses the comparison and not the numbers being compared.

Grader blinding deserves the same attention. Model graders have a documented preference for longer, more hedged answers, and a long-context arm produces longer answers on average, so an unblinded rubric that rewards thoroughness measures answer length. Strip any marker of which arm produced an answer, shuffle the presentation order, and check the grader against a human-labelled subset of fifty items before trusting it on the other eight hundred.

The grader is an instrument, and instruments get calibrated#

Holding the grader constant makes it a fair judge of both arms only if it is not systematically wrong in a direction that one arm exploits. Stability and accuracy are separate properties, and a rubric that reliably overrates hedged answers will reliably hand points to whichever arm hedges more.

This is the step easiest to believe you have covered. The defence sounds airtight: one grader, one snapshot, one rubric text, applied to both arms, so whatever it gets wrong it gets wrong equally and the error cancels. It cancels only if the thing the grader is wrong about is present in equal measure in both arms — and here the grader is held constant while the property it is biased toward, answer length, is precisely the property the designs differ on. A constant instrument removes drift, the error that changes between measurements. It does nothing about bias, the error that stays put, and bias survives a subtraction whenever the two things subtracted present differently to the instrument.

Calibrating it takes fifty items and about three hours. Draw them stratified across the question types in the set, have two people label each one blind to arm and to the grader's opinion, and compare. On this migration the two humans agreed with each other on 46 of 50, which sets the ceiling for what the grader can be asked to achieve.

That ceiling follows from what a disagreement between two careful labellers means. On the four items where they diverged there is no fact for an instrument to be right about, only a judgement settled by argument, so a grader scored against a manufactured consensus is scored partly against a coin somebody else called.

The grader agreed with the human consensus on 43, a Cohen's κ of 0.70, that being an agreement score corrected for how often two judges would coincide by luck alone, where 1.0 is perfect agreement and 0 is no better than chance, which sounds acceptable until the seven disagreements are read individually. Six of the seven were the grader marking correct an answer both humans marked wrong, and five of those six came from the long-context arm.

The correction is what makes that κ smaller than the raw agreement suggests. Two judges who both mark most answers correct land on the same verdict often without exercising judgement at all, because "correct" is the common answer. κ subtracts the agreement that pattern alone produces and divides by the room left above it, so a raw agreement in the middle eighties sits at 0.70 once the free agreement comes out.

The mechanism was visible once the answers were laid side by side. Long-context answers averaged 210 words against retrieval's 120, and the extra length went into enumerated qualifications — "if the request was filed before the deadline, the 30-day window applies; for late filings the review period may extend" — where one clause was right, one was invented, and the rubric's instruction to judge whether the answer "addresses the question correctly" let a model grader resolve the mixture generously. Retrieval's shorter answers offered the grader nothing to be generous about.

Five of six lenient calls landing on one arm is a lean rather than a finding, since six items would split three and three under a rubric with no arm preference and a five-to-one split is well inside what a fair coin produces. What raises it above a lean is a mechanism identified first and then checked by a measurement sharing none of its steps.

Two rubric changes closed most of the gap. The grader was required to quote the specific span of the answer it was scoring, which forces it to commit to one claim rather than to an impression of the whole. Unsupported qualifications were scored as errors rather than as neutral colour, on the grounds that a user reading them will act on them. Re-running all 820 items under the revised rubric moved the long-context arm from 76.5% to 74.3% and the retrieval arm from 71.4% to 71.2%, shrinking the reported difference from 5.1 points to 3.1. Nothing about either system changed; two of the five points had been an artefact of how the answers were read.

Those four figures close on each other exactly, and the closure carries the attribution. The long-context arm lost a little over two points, the retrieval arm lost two tenths, and the gap lost the difference between those movements — the two points it shed, with no residual. Had both arms fallen by the same amount, the revision would have been a uniform re-scaling, every answer marked more harshly and the gap untouched, and the re-run would have said nothing about which arm the old rubric favoured. The retrieval arm barely moving is the control that makes the long-context arm's movement mean something, and the ratio between the two is about ten to one. A fifty-item hand audit and an eight-hundred-item re-run share no step, and they name the same arm.

What the closure does not license is the 3.1 points that survived. The instrument producing them still agrees with human consensus at 0.70 against a ceiling of forty-six in fifty, and a second bias neither revision touched would be invisible to this test, whose only evidence is that the arms moved unequally under a change aimed at the first. The fifty human labels are worth re-collecting whenever the rubric text or the grader snapshot changes, since both alter the instrument. They are also the only part of this comparison that cannot be automated, and their absence is the most common reason two careful teams produce incompatible numbers from identical systems.

Four numbers, reported together#

Reporting accuracy alone is what produced the two contradictory recommendations, because a single number invites a single verdict. The decision needs four, and they have to appear in the same table so that nobody can quote one without the others.

Effective input tokens rather than raw input tokens, because chapter 2's multipliers — the different rates a provider bills for writing a cache entry, reading one and reading nothing at all — are what the invoice applies. Measured hit rate rather than assumed, because chapter 2's team believed theirs was 100% and it was 4%. And p95 rather than mean, because, as chapter 4 of this topic showed, the mean hides the cold reads that the tail is made of.

The table is read down the gaps rather than across the levels. Correctness differs by three points and a bit, effective input tokens by not quite sevenfold, and the p95 figures by exactly a factor of four. Read as levels, the rows invite the question "is 74.3% good", which no one can answer; read as gaps, they ask what three points is worth against four times the wait, which a product owner can hold an opinion about.

The interval on the difference, and the margin it has to clear#

An accuracy estimated from a finite set of questions would come out somewhere else on a different set drawn the same way, and the amount it would move can be worked out in advance. For one arm answering n items with accuracy p, that wobble — the standard error — is p times one-minus-p, divided by n, all under a square root. It shrinks with the square root of n rather than with n, which is the fact governing every sample-size decision below.

A difference between two independently measured arms wobbles more than either arm does, because both estimates are wobbling and their errors do not know about each other. The variances add, so the standard error on the difference is the square root of the sum of the two arms' squared standard errors. With 240 items and accuracies near 70%, that comes to about 4.6 points, and the 95% confidence interval on the difference — the range the true difference plausibly lies in — is roughly two standard errors each side, about ±9 points. A reported 4.1-point win is indistinguishable from a 4.1-point loss. That 4.6 is the conservative end of the calculation, since p times one-minus-p is largest when p is one half and falls away in both directions; it errs toward saying the experiment resolves less than it does, which is the direction a power calculation should err in.

The interval that matters is the one on the difference, not two intervals on two arms compared by eye. Arms whose intervals overlap can still differ significantly, and arms whose intervals do not overlap differ by more than significance requires, so reading overlap off a chart of two error bars gives the wrong answer in both directions. The difference is a quantity in its own right, with its own interval, and that interval is the only one the decision consults.

Pairing helps substantially, since both arms answer the same questions and item difficulty cancels. The paired test looks only at the questions where the arms disagree.

The cancellation is exact and it happens item by item. Score each arm on each question as one or zero and take the difference per question rather than per arm. A question both arms answer correctly contributes one minus one; one both get wrong contributes zero minus zero. Either way the contribution is zero, whatever made that question easy or hard, so every property of the question itself drops out before any averaging happens. Only questions where the arms disagree contribute anything, each of them plus one or minus one. The difference in accuracy is therefore the imbalance between the two disagreement counts divided by the number of questions, which is why the table below has four cells and only two of them matter.

Forty-seven discordant items out of 240, a 13-item imbalance, McNemar's χ² of 3.06 and p ≈ 0.08. McNemar's test is the one built for paired pass-or-fail outcomes: it ignores the questions both arms got right or both got wrong and asks whether the imbalance among the disagreements is larger than chance would produce, and a p of 0.08 says an imbalance this large would turn up about one run in twelve even if the two designs were identical. That is not a result; it is an observation that the arms differ on a fifth of the questions and that the direction of the imbalance is not yet distinguishable from a coin.

Every one of those figures comes out of the four cells with nothing left over. The cells sum to the 240 questions. The two off-diagonal cells sum to the forty-seven discordant ones, and their difference is the 13-item imbalance. The statistic is that imbalance reduced by one, squared, and divided by the discordant count, and it lands on 3.06 exactly. Reducing the imbalance by one before squaring is a continuity correction: the counts are whole numbers while the distribution the p-value is read from is smooth, and without it the same cells give a larger statistic and a smaller p. The correction stops a test built on forty-seven items claiming more precision than forty-seven items contain.

The direction of that imbalance is worth reading off the table rather than assuming. The cell for questions retrieval got right and long context got wrong holds 30; the reverse cell holds 17. This 240-item run leans toward the retrieval arm, while the eight-hundred-item table above has long context ahead by 3.1 points. The two are not in conflict; the tension is the section's own claim arriving as evidence, since at 240 items the sign is not established and a later, larger run landing on the other side of zero is what "not established" predicts.

The same derivation exposes something easy to miss, because it moves a number without any system changing: adding questions both arms answer correctly leaves the imbalance untouched and raises the denominator, so the gap measured in percentage points shrinks. Reporting the gap in points is a design decision about the metric, and the requirement underneath is only that one design beat the other — the two come apart the moment somebody adds straightforward questions to make the set more representative.

Reaching ±2.5 points at this discordance rate takes roughly 800 items, which is why the table above says n=820 — the team wrote 580 more questions, which took two people about a week and was the single highest-value activity in the whole migration.

That target is met in the reported table, at ±2.4, and getting there took more than the extra questions. The interval shrinks with the square root of the item count, so 240 items to 820 is under a fourfold increase and buys under a twofold reduction. Pairing supplies a further chunk, since the paired interval starts smaller than the unpaired ±9 above. The remainder is the enlarged set itself, recorded below as 41% multi-passage: a set on which the arms disagree less often has fewer discordant items per question and so a narrower interval. Two of those three contributions are calculable and the third is not, so "roughly 800 items" holds on the condition that the enlarged set discords at a lower rate than the pilot did.

None of this yet licenses the words "at least as good". Failing to find a difference is not the same as finding no difference: a wide enough interval is compatible with the new design being substantially worse, and the wider the interval the more comfortably it hides a real loss. The claim that licenses a switch is narrower and checkable. It is that the lower end of the interval on the difference sits above a margin agreed in advance — a margin saying how much accuracy the organisation will trade for whatever else the new design buys. Choose that margin after seeing the interval and the interval chooses the margin, since there is always a threshold the observed lower bound clears. It has to be a number somebody committed to while the outcome was still unknown, which is the only condition under which it can fail.

Better accuracy and a p95 four times worse#

Take the result in the table at face value: long context wins by 3.1 points and loses on p95 by a factor of four. There are four things to try before treating that as a verdict, in order of how much they cost to try.

Check the prefix layout first, because it is an hour of work and the measured hit rate is the diagnostic. At 88.4% the layout is basically right; had it come back at 4%, the fix from chapter 2 would have changed both the cost and latency columns before any harder question needed answering.

Then split the latency distribution by cache state, which the harness already records. If p95 conditional on a cache hit is 6.2 seconds and conditional on a miss is 21.3 seconds, the population p95 is being set by the 11.6% of requests that miss, and the productive move is raising warmth rather than shortening prompts: a longer TTL tier, prefix-affinity routing so that requests sharing a prefix land on the same shard, or a warming call at session start that pays a write in exchange for every subsequent turn in that session reading warm. Getting the hit rate to 97% moves the 95th percentile out of the cold region entirely, and no prompt change achieves that.

Two figures there are derived rather than measured. The miss fraction is not a separate counter; it is one minus the measured hit rate, so 88.4% warm implies 11.6% cold with nothing to reconcile. And 11.6% is not the 11.6 seconds in the table above — one is a share of requests, the other a latency, and they share their digits by coincidence.

The rest follows from what a percentile is. Assume, as the two conditional figures strongly suggest, that a cold request is slower than very nearly every warm one. Then the slowest 11.6% of requests are the misses, and the 95th percentile — which by definition has only the slowest five per cent above it — lies inside that miss region. The population p95 is therefore a percentile of the misses, landing a little past their halfway mark, which is why 11.6 seconds sits so far below the misses' own 21.3-second figure. The rule that falls out compares two numbers and nothing else: the 95th percentile is a warm-path figure exactly when the miss rate is below five per cent, because five per cent is what the 95th percentile leaves above it. At 11.6% cold the tail is entirely cold; at the 97% hit rate suggested above the cold share drops under five per cent, the top of the distribution is warm again, and p95 falls toward the 6.2-second warm figure without a token leaving the prompt. The assumption fails on a system with a slow warm path, where the tail is a mixture and raising warmth moves p95 less than this predicts.

Third, reduce tokens in the window using chapter 5's design, which retrieves into the window and caches what it retrieved, and which is the only lever that acts on prefill time itself rather than its constant factor. That is a week of work and it usually moves p95 by more than the other two together.

Fourth, check that p95 end-to-end is the metric the product actually cares about. If answers stream, the user-visible number is time to first token, and if the interface shows retrieved sources while the answer generates, the perceived wait is different again. Measure the product outcome — abandonment rate, time to task completion — on a small live slice before optimising a percentile that no user experiences. This is the step most likely to reveal that the latency objection was about a number on a dashboard rather than about a user.

Write down what would settle it, before you run it#

The reason the two teams argued is that neither had stated in advance what result would change their mind, so both interpreted their evidence toward the position they already held. A short pre-registered record fixes that, pre-registration meaning the decision rule is written down and agreed before any data arrives, in the sense the word carries in clinical trials, and it becomes the artefact chapter 8 relies on when the next model release reopens the question.

comparison: policy-assistant-long-context-vs-retrieval
frozen:
  model: model-2026-04-17          # dated snapshot, both arms
  temperature: 0.0
  eval_set: policy-eval-v3 (820 items, 41% multi-passage)
  corpus_hash: 3b9f...c07
  grader: judge-2026-02-11, rubric v4, blind, order-shuffled
decision_rule:
  switch_if:
    - correctness_delta_lower_ci: ">= +1.0 pts"
    - p95_end_to_end: "<= 5.0 s"
    - cost_per_1k_queries: "<= 1.6x retrieval"
    - measured_cache_hit_rate: ">= 0.95"
  reject_if:
    - correctness_delta_upper_ci: "< 0 pts"
  otherwise: "iterate on the failing dimension; do not ship on partial wins"
review_trigger: [model snapshot change, corpus > 2M tokens, price change]

Written this way, the observed result reads unambiguously: put each measured figure next to the line agreed for it. The correctness clause asks for the lower end of the interval on the difference, not the difference itself. The table reports +3.1 points with a 95% interval of ±2.4, so that lower end is a little over half a point above zero and the clause asked for a full point. Correctness does not clear its bar. Nor does it trigger the rejection clause, which fires only when the upper end falls below zero, and that end is above five points. The p95 clause asks for 5.0 seconds and the measurement is 11.6. The cache clause asks for 0.95 and the measurement is 88.4%. The cost clause names a cost per thousand queries the four-number table does not carry — it reports effective input tokens instead — so that criterion cannot be evaluated from this report, a defect in the report rather than in the rule.

Three clauses fail outright, the fourth cannot be answered, and the otherwise branch applies: iterate on the failing dimension, do not ship on partial wins. The switch does not ship and the work is on warmth and window size. Nobody has to relitigate whether 3.1 points is worth 11.6 seconds, because that trade was priced before anyone knew which way the numbers would fall. A 3.1-point headline reads as a win in a summary slide, and the same 3.1 points with its interval attached does not reach the bar the team set while the answer was unknown; both descriptions are true of one measurement, and only the rule stops them being argued into different decisions.

Comparisons too small to resolve their own headline#

The uncomfortable corollary is that most published internal comparisons of these two designs are underpowered and their headline differences are noise. If your evaluation set is 150 questions and the arms are three points apart, the honest report is that they are indistinguishable on quality, and the decision falls to cost, latency and freshness, which are measured with far less uncertainty because a token count and a latency are read off counters rather than estimated from a sample. The cost clause going unanswered above generalises too: a criterion naming a quantity nobody instrumented reads at review time as though it passed, since nothing contradicts it. Treat a missing measurement as a failure to answer rather than as an absence of objection.

What the comparison still cannot tell you is how either design behaves outside the distribution the evaluation set was drawn from. Both arms were measured on a corpus of a fixed size, documents that did not change mid-run, and questions written by people who wanted the feature to work. Each is a condition on the result rather than a caveat about it: the 3.1 points holds for a corpus of that size, for documents stable across a run, and for questions a friendly author wrote. Chapter 7 measures what happens when the corpus is larger than the ceiling, when it changes underneath the cache, and when a document in it contains instructions addressed to the model.

Approval for a switch is a claim that one design is at least as good as another, carried by a single interval on a single quantity — the difference between the arms — measured where nothing but the prompt-assembly strategy varied, graded by an instrument calibrated against human labels, and judged against a margin agreed while the answer was unknown. Every part of that sentence is something an experiment can get wrong quietly. An unpinned model puts an unmeasurable offset in the arm column. A constant but biased grader hands points to whichever arm presents the way it likes. Two hundred and forty questions cannot resolve a five-point difference. Four numbers in one table stop any of it being quoted selectively, and a pre-registered rule stops the reading of them being negotiated afterwards: here correctness fell short of its margin, latency and warmth failed theirs, cost went unmeasured, and the design stays as it is while the hit rate and the window size get the work.