More context can lower accuracy, not just raise the bill

Filling a context window changes two things at once, and only one of them is the bill. The other is what the model has to do with what it has been given. Both changes are worth measuring before a design decision rests on either, and the case below measures them on one workload: same 300,000-token corpus, same 240-question evaluation set, same grader, same model, with the only change being that an assistant stopped retrieving eight ranked chunks and started receiving the corpus whole.

Accuracy went from 74.2% to 68.3%. The corpus contained the answer to every question in both conditions, and the condition that contained more of it did worse by 5.9 points. Two words do specific work in everything that follows and are worth pinning down first. Recall here is the share of answerable questions whose supporting evidence the model actually brought to bear, which is a property of what reached the prompt. Dilution is what happens to the model's job once material that resembles the evidence reaches the prompt alongside it, which is a property of what else is there. The first is what stuffing fixes. The second is what stuffing causes.

Where the six points went#

An aggregate score does not tell you anything you can act on, so the first move is to bucket the evaluation items by where the supporting evidence sits in the assembled context. That requires knowing, per question, which passage answers it and what character offset that passage lands at once the corpus is concatenated — which is a property of your evaluation set that you have to record deliberately, because nothing else will record it for you.

def evidence_position(assembled: str, gold_span: str) -> float | None:
    """Fraction of the way through the prompt at which the gold evidence sits."""
    idx = assembled.find(gold_span)
    return None if idx < 0 else idx / len(assembled)

for item in eval_set:
    prompt = assemble_corpus(item.corpus_docs)
    item.position = evidence_position(prompt, item.gold_span)
    item.bucket = next(b for lo, hi, b in BUCKETS if lo <= item.position < hi)

Bucketing the 240 items gives a shape that the single number hides completely.

Three properties of that table are worth checking before its rows are used to argue anything, because each one licenses a different move. The item counts add to 240 with nothing left over, so the buckets are a partition — every question sits in exactly one of them, and the bottom row is a weighted average of the five above it rather than a sixth measurement. Weighting the five retrieval cells by their item counts and averaging reproduces 74.2% exactly. Doing the same to the stuffed cells lands within a fifth of a point of 68.3%, and that gap is what five cells rounded to whole numbers cost when they are averaged into a figure carried to one decimal place.

The third property is that the Δ column is the stuffed cell minus the retrieval cell in every row, the bottom one included. It is therefore a view of the other two columns rather than evidence of its own, and a reader who treats a large Δ as a third reading is counting the same measurement twice.

The retrieval column is flat, which is what makes the stuffed column interpretable. Under retrieval, "position" is a property of the source document and has nothing to do with the prompt, because whatever chunk gets retrieved lands within the first five thousand tokens regardless of which page it came from. So the flat column is the control: these question groups are not of differing difficulty, and any variation in the stuffed column is produced by the assembly, not by the questions.

How much weight that control carries is not the same in every row, and the difference matters. The retrieval cells span three points across the five buckets, which on buckets of twenty and thirty-four items is inside the range that re-drawing the same questions would produce anyway. The control is therefore weakest exactly where the buckets are smallest, at the two ends, and strongest on the 96-item middle bucket. That is where the finding is, which is the fortunate arrangement rather than the usual one. A thirteen-point effect sitting on a bucket of twenty would not have survived the same check.

The stuffed column is not flat. Evidence near either end is recovered slightly better than retrieval manages, and evidence in the middle 40% of the window is recovered thirteen points worse. That is the position effect from context degradation showing up as a budgetable number rather than a warning: filling the window did not merely waste tokens on irrelevant text, it actively cost recall on the subset of questions whose answer happened to land in the middle. Since 40% of the items fall in that bucket, a thirteen-point loss there is enough to swamp the small gains at the edges.

Where the six points went has an exact answer, and it is worth writing out because it decides what is worth fixing. Each bucket moves the headline by its share of the items multiplied by its own Δ, and those five products add back to the headline. The middle bucket's share is forty per cent of the items and its Δ is thirteen points, so on its own it moves the total by more than five of the six points. The other four buckets together move it by less than one, because their two gains cancel about half of their two losses. The reconstruction comes out at a little over six points where the headline says 5.9, and the tenth of a point between them is the same rounding residual as before.

What that licenses is a hard statement about leverage. An intervention that recovers the middle of the window recovers essentially the whole regression; an intervention that improves the two ends is capped at under a point however well it works. That is a rare thing to be able to say from an evaluation, and it is available only because the buckets partition cleanly and the control column is flat.

Two different quantities in that table are both 40%, and fusing them is the easiest mistake on the page. The middle bucket spans 30–70% of the prompt, so it is 40% of the window measured by position. It also holds 96 of 240 items, so it is 40% of the evaluation set measured by count. Those two agree here because evidence positions in this corpus happen to be spread almost evenly across the concatenation — and the end buckets show that they are not spread perfectly evenly, since the first holds more items than its width alone would give it and the last holds fewer. The width share is fixed by where the bucket boundaries were drawn. The count share is a property of the corpus.

Two consequences of that position effect follow, and both are easy to miss. The damage is a function of where the corpus put the evidence, which is an arbitrary property of document ordering, so re-sorting the corpus alphabetically instead of by upload date will move which questions fail without changing how many. And because the degradation is concentrated, the aggregate number understates the risk for any user whose work happens to concentrate in the affected region — a compliance team that only ever asks about the two documents that sit in the middle of the concatenation experiences this feature as substantially worse than the evaluation says.

The first of those holds in expectation rather than exactly, and the count share above is what would move. A re-sort redistributes items across the buckets, and a particular ordering can leave more or fewer of them in the middle band than the last one did. What is stable across orderings is the per-bucket effect; what is not stable is how many questions each bucket catches. An evaluation quoted without its ordering is quoting one draw.

Dilution is separate from position, and it is bigger#

Slicing the same 240 items by how many distinct passages the answer requires separates a second effect.

Single-passage questions barely move. Questions needing several pieces of evidence combined lose nine to twelve points, and the mechanism is different from position: the corpus contains dozens of passages that discuss the same subject in nearly the same words, and picking the three that actually bear on the question is a discrimination problem that gets harder as the number of candidates grows. A reranked top-8 has already performed that discrimination with a model trained for it. A stuffed window asks the answering model to do it inline, as a side effect of generating, over 300,000 tokens of candidates.

That table repays being read as ratios rather than as differences, because a per-passage penalty compounds multiplicatively and differences hide compounding. Divide each stuffed cell by the retrieval cell beside it. The one-passage row keeps most of what retrieval got, the two-passage row keeps noticeably less, and the three-or-more row less again.

Now suppose each required passage carried its own independent chance of being lost to the competition. Then the ratios would compound: a two-passage question would keep the square of what a one-passage question keeps, and a three-passage question the cube, because independent events multiply. Squaring the one-passage ratio gives a number comfortably above the two-passage ratio the table actually reports, and cubing it gives one well above the three-or-more ratio. The observed loss is worse than independence predicts.

There is a reason for that, and it is the mechanism above sharpened. The passages a multi-passage question needs are usually about the same subject as one another, which makes them each other's closest competitors as well as competitors against every other near-duplicate in the corpus. Independent losses would have been the optimistic case. Here they reinforce, so the penalty per required passage grows with how many are required.

Two caveats hold that conclusion in place rather than weakening it. The bottom row is a bucket of three or more, so its ratio mixes questions with different passage counts and the cube is only an approximate benchmark for it. And the one-passage ratio is contaminated by the position effect, which does not compound with passage count at all — so the true per-passage selection penalty is smaller than the one-passage row implies, which would push the predicted two- and three-passage ratios higher still and widen the shortfall. Both caveats push the conclusion the same way.

The advantage a well-tuned retrieval pipeline sometimes has over a long window on quality, rather than only on cost, comes from that selection step. The smaller volume of evidence it sends is incidental; the operative difference is that a model trained to rank has already chosen between the candidates before any of them reach the prompt. The stuffed design's advantage is that it cannot miss; its disadvantage is that it cannot select. Which one dominates is an empirical question about your corpus, and it turns on how much near-duplicate material the corpus contains. A corpus of 300,000 tokens on twelve unrelated subjects loses very little to dilution; a corpus of 300,000 tokens of forty versions of the same policy loses a great deal.

Recall saturates and selection does not#

The two tables above are two terms of one product, and putting them together predicts the shape of every budget sweep in this topic without needing another evaluation run.

The first term is recall: the probability that the passage carrying the answer is in the prompt at all. Retrieval can fail to include it. Stuffing cannot. Recall therefore rises as more tokens are sent, and it is bounded above, because once the passage is in the window sending more tokens cannot put it there a second time. Recall saturates, and the closer it already is to its ceiling the less another thousand tokens can buy.

The second term is selection: given that the passage is present, the probability that the model answers from it rather than from something else that looked like it. Every extra token of corpus brings extra passages, and some share of them discuss the same subject in similar words. Call that number the competitor count. Selection gets harder as the competitor count rises — a model that could not tell them apart at all would be choosing one candidate out of the competitors plus the real one — and the competitor count has no ceiling short of the whole corpus. Selection does not saturate.

Accuracy is the product of the two, because an answer needs the passage present and needs it chosen. So whether another thousand tokens helps is a race between the proportional gain in the first term and the proportional loss in the second. Early on, recall is well below its ceiling and each extra token buys a real chance of including something that was missing, so the gain wins. Once recall is near its ceiling the gain is close to nothing, while the loss continues at whatever rate the corpus's redundancy sets. The balance therefore tips exactly once, which is why accuracy against window size has a single interior maximum rather than a plateau or a monotone climb. Nothing in that argument depends on which model is answering. What depends on the corpus is where the tip falls.

That product is also where the tempting wrong belief lives, and it is worth naming precisely. In the stuffed condition recall is pinned at its ceiling by construction: the corpus contains the answer to every question, and the whole corpus is sent. A team watching retrieval recall as the guard on quality therefore reads a constant maximum through the entire six-point regression, because the quantity being watched cannot move, while the quantity that broke — selection among competitors — has no meter on it at all. A defence that holds one number at its best value while the dangerous property varies freely underneath it is the most convincing kind of defence and the least useful.

The tail is where the latency argument actually happens#

Prefill time scales with tokens in the window, so time to first token — the wait between sending the request and the first word of the answer appearing — separates the two designs by more than an order of magnitude before any caching is applied. The two percentiles below are how that wait is normally reported: p50 is the median request, and p95 is the slowest one request in twenty, which is the figure users are describing when they say a feature is slow while the average looks healthy. Measured on the same model and hardware tier, with 180-token answers generated at about 90 tokens per second:

The generation row is the only one in that table that is not a measurement. 180 tokens at about 90 tokens a second is 2.0 seconds, and it is 2.0 seconds in all three columns because none of the three changes how long the answer is. That has two consequences. The end-to-end row is then an exact sum in every column — 0.9 and 2.0 giving 2.9, 6.2 and 2.0 giving 8.2, 21.3 and 2.0 giving 23.3 — with no residual anywhere. And the addition is legitimate, which adding percentiles usually is not: a p95 plus a p95 is only a p95 when the same requests are slowest in both, and here the second term is a constant, which is the degenerate case where the objection cannot arise. Read the column by its gaps and the constant drops out, leaving the gaps as pure prefill.

A cache read is roughly three and a half times faster to first token than a cold read of the same 300,000 tokens, because the provider skips the attention computation over the cached span and pays mostly for transferring the stored state. It is not free, and it does not approach the 0.9 seconds that a five-thousand-token prompt achieves, because 300,000 tokens of state still has to be materialised.

That factor turns up twice, at two percentiles that share no step with each other. The median and the slowest one in twenty are different questions — one asks what a typical request waits, the other what the unlucky request waits — and cold divided by warm comes out at roughly three and a half in both. Two instruments agreeing licenses treating the warm row as the cold row divided by a constant, which is a much stronger statement than either row alone: the cache scales the whole prefill distribution rather than clipping its tail. What it does not license is carrying that constant to a different window size, because the transfer the warm read still pays for grows with the stored state while the attention computation it skips grows faster.

The number that ends up in the incident review is the population p95, and it lands somewhere none of those three columns show. At a 12% cache miss rate — which chapter 3's arrival profile produces comfortably, and which is a good hit rate — the slowest 12% of requests are the cold ones, so the 95th percentile of the mixed distribution sits inside the cold-read tail at around 12 seconds. Against retrieval's 2.9 seconds that is a factor of four, produced entirely by the interaction between a tail statistic and a miss rate that looked excellent on the cost dashboard. Improving the hit rate from 88% to 97% moves the p95 more than any prompt change will, because it moves the cold reads out of the percentile rather than making them faster.

The rule underneath that is worth having in general form, because it is counter-intuitive in a way that survives being told once. A percentile of a mixture is not a mixture of percentiles. When every miss is slower than every hit, the requests above a given percentile are drawn from one population or the other and never from a blend of both, so the reported figure is a point taken from the cold distribution alone, at a place inside it given by the size of the tail being reported divided by the miss rate. The tail above a p95 is one request in twenty. As long as the miss rate is larger than that, the p95 is a cold request and no amount of improving warm-read latency touches it. Drop the miss rate below one in twenty and the same tail stops reaching the cold reads altogether — the p95 becomes a warm request, sitting just above the warm column's own p95. Nothing about the cold reads changed; they simply stopped being reported.

There is a third cost of a full window that shows up as a bug report rather than a metric. Input and output share the same budget, so a 300,000-token corpus in a 400,000-token window leaves 100,000 tokens for instructions, conversation history and the answer, and a thirty-turn session with 800-token exchanges consumes a quarter of that headroom before anything goes wrong. What the user sees is an answer that stops in the middle of a sentence, or a request rejected outright at turn twenty-six of a conversation that worked fine at turn ten. Whatever assembles the prompt has to reserve output space explicitly and drop context to preserve it, which is one more thing the retrieval design gets for free because its prompt never approaches the ceiling.

The erosion in that paragraph is monotone, which is what makes it a schedulable problem rather than a mysterious one. Headroom is consumed turn by turn and never returned, so the turn at which a session becomes unservable follows from the exchange size and the reserved output space, and both are known before a single user arrives. A design that reserves nothing has not avoided the question; it has moved the answer into production.

What a needle benchmark certifies#

A synthetic needle is a verbatim, lexically distinctive sentence — "the best thing to do in San Francisco is eat a sandwich in Dolores Park" — inserted into filler that shares none of its vocabulary. Finding it requires matching a string against a haystack that contains nothing similar to it. A 98% score at 200,000 tokens tells you the model's attention reaches the whole window, that the context length in the model card is not a fiction, and that if a fact appears once, verbatim, in distinctive language, it is findable wherever it sits. That is a genuine and non-trivial capability, and it is a precondition for everything in this topic.

It tells you nothing about the three-or-more-passages row of the second table, where accuracy fell twelve points. Real questions need several dispersed pieces of evidence, reasoning over them, and discrimination between passages that resemble each other closely — which is the opposite of a needle's defining property. Models that score near-perfectly on needles routinely lose exactly those points, and the discrepancy is well enough documented that there are benchmark suites built to expose it; U-NIAH, which runs controlled comparisons of stuffing and retrieval with failure analysis, is a reasonable place to look at what the harder variants measure. The practical rule is that a needle score is a sanity check on the plumbing and never evidence about your workload, so a design decision that cites one has skipped the measurement rather than made it.

The needle score and the pinned recall figure are the same defence twice, which is why both were convincing. A needle benchmark holds the competitor count at zero by construction — that is what makes a needle a needle — and stuffing holds recall at its ceiling by construction. Each measurement therefore sits at its best possible value and stays there while the corpus's redundancy, the one property that actually moved, varies underneath. Any quantity that is pinned by the design of the measurement rather than by the health of the system is worth treating as decoration until something is shown to move it.

Where the finding stops applying is set by the same term. A corpus with little near-duplicate material has a small competitor count at every window size, so its selection term barely declines and stuffing costs it almost nothing in quality; the argument against stuffing there is the bill and the tail, not the score. The steeper the redundancy, the earlier the interior maximum falls, and the redundancy is a property nobody chose — it is what the organisation happened to write down.

The finding to carry out of this is the product. Sending a corpus whole makes the recall term perfect and hands the selection term over to whatever the corpus's own redundancy does to it, which is why a change that cannot lose evidence can still lose answers, and why the loss arrived concentrated rather than spread — the middle of the window and the multi-passage questions are where selection was already hardest. Every defence available before the measurement was a reading of the recall term: guaranteed inclusion, and a needle score at the full window length. Neither could have moved, and neither was about the thing that broke.

Both the position effect and the dilution effect are functions of how many tokens are in the window, which means neither is a fixed property of the corpus. Chapter 5 uses that: select a subset of a corpus too large to fit, put the subset in a large window, and cache it, which trades a little of retrieval's selectivity for a lot of the stuffed design's recall — and moves the prefill term that this chapter showed is the only lever on a p95 regression.