Needle-in-a-haystack and multi-fact integration

Two curves about long context circulate at the same time and appear to contradict each other. A model vendor publishes a heat map of retrieval accuracy against context length and depth, solid green from corner to corner at a million tokens. A team measuring its own summarisation quality watches a judge score slide from 0.81 at 8,000 tokens to 0.66 at 60,000 and to 0.52 at 200,000. The tempting resolution is to call the vendor chart marketing, and that answer is both rude and wrong: the chart is a real measurement of a real capability, produced by a defensible methodology, and it will not survive the follow-up question.

The two results are compatible, and seeing why requires looking at what each evaluation asks the model to do. This chapter puts the two prompts next to each other, sets out what a needle score does and does not establish, produces the numbers that belong on the same axes, and describes a two-needle variant that changes exactly one variable in the vendor's protocol and reproduces most of the disagreement in an afternoon. The output is something you can hand to whoever forwarded the chart.

The two prompts, side by side#

A needle-in-a-haystack evaluation constructs a prompt like this:

[…62,000 tokens of Paul Graham essays…]

The best thing to do in San Francisco is eat a sandwich and sit in
Dolores Park on a sunny day.

[…62,000 tokens of Paul Graham essays…]

What is the best thing to do in San Francisco?

The evaluation runs that prompt at every combination of haystack length and needle depth — depth being how far into the haystack the planted sentence sits, expressed as a percentage, which is the vertical axis of the green rectangle — and scores whether the response contains the planted sentence's content. The question and the needle share a phrase, "the best thing to do in San Francisco", so the task is well specified by string overlap alone.

A production summarisation prompt looks like this:

[…a 62,000-token incident channel export: 1,400 messages…]

Write a post-incident summary: what failed, when it started, what
the contributing factors were, which mitigations were tried and in
what order, and when customer impact ended.

Nothing in that prompt is planted, no phrase in the question appears verbatim in the evidence, the answer is not in any one message, and correctness depends on ordering claims from messages 112, 340, 806 and 1,290 against each other and noticing that the mitigation described at 806 was reverted at 941.

The difference has a familiar shape. The first prompt is grep: one query, one match, and the work is locating it. The second is closer to writing a literature review from a stack of papers, where finding any individual paper is trivial and the difficulty is entirely in holding forty claims side by side, resolving where they contradict each other, and producing something coherent. A tool that is excellent at grep tells you very little about the second job.

Both halves of that comparison are pictures rather than descriptions of what the model does, and each breaks somewhere worth knowing. grep matches a literal string and either finds it or does not, while the needle evaluation is scored on whether the response carries the planted content, so a model can pass by paraphrasing what it found and can fail while having located it perfectly well. The literature-review half breaks in the model's favour as much as against it: a person writing a review can take notes, re-read a paper on the third day, and go and fetch one that was never in the stack, and none of those is available here. What survives both breaks is the only part the comparison is doing any work for, which is that the number of things that have to be brought together is the axis along which the two prompts differ.

What a green chart establishes#

Reading that table downwards explains why the needle score barely moves as the haystack grows — the chart solid green from corner to corner that the opening described, and that the measurements further down put figures on. Each row names a dimension along which a task can be hard, and the needle protocol has had every one of them switched off at once. The answer depends on a single fact, so there is no second retrieval to fail and no relationship between facts to get wrong. The question and the planted sentence share a distinctive phrase, so the span that bears on the question can be identified by wording alone, without the model having to establish what the question is about. And the haystack is off-topic by construction, so nothing else in the prompt is a candidate for that identification.

Those three properties compound rather than merely adding up. Locating the needle is not a search through competing candidates but a match against the one span that shares wording with the question, and once it is matched there is nothing further to do but restate it. A task built that way contains no step that gets harder as the haystack grows, which is why the curve is flat, and a flat curve is the result to expect rather than a surprise to explain. The second-to-last row names what is left once the three difficulties are removed: what remains resembles search, and search over a collection holding exactly one relevant document is the easiest search there is.

The last row is the one that resolves the disagreement. A perfect needle score establishes that information at any position in a one-million-token input remains addressable: attention can reach it, the positional encoding has not broken down, and the serving stack is not silently truncating. That is a genuinely useful thing to know, and it is not free — early long-context models failed exactly this test, producing charts with a red band in the middle that made the limitation obvious.

What it cannot establish is anything about performance when the number of retrieval operations required is not one. If a task depends on eight facts and each is recovered independently with probability 0.97, the joint probability of getting all eight is 0.78 before any reasoning over them has happened. That sentence is worth doing slowly, because it is the whole quantitative content of the disagreement. The answer is right only if the first fact is recovered and the second and the third, through to the eighth, so what is wanted is the probability of a conjunction rather than the probability of any one event. The probability of a conjunction is the product of the individual probabilities, and forming that product is permitted only when the events are independent — when whether one fact was recovered tells you nothing about whether another was. That independence is an assumption rather than an observation, and the paragraphs below depend on it.

Multiplying it out shows where the loss comes from. Recovering the first fact leaves 0.97 of the cases still alive. Recovering the second multiplies that by 0.97 again, which removes a further three per cent of whatever survived rather than three per cent of the original. The third multiplies the remainder again, and so on through all eight. Nothing about any individual retrieval got harder at any point, since each one is still running at 0.97, and the compounding on its own is enough to bring the joint probability to 0.78. A per-fact rate that reads as excellent has produced a whole-answer rate that reads as poor, purely by being required eight times over.

The assumption is worth being honest about in both directions. Real failures are likely to be correlated rather than independent — a length that hurts one recovery tends to hurt the others in the same prompt — so the true joint figure will not be exactly the product, and correlation can move it either way depending on whether the failures cluster on the same items or spread across them. The product is also a ceiling rather than a prediction, because integration introduces failure modes with no analogue in the needle test at all: two facts can both be found and still be combined in the wrong order, and no arrangement of per-fact recovery rates accounts for that.

The two curves are one mechanism seen with something switched off and then on. The first chapter of this topic argued that a passage in the middle of a long prompt loses because attention is a rivalrous budget and that passage has to win a share of it against near-identical neighbours. A needle test removes both halves of that contest. Nothing in an off-topic haystack resembles the planted sentence, so there are no near-identical neighbours to lose to, and the verbatim overlap between the question and the needle supplies the selection cue for free, so the model is never asked to work out which span matters. An integration task over an incident channel restores both at once: every message is on-topic, so the competitors are dense, and no phrase in the question appears in the evidence, so the selection has to be made on meaning. The vendor's chart and a declining in-house curve are therefore not two facts about long context that have to be reconciled. They are one fact, measured with the competition taken out and then left in.

The numbers that belong on the same slide#

Running both evaluations on the same model, the same stack and the same day is what turns this from an argument into a table. Here is one such run, using a team's own archive of incident channel exports for the integration tasks and a standard planted-sentence protocol for the needle:

Read across the top row and the columns look like different metrics of the same thing. Read down and they diverge: the needle column loses 1.1 points across a 125-fold increase in length while the timeline column loses 53.5. The two-needle column is the informative middle, because it changes exactly one variable relative to the vendor's protocol — the number of planted facts the answer requires — and it degrades an order of magnitude faster than the single-needle version. That column is the cheapest possible demonstration that the shortfall is about integration rather than about the vendor being dishonest, and it takes an afternoon to produce because it reuses the needle harness with a second needle inserted.

Attribution is what makes that column worth more than it costs. Everything else about the protocol is held fixed — the same haystack, the same lengths, the same scorer, the same model — so the gap between the single-needle column and the two-needle column has exactly one candidate cause, which is the extra fact the answer now requires. Had the variant also moved to real incident exports, or to a paraphrased needle, or to a judge instead of a string match, the resulting gap would have been just as real and entirely unattributable, and whoever forwarded the vendor chart would have been right to say so. That is the same controlled substitution the position probe earlier in this topic runs on, pointed at a different variable.

Building the two-needle variant is a small edit with a few decisions in it. The two planted sentences should be placed at different depths, because putting them adjacent turns the task back into a single retrieval over a slightly longer string. The question has to require both — "which two cities were mentioned as having the same policy, and what was it" rather than a question either needle could satisfy — and the scorer has to demand both, since a partial-credit scorer will report a number that sits between the single-needle and two-needle curves and obscures the effect. Fixing the depths across lengths rather than randomising them keeps the columns comparable, and a handful of depth pairs averaged together removes the risk of having accidentally chosen a lucky position.

The needle column is also not quite as flat as it looks in most published charts, and the reason is a detail worth knowing. When the planted sentence shares wording with the question, matching is easy; when the needle is paraphrased so that the question and the evidence share no distinctive terms, needle scores fall with length in a way the standard protocol never shows. The Chroma Context Rot work varied precisely this and found needle–question similarity to be one of the strongest moderators of long-context performance. A needle chart is therefore a measurement of a task that has been made easy along several dimensions at once, all of which your task is harder along.

Where the comparison stops#

It is not a claim that long context windows are useless or that vendors are overselling them. A million-token window is transformative for tasks that genuinely are search — find the clause, locate the log line, answer a question whose evidence is a single passage — and for those, the vendor's chart is the relevant measurement and an integration eval is not. It also helps to be generous about why that chart gets published. A needle test is one of the few long-context evaluations that is cheap to run, deterministic to score, comparable across models and directly sensitive to the engineering failures that matter most to whoever is serving the model: attention implementations that silently drop distant tokens, positional schemes that stop extrapolating, batching bugs that truncate. It is a regression test for the infrastructure, and a green chart is a claim about the infrastructure being correct. Reading it as a claim about task performance is an inference the chart does not make.

Setting that inference out in steps is what makes it possible to grant the chart everything and still decline its conclusion. The chart supports one claim: at every length and depth tested, a planted sentence remained addressable. The belief a reader forms from it is a different claim, which is that the model will use a million tokens of their own material well. Getting from the first to the second needs two further premises the chart never tested — that the task depends on one fact rather than several, and that nothing else in the context competes with the fact that matters. Both are false for most production workloads, and the reason the gap goes unnoticed is not that anyone reasoned carelessly. The chart's horizontal axis is labelled in the units the product cares about, the task on it is shaped like question answering, and it is green from corner to corner, so the two missing premises never present themselves as premises at all.

The decline is not inevitable at every length either. The curves above are one model on one collection of exports, and both the level and the slope move with the model, the task and how the context is structured; material that arrives pre-sorted by timestamp makes the timeline task substantially easier at every length, which is a property of the prompt rather than of the model.

And this is not the same argument as the filler control from the previous chapter. That held the evidence fixed and grew irrelevant padding, isolating length. This one holds length as the axis and varies how many pieces of evidence the answer requires, isolating integration load. Both curves decline, and a system can be limited by either; distinguishing them tells you whether to shorten the prompt or to restructure the task into smaller pieces whose results you combine yourself.

Two honest evaluations of the same model disagree when they are asked different questions, and the difference here is the number of retrieval operations the answer requires. One is search and stays flat; the other is synthesis and falls with length, on every model anyone has published this comparison for. The practical consequence is that a long context window is a capability you have to evaluate on your own task shape before you plan around it, and the cheapest way to do that is the two-needle variant — one extra planted sentence, one scorer that demands both, an afternoon of compute — which sits between the two protocols and shows you the slope.

Everything measured so far has been about evidence: where it sits, how much competes with it, how far apart the pieces are. The other thing in a long prompt is instruction, and instructions are not scored the same way, because there is usually no single correct output to match against. Where to put the sentence that must be obeyed, and what to do when a regulator rather than a product manager is the one asking, needs its own probe.