Needle tests and context rot

The needle-in-a-haystack format comes from a 2023 evaluation by Greg Kamradt that became the standard long-context marketing chart: plant a single unrelated sentence — the classic is a line about the best thing to do in San Francisco — at a known depth inside a long block of filler text, then ask a question whose answer is that sentence, verbatim. Score is exact retrieval of a planted string. It was a good probe when models routinely failed it, and models routinely failed it in 2023.

What the needle actually asks#

The task has three properties that make it a poor proxy for anything an agent does. The needle is lexically distinctive against its background, so finding it is close to a string match rather than a semantic search. The question maps to exactly one span, so there is no composition: nothing has to be combined, compared or excluded. And the filler is topically unrelated to the needle, so nothing in the haystack competes for the answer. Real prompts invert all three — the evidence looks like everything around it, the question needs two or three facts joined, and the surrounding passages were retrieved precisely because they resemble the question.

The reason it gets reached for anyway is not carelessness, and naming it is what makes it resistible. A needle test has every surface property of a good experiment: it is cheap, it varies exactly one thing, it produces a chart, and it can be run on your own corpus — which answers the strongest objection anyone raises against a published result, that the published result was measured on somebody else's data. What it does not have is a task whose difficulty resembles the task you serve. Running it on your own corpus does not repair that, because the defect is in the task rather than in the data, and the right data under the wrong measurement still measures the wrong thing.

The empirical picture is better documented than the argument usually admits. Context Rot: How Increasing Input Tokens Impacts LLM Performance (trychroma.com) runs the degradation question across eighteen models and finds that performance falls as input grows even on tasks that are trivially easy at short length, and that the size of the fall depends on how similar the distractors — the non-answering material sharing the input with the evidence — are to the target and how similar the question is to the evidence, with length alone the weakest predictor. Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell (arxiv.org) separates two failure modes that a single score conflates: the model may fail to locate the relevant span, or it may locate it internally and fail to use it in the answer. A needle test is blind to the second kind by construction, since locating and answering are the same act.

Vary volume and nothing else#

The design is the single-variable rule applied to a continuous variable instead of a binary one. Fix the questions. Fix the evidence those questions require, so that every arm contains exactly the passages needed to answer. Fix the answer budget. Then vary only the amount of surrounding material, and plot per-case score against total input tokens.

import random

LENGTHS = [8_000, 16_000, 32_000, 60_000, 90_000]

def build_arm(case, filler_pool, target_tokens, rng):
    """One case at one input length. Evidence and question are constant."""
    blocks = list(case.gold_passages)          # identical in every arm
    used = case.base_tokens + sum(p.tokens for p in blocks)
    # pad with real passages from the same corpus that do NOT answer the question
    while used + 400 < target_tokens:
        p = rng.choice(filler_pool[case.topic])  # same topic, non-answering
        if p.id in case.gold_ids:
            continue
        blocks.append(p)
        used += p.tokens
    rng.shuffle(blocks)                        # evidence position randomised
    return {
        "prompt_blocks": blocks,
        "question": case.question,             # constant
        "max_tokens": 700,                     # constant answer budget
        "target_tokens": target_tokens,
        "actual_tokens": used,
    }

def sweep(cases, filler_pool, seed=20250812):
    for length in LENGTHS:
        for repeat in range(3):                # three repeats, per the churn floor
            rng = random.Random(seed + repeat)
            for case in cases:
                yield build_arm(case, filler_pool, length, rng)

Three task families run through the same sweep, which is what makes the result interpretable rather than just a curve. The first is a verbatim needle, included as a control so you can show that the classic result reproduces on your data. The second is single-hop question answering: one retrieved passage contains the answer, and the model has to use it. The third is two-hop: the answer requires combining a fact from one passage with a fact from another — a refund eligibility that depends on both the purchase date in one document and the category-specific window in another.

The curve, and where it turns#

Run over 200 cases per family, three repeats, with everything else pinned:

The needle column is accurate and useless: 0.99 at 60k, essentially flat. The two-hop column loses 27 points between 8k and 60k on the same inputs, with the same evidence present, under the same answer budget. The single-hop column sits between them, which is what you would expect if the mechanism is interference from similar material rather than a hard capacity limit — one lookup among many plausible candidates is harder than one lookup among unrelated filler, and joining two lookups is harder again. The turn in the two-hop curve is between 16k and 32k, and no published chart contains that number, because it is a fact about this corpus, these questions and this model.

The figure in that reading comes straight off the table. The two-hop column holds eighty-eight hundredths at eight thousand tokens and sixty-one hundredths at sixty thousand, and the second subtracted from the first is twenty-seven hundredths, which is the twenty-seven points. The needle column over the same span moves by a single hundredth, which on a two-hundred-case family is two cases — the control is not merely high, it is flat to within the resolution of the family it was measured on.

The three columns support one more step, and it is the one that says where the loss is happening. The families ran on the same inputs, and a two-hop question is structurally two single-hop lookups plus a join. If the two lookups succeeded or failed independently, each at the single-hop rate, the two-hop rate would be the single-hop rate multiplied by itself. Do that multiplication at eight thousand tokens and the product lands on the measured two-hop score almost exactly, which says that at short lengths composing two facts costs nothing beyond finding them. Do it at sixty thousand and the product comes out well above the measured score; do it at ninety thousand and it comes out further above still. The gap between the product and the measurement is therefore a third quantity, absent at the short end and growing with length, and it is not explained by either lookup failing on its own. That residue is the composition step getting harder — and it is the exact thing a needle test cannot contain, because a needle test has one lookup in it and no join.

The p95 latency column is not decoration either. It more than triples across the same sweep, from 2.9 seconds at 8,000 tokens to 9.5 at 90,000, which means the cost of volume is being paid twice — once in the two-hop column and once in a number the user experiences directly. A decision about k that reads only the quality curve is leaving half the evidence on the table.

The needle family earns its place in the same run for a reason that is about arguments rather than about models. Anyone can dispute a published chart measured on somebody else's data; nobody disputes the control column when it sits beside the task column, produced by the same harness on the same corpus in the same afternoon. Reproducing the classic result and then showing what it fails to predict is what stops the disagreement from recurring at the next standup.

There is also a result the sweep can produce that means the experiment is broken rather than the model is: a two-hop curve that rises with length. That happens when the padding is accidentally helpful — filler drawn from the same documents as the gold passages, carrying context that helps the model interpret them — and it is the most common way this design goes wrong.

The confounds that make or break it#

Five of them, each capable of producing a curve that means something other than what you will read into it.

Evidence position has to be randomised across repeats rather than fixed, because models attend unevenly across a long input and a design that keeps gold passages at the front while padding grows behind them measures position, not volume. Shuffling per repeat and averaging separates the two; holding position fixed and varying it as a second sweep is how you measure position deliberately.

Why shuffling works is worth stating rather than trusting, because it is the general move for any variable you are not trying to measure. Position affects the score. You are not asking about position. What you need, then, is for position to have the same distribution in every arm, so that whatever it contributes contributes equally on both sides of the comparison and cancels when one arm is subtracted from the other. Shuffling per repeat and averaging over repeats produces exactly that sameness. Pinning the gold passages at the front produces its opposite: as padding grows behind them, their position relative to the whole input drifts steadily toward the opening, so position and volume move in lockstep and no subtraction can pull them apart afterwards.

Padding has to be plausible and non-answering. Lorem ipsum, repeated boilerplate, or passages from an unrelated corpus are trivially ignorable, and a sweep padded that way will show a flat curve that tells you nothing about production, where every retrieved passage was returned because it resembled the question. Drawing filler from the same topic in the same corpus, with gold IDs excluded, is what makes the distractors realistic.

The answer budget has to stay constant. Letting max_tokens grow with input length is the most natural-feeling mistake in the design, and it confounds volume with output space: a longer allowance changes how the model structures the answer, and the judge scores that difference too. The evidence set has to stay identical for the mirror-image reason — adding more relevant passages as you add tokens makes the task easier and the volume larger at the same time, and the two effects partly cancel into a flat curve.

And the judge has to see the same thing in every arm: the question, the gold answer and the model's response, never the length or the arm label. A judge that can tell it is grading a 90k-token run will grade it differently, and that is a measurement of the judge.

Corpus-specific, snapshot-specific, mechanism-blind#

It is not a claim that long context is useless, and reading it that way leads to systems that under-retrieve and fail from missing evidence instead of failing from interference. The 60k arm still answers 61% of two-hop questions correctly with all the evidence present; an 8k arm that drops the second gold passage answers 0% of the ones it dropped.

It is also not transferable. The turn at 16–32k belongs to this corpus, these two-hop questions and this model snapshot. A model release moves it, usually upward, and a change in what your passages look like moves it too — the effect is driven by distractor similarity, so a corpus of near-duplicate policy documents will turn down earlier than a corpus of distinct product manuals. Re-running the sweep after a model change costs about two hours and is the only way to know which direction it moved.

Finally, "context rot" names a symptom rather than a mechanism. The curve is consistent with position effects, with distractor interference, and with the model locating the right span and failing to use it — the distinction the arxiv paper draws. Which one you have determines the fix, and telling them apart needs a further experiment: present the same two-hop question with only the gold passages, and if accuracy jumps back to 0.88, the model could not find or hold the evidence among the distractors; if it stays near 0.61, the failure is in composing the answer, and no amount of retrieval tuning will help.

A needle test and a two-hop sweep answer different questions, and only one of them is about your system. Running the sweep costs a few hundred requests per length per family and produces a number that belongs to your corpus: the input size past which the tasks you actually serve start losing points while a planted string keeps coming back perfectly. That number is what an argument about retrieval volume needs, and it is the one thing neither published chart in the standup contained.

What it still does not do is decide anything. A curve turning down at 32k does not by itself say whether to rerank, compact, or leave the system alone, because that decision depends on a threshold and a constraint that have to exist before the numbers do.