Finding the retrieval crossover

A crossover is the value of k past which the marginal passage costs more than it contributes. It exists because both effects are real at once: each extra passage raises the chance the answer-bearing article is present, and lowers the odds the model uses it, by restating what is already there and by pushing the governing instruction further from the question. Two measurements at 8 and 20 establish that a curve slopes downward somewhere between them and nothing else.

A turn has to exist for a reason, and the reason is that those two effects are not merely opposed but shaped differently. Adding passages helps by supply: each extra passage is another chance that the article carrying the answer is in the prompt at all, and that chance is subject to diminishing returns, because the passages a retriever adds late are the ones it scored worst and they are increasingly likely to duplicate something already admitted. Adding passages hurts by dilution, and dilution does not diminish. Every extra passage restates material already present and pushes the instruction further from the question, and the twentieth restatement costs about what the second one did. A term that flattens plus a term that does not eventually produces a maximum, and where it falls depends entirely on how quickly the first term flattens — which is a fact about how much a particular corpus repeats itself.

Finding the turn matters more than the point value it produces, because the shape around the peak is what tells you how much slack you have. A flat plateau means the exact number is not worth arguing about; a sharp peak means the default someone inherited is probably on a slope. The measurements below come from a support bot on a help-centre corpus, scored on a frozen 500-question set against a 12,000-token input budget.

Why reordering cannot explain the drop#

The first explanation offered when an eval regresses on a longer context is lost in the middle. It arrives fast, it sounds informed, and the proposed fix is cheap: models attend better to the start and end of a long context than to its middle, so reorder the passages so the top-ranked ones sit at both ends and the weak ones sink into the trough.

The explanation is tempting for a good reason, which is that its premise is true. Models really do attend unevenly across a long input, the effect has a name and a literature, and a longer prompt really was the thing that changed. What makes it the wrong diagnosis here is the step nobody says out loud: that because the prompt got longer, the material that matters ended up somewhere worse. A reorder is a permutation, and a permutation only helps if the thing you care about is currently in a bad position and would move to a better one. Whether that is the case is not a question about models. It is a question about where the bytes actually sit, and that can be answered by counting.

That explanation can be ruled out before running anything, with a column of token offsets. The prompt is assembled in a fixed order — system instructions, tool definitions, the retrieved passages in rank order, the conversation history, then the user's question — and the two arms share every component ahead of the passages. So the top-ranked passages sit in the same place in both.

Every passage present in the eight-passage arm occupies an identical position in the twenty-passage arm. Nothing moved. If the model were failing to attend to material in the middle, the eight passages that carry the answer are in exactly the same middle in both arms, and the arm with the higher accuracy is the one where they are surrounded by less. Positional effects are real and worth knowing about, but they cannot explain a difference between two prompts that agree, byte for byte, on their first 5,600 tokens. Reordering rearranges those 5,600 tokens; it does nothing about the 2,635 that were appended after them.

Every offset in that table can be reconstructed from the service's own token allocation — 1,400 tokens for the system prompt, 900 for tool definitions, 4,000 for retrieval and 3,700 for conversation history — and reconstructing them is what makes the argument airtight rather than merely plausible. The system instructions start at zero and are allocated 1,400 tokens, so the tool definitions start at 1,400. Their allocation is 900, so the first passage starts at 2,300 — in both arms, and necessarily in both arms, because neither component's size has any dependence on k. From there the passages are laid down in rank order at whatever each one weighs, which is why ranks 4 and 8 also land on identical offsets. The same passages, in the same order, behind the same prefix, arrive at the same place.

The two rows below the passages are where the arms finally separate, and they carry more than a position. In the eight-passage arm history begins at 5,600 and the question at 9,300, a gap of 3,700, which is exactly the history allocation — nothing has been taken from it. In the twenty-passage arm history begins at 8,800 and the question at 10,000, a gap of 1,200, which is exactly the floor the eviction policy is not allowed to cut history below. Read that way the offset table is not only a positional argument. It is a record of the assembler having emptied a component down to its floor, legible before a single answer was generated and from a column somebody built for a different purpose.

Designing the sweep#

What is actually needed is the curve, and three details decide whether the numbers mean anything.

The first is that every arm runs against the same frozen question set with the same model, the same settings, and the same judge prompt — the judge being a second model call that reads the question, the reference answer and the produced answer and scores the last against a written rubric, which is how 500 questions get graded in twenty minutes instead of a week of human reading. This sounds obvious and is where most sweeps go wrong, because it is tempting to regenerate the eval set when the corpus changes and then compare an arm from last month against one from today.

The second is that you record what the assembler actually built, not what you asked it to build. The service enforces a 12,000-token input budget, allocated as 1,400 for the system prompt, 900 for tool definitions, 4,000 for retrieval, 3,700 for history and 2,000 held back for the answer. Fifteen 430-token passages do not fit in a 4,000-token slice. The eviction policy lets retrieval borrow from history down to a floor of 1,200 tokens and then trims retrieval, so the arm labelled k=20 is not running twenty passages.

The number it does run can be worked out in advance from those two rules, which is worth doing because it predicts a result in the table below. Retrieval's own slice holds 4,000 tokens, and at roughly 430 tokens a passage that is nine passages with a little left over. History is allocated 3,700 and may be cut to 1,200, so the most retrieval can borrow is the difference between those two, which is a little over two thousand tokens. The slice plus that borrowing comes to around six and a half thousand tokens, and at 430 tokens a passage that is fifteen passages. Fifteen is the ceiling on realised k for this service, and it is a property of the budget and the chunk size rather than of anything anybody typed into a config. Any requested k at or above fifteen produces the same prompt.

The third is that you run the sweep twice: once at the production budget, and once with the budget raised high enough that nothing is ever evicted. The two curves differ by exactly the amount that eviction is costing you, which is how you tell dilution apart from budget pressure without arguing about it.

The harness is small enough to read in full. The part that earns its keep is the instrumentation, because it is what turns a confusing curve into a legible one.

import json

def run_arm(questions, k, budget_tokens, assemble, generate, judge):
    rows = []
    for q in questions:
        ranked = retrieve(q.text, top_k=k)
        prompt, trace = assemble(q, ranked, budget_tokens)  # trace: what survived
        answer = generate(prompt)
        rows.append({
            "question_id": q.id,
            "requested_k": k,
            "realised_k": trace.retrieval_chunks,     # not always == k
            "history_tokens": trace.history_tokens,   # first thing evicted
            "prompt_tokens": trace.total_tokens,
            "distinct_claims": len(trace.claim_union),
            "score": judge(q, answer),
        })
    return rows

with open("sweep.jsonl", "w") as f:
    for k in (2, 4, 6, 8, 12, 16, 20, 28):
        for budget in (12_000, 20_000):
            for row in run_arm(QUESTIONS, k, budget, assemble, generate, judge):
                row["budget"] = budget
                f.write(json.dumps(row) + "\n")

Reading the curve#

At the production budget, the sweep comes out like this. The realised k and history columns are the ones that explain the shape, and the last column is p95 latency — the time under which 95 of every 100 requests complete, which is the figure a latency budget is normally written against because the mean hides the slow tail that users notice.

The maximum is at six, and it is a modest one — 0.79 against 0.78 at eight, a difference of five questions out of 500 and not on its own a reason to change anything. What the curve does establish is that the region from four to eight is a plateau and everything past twelve is a decline, so the original default of eight was already at the top of the hill and the change to twenty walked down the far side of it.

Converting the gap back into questions is what stops a decimal from being mistaken for a result. Judged accuracy here is a share of a fixed 500-question set, so one hundredth of a point is five questions, and the whole distance between the best arm and the incumbent is five graders' verdicts. Read the same way, the drop from eight to twenty is seven hundredths, which is thirty-five questions, and the difference between five and thirty-five is the difference between a number inside anybody's run-to-run wobble and a number that reproduced across three runs. The unit conversion is the cheapest guard against arguing over a curve's third decimal place.

The plateau matters more than the peak for a reason the shape makes plain. Four, six and eight sit within a hundredth of each other, so an inherited default anywhere in that region costs nothing measurable, and moving between them is not work worth scheduling. Past twelve the drop is monotonic and large. A default of eight was one step from the edge of that region rather than in the middle of it, which is the useful thing to know: the risk was never that eight was wrong, it was that eight had no margin on the side the corpus drifts towards.

The last two rows are the most informative. k=20 and k=28 produce identical numbers because they produce identical prompts: both requests are trimmed to the same fifteen passages against the same 1,200-token history floor. Setting top_k: 20 did not ship twenty passages. It shipped fifteen passages and deleted two-thirds of the conversation history, and in a support bot where the customer's previous three messages contain the order number, that second part is not a detail.

Those rows are the ceiling worked out above arriving in the data. Fifteen was predicted from the slice, the floor and the chunk size before any arm ran, and two requested values landing on one realised value is what a hard ceiling looks like from the outside. The two-thirds figure comes from the same place: history is allocated 3,700 tokens and was cut to its floor of 1,200, so it lost the difference between them. That is the part a realised_k column alone would not have shown. Logging what retrieval got tells you the request was clamped; logging what history was left tells you what paid for the clamp.

The raised-budget arm separates the two effects:

Dilution is the larger term. Removing budget pressure entirely recovers two points of the seven that were lost, and the other five are the model reading a prompt where the answer is stated once and its neighbours are stated seven times. The k=28 row inverts because at 20,000 tokens all twenty-eight passages actually fit, so it is the first arm in the table that is genuinely more diluted than the arm below it — a useful check that the raised-budget arm is measuring what it claims to.

The split into two and five is a subtraction, and it is worth walking because each term has a name and a different fix. Total loss from the best production arm to the worst is 0.78 at eight against 0.71 at twenty, which is seven hundredths. Raising the budget removes eviction as a possible cause while changing nothing else, and the same requested k then scores 0.73 rather than 0.71: two of the seven hundredths were eviction. Whatever is left after removing a cause is attributable to the causes that remain, so the other five hundredths belong to the prompt the model actually read. Eviction is fixed by an allocation, and there is a config line for it. Dilution is fixed by selecting differently, and there is not.

The reason to run an arm at a budget you have no intention of shipping is worth being explicit about, because the arm looks like wasted eval spend. It is not there to be a candidate. It is there to be a control: it differs from the production arm in exactly one respect, so the difference between them isolates that one respect. Without it the two effects arrive summed, and every argument about the curve becomes an argument about which of the two is doing the work, settled by whoever is most confident. The k=28 row is the same trick used as a sanity check in reverse — an arm where the raised budget is expected to score worse, because at 20,000 tokens nothing is trimmed and the model reads all twenty-eight passages. It does score worse, which is weak evidence that the raised-budget arm is measuring dilution rather than measuring nothing.

The levers that are not k#

k is the crudest control available, because it selects a count rather than a property. Four others act on the same problem and compose with each other.

The second row's token figure is the one that ties this table back to the sweep, and it is smaller than it first looks for an instructive reason. Claim-level selection is being compared against k=20, and k=20 does not ship twenty passages on this service — it ships the fifteen the ceiling permits, at roughly 430 tokens each, which is around six and a half thousand tokens of retrieval. Cutting that down to the couple of thousand tokens a claim-level pass admits is the −4,300. Had the comparison been drawn against the twenty passages the config asks for, the saving would have looked larger and would have been describing a prompt the assembler never built. A lever table is only honest if its baseline is the realised arm.

The row worth staring at is the third. Chunk size and k are the same knob wearing different hats: at a fixed token spend, halving the chunk size doubles the number of retrievable units, which raises the odds that the specific sentence you need is in a unit whose whole content is about that sentence. It also destroys passages whose meaning depends on the paragraph above them, which is why the gain is +0.03 and not +0.10 — smaller chunks help the questions with a single crisp answer and hurt the ones that need a procedure read end to end. The way to find out which population dominates is the same sweep with chunk_tokens on the x-axis.

The mechanism under that row is worth spelling out, because "smaller chunks retrieve better" is the kind of rule that gets carried to a corpus where it is false. A retriever compares one vector for the query against one vector per unit, and a unit's vector is an average of everything in it. A 430-token passage that is mostly about refunds and mentions the issuer exception in one sentence has a vector dominated by refunds, so a question about issuer holds scores it poorly even though the answer is inside it. Halve the unit and that sentence can land in a passage whose whole content is about issuer holds, whose vector therefore points at the question. That is the gain, and it applies to any question whose answer is one locatable statement. The loss has the same cause read backwards: a procedure whose third step is meaningless without its second is one idea spread across several sentences, and cutting it in half produces two units each of which is a partial account of something. The knob does not make retrieval better. It moves quality from one population of questions to another, and which way that trade falls is a fact about what your users ask.

The reranker row is the one most teams reach for first, and it is a real gain, but note what it costs and what it does not do. A cross-encoder gives you a better ordering of forty candidates. Order six copies of the same claim however you like and they are still six copies. Reranking and marginal-contribution selection are complements: rerank to improve what enters the selector, select to control what leaves it.

The crossover is a property of your corpus, not a number to quote#

The temptation, once you have a curve, is to write down that eight is too many and six is right, and to treat that as knowledge. It is a measurement of one corpus at one point in time against one model, and every term in that sentence moves.

Corpus redundancy is the dominant variable and it drifts upward. Every localised copy, every FAQ that restates a policy article, every migration that leaves the old document in the index pushes the crossover left, because the marginal passage is more likely to be a restatement. Support corpora become more redundant over time as a rule, not as an accident. A crossover measured at six passages this quarter is plausibly at four next year with no change to the retriever at all.

Changing the model moves it too, and not always in the direction you expect. A model that follows a distant instruction more reliably tolerates a longer evidence block, which pushes the crossover right; a model that weights repeated assertions more heavily pulls it left. There is no way to read this off a release note. Re-running the sweep costs one afternoon of eval spend and is the only thing that answers it.

And the sweep says nothing about the components that are not retrieval. It varies k and holds the system prompt, the tool definitions and the few-shot block fixed — few-shot meaning the handful of solved examples of the task that sit in the prompt to show the model the shape of a correct answer — because holding them constant is what makes it a controlled experiment.

The curve, then, is the deliverable rather than its maximum. It tells you whether the number you inherited sits on a plateau or a slope, how much of the loss is dilution and how much is the assembler quietly evicting something else, and which of the levers that are not k is worth the engineering. Two points tell you none of that, and a reordering will not reveal it, because a permutation of a prompt does not change what was appended to the end of it.

Those fixed blocks are also spending tokens, they are also on every request, and nobody has asked what they add. The natural next move is to take one of them out and see whether anything happens — which turns out to require a different kind of evidence than a curve, because a curve compares aggregates and a deletion has to be defended case by case.