Higher recall of the gold passage can lower answer accuracy

A retrieval system is asked for a fixed number of passages per question, and that number is conventionally called k. Raising it is a one-line change with two consequences that get reported as one. Recall of the gold passage is the share of questions where the paragraph known to contain the answer ended up inside the prompt; it is a property of search. Answer accuracy is the share of questions answered correctly when graded; it is a property of what a model did with a prompt it was given. The first is an input to the second and is not a proxy for it, and the arithmetic below is what happens when a team treats it as one.

Separating the two also separates a requirement from a design decision, which is worth doing explicitly because they get argued about as though they were the same kind of statement. That a system must answer questions from a company's own documents is a requirement: it comes from outside and it is not negotiable by the team building it. That the system retrieves a fixed k passages per question and puts all of them into the prompt is a design decision, made early, usually without discussion, and reversible. Everything in this chapter is a consequence of that decision rather than of the requirement, which means every finding here has a knob attached to it.

Two stages, two numbers, opposite directions#

Recall and accuracy are measuring different stages, and they are not on the same axis. Recall is a property of retrieval: did the evidence reach the prompt. Accuracy is a property of generation given that prompt: did the model use the right part of it. Reporting them as a pair of percentages side by side invites the reading that they are two views of one quantity, and they are two readings taken at two points on a chain, with a stage in between that can undo the first.

An answer is correct when two things both hold: the evidence reached the prompt, and the model used the right part of it. That makes end-to-end correctness a product of two factors rather than a sum, and raising k acts on both of them at once, in opposite directions. It raises the numerator of the first, because a longer list is more likely to contain any particular passage while the count of questions it is divided by stays fixed. It lowers the second, because the model's task changes from reading one candidate to discriminating between many, and every passage added raises the amount of plausible, topically adjacent, wrong material the model has to choose against, which is a load the model carries at generation time. Above some k the second effect exceeds the first, and where that point sits depends on how much near-duplicate material the corpus contains — a corpus with one canonical document per topic tolerates a large k, and a corpus with five years of superseded drafts does not.

Raising k from 5 to 25 on a 300-question labelled set did the first thing exactly as advertised: recall of the gold passage went from 82% to 96%, meaning that on 42 more questions the paragraph containing the answer was now inside the prompt. End-to-end answer accuracy went from 74% to 72%. The paired breakdown showed 11 questions newly correct and 17 newly wrong, and every one of the 17 had the gold passage in the prompt.

Two arithmetic checks are worth making on those figures before reading anything into them, because both of them pin down what the numbers are measured over. The recall move is fourteen points, and fourteen points of a 300-question set is forty-two questions, which is exactly the count reported. That tells you the denominator of the recall figure is all 300 questions rather than the subset whose answers exist in the corpus at all — a distinction that matters, because a recall figure computed over the answerable subset would be higher and would not reconcile with the count. The accuracy move is two points, which is six questions of the 300, and the paired breakdown gives seventeen newly wrong against eleven newly correct, a net of six. Two independent instruments landing on the same six is what licenses reading the breakdown as complete: no question changed grade in a way those two counts miss.

The gross and the net are the finding, and the headline hides it. Twenty-eight questions changed grade while the reported accuracy moved by six questions' worth. A team reading only the aggregate sees a two-point wobble and calls it noise. The paired view shows a system that was substantially rearranged, gained on one set of questions, lost on a larger one, and happened to land close to where it started. That every one of the seventeen regressions had the gold passage in the prompt settles which factor moved: the retrieval factor went up for those questions, to its maximum, and correctness went down anyway, so the loss is entirely in the second factor.

Reading five of the regressions showed the same shape each time. A question about the refund window on the legacy plan had retrieved, at rank 1, the legacy refund policy, and at ranks 6, 9, 14 and 22, the current refund policy, a regional variant, a superseded draft, and an internal FAQ that summarised all three imprecisely. Under top-5 the model saw one policy and quoted it. Under top-25 it saw four documents making four similar-but-different claims about a 14-day window, a 30-day window, and an exception, and it produced an answer that blended them.

Those four ranks are the mechanism rather than an illustration of it. Every one of them falls above 5 and at or below 25, which is precisely the band the change opened up, so the four distracting documents are not documents that happened to be there — they are documents the change admitted. The gold passage never moved; it was at rank 1 before and at rank 1 after. What changed is the company it was keeping. A reader who takes away only "large k is risky" has taken away less than the case contains: the risk is specific to corpora holding several near-duplicate documents per topic, and it is measurable in advance by asking how many distinct documents in the candidate band make overlapping claims.

Reading the sweep as gaps rather than levels#

Four settings over the same 300 questions show both effects at once. The window figure is the model's context window, meaning the total amount of text a single call can hold, prompt and answer together, and it is the hard ceiling on how large a k can be.

The levels in that table say very little and the gaps between rows say all of it. Step down the recall column and each step buys less than the one before: eleven points, then nine, then five. Step down the accuracy column and the steps are four points, then one, then a loss of three. The two columns therefore cross somewhere between the third and fourth rows, and the crossing is the entire result — recall is still climbing at the point where accuracy has already turned over.

The gap measured across each row rather than down each column makes the same point more sharply. Subtract accuracy from recall row by row and the difference is one point, then eight, then sixteen, then twenty-four: it widens monotonically, by roughly the same amount at every step, and it never once narrows. That widening quantity is recall that did not convert into an answer, and it is worth naming honestly rather than treating as a clean count of failures. It is a net of two opposite things: questions where the evidence was present and the answer was still wrong, less questions answered correctly without the evidence being present at all. The second of those is not zero — a model can answer a question from what it already knows, or from a passage that is not the labelled one — and nothing in this table separates the two components, so the column is a difference and not a count. What it shows is that as k grows, the first quantity grows faster than the second, whatever their individual sizes are.

Reporting only the first column makes the last row look like the best system in the table. Reporting only the second makes top-10 look like a free win, when the step from top-5 to top-10 roughly doubles the prompt tokens and multiplies the price per query by 1.8, for one point that sits well inside the noise of a 300-question set. Both columns together are the actual finding, which is that this corpus's usable range is somewhere between 5 and 10 and that the remaining 9% of recall is not convertible into accuracy by adding context.

That 9% is measured from the top of the usable range rather than from the best recall in the table, and the choice of vantage point is what makes it the useful figure. At top-10 the accuracy column has peaked, and nine points of recall are still missing. Pushing on to top-25 recovers five of those nine and gives back three points of accuracy in exchange. So the nine points are not unreachable — they are unreachable by this lever, and the honest reading is that whatever is going to close them is not a larger k.

What a design has to beat besides the other design's score#

The comparison that follows from this is between two full designs rather than two values of k. Design A is one dense retriever, top-5, one model call, p95 2.4 s, $0.006 per query, scoring 71% on the 300-question set. Design B adds LLM query rewriting and a cross-encoder rerank over 50 candidates down to top-5, p95 5.1 s, $0.021 per query, scoring 79%.

The method is the paired comparison from the labelled-set chapter, run on identical questions, with the interval computed on the per-question difference. Here that gives 8.0 points with a 95% interval of roughly 3.4 to 12.6. The interval excludes zero, so B is genuinely better, and its width says the improvement could plausibly be three points, which changes how much latency it is worth. The 79% on its own is an estimate and not a measurement, and treating it as a measurement is how a two-point difference gets defended in a meeting for a month.

That interval is worth looking at rather than past, because its shape says what kind of instrument produced it. It reaches the same distance either side of the point estimate — the same number of points from 8.0 down to 3.4 as from 8.0 up to 12.6 — which is the signature of a symmetric margin laid around a mean difference, rather than a range read off two separate measurements. What that buys is a reading rule: the point estimate is the centre of the interval and carries no more authority than any other value in it, and the width, not the centre, is what a decision has to survive.

Three things have to clear beyond the point estimate. The regression set is the first: B answers 14 questions that A got right incorrectly, and the mechanism behind them matters more than their count. A rewriter that turns "how do I cancel a trial on the legacy plan" into "how do I cancel a trial" produces a fluent, confident, wrong answer about the current plan, and confident wrongness on a question the simpler system handled is a worse user experience than an equal number of don't-knows. The second is latency measured against the session it lives in rather than as a flat tax. The third is cost at projected volume, where $0.021 against $0.006 is 3.5× and only becomes a real argument once multiplied: at 4,000 queries a day it is $60 a month against $17, which nobody should optimise; at 400,000 a day it is $6,000 against $1,700 and the conversation changes.

The reason volume has to be named before the cost argument means anything is that a ratio is scale-free. Divide one per-query price by the other and the answer is the same at one query a day and at a million, so the ratio alone can be quoted in either direction by whoever wants to win the meeting. It becomes an argument only when multiplied by a number of queries, and at that point the same 3.5× is either an amount nobody should spend an engineer's afternoon on or a line item somebody has to sign off.

Buying back the latency, priced individually#

When a design loses on latency, the question that follows is whether its quality can be had at the cheaper design's latency. The levers are concrete and each one hands back a slice of the eight points. B's 5.1 s decomposes into roughly 0.7 s of query rewriting, 0.15 s of first-stage retrieval, 1.4 s of cross-encoder scoring over 50 candidates, and 2.8 s of generation.

Those four parts sum to the whole to within the precision they are quoted at, and the small residual is rounding rather than a fifth stage nobody measured. That matters more than it sounds. A decomposition that closes is a licence to treat the list as exhaustive: every second of the p95 is owned by a named stage, so a lever that does not appear against one of these four cannot exist, and any proposal to save time somewhere else is a proposal about something not in the measurement.

One row in that table costs no accuracy at all, and it is the one to read first. Every other lever that moves p95 pays for the time in points: two for the smaller rerank set, one and a half for the smaller cross-encoder, an unquantified amount on long-tail questions for the threshold. Running the rewrite concurrently with first-stage retrieval pays in vector-store throughput and in a candidate set that is a union rather than a single list, and in nothing else. It is a lever whose cost is denominated in a currency the quality measurement does not use, which is why it should be spent before any of the others are considered.

The concurrency lever is the one that is usually available and usually not taken, because the rewrite is drawn as step one in the diagram and people implement the diagram. The sequencing in the drawing is a picture of data dependency, and the dependency is not actually there: first-stage retrieval on the raw query does not need the rewritten query, so the two can start together and the slower one sets the clock.

async def retrieve_b(question: str, principal: Principal) -> list[Hit]:
    rewrite_task = asyncio.create_task(rewrite_query(question))       # ~0.7 s
    raw_task = asyncio.create_task(search(question, principal, k=50)) # ~0.15 s

    rewritten = await rewrite_task
    candidates = {h.payload["chunk_id"]: h for h in await raw_task}
    for h in await search(rewritten, principal, k=50):
        candidates.setdefault(h.payload["chunk_id"], h)

    return rerank(question, list(candidates.values()))[:5]

That version pays for the rewrite only where it exceeds the retrieval it overlaps, and it has a side benefit worth measuring separately: because the raw query's candidates are unioned in, a rewrite that drops a constraint can no longer remove the gold passage from the candidate set, which addresses the regression mechanism directly rather than by hoping the rewriter behaves. It costs a second search per query and a slightly larger reranking input unless the union is capped.

The side benefit is worth stating as a mechanism rather than as a happy accident, because it is the same product argument in a smaller frame. A rewritten query that drops "on the legacy plan" produces a candidate list whose existence factor for the legacy passage is zero, and a reranker downstream cannot recover it. Unioning the raw query's candidates in means that factor can only be zeroed if both queries miss the passage, and the raw query is the one that still contains the constraint. Two chances at a factor is a different design from one chance at it, and no amount of improving the rewriter reaches the same guarantee.

Streaming changes nothing about end-to-end duration and changes almost everything about the experienced wait, since first token at 0.4 s reads as responsive regardless of when the last token lands. It constrains the design in one specific way: anything the application does after generation — the citation resolution, any post-processing — has to be restructured, and the permission decision cannot live there, which the third chapter already established for other reasons.

Ordering inside the prompt is a lever with no latency cost at all, which makes it the first thing to try when distraction is the diagnosis. Models attend unevenly across a long context, and the passages at the beginning and end of a block are used more reliably than the ones in the middle, so a top-25 prompt buries ranks 10 through 18 in the least-attended region. Reordering so that the highest-ranked passages sit at the two ends, or simply trimming to the range where accuracy peaked in the table above, costs nothing and is measurable on the same 300 questions in four minutes. It is also worth labelling each rendered passage with its source and date, since a model given two conflicting policies and no way to tell which is current has no basis for choosing, and an explicit "superseded" marker in the rendered text converts a coin flip into a decision.

Where these numbers stop holding#

The same two designs in a different container flip the decision without any new measurement. An internal research tool where an analyst asks one question and reads for four minutes has latency headroom that a help widget does not, and there B is obviously right. Nothing about B changed; the denominator the latency is measured against did.

None of this says B's architecture is wrong. It says that on this corpus, this question set, and this session shape, eight points did not buy 2.7 seconds. Change the corpus to one where the first-stage retriever is genuinely weak — many near-synonymous documents, heavy jargon, queries phrased unlike the corpus — and the reranker's contribution grows while its cost stays flat. The measurement is local, and re-running it after the corpus doubles is not optional.

The accuracy-versus-recall result is also specific to the models in use at the time it was taken. Long-context models have improved markedly at ignoring irrelevant material in a large window, and the k at which distraction overtakes recall has moved upward over the last two years and will move again. Which means the table at the top of this chapter is a measurement procedure rather than a finding: re-run it when you change models, and expect the optimum to shift.

There is also a design in the space that neither A nor B occupies, and it becomes obvious once the top-25 result is in front of you. If the corpus contains many near-duplicate policies, the fix that helps most is not a better ranker over the duplicates but removing the superseded ones from the index, which is a reconciliation problem rather than a retrieval one. Quality work aimed at the retriever is frequently work that ingest should have done.

The belief this chapter is written against is that a retrieval metric is a safe stand-in for answer quality, and it survives because it is a defence that stays constant while the thing it must discriminate varies. Recall of the gold passage reads identically whether the model used that passage or wrote its answer out of the four documents sitting beside it, because recall stops looking at the point the passage enters the prompt. That is why the two numbers have to be carried together and why a single-column report is worse than no report: it moves in the right direction while the product gets worse, and it will keep doing so for as long as anybody is steering by it.

The most extreme version of that thought is that some corpora do not need retrieval at all, either because a key already determines the answer or because the whole corpus now fits inside a single prompt. Both cases are decided on the same labelled set and the same three axes, which is where this topic ends.