Recall@k and answer accuracy are two different curves

A question-answering system built over a company's own documents has one dial that looks free. It is the number of passages the search step hands to the language model, and turning it up is a one-line change with an obvious argument behind it: more passages, more chance the answer is somewhere in there. That argument is correct about one measurement and silent about the other. Turning the dial up improves the measurement that asks whether the evidence arrived, without exception, while past a certain setting it drives the measurement that asks whether the answer was right steadily downwards.

The machine those curves are measured on is worth spelling out once, because every decision in this topic sits inside it. A support assistant's documentation, policy pages and support macros have been cut into passages and stored in an index, each held as an embedding — a list of numbers standing in for its meaning, so that searching means finding the passages whose numbers lie closest to the question's. When a user asks something, the system searches that index, pastes the best handful of passages into the prompt above the question, and asks a language model to answer from them. That arrangement is retrieval-augmented generation, almost always written RAG, and it exists because a model's weights carry its training data rather than what is in your wiki this morning: retraining whenever a refund policy changes is impractical, so the current policy is put in front of it at question time instead. The collection of documents being searched is the corpus, the component that searches it is the retriever, and top-k — written k — is how many passages the retriever is asked to hand back. A retrieval strategy is the set of decisions around that step: what a stored passage contains, how many come back, and what text each carries into the prompt. Documentation search, internal question-answering over a company's own files and code assistants reading a repository are the same machine with different corpora.

What each curve counts#

Take one support assistant with a judged set of 400 production queries, meaning 400 questions for which a person has gone through the corpus and marked which passages contain the answer. Raising k from 5 to 20 moved recall@k from 0.72 to 0.89, and moved end-to-end answer accuracy from 71.0% to 67.0%. Both figures are correctly computed. They disagree because they are averages of different things.

Recall@k is built one query at a time. For a single query, count how many of that query's judged-relevant passages appear anywhere in the top k the retriever returned, and divide by how many judged-relevant passages that query has in total. The result is a fraction between zero and one for that one query, and recall@k for the set is the mean of those fractions across the 400 queries. The harness later in this chapter computes exactly that: hit / len(relevant_ids), where hit is the size of the overlap between what came back and what was marked relevant. A query with one relevant passage therefore scores zero or one, while a query with four can score a quarter or a half and still be unanswerable.

Multiplying the mean by the set size converts it into a count, which is how the improvement gets described concretely: 400 queries at a mean of 0.72 is 288, and 400 at 0.89 is 356, so the usual sentence is that the evidence now reaches the prompt on 356 queries instead of 288. The arithmetic is right and the phrasing needs a condition. Because recall@k averages fractions rather than verdicts, 288 is the number of queries' worth of evidence retrieved, not the number whose evidence was retrieved in full. Those coincide only where every query has exactly one relevant passage.

Answer accuracy is built with no fractions in it at all. The system produces an answer, and an LLM judge — a second model shown the question, the produced answer and a hand-written reference answer, and asked whether the two agree — returns a verdict. On this set that judge was checked against a 60-query human audit and matched it to within two points. The verdict is binary, so a query needing three facts and getting two contributes zero, exactly as a query that retrieved nothing does. That asymmetry is the engine of the divergence: a change turning a quarter-answered query into a half-answered one moves recall and cannot move accuracy.

A second reason the divergence is possible is stronger, because it says the recall curve cannot turn over whatever the system does. When the retriever is asked for more passages, the shorter list is contained inside the longer one: the top 5 are the first five of the top 20, in the same order. The overlap between the returned list and the relevant list is therefore a count that can only grow as k grows, and the denominator does not move at all, since it depends on the judgments rather than on k. Every query's fraction is non-decreasing in k, and so is their mean. Recall@k rises with k as a matter of arithmetic, on any corpus, for any retriever, with no assumption about quality anywhere.

Answer accuracy carries no such guarantee. The prompt gained text, the added text is evidence for questions other than the one being asked, and nothing about a larger prompt constrains the model to answer at least as well as it did from a subset of it.

What the fifteen extra passages actually do#

The passages ranked sixth through twentieth are not random text. They are what a competent retriever thought was plausibly about the same thing, and being about the same thing both earned them their rank and makes them confusable with the answer.

A query like "how long do refunds take to appear on the customer's statement" pulls back the refund SLA (the answer), the chargeback SLA (a different number, same shape), the refund policy for annual plans (a different number, same shape), a changelog entry announcing that the refund SLA changed last March, and a support macro quoting the old value. The generator has to arbitrate between four numbers that all look like answers, and its arbitration is not free.

Three mechanisms drive the decline, worth separating because they respond to different fixes.

The first is competing facts. A near-miss passage — a distractor — states something in the same shape as the answer, so the model has a plausible candidate for every reading of the question rather than only for the correct one. Having a plausible candidate is what stops it declining to answer, which is why this mechanism shows up as confident wrong answers rather than as refusals.

The second is staleness, a special case of the first that gets worse specifically as k rises. An old version of a document says nearly what the current version says, in nearly the same words. Nearly the same words produce nearly the same embedding. Nearly the same embedding produces a rank immediately below the current version's rather than far down the list. So the passages a rising k admits first are disproportionately superseded copies of passages it already had, each carrying a value that used to be true.

The third is positional and is independent of what the passages say. Each stored passage here is a 512-token chunk, so k=20 hands the model a 10,240-token evidence block, which is what the table below records for that row. Attention over a long block is measurably less reliable in the middle than at the ends, so the passage ranked twelfth is read less carefully than the one ranked second even when it is the right one. Raising k moves evidence that used to sit at the top of a short block into the middle of a long one, a loss inflicted on the correct passage by passages that have nothing to do with it.

What the table below cannot do is apportion the decline between those three. It records the sum of their effects, so they are a claim about causes rather than a decomposition anybody measured here. Separating them takes different instruments — checking whether wrong answers cite a superseded passage, or re-running one setting with the evidence pinned to the top of the block.

Partial credit, and the queries it flatters#

The definitional trap sits inside the recall number itself. Recall@k gives a query with four relevant passages partial credit for finding one, and answer accuracy gives none when the question requires three of the four — a comparison across two regions, a policy plus its exception, a value plus its effective date. The two metrics disagree most where a partly-successful retrieval is worth nothing, and the disagreement flatters larger k. The variant that removes it is a one-line change to the same harness: instead of returning the fraction hit / len(relevant_ids), return one when hit equals len(relevant_ids) and zero otherwise. That is all-evidence recall, the fraction of queries where every judged-relevant passage is present, and it is a binary per-query verdict of the kind answer accuracy uses. On the set below, all-evidence recall at k=5 was 0.48 against the 0.72 headline figure.

Subtracting one from the other gives a quantity with an exact identity rather than an approximation. On a query with a single relevant passage the two definitions return the same value, since a fraction of one is one and a fraction of zero is zero. Every hundredth of difference between 0.48 and 0.72 is therefore contributed by multi-evidence queries scoring strictly between zero and one: some of their evidence found, scored as most of the way there, unanswerable. Just under a quarter of the headline recall here is that partial credit.

What the identity licenses is a ceiling rather than an estimate. At k=5 the share of queries holding all of their evidence is 0.48, so the share the generator could possibly answer in full is at most 0.48 — and answer accuracy there is 71.0%. The two are not in conflict, because the judged set marks every passage that supports an answer while a model can often answer correctly from fewer than an annotator marked. What the ceiling establishes is the direction of the error: a headline recall figure describes retrieval more optimistically than a query-by-query count does, and the strict variant is the one that tracks answer accuracy at small k.

Tokens, money and seconds as functions of k#

The cost side is simpler than the quality side and easy to forget while staring at a recall curve.

Three of those six columns are functions of the first and can be reconstructed from it, which is worth checking rather than assuming: each identity licenses a different reading of the table.

The prompt-token column is k multiplied by the chunk size. Every passage here is a 512-token chunk, and three of them is 1,536, five is 2,560, eight is 4,096, and so on to forty, which is 20,480. That holds on all seven rows exactly, and the absence of a remainder is the informative part: a real prompt also contains the user's question and whatever instructions wrap it, and none of that appears here. This column is retrieved text alone, so a production prompt is larger than every row by the same constant, the ratios between rows on a live system are gentler, and a capacity estimate built straight off these numbers understates the floor and overstates the growth.

The cost column is the token column multiplied by one fixed price per token, the same price on every row — a shade under three dollars a million input tokens on all seven, once each product is rounded to the nearest tenth of a cent. That second identity says something narrow and useful: nothing in this pricing is tiered, so no row is cheaper per token than any other. The rounding it hides makes a true relationship look false. The k=3 row's true product is a little under half a cent and prints as $0.005, rounded up; the k=8 row's is a little over a fifth of a cent above $0.012 and prints as $0.012, rounded down. Divide the printed figures against the k=5 row and the cost ratios come out slightly below the token ratios every time. So k=8 buys 2.5 points of accuracy over the shipped k=5 and costs 1.5× the input bill as the column prints it, while the underlying quantity of text is eight fifths — the ratio the token column gives, and the one to use for anything that has to hold at another price.

The latency column is the token column again in a different unit. Across the two ends of the table the p95 climbs by two and a half seconds while the prompt grows by a little under nineteen thousand tokens, which puts the marginal cost of a token at a fixed rate — the model working through the prompt at a little over seven thousand input tokens a second. Extending that rate back to an empty prompt leaves about a second and a half of latency not attributable to input: the time to produce the output, which is the same work whatever k is. Fixed cost plus a constant per-token rate reproduces all seven rows to within the rounding shown.

That third identity rules something out, which is why it matters most for reading the quality columns. Latency is affine in the token count from end to end, with no knee, no cliff and no discontinuity in the range, so nothing is timing out, nothing is being truncated, and no retry is firing at the far end of the sweep. The accuracy collapse between k=8 and k=40 is therefore a quality effect rather than an infrastructure artefact — the first alternative explanation anybody should want eliminated.

With the mechanical columns accounted for, the measured ones can be read against each other. Recall climbs the whole way and is still climbing at 40; answer accuracy peaks at k=8 and is below its k=3 value by k=20. The gap is the interesting thing rather than either level: from k=8 to k=40, recall gains fourteen hundredths while accuracy loses twelve points. The evidence reaching the prompt strictly improved over exactly the range where the answers got worse, which rules out the explanation everyone reaches for first — whatever is going wrong at k=40, it is not that the model lacked the material. Cost and latency are monotonic in the direction you do not want, so every setting past k=8 is bought with money, seconds and accuracy at once.

The sweep that produces a number you can defend#

The sweep is not complicated, and teams skip it because they have no labelled set rather than because the harness is hard. Two hundred queries with relevance judgments is enough to see the shape; 400 is enough to trust the peak's location to within one or two values of k. The queries have to come from production logs, sampled across the actual query mix, because a peak measured on hand-written questions sits somewhere else than a peak measured on what users type.

import json
from statistics import mean

K_VALUES = [3, 5, 8, 10, 15, 20, 40]

def recall_at_k(retrieved_ids, relevant_ids):
    if not relevant_ids:
        return None
    hit = len(set(retrieved_ids) & set(relevant_ids))
    return hit / len(relevant_ids)

def sweep(queries, retriever, generator, judge, k_values=K_VALUES):
    rows = []
    for k in k_values:
        recalls, correct, tokens = [], [], []
        for q in queries:
            # Retrieve once at max k, slice down — one index call per query,
            # not one per k. The ordering is identical either way.
            hits = retriever.search(q["text"], top_k=max(k_values))[:k]
            recalls.append(recall_at_k([h.id for h in hits], q["relevant_ids"]))
            answer = generator.answer(q["text"], [h.text for h in hits])
            correct.append(judge.is_correct(q["text"], answer, q["reference"]))
            tokens.append(sum(h.n_tokens for h in hits))
        rows.append({
            "k": k,
            "recall_at_k": round(mean(r for r in recalls if r is not None), 3),
            "answer_accuracy": round(mean(correct), 3),
            "mean_prompt_tokens": round(mean(tokens)),
        })
    return rows

print(json.dumps(sweep(labelled_queries, retriever, generator, judge), indent=2))

Two details in that harness matter more than they look. Retrieving once at the maximum k and slicing makes the top 5 at k=5 the same five documents as the top 5 at k=20, so the only variable across rows is how many you kept — and it is what makes the monotonicity argument above true of the measurement rather than only of the idea. Search the index separately per k and a retriever with any nondeterminism can return a document at k=5 that it does not return at k=20, at which point recall can fall between rows and the table stops meaning what it appears to.

The other detail is the if r is not None filter in the mean. recall_at_k returns None for a query with no relevance judgments, and the filter drops those from the average rather than scoring them zero. An unjudged query scored as a miss drags the mean down in proportion to how many the set contains, which is the most common way these tables end up wrong, and it is invisible afterwards because the curve has the right shape and the wrong height.

The judge has to be trusted before the curve means anything, since every accuracy number here comes from a model grading a model. Have a person grade 60 answers spread across the k values, compare against the judge, and report the disagreement rate. Two figures about that judge look as though they contradict each other and do not: it agrees with the human grader on 92% of items, and the audit put it within two points on the aggregate score. A disagreement rate bounds nothing on its own, because disagreements point in both directions and mostly cancel when averaged; the two-point figure survives that cancelling and governs a comparison between rows. Two points is fine for a four-point difference between settings and not for a half-point one, and a 60-item audit pins down a rate loosely — both of which argue for reading the peak's location rather than its height. Grading with the reference answer visible, and asking for a binary correct-or-not rather than a 1-to-5 score, removes most of the variance that makes these comparisons unstable.

Plotted on shared axes with k on the x-axis, the recall curve rises and flattens while the accuracy curve rises and turns over, and the gap between the peaks is diagnostic. Recall still climbing at 40 while accuracy peaks at 8 means the pipeline is finding relevant material deep in the ranking and failing to use it, so the ordering between rank 8 and rank 40 carries signal nothing is exploiting. That is a reranking problem rather than a k problem.

Rerank depth is the second axis, and it is where the tempting answer breaks#

The instinct that follows is to retrieve deep and rerank shallow: pull 200 candidates so recall is high, let the cross-encoder from the reranking work in topic 8.14 sort them, and pass the top 8 to the generator, giving the recall of a deep pool and the prompt of a shallow one. It is the right shape of idea and not automatically true, because it treats the reranker as a defence whose quality is constant while varying the one property that defence has to cope with. The depth axis changes how many plausible impostors are in the pool, and the reranker's accuracy at telling an impostor from the answer was measured somewhere else, at another depth. A quantity measured once and applied across a range it was never measured over is not a defence, it is a number wearing one.

That assumption fails in two ways. Rerankers are trained and validated on candidate pools of a particular depth, typically the top 100 or top 1000 from BM25 — the keyword scoring function that ranks a passage by how many of the query's terms it contains and how rare they are — and their calibration at rank 150 of a dense retrieval pool, drawn by embedding similarity rather than shared words, is not something anyone has measured on your corpus.

The second way is concrete and does not depend on training data at all. A deeper pool contains more near-duplicates: old versions of a page, a document and its translation, the same policy quoted in three onboarding guides. A cross-encoder scores a passage by reading it together with the query, so two passages differing in a few words score within a few points of each other, and a near-duplicate of the second-best answer scores just below it rather than far below. Widen the pool enough and there are three such copies, all scoring above the single correct passage, and the reranker promotes all three — filling the shortlist with variations on one wrong answer while the right one sits just outside.

The measurement that settles it is the same sweep with retrieval depth as a second axis: fix the reranker, vary how many candidates it sees and how many survive, and read answer accuracy off the grid.

Recall at depth keeps climbing and answer accuracy peaks at a retrieve depth of 50, the pattern to expect when a reranker meets a pool wide enough to contain plausible impostors. The numbers will differ on your corpus; the two-peaked shape usually does not.

The two tables agree where they overlap, which is worth noticing before either is used against the other. Retrieving 20 candidates gives recall 0.89 on this grid, and k=20 gives recall 0.89 on the sweep above. Different code doing different work lands on the same figure at the one setting they share, which is what licenses reading their accuracy columns together; without it, a difference between the tables could be a difference in the judged set rather than an effect of reranking.

With it, the comparison is exact. The best cell on this grid is 76.5% and the best on the single-axis sweep is 73.5%, and both put eight passages in front of the generator. Eight passages is 4,096 tokens on both, so the three points cost nothing extra — the same money, the same prefill. That is the actual argument for reranking here: it moves the accuracy peak up rather than sideways.

Rerank cost is a fixed per-call overhead of a couple of milliseconds plus a constant charge per candidate scored, just under two milliseconds each, which reproduces every row of the latency column. The depth-20 row divides out to a slightly higher cost per candidate than the rows below it, which is the fixed overhead spread across fewer candidates rather than inefficiency at shallow depth. Reranking is priced linearly in depth, so the accuracy peak at 50 is not a latency artefact. What that column will not support is addition: it reports p50 while the sweep reports p95, and adding 95 ms to a 1.9-second p95 does not give the p95 of the combined system, since the slowest requests overall are not the requests with the median rerank cost.

A tuned k is a property of one pipeline#

A tuned k is not a property of your problem domain and does not transfer between systems that look similar. It is a property of one combination of chunker, embedder, index, reranker, generator and query mix, and it expires when any of them changes. Re-chunking from 512-token windows to heading-aware sections changes the token cost of each unit and the context each already carries, moving the peak down, since larger units mean fewer are needed. Swapping the embedder changes what sits at ranks 6 through 20, precisely the region the peak is sensitive to. A better long-context generator moves the peak up, sometimes substantially, and a team that tuned k=8 against a 2023 model is leaving accuracy on the table in the opposite direction from the one this chapter opened with.

Two beliefs make that expiry easy to miss, and both have the same shape: a defence that stays constant while the thing it is supposed to discriminate moves. The first is the framework default, whose k is the same integer for every reader, every corpus and every model — constant by construction. The property it would need to track varies enormously between corpora: how densely near-misses cluster around an answer, how many superseded copies of a document the index holds, how many facts a question needs. A number that cannot vary cannot be tracking any of that, and appearing in a widely used library's documentation is not evidence about your corpus.

The second is the absence of an alarm. After an embedder swap nothing fires — no error, no failed deployment, no page. That silence gets read as confirmation that k survived, and it is evidence of nothing, because no part of the system measures whether k is still at the peak. A pipeline two settings past optimal and one that is exactly right emit the same signal, which is none, and the only instrument that separates them is the sweep, re-run. The residue is a piece of hygiene: record the chosen k next to a hash of the config that produced it, and treat a config change as invalidating the number. Teams standardise on k=10, the reason it was 10 leaves with the engineer who chose it, and eighteen months later the number is defended on the grounds that it has always been 10.

Queries no deployable k reaches#

Tuning k does not fix a retrieval failure either. If the evidence for a query is ranked 340th, no deployable value of k reaches it, and the recall curve flattening below 1.0 tells you how many queries are in that state.

On the set above, recall at 40 is 0.93, and reading that as "7% of queries have evidence the retriever essentially cannot find at any sane depth" is right in spirit and needs its condition. What 0.93 measures is the mean per-query fraction, so 7% is the share of judged-relevant passages that never surface, averaged over queries. The share of queries touched by that miss is at least 7%, and larger when the misses are spread across many queries rather than concentrated in a few — which the gap between all-evidence recall and headline recall says they are here. Either way, those queries need a different index, a different chunking decision, or a hybrid lexical channel — a keyword search running alongside the embedding search, with the two result lists combined.

Two curves over one dial, then, and only one is the one to tune against. Recall@k rises with k by arithmetic and keeps rising past every setting worth deploying, so a rising recall number is not news and cannot on its own justify a change. Answer accuracy rises, turns over and falls, and where it turns over is the number to ship — with rerank depth as a second axis when the two peaks are far apart, since that distance is the system telling you it is finding material it is not using.

What the sweep cannot tell you is whether the unit being counted is the right unit. Every row assumed that passing 8 passages means passing 8 fixed-size slices of text, the kind the chunking strategies in topic 9.5 produce. Changing what a retrieved unit contains is a larger lever than changing how many you pass, and the first move in that direction is to stop assuming that the thing you match against and the thing you send to the generator have to be the same object.