Index recall and retrieval relevance are two different numbers

Two measurements in a retrieval system are both called recall, they are computed against different reference points, and neither of them responds when the other one moves. One compares what the system returned against what a human judged relevant to the question. The other compares what an approximate index returned against what an exhaustive scan of the same vectors would have returned. An approximate index is a structure that finds a query's nearest stored vectors without comparing the query against all of them, which it achieves by declining to look at most of the space, and declining to look is what makes it capable of being wrong.

What makes the confusion durable is that both numbers are reported as a fraction close to one, both are called recall at k, and only one of them is usually on a dashboard. The sentence that follows from that arrangement is easy to write and false: our recall@10 has been steady for six months, so retrieval is not the problem. The relevance number is steady because the things it measures — the chunking, the embedding model, the coverage of a corpus of support articles — have been steady, and the index it is being used to exonerate is not among them.

Two things called recall#

Retrieval relevance asks whether the system found documents that answer the question. It needs human judgments, or a proxy for them, and its ceiling is set by whether the corpus contains an answer at all. A recall@10 of 0.79 on a judged set means that for the average query, roughly four out of five of the documents a human marked as relevant made it into the top ten — and a perfect index cannot push that past whatever the embedding model and the chunking allow. The companion measure quoted alongside it, nDCG — normalised discounted cumulative gain — scores the same list but discounts each relevant document by how far down the ranking it sits, so it moves when good documents sink as well as when they disappear.

Index recall asks a much narrower question: for this query, did the index return the same documents that a full scan would have returned? It is defined against the exact top-k from chapter 1, where a linear scan over every stored vector returns the true nearest neighbours by construction, and not against any human opinion. It is written R@k:

R@k  =  |approx_top_k(q)  ∩  exact_top_k(q)|  /  k

The divisor is k for a reason worth pausing on, because it is what makes the number readable without a second figure beside it. Both sets have exactly k members: the index returns k documents and the scan returns k documents. Their overlap can therefore be no larger than k and no smaller than zero, so dividing by k lands the score between zero and one, with one meaning the two sets are identical. It also collapses two measures into one. Precision would divide the overlap by the size of what was returned and recall by the size of what should have been returned, and here those two denominators are the same number, so there is nothing to report separately.

If exact search says the ten nearest chunks to a query are IDs {41, 903, 12, 5588, 77, 210, 3391, 8, 1204, 66} and the index returns eight of those ten plus two others, R@10 for that query is 0.8. Eight correct out of ten slots is 0.8, and the two intruders are not counted a second time as false positives — they cannot be, because a list of ten with eight correct entries has exactly two slots left, so every miss creates one intruder and every intruder displaces one miss. Under this definition the two are one event counted once. Averaged over a query set, that average is what people mean when they say "HNSW gets 99% recall". It is a property of the index and the parameters it was built and searched with, and nothing else. Swap the embedding model and it barely moves. Swap the corpus for one with a different geometry and it can collapse.

The row that does the work is the last one. A relevance regression shows up as user complaints and a moving dashboard. An index recall regression shows up as nothing, because the substituted neighbours are plausible — they are, after all, still near the query in embedding space, just not the nearest — and a language model handed the seventh-best chunk instead of the second-best writes a confident answer either way. This is the whole reason a team can carry an index at 0.79 index recall for a year while believing the folklore figure of 0.99.

That last row is a conclusion rather than an observation, and the chain under it is worth following link by link, because each link is checkable and the final one is a choice somebody made rather than a fact about retrieval. The index drops a chunk that was genuinely among the nearest and puts a slightly more distant one in its place. The substitute is still near the query in embedding space, since candidates near the query are the only ones the structure ever examines. Near in embedding space means near in meaning, since producing that correspondence is what the embedding model was trained for. So the substitute is usually another chunk of the same article, or a near-duplicate paragraph from a second version of it. A judgment set marks whole articles relevant rather than individual chunks, so the substitute scores as a hit exactly as the chunk it displaced would have. The relevance metric therefore does not move.

The last link is the design decision, and it is the only one that could have gone otherwise. Judging at article granularity is what makes human judging affordable, because a person can say whether an article answers a question far faster than they can rank the passages inside it. Nothing about retrieval requires it. It is also the exact step that renders this defect invisible, so a team that judges at chunk granularity buys sensitivity to precisely this failure and pays for it in judging time. Naming which link is a choice is what tells you where the instrument could be changed, as against where the physics is fixed.

The stumble this chapter exists to prevent is a belief that a defence works. The relevance dashboard is the defence: it is the instrument a team points at retrieval in order to decide whether retrieval is at fault. A number that reads the same whether the index is faithful or is substituting a fifth of every result set cannot separate those two states, and a quantity that does not move when the dangerous property moves cannot be used to order by. Stability on that dashboard is evidence that the corpus and the model are stable. It is not evidence about the index, and reading it as though it were is how the index becomes the one suspect nobody ever interviews.

What the substitution actually looks like#

A support-search team ran both measurements on the same 240 judged queries, once against a production HNSW index and once against an exact scan over the identical vectors.

Two rows of that table are the same measurement written twice, and checking that they close is what tells you the whole table came out of one run rather than two experiments stitched together. Index recall of 0.79 means that on average a shade under eight of the ten returned documents were also in the exact top-10. The rest — 2.1 of them — are what the index substituted, and 2.1 of 10 is exactly what the last row prints, with no residual anywhere. What that identity licenses is a change in how the first row is read. It is not a score. It is a count of substituted documents divided by ten, which is the form product can act on, and the argument in the next paragraph closes only because the two rows are the same fact.

The two 0.79s in the first column are a coincidence of this dataset and they are the reason the confusion is so durable — the number on the dashboard happened to match the folklore figure for index quality closely enough that nobody looked twice. Switching to exact search changed on average 2.1 of the 10 returned documents per query and moved the relevance metrics by three or four hundredths, which is inside the run-to-run noise of an LLM-judged evaluation and would have been dismissed as flat. That is the mechanism in full: the index drops a genuinely nearest chunk and substitutes one that is a little further away, the substitute is usually from the same article or a near-duplicate of it, and a judgment set that marks whole articles relevant cannot tell the two apart.

The four hundredths are not nothing, and on a subset of queries they were much larger. Segmenting by query type showed judged recall@10 rising from 0.71 to 0.82 on queries mentioning a specific error code, where exactly one chunk in the corpus is the right answer and substituting a neighbour means substituting a wrong answer. That segment gained around a tenth while the whole set gained 0.04, and an average is a weighted sum of its parts, so a part that gained a tenth against a whole that gained well under half of that forces the remaining queries to have gained close to nothing. Averaged over a query mix dominated by broad questions with many acceptable answers, that improvement disappeared — and "dominated" there is a consequence of the arithmetic rather than an assumption about the traffic. A relevance metric can therefore be simultaneously correct, stable, and completely blind to a defect that is costing real users real answers, which is why the index number has to be measured separately rather than inferred.

Building the ground truth#

The comparison needs an exact answer per query, computed once and stored. That is the same scan from chapter 1, where the cost of exhaustive search is worked out as bytes divided by memory bandwidth, run offline over a fixed query set, and its cost is negligible: 500 queries against a 200k corpus at 8 ms each is four seconds of compute. Against a 3M-vector corpus it is 500 × 120 ms, or a minute. There is no scale at which this is the expensive part of your evaluation.

Neither per-query figure there was measured for this purpose. Both are the previous chapter's model evaluated at a corpus size, and three million vectors sit between the one-million and five-million rows of that chapter's table, which is where the 120 ms comes from — between the two scan times those rows print, exactly where growth in proportion to the byte count puts it. Multiplying each by 500 queries gives the four seconds and the minute.

import numpy as np, json

def build_ground_truth(queries: np.ndarray, corpus: np.ndarray, k: int = 10):
    """queries (nq, d) and corpus (n, d), both L2-normalised, inner product."""
    gt_ids, gt_scores = [], []
    for i in range(0, len(queries), 64):                 # batch to bound memory
        block = queries[i : i + 64] @ corpus.T           # (64, n)
        idx = np.argpartition(-block, k, axis=1)[:, :k]
        rows = np.arange(len(idx))[:, None]
        order = np.argsort(-block[rows, idx], axis=1)
        top = idx[rows, order]
        gt_ids.append(top)
        gt_scores.append(block[rows, top])
    return np.vstack(gt_ids), np.vstack(gt_scores)

def index_recall(approx_ids: np.ndarray, gt_ids: np.ndarray) -> np.ndarray:
    """Per-query R@k. Mean it yourself; the distribution matters more."""
    k = gt_ids.shape[1]
    return np.array([
        len(set(a[:k]).intersection(g)) / k
        for a, g in zip(approx_ids, gt_ids)
    ])

Two choices inside that function repay reading rather than skimming. The loop steps through the queries 64 at a time because block holds a score for every query in the step against every stored vector, so its size is the step size multiplied by the corpus size — a matrix that is ordinary at 64 rows and unmanageable if a whole query set were scored at once. And the selection happens in two stages for the same reason chapter 1's baseline used argpartition rather than a sort: the partition drags the k largest scores to the front without ordering anything else, and the argsort after it has only k values to put in order instead of a whole row.

Three details decide whether the resulting number means anything.

The query set has to come from production logs, not from a synthetic generator and not from the golden relevance set. Judged queries were chosen because a human could evaluate them, which biases them toward well-formed questions with clear answers, and those are exactly the queries an approximate index handles well. A sample of 500 real queries drawn uniformly from a month of traffic will include the malformed ones, the two-word ones, and the ones about a product area added last week — and those are where the index loses. Two hundred queries is enough to see a mean; 500 is enough to see the distribution, which chapter 5, on the population of queries approximation loss lands on, argues is the number you actually need.

How large that set has to be is the same detail seen from the other side, and it deserves a moment of arithmetic rather than a guess, because "we ran it on 50 queries" is common and 50 is not enough. Per-query R@10 is bounded in [0, 1] and, on a real index, has a standard deviation around 0.25 to 0.35 because of the bimodality described later in this topic, meaning that the per-query values pile up at the two ends of the range rather than clustering around the average. A quantity confined between zero and one has a hard ceiling on how far it can spread at all, and it only comes near that ceiling when its mass sits at both ends, so a standard deviation in that band is itself the evidence for the word bimodal. The standard error of the mean is that divided by the square root of the query count, so 50 queries give ±0.10 at 95% confidence, 200 give ±0.05, and 500 give ±0.03. Those three intervals use the pessimistic end of the standard-deviation range: take 0.35, divide by the square root of the query count, and double it to reach 95% confidence. The square root of fifty is a little over seven, and 0.35 divided by that and doubled is 0.10. The three figures also stand in a fixed relation to each other without the standard deviation entering at all — 50 to 200 is a fourfold increase whose square root is two, which is why 0.10 halves to 0.05 — so the set is one measurement and a scaling law rather than three separate results.

A sweep that is trying to distinguish recall of 0.94 from 0.96 needs the 500-query set; a check that is trying to notice a collapse from 0.95 to 0.70 is fine with 100. Read against the intervals above, the first of those looks impossible, since the gap between 0.94 and 0.96 is smaller than the ±0.03 that a 500-query set puts around either of them. The resolution is that the two quantities answer different questions. An interval of ±0.03 is uncertainty about where the mean over all possible queries sits. Fixing the query set rather than resampling it each run removes most of the remaining variance from comparisons, since the same queries are being asked of every configuration, and a comparison on fixed queries asks only whether one configuration beat another on these particular queries. Every query that both configurations handle identically contributes nothing to that difference and nothing to its uncertainty. The 500-query set is needed for a separate reason: a smaller fixed set leaves too few queries on which the two configurations disagree at all.

The ground truth is tied to a specific corpus state. If a document is added, deleted or re-embedded, the exact top-10 for some queries changes, and a stored ground truth computed before that change will report a recall loss that is really a corpus diff. The fix is to version the ground truth with the corpus snapshot it was computed from and recompute it whenever you rebuild — which is cheap, per the arithmetic above, and is the reason the recomputation should be a step in the reindex job rather than a manual task somebody remembers.

Ties need a decision. When several chunks sit at identical or near-identical scores — common with duplicated boilerplate, or with the same paragraph appearing in three versions of a document — the exact top-10 is one arbitrary ordering among many equally correct ones, and an index that returns a different arbitrary ordering is penalised for nothing. Recording the exact scores alongside the IDs, as the function above does, lets you compute a tie-tolerant variant: count an approximate result as a hit if its score is within a small epsilon of the k-th exact score. That is what the second return value of build_ground_truth is for, and discarding it to save storage is a decision that costs you this check months later. The two numbers usually differ by a couple of hundredths, and when they differ by more than that it is itself informative, because it means your corpus has a duplication problem.

Fidelity is not quality, and membership is not order#

Index recall of 1.0 does not mean retrieval is good, and the sentence to watch for is the mirror image of the one this chapter opened on: our index gets 99% recall, so retrieval is solved. An exact scan over a corpus that has been chunked at the wrong granularity, or embedded with a model that does not understand your domain vocabulary, returns exactly the wrong documents with perfect fidelity. The two numbers are independent in both directions, which is precisely why you need both: relevance tells you whether the pipeline answers questions, index recall tells you whether the index is the reason it does not. Reporting only one of them leaves a permanent unresolved suspect in every retrieval investigation.

Index recall is also silent about ordering within the returned set. R@10 counts membership, so an index that returns the correct ten documents in reverse order scores 1.0. For a RAG pipeline — retrieval-augmented generation, where a question is answered by placing retrieved passages in a language model's prompt — that is the right thing to measure, because all ten chunks go into the prompt and ordering barely matters once they are in context. For a reranker, meaning a slower and more accurate model that rescores whatever the index handed it, and which only sees the top 50 candidates, or for a UI that shows three results, ordering matters and the metric to use is rank-sensitive — R@1 alongside R@10, or an ordering-aware measure. Choosing k to match how the results are consumed is part of the setup, and a recall figure quoted without its k is not a number.

Finally, a single averaged R@10 hides a great deal. A mean of 0.86 is consistent with every query losing one or two of its ten neighbours, and it is equally consistent with 85% of queries being perfect and 15% returning almost nothing correct. Those two worlds call for entirely different responses — the first is a parameter slightly too tight everywhere, the second is a region of the corpus the index cannot reach — and the average cannot distinguish them. That is why the harness above returns per-query values rather than a mean.

Two reference points, then, and no substitute for either. Relevance is measured against what a person judged useful and answers the question of whether the pipeline is worth anything. Index recall is measured against what an exhaustive scan returns and answers the question of whether the index is faithful to the vectors it was built over. Because they are computed against different things, one can sit perfectly still while the other collapses, and any argument that reasons from one to the other is reasoning across a gap that is not there to be crossed.

With ground truth in place there is a number that responds to what the index is doing, and the next question is what makes it move. That means opening up the indexes themselves, starting with IVF, whose structure makes the source of the loss unusually easy to see: the corpus is cut into cells at build time, and a query only ever looks inside a few of them.