Recall over 41 candidates is not recall over 10

Multi-query expansion asks a model for several phrasings of the same question, retrieves for each of them, and merges the results into one candidate pool. It targets the vocabulary gap: when a document uses words the question never does, one of several phrasings has a chance of using the document's words even though the original did not. It is the largest single bucket in chapter 2's partition, so the repair matters.

The difficulty is that this repair changes the shape of the thing being measured, and the standard measurement does not notice. Recall@k is the share of questions whose gold chunk — the chunk marked by hand as containing the answer — appears in a candidate set of size k. The k in that name is a parameter of the system being measured, and an intervention that merges several result lists raises it as a side effect. A comparison that lets k move is not a comparison of the intervention.

VARIANTS = """Write 5 different search queries for this question.
Vary the vocabulary. Return one per line."""

def multi_query(question: str, index, k: int = 10) -> list[Chunk]:
    variants = llm(VARIANTS, question).splitlines()
    pool: dict[str, Chunk] = {}
    for v in [question] + variants:
        for chunk in index.search(v, k=k):
            pool[chunk.id] = chunk          # dedupe by chunk id
    return list(pool.values())              # typically 41 chunks

Six queries at k=10 is 60 retrievals, and after deduplication the pool is about 41 distinct chunks, all of which go to the generator. The baseline it was measured against sends one query, retrieves 10, and passes those 10. Reported against that baseline, recall@k moves from 0.61 to 0.84 on a 220-question labelled set — a 23-point gain, for 900 ms of added p95 latency and 3.4x the cost per query.

The two numbers are not measuring the same thing#

Recall@k is the share of questions whose gold chunk appears in the candidate set of size k. The baseline's candidate set has 10 members and the treatment's has 41, so the comparison is between a system allowed 10 guesses and a system allowed 41. A pool four times larger contains the gold chunk more often for reasons that have nothing to do with how the queries were phrased.

As an analogy, this is two candidates comparing exam marks when one of them was allowed to attempt four times as many questions. The analogy is worth disclosing and worth breaking in the same breath: on an exam the number of questions attempted is written on the paper, and here it is not. recall@k prints the same label whether k is ten or forty-one, so the denominator moves silently and the two numbers sit next to each other in a pull request description looking comparable.

The control that settles it costs one line: run the original question alone at k=41 and measure the same way.

Eighteen of the 23 reported points were available from the original query by asking for more results, at no additional generation call and no additional retrieval. The paraphrases contribute five points at a matched pool of 41, and eight points at a matched generator budget of 10, the difference being that a reranker over a pool containing genuinely different material has more to work with than one over a pool of near-duplicates.

Those two contributions add to the reported gain exactly, and the exactness is the argument rather than a coincidence worth noting in passing. The reported movement was measured between two configurations differing in two ways at once — the query set changed and the pool size changed — and the control run holds the query set fixed while moving only the pool. What it recovers is the share of the movement attributable to pool size alone. Subtracting that from the total leaves the share attributable to the query set, and because the two shares reconstruct the whole with nothing left over, there is no third effect hiding in the original comparison. That is what licenses saying the reported number was mostly a measurement artefact: not that it was implausible, but that its parts have been separated and counted.

The two matched comparisons disagree with each other, and the disagreement is a real residual rather than rounding. At a matched pool the paraphrases are worth five points, and at a matched generator budget they are worth eight — the smaller budget shows the larger effect, which is the opposite of what a reader expects. The mechanism is in the reranker's input. Reranking down to ten is a selection from whatever the pool contains, and a pool assembled from six near-identical legs offers forty-one chunks that are mostly variations on the same ten. A pool assembled from genuinely divergent legs offers forty-one chunks covering more distinct material, so the selection has more to choose between and the ten it returns are better. The paraphrases are not only adding gold chunks to the pool; they are improving the quality of the choice made afterwards, and that second effect only shows up when a choice is actually made.

Which of those two matched comparisons you care about depends on what is downstream. If the generator receives everything in the pool, the pool size is the budget and the honest figure is 0.84 against 0.79. If there is a reranker and the generator sees a fixed 10, the budget is 10 and the honest figure is 0.69 against 0.61. In both readings the intervention is worth somewhere between five and eight points, and in neither is it worth 23.

The reason the mistake is so easy to make is that recall@k with a floating k reads like a system metric when it is a metric of a stage. Nothing in the name warns you that the denominator moved. Reporting it as recall@10 (rerank) and recall@41 (pool) as separate lines, always with the candidate count in the label, removes most of the opportunity to fool a room.

The belief worth naming is that a recall figure is a check on a retrieval change. It is tempting because recall is the right property to care about and because the number did move in the right direction. What makes it false is that the figure is identical in form whether the change improved the queries or merely widened the window, and a check whose output looks the same in the case it should approve and the case it should reject is not checking anything. The candidate count in the label is what makes the two cases distinguishable, and it is the whole of the repair.

What the intervention costs#

The 3.4x is worth breaking apart, because most of it is not where people assume.

The five component rows add to the total row with nothing unaccounted for, which is worth checking before trusting any conclusion drawn from the table. A latency budget that does not close usually means a step nobody listed — a serialisation point, a retry, a cold cache — and any argument about which component to attack is worthless while that step is unnamed. Here the column closes, so the ranking of the components is the ranking of the opportunities.

The fan-out itself is nearly free, because six ANN searches over a 400k-chunk index run concurrently and the index was never the bottleneck. ANN stands for approximate nearest neighbour, which is the vector lookup itself: the index trades an exact scan of every chunk embedding for a structure that finds almost all of the closest ones in a few milliseconds. The cost is at the two ends: one generation call to write the variants, and a generator prompt four times longer if the pool goes in unfiltered. The token multiple is what produces 3.4x on the bill, so a reranker that cuts 41 back to 10 removes most of the added cost as well as fixing the measurement, which makes it the first thing to build rather than an optimisation for later.

Read the added column as gaps rather than as five levels and the shape is clearer still. The two ends of the pipeline dominate, and the three middle rows together are a small fraction of either of them. Both dominant rows are generation, and both scale with tokens rather than with the number of legs. So the intuition that multi-query is expensive because it does six retrievals is exactly backwards: doubling the number of legs would barely move the total, while doubling the number of chunks handed to the generator would move it a great deal.

What the deduplication rate is telling you#

The pool comes out at 41 rather than 60, so the six legs overlap by about 32%. That ratio is a diagnostic and worth logging per query rather than averaging away.

A pool near 15 means the paraphrases were lexical shuffles — "how do I rotate the signing key", "what is the process to rotate the signing key", "steps for rotating the signing key" — which embed to nearly the same point and retrieve nearly the same chunks. You paid for a generation call and received one query. A pool near 55 means the variants diverged so far that they are retrieving disjoint material, which happens when the model drifted off intent, and the extra chunks are distractors rather than second chances. On this eval, questions with a pool between 30 and 45 gained a mean of 0.11 recall; questions with a pool above 50 gained 0.03 and lost precision.

Deduplicating by chunk id is also the weakest available merge, because it throws away the ranks. A chunk that came back at rank 1 for four of the six legs is a much stronger candidate than one that appeared at rank 9 for a single leg, and set union treats them identically. Reciprocal rank fusion keeps that information and is about as much code as the dictionary was. It scores every chunk by one divided by the rank it achieved, summed across each of the lists it appears in, so a chunk near the top of several lists outscores a chunk buried in one, and the scores are comparable across legs without anyone having to calibrate similarity values against each other.

def rrf(runs: list[list[Chunk]], k: int = 60, top: int = 10) -> list[Chunk]:
    scores: dict[str, float] = defaultdict(float)
    seen: dict[str, Chunk] = {}
    for run in runs:
        for rank, chunk in enumerate(run, start=1):
            scores[chunk.id] += 1.0 / (k + rank)
            seen[chunk.id] = chunk
    best = sorted(scores, key=scores.get, reverse=True)[:top]
    return [seen[cid] for cid in best]

Why reciprocal rank rather than an average of the similarity scores is the design decision inside that function, and it is worth separating from the requirement, which is only that a chunk appearing well-placed in several legs should outrank one appearing badly in a single leg. Similarity scores would satisfy that requirement if they were comparable, and they are not: each leg is a different query, its score distribution has its own centre and spread, and a chunk scoring high against a narrow query is not doing better than one scoring lower against a broad one. Ranks discard the scale entirely, which is what makes them addable across legs without calibration. The reciprocal is what makes the top of a list matter more than the middle, since one divided by a small rank is much larger than one divided by a large one, and the differences shrink as you go down.

At the matched budget of 10, fusion lands at 0.71 against the reranked union's 0.69 and the baseline's 0.61, and it costs microseconds rather than a cross-encoder pass. The k=60 constant damps the influence of top ranks so a single leg cannot dominate; it is the value from the original fusion literature and is not worth tuning until everything else is settled.

Precision moves the other way#

Merging more candidates puts more material in front of whatever comes next, and not all of it is good material. On the same set, precision@10 after reranking falls from 0.42 to 0.38 while recall rises from 0.61 to 0.69, and end-to-end answer accuracy moves from 71% to 73%. Precision@10 is recall's counterpart over the same ten chunks: recall asks whether the passage you needed is in there, and precision asks what fraction of the ten deserve to be, so 0.38 describes a window in which roughly six of the ten chunks are material the generator has to read past.

Two points of end-to-end accuracy from eight points of recall is a normal conversion rate and it is worth understanding why it is not higher. Some of the newly retrieved gold chunks arrive alongside a near-miss chunk from another variant's leg — a passage about the adjacent product, or the previous version of the same runbook — and the generator now has to choose. On 6 of the 220 questions the baseline answered correctly and the multi-query version did not, and reading those six shows the same shape each time: the gold chunk was present in both runs, and in the multi-query run it was accompanied by a plausible competitor that the answer drew from instead.

Those six put a floor under the gross gain, which is a more useful reading than the net. Six questions out of a set that size are worth nearly three points of end-to-end accuracy on their own, and they moved downwards. For the net to be two points upwards, the questions that improved must be worth closer to five points than to two. So the layer is not gently lifting a fifth of the recall gain into accuracy; it is producing a substantial gain and a substantial loss whose difference happens to be small. Those two situations call for different work — one for more of the same, the other for finding out what the losses have in common — and the net number alone cannot tell them apart.

That is the cost that does not appear in any recall number, and it falls hardest on the queries that were already working. A precise query with a clear intent retrieves a tight, correct top-10; widening it to a fused pool of 41 can only add material that was ranked lower for a good reason.

Whether eight points is worth shipping#

Not on all traffic. The gain is concentrated exactly where chapter 2's partition said it would be: on the vocabulary-gap slice the mean gain is 0.19 recall, on compound queries it is 0.02, and on the well-specified questions that were already succeeding it is -0.01 with the precision loss above.

Routing it to the vocabulary slice changes the economics rather than the effect. If that slice is 20% of failures and failures are 38% of traffic, the layer runs on roughly 8% of queries, which turns 3.4x cost per query into about 1.2x overall and leaves p95 for the other 92% untouched. The decision then rests on whether the routing signal is good enough, which is the document-frequency check from chapter 1 and is a prior rather than a verdict.

That overall multiple is worth reconstructing rather than accepting, because it is the number a budget conversation will turn on. The routed traffic pays the full per-query multiple and the unrouted traffic pays exactly what it paid before, so the blended figure is the routed share at the higher rate plus the remaining share at the original rate. Multiplying out and adding gives the stated overall multiple to the tenth, which is the model closing on its own arithmetic. What it licenses is a specific rebuttal: the cost objection to multi-query is an objection to running it everywhere, and it evaporates under routing without anything about the layer itself changing.

The other input to the decision is the same paired regression list from chapter 3. Eight points net is a net; the six questions that broke are the ones to read before shipping, because if all six are from a single high-value query shape, a two-point average gain can still be the wrong trade.

Before swapping in a smaller model to write the variants#

The proposal to generate the five variants with a small local model removes most of the 340 ms and nearly all of the marginal cost, and it is a good idea that needs three checks first, none of which is "does it still produce five lines".

The first is constraint retention, which is the validator from chapter 3 applied to every variant. A paraphrase that drops the year or the version string is a leg retrieving the wrong period, and a small model instructed to "vary the vocabulary" varies whatever is easiest to vary. The second is the overlap ratio, because the entire value of the step is lexical divergence, and a small model's paraphrases tend to be syntactic — reordering clauses and swapping determiners while keeping every content word. If the pool size drops from 41 to 18 after the swap, the layer has stopped working while continuing to run. The third is domain vocabulary, which is the actual mechanism: the reason a variant retrieves the runbook is that the model knew certificate rotation is sometimes called rollover, and that knowledge is what a smaller model is most likely to lack in a specialised corpus.

The check that was ruled out at the top of this section deserves its own sentence, because it is the one a hurried substitution will actually run. Counting the lines returned is a test whose answer is identical for a model producing five genuinely divergent phrasings and a model producing five near-copies with the determiners swapped. It passes in both cases, so it distinguishes nothing that matters, and it feels like verification because something was checked. The overlap ratio is the test that moves between those two cases, which is why it is on the list and the line count is not.

All three collapse into one instruction, which is to re-run the matched-budget eval with the small model in place, on the routed slice, and compare against the same 0.61 baseline. Substituting a component in a pipeline whose value has been measured once does not inherit the measurement.

Five phrasings of a question are still five questions#

Every variant in the pool is interrogative. how do I rotate the signing key, steps to rotate the signing key, signing key rotation procedure — the model varied the words and preserved the form, because it was asked for search queries and that is what a search query looks like.

The document that answers it is not a question. It is a procedure written in the imperative, with headings, prerequisites and command lines, and its vocabulary is the vocabulary of doing the thing rather than of asking about it. Sampling more phrasings of the question moves you around inside question space, and the gap that is actually costing recall on the vocabulary slice runs between question space and answer space.

What this chapter leaves you with is therefore two separable results. One is about measurement and applies to every retrieval change, not only this one: a comparison in which the candidate count moves is not a comparison, and the fix is to state the count in the label every time. The other is about the layer itself, which repairs a real defect on a real slice of traffic and repairs it by a margin small enough that it has to be routed rather than deployed everywhere.

Closing the remaining gap means embedding something shaped like the document you want, which requires writing one.