Aggregate recall hides the query class where fusion actively hurts

A query class is a group of queries that behave alike against a retriever — identifiers, rare proper nouns, paraphrased questions, short keyword strings. The classes are a property of the traffic rather than of the system, and the reason they matter here is that the two arms of a hybrid retriever were chosen precisely because they behave differently across them. A single mean over a mixture of classes is therefore averaging over the exact variation the system was built to exploit.

This chapter is about what that mean conceals, how to declare the classes so the breakdown is evidence rather than rationalisation, and why the depth at which you measure changes the answer more than the fusion rule does.

Classes defined before the numbers are looked at#

The discipline that makes a per-class breakdown evidence rather than post-hoc rationalisation is defining the classes from the query log before running the comparison. On this robot-documentation corpus the taxonomy came from sampling 2,000 production queries and grouping them by what the user appeared to be doing, then writing rules that reproduce the grouping well enough to apply automatically:

import re

IDENT = re.compile(r"\b[A-Z]{1,3}-?\d{3,}[A-Z]?\b")

def classify(query: str, idf: dict[str, float]) -> str:
    if IDENT.search(query) or any(t in SKU_SET for t in query.split()):
        return "identifier"
    terms = query.lower().split()
    if terms and max(idf.get(t, 0.0) for t in terms) > 9.0:
        return "rare_term"
    if len(terms) >= 6 and terms[0] in WH_WORDS | AUXILIARIES:
        return "paraphrase"
    return "short_keyword"

Four classes, fixed, applied identically to every arm of the ablation from the last chapter. Pre-registering them matters for a reason worth deriving rather than asserting, because "with enough slices, something always regresses" is exactly the kind of claim that gets waved through.

Suppose each class is tested at the 5% level, meaning each test is allowed a one-in-twenty chance of calling a difference real when nothing is there. Then each test individually comes back clean nineteen times in twenty. If the four tests are independent, all four come back clean with probability nineteen twentieths multiplied by itself four times, which is a bit over eight tenths — so roughly one time in five, at least one of the four looks bad by accident. Four pre-declared classes therefore carry about a 19% chance of a spurious regression, which is a risk you can state and live with.

Now run the same arithmetic on forty slices invented after seeing the aggregate. Nineteen twentieths multiplied by itself forty times is a small number, so at least one spurious regression is close to certain, and a regression discovered that way is not information. The rule that keeps this honest is that the classes are part of the evaluation harness, checked into the repository next to the judged set, and changing them is a change to the harness rather than a step in an analysis.

The breakdown#

Same budget, same snapshot, same 412 queries, split four ways:

Check the last row against the four above it before drawing any conclusion from it, because that check is what makes the rest of the chapter safe. The class sizes sum to exactly 412, so every judged query is in one class and none is in two. Multiply each class's BM25 figure by its class size, add the four products, divide by 412, and the result is 0.66 — the aggregate on the bottom row. Do the same with the dense column and it comes to 0.71. Do it with the hybrid column and it comes to 0.78.

All three close, with no residual. That is not a coincidence and it is not a sanity check that could have failed for interesting reasons; it is the definition of a mean over a partition. What it licenses is the whole argument: the aggregate contains no information the breakdown lacks, the breakdown contains a great deal the aggregate lacks, and "the headline rose while a class got worse" is not a paradox to be explained but arithmetic to be read.

The bolded cells are not in the hybrid column. Fusion is not the best arm on any single class, and it is the best arm on the aggregate, and both statements come from the same six numbers. What hybrid does is never be terrible: on identifier queries it gives up 0.11 against BM25 but gains 0.31 over dense, and on paraphrased queries it gives up 0.04 against dense but gains 0.37 over BM25. Averaged over a mixed workload, never being terrible beats being excellent on 42% and hopeless on 23%.

That is a genuine and useful property, and it is also a much weaker claim than the one on the slide. The upper bound sitting next to it makes the point. Take the best arm in each row — 0.94 for identifiers, 0.85 for rare terms, 0.79 for paraphrases, 0.80 for short keywords — and average those four by the same class sizes. An oracle that knew each query's class and sent it to the best arm for that class would score 0.836, nearly six points above fusion. Some of that gap is unattainable, because a real classifier is imperfect and because "best arm for the class" is not "best arm for the query". Enough of it is attainable that "hybrid, globally, as the default" should be argued against routing rather than only against the single-arm baselines.

The mechanism behind the identifier row is the one chapter 3 traced on E-4417, now visible at scale. When the lexical arm returns the exact document with a large score margin and a short list, and the dense arm returns fifty plausible neighbours that agree with each other and with the lexical arm's second and third entries, rank fusion promotes the corroborated near-misses. The gold document was at rank 1 in a list of four; it ends up at rank 2 or 3 in a fused list of seventy.

Now count what the dashboard could see of that. The metric being watched is recall@10, and a document moving from rank 1 to rank 3 does not change recall@10 at all — it was in the top ten before and it is in the top ten after. So the metric is constant across precisely the degradation the users were reporting, and a quantity that does not move when the harm moves cannot rank configurations by harm. Only the queries pushed clean out of the top ten register, and the class figure says how many that was: 0.94 down to 0.83 across 96 queries is a loss of about a tenth of them, which is 10 of the 96 identifier queries losing the document entirely. The other 40-odd degradations were invisible to the metric being watched.

Measuring at the depth the consumer actually uses exposes the rest of it. The generation step on this system receives the top three chunks. At depth 3, the identifier class scores 0.91 for BM25 and 0.68 for hybrid — a gap of 0.23 rather than 0.11. Nearly a quarter of the queries that the lexical arm answered in the first position now fail to put the answer anywhere the model will see it. Recall@10 was never wrong; it was measuring a depth nothing in the product consumes.

Which of those deltas are real#

Three of the six comparisons in the table are worth arguing about and the rest are noise, and the arithmetic from chapter 4 sorts them. The smallest detectable paired difference is about 2.8 × σ_d / √n, with σ_d around 0.20 for per-query recall differences on this corpus:

The four class rows follow from the formula directly: the square roots of 96, 62, 172 and 82 are roughly ten, eight, thirteen and nine, and dividing 2.8 times 0.20 by each gives the fourth column. The last row is the one to read with a caveat. Its threshold is the nDCG figure carried over from chapter 4's sweep, computed with a smaller spread than the 0.20 this table uses for recall, so the honest bar for the aggregate row sits a little above what is printed. The verdict does not move — +0.07 clears either — but the row is not derived the same way the four above it are.

The identifier regression survives every correction you can reasonably apply to it, including a Bonferroni adjustment for four pre-declared classes, which divides that one-in-twenty allowance by the number of classes tested so that four slices have to clear a stricter bar than one would. The rare-term and short-keyword rows do not, and reporting them as regressions would be exactly the failure the pre-registration rule exists to prevent.

The paraphrase row sits on the boundary — the observed 0.04 against a bar of 0.043 — and the correct thing to say about it is that it is unresolved on this evaluation set. Settling it means pushing the bar comfortably below the effect rather than level with it, and the formula prices that: the bar falls with the square root of n, so halving it costs four times the queries. Four times the 172 paraphrase queries already judged is roughly 700, which is what it would take.

The other half of the evidence is not statistical. The two queries from chapter 1 — HX-2140R and the thermal question — are what turn a number into an explanation. The table says the identifier class dropped 0.11; the traces say the gold document went from rank 1 in a four-document lexical list to rank 3 behind a family overview and a compatibility matrix that both arms happened to agree on, which identifies the mechanism, points at rank fusion's treatment of margin as the cause, and suggests the two or three fixes that follow. An aggregate cannot do any of that, and neither can a per-class table on its own. This is why the concrete pair is worth more than the improvement number: the improvement number tells you something changed, and the pair tells you what and why, which is the only form of evidence you can act on and the only form that keeps working after the corpus, the analyser and the model have all moved.

Those pairs also make a regression test. Forty query/document/expected-rank triples, drawn from the classes that matter and asserted in CI, catch the identifier regression on the pull request that introduces it rather than eleven days after rollout:

CANARY = [
    ("HX-2140R",  "cat/hx-2140r/datasheet#lubricant",   1),
    ("E-4417",    "ref/fault-codes/E-4417",             1),
    ("Kirchgessner escalation", "esc/2025-08/kirchgessner", 1),
    ("why does the arm stop moving when the shop floor gets cold",
                  "man/px-arm/safety#thermal-interlock", 3),
]

def test_canary_ranks(search):
    for query, gold, max_rank in CANARY:
        hits = [h.id for h in search(query, k=10)]
        assert gold in hits[:max_rank], f"{query!r}: {gold} not in top {max_rank}"

The max_rank column is what makes this catch what recall@10 could not. Asserting that the gold document is in the top ten would pass on every one of the 40-odd degraded queries above. Asserting rank 1 for an identifier fails the moment fusion demotes it, which is the whole point.

Routing, pinning, or shipping nothing#

Declining a rollout is a decision about a specific configuration, not a verdict on hybrid retrieval, and there are three responses to this table that are all defensible.

Routing is the strongest. Send queries the classifier calls identifier to the lexical arm alone and everything else to the fused ranking. The classifier catches 92% of the class at 97% precision, and that first figure is what sets the realised quality: 92% of the class gets BM25's 0.94 and the remaining twelfth or so that the classifier misses stays on fusion's 0.83, so the class lands at 0.93 rather than BM25's 0.94. Feed 0.93 back into the identifier row and re-average by class size, and the aggregate lands at 0.81 — above the fused default, with the regression eliminated. The cost is a branch in the query path and a classifier whose behaviour has to be monitored, which is a smaller and more legible liability than a tuned fusion weight.

Exact-match pinning is the cheaper variant. When any query token matches the identifier pattern and a candidate document contains that token verbatim, place that candidate above the fused ranking. It requires no classifier, handles mixed queries where an identifier appears alongside natural language, and it is the behaviour users expect from a search box anyway. On this corpus it recovers the identifier class to 0.91 at depth 3, which is most of the loss.

The third response is to ship nothing and go back to the single best arm. Learned sparse alone scored 0.74 aggregate with a much simpler serving story than three indexes and a fusion layer, and on a team without the appetite to operate two retrieval systems that is a rational place to stop.

What would not be defensible is shipping the fused default on the strength of +0.07 with a measured, mechanistically explained, statistically resolvable regression on the class that the lexical arm was installed to serve — visible at the depth the product consumes, on 23% of traffic, with the ticket already filed. The honest version of the review slide has six rows, not one, and the recommendation attached to it is routing.

Everything decided so far has been decided offline, on a fixed snapshot, against a judged set whose class mix was chosen by whoever assembled it. The final chapter takes the winning configuration into production, where the fused query costs the slower of two retrievers rather than the average, where the two indexes drift out of sync with each other, and where the interesting question becomes what the system does when one of the arms stops answering.