Stratifying by query shape

An aggregate recall figure collapses each query to a bit and averages the bits, so it is invariant to which queries failed. Two runs can produce the same number by failing on disjoint sets of users, and the average has no way to say so. Stratification is the repair: partition the labelled queries into groups defined by a property that changes how a ranker scores, then report the metric per group.

The properties worth partitioning on come from the mechanisms in chapter 4, on what each ranking family scores. BM25 scores by rare-term overlap, so its behaviour changes sharply when a query contains a rare exact token or contains no exact match at all. A dense retriever — the embedding path, called dense because each vector is a full array of numbers rather than a sparse list of term counts — scores by learned proximity, so its behaviour changes with paraphrase distance and with tokenization damage. Those are hypotheses about causes, and they are what makes a stratum legitimate rather than a slice invented after seeing the data.

Where the failure sets overlap#

Before any stratification, one computation is worth two minutes, because it settles whether the two systems are interchangeable at all.

bm25_miss = {q for q in qrels if not set(bm25_run[q][:10]) & qrels[q]}
dense_miss = {q for q in qrels if not set(dense_run[q][:10]) & qrels[q]}

print(len(bm25_miss), len(dense_miss))          # 78 78
print(len(bm25_miss & dense_miss))              # 31
print(len(bm25_miss | dense_miss))              # 125
print(1 - len(bm25_miss & dense_miss) / 300)    # 0.897

Only 31 queries defeat both systems. The two failure sets overlap on 40% of their members and are otherwise disjoint, which means an oracle that could pick the better ranker per query would score 0.897 — sixteen points above either system alone, and about as much headroom as retrieval work on this corpus is likely to produce all year. It also means the two systems are not interchangeable in the way the identical averages suggested: swapping one for the other changes which 78 users get a bad answer, not how many, and the two groups of users are mostly different people asking different kinds of question.

Every one of those four printed lines follows from the two sets, and working them through is what makes the last one trustworthy. The two miss counts are the same because both systems hit 222 of 300, and 300 minus 222 is 78 either way. The union comes from counting the two sets and then subtracting what was counted twice: seventy-eight plus seventy-eight is a hundred and fifty-six, minus the thirty-one queries in both, gives 125. The oracle line is the one worth dwelling on. An oracle that sees both result lists and picks the better one per query fails only where there is nothing to pick — on the queries both systems missed. Its miss count is therefore exactly the size of the intersection, and its recall is one minus thirty-one over three hundred, which is the 0.897 printed. Nothing about the oracle is implementable; it is a ceiling, and it says that everything routing or fusion could ever buy on this corpus lives in the roughly ninety-four queries that one system found and the other did not.

That ceiling is what turns the identical averages from a curiosity into a warning. An average recall is a defence in the sense that people use it to decide two systems are equivalent, and here it reports 0.74 whether the two systems fail on the same users or on entirely different ones. A number that does not change when the thing it is being asked about changes completely cannot answer the question it is being asked. The overlap count does change, which is why two minutes of set arithmetic is worth more than the metric it is checking.

Splitting by the shape of the question#

The strata have to be defined before you look at the results, because a slice invented after seeing the data will find a difference somewhere and mean nothing. Six of them cover the traffic on a technical documentation corpus. Assigning them is mostly mechanical — a regex for identifier-shaped tokens, a spellchecker pass against the corpus vocabulary, a negation-word list — with a manual pass over the residue that took one person about forty minutes for three hundred queries.

import re

IDENT = re.compile(r"[A-Z]{2,}_\d+|\b\w+_\w+\b|\b[a-z]+[A-Z]\w+\b|/v\d+/")
NEGATION = {"not", "without", "except", "excluding", "unsupported", "cannot"}


def stratum(q: str, vocab: set[str]) -> str:
    toks = q.lower().split()
    if IDENT.search(q):
        return "exact_identifier"
    if NEGATION & set(toks):
        return "negation"
    if any(t.isalpha() and t not in vocab for t in toks):
        return "misspelled"
    if len(toks) <= 3:
        return "short_keyword"
    if sum(1 for t in toks if t.istitle()) >= 2:
        return "multi_entity"
    return "paraphrase"

Every query lands in exactly one stratum here, by first match, which loses some information — a misspelled negation query is filed as negation — and buys a table whose rows sum to the total. Overlapping strata are defensible too, provided you stop reporting a column that adds up to the headline figure and say so explicitly, because a reader who assumes the rows partition the set will draw wrong conclusions from the counts.

Before reading any row, check that the table closes, because a stratified table that does not close is reporting overlapping slices and its rows cannot be compared with each other. Add the n column downwards: fifty-four, sixty-one, twenty-two, ninety-six, forty-nine and eighteen come to exactly three hundred, with no residual and no "other" row absorbing the difference. Add the BM25 hits column: fifty-one, fifty-three, nine, sixty-five, thirty-six and eight come to exactly 222. Add the dense hits column and it comes to 222 as well. Both hit totals divided by three hundred give 0.74, which is the aggregate the whole exercise started from, and three hundred minus 222 gives 78, which is the miss count the set arithmetic printed a page ago.

Three columns closing exactly is not decoration. It licenses two things the table would otherwise only appear to say. Because the counts partition the set, the aggregate is a weighted average of the six rows and nothing else, so no stratum can be hiding outside it and no query is contributing to two rows at once. And because the same partition holds for both systems, a gain in one row is a real gain rather than a query borrowed from another row — which is exactly the confusion an overlapping set of slices produces, where improving one slice's number can silently worsen a slice that shares its members.

Every row is a decision the aggregate was hiding. Users pasting a function name, an error code or an endpoint path — 18% of this traffic — are served correctly 94% of the time by the system somebody was about to delete and 61% of the time by the one that was going to replace it. Users who cannot spell pagination are served almost twice as well by the dense retriever, and there are enough of them to notice.

Reading three individual failures makes the rows concrete in a way the percentages do not. The query ERR_4021 on webhook delivery is one of the 21 identifier queries the dense retriever missed: its top three results are the webhook overview, the retry-policy page and a troubleshooting guide for ERR_4019, all of them plausibly about webhook errors and none containing the code. BM25 put the correct page first, on the strength of a single term with a document frequency of 2. Going the other way, how do I stop the sync from running twice is one of the 31 paraphrase queries BM25 missed entirely: the page that answers it is titled "Idempotency keys" and shares only the stopwords how, do, the and from with the query, so there is nothing for a term-matching scorer to score. And paginaton parameters — a real query, typed as shown — returns nothing useful from BM25 because the misspelled token has a document frequency of zero, while subword tokenization leaves the dense retriever with most of the meaning intact.

Both of those counts come straight out of the table rather than from a separate tally, which is worth checking because it is how you know the examples belong to the rows they are being used to illustrate. Fifty-four identifier queries minus the thirty-three the dense retriever hit is twenty-one. Ninety-six paraphrase queries minus the sixty-five BM25 hit is thirty-one. That second thirty-one is not the thirty-one from the set arithmetic above, and the collision is worth naming before it causes trouble: one is the number of queries both systems missed across the whole set, the other is the number of paraphrase queries BM25 alone missed. They are different populations that happen to be the same size, and a reader who conflates them will conclude that every shared miss is a paraphrase query, which the negation row alone disproves.

Reporting where each system fails, rather than only where it wins, is what makes those examples reachable later. Alongside the table, the run writes out the failing query identifiers per system per stratum, so the 78 BM25 misses can be read as a list rather than as a number. That list is also the raw material for the trap cases in chapter 3, on constraints similarity cannot carry: the failures that recur across releases are the ones worth promoting into named regression cases, and you can only notice recurrence if the identity of the failing queries is recorded on every run rather than recomputed and discarded.

The negation row is the one worth sitting with. Both systems fail more than half of those queries, and combining them cannot fix it, since fusing two rankers can only promote a document that at least one of them retrieved, and on this row there is very little to promote. BM25's eight hits leave ten misses out of eighteen; the dense retriever's seven leave eleven. The queries neither system rescued therefore number at most ten, because two sets cannot overlap on more than the smaller of them — an upper bound the table hands you without any extra measurement, and the one an instinct to add the two miss counts together gets badly wrong. No weighting scheme, no reciprocal rank fusion — the standard way of merging two ranked lists, which scores each document by adding up 1 divided by its rank in each list it appears in — and no threshold produces a document that is not in either list. That row is a requirement for a different mechanism — a reranker that reads query and passage jointly, or a query rewrite that turns "which regions are not covered" into a lookup against an exclusions field — and it stays visible only because it has its own row.

It is also the row that most needs a caveat. With n = 18 and an observed recall of 0.44, the standard error is sqrt(0.44 × 0.56 / 18) = 0.117, so the 95% interval runs from roughly 0.21 to 0.67. The honest reading of BM25's 0.44 against dense's 0.39 in that stratum is that the set cannot tell them apart. The same arithmetic on the paraphrase row, with n = 96, gives a standard error near 0.048 for each system, and an 18-point gap there is real. Reporting the interval alongside the point estimate for every stratum, or simply refusing to draw conclusions from strata under about 40 queries, is what stops a stratified table from becoming a machine for generating confident nonsense out of small cells.

Where the forty comes from is the same expression read backwards, and it is worth having, because the threshold is otherwise a number to memorise. The stratum size sits under a square root in the denominator, so the interval narrows as the square root of the count and no faster. At eighteen queries the interval is nearly half the scale wide, which is another way of saying the row is compatible with almost any true value. Quadrupling that stratum to seventy-two would halve the interval and no more. Forty is where two systems separated by a gap of the size these rows actually show — a fifth of the scale or so — start to be distinguishable at all, and below it the row is worth reporting as a count of failures to go and read rather than as a rate to compare. That is also the honest reason the negation row survives in the table despite being uninterpretable as a rate: eight and seven are facts, 0.44 and 0.39 are noise, and the row is in the table because eighteen queries nobody can serve is a finding regardless of which system is nominally ahead.

Reading the table for a decision#

The table supports three different actions, and which one it recommends depends on the shape of the gaps rather than on their size.

Routing looks attractive on this table because the identifier stratum is detectable with a regex whose precision on this traffic was 0.97. It is the option that fails silently, though: a query the classifier misroutes gets the worse system with no signal anywhere that a choice was made badly, and the classifier's error rate multiplies into the retrieval error rate. Fusion is unconditional and needs no classifier, at the cost of running both retrievers on every query — on this corpus, about 8 ms for BM25 and 14 ms including the embedding call for dense, so roughly 14 ms wall-clock when run concurrently and two systems to keep in sync at index time.

What "multiplies into the retrieval error rate" means is worth making concrete, because the phrase sounds like a caution and is arithmetic. A precision of 0.97 means three queries in every hundred the regex flags as identifiers are not identifier queries at all. Those three are drawn from the strata the regex was not built for, where BM25 is the weaker system by a wide margin, so they arrive at a ranker scoring 0.68 or 0.41 rather than one scoring 0.86 or 0.77. Blend the two groups and the identifier route's realised recall falls from 0.94 to a little over ninety-three percent. One point is not the problem. The problem is that the point is invisible: no log line distinguishes a query the router sent to the wrong system from a query the right system happened to fail, so the loss never appears in the place anyone would look for it.

The latency comparison also has a step in it that gets skipped. Running both retrievers costs 8 ms and 14 ms, which is twenty-two milliseconds one after the other and 14 ms when they are issued concurrently, because concurrent work costs the larger of the two rather than their sum. The embedding call is already inside the 14 ms, so the dense side sets the floor whether or not BM25 runs beside it, and fusion is close to free in wall-clock terms. What it is not free in is operations: two indexes have to be built from the same chunks at the same time, and an index pair that drifts produces a fused list assembled from two different corpora.

What settles it is re-running the stratified table for the fused system rather than reasoning about it. Reciprocal rank fusion over these two runs scored 0.86 overall, with identifier recall at 0.91 — three points below BM25 alone, because dense results dilute a list BM25 had right — and paraphrase at 0.87. Trading three points on 54 queries for eighteen points on 96 is a decision the table makes arithmetic instead of rhetorical, and the negation row stayed at 0.44, exactly as the disjointness analysis predicted.

Converting those percentages into queries is what makes the trade obvious rather than arguable. Three points on fifty-four queries is under two queries. Eighteen points on ninety-six is a little over eighteen. Losing two users to gain eighteen is not a judgement call, and the only reason it ever reads as one is that percentages of different denominators look comparable and are not. The overall figure closes the same way. Fusion moved the aggregate from 0.74 to 0.86, which is twelve points of the roughly sixteen the oracle showed were available, so fusion collected about three quarters of the headroom the failure-set overlap said existed. That last comparison is the one to keep, because it turns a good result into a bounded one: the remaining four points sit inside the thirty-one queries both systems missed, and no scheme for merging these two lists will ever reach them.

Where stratification stops helping#

Strata are hypotheses about mechanisms, and they are only as good as the mechanism they name. Slicing by an attribute with no causal relationship to how the rankers score — the day of the week, the requesting team, the length of the document — will still produce a table with variation in it, and with six slices and two systems there is a good chance one of the cells shows a gap that will not reproduce. The discipline is to write the strata down before running anything, to justify each one by a mechanism you can state, and to treat any post-hoc slice as a hypothesis needing a fresh sample rather than a finding.

The table also says nothing about the size of each stratum's contribution to user experience unless the sample matches production traffic. If identifier queries are 18% of the labelled set because that is what the sampling produced, but 45% of real traffic because power users dominate volume, the aggregate row is the wrong weighting and every decision above shifts. Sampling the eval set proportionally from real logs, and re-checking the proportions quarterly, is what makes the "All" row mean anything.

The size of that distortion is computable from the rows already in the table, which is the useful part. BM25 leads the dense retriever by 0.33 on the identifier stratum. Move that stratum's weight from 18% to 45% and its contribution to the aggregate gap grows by the difference in weights times the gap itself, which is a shift of roughly nine points in BM25's favour, from one row alone and with no ranker changed. Nine points is larger than every gap in the "All" row of any comparison in this chapter. A mis-weighted stratum is therefore not a rounding concern to note in a footnote; it is enough on its own to reverse which system the table appears to recommend, which is why the sampling proportions belong next to the score rather than in the appendix.

The rows, then, are the report. An average over a labelled set is a single bit per query smeared into one number, and it is the wrong instrument for every decision anyone actually takes: whether to keep two indexes, whether to route, which stratum deserves engineering next, and which population a cost saving would fail. Six rows and a failure-set overlap cost an afternoon and answer all four.

And a table of retrieval numbers, however finely sliced, measures only whether the evidence reached the context window. Push recall@10 from 0.74 to 0.92 and the complaints do not stop, because a system can put the right passage in front of the model and still be wrong, in ways this measurement cannot distinguish from a retrieval miss.