Filtered recall has to be measured apart from unfiltered recall

Once a retrieval service routes queries to different plans depending on how many documents their predicate matches, its quality stops being one number. Some queries are answered exactly, some approximately at a selectivity the index handles well, and some approximately near the cliff — and an average over that mixture moves when the traffic mix moves rather than when quality moves. This chapter is about the measurement that survives a router: what ground truth means for a filtered query, how to cut the result so a failure can appear in it, and which queries to evaluate in the first place.

Two definitions carry the rest. An eval harness is the offline job that replays a fixed set of queries against retrieval and scores each answer against a ground-truth list, the set of documents an exhaustive, exact computation says should have been returned. Recall@k, from earlier in this topic, is the fraction of that ground-truth list the search actually found. All three are ordinary and all three go subtly wrong the moment a predicate is involved.

Why an average cannot report a slice failure#

Start with the arithmetic that makes a blended tile useless, because it settles the question before any measurement is taken.

A mean over a mixture is each group's value weighted by its share of the population. So if one group holds a fraction f of the traffic and its recall falls from healthy to broken, the blended mean falls by f multiplied by the size of that drop, and by nothing else. The other group's contribution has not changed, so it cannot contribute any movement. That product is the whole budget the failure has to work with.

Put a small tenant in it. Suppose a customer issues four queries in every thousand — a fraction of 0.4% — and their recall collapses from around 0.95 to around 0.20. The drop is three quarters of a point of recall. The blended figure therefore falls by 0.4% of three quarters, which is three thousandths. Whatever else is true, three thousandths is the largest movement this failure is capable of producing, and it is capable of producing it only if the collapse is total.

Now compare that against the number's own noise. A tile computed over a resampled query set fluctuates by more than three thousandths from one regeneration to the next, so the entire collapse of a paying customer's retrieval quality is indistinguishable from the tile being recomputed. The tile is not merely insensitive, it is structurally incapable of showing this. That is the shape to recognise whenever a monitoring signal is proposed: the defence is bounded by the traffic share of the thing it is defending against, so it stays effectively constant across exactly the axis it needs to vary along. A monitoring signal that cannot represent the failure it is supposed to catch is not a weak signal; it is a decoration.

The consequence is worth stating plainly before the rest of the chapter earns it. The fix is not a better threshold on the blended number, because no threshold on a quantity with a three-thousandth range can be both sensitive and quiet. The fix is to stop blending.

Filtered ground truth is not filtered from unfiltered ground truth#

Before the slicing question comes a definitional one that a surprising number of eval harnesses get wrong, and getting it wrong makes the numbers look better than reality in exactly the cases that matter.

Ground truth for a filtered query is the exact top-k within the eligible set, which is the exact_top_k computation from chapter 2 run against the documents the predicate matches. The tempting shortcut is to compute one unfiltered ground-truth list per query, cache it, and then apply the predicate to that cached list. That is post-filtering the ground truth, and it inherits post-filtering's arithmetic exactly — the same k × s product from the first chapter, with k now the length of the cached list. At 0.1% selectivity a cached unfiltered top-100 yields 100 multiplied by one thousandth, which is a tenth of a document: about zero eligible documents. So the "ground truth" for that query is an empty list.

Follow what an empty truth set does to the score. Recall is an intersection divided by the size of the truth set, and dividing by zero either raises or, in the version people actually ship, causes the query to be skipped. The harness quietly drops the hardest queries and reports the average over the easy ones, which is the second time in two pages that a measurement has been made to agree with itself by removing the disagreement.

The denominator needs the same care for a different reason. Recall@10 for a query whose predicate matches only 4 documents cannot exceed 0.4 if you divide by 10, because there are only four documents in existence that could have been found. That quantity is capped by the predicate rather than by the search, so it cannot be used to compare searches. The correct denominator is min(k, n_eligible):

def recall_at_k(returned_ids, truth_ids, k, n_eligible):
    denom = min(k, n_eligible)
    if denom == 0:
        return None                     # no eligible docs: not a failure
    return len(set(returned_ids) & set(truth_ids)) / denom

def evaluate(queries, k=10):
    rows = []
    for q in queries:
        n_elig = cardinality.exact(q.predicate)       # not an estimate
        truth = exact_top_k(q.vec, q.predicate, k)    # scan eligible set
        hits = retrieval.search(q.vec, q.predicate, k=k)
        rows.append({
            "query_id":    q.id,
            "filtered":    q.predicate is not None,
            "tenant":      q.predicate.get("tenant_id") if q.predicate else None,
            "attrs":       tuple(sorted(q.predicate)) if q.predicate else (),
            "selectivity": n_elig / CORPUS_SIZE,
            "plan":        hits.plan,                 # scan | ann | postfilter
            "n_returned":  len(hits.ids),
            "recall":      recall_at_k(hits.ids, truth, k, n_elig),
        })
    return rows

Note that the evaluation uses an exact cardinality rather than the router's estimate. The router's estimate is part of the system under test — if it is wrong, as the correlated-predicate case in the previous chapter shows it can be by more than an order of magnitude, that shows up as a bad plan choice and therefore as low recall, which is what you want the harness to reveal rather than to launder.

Cutting the number so failures can surface#

Once each query carries its attributes, its selectivity and its plan, the reporting job is to stop averaging across things that behave differently. The minimum useful cut separates unfiltered from filtered, and then breaks filtered down along the dimensions on which the system's behaviour actually varies: selectivity band, filter shape, and the individual values that carry meaningful traffic — which in a multi-tenant product means per tenant.

Add the query column downwards before reading anything else: 1,180 and 402 and 251 and 118 and 49 come to exactly the 2,000 on the blended row, with nothing left over. That closure says the cut is a partition — every query in the eval set lands in exactly one slice, none is counted twice, none is missing — which is what licenses reading the rows as independent populations rather than as overlapping views. A slicing where the parts overran or undershot the whole would mean some queries were being double-counted or dropped, and every comparison between rows would be suspect.

The direction of recoverability runs one way only, and the latency column is the easiest place to see it. A median is not an average, so the blended p50 cannot be rebuilt from the five slice medians no matter what weights you use — the position of the middle query in the combined set is simply not a function of where the middle sat in each part. The slices determine the whole; the whole does not determine the slices. That asymmetry is the argument for publishing the slices.

Every row above carries information the blended row destroys. Read the recall column as gaps and the destruction is quantified: from unfiltered down through the loose filters, recall gives up a hundredth and then six hundredths, and then between the 1-10% band and the 0.1-1% band it gives up twenty-seven hundredths in a single step. The bottom two bands are 8% of the eval set — 118 and 49 queries out of two thousand — and are where the product is broken.

It is worth knowing how much worse that would have to get before the blended tile noticed. The healthy slices average around 0.96 between them and the two broken bands average a little over one half, so the blend falls by the broken share multiplied by the gap of roughly four tenths. Getting the blend down to 0.85 therefore needs a broken share of about eleven hundredths divided by four tenths, which is a bit over a quarter. A dashboard showing only the last row would have to wait for those bands to reach a quarter of all traffic before the number dropped below 0.85 — by which time a quarter of the product does not work.

The same cut by tenant is what catches a single collapsed customer. Reporting 400 individual tenant means is not usable as a dashboard, so the aggregate worth publishing is an order statistic rather than a mean, a figure defined by rank in the sorted list rather than by averaging — the 5th percentile of per-tenant recall, or more bluntly the count of tenants below an agreed floor. The reason that works is precisely the reason the mean does not: an order statistic is not weighted by traffic, so one tenant crossing the floor moves it by a full unit whatever their query volume. A tile reading tenants below 0.80 recall: 3 moves the moment one tenant breaks and cannot be dragged back up by the other 399. Pair it with the identity of the worst three and the on-call engineer has a starting point instead of a mystery.

Slicing also changes what a regression gate can say, a gate here being the automated check that refuses a release when the measured number falls below an agreed threshold. Work out its noise floor rather than picking a threshold by feel. The standard error of a mean is the per-query standard deviation divided by the square root of the number of queries. With 2,000 queries and a per-query recall standard deviation around 0.25, the square root of two thousand is about forty-five, so the standard error is 0.0056 — any threshold tighter than about a point of recall fires on noise.

Per-slice gates have fewer queries each and therefore wider error bars, and the same division says how much wider: 118 queries has a square root of about eleven, so that slice's standard error is a little over two hundredths. But it is testing a much larger effect. A plan regression in the 0.1-1% band moves that slice's mean by 0.2 or more, which is roughly nine standard errors across 118 queries and therefore unmistakable. The gate that works in practice is a set of per-slice floors, each set a few points below the current value, rather than one tight bound on the aggregate.

Choosing which queries to evaluate#

Sampling the eval set the way traffic is distributed reproduces the blindness you are trying to remove. If 96% of queries carry no filter, a proportional sample of 2,000 queries contains 80 filtered ones spread across every attribute combination in the product, which is not enough to say anything about any of them.

Stratify instead. Enumerate the distinct filter shapes in a week of logs — the sets of attributes used together, ignoring values — and sample a fixed number of queries per shape, then within the shapes that carry a value dimension like tenant or document class, sample across the value range rather than by popularity. A workable allocation for a 2,000-query set is a few hundred unfiltered queries as a control, then roughly equal quotas across the selectivity bands, with the lowest bands deliberately over-represented because they are where variance is highest and where a small number of queries has to support a conclusion.

The cost is the exhaustive scans. At 10.2M vectors of dimension 768, an exact scan for one query costs 4-8 seconds single-threaded, and the eval set needs one per query. Two thousand of those is between two and four and a half hours of scan time, and the temptation is to divide that by the core count and promise twenty-odd minutes. It does not divide. The previous chapter established that this scan is bound by memory bandwidth rather than by arithmetic, and cores added to a bandwidth-bound loop contend for the same bus, so eight of them buy a good deal less than eight times the throughput. Roughly three hours on eight cores is the honest expectation, which makes this a nightly batch job rather than a CI gate.

Two things make it cheaper. Ground truth for a query is stable until the documents change, so it can be cached keyed by (query_id, predicate, corpus_version) and recomputed only for the shard that changed. And the high-selectivity queries, which are the ones you most want in the set, are the cheapest to compute exactly, because scanning 5,000 eligible vectors takes under a millisecond. The expensive ground truth is for the loose filters, which are also the ones where the system is least likely to be broken.

Recall, relevance and the queries nobody ran#

Recall measures whether retrieval found the documents that the exact computation says are nearest. It says nothing about whether those documents answer the question. An embedding model that maps the query "what is our termination notice period" closer to procurement boilerplate than to the termination clause will produce recall@10 of 1.0 and an unusable answer, because the retrieval system faithfully returned the ten nearest wrong things. Recall is a measure of the index and the plan; relevance is a measure of the embedding and the chunking, and improving one has no mechanical effect on the other.

Offline recall also does not transfer to production without an assumption. The eval set contains the filter shapes that appeared in last week's logs, and the space of shapes is combinatorial — eleven filterable attributes with a handful of values each generate more combinations than you will ever evaluate. A user who invents doc_type IN ('sow','amendment') AND jurisdiction = 'AT' AND signed_after = 2024-06-01 is running a plan against a selectivity nobody measured. Stratified sampling by shape reduces this exposure and does not eliminate it, which is the honest limitation of the whole approach: an offline harness proves things about the queries it contains.

The per-tenant cut carries its own caveat, which is that a tenant with a small document set can look terrible for a reason that is not a defect. A customer with 300 documents has a per-tenant selectivity of 0.003% against a 10.2M index, so every one of their queries lands in the bottom band by construction. If the router is sending them to an exact scan their recall is 1.0 while their neighbours in the same selectivity band sit at 0.34, and if it is not, they are on the wrong side of a threshold. Reporting recall by tenant without also reporting plan makes those two situations indistinguishable, and someone will spend a week tuning ef_search for a tenant whose queries never touch the graph. Carrying the plan column into the slice keys costs nothing and removes the ambiguity.

Every slice above is measured offline, against a ground-truth set that took hours to build, on queries chosen last week. That is the right instrument for deciding whether a change made retrieval better or worse, and it is the wrong instrument for knowing whether retrieval is working right now. Production has neither the ground truth nor the time.

What production does have is the harness's other column. The code above records n_returned alongside recall, and those two fail differently. A query that returns ten results with recall 0.6 looks fine to a user and bad in the eval; a query that returns two results with recall 1.0 looks broken to a user and perfect in the eval, since both of its eligible documents were found. Users do not report recall. They report counts, and a result list shorter than the page size is the signal they react to — which is detectable on live traffic, without a ground-truth set, and is the subject of the next chapter.