A 12% corpus cut that improves your metrics deserves suspicion first

Offline retrieval evaluation compares a ranking against a judgement file, which holds one relevance grade per query-document pair for the queries somebody chose to judge. Every document that was never judged is scored as non-relevant, because there is no third option available and treating unjudged as relevant would be worse. That convention is what makes offline evaluation possible at all, and it carries a consequence that is obvious once stated and almost never checked.

An unjudged document can only ever hurt a score. It cannot be a hit, so it contributes nothing when it is retrieved, and it can outrank a judged positive and push it down. Removing unjudged documents from an index therefore raises offline retrieval metrics unconditionally, regardless of whether those documents were duplicates, useful, or the only place in the company where a policy was written down. Any change that shrinks a corpus and is then scored this way is being measured by an instrument that rewards shrinkage on its own. Distinguishing the two possible explanations takes about a day of work that is much easier to do before the change is celebrated than after.

Deleting random documents also improves your metrics#

On the payments evaluation, 1,200 queries had judgements over 14,800 chunks — 0.35% of a 4.2M-chunk index. Everything else is unjudged, and the standard scoring convention treats unjudged as non-relevant.

Take one query where the judged positive sits at rank 7 with gain 1. Its contribution to DCG is 1/log2(8) = 0.333. Remove two unjudged chunks from ranks 3 and 5 and the positive moves to rank 5, contributing 1/log2(6) = 0.387. The judged positive did not become easier to find, the retriever did not improve, and nDCG went up.

Walk the four steps under that arithmetic, because they are what generalise beyond the one query. A document's contribution to the score depends on its rank rather than on anything about the document. Rank is a count of what sits above it. Removing anything above it therefore moves it up, whatever the removed thing was. And the discount curve is steepest near the top, so the same one-place promotion is worth more the higher it happens — which means the effect is largest exactly where the metric pays most attention. None of those four steps mentions duplication, which is why none of them can distinguish a deduplication run from vandalism.

The control that makes this concrete costs one afternoon: delete 12% of the corpus at random, protecting the judged documents, and score the same query set.

Randomly destroying half a million chunks bought +0.026 recall and +0.026 nDCG. Read the deduplication row against the baseline row and its own gains are four hundredths of recall and a little over three hundredths of nDCG. Divide one pair by the other and vandalism reproduces about two thirds of the recall gain and about four fifths of the nDCG gain, so the metric that looks most impressive is the one where the control does best. The celebrated numbers are barely distinguishable from vandalism, and the two rows that actually differ from the baseline in a way anyone would want are the last two columns, neither of which is on the retrieval dashboard.

The bottom row is the configuration from chapters 4, 6 and 7. It removes nothing, so it gets no free lift; its recall and nDCG sit inside noise of the baseline, and its answer accuracy is the best of the four. A reviewer who judges runs by recall@10 would rank it last. That inversion is the second stumble in this chapter and it is worth stating in its own right: a metric that rewards deletion will rank the configuration that deletes nothing beneath the configuration that deletes half a million chunks at random, and it will do so with numbers that are entirely correct.

Running the random control once, at the same removal volume as the real run, converts the whole question from an argument into a comparison. If the deduplication run does not clearly beat random deletion on the metrics being quoted, those metrics are measuring the deletion rather than the deduplication.

Reading the removed set instead of the aggregate#

Chapter 4's calibration table predicts a false-merge rate, and the prediction is about the labelled sample, drawn from a candidate file that was built weeks earlier under a particular model and chunker. What actually shipped may differ, so the rate is worth re-measuring on the run that executed.

Sample the executed merges the same way the calibration sampled candidates — stratified across the score range, not from the top — and apply the same written decision rule. On this run, 300 sampled clusters yielded 21 that grouped documents a reader would want kept apart: 7.0% against a predicted 3.8%. The discrepancy came from the operating cutoff being 0.95 rather than the 0.96 the calibration recommended, a change made during the rollout to increase the corpus reduction. Doubling the false-merge rate to gain three percentage points of index shrinkage is a trade someone might defend, but it was not made deliberately and nobody had the number until it was measured.

The judged set deserves its own check, because a judged relevant document removed by deduplication corrupts both the users' experience and the evaluation that is supposed to detect the problem:

removed = set(run.removed_chunk_ids)                 # 503,000
judged  = {c for c, rel in qrels.items() if rel > 0}  # 14,800

casualties = removed & judged                         # 34
orphans = [
    c for c in casualties
    if run.representative_of[run.cluster_of[c]] not in judged
]                                                     # 3

The two lines of that snippet ask different questions and the second is the one that matters. The intersection counts judged documents that were removed, which sounds alarming and mostly is not: a judged document folded into a cluster whose surviving representative is also judged relevant is still retrievable, and the merge did what it was meant to do. The filter underneath narrows that set to the cases where the survivor was not judged relevant at all, which means the evidence has left the reachable index without a judged replacement.

Thirty-four judged relevant chunks were removed and 31 of them fell into the harmless category. Three did not, and reading those three clusters found the failure mode from chapter 3 in the wild — templated siblings, where the judged page had been merged into a sibling that answered a different question. The three that break the pattern are worth more than the 31 that follow it.

Three out of 14,800 is a small contamination of the evaluation and a large signal about what the threshold is doing across the other 503,000 removals. The step from the first reading to the second is one division. The judged set is 0.35% of the corpus, so it is a sample of roughly one chunk in three hundred; three casualties inside a sample that size scales to something on the order of 900 across the whole index. That is why three is a large number rather than a small one, and the multiplication is the only reason anybody would think so.

Per-query deltas the average absorbed#

An aggregate of +0.03 nDCG across 1,200 queries can be a uniform small gain or a large gain on 400 queries paid for with a large loss on 190. The two situations call for opposite decisions and look identical on a dashboard.

deltas = {q: after[q] - before[q] for q in queries}     # nDCG@10 per query
improved = [q for q, d in deltas.items() if d > 0.005]
regressed = [q for q, d in deltas.items() if d < -0.005]
severe    = [q for q, d in deltas.items() if d < -0.10]

print(len(improved), len(regressed), len(severe))       # 401 187 46
print(sum(deltas[q] for q in improved) / len(queries))  # +0.061
print(sum(deltas[q] for q in regressed) / len(queries)) # -0.031

Four hundred and one queries improved, 187 regressed, and 46 of those regressed by more than 0.10 nDCG. The last two printed figures are worth adding together, because their sum is the number the whole project is being judged on. Both are divided by the same denominator — every query, not just the ones in each group — so they are contributions to the overall mean rather than group averages, and contributions add. The improvements contribute +0.061 and the regressions contribute −0.031, which leaves the +0.03 that appears on the dashboard. The headline is those two quantities added, and it displays neither of them.

Reading the 46 is a two-hour job and it found a concentrated cause: 29 of them traced to false merges in 11 clusters, all of them templated document families — release notes, quarterly reports, and per-region variants of a shipping policy that differ only in the country name and the delivery window. Eleven clusters, correctly excluded, would have removed most of the regression while costing 0.002 of the aggregate gain. Those two figures are the argument for doing the work: the damage the eleven clusters caused runs into hundredths of nDCG on dozens of queries, and undoing it costs thousandths.

Splitting the deltas by query frequency finds the second pattern. Head queries — the top 5% by volume — improved by +0.048 on average, while the bottom 50% by volume improved by +0.011 and contained 78% of the severe regressions. That is the expected shape when duplicates are concentrated in heavily-copied popular documentation and false merges are concentrated in the long tail of specific pages, and it matters because the tail is where a knowledge assistant earns its keep. Anyone can find the refund policy; the assistant exists for the question about the 2019 regional variant.

Where the removals came from#

The last check needs no labels and takes ten minutes. Compare each connector's share of the index against its share of the removals.

Both percentage columns sum to exactly a hundred with nothing left over, and that closure is what makes the Ratio column mean something. Because each column is a complete division of one whole, no connector can take a larger share of the removals without some other connector taking a smaller one. The ratio is therefore a redistribution rather than a set of independent measurements: the two rows above unity are paid for by every row below them, and the five low rows are low precisely because those two are high. That is what licenses reading a single high ratio as a claim about the whole corpus, and it is why the same figure would mean different things on different rows.

Every row here has an explanation that matches what is known about the corpus, which is the outcome you want. A ratio of 7.6 for a decommissioned wiki that was mirrored wholesale into Salesforce is the pipeline working. The same 7.6 on the Notion connector, where nothing was mirrored from anywhere, would mean something else entirely — most likely that Notion's exporter emits a large shared template that chapter 3's residual test should have caught, and that the run had quietly deleted a source rather than deduplicating one. A single connector supplying the large majority of removals is worth understanding before shipping, whatever the explanation turns out to be.

What offline evaluation cannot reach#

These checks establish that the removals are what you think they are and that the metric movement is not an artefact of a smaller judgement pool. They do not establish that users are better off, and offline evaluation cannot, because the entire mechanism at issue is that the judgement file does not cover the corpus. An honest verdict needs either an end-to-end answer evaluation of the kind used throughout these chapters, where a human or a strong grader scores the assistant's response rather than the ranking, or an online comparison on live traffic with a behavioural outcome — escalation rate to a human agent, follow-up question rate, thumbs-down rate. The payments team ran the second and saw escalations fall from 18.2% to 15.4% over three weeks at 50% traffic, which is the only number in this chapter that describes a benefit to anyone outside the retrieval team.

None of this is an argument against deduplication. The last row of the first table is a substantial win — 0.71 to 0.86 answer accuracy — bought by a pipeline that removed nothing at all. The argument is against the specific inference from a smaller index and larger retrieval metrics to a better system, which does not hold, and which is available to be tested cheaply.

Running this on your own corpus#

The order that works is the order these chapters took, and the first three steps produce a decision about whether to continue.

Start with the exact-hash pass, because it is linear, needs no labels, and tells you within an hour what fraction of the corpus is byte-identical duplication. If that number is 1%, the embedding pipeline may not be worth building; if it is 11%, the near-duplicate layer above it is usually larger still. Then run the ANN self-join at a loose floor, apply the same-document, template and short-chunk exclusions before looking at a single score, and measure the self-join's own pair recall on a subset small enough for exhaustive comparison, since that number bounds everything after it.

Label 700 pairs stratified across the bands, publish the table, and pick the cutoff with the residual-similarity condition attached. Write cluster ids and a run id into the index rather than deleting anything, put the cluster id in the vector payload while you are touching the schema, and build the collapse with an over-fetch factor read off the logs. Instrument unique clusters per top-k before any of it ships, because it is the only metric in the stack that responds to the thing being fixed, and it costs nothing to compute on traffic you are already serving.

For getting started on a corpus this week, SemHash will find near-duplicates over a dataset with a few lines of setup and is a reasonable way to see the size of the problem before committing to a pipeline; NVIDIA's Semantic Deduplication documentation is the most complete production reference for clustering, thresholds and representative selection at scale. Both assume the decisions in chapters 4, 6 and 7 have already been made, which is the part that belongs to your corpus rather than to a library.