Recall and nDCG are blind to crowd-out by construction

An evaluation metric answers one question and is silent on every other, and the useful skill is knowing which question a given metric asked. Two of them are what almost every retrieval system is watched by. Recall@10 is the fraction of the documents a human assessor marked relevant to a query that turn up in the top ten. nDCG@10 — normalised discounted cumulative gain — additionally cares where in the ten they land, rewarding a relevant document more the higher it sits and reporting the result as a fraction of the best score that ranking could possibly have achieved.

Neither question mentions the other results in the window. That omission is not an oversight and it is not something a threshold or a parameter can restore, because there is no place in either formula where one result could be compared against another. What it costs, on a corpus that contains copies, is the subject here. The replacement measurement needs exactly one thing the two familiar metrics do not have: a duplicate cluster identifier, naming the group of chunks that chapters 2 through 4 established as carrying the same content.

nDCG 1.00 on a window holding one document four times#

Work the arithmetic on a single query and the reason becomes mechanical. Suppose the judgement file for the query invoice dispute window — the file holding one grade per query-document pair, written by an assessor who read the document and rated how well it answers the query — marks eight documents relevant, graded 3, 3, 3, 3, 2, 2, 2 and 1, and the retriever returns this top-10:

Three columns of that table are computed rather than given, and each of them is a design decision worth naming before the total is read.

The Gain column raises two to the power of the grade and subtracts one, so a grade of 3 becomes 7, a grade of 2 becomes 3, and a grade of 1 becomes 1. Raising to a power is a choice about how much better a strongly relevant document is than a moderately relevant one: under this rule one document graded 3 is worth more than two documents graded 2, which would not be true if the grade were used directly. Subtracting one is what makes a grade of 0 contribute nothing rather than one.

The Discount column takes one divided by the base-two logarithm of the rank plus one, which gives exactly 1.000 at rank 1, since the logarithm of two is one, and exactly 0.500 at rank 3, since the logarithm of four is two. Rank 7 gives one over the logarithm of eight, which is a third. The logarithm is a choice too, and a mild one: it makes rank 1 worth a little over three times rank 10 rather than ten times, which encodes a belief that users read some way down a list rather than only the first item.

The Contribution column is the product of the two, and one row is enough to see it. At rank 7 the gain is 3 and the discount is a third, so the contribution is exactly 1.000.

DCG@10, the discounted cumulative gain, is the sum of that last column: 21.475. The eight non-zero contributions add to that figure with nothing left over, which is worth checking by hand once, because the total is about to be divided by itself and the identity is easier to trust when the column has been read downwards.

The ideal ranking puts the same eight judged documents in grade order, which is precisely the order they already appear in, so the ideal DCG — IDCG@10, the denominator that turns the raw sum into a fraction — is also 21.475 and nDCG@10 is exactly 1.000. Recall@10 is 8 of 8, or 1.000. A perfect score on both metrics.

That 1.000 is an identity rather than a rounding, and the reason is structural. The grades in the judgement file are 3, 3, 3, 3, 2, 2, 2 and 1, in that order, and the ranking presents them in exactly that descending order. The ideal permutation of those eight documents is therefore the permutation the retriever already produced, so the denominator is computed over the same numbers as the numerator, term for term. No arrangement of these documents could have scored higher, and the metric says so correctly.

The first four documents are the same paragraph. One of them is the Zendesk article, one is a Confluence page pasted from it, one is a PDF export of that page, and one is a MediaWiki copy left over from a migration. Eight documents were judged relevant and four of them are one thing, so the eight carry five distinct pieces of evidence: eight minus the three redundant members of that one group. The window carries five distinct pieces of evidence in ten slots, and both metrics report flawless performance, because each of those four documents genuinely is relevant to the query and was judged so by an assessor who saw it in a pool and marked it as answering the question. The assessor was right. The metric did what it was designed to do.

That is the answer to why the dashboard did not see it. Recall counts how many relevant documents were found, and a copy of a relevant document is a relevant document. nDCG compares the achieved gain ordering against the best possible gain ordering over the same judgement file, and since duplicates carry relevance grades, they raise the ideal as much as they raise the actual. Neither metric contains any term that could distinguish four witnesses from one witness photocopied four times, because relevance in the Cranfield tradition these metrics come from — the test-collection method, named after the aeronautics college where it was worked out in the 1960s, in which a fixed query set and a fixed judgement file let two retrieval systems be compared offline — is a property of a single document assessed against a single query, evaluated independently of everything else in the ranking. Independence of documents is an assumption baked into the formula, and duplicates are exactly the case where it is false.

The temptation these two numbers create is worth naming as a defence rather than as a measurement error. A team watching recall and nDCG believes it has an alarm that would sound if result quality fell. That alarm produces the same reading whether the ten slots hold ten independent sources or one source repeated four times, because the only inputs it has are per-document grades and ranks, and both of those are unchanged by duplication. A check whose output does not vary across the two states it is supposed to separate cannot be a weak check or a lagging one; it carries no information about that distinction at all.

The hypothesis to state, when someone hands you healthy metrics and unhappy users, is therefore not that the ranker regressed. It is that the ranker is working and the corpus contains redundant documents that the ranker is correctly retrieving, and that the evaluation is scoring a property nobody cares about.

Counting distinct sources instead#

The measurement that sees the problem needs a notion of identity across documents, which chapters 2 through 4 produced as cluster ids. Once every chunk carries the id of the duplicate cluster it belongs to, a top-k can be scored on how many distinct clusters it contains:

def unique_clusters_at_k(results, k: int) -> int:
    return len({r.cluster_id or f"single:{r.chunk_id}" for r in results[:k]})

def cluster_recall_at_k(results, judged_clusters: set[str], k: int) -> float:
    got = {r.cluster_id or f"single:{r.chunk_id}" for r in results[:k]}
    return len(got & judged_clusters) / len(judged_clusters)

Both functions turn a list into a set, and the set is doing the entire job. Two chunks from one cluster map to the same identifier, so they collapse to one member; a chunk with no cluster falls back to a key built from its own id, which is unique and therefore never collapses with anything. Counting the set rather than the list is what makes copies free and independent documents expensive, which is the exact inversion of what recall does.

The first counts evidence delivered; the second counts distinct evidence found against a judgement file that has itself been collapsed to clusters, which is the version of recall that stops rewarding copies. Neither requires new labelling, and the first requires no labelling at all, which matters more than it sounds: unique-clusters@k can be computed over live production query logs, on every query the system has ever served, with no relevance judgements in sight. On the payments assistant it took an afternoon to backfill across three months of traffic, and the result was a distribution rather than a single number — a median of 6 distinct clusters at k=10, with 14% of queries returning 4 or fewer.

Running the same query set against a deduplicated index gives the comparison that settles the argument:

The first two rows move by the same amount in opposite directions, and that pairing is what makes them maximally misleading rather than merely uninformative. They do not fall, so nothing looks broken. They do not rise, so nothing looks fixed. A number that stays put is normally evidence that nothing happened, and here it is evidence about the metric instead. The distinct-evidence measures move by amounts no one would attribute to noise, and the end-to-end answer accuracy moves with them. That last row is the one that justifies the work, and it is worth building the 200-question evaluation for that reason alone, since a retrieval metric of any kind is a proxy and the assistant's correctness is the thing being bought.

Cluster-recall@10 rising to 0.793 rather than to 1.0 is also informative. The remaining gap is ordinary retrieval difficulty — evidence the ranker did not surface for reasons that have nothing to do with duplication — and separating the two is most of the value of having both numbers.

Raising k buys the same ratio at twice the price#

The reflexive fix is to enlarge the window. If ten slots hold six distinct clusters, ask for twenty and there will be room. Measured on the same corpus, the ratio barely moves:

Two columns of that table describe the same data and they disagree about whether anything is improving. The Distinct fraction column falls steadily. The Marginal column, which is the increase in distinct clusters divided by the increase in slots, is flat and if anything rises slightly. Read as levels the table says the window is getting worse; read as steps it says every additional slot delivers the same thing every other additional slot delivered.

The steps are the true reading and the levels are an artefact. The first slot is always distinct, because there is nothing above it for it to be a copy of, so the fraction starts with a free point that the marginal rate does not enjoy. Spread that one free point across a larger k and its contribution to the fraction shrinks, which drags the fraction downwards towards the marginal rate without the marginal rate having changed at all. What the two columns together say is that the corpus delivers a little over half a new piece of evidence per slot and has done so throughout, and that the fraction is converging down onto that constant rather than deteriorating.

Each additional slot delivers a little over half a new piece of evidence, and that rate is essentially flat, because the duplicates are distributed through the ranking rather than concentrated at the top. Going from k=10 to k=20 does add about six distinct clusters, so the tactic is not useless, and it is exactly this partial success that keeps the idea alive.

What it costs is a doubled prompt, and the cost side is exactly linear where the benefit side is not. Prompt tokens per slot are about the same at every k in the table, so doubling k from 10 to 20 doubles the prompt from 5,100 to 10,200 tokens. On a mid-sized model at roughly 3 USD per million input tokens, 5,100 tokens is 0.0153 USD, and the doubled prompt is 0.0306 USD; multiplying the difference by 40,000 queries a day gives an extra 610 USD daily to deliver the same evidence-per-token as before. Longer contexts also make position effects worse, and the material stranded in the middle of a 25,000-token prompt is not reliably read.

Enlarging k treats the symptom by buying more of a window that is being wasted at a constant rate. Removing the redundancy changes the rate, and then a smaller k does more work than the larger one did.

Diversity re-ranking is a different instrument#

Maximal marginal relevance and its relatives are the standard tool for making a result list less repetitive, and they are not this. MMR selects each next result by trading its query relevance against its maximum similarity to what has already been chosen, which suppresses anything semantically close to an earlier pick regardless of whether it is a copy. On a query like invoice dispute window, the second-best genuinely independent source — the billing handbook stating the sixty-day rule — is semantically close to the first result precisely because it is about the same thing, and an MMR pass tuned aggressively enough to suppress a 0.98 duplicate will also penalise a 0.86 independent corroborating source.

That is the second stumble in this chapter and it is the mirror image of the first. Recall and nDCG cannot see identity at all; MMR can only see topicality and treats it as identity. Both readings are constant across the distinction that matters, and reaching for the second as a remedy for the first swaps one blindness for another. The distinction is between identity and topicality. Deduplication removes copies, a decision made once, offline, against calibrated evidence, and it leaves every genuinely distinct document in place no matter how close it sits to another. Diversity re-ranking reshapes a result list toward covering more of the query's possible intents, which is a useful and separate goal that becomes tractable once the duplicates are gone — on a corpus full of copies, an MMR pass spends its diversity budget rejecting the same document four times and never reaches the intents.

The evaluation literature makes the same split. α-nDCG and subtopic recall measure whether a ranking covers distinct information needs, which is the diversity question and requires per-query subtopic labels that most teams will never produce. Unique-clusters@k measures whether a ranking covers distinct documents, which is the duplication question and requires only the cluster ids already sitting in the index.

One honest limitation of the cluster-based measures: they inherit every error from the detector. If the threshold merged two release-notes pages that were not duplicates, unique-clusters@10 will report a window as less diverse than it truly is, and if the candidate search missed a pair, it will report more diversity than there is. The measurement and the thing measured share a failure mode, which is a reason to keep the calibration table from chapter 4 next to the diversity numbers and to re-run both when the model changes.

At this point the pipeline knows which chunks are copies of which, with stated error rates, and there is a measurement that responds when the situation improves. What has not been decided is what actually happens to a duplicate — whether the extra copies are deleted from the index, folded into one canonical record, or kept with a marker and handled at query time. Those three options are usually presented as a matter of preference, and they are not: each one destroys a different thing, and the losses are not recoverable after the fact. That is the next chapter.