Five accounts of one cause spend the budget on one viewpoint

Every strategy so far in this topic has treated a retrieved passage as too small a piece of the truth, and has responded by making the returned unit bigger. There is an opposite failure, and it is at least as common: the retriever returns five units, every one of them relevant, every one of them sufficient on its own, and all five say the same thing. Nothing is missing from any individual passage. What is missing is everything a corpus says that is not the most-documented view of the question.

That failure is a property of how a ranked list is built rather than of any passage in it. Cosine similarity — the standard measure of how close two embeddings point in the same direction, running from 0 for unrelated text to 1 for identical text — is computed between a query and one passage at a time. The score of a candidate does not depend on what has already been selected, so nothing in the ranking has any term for redundancy. This chapter is about adding one: what it buys, what it costs, how to measure both, and the shape of corpus for which it is the first thing to reach for rather than the last.

Relevance ranking has no term for what it already has#

A similarity-ranked top-k selects each item independently: the score of passage 7 does not depend on whether passages 1 through 6 already said the same thing. Follow that one step further and the failure becomes a prediction rather than a surprise. A near-duplicate of a good match is itself a good match, because the two texts differ in a few words and therefore have nearly the same embedding. So when a corpus contains thirty near-duplicates of one document, independence guarantees they occupy the top of the ranking together. The retriever is not making an error. It is answering a different question from the one the prompt needs answered — "which passages are most relevant" rather than "which set of five passages best covers what the user asked".

Maximal marginal relevance is the standard patch and it is old: Carbonell and Goldstein proposed it at SIGIR, the main information-retrieval research conference, in 1998 for exactly this problem in document summarisation and IR, long before dense embeddings existed. It scores candidates against a shrinking objective that subtracts similarity to what has already been chosen.

$$\text{MMR} = \arg\max_{d \in R \setminus S}\ \big[\lambda \cdot \text{sim}(d, q) - (1 - \lambda) \cdot \max_{s \in S} \text{sim}(d, s)\big]$$

Every symbol there is doing one job. $d$ is a candidate passage drawn from the pool $R$ of everything retrieved so far, minus the set $S$ of what has already been selected. $\text{sim}(d, q)$ is the candidate's similarity to the query, which is the ordinary relevance score. $\text{sim}(d, s)$ is its similarity to an already-selected passage $s$, and the $\max$ over $S$ takes the worst case: a candidate is penalised by how close it is to the single selected passage it most resembles, rather than by its average closeness to all of them. $\lambda$ weighs the two terms against each other. With $\lambda = 1$ the second term vanishes and this is ordinary relevance ranking. As $\lambda$ falls, a candidate very close to something already chosen loses more, so the second write-up of a well-documented cause — nearly identical to the first — can lose to a write-up of a different cause despite a lower query similarity. The selection is greedy, one passage at a time, and it runs over a candidate pool you fetch first, typically 5 to 10 times the final k.

import numpy as np

def mmr_select(query_vec, cand_vecs, cand_ids, k=5, lam=0.6):
    """cand_vecs: (n, d), L2-normalised. Returns k ids."""
    rel = cand_vecs @ query_vec              # (n,) cosine to query
    sim = cand_vecs @ cand_vecs.T            # (n, n) pairwise cosine
    selected, remaining = [], list(range(len(cand_ids)))

    while len(selected) < k and remaining:
        if not selected:
            best = max(remaining, key=lambda i: rel[i])
        else:
            def score(i):
                redundancy = max(sim[i][j] for j in selected)
                return lam * rel[i] - (1 - lam) * redundancy
            best = max(remaining, key=score)
        selected.append(best)
        remaining.remove(best)
    return [cand_ids[i] for i in selected]

# Fetch wide, select narrow: MMR can only diversify what it can see.
pool = index.search(query_vec, top_k=50)
final = mmr_select(query_vec, np.stack([p.vec for p in pool]),
                   [p.id for p in pool], k=5, lam=0.6)

The pool size is the parameter people get wrong, and it is the one place where the arithmetic is exact rather than approximate. A diversity term can only reallocate the budget among passages it was given; it cannot summon a passage the pool does not contain. So the pool has to be at least as deep as the rank of the material you want reallocated to. On the checkout question below, the best write-up of the second cause sits at rank 9 and the best write-up of the third at rank 14. A pool of 50 therefore contains both and either can be selected. A pool of 10 contains the first and not the second, so no setting of $\lambda$ can produce coverage of the third cause. A pool of 5 contains neither and MMR is reduced to reordering five write-ups of one cause. Running MMR over a pool of 8 to pick 5 has almost nothing to reallocate, because the pool is already dominated by the redundant cluster — the pool depth, not the parameter, is what decides the ceiling.

What the diversity term costs, measured on two query types#

The evaluation that makes this decision has to be stratified, because the same change helps one kind of question and hurts another. Split the judged set into questions with multiple distinct correct components — causes, options, affected regions — and questions with one true answer. Then measure coverage on the first stratum (fraction of the distinct judged facets that appear anywhere in the retrieved set) and accuracy on the second.

Read the first two columns as gaps between rows rather than as levels, and the shape of the trade appears immediately. Coverage rises fastest at the top of the table and then flattens: the first two steps down from 1.0 buy more than four fifths of the total gain between them, and each of the last two buys a few hundredths. Mean relevance does the opposite, giving up three hundredths on the first step and nine on the last. So the two columns cross somewhere in the middle: early reductions in $\lambda$ buy a great deal of coverage for very little relevance, and late ones buy almost no coverage for a great deal. That crossing is the whole argument for a middle setting, and it is why the multi-cause accuracy column peaks at 0.6 rather than at the bottom of the table where coverage is highest.

The single-answer column is the honest cost. On a question with exactly one correct answer, the second-best passage is usually a supporting statement of the same fact — a corroborating source, the same policy quoted elsewhere — and a diversity term evicts it in favour of something deliberately different, which on a single-answer question means something about a different topic. You have traded a supporting passage for a distractor, and at $\lambda = 0.4$ that costs six points.

A single global $\lambda$ has to be judged against the query mix, and the arithmetic decides more than it looks as though it will. Suppose 70% of traffic is single-answer and the rest multi-cause. Relevance-only ranking then scores seven tenths of 77.6% plus three tenths of 58.9%, which is a little under seventy-two per cent overall. The same weighting at $\lambda = 0.4$ gives seven tenths of 71.4% and three tenths of 68.9%, a little under seventy-one — a net loss, even though the multi-cause numbers look excellent. But the same weighting at $\lambda = 0.6$ gives seven tenths of 75.2% and three tenths of 71.1%, which is a shade under seventy-four, and that is two points better than relevance-only on the identical mix. The mix does not decide whether to use a diversity term. It decides how much of one, and a value one step too aggressive turns the same change from a two-point gain into a one-point loss.

Routing is the obvious response and it is only partly satisfying. A cheap classifier over the query — does this ask for causes, options, differences, a list, or does it ask for one fact — can select $\lambda$ per query, and on the set above a two-class router at $\lambda \in {1.0, 0.6}$ recovered most of both columns. It also introduces a component that is wrong some of the time, and a misrouted single-answer question gets the worse configuration, so the router's error rate has to be small relative to the six-point gap it is arbitrating.

Defining the coverage number, and a proxy for production#

Facet coverage is not a metric any library computes for you, because it needs an annotation that relevance judgments do not contain. For each multi-component query, someone has to enumerate the distinct components — the three checkout causes, the four affected regions, the two competing policies — and mark which passages support each. Ninety queries at around eight minutes apiece is twelve hours of work, done once, and it is the only way to state what fraction of the answer the retrieved set could possibly support. Coverage@k is then the fraction of a query's facets with at least one supporting passage in the top-k, which is a different quantity from recall@k because it weights by facet rather than by passage and therefore refuses to reward thirty documents about one cause.

There is a cheaper proxy that needs no annotation and is the right thing to run in production: the mean pairwise cosine similarity among the retrieved set. On the checkout query it was 0.79 with relevance-only ranking and 0.61 at $\lambda = 0.6$, and across the judged set it correlated with facet coverage at about $r = -0.6$. That is strong enough to alarm on and far too weak to optimise, and the reason is worth stating precisely, because it is the same shape of error this topic keeps producing. The proxy reads the geometry of the retrieved set and nothing else. A set of three genuinely different causes and a set of five mutually unrelated passages that have nothing to do with the query both score well on it, because the property it measures — how far apart the passages are — is identical in the two cases. A retriever tuned to minimise mean pairwise similarity will therefore happily return the second set, scoring perfectly on the proxy and answering nothing. The number is constant in exactly the property that separates a good result from a useless one, which is why it belongs on an alarm and not in an objective. Watch it per query, investigate the queries where it sits above 0.85, and keep the decision on the annotated set.

A third measurement lives on the answer side rather than the retrieval side, and it is the one that actually matters to users. Take the reference answer for each multi-component query, decompose it into its claims, and have the judge check which claims the generated answer makes. That gives answer coverage directly and is robust to a retrieval set that technically contained a facet in a passage the generator ignored — a real failure mode when a diversity term surfaces a passage at rank 5 that the model treats as an aside.

The compute cost of the diversity term itself is negligible and should not enter the decision, and the arithmetic is worth doing once so that nobody argues about it twice. The expensive object is the pairwise similarity matrix, which compares every candidate against every other, so its size is the square of the pool. A pool of 50 candidates at 1,024 dimensions is 2,500 dot products, well under a millisecond; a pool of 500 is 250,000 dot products and around 5 ms. Ten times the pool is a hundred times the work, which is what a square means, and 5 ms is still smaller than the network round trip to fetch the candidates. What limits pool size is the reranking stage, since a wide pool that must be cross-encoded costs real latency. The sensible arrangement is to rerank a moderate pool and apply the diversity selection to the reranked list, so the geometry operates on passages whose relevance has already been established properly.

Cheaper things that solve part of the same problem#

MMR is not the only way to stop a result set repeating itself, and the alternatives differ in what they can see.

The four rows are ordered by what each technique is allowed to look at, and that is what fixes its blind spot. A hash sees bytes, so it catches copies and nothing else. A threshold at 0.95 sees one pair of embeddings at a time, so it catches near-identical text and cannot see that two differently worded write-ups describe one cause. A per-source cap sees metadata, so it catches one document dominating a list and is blind to thirty documents agreeing. Only the diversity term sees the geometry of the whole selected set, which is the level at which the checkout failure lives.

The per-source cap deserves more use than it gets, because a large share of production redundancy is one long document contributing six chunks to a top-10, and a metadata rule fixes that with no similarity computation and no tunable that can go wrong. It does nothing for the checkout case, where the redundancy is across thirty separate incident reports.

Deduplication at index time is worth doing regardless and it is not a substitute, whether it is done by hashing the text to catch exact copies or with MinHash, a compact fingerprint of a document's word shingles that stays similar when the text has been edited slightly. What makes it a poor defence against the failure in this chapter is that it operates before any query exists, and redundancy is a property of the relationship between a query and a corpus rather than a property of the corpus alone.

The corpus this is the first move for#

Forty thousand support-ticket resolutions averaging 120 tokens each, many describing the same handful of recurring problems in slightly different words, is a corpus where diversity is the first thing to reach for rather than the last. Every earlier strategy in this topic is unavailable on it, and for one reason each. At 120 tokens a ticket is already a whole unit — a symptom, a diagnosis and a resolution — so the fragment problem that motivates expansion does not arise. There is no parent worth expanding to, because the container above a ticket is the queue it was filed in, which groups by routing rules rather than by meaning. Sentence-window expansion on a 120-token record returns most of the record. Auto-merging has no hierarchy to merge within, since a flat collection of records has no parents to accumulate siblings under.

What that corpus does have, in abundance, is the failure this chapter describes: five tickets in a top-5, all describing the same recurring problem, differing in the customer's name and the agent's phrasing. Flat chunks — one ticket, one chunk, no splitting at all — with a diversity constraint over a wide pool is the configuration that matches the shape of the data, and the measurement that confirms it is facet coverage on a deliberately redundant eval slice rather than mean relevance.

A diversity term, then, is a decision about what a context budget is for. Spent by relevance alone it buys the best available account of a question, repeated until the slots run out, which is the right purchase when the question has one answer and the wrong one when it has three. Spent with a subtraction for what has already been chosen it buys a set rather than a ranking, at a measurable cost in average relevance and a measurable gain in how much of the answer the generator could possibly assemble. Which of those is the better purchase is not a matter of taste; it is the query mix, the pool depth and the two accuracy columns above.

That is two corpora now with opposite prescriptions: structured reference documents that want expansion, and short repetitive records that want diversity and no expansion at all. Production systems rarely have the courtesy to contain only one, and a single index with a single top-k serving both is where these strategies stop being independent choices and start interfering with each other.