Choosing the headline metric

One retrieval service can have two consumers with incompatible needs. A command palette takes whatever the user types, shows a single top suggestion, and jumps there when they press enter. A support assistant takes the same query, pulls eight chunks, and hands them to a model that writes an answer. The first cares only about what sits at rank 1; the second cares about the composition of an eight-document set and about which of those documents survive context truncation.

The metrics in this chapter all read the same ranked list and the same judgments. Hit rate@k asks whether anything relevant appeared in the top k. Precision@k is the fraction of the top k that is relevant. MRR, mean reciprocal rank, is the average of $1/r$ where $r$ is the rank of the first relevant document. Recall@k and nDCG@k are the pair from the previous chapter. Choosing among them is not a matter of rigour, because they are all rigorous; it is a matter of which feature of the list the thing downstream is sensitive to.

Five metrics, one run#

The differences are easiest to see computed on the same handful of queries. Take five from a golden set, binary relevance for simplicity, with the ranks at which relevant documents were returned in the top 10 and the total number of relevant documents each query has.

Hit rate@3 asks a yes-or-no question per query: was anything relevant in the top three? That is true for q1, q2 and q4 and false for q3 and q5, so the run scores 0.60. Precision@5 is the fraction of the top five that is relevant, computed per query and then averaged: 0.2, 0.2, 0.0, 0.4 and 0.2, giving 0.20. MRR takes the reciprocal of the rank of the first relevant document for each query — 1, $\tfrac{1}{3}$, 0, 1 and $\tfrac{1}{4}$ — which sums to 2.583 over five queries, so 0.517. Recall@10 and nDCG@10, computed as in the previous chapters, come out at 0.60 and 0.501.

Recall is worth writing out because it is the one whose per-query values are not all the same shape. Each query contributes the fraction of its own relevant documents that appeared: q1 found its single one, so it contributes a whole; q2 found one of two, so a half; q3 found none, so nothing; q4 found all three, so a whole; q5 found one of two, so a half. Those five contributions total three, and three over five is the 0.60. Notice that q4 and q1 contribute identically despite q4 having three times as much to find — recall normalises each query by its own denominator, which is the choice that makes queries comparable and the choice that hides how much work q4 required.

nDCG's five per-query values follow from the discount table rather than from counting. q1 has one relevant document at rank 1, so it is at its ideal and scores a whole. q4 found all three, but its third sits at rank 6 where the ideal would have put it at rank 3, so it comes out a little over nine tenths. q2 found one of two and put it at rank 3, which is worth about three tenths of an ideal that expected two documents at the top. q5 is similar and slightly worse for the same reason. q3 scores nothing. Averaged, those five come to 0.501. The whole table above is five queries and ten facts, and every one of the five headline figures is derivable from it — which is what makes the disagreements between them arithmetic rather than opinion.

Two of those rows deserve a second look. Precision@5 of 0.20 sounds like a disaster and is close to the arithmetic maximum for this set: a query with one relevant document cannot exceed 0.2 at $k=5$ no matter how perfect the ranking, so averaging precision@k over queries with different numbers of relevant documents averages in a per-query ceiling that has nothing to do with quality. R-precision, which evaluates precision at a rank equal to that query's number of relevant documents, exists to remove that artefact, and it is the reason precision@k is rarely the headline for a corpus where relevance counts vary.

MRR's blind spot is the opposite. It read q4 as a perfect 1.0 because the first hit was at rank 1, and the two other relevant documents at ranks 2 and 6 contributed nothing at all. MRR also moves in discrete steps — the per-query values available are 1, 0.5, 0.333, 0.25, 0.2 and so on — so a move between two adjacent positions deep in a list is worth around 0.021 while a move from rank 2 to rank 1 is worth 0.5. Those two figures come from one expression. The value of moving a document from rank $r+1$ to rank $r$ is $1/r - 1/(r+1)$, which simplifies to $1/(r(r+1))$ — so the worth of a single-position move falls with the square of the rank. At the top of the list that product is one times two, giving a half. Around rank 7 it is seven times eight, giving something near a fiftieth. The same move is therefore worth roughly thirty times more at the top of a list than a few positions down, and the reason is not a design choice layered on top of MRR; it is what taking a reciprocal does. That heavy weighting of the very top is a defect when several documents matter and precisely the property you want when only the first one is ever seen.

Hit rate@k and recall@k coincide exactly when every query has one relevant document, which is worth knowing because a great many published benchmarks for RAG — retrieval-augmented generation, abbreviated that way almost everywhere, the arrangement where retrieved chunks are placed in a model's prompt before it answers — are built that way and report "hit rate" for what is arithmetically recall. On a golden set judged completely, as the first chapter of this topic insisted, they diverge: q4 counts as a hit at $k=3$ while contributing 0.67 to recall@3, because two of its three relevant documents made the cut.

Which sets up the trap in the results above, and it is worth naming before somebody falls into it. Hit rate@3 and recall@10 both came out at 0.60 on this run, and they did so for unrelated reasons. Hit rate counted three queries out of five as successes and two as failures. Recall added up five fractions — one, a half, nothing, one, a half — that happened to total three. Nothing connects the two calculations, no property of the set makes them agree, and the next run will separate them. A reader who takes two metrics reporting the same number as evidence that they measure the same thing has drawn a conclusion from a coincidence, and the way to avoid it is to keep the per-query values rather than only the means, because two means can collide where ten per-query numbers cannot.

Matching the metric to the consumer#

Writing down what each consumer does with the list settles which number belongs on the front page.

MRR earns the headline when the consumer reads exactly one result and there is a single correct answer, because under those conditions the reciprocal rank is the user's experience: rank 1 means they got it, rank 4 means they scrolled or gave up. nDCG earns it when several documents contribute in graded amounts, because it is the only metric here that reads both the grade and the position. The two ways of getting this wrong are symmetrical — reporting nDCG@10 for a one-result surface, where nine of the ten positions are never rendered, and reporting MRR for a RAG pipeline, where a run that puts one perfect chunk first and seven irrelevant ones after it scores 1.0.

The reason those two can be pulled in opposite directions by one change is that they read overlapping parts of the list with wildly different weights. MRR reads position 1 and, once a relevant document is there, is indifferent to everything below it — the query is already at its maximum and no further improvement or damage registers. nDCG at a cutoff of eight reads all eight positions, with the deepest of them still carrying about a third of a top slot. So a change that lifts one document into position 1 by displacing others downward is pure gain to MRR and a mixture to nDCG: the promoted document gains the difference between two adjacent discounts, and any document pushed past the cutoff loses its entire contribution rather than a fraction of it. Falling off the end is a step change, and a promotion at the top is a small increment, which is why the mixture can come out negative while the first metric records a clean win.

For a compact tour of what each rank-aware metric rewards, including mean average precision, which sits between precision@k and nDCG and is the natural choice when judgments are binary and you care about the whole ranked list, Every Ranking Metric: MRR, MAP, NDCG by ritvikmath covers the family in about fifteen minutes.

def headline_metrics(rows, weights):
    """rows: per-query dicts from evaluate(); weights: traffic share per stratum."""
    by_stratum = defaultdict(list)
    for qid, r in rows.items():
        by_stratum[STRATUM[qid]].append(r)
    report = {}
    for stratum, rs in by_stratum.items():
        report[stratum] = {m: mean(r[m] for r in rs) for m in rs[0]}
    overall = {
        m: sum(weights[s] * report[s][m] for s in report)
        for m in next(iter(report.values()))
    }
    return {"overall": overall, "by_stratum": report}

The traffic weighting from the sampling chapter applies to whichever metric becomes the headline, and it applies per stratum rather than to the pooled mean, because a set that over-samples migration queries will otherwise report a headline that no user population experiences.

Sensitivity is a second criterion for the headline slot, and it is easy to overlook. A metric earns its place partly by being able to detect the changes you make, and these differ substantially in how noisy they are on the same 160 queries. MRR's per-query values live on the discrete ladder described above, so its standard deviation across the set was 0.41 and the standard error of its mean 0.032. nDCG@10 varies continuously and came out with a standard deviation of 0.29 and a standard error of 0.023. Hit rate@3 is a Bernoulli variable per query and sits between them. Both standard errors are the standard deviation divided by the square root of the set size, and on 160 queries that divisor is a little under thirteen: 0.41 over it gives 0.032, and 0.29 over it gives 0.023. The two figures are therefore not independent facts about the metrics — they are one fact, the spread of per-query values, seen through the same divisor. What makes MRR the noisier of the two is entirely its per-query spread, and the ladder above is why: values that can only be 1, a half, a third or zero cluster at the extremes and spread widely, while nDCG's per-query values fall anywhere in between.

The practical consequence is that a change worth two points is detectable on nDCG and buried in MRR, so a team whose only headline is MRR will run experiments that come back inconclusive and conclude their ideas do not work. That comparison is about the ratio between the two rather than about either in isolation, and the qualification from the sampling chapter applies here too — these bands describe drawing a fresh sample of queries, and a before-and-after comparison on the same queries is far more sensitive than they suggest, because query difficulty cancels. What survives the qualification is the ordering: whatever resolution a paired comparison achieves, nDCG achieves roughly half as much again of it as MRR does, on the same queries, for the same effort.

Two consumers, one index#

When two surfaces share an index and a query pipeline, the metrics separate by consumer and the honest options become visible. The first is to accept a single configuration and gate on both metrics, refusing any change that improves one by less than it damages the other — workable, and it slows both teams down. The second is to serve the two consumers from the same index with different post-retrieval configurations: one takes the top result after a title-boosted rerank, the other takes eight chunks after a cross-encoder pass with no title boost. The index is shared, which is where the cost is, and the ranking stage is not, which is where the disagreement was.

The second option is usually right and it has a consequence for evaluation. Two configurations mean two runs on every golden-set evaluation, with the single-result run scored on MRR and hit rate@1 over the navigational and entity strata, and the assistant run scored on nDCG@8 and recall@50 over all six. The golden set is shared; the metric, the cutoff and the relevant strata are not. Reporting a single number across both would average an experience nobody has.

What goes on the wall and what goes in the report#

The dashboard a team looks at every day carries two numbers, not five. The headline is nDCG@10 for a system whose primary consumer is an assistant, and next to it, smaller, is recall@50 at the candidate depth. Everything else — precision@10, MRR, hit rate, the per-stratum breakdown, the distribution of per-query scores — lives in a report anyone can pull up and that gets read when the headline moves.

The pairing is not decoration. The argument for dropping recall is that nDCG accounts for both presence and position and therefore contains everything recall contains, and it fails for the reason the previous chapter established: nDCG@10 cannot express what was never in the candidate list. Recall@50 on the same wall costs one tile and answers the question "is there anything left to rank". When nDCG@10 falls and recall@50 holds, the ordering broke and the fix is in the reranker or the fusion weights. When both fall, something upstream changed — an analyzer, a chunking rule, an ingestion job — and the reranker is innocent. That two-tile layout is a diagnosis, and either tile alone is a symptom.

Two tiles that each go up or down give four combinations, and the layout is only worth the space if all four say something. Two of them are covered above. The third is both holding, which is the ordinary state and the reason a dashboard can be glanced at. The fourth is the interesting one: recall@50 falling while nDCG@10 holds. That means documents are leaving the deep part of the candidate list without yet disturbing the top ten, which is what a slow ingestion or analyser problem looks like before it reaches a user. It is the only one of the four that is a warning rather than a report, and it is invisible on any dashboard carrying the headline alone — which is the whole argument for the second tile, stated as a case rather than as a principle.

None of these numbers is a business metric, and none of them should be presented as one. The online counterparts — click-through at position 1, the rate at which a session ends without a follow-up query, the fraction of assistant answers marked unhelpful — are what the company actually cares about, and offline metrics are a cheap proxy that lets you iterate hourly instead of fortnightly. The proxy is worth having precisely because it is fast, and it is worth distrusting whenever it moves in the opposite direction from the online numbers.

The metric family is small, well understood, and not in dispute; what is in dispute, whenever two people quote different numbers at each other, is which feature of a ranked list matters for the thing consuming it. Answer that first, in one sentence about what the consumer reads and how many correct answers exist per query, and the headline picks itself. Compute the rest anyway, because they cost nothing once the ranked lists are stored and they are what you reach for when the headline moves and nobody knows why.

There is a condition underneath everything in the last four chapters that has been assumed rather than stated. Comparing 0.66 against 0.58 presumes the two numbers were produced over the same documents, the same queries, the same judgments and the same cutoff, and none of those four is stable in a live system by default. Making a number from March mean something in September takes deliberate work.