The comparison that decides this is against single-vector plus a reranker

Two things have to be named before any comparison of these architectures reads correctly. A cross-encoder is a model that takes a query and one document as a single joined input and emits one relevance score for that pair, having attended across both at every layer; it stores nothing and can be run only on pairs someone has already assembled. Late interaction keeps one vector per token of every document, and scores a pair with MaxSim — each query token takes its best match among that document's stored vectors, and those bests are added up — which lets the document side be computed and stored in advance. The first sees more of the pair and can be applied to almost nothing; the second sees less and can be applied to a whole corpus.

The comparison that decides an adoption therefore has three arms rather than two, and the middle one is the one that goes missing. Dense retrieval on its own is a candidate stage with no careful scorer at all, and no serious search system ships that way; what production usually runs is dense retrieval feeding a cross-encoder over a shortlist. Judging any of it needs relevance judgments — human labels saying which documents count as answers to which queries, graded when a labeller assigns degrees of relevance and binary when each document is simply marked relevant or not, with no grades in between — and the quality of those labels bounds the quality of every number below.

Two paths that both end in a reordered list#

Drawn side by side, the structural difference is where the expensive stage gets its input. The cross-encoder can only reorder what dense retrieval handed it, so its ceiling is the recall of the top fifty; a document that ranked 300th is invisible to it no matter how well cross-attention would have scored it. Late interaction's expensive stage draws from a candidate pool built by its own token-level search, which is a different pool with different recall. That is the axis on which the two systems genuinely differ, and it is not the axis most evaluations measure.

The word "reorder" carries the whole argument, so it is worth taking apart. A reranker receives a list of fifty documents and produces a list of the same fifty in a different order. Nothing in that operation adds a document, because the model is only ever called on pairs assembled from the list it was given, and nothing removes one, because every input gets a score. The output is therefore a permutation of the input: a different arrangement of an identical set.

Recall at a depth is a property of a set. recall@50 asks what fraction of the judged-relevant documents appear anywhere in the top fifty, and "anywhere" is what makes it order-blind — a relevant document at rank 2 and the same document at rank 49 both count once. Put the two facts together and the conclusion is forced rather than measured: a reranker over the top fifty produces a set identical to its input, and recall@50 depends only on the set, so a reranker over the top fifty cannot move recall@50 by any amount, for any query, with any model.

That is the sense in which the first stage sets a ceiling. Everything the cross-encoder can achieve is achieved by arranging documents it was handed, so the fraction of queries whose answer was never handed to it is a fraction it scores zero on however good it is. The same reasoning does not extend to a reranker fed a deeper list: rescore the top 200 and the top fifty afterwards is a different set, drawn from a larger one, and recall@50 can rise as far as the first stage's recall@200. Depth is what makes the ceiling move; the model never does.

The practical consequence is a diagnostic you can run before building anything. Measure recall@50 of your dense retriever against your relevance judgments, meaning the fraction of judged-relevant documents that appear anywhere in its top fifty, which is the ceiling on everything a later stage can do. If it is 0.94, the reranker path is leaving at most six points of recall on the table and no retrieval change can be worth more than that; if it is 0.62, a third of the answers are unreachable by the reranker and late interaction has something real to win. The whole decision tends to turn on that single number, and it costs an afternoon to produce.

Equalising the comparison#

Four things have to be held constant before any quality number is worth reading, and each of them is a way people accidentally cheat.

The corpus snapshot and the chunking must be identical. This is harder than it sounds after the previous chapter, since a late-interaction index is often built with chunking that the incumbent dense index never had, and chunking alone moves nDCG. If you change the chunking, change it on both.

The query set must be large enough and judged independently. Four hundred queries with graded judgments is a reasonable floor; 120 binary judgments written by the person hoping for a result is not evidence. Sample the queries from production traffic by frequency rather than picking interesting ones, and keep a held-out slice.

The candidate budget must be reported for every system, and swept rather than fixed at one point. There is no honest single value of "equal candidates" between a path that reranks 50 with a cross-encoder and a path that rescores 1,024 with MaxSim, because the per-candidate costs differ by three orders of magnitude. What is comparable is a curve: sweep the reranker's depth over 50, 100, 200, 500, 1000 and sweep ndocs over 256 to 16,384, measure quality and p95 at every point, and compare the two systems at equal p95 and at equal monthly cost.

Choosing which of those two equalities to compare at is a design decision rather than something the measurement forces. Equal p95 answers "which system is better inside the latency budget users already experience"; equal monthly cost answers "which system is better for the money already being spent". They can disagree, because one architecture spends on GPU seconds that scale with traffic and the other on RAM that scales with the corpus, and a team that reports only the comparison that favours its preferred answer has not lied about anything.

The hardware and the measurement must match. Both systems need the same instance class, the same warm cache state, the same concurrency during the latency run, and p95 reported rather than mean, since chapter 5 established that the tail is where the difference lives. The latency run also has to use the production query mix rather than the evaluation query set, because a late-interaction system's tail is a function of which documents its candidate stage pulls in, and an evaluation set curated for coverage will not reproduce the length distribution of what real users ask about. Running the two side by side under replayed traffic for an hour is worth more than any number of single-query timings.

Two rows change the reading of this table. Reranking cannot move recall@50, which is why the second column repeats the first, and that identity is the structural point drawn in the diagram rather than an artefact of these numbers. And the nDCG@10 gap between the reranker path and late interaction is 0.013, which on 400 queries is inside the noise.

The recall row deserves one clarification, because it is the row a reader is most likely to misread. The figure quoted for each column is the recall of that column's candidate stage — the ceiling the column's scorer inherits — which is why the fourth column also reads 0.86 despite rescoring two hundred documents rather than fifty. For the second column the equality is exact and unavoidable, since the reranker's input depth and the measurement depth are both fifty. For the fourth it is a statement about the ceiling rather than about the final ordering, and a post-rescoring recall@50 for that column would be a different and slightly larger number. Two columns of a comparison table agreeing to the last digit is a result to be pleased about: a controlled experiment should reproduce the quantities it was not supposed to change, and if this row had differed, something other than the scorer had moved between the runs.

The Index size row closes in the same way, and it closes across the whole table. The fourth column stores the token vectors and keeps the existing dense index to retrieve with, so its footprint should be the sum of the two — and 144 GB plus 15.4 GB is the 159 GB printed. Nothing was measured twice there; the third figure is the first two added. The 15.4 GB itself divides back into a per document cost: five million documents holding one vector each works out at roughly three kilobytes apiece, which is one vector plus its overhead and is the same whatever any of those documents contains. That is the axis each architecture pays on, visible as arithmetic — the single-vector index charges by the document, the token index charges by the token, and the hybrid pays both bills because it runs both structures.

Read the latency row as gaps rather than levels and the same split appears in time. Going from dense alone to dense plus a cross-encoder costs over a hundred milliseconds at p95 for fifty pairs of GPU work, and that cost is paid on every query for as long as the system runs. Going from dense alone to late interaction costs less than that at p95 while doing no GPU work beyond encoding the query, and the price appears instead as a fixed monthly figure in the row underneath. One architecture converts quality into a marginal cost per query; the other converts it into a capital cost per corpus.

The fourth column is the cheapest experiment in the table and the one most often skipped. It stores the token vectors but builds no centroid index over them, retrieves candidates with the existing dense index, and runs MaxSim as a rescoring pass over the top 200 by fetching each candidate's vectors by document id. Reads are sequential within a document and keyed by a primary key, so the vectors can sit on ordinary block storage with a cache in front rather than in RAM, and there is no k-means build, no residual compression to tune and no new query path. Its recall is capped by the dense retriever, which is exactly why it is informative: it isolates the ranking contribution of MaxSim from the retrieval contribution of the token-level candidate stage. If this column already matches the full system, your gain is in scoring and you do not need the token index; if it lands well below, the gain is in candidate generation and the index is the whole point.

Judgments are the input that most teams do not have and the one that determines whether any of this is measurable. Graded relevance labels over 400 production queries are perhaps three days of annotator time, and the cheaper substitutes have known distortions: click logs are biased toward what the current system already ranks highly, which systematically understates any change in recall, and model-generated labels agree with human graders well enough for coarse comparisons and poorly on exactly the near-duplicate near-miss cases where these systems differ. Pooled judging avoids the worst of it — take the union of the top 20 from every system under test, judge that pool blind to which system produced each document, and accept that anything outside the pool counts as irrelevant.

Whether 0.013 is a result#

A difference in mean nDCG is a sample statistic and needs an interval before it can be argued about. The standard tool is a paired bootstrap over queries, which draws thousands of resampled query sets from the one you have — sampling with replacement, so some queries appear twice and others not at all — and reports the range the difference moves over. It respects the fact that both systems answered the same queries and that per-query nDCG varies far more than the difference between systems does.

Two details of that procedure carry the reasoning. The resampling is over queries rather than over scores, because the thing you want to generalise to is a different sample of queries from the same traffic, and shuffling scores would be asking a question about a different world. And the two systems are resampled together, on the same drawn indices, because they answered the same queries: a query that is easy for both contributes a large score to both and a difference near zero, and pairing lets that shared difficulty cancel instead of being counted as noise twice.

import numpy as np

def paired_bootstrap(a: np.ndarray, b: np.ndarray, rounds: int = 10_000) -> tuple:
    """a, b: per-query nDCG for the two systems, aligned by query."""
    rng = np.random.default_rng(0)
    n = len(a)
    diffs = np.empty(rounds)
    for i in range(rounds):
        idx = rng.integers(0, n, n)          # resample queries, not scores
        diffs[i] = a[idx].mean() - b[idx].mean()
    return diffs.mean(), np.percentile(diffs, [2.5, 97.5])

mean_diff, ci = paired_bootstrap(late_interaction_ndcg, reranker_ndcg)
print(f"{mean_diff:+.3f}  95% CI [{ci[0]:+.3f}, {ci[1]:+.3f}]")
# +0.013  95% CI [-0.008, +0.034]

An interval spanning zero means the ranking quality of the two systems is indistinguishable on this query set, and it would be dishonest to present the 0.013 as a gain. The recall difference is a different matter: 0.93 against 0.86 on the same 400 queries came out at +0.07 with a 95% interval of [+0.041, +0.099], which is a real effect and points at where the value actually sits. Late interaction was finding relevant documents that the dense first stage never surfaced, and then failing to convert that advantage into ranking quality because the cross-encoder was better at ordering the documents it did see.

Comparing the two intervals side by side shows why one is a result and the other is not, and the comparison needs no statistics beyond subtraction. Each interval is centred exactly on its own point estimate — the distance from the lower bound up to the estimate equals the distance from the estimate up to the upper bound, in both cases — so the only information either interval carries beyond the estimate is its half-width. For the nDCG difference that half-width is larger than the difference itself, which is what drags the lower bound below zero. For the recall difference it is smaller than the difference, which is what keeps the whole interval on one side of it. The two results are not distinguished by their sizes; the recall gap and the nDCG gap are the same order of magnitude. They are distinguished by how much the per-query numbers wobble underneath them, and recall wobbles less because it is a count of presences rather than a discounted sum over positions.

That observation suggests the configuration that usually wins, which is not either column of the table. Use the late-interaction candidate stage to build a pool with recall 0.93, then rerank its top 25 with the cross-encoder you already own. On this corpus that combination scored nDCG@10 0.574 at p95 196 ms, which beats everything else and costs both the 144 GB index and the GPU pass. Whether that is worth having depends on how much of your quality problem is retrieval rather than ranking, which is precisely what the recall numbers tell you.

That p95 figure looks too high on first inspection and is not. Halving the reranker's depth from fifty to 25 does not halve its contribution to the tail: the cross-encoder pass carries a fixed cost per call — the request hop, the batch assembly, the kernel launches — that is paid whether it scores fifty pairs or 25, and only the per-pair work scales. A stage that is cheaper in total and more expensive per candidate is what batching looks like from the outside, and a depth sweep that assumes proportionality will misprice every point on it.

A gain the reranker path cannot reach#

The result that justifies adoption is a quality gain the reranker path cannot reach at any latency you can afford. Producing it means sweeping the reranker's depth until it either closes the gap or stops trying: if reranking the dense top-500 recovers the recall and lands at nDCG 0.549 with p95 620 ms, then the two systems are equivalent in quality and the choice is between 620 ms of tail latency and $3,000 a month, which is a budget conversation with a clear answer either way. If reranking the top-1000 still sits at 0.51 because the dense retriever's recall@1000 is itself capped at 0.88, the gap is first-stage recall that no amount of reranking depth can buy, and that is the case where late interaction is the only thing on the table that fixes it.

Those two outcomes are separated by one question, and the depth sweep is what answers it: does the incumbent's ceiling rise fast enough with depth to overtake the challenger before the latency budget runs out. A ceiling that keeps rising means the incumbent was under-configured and the fix is free. A ceiling that flattens while quality stays short means the missing documents are missing from the dense index's neighbourhood entirely, and no depth reaches them.

A yes is also allowed to rest on latency rather than quality. Equal nDCG at 118 ms against 148 ms, with the GPU inference removed from the serving path entirely, is a genuine operational win for a service under strict tail-latency requirements, and it converts a per-query variable cost into a fixed monthly one. That trade improves as query volume rises and worsens as the corpus grows, so the sensible way to present it is as a crossover point rather than as a verdict.

This comparison does not establish that cross-encoders are obsolete, and any presentation that implies it should be distrusted. It also does not transfer between corpora: everything in the table is a property of a specific document distribution, a specific query mix and a specific pair of checkpoints, and the same experiment on 200-token support articles produces different numbers with a different conclusion. The one part that does transfer is the method — the recall@50 diagnostic, the swept depth, the paired interval — and the discipline of comparing against the system you actually run.

The three arms of that method answer three separate questions, and keeping them separate is what stops a comparison collapsing back into a slide. Recall says how much of the answer set either architecture can reach. The swept depth says whether the incumbent can reach it more cheaply than the challenger. The paired interval says whether the remaining difference is a difference at all.

Running that method carefully sometimes produces a clear negative, and a negative result argued badly gets overturned in the next planning cycle. The last chapter is about making the case against adoption in a way that survives contact with a team that wants this.