Two-hundred-token support articles have no pooling loss to recover

Late interaction earns its cost by undoing a specific loss. A single-vector index represents a document by pooling its token vectors into one — averaging them — and whatever distinguished one token from another is gone after the average. Pooling loss is the name for what that average throws away, and MaxSim, which lets each query token find its own best match among a document's stored token vectors, is the operation that recovers it. The size of the prize is therefore the size of the loss, and the size of the loss is a property of the documents rather than of the models.

That makes an argument against adoption a measurement rather than a matter of taste, which matters when a team arrives with a proposal to migrate and a working prototype behind it. A corpus of short support articles, each written about one thing, has very little for an average to blur, and a first stage that is already finding the answers has very little room left to find more of them. Both of those are numbers somebody can produce in an afternoon without building an index, and both of them are stronger ground than a budget.

Budget as an objection, and what it concedes#

Cost is never going to be the argument here, and it usually is not. Even at the terabyte scale of chapter 4, where the storage arithmetic for a token index was worked out, a team that believes there is a quality gain waiting will find $3,500 a month, and framing the objection as budget concedes that the gain exists and reduces the discussion to whether it is affordable. That is a discussion the proposer wins, because a search quality improvement is worth more than a mid-size instance to almost any business. The objection that holds is about the mechanism: on this corpus, there is very little pooling loss for MaxSim to recover, and the failures the team wants to fix are not retrieval failures.

On a corpus of 340,000 support articles averaging 200 tokens each, the budget objection is not merely weak — it is arithmetically empty, and working the arithmetic shows why. A token index stores one vector per token, so the vector count is the document count times the mean tokens per document. Each stored vector costs a fixed number of bytes after compression, which for this family of indexes is 36. Multiplying the three together, 340,000 × 200 × 36 bytes is 2.4 GB, which fits in the page cache of a machine the team already owns.

The step that makes that multiplication legitimate is worth naming, because the previous chapter spent itself on cases where it is not. Using a mean to get a total is only safe when the total is what you actually want and the mean was computed over the same population — which it is here, since bytes add up across documents and no document is being singled out. What the mean cannot tell you is the shape underneath, and that matters when a small tail of very long documents dominates the sum. This corpus has no such tail: a mean of 200 tokens with a p99 of 480 means the longest one percent of articles are barely more than twice the typical one, so the distribution has nowhere to hide a surprise. The mean is a sound capacity input here for the same reason it was a trap in chapter 5 — the condition it depends on happens to hold.

Measuring how much pooling actually discards#

Chapter 2 located the advantage precisely, in documents where the query's evidence occupies a small fraction of the token mass and gets averaged into a centroid dominated by everything else. That framing is measurable directly, without building an index, by asking how far a document's token vectors sit from their own mean.

import numpy as np

def pooling_coherence(token_vecs: np.ndarray) -> float:
    """Mean cosine between each token vector and the document centroid.
    High = the centroid represents every token well = little pooling loss."""
    centroid = token_vecs.mean(axis=0)
    centroid /= np.linalg.norm(centroid)
    return float((token_vecs @ centroid).mean())

# measured over 2,000 sampled documents per corpus
# support articles,   mean 200 tokens : 0.71  (p10 = 0.63)
# product guides,   mean 6,000 tokens : 0.38  (p10 = 0.24)
# hardware manuals, mean 56,000 tokens: 0.29  (p10 = 0.17)

Three steps in that function are doing the work, and each one answers a question about what pooling is. Taking the mean of the token vectors reproduces exactly what a single-vector index stores, so the centroid in the code is the same object the production index holds. Dividing that centroid by its own length turns the dot products that follow into cosines, which strips out how long the centroid happens to be and leaves only how well its direction agrees with each token's — length being a property of how many tokens were averaged rather than of what they said. And averaging those cosines over every token asks the pooling question directly: across the whole document, how typical is the summary of the things it summarises.

Read the answer back and the interpretation follows without any further machinery. A coherence of 0.71 means the pooled vector is a close stand-in for essentially every token in the article, which is what you would expect from a 200-token document about one thing. There is no buried clause, because the article is the clause. The manuals from chapter 5 sit at 0.29, and that gap is the entire mechanism this topic has been describing, expressed as a number you can compute in an afternoon on any corpus.

The three lines of that comment block are also a curve, and reading them as gaps rather than as levels is what turns the measurement into a prediction. Mean length rises from 200 tokens to 6,000 and then to 56,000 — roughly thirty-fold, then roughly ninefold — while coherence falls from 0.71 to 0.38 to 0.29. The first jump in length costs most of the coherence; the second, larger in documents but applied to a corpus that has already lost its coherence, costs much less. Averaging blurs fastest when a document first stops being about one thing, and a document that is already about forty things is not made much blurrier by being about eighty. So the useful reading is that the mechanism's prize is won and lost early, somewhere between a short article and a long guide, rather than accumulating steadily with length.

The second measurement is a rank correlation between the two scoring functions on the same candidates. Take the dense top 50 for each of the 1,000 queries, rescore with MaxSim, and compute Kendall's tau against the pooled ordering — a rank correlation running from -1 to 1 that counts, over every pair of documents, how often the two rankings agree about which of the pair comes first. On the support corpus it came out at 0.93, meaning the two functions largely agree about how to order the documents the retriever already found. On a long-document corpus that figure typically sits between 0.5 and 0.7. Two functions that agree at 0.93 are not going to produce different search results often enough to matter.

The reason tau is the right instrument rather than a score comparison is that nothing downstream consumes a score. A search result page consumes an order, so two scorers that disagree violently about magnitudes and agree about order are the same system as far as any user is concerned. Tau measures exactly the disagreement that can reach a user, and it does so without needing the two scorers to be on a common scale — which they are not, since one is a cosine and the other a sum of maxima over query tokens.

Where the failures actually are#

Recall@50 at 0.94 bounds the argument from the other side. Sixty of the thousand judged queries did not have a relevant article in the top fifty, so sixty is the absolute maximum any retrieval change can address, and reading those sixty tells you how many of them are addressable by this particular change.

Two properties of that table are what make it an argument rather than an anecdote, and both are checkable by addition. The Queries column adds to exactly sixty, with no residual row and no "other" bucket, so the categorisation is exhaustive over the failures it was built from — every one of them was read and placed. And sixty out of a thousand judged queries is six percent, which is exactly what a recall@50 of 0.94 leaves unaccounted for. The count and the recall figure are the same fact arrived at by two routes: one by running a retrieval metric over a judged set, the other by a person reading queries one at a time. When two instruments that share no machinery land on the same number, the number is a population rather than a sample, and a ceiling computed from it is a real ceiling rather than an estimate with an error bar.

What the categories describe is why each query failed the user, which is a wider question than why the retriever missed. The first row is a query whose answer the retriever did find and put below the fold, so it is a ranking failure sitting inside the same sixty; the rows beneath it are content gaps, ambiguity, and the two shapes of retrieval problem. Keeping that distinction visible is what stops the table being read as sixty arguments for a new index.

Two queries in a thousand. Even assuming late interaction fixes both perfectly and breaks nothing, the ceiling on end-to-end improvement is 0.2% of traffic, against a quarter of engineering time, a second index to keep in sync with every content change, a re-encode on every model upgrade, and a new failure domain in the serving path. The thirty-eight ranking failures are the real opportunity, and they are addressable by a cross-encoder over the top fifty, which this team does not yet run and which costs no index at all.

That reading is what makes the argument survive. It does not claim that late interaction is bad or expensive; it claims that the specific quantity it improves is close to zero on this corpus, and it puts a different, larger number next to it.

The three vocabulary-mismatch failures point at a cheaper answer as well. A user typing "card declined" against an article titled "authorization failure handling" is a lexical gap, and the standard remedies are fusing a BM25 ranking with the dense one or expanding queries with learned synonyms, both of which are configuration changes to an existing search stack rather than a new index. Late interaction would help with some of these, since token-level matching does catch surface forms that a pooled vector smooths over, and it is a heavy instrument for a problem that a lexical channel addresses at roughly zero marginal cost.

Operational burden is the part of the case that is usually waved through and deserves a paragraph of specifics. This documentation team publishes around 200 article edits a day, and every edit means re-encoding a document and replacing its token vectors, which in a centroid-and-residual index means a delete plus an append against a codebook that was fit to a snapshot of the corpus. That works for a while and degrades: as the content drifts away from the centroids it was clustered against, residuals grow, compression error rises and the candidate stage's recall quietly declines, so the index needs periodic full rebuilds on a schedule nobody can derive in advance. A single-vector index has none of this, since a document is one row and replacing it is a write. The argument here is not the dollar cost of the rebuild; it is that the team acquires a maintenance obligation whose failure mode is a slow, unalarmed decline in retrieval quality.

That last clause names the shape of the risk precisely. A rebuild that is overdue does not fail, throw or page. It returns slightly worse candidates, which show up as slightly worse answers, which show up as nothing at all unless somebody is running a judged set on a schedule. The world in which the codebook has drifted and the world in which it has not produce the same dashboards, so "we have had no problems" is a sentence both worlds emit and neither is distinguished by.

The probe that settles it in a day#

None of the above requires an index build, and neither does the direct test. An off-the-shelf ColBERT checkpoint can rescore the existing dense top-50 by encoding those fifty documents at query time, which is far too slow for production and perfectly adequate for an experiment over a fixed evaluation set.

from ragatouille import RAGPretrainedModel

colbert = RAGPretrainedModel.from_pretrained("colbert-ir/colbertv2.0")

per_query_ndcg = []
for q in eval_queries:                       # 1,000 judged production queries
    candidates = dense_index.search(q.text, k=50)
    reordered = colbert.rerank(query=q.text,
                               documents=[c.text for c in candidates], k=10)
    per_query_ndcg.append(ndcg_at_10(reordered, q.judgments))

# baseline (pooled ordering) : 0.612
# MaxSim rescoring           : 0.619
# paired bootstrap           : +0.007, 95% CI [-0.011, +0.026]

Encoding fifty documents per query at query time is the step that makes this cheap to run and useless to serve, and separating those two facts is what licenses the experiment. Serving needs the document side precomputed, which is the entire reason a token index exists. An offline evaluation over a fixed set of queries has no such requirement — it can spend a second per query and finish overnight — and it produces exactly the same scores the served system would produce, because the model is the same and the candidates are the same. Slowness is a deployment property, not a correctness one.

Read the three comment lines together and the bootstrap turns out to be adding only one thing. Subtracting the pooled baseline from the MaxSim rescoring gives 0.007, which is precisely the point estimate the bootstrap reports — as it must, since a paired bootstrap resamples the queries you already have and its average difference converges on the difference you already measured. The bootstrap invents no new estimate. What it contributes is the interval, and the interval spans zero, which means MaxSim scoring produces no measurable improvement in ranking on this corpus even when handed the same candidates for free. This does not test the candidate-generation half of the system, and it does not need to: recall@50 is 0.94, so there is at most six points of recall to gain and the failure table has already shown that five of those six points are content gaps and ambiguity rather than retrievable documents the index missed. A probe that comes back flat on ranking, over a corpus whose recall is already high, has closed both halves of the case.

There is a way to run this conversation that does not turn it into a contest. Hand the two scripts to the engineer who wrote the proposal and let them produce the coherence figure and the probe result, since a number someone generated themselves settles a question that the same number quoted at them does not. Pair it with the alternative that the failure table pointed to: reranking the dense top fifty with a cross-encoder addresses thirty-eight of the sixty failures, is a week of work rather than a quarter, and can be measured with the same harness the probe just used. A proposal redirected toward a larger effect is easier to accept than one that is simply declined.

The conditions that would reverse this#

A recommendation against something is worth more when it names its own expiry conditions, and these are specific enough to be monitored rather than revisited by memory.

Every threshold in that table sits between two corpora this topic has already measured, which is what makes it a tripwire rather than a round number somebody liked. The coherence row is the clearest case: the support corpus's p10 is 0.63 and the guides corpus's p10 is 0.24, and the threshold of 0.45 falls close to midway between them, so the wire trips when a corpus stops resembling the one the decision was made on and starts resembling the one where the mechanism pays. The same holds along the other rows. A p95 length of 1,500 tokens sits above this corpus's p99 of 480 and well below the guides' mean of 6,000. A tau of 0.75 sits below the 0.93 measured here and above the 0.5 to 0.7 band that long documents produce. Each threshold is calibrated on measurements the topic has already taken rather than on a guess about the future.

The first row is the one most likely to fire, because corpora acquire long documents through business decisions rather than through search decisions. A partnership that brings 4,000 policy PDFs into the same index, or a move to indexing whole knowledge-base exports instead of curated articles, changes the coherence number overnight and changes the answer with it. The right cadence is to recompute the table quarterly and to recompute it unconditionally whenever a new content source is onboarded, which takes under an hour once the scripts exist.

Recomputing on a schedule is what makes the recommendation durable, and skipping it fails silently in the same way the codebook drift does. A team that decided against this two years ago and has heard no complaints since has learned nothing, because a corpus that quietly acquired a long-document source produces no complaint either — the search results get worse in a way users attribute to the documentation rather than to the index. The quarterly recomputation exists so that the answer is re-derived rather than remembered.

Two things this argument deliberately does not claim. It does not say the technique fails, and a version of it that did would be wrong and would age badly; the same probe on the guides corpus at coherence 0.38 returned +0.041 with an interval clear of zero. It also does not say the team was wrong to prototype. The prototype is what made the coherence and tau numbers cheap to produce, and a proposal that arrives with a working prototype and no measurement of the mechanism is one measurement away from being decidable.

The material in this topic reduces to four numbers you can produce without committing to an index. Compute the true token-length distribution of your corpus and the fraction that would be truncated at your doc_maxlen, which gives you the storage estimate and the truncation risk together. Compute recall@50 for your current first stage against real judgments, which bounds every retrieval improvement available to you. Compute pooling coherence over a sample, which tells you whether the mechanism has anything to act on. Rescore your existing candidates with an off-the-shelf checkpoint and bootstrap the difference, which tells you whether it does act.

If those four say go ahead, the build has a defined shape: measure sum(min(length, doc_maxlen)) before and after every index build, emit vectors_rescored on every query and alert on its p99, sweep ndocs against your own quality curve rather than adopting a default, and compare the result against dense plus a cross-encoder at equal latency. If they say stop, the tripwire table is the artefact to leave behind, since the corpus that does not need this today is frequently not the corpus you will have in a year.