Late-interaction and multi-vector retrieval

Read these first

• Reranking

What you will know

• Build a ColBERT-style late-interaction retriever that stores token embeddings and scores query-document pairs with max-sim aggregation.

  The interaction preserves separate evidence for different query terms, while every additional document token increases index and scoring work.

• Instrument per-document token counts, index bytes, candidate counts, and p95 retrieval latency in a multi-vector implementation.

  Averages conceal the long-document tail, where storage and latency can grow far faster than the document count suggests.

• Compare it against single-vector retrieval and a reranker on the same corpus, query set, candidate budget, and recall or nDCG, latency, and storage measurements.

  A quality gain is only useful if it survives equal evaluation conditions and justifies the extra index and serving cost.

• Reproduce a quality and latency regression caused by increasing document token count, then patch it with chunking or token-budget controls.

  Long documents can exhaust token limits or dominate MaxSim work, causing both missed evidence and unpredictable tail latency.

• Decide against it for a workload that doesn't need it

  Short, focused chunks often already expose the relevant evidence, so extra vectors add operational burden without a measurable retrieval gain.

Chapters

1. 1

   A late-interaction index stores one vector per token, not one per document

   ColBERT-style retrieval keeps the encoder's per-token output rather than pooling it away, typically projected down to around 128 dimensions and often held in fp16. A thousand-token document becomes a thousand vectors. Before any quality argument, hold that fact still: the unit of storage has changed from the document to the token, and everything expensive about this approach follows from it.

   Read chapter
2. 2

   MaxSim keeps the per-term evidence that pooling averages into one number

   For each query token, take its best match among the document's tokens; sum those maxima. Nothing forces one term's strong match to be diluted by another term's weak one, which is precisely what mean-pooling does. Expect the difference on queries carrying two independent constraints — a specific error code alongside a conceptual phrase — where a pooled vector returns documents that are moderately about everything and exactly about nothing.

   Read chapter
3. 3

   Late interaction is itself retrieve-then-rescore, one search per query token

   MaxSim scores a pair, and four billion token vectors are not pairs, so something has to choose which documents get scored at all. Each query token runs its own approximate nearest-neighbour search over the token index, the documents owning those token hits are gathered into a candidate set, and only that set is rescored with full MaxSim over all of its token vectors. The per-token depth is the knob that bounds everything downstream — how many candidates you gather, how much rescoring work each query costs, and where the tail latency comes from — and it is the candidate count the later chapters ask you to instrument and hold equal across systems.

   Read chapter
4. 4

   The token multiplier, not the dimension drop, sets the storage bill

   Going from 768 float32 to 128 fp16 looks like a 12x saving per vector, and reasoning stops there at your peril. Five million documents averaging 800 tokens means four billion vectors against five million: roughly a terabyte where single-vector retrieval used fifteen gigabytes, about 70x more. Do this arithmetic before the meeting, because the plausible version of it is wrong by an order of magnitude.

   Read chapter
5. 5

   Long documents are where the footprint and the tail latency actually live

   An 800-token average built from mostly-short documents and a handful of 20,000-token manuals behaves nothing like a uniform corpus: those few dominate index bytes and MaxSim work, and they hit token truncation limits that silently drop evidence. Instrument per-document token counts, index bytes, candidate counts and p95, then reproduce the regression by admitting long documents and patch it with chunking or an explicit token budget.

   Read chapter
6. 6

   The comparison that decides this is against single-vector plus a reranker

   Late interaction is not competing with naive dense retrieval; it competes with dense retrieval followed by a cross-encoder over the top fifty, which already sees query and document together. Hold corpus, query set and candidate budget identical, then compare nDCG, p95 and index bytes across all three. A yes looks like a quality gain the reranker path cannot reach at any latency you can afford.

   Read chapter
7. 7

   Two-hundred-token support articles have no pooling loss to recover

   The argument against is not that it costs too much — a team told that will simply go and find the budget. It is that the mechanism has nothing to act on: pooling a short, single-topic chunk discards almost no per-term evidence, and recall@50 at 0.94 says the retrieval is not what is failing. What would change your mind is a query class where the evidence sits in one clause of a long, mixed-topic document.

   Read chapter