A late-interaction index stores one vector per token, not one per document

A conventional dense retriever stores one vector per document: a list of numbers, produced by a neural text encoder, arranged so that texts about similar things sit near each other. Everything a document says is squeezed into that one summary, and the weakness this topic exists to fix is the one that squeezing creates. A query carrying a rare identifier alongside an ordinary topic — an error code together with "retrying multipart upload" — tends to be answered by whichever document is broadly on topic, while the one containing the exact clause about that error code ranks below it. The clause is in the document; it is simply not addressable in the summary, at any threshold and in any ranking. Per-token vectors are what keep that clause addressable.

Late-interaction retrieval keeps the encoder's output for every token — every word or word-piece the tokeniser emits — so a passage is stored as many vectors rather than one, and the query is itself a set of vectors. The two sets are compared piece by piece at search time, after each side has been encoded without sight of the other, which is what the word "late" describes. ColBERT, the reference model, is short for Contextualized Late Interaction over BERT, BERT being the transformer text encoder underneath it. The same family goes by multi-vector retrieval, named after what it stores. You meet it in practice as the ColBERT and ColBERTv2 checkpoints and the RAGatouille library that wraps them; as the multi-vector index types Vespa, Qdrant and Weaviate have added; and as ColPali, which applies the same idea to page images. Teams reach for it when retrieval keeps returning documents that are plausibly relevant and do not contain the answer.

The pooling step is where the document became a point#

A transformer encoder never produces a single embedding. Feed it a passage and it produces one contextual vector per input token: a 420-token chunk through BERT-base comes out as a 420 × 768 tensor, where the vector at position 17 is the model's representation of that particular word in that particular sentence. The single embedding that a conventional dense retriever stores is the output of a pooling step bolted onto the end — a mean over the token vectors, or the vector at the [CLS] position, or a weighted variant of the two.

It is worth being precise about why that step is there, because the reason is not a property of the encoder. The chain runs: an approximate nearest-neighbour index — the structure which, given a query point, returns the stored points closest to it without comparing against every one of them, trading a small chance of missing a true neighbour for a search that finishes in milliseconds over millions of vectors — is built to hold points. A sequence of token vectors is not a point. Something therefore has to collapse the sequence before it can be stored. Pooling is that collapse. So the single vector per document is an artefact of the index's data model rather than something the encoder wanted to produce, and it is a design decision rather than a requirement: the encoder's per-token output was there the whole time and was discarded on the way to the index.

ColBERT removes that step. The per-token vectors go through a linear projection down to 128 dimensions, are L2-normalised — rescaled to unit length — so that a dot product between two of them is the cosine of the angle between them, a similarity running from -1 to 1, and are written to the index as they are. The projection is the only compression the architecture applies by default, and its purpose is to make the storage bearable rather than to improve quality; the released ColBERT checkpoints use dim=128, and the difference in retrieval quality between 128 and the full 768 is small enough that nobody stores the full width.

import torch
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("bert-base-uncased").eval()
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
proj = torch.nn.Linear(768, 128, bias=False)   # ColBERT's projection head

text = "Retries on multipart upload fail when the part size is below 5 MiB."
enc = tok(text, return_tensors="pt")

with torch.no_grad():
    hidden = model(**enc).last_hidden_state      # (1, 19, 768)

single = torch.nn.functional.normalize(hidden.mean(dim=1), dim=-1)    # (1, 768)
multi = torch.nn.functional.normalize(proj(hidden), dim=-1)           # (1, 19, 128)

print(tuple(single.shape), single.numel() * 4, "bytes at float32")
print(tuple(multi.shape), multi.numel() * 2, "bytes at float16")
# (1, 768)      3072 bytes at float32
# (1, 19, 128)  4864 bytes at float16

The two byte counts in that output are worth deriving rather than reading off, because the derivation is the whole cost model. A pooled vector holds 768 values at four bytes each, which is 3,072 bytes, and that figure is a constant: feed the encoder a sentence or a treatise and the pooled output is the same shape, so the same 3,072 bytes come out. A late-interaction representation holds 128 values at two bytes each for every token, which is 256 bytes per token, and that figure is a rate rather than a constant. The nineteen-token sentence therefore costs nineteen times 256, which is the 4,864 bytes printed above.

Divide the second by the first and the ratio between the two representations falls out as the token count times 256, over 3,072 — which is the token count divided by twelve, since 3,072 is twelve lots of 256. That single division is the arithmetic behind every number in this topic. Nineteen tokens divided by twelve is a ratio of 1.6x, which is why the multi-vector version looks harmless on one sentence. A 420-token chunk divided by twelve is 35x. An 800-token document divided by twelve is 67x.

Notice which of the two terms in that fraction can move. The twelve is fixed by the model card, and a deployment does not get to choose it once a checkpoint is chosen. The token count is set by the documents, and across the three ratios just computed it moves by more than an order of magnitude. So the divisor cannot respond to anything: it is the same twelve for a corpus of tweets and a corpus of legal opinions, and an argument resting on it reads identically in the case where the storage is trivial and the case where it is ruinous. Document length is the only term that can push the ratio anywhere, and it is the term such arguments leave out.

The mental model that produces this reading is the one where a vector index is a table with one row per document, and where changing retrieval strategy means changing what goes in the row — a different model, more dimensions, a different distance metric. Under that model the only lever on index size is the width of the vector, so a move from 768 float32 values to 128 fp16 values reads as a straightforward saving, fp16 being half-precision floating point, two bytes per value against float32's four. Late interaction does not change what goes in the row. It changes how many rows there are, from one per document to one per token, and that multiplier sits in front of every other term in the cost. Stated as one sentence: a ColBERT-style index stores one reduced-dimension vector per token of each document, and the document-level row you are used to does not exist.

Why the interaction is called late#

The name describes when the query meets the document, and it is a position on a scale with two established ends. A dual encoder has no interaction at all: the document is encoded without ever seeing a query, the query is encoded without seeing a document, and the only thing that ever combines them is a dot product between two finished points. A cross-encoder has early interaction: the query and the document are concatenated into one input and every transformer layer attends across the boundary, so the model can notice that the number in the query matches the number in the third sentence of the document. The accuracy ordering between those two is not controversial, and neither is the cost ordering, which is why the standard architecture puts one in front of the other.

Late interaction takes the position in between. Encoding stays independent, so document vectors are computed once at index time exactly as in a dual encoder, and the interaction is deferred to a scoring function that runs over the two sets of finished vectors. Khattab and Zaharia's 2020 ColBERT paper made the case that this ordering preserves most of what makes cross-attention accurate while keeping the property that makes dual encoders servable, and ColBERTv2 in 2021 made it practical by attacking the storage directly with residual compression and by training with distillation from a cross-encoder. The lineage matters when you are choosing which paper's numbers to believe: results reported for the 2020 model describe an index that is several times larger than what the 2021 engine produces for the same corpus.

The consequence of encoding independently is the one that shows up in your ingest pipeline, and it follows in two steps. Because the document side never sees a query, document encoding can happen once, offline, at a cost proportional to the number of tokens in the collection rather than to the number of searches. And because the stored artefact is token vectors rather than document vectors, a change of checkpoint invalidates every one of them at once. There is no cheap partial migration and no way to re-pool what you already have, so a model upgrade rewrites every byte of a terabyte-class index. Teams that treat their embedding model as a thing they swap quarterly should price that in before the first index build, since the rebuild window is the operational cost that surprises people after the storage cost has already been accepted.

What the encoder is fed is not quite the text#

Two preprocessing details change the token counts enough to matter when you are estimating, and both are visible in the config rather than in the model.

Queries and documents are encoded by the same weights but marked differently. A special [Q] token is prepended to queries and a [D] token to documents, which lets one set of weights learn two behaviours. The query is then padded to exactly query_maxlen positions with [MASK] tokens rather than being padded with ignored padding, and those mask positions are encoded and kept. This is query augmentation, and it means a three-word query costs the same 32 vectors as a twenty-word one, with the mask positions acting as learned slots that latch onto plausible expansions of the query. On the document side, the tokeniser output is filtered to drop punctuation before storage, which removes roughly 5-10% of positions on prose and rather more on code-heavy text.

from colbert.infra import ColBERTConfig

config = ColBERTConfig(
    dim=128,           # projection width; 128 is the released default
    doc_maxlen=256,    # tokens kept per document — the rest is discarded
    query_maxlen=32,   # queries are padded to exactly this many vectors
    nbits=2,           # residual bits per dimension in the compressed index
    index_path="/mnt/idx/docs.v1",
)

doc_maxlen deserves a note now and a chapter later. The original ColBERT repository ships a default of 180 tokens and RAGatouille raises it to 256, which means that on an unconfigured index a 2,000-token document contributes 256 vectors and the remaining 1,744 tokens are never encoded at all. That truncation is what keeps a naive index from being as large as the arithmetic above suggests, and it does so by silently discarding most of the evidence in exactly the documents where per-token matching would have helped most. For now, treat 256 bytes per stored token as the unit and remember that the number of stored tokens is a configured quantity rather than a property of your corpus.

The vectors are not the only thing an index of this shape has to hold. Every stored token vector needs a route back to the document that owns it, which in practice is a contiguous layout plus a per-document offset table rather than a four-byte document id attached to each vector; the layout choice is worth 16 GB on a four-billion-vector index, so it is not a detail the implementation gets to be careless about. There is also whatever structure supports approximate search over the token vectors themselves, since a query has to find candidate tokens among billions of them, and in ColBERTv2 that structure is a k-means codebook over the token embedding space with the vectors stored as compressed residuals against their assigned centroid. The centroid list is small and hot, the residuals are large and cold, and the split between them is what makes the index servable at all.

The neighbours this gets confused with#

"Multi-vector retrieval" is used loosely enough that two teams can agree to adopt it and build different systems. The specific claim of late interaction is that the multiple vectors are token vectors from a single encoder pass, and that they are combined with the query's token vectors at scoring time rather than at indexing time. Several other things also produce more than one vector per document and none of them behave like this.

Read that last column as gaps rather than as levels and two of the rows turn out to be the same formula. The chunk-level row stores four vectors of the same 768-d float32 shape as the single-vector row, and its index is four times the single-vector one — 369 MB multiplied by four is the ~1.5 GB in the table, with nothing left over. The late-interaction row stores 420 vectors of a shape that is twelve times smaller, and its index is 369 MB multiplied by 35, which is the 12.9 GB in the table, again with nothing left over. Both rows are the count of stored vectors times the size of one, and the only difference between them is that late interaction gets its multiplier divided by twelve on the way through while chunk-level multi-vector pays it in full. That the two products land exactly on the printed figures also says something about what the printed figures contain: they are token-vector payload and nothing else, with no allowance in them for the offset table, the codebook or the search structure described above.

Chunk-level multi-vector is the closest impostor, because splitting a document into four passages and storing four embeddings genuinely does recover some of what pooling loses. It differs in resolution and in cost by two orders of magnitude, and it needs no new index machinery, which makes it the sensible thing to try first. Learned sparse retrieval attacks the same weakness from the other direction, keeping term-level evidence but expressing it as weights over a vocabulary that an ordinary inverted index can serve. The cross-encoder you already met as a reranker sits at the opposite extreme: it indexes nothing and computes everything at query time, which is why it is accurate and why it can only see a few dozen candidates. Its 0 B is not a saving; it is the same cost moved from index time to query time, where it is paid once per candidate per query rather than once per document ever.

Neither a compression scheme nor a drop-in encoder#

Late interaction is not a compression technique, despite the 128-dimension projection that makes it look like one. Nothing about it reduces what you store relative to a single-vector index; the projection only limits how badly the token multiplier hurts. The sentence a reader is likely to write after meeting the projection — that ColBERT stores its vectors in a compressed form, so the index is smaller than a dense one — inverts the direction of the effect, and the table above is the correction.

It is likewise not an alternative encoder you can drop into an existing index. The token vectors need their own approximate-nearest-neighbour structure, their own mapping from token back to document, and their own scoring path, none of which your current vector database is necessarily willing to provide. A migration here is a migration of the storage layer and the query path together, which is a different size of project from swapping a model name in a config.

There is a regime where the multiplier stops dominating, and it is worth locating precisely rather than dismissing. The ratio is the stored token count over twelve, so a corpus of very short chunks pushes it towards one, and a corpus whose stored token count is set by a low doc_maxlen rather than by the documents themselves has already been capped without anybody deciding to cap it. Both cases make the storage argument mild. Neither makes it disappear, and the second one buys its mildness by throwing away the tail of every long document — which is the material late interaction was adopted to reach.

Keeping 420 vectors where one would do is only defensible if the retained detail buys retrieval quality, and everything above establishes the bill without the benefit. The benefit lives entirely in the scoring function, because token vectors are useless unless something compares them to the query's token vectors in a way that a single pooled dot product cannot reproduce.

That function is MaxSim, and it is where the argument for spending 35x on storage either holds or falls apart. It is also the only place the spending can be justified, since nothing else in the system does anything with a token vector that a pooled vector could not have done more cheaply.