The token multiplier, not the dimension drop, sets the storage bill

Two ratios decide what a late-interaction index weighs, and only one of them is usually quoted. The first is the ratio between one stored vector and the one it replaces: a pooled vector holds 768 float32 values at 3,072 bytes, a ColBERT token vector holds 128 fp16 values at 256 bytes, and 3,072 divided by 256 is 12. The second is the ratio between the number of vectors on each side, which is one per document against one per stored token, so it is the average stored token count of a document. For a corpus of technical documentation that figure runs into the hundreds.

Those two ratios do not compete. One divides the bill and the other multiplies it, and when the multiplying one is two orders of magnitude larger than the dividing one, an estimate that omits it is not approximate. It is wrong in a direction and by an amount that no safety margin covers, which is why a capacity conversation about this architecture goes wrong in the same place almost every time.

The arithmetic, in full#

An index's byte count is one multiplication with four terms in it: how many documents, how many vectors each contributes, how many values in a vector, and how many bytes in a value. Nothing else belongs in the first estimate, and writing it as a function makes the single-vector case a special value of the same expression rather than a different formula.

def index_bytes(n_docs: int, tokens_per_doc: int, dim: int, bytes_per_value: float) -> int:
    """tokens_per_doc = 1 for a pooled single-vector index."""
    return n_docs * tokens_per_doc * dim * bytes_per_value

single = index_bytes(5_000_000, 1,   768, 4)     #  15,360,000,000  = 15.4 GB
multi   = index_bytes(5_000_000, 800, 128, 2)     # 1,024,000,000,000 = 1.02 TB

print(f"{multi / single:.0f}x")   # 67x

Follow the second call through. Five million documents at 800 stored tokens each is four billion vectors, which is the count. Each of those is 128 values at two bytes, which is 256 bytes. Four billion times 256 bytes is the 1.02 TB in the comment. The first call is the same expression with the token count set to one and a wider, heavier vector: five million times 768 times four bytes is 15.4 GB.

The ratio has a closed form worth carrying around, because it removes the need to redo the multiplication for every corpus you are asked about. With t tokens stored per document, d dimensions and b bytes per value on each side, the ratio is (t × d_multi × b_multi) / (d_single × b_single), which for the standard configurations is t × 256 / 3072, or t / 12. Divide the average stored token count by twelve and you have the multiple.

The document count cancels out of that expression, since it appears identically on both sides. That is the reason the ratio is a property of your documents rather than of how many of them you hold, and it is why the same multiple applies to a pilot and to the full corpus.

Check that against the two ratios the chapter opened with and they close on each other exactly. The ratio between the vector counts is 800, and the ratio between the vector sizes is 12. Eight hundred divided by twelve is the 67x that the code prints, with nothing left over. What that exactness licenses is a statement about what the 67x does not contain: no compression, no per-vector document id, no approximate-search structure, no filesystem overhead. It is two ratios and nothing else, so every one of those omitted terms is still to be added, and none of them can be the explanation when the figure surprises somebody. Eight hundred tokens gives 67x; 240-token chunks give 20x; 120-token chunks give 10x. The dimension drop from 768 to 128 is present in that formula as the constant 12, and its entire contribution is to divide the token count by twelve rather than to save anything on its own.

The ratio is flat down the document-count axis and linear along the token axis, which is the single most useful thing to know when someone asks whether this scales. Doubling the corpus doubles both indexes and changes nothing about the decision. Doubling the length of your documents doubles only one of them. The last row is the cleanest demonstration: ten times the documents at the same length moves both columns by a factor of ten and leaves the ratio on 67x, unchanged, while the row above it holds the document count still and moves the ratio by a factor of two and a half.

That flatness is what makes the dimension drop such a persuasive wrong answer. A team reasoning from "12x smaller per vector" has picked the one term in the formula that cannot vary with their corpus — the projection width is fixed by the checkpoint, identical for a corpus of tweets and a corpus of legal opinions. The term that does vary, and varies over more than an order of magnitude between the top and bottom rows of that table, is the one the argument leaves out. A defence that reads the same number whatever the dangerous property is doing is not a defence, and document length is the dangerous property here.

Where the token count comes from#

Since the whole estimate now hangs on t, it is worth being careful about where that number comes from. Document length is usually known in words or in characters, and subword tokenisers expand English prose by roughly 1.3 tokens per word, more on identifier-heavy technical text where EntityTooSmall becomes three pieces and a UUID becomes a dozen. A corpus described internally as "about 600 words per article" is an 800-token corpus — six hundred words at that expansion rate lands just under eight hundred tokens, which is exactly the gap between a comfortable estimate and an uncomfortable one. Punctuation filtering gives a few percent back, and doc_maxlen caps the figure from above, so the quantity that belongs in the formula is the number of vectors the indexer will actually write.

from transformers import AutoTokenizer
import numpy as np

tok = AutoTokenizer.from_pretrained("bert-base-uncased")
sample = load_random_documents(n=2000)          # a real sample, not the first 2000

lengths = np.array([len(tok(d.text).input_ids) for d in sample])
stored = np.minimum(lengths, 256)               # doc_maxlen

print(f"mean {lengths.mean():.0f}  p50 {np.percentile(lengths, 50):.0f} "
      f"p95 {np.percentile(lengths, 95):.0f}  p99 {np.percentile(lengths, 99):.0f}")
print(f"vectors written per doc at doc_maxlen=256: {stored.mean():.0f}")
print(f"truncated documents: {(lengths > 256).mean():.1%}")

Running that on a sample of two thousand documents takes a minute and replaces the most load-bearing assumption in the estimate with a measurement. The np.minimum is the line that matters: it is doc_maxlen applied in arithmetic exactly as the indexer applies it on disk, so the mean of stored is the number to put in the formula and the mean of lengths is not. Print the percentiles rather than only the mean; the reason is the subject of the next chapter, and the output of this snippet is the evidence for it.

Compression turns 67x into 9x, and takes a payment#

The terabyte figure describes an uncompressed fp16 index, and nobody serving late interaction at this scale stores one. ColBERTv2's contribution on the cost side was residual compression: the k-means codebook from chapter 3 gives every token vector an assigned centroid, and instead of the vector you store the centroid id plus the residual quantised to one or two bits per dimension, quantisation here meaning that each number is replaced by the nearest value from a very short list of allowed ones. At nbits=2 that is 128 dimensions × 2 bits = 32 bytes of residual plus roughly 4 bytes of centroid reference, so about 36 bytes where fp16 used 256.

Take the two residual rows and subtract one from the other. The difference is 16 bytes, which is 128 bits, which is exactly one bit for each of the 128 dimensions — so the whole of the gap between nbits=2 and nbits=1 is the second bit, and the centroid reference is identical in both. That is a consistency check on the roughly 4 bytes: it is a real, constant per-vector overhead rather than a fudge absorbing slack, because if it were absorbing slack the two rows would not differ by precisely the payload.

Read the top three rows as gaps rather than as levels and the shape of the lever appears. Halving the bytes per vector halves the multiple every time — 133x to 67x to 33x — with no threshold and no diminishing return, because nothing in the formula is nonlinear in the width of a value. And the quality column barely moves across those same two steps. The cheap part of the compression story is therefore exhausted before the residual rows begin: they buy a further factor of three and a half, then a further factor of under two, and each of those is the first purchase that costs measurable quality. The level to quote is the bottom of the range; the gaps are what tell you there is nothing left to find.

Two things in that table matter more than the individual rows. The best available compression leaves the index at five to nine times the single-vector one, so compression changes the decision from impossible to expensive rather than from expensive to free. And the codebook itself is negligible — 262,144 centroids at 128 fp16 dimensions is 67 MB — which means the memory that has to be hot is small even when the residuals are not, and that split is what makes a partly disk-resident index workable.

The quality column deserves the scepticism you would apply to any vendor number, starting with what it measures: MRR@10 is the mean reciprocal rank of the first relevant result within the top ten, so a first-place hit scores 1, a third-place hit scores a third, and a point of MRR is one hundredth of that average. Published residual-compression results are measured on MS MARCO passages — a large public web-search benchmark of real Bing queries — which average around 60 tokens, and quantisation error interacts with the MaxSim margin described in chapter 2 — a margin that was 0.56 out of 21.5 on the worked example. Perturbing every stored vector slightly perturbs exactly the quantity the ranking depends on, and there is no reason to assume the loss on your corpus matches the loss on a passage benchmark. Measure nbits=2 against fp16 on your own query set before you put the compressed number in a capacity plan.

The projection width is the one remaining lever, and the field has been pushing on it. Later checkpoints in this family ship narrower: some small ColBERT-style models project to 96 or 64 dimensions with a backbone an order of magnitude smaller than BERT-base, trading a little quality for an index that is a quarter of the size and an encoder that runs on CPU. The movement in the other direction is worth knowing about because it changes the arithmetic dramatically: ColPali and ColQwen apply the same late-interaction idea to document images, encoding a page as roughly a thousand patch vectors and skipping OCR — optical character recognition, the step that would otherwise turn the page image back into text — entirely. Weaviate's overview of the late-interaction family is a reasonable map of who is where. The point for a capacity plan is that t in the formula is a property of the model as much as of the corpus, and a page-image model has a t of about 1,030 whether the page holds ten words or a thousand.

What the bytes cost once they have to be served#

Storage price is the least interesting part of this, and quoting it is how estimates get waved through. A terabyte of gp3 block storage is about $80 a month, which sounds like nothing, and it would be nothing if the workload were archival. The workload from chapter 3 moves 29 MB of compressed residuals per query at ndocs=1024, in thousands of small random reads, at a p95 budget measured in tens of milliseconds. That is a memory-residency problem wearing a storage costume.

The comparison that means something is the serving footprint, and it turns on a threshold rather than on a ratio. A 15.4 GB single-vector index fits in RAM on a 32 GB instance with room for the process, so a three-replica deployment is three small machines. A 144 GB compressed late-interaction index does not fit on anything commodity: you are looking at a 256 GB memory instance per replica, or a local-NVMe instance where the residuals live on disk and the centroids and offsets live in RAM. On current on-demand pricing that is roughly $180 a month per replica for the single-vector service and $1,000-1,500 a month per replica for the late-interaction one, so a three-replica production deployment moves from about $550 a month to about $3,500. Adding the GPU for query encoding, which the single-vector service also needs, does not change the ordering.

Index construction is the part that turns out not to be a problem. Encoding four billion tokens at around 21,000 tokens per second per A10G is 53 GPU-hours, which is under seven hours on eight GPUs and roughly $60-130 of compute depending on instance type, plus k-means over a sample of the token vectors and a compression pass. That is a nuisance to schedule and cheap to buy. The recurring cost is residency, and the recurring risk is that every model change repeats the whole build against a live index.

The comparison the number licenses#

Having the right number does not settle anything by itself, and treating 67x as a verdict is its own mistake. A 9.4x compressed index on a corpus where late interaction genuinely fixes a class of failing queries can be an easy yes, particularly if the alternative is a cross-encoder pass that costs GPU seconds per query at serving time rather than dollars per month at rest. Storage is a fixed cost that scales with the corpus and can be planned; query-time model inference is a variable cost that scales with traffic and shows up in tail latency. Those two live in different budgets and fail in different ways, and a comparison that reduces both to a monthly figure has thrown away the property that distinguishes them.

What the number does settle is which comparison you are allowed to make. The single-vector index is not the thing to compare against, because a team that has 15.4 GB and a quality problem is not choosing between doing nothing and spending 144 GB — they are choosing between spending 144 GB and adding a reranker to the pipeline they already run, which costs no index bytes at all. Chapter 6 sets that comparison up properly.

There is also a term in the estimate that has been treated as solid and is not. Every figure here rests on "5M documents averaging 800 tokens", and an average is a summary of a distribution. If those 800 tokens come from documents that are all 700 to 900 tokens long, the terabyte is a terabyte. If they come from four million 200-token chunks and eight hundred thousand documents that run to twenty thousand tokens each, the same average describes a corpus whose bytes and whose tail latency are concentrated in a small fraction of the documents, and whose behaviour under a default doc_maxlen is not what either the arithmetic or the quality claim assumed.

The formula is indifferent to which of those two corpora it is handed, and that indifference is exactly its limit. t enters as a single number, so every estimate in this chapter is an estimate of a total, and a total is the one statistic that a heavy tail leaves undisturbed while it rearranges everything a service actually experiences.