Long documents are where the footprint and the tail latency actually live

A late-interaction index stores one vector per token rather than one per document, and scores a query against a document with MaxSim: for each query token it finds that token's best match among the document's stored vectors, and adds those best matches up. Both halves of that sentence are proportional to document length. A document twice as long occupies twice the index and costs twice as much to score, which is a property no other component of a retrieval stack has. In a single-vector index a document is one vector whatever its length, and in a lexical index a document is a set of postings whose cost falls on the terms rather than on the document.

That proportionality turns document length from a content property into an operational one, and it does so through a statistic that capacity plans almost never carry. The plan is built from a mean, and a mean over a corpus — the whole set of documents that have been encoded and stored — is the total token count divided by the document count. Two totals, one quotient. This chapter is about what that quotient cannot see, and about doc_maxlen, the per-document token ceiling a late-interaction indexer applies while encoding, which is the setting through which the invisible part does its damage.

Two populations under one average#

Start with the arithmetic that closes, because it is what licenses everything else read off this table. The Documents column adds to exactly 5,000,000 with nothing left over, and the shares beside it add to exactly a hundred percent. The Total tokens column adds to exactly 4,000M, and its shares add to exactly a hundred percent as well. No bucket is a rounding remainder and no document is counted twice, so every proportion below is a statement about the whole corpus rather than about a sample of it.

The mean is the next thing to derive rather than to read. Dividing the total token count by the total document count — 4,000M over 5,000,000 — gives exactly the 800 in the bottom row. That is not a fifth measurement. It is the two column totals put over one another, which means the mean is a function of those two numbers and of nothing else in the table. Move a billion tokens out of the short articles and into the manuals, leaving both totals alone, and the mean does not shift by a single token. The statistic is blind to the distribution by construction, which makes the average the wrong statistic for this job in a way that measuring it more carefully cannot repair.

Reading the table as gaps rather than as levels shows what the mean is hiding. Step from each row to the one below it and two things move at once. Mean length rises by a factor of a little under five, then a little over five, then a little over nine. Document count falls by a factor of five, then about four and a half, then nine. Each step multiplies the length by roughly the same factor it divides the count by, so the product — which is the total token count for that band — barely moves. That is why the last column reads 23, 22, 27 and 28 instead of collapsing toward one end: four bands of wildly different character each hold about a quarter of the token mass, because rarity and length cancel.

Once the bands are read as near-equal contributors, the headline follows by addition. Four tenths of one percent of the documents hold 28% of the token vectors, and the top 4% hold 55% of them. The second figure is the last two rows of the table added together and nothing more: 3.6% plus 0.4% of documents, 27% plus 28% of tokens. That it is a sum rather than a fresh measurement is what makes it safe to act on — there is no third population hiding behind it, because the shares already added to a hundred.

Nothing about this shape is exotic. It is what nearly every real corpus looks like when generated short content is mixed with imported source documents, and it is invisible in the one number capacity plans are built from. The same distribution under a single-vector index is entirely harmless, and the reason is worth stating as a chain rather than as a fact. A single-vector index stores one fixed-width vector per document, so a 56,000-token manual and a 230-token article both cost 3,072 bytes. Scoring compares the query vector against one document vector, so both cost one dot product. Neither storage nor compute carries a length term, so the length distribution never reaches either of them. Late interaction is the first component in the stack whose cost is proportional to document length, and therefore the first one for which that distribution is an operational property rather than a content property.

The stumble worth naming here is the belief that a mean tested against reality is a mean that can be trusted. A team that sizes on 800 tokens, ships, and sees the index land near the estimate has evidence that its planning number works — and the evidence is worthless, because the mean would have produced the same prediction for a corpus of five million identical 800-token documents and for this one. A defence that returns the same answer whatever the dangerous property does is not discriminating between the two worlds it is meant to separate. What discriminates is a percentile.

The tokens that were never encoded#

doc_maxlen is where the distribution first draws blood, and it does so quietly. A late-interaction indexer encodes at most that many tokens of any document; everything past the ceiling is dropped before encoding, and the evidence it carried has disappeared from the index without appearing anywhere else. At the RAGatouille default of 256, every manual contributed 256 token vectors — the title page, the copyright notice and the first page of the table of contents — and the remaining 55,744 tokens were never encoded.

This is the failure mode worth naming precisely, because it produces no signal anywhere. The ingest succeeds, the document count is right, the index is smaller than the estimate rather than larger, and the only symptom is that a whole class of queries cannot be answered by documents that plainly contain the answer. It also inverts the sizing arithmetic: with the cap applied, the stored token count is sum(min(length, 256)), which for this corpus is 1.23 billion vectors rather than 4 billion, so the index came in at 44 GB compressed instead of 144 GB and the under-run looked like good news.

Work through what that min does to each band and the inversion stops being a surprise. A document shorter than the cap contributes its own length, so a band sitting well below the ceiling contributes what it always did. A document longer than the cap contributes exactly the cap, whatever its true length, so a band whose documents are nearly all longer than the ceiling contributes its document count times the cap and not one vector more. Three of the four bands here have a mean above 256, and the manual band is the extreme case: its per-document contribution shrinks by more than two orders of magnitude, from tens of thousands of vectors to a few hundred. The rarest and longest documents, which dominate the uncapped estimate, are exactly the ones the cap erases hardest.

An index size is also a vector count in disguise, which makes the under-run readable without any instrumentation at all. Divide 144 GB by four billion vectors and divide 44 GB by 1.23 billion, and the two quotients agree — the same fixed number of bytes per stored vector, since compression is applied per vector and knows nothing about which document a vector came from. So a figure in gigabytes converts straight back into a figure in vectors, and any gap between the built index and the formula is a gap in what was encoded.

The interaction with the sizing estimate of the previous chapter is worth stating plainly. An index that is smaller than your formula predicted is not a saving; it is a truncation report. The first check on any late-interaction index build is whether sum(min(length, doc_maxlen)) equals sum(length), and the fraction of documents where it does not.

The tempting defence at this point is operational silence, and it is the weakest evidence available. Nobody was paged, no ingest job failed, no error counter moved — but that is precisely what the diagram above predicts, because truncation is a branch that discards and continues. A world where every manual lost fifty-five thousand tokens and a world where none did produce identical logs, identical document counts and identical dashboards. Absence of a report is compatible with both, so it separates them not at all, and the only thing that does separate them is the comparison of two sums that nothing computes unless you ask for it.

Instrument the two halves and join them#

Two measurements make this diagnosable, and they live in different places. At index time, record the true and stored token counts per document. At query time, record the counters introduced in chapter 3, where a late-interaction query was broken into a candidate stage and a rescoring stage — in particular vectors_rescored, which is the quantity that predicts latency.

-- index time: one row per document, written alongside the index build
INSERT INTO doc_index_stats (doc_id, true_tokens, stored_tokens, truncated)
VALUES ($1, $2, $3, $4);

-- query time: join the trace back to the documents it paid for
SELECT
  q.query_id,
  q.rescored_docs,
  q.vectors_rescored,
  q.latency_ms,
  MAX(d.stored_tokens)        AS longest_candidate,
  SUM(d.stored_tokens > 4000) AS long_candidates
FROM query_traces q
JOIN query_candidates c ON c.query_id = q.query_id
JOIN doc_index_stats d  ON d.doc_id   = c.doc_id
GROUP BY q.query_id
ORDER BY q.latency_ms DESC
LIMIT 50;

Why vectors_rescored rather than rescored_docs is the counter that matters follows from what the rescoring stage actually does. MaxSim compares each query token against each stored token vector of a candidate, so the work for one candidate is proportional to that candidate's stored token count. Summing over candidates gives total work proportional to the sum of their stored token counts, which is the definition of vectors_rescored. The document count appears nowhere in that product. It is a budget only under the assumption that documents cost the same as one another, and the table above is a table of a corpus where they do not.

The join is what turns "p95 is 640 ms" into "p95 is 640 ms on the 3% of queries whose candidate set contains at least one document over 4,000 tokens". Running it after doc_maxlen was raised to 8,192 to fix the quality problem produced exactly that: the median query rescored 1,024 documents and touched 254,000 vectors, while the slowest fifty rescored the same 1,024 documents and touched 2.9 million, because between eleven and thirty of those candidates were manual fragments at the 8,192-token cap. Same candidate budget, eleven times the bytes.

Both of those figures are worth dividing before moving on. Spread 254,000 vectors over 1,024 candidates and the average candidate is a couple of hundred tokens long, which is the short-article band and exactly what the corpus profile predicts a typical query to pull in. Spread 2.9 million over the same 1,024 and the average candidate is thousands of tokens long instead. Thirty documents pinned at the 8,192-token cap come to roughly a quarter of a million vectors between them, which is a floor on the slow queries' work rather than the whole of it; the rest arrives from the wider population the long_candidates column exists to count, everything past 4,000 tokens. Either way the candidate budget was met and the work was an order of magnitude apart, which is the failure of rescored_docs as a budget stated as a measurement.

Long documents compound the problem from both directions. Chapter 2, where MaxSim's mild length bias was measured, noted that more tokens means more chances to win each query token's maximum, so long documents are over-represented in the candidate set relative to their share of the corpus — and each one of them is also the most expensive thing in it. A candidate budget expressed in documents is therefore not a budget at all on a skewed corpus, which is why vectors_rescored rather than rescored_docs is the counter to alert on.

Reproducing it on purpose#

The regression this chapter is built on was discovered in production, which is the expensive way to find it. It reproduces deliberately in about a day, and the reproduction is worth having because it gives you the curve rather than a single point.

# same corpus, same query set, same candidate budget; only the cap moves
for maxlen in 256 1024 2048 4096 8192; do
  colbert index --doc-maxlen "$maxlen" --nbits 2 \
                --collection corpus.tsv --index "idx_$maxlen"
  colbert search --index "idx_$maxlen" --queries eval_400.tsv --ndocs 1024 \
                 --trace-out "trace_$maxlen.jsonl"
done

Hold ndocs fixed across the runs so that the only variable is tokens per document, and evaluate on two query sets rather than one: a general set that the short articles serve, and a set of forty or so queries whose answers are known to sit deep inside long documents. The general set will show flat quality and rising latency across the sweep, since its candidates are short at every setting and the extra vectors are pure cost. The long-document set will show quality climbing steeply from 256 to about 2,048 and then flattening, because most of the answers were in the first few thousand tokens. The gap between where quality stops improving and where latency keeps growing is the setting you want, and it is corpus-specific enough that copying someone else's doc_maxlen is guesswork.

The same sweep gives you the storage curve for free, since each index reports its own size, and those five points are the empirical version of the sum(min(length, doc_maxlen)) calculation. Where the measured size falls short of the formula, the difference is content you are not indexing.

Length distribution as a production input#

The operational lesson generalises past this one incident, and it reduces to a single instruction. Watch the distribution rather than the mean. A corpus's token-length distribution is a production input to a late-interaction system in the way that request size is a production input to an HTTP service, so it belongs on a dashboard rather than in a design document. The useful panels are the p50, p95, p99 and maximum of true_tokens per ingest batch, the fraction of documents hitting the cap, and the running total of stored vectors, all sliced by source. Sliced by source is the part that catches things early: a mean that moves from 800 to 830 after an import is unremarkable, while a new source whose p99 is 40,000 is the entire future of your p99 latency arriving in one commit.

An alert on the truncation fraction is worth more than an alert on index size. Index size going up is visible in the bill and gets attention; index size failing to go up as fast as the document count is the signature of evidence being discarded, and nothing else in the stack reports it.

Four ways to bound it#

Raising the cap fixes the quality and buys the latency problem. Adding a per-query vector budget — stop admitting candidates once vectors_rescored would exceed, say, 400,000, taking them in centroid-score order — bounds the tail at a small and measurable quality cost, and it is the right emergency patch because it ships in an afternoon and cannot make anything worse than a timeout. It leaves the underlying situation intact: a manual is still one document, so retrieving it tells the reader that the answer is somewhere in 56,000 tokens.

Chunking is the durable fix and it changes the shape of the corpus rather than the settings. Splitting the manuals into 300-token windows with 64 tokens of overlap turns 20,000 documents into 3.7 million, so the corpus grows to 8.7 million documents while the token count grows only by the overlap ratio, about 27%. Every candidate now costs at most 300 vectors, which removes the variance that was producing the tail; the retrieved unit is a passage a reader can act on; and the truncation cap becomes irrelevant because nothing reaches it. The index is larger than the truncated one by design, since it is now storing the content that was previously being thrown away.

Both figures in that paragraph are derivable, and deriving them is the difference between accepting the option and being able to price a different one. A window of 300 tokens that advances by all but 64 tokens each time stores those 64 twice, so the stored token count exceeds the original by the ratio of the repeated tokens to the fresh content each window brings — the 27% quoted. Applying that growth to the manual band alone, since only the manuals are being split, and leaving the other three bands untouched, gives a total that converts at the same bytes-per-vector rate into the 155 GB in the last row. Read the other way round, the difference between 155 GB and the 144 GB of a full untruncated index is the duplicated overlap and nothing else. That identity is what licenses treating overlap as a dial: halve it and you can predict the new footprint without building anything.

Chunking has a cost that is easy to miss and directly relevant to the rest of this topic. A 300-token chunk about one subsection has very little internal topic mixing, which means pooling it into a single vector discards much less per-term evidence than pooling a 56,000-token manual would have — so the same operation that bounds the latency also shrinks the advantage late interaction was adopted for. Chunk small enough and you have rebuilt the corpus into the shape where a single-vector index performs well. That tension is real and it does not have a general resolution; what it has is a measurement, which is whether the chunked late-interaction index still beats the chunked single-vector one.

Facts that sit at opposite ends of a manual#

Bounding the tail is not the same as making long documents work. A query whose answer requires two facts from opposite ends of a manual is served no better by 187 independent chunks than by a truncated document, because MaxSim scores one candidate at a time and no chunk contains both facts; that is a job for whatever composes the retrieved passages afterwards. Overlap reduces the chance of splitting a single fact across a boundary and does nothing for facts that are genuinely far apart.

That chunk count is itself a division worth doing, because it says how far the problem has been moved rather than solved. A 56,000-token manual cut into 300-token windows is 187 of them, so a reader who would previously have been handed one document containing the answer somewhere is now handed one window containing part of it. The retrieved unit got small enough to act on and the number of units that have to be assembled to answer a two-fact question went up by the same factor.

The token budget is also not a quality control. It caps the work per query by dropping candidates that the centroid stage ranked lowest, which is a reasonable heuristic and an unprincipled one, and on the 3% of queries that hit the cap it changes the result set in a way no offline evaluation will see unless the evaluation harness replays production candidate counts. The evaluation runs on a fixed candidate list; production runs on a list that gets truncated under load. Those are different systems, and only one of them is being measured.

With the distribution measured, the truncation closed and the tail bounded, the system finally produces numbers that mean something across runs. What every one of those numbers depends on is the shape of the corpus that produced them: the 800-token mean is a quotient of two totals, the 44 GB was a truncation report, the 640 ms belonged to 3% of traffic, and each became legible only once the distribution behind it was on a dashboard.

The question those numbers exist to answer has not been asked yet, because comparing a tuned late-interaction index against the naive single-vector baseline it replaced is not the comparison anyone should accept.