A chunk-size sweep separates truncation damage from chunking effects

Two positions get stated in the same meeting. One is that chunk size should come down to 384 tokens because the model's limit is 512 and the current 1,200-token chunks are obviously being clipped. The other is that chunk size is a red herring, that the team has never seen a single error from the embedding client, and that the real problem is the retriever, or the query rewriting, or the mix of documents in their own corpus. Both positions are arguments. The way to end the meeting is to convert them into a prediction and a measurement.

That measurement is a chunk-size sweep: index one body of text twice, once with chunks comfortably under the model's input limit and once comfortably over it, evaluate both the same way, and read the difference. Two separate defects are in play and telling them apart is the job. Truncation is text that never reached the encoder at all, because a chunk arrived longer than the limit and the client silently clipped it. Dilution is the opposite situation, in which every token did reach the encoder and the single vector that comes back has to stand for so much material that it sits close to nothing in particular. Both defects make long documents retrieve worse than short ones, and both are partly cured by making chunks smaller, which is exactly why a careless sweep produces a delta nobody can attribute.

The cheap measurement that comes first#

Before building any arms — an arm being one complete configuration of the indexing pipeline, built and evaluated end to end — two numbers are worth having, because they take twenty minutes and they can make the sweep unnecessary. The first is the effective limit, measured against the deployed path rather than read off the model card, which is what the boundary sweep (chapter 2, where an input is grown a little at a time until the returned vector stops changing) produces. The second is the distribution of token counts over the chunks already in the index, tokenized with the model's own tokenizer, split by the length of the source document.

For 1,200-token target chunks against a 512-token limit, the prediction is specific: the over-limit fraction is high and it is concentrated. Something like 75 to 85 percent of chunks exceed the limit overall, approaching 100 percent for documents long enough to produce full-size chunks, and near zero for documents shorter than the target size.

Concentrated rather than spread evenly is the surprising half of that, and it follows from how a fixed-target chunker divides a document. The chunker walks a document and emits pieces at the target size, then emits whatever is left over at the end as a final short piece. A document shorter than the target therefore produces exactly one chunk, that chunk is shorter than the target, and a chunk shorter than the target is under the limit here — so it can never be clipped, whatever the client does. A document long enough to produce full-size chunks emits pieces sitting at the target, and the target is more than twice the limit, so almost every one of those pieces is clipped. The population of chunks splits by the length of the source document rather than smearing across it, and that split is the whole explanation for "good on short documents, poor on long ones". It arrives before any evaluation runs.

The other position in the meeting deserves its strongest form, because it is the belief most likely to survive this chapter: we have never seen a single error from the embedding client. The belief underneath it is that an absence of errors is a defence, and it is tempting for an ordinary reason — across most of a system, a fault that is present announces itself. Clipping does not announce itself. The client shortens the input, returns a well-formed vector of the right dimensionality, and logs nothing, so the error count is zero when every chunk is clipped and zero when none is. A check whose output stays constant while the thing it must discriminate varies cannot discriminate it, and silence is precisely what clipping produces. The absence of errors is evidence about the library's defaults, and it is not evidence about the data.

If the over-limit fraction comes back near zero, the sweep is the wrong experiment and the gap is dilution or something else entirely. If it comes back at 78 percent, the sweep is worth running to size the effect rather than to establish it.

What has to be held still#

Changing the target chunk size changes the number of chunks, and that alone breaks three things if you let it.

The first is the meaning of recall@k — the fraction of evaluation queries for which at least one item judged relevant appears among the top k results the retriever returns. With 1,024-token chunks a corpus of 6,200 documents produces 22,400 chunks; at 384 tokens it produces 58,900. Retrieving the top 10 chunks therefore retrieves a smaller fraction of that corpus in the second configuration. It also retrieves fewer total tokens, and the multiplication that shows this is the one worth doing by hand: ten chunks at 384 tokens each is 3,840 tokens of evidence, while ten chunks at 1,024 tokens each is 10,240. A raw comparison of chunk-level recall@10 is comparing two piles of evidence that differ by that ratio, and any difference in recall could be the ratio talking.

Two separate repairs are needed, one for each half of the damage. Scoring at the document level fixes the first half: mark a query as recalled if any retrieved chunk belongs to a document judged relevant, so a hit counts once whichever chunk of that document produced it, and the units become comparable across arms. Matching the token budget rather than k fixes the second half, and the matching is the same multiplication run backwards. Ten chunks at 1,024 tokens each is the 10,240 above; twenty-seven chunks at 384 tokens each lands within a rounding of that same figure. That is why k=27 is the right partner for k=10 rather than some rounder number, and reporting both — fixed k and fixed token budget — costs one extra run and pre-empts the objection.

The second is overlap, meaning the tokens each chunk repeats from the end of the chunk before it, so that a passage cut in half at a boundary still appears whole somewhere. An overlap expressed in absolute tokens is a different fraction of a 384-token chunk than of a 1,024-token chunk, so holding it at "200 tokens" quietly triples the redundancy in the small-chunk arm. Express it as a fraction of chunk size and hold the fraction fixed.

The third is corpus coverage. If the small-chunk arm indexes documents the large-chunk arm dropped, or the extraction pipeline was re-run between the two, the delta includes changes that have nothing to do with size. The safe procedure is to snapshot the extracted text once and chunk both arms from the same snapshot, in the same run, with the same code.

The query set, the relevance judgements — the recorded labels saying which documents count as answering which query — the embedding model and version, the vector index parameters, and the similarity function all stay fixed for the same reason.

The prediction written down before the run#

Writing down the expected direction and rough magnitude before running is what turns the sweep into evidence rather than a story assembled afterwards. For the configuration above — 1,200-token chunks, a 512-token limit, long-document recall visibly poor — a reasonable prediction is that short-document recall is essentially unchanged, because short documents produce chunks under the limit in both arms and their chunking barely differs, and that long-document recall improves by 15 to 30 points if truncation is the dominant defect and by 3 to 5 points if it is not. Those two ranges do not touch, and the space between them is wide, so whichever one the result lands in the result has discriminated. A pair of ranges that overlapped would be unfalsifiable in the direction that matters and would teach nothing.

Getting it wrong is informative. A prediction of +20 that comes back as +4 says truncation was a real but minor contributor and something else dominates; a prediction of +20 that comes back as +35 usually means the over-limit fraction was higher than assumed and worth checking against the token counts.

Three arms, not two#

Two arms answer "does sizing under the limit help", and they cannot separate truncation from every other effect of chunk size, because the small-chunk arm changed both at once. The chain is worth walking. Making chunks smaller removes the clipping, and making chunks smaller also narrows what each vector has to summarise. Both of those push long-document recall in the same direction. A single difference between two arms therefore carries both effects added together, and nothing inside those two numbers can split the sum. A third arm splits it by moving one cause and holding the other: keep the 1,024-token chunks exactly as they are and embed them with a long-context encoder whose limit is 8,192, so the chunking is identical to arm A and the truncation is gone.

ARMS = [
    Arm("A", target_tokens=1024, overlap_frac=0.10, model="bge-base-en-v1.5"),
    Arm("B", target_tokens=384,  overlap_frac=0.10, model="bge-base-en-v1.5"),
    Arm("C", target_tokens=1024, overlap_frac=0.10, model="bge-m3"),  # 8192 limit
]

snapshot = load_extraction_snapshot("2026-03-18")   # identical text for every arm

for arm in ARMS:
    chunks = chunk_by_tokens(snapshot, arm.target_tokens,
                             overlap=int(arm.target_tokens * arm.overlap_frac),
                             tokenizer=tokenizer_for(arm.model))
    index = build_index(embed(chunks, arm.model))
    for k in (10, 27):
        report(arm, k, evaluate(index, QUERY_SET, JUDGEMENTS, level="document"))

Arm C is not a free comparison, because it changes the model, and topic 8.6 — on choosing an embedding model — established what that entails: a different training mixture, different prefixes, a different dimensionality, and quality differences unrelated to sequence length. The way to keep it interpretable is to read arm C's short-document recall as a calibration. Short documents were never clipped in any arm, so any movement in their recall between arm A and arm C is the model swap and nothing else. If it matches arm A's short-document recall within noise, the model swap is roughly neutral on this corpus, and the long-document movement can be read as the truncation effect.

Attributing the movement, and the gap it leaves#

The three arms decompose that 0.22 exactly, and the exactness is the whole reason for running three of them. Arm C's long-document recall sits partway between the other two. Arm C minus arm A is 0.14, and since arm C's chunking is byte-identical to arm A's while only the clipping is gone, that 0.14 is truncation damage and nothing else. Arm B minus arm C is 0.08, and since arm B differs from arm C by having smaller chunks, that 0.08 is the chunking change itself, which is dilution reduced. Fourteen hundredths plus eight hundredths is twenty-two hundredths, which is the whole arm-A-to-arm-B movement with no residual row left over.

What that closure licenses is worth stating plainly, because it is the answer to "why not just run two arms". The improvement is fully accounted for by two named causes, so no third unnamed cause is hiding inside the delta. Had the two parts summed to less than the whole, something else would be doing work — a lucky interaction, a difference in how the two indexes were built, an unnoticed change in the query path — and the attribution would be a story rather than a measurement. A decomposition that closes is what converts an argument in a meeting into a number somebody else can check.

The gap that survives all three arms is the one the sweep does not close, and its figure repays a second look. Arm B's long documents still sit 0.24 below where short documents sit, which invites a correction, since arm B's own short-document recall is 0.85 and its long-document recall is 0.63, a difference of 0.22. The 0.24 is defensible. The short-document level for this corpus is 0.87, which is what the arm measuring that population best reports, and short-document recall moves by at most 0.02 across all three arms — a movement that the confidence interval computed in the next section places inside the noise band. To the precision this experiment supports, 0.22 and 0.24 are the same number, and picking between them is reading the table past its real precision. Whichever of the two you write down, the quantity is dilution that smaller chunks did not remove.

The noise floor on a 200-query evaluation#

Every figure above is a proportion measured on a sample, so the first question about any difference is whether the sample was large enough to see it. With 200 queries and document-level binary recall, each query is one outcome that lands either as a hit or as a miss, and an arm's recall is the share that landed as hits. The uncertainty on a proportion has a standard form, and the form is worth deriving in words rather than quoting. A single yes-or-no draw varies most when the two outcomes are near equally likely and varies less as the proportion approaches either extreme, and the product of the proportion with its complement is the quantity that behaves that way. Averaging over independent draws divides that quantity by the number of draws. Taking the square root converts it back from a squared scale onto the scale of the proportion itself. So the standard error is the square root of 0.6 multiplied by its complement, divided by the 200 queries.

Carrying that through for a proportion near 0.6 gives about 0.035, and roughly two standard errors either side of a figure is the 95 percent confidence interval, which comes to about ±0.07 on any single number in that table. That interval is what licenses reading the table at all. A difference of 0.02 sits well inside it and is indistinguishable from no difference, which is why the short-document row counts as a control rather than as a small effect. A difference of 0.22 sits many times outside it and cannot be produced by sampling variation on this query set. The honest reporting of the 0.14 attribution is that it carries a wide interval of its own — a bootstrap over the query set, resampling with replacement 1,000 times, is the cheap way to put a real interval on it rather than an assumed one.

The arms share a query set, so paired comparison is available and considerably tighter than treating them as independent samples. The reason it is tighter is that a query which is hard for one arm is usually hard for all of them, so most of the spread behind that ±0.07 is query-to-query difficulty rather than anything the arms did, and looking only at the queries whose outcome changed cancels that difficulty out. Score each query in both arms, count the queries that flipped in each direction, and test on those. Twenty-nine queries improved and four regressed in the A-to-B comparison, which is decisive at any threshold anyone will ask for.

Two points either side of the limit, not a curve#

A sweep built this way does not find the optimal chunk size. Two or three points do not describe a curve, and the arms were chosen to sit clearly on either side of the limit precisely so the delta would be attributable, which is a different design goal from optimisation. Sweeping 128, 192, 256, 384 and 512 to find a maximum is a legitimate follow-up experiment and it answers a question this one was not asked.

What three arms cost, and the sample that reduces it#

Nor does a sweep justify itself on cost in every situation. Three arms over 6,200 documents cost 157 minutes of index build plus three evaluation passes, and that total is itemised in the table's last row: 34 minutes for arm A, 71 for arm B and 52 for arm C add to exactly the 157, with no unattributed remainder. The figure is three index builds and nothing else concealed inside it. On a rented GPU that is a few dollars. On a hosted embedding API it is a real invoice, because 58,900 chunks at 341 tokens each — the chunk count and the mean chunk length from the same column of the table, multiplied together — is about 20 million tokens for arm B alone.

Re-embedding a ten-million-document corpus three times is not a decision made in a retro. Sampling is the answer at that scale: run the sweep on a stratified 5 percent of that corpus, sampled to preserve the mix of document lengths and source systems so the short and long slices stay comparable, with the full query set. Accept that absolute recall figures will be optimistic, since a smaller index holds fewer competing documents for a relevant one to outrank. Then read only the difference between arms, which is what the experiment was designed to produce and the part a smaller index leaves intact.

Query-side truncation, and a corpus that drifts#

A sweep of this shape says nothing about the query side. Both arms embed the same queries through the same path, so query-side truncation, if any, is present identically in both and cancels out of the comparison. A query long enough to be clipped would damage every arm equally and would leave the reported delta unchanged, so the sweep reports nothing about it either way. That is a limit of the design rather than a reassurance about queries, and a query path long enough to be at risk needs its own boundary sweep.

The result is also a snapshot. It holds for the body of documents that was indexed, at the composition it had on the day the extraction snapshot was taken, and a corpus that shifts towards longer documents or towards code will drift back over the limit without anyone re-running the evaluation. The over-limit fraction is a property of the interaction between document lengths and the chunk target, and both sides of that interaction move on their own schedule.

With those three variables held still, a sweep converts a meeting into a decomposition: a movement in long-document recall, split cleanly into the part truncation caused and the part chunk size caused, with a named residue that neither explains. The discipline is the unglamorous half — one extraction snapshot chunked by every arm, overlap expressed as a fraction, scoring at the document level, a token budget matched rather than a k matched, and the prediction written down before the run so that the result can contradict it.

A corpus that drifts is also the argument against treating evaluation as the detection mechanism. A sweep costs a couple of hours of engineering and an hour of compute, it answers the question once, and it will not tell you when the answer changes. The defect needs a continuous signal, which means emitting numbers at index time.