A 128-token chunk cuts a procedure into steps that arrive alone

A procedure — a runbook, a checklist, any numbered sequence somebody has to follow in order — is written on the assumption that it will be read from the top. Step 6 does not restate what is being rotated, because step 1 already said it and prose does not repeat its antecedent. That assumption is invisible while a person reads the page, and it becomes load-bearing the moment the page is cut into fixed-size pieces and those pieces are ranked independently against a question.

Chunking is the step that does the cutting. Before anything is indexed, each document is split into units of a bounded size, and it is the unit rather than the document that gets a vector and a place in the ranking. The bound is usually written in tokens — the sub-word pieces a model counts text in — and 128 is a common choice, on the reasoning that a smaller unit is a purer match for a short question. The measurements below come from a dense index over a corpus of 312 documents, one key rotation runbook inside it, and one question a user actually asked.

Where the boundaries actually fall#

The key rotation runbook is 2,100 tokens long. Split at 128 tokens with no overlap, it becomes seventeen chunks, eleven of which carry a numbered step that the correct answer needs. That count is not a measurement. Dividing 2,100 by 128 gives sixteen units and a remainder, and a remainder still has to be stored, so the seventeenth chunk is the leftover. Nothing about the encoder, the query or the corpus enters that calculation: the number of units a document becomes is fixed by two design decisions — the budget and whether there is overlap — and is knowable before a single vector is computed.

Here is where each of those eleven lands in a corpus-wide ranking for the query "how do I rotate the signing key?", against a dense index over the whole 312-document corpus.

chunk  rank   cos    first line
  #7      1  0.834   "Step 8. Update the JWKS endpoint and wait for the verifiers…"
  #3      2  0.831   "Step 4. Drain the token cache before issuing tokens against…"
  #0      6  0.812   "Key rotation runbook. Applies to the production signing key…"
  #5     47  0.771   "Step 6. Confirm every verifier reports the new kid in its…"
  #2     88  0.762   "Step 3. Record the outgoing key id in the rotation ledger…"
  #9    121  0.755   "Step 10. Remove the retired key from the JWKS document once…"
  #4    174  0.748   "…set grace_seconds to at least twice the longest token TTL…"
  #1    203  0.744   "Step 2. Open a change record and attach the current kid and…"
  #8    266  0.739   "Step 9. Watch the verification error rate for one full cache…"
  #6    311  0.733   "…if the staging rotation produced any 401s, stop here and…"
  #10   402  0.728   "Step 11. Close the change record and note the new kid in the…"

Read the chunk column and the step numbers together before reading the ranks, because the two columns close on each other exactly. Chunk #1 opens step 2, #2 opens step 3, #3 opens step 4, #5 opens step 6, and the pattern holds all the way to #10 opening step 11: each chunk carries the material of the step one higher than its own index. Chunk #0 carries the heading and step 1 with it. Eleven chunks map onto eleven steps with nothing left over on either side, which is what makes the two exceptions worth staring at.

Two rows in that listing do not begin with a step number. Chunk #4 opens mid-sentence on grace_seconds, and chunk #6 opens mid-sentence on staging 401s. Under the mapping just established those two chunks carry steps 5 and 7, and the reason their first line is a fragment is that steps 5 and 7 began inside the chunk before them. The boundary landed in the middle of a step. That is not an anecdote about two unlucky rows; it is the budget doing exactly what a budget does, and it is visible in the artefact without running anything.

The indices also tell you where the gold material sits in the document. The eleven run from #0 to #10 with no gaps, a contiguous block from the start of the runbook through step 11, which leaves the remaining six chunks of the seventeen carrying nothing the answer needs. Contiguity is a property of this document rather than a general one, and it matters later: it is the precondition under which a fix based on neighbours can work at all.

Two of the eleven are in the top five, a third sits at rank 6, and the remaining eight are scattered between rank 47 and rank 402, which at k=20 means those eight are not in the candidate set, and no reranker can promote a candidate that was never returned. The scores that put them there sit between 0.728 and 0.771, a range that also contains several hundred unrelated chunks from elsewhere in the corpus, so there is no threshold that admits step 6 without also admitting the meeting minutes.

The listing turns that claim into a measurement on its own. From the chunk at rank 6 to the chunk at rank 47 the score falls by about four hundredths, and about forty ranks fit in that interval. From rank 47 to rank 402 the score falls by roughly the same four hundredths, and three hundred and fifty-five ranks fit in it, because the corpus piles up in the lower band. A score gap is therefore not convertible into a rank gap at any fixed exchange rate, and a threshold set on score has completely different selectivity depending on where it is set.

The belief this makes hardest to give up is that a high similarity score is a defence against an unusable answer, so a system whose retrieved chunks all score well is a system that retrieved well. Every one of the eleven gold chunks scores between 0.728 and 0.834 — a spread of about a tenth, tight enough that a person reading a score log would call all eleven good matches — while the property that decides the answer, whether the chunk is inside the candidate set, swings across four hundred ranks. The quantity on the dashboard barely moves. The quantity that matters moves by two orders of magnitude, and it is not on the dashboard.

Why the surrounding steps score lower#

The query contains three content words: rotate, signing, key. Chunk #7 contains "signing key" and "rotation" because step 8 happens to restate the goal. Chunk #3 mentions the token cache and tokens issued against the new key, close enough to score 0.831. Step 6, in chunk #5, says to confirm that every verifier reports the new kid in its metadata endpoint, and it does not contain the words rotate, signing or key at all, because the document already established what is being rotated in the title and in step 1 and does not repeat itself. Human prose is written with the antecedent stated once, and a chunker with no overlap and no title propagation throws that antecedent away for every chunk after the first.

The result is that each step is embedded as if it were a standalone document about whatever that step happens to mention. Step 4 becomes a document about token caches, step 6 about verifier metadata, step 9 about error rates. Their vectors land near other documents about token caches and error rates, which is the correct behaviour for the encoder — the embedding model that turns a chunk into the numeric summary the index stores — given what it was shown.

Follow that through one link at a time, because the conclusion is stronger than "the steps score lower". A fixed budget cuts wherever the count runs out, so the boundary lands mid-procedure, which the two fragment rows in the listing demonstrate. A step and the condition under which that step applies therefore end up in different chunks. A retriever scores each chunk against the query and has no representation of "these two belong together", so it has no reason to fetch the condition once it has fetched the action — the condition is not more similar to the query for being the condition of something similar. The model consequently receives an action with no condition attached. The fragment that arrives is not merely shorter than the step; it is unconditioned, and an unconditioned instruction reads as unconditional.

Chunk #4 is the clean case. It carries grace_seconds, which has to be at least twice the longest token TTL, and it sits at rank 174 — far outside the candidate set. The step that survives is step 8, which updates the JWKS endpoint and waits for the verifiers. An answer built from what arrived tells the reader to update the endpoint and wait, and says nothing about how long the grace period has to be, because the sentence that set it was scored on its own and scored as a document about token lifetimes. The similarity ordering for a procedural query selects the one or two steps that restate the topic and drops the preconditions, so the model receives the middle of a procedure with no way to know that anything preceded it.

An image helps here, and it is an analogy rather than a description of the mechanism. Reading the runbook through this index is like reading it through a slot that shows one paragraph at a time. The place the analogy breaks is the place that does the damage: a slot preserves the order of the paragraphs, and retrieval does not. The fragments arrive sorted by resemblance to the question, so the reader gets step 8 first and step 4 second and steps 5 and 6 not at all. The analogy understates the failure rather than illustrating it.

Smaller chunks were supposed to raise precision, and in the narrow sense they do: chunk #7 is a tighter match to the query than the full 2,100-token runbook would be, and it wastes fewer tokens. What was traded away is the property that a retrieved unit is self-sufficient. Precision at the chunk level and sufficiency at the answer level are different things, and 128 tokens optimises the first at the cost of the second on any corpus whose documents have internal dependencies.

Adding overlap is the reflex fix and it addresses a different problem. Sixty-four tokens of overlap on 128-token chunks grows the index by about 50% and rescues sentences that were cut in half at a boundary, which is a real defect and not this one. The material step 6 is missing sits four hundred tokens upstream, so no overlap short of half a chunk-length would carry it, and an overlap that large is a 512-token chunking with extra storage. That last equivalence is arithmetic rather than rhetoric, and it is the part to hold on to: a 128-token chunk that also has to carry four hundred tokens of preceding text is a window of a little over five hundred tokens, which is the 512-token chunking the sentence names. Overlap large enough to solve this problem has stopped being overlap and become a different chunk size. The cheaper move in the same family is to prefix every chunk with its document title and section heading before embedding, which restores the antecedent for a few tokens of cost; Anthropic's write-up "Introducing Contextual Retrieval" takes this further by generating a sentence of surrounding context per chunk at index time and reports a substantial drop in retrieval failures from that plus lexical search. Both belong in the list of candidate fixes, and neither is worth committing to before the mechanism is confirmed.

The smallest experiment that tests this#

The tempting experiment is to re-chunk the corpus at 512 tokens, re-embed everything, and compare answer quality before and after. It is the largest experiment available for the smallest information gain. It takes most of a week once you include re-indexing production-scale data and re-grading answers, and it moves four things at once: chunk size, the number of chunks per document, the tokens in context at fixed k, and therefore the effective amount of material the model reads. A positive result cannot say which of those mattered, and a negative one cannot distinguish "chunk size was not the problem" from "512 with k=20 blows the context out to 10,000 tokens and the precision loss cancelled the sufficiency gain".

That context figure is worth deriving rather than accepting, because the shape of it explains why the four changes cannot be separated. Twenty chunks of 512 tokens is a little over 10,000 tokens, and twenty chunks of 128 tokens is a little over two and a half thousand. Take the ratio of those two and the k cancels: it appears once in the numerator and once in the denominator, so the blow-up factor is the ratio of the chunk sizes alone, which is four. Turning k down to keep the context small therefore does not undo the change — it trades the same factor of four back out of the number of distinct documents the model gets to see. Chunk size and context size are not two knobs; they are one knob with k attached to it.

Two smaller experiments come first, and neither touches the index.

The table is easier to read as gaps than as levels. The first row and the last differ by most of a working week, and what that week buys is a result about four coupled variables rather than one. The ordering follows from that gap: run the cheap arms first, because either of them can make the expensive one unnecessary and neither can be made unnecessary by it.

The oracle run is the cheapest thing in the table and it is the one people skip. Assemble the context from the gold spans alone, generate, and grade. If procedural answers are still wrong with perfect context, the defect is in generation or in the prompt and every retrieval change you were about to make is wasted effort. On the fixture, the oracle arm gets nine of nine procedural queries right, which establishes that the ceiling is high and the loss is happening on the retrieval side.

Neighbour expansion then tests the specific mechanism. Retrieve as normal with the existing index, and for every hit, pull the chunks that sit immediately before and after it in its source document, merging overlaps and stopping at document boundaries.

def expand(hits: list[Chunk], store: ChunkStore,
           window: int = 2, budget_tokens: int = 2_500) -> list[Chunk]:
    """Widen each hit to its document-contiguous neighbours, best hits first."""
    chosen: dict[str, Chunk] = {}
    for hit in hits:                       # hits are in rank order
        lo = max(0, hit.chunk_ix - window)
        hi = hit.chunk_ix + window
        for c in store.range(hit.doc_id, lo, hi):
            if c.id in chosen:
                continue
            if sum(x.tokens for x in chosen.values()) + c.tokens > budget_tokens:
                return order_by_document(chosen.values())
            chosen[c.id] = c
    return order_by_document(chosen.values())

Two details in that function are the experiment rather than the implementation. The token budget is fixed at the baseline's 2,510 so that both arms put the same amount of text in front of the model, which is the only way the comparison attributes anything to contiguity rather than to context size. And the final ordering is by document position, not by score, because a procedure whose steps arrive shuffled is a different failure from one whose steps are missing, and you do not want to introduce the second while testing the first.

The 2,510 is a measurement rather than a round number chosen for tidiness. Two ways of arriving at it agree: twenty chunks of at most 128 tokens comes to a little over two and a half thousand, and totalling the tokens the baseline really put in a prompt lands in the same place, a per cent or two lower because the chunker stops at a sentence boundary before it reaches its ceiling. The two instruments share no step, and the gap between them is accounted for. That is what licenses holding the number fixed across both arms — it is a property of the baseline configuration rather than a cap that might advantage one of them — and holding it fixed pins the number of chunks in front of the model at about twenty either way. What changes between the arms is how that text is arranged rather than how much of it there is, so any difference in answers is attributable to arrangement.

The confounds worth controlling, and the ones you accept#

Holding the token budget constant means the expanded arm retrieves fewer distinct hits — roughly six seeds at ±2 instead of twenty independent chunks — so the effective k has moved. That is unavoidable and it is the right trade, because tokens in context is the variable that most directly changes what the model does, and letting it float while claiming to test chunking would make the result uninterpretable in the same way the full re-chunk is.

Six seeds at ±2 would be thirty chunks if none of the windows touched, and thirty will not fit in the budget. They fit because the windows overlap: the gold material in the runbook is contiguous, so seeds landing near each other claim some of the same neighbours, and the if c.id in chosen guard charges each chunk to the budget once. The seed count is an outcome of the experiment rather than a setting, and it will differ on a corpus whose relevant material is spread across documents.

The second confound is the query mix. Complaints are concentrated on procedural questions, and the fixture's forty queries contain nine of them, so an effect that is large on procedural queries and zero elsewhere shows up as a small effect on the aggregate. Report per class, always, and treat the aggregate as a summary rather than as the result. The third is grading: if a human grades the answers and knows which arm produced them, the effect will be larger than it is, so grade blind with the arms shuffled, or use an automatic judge — a second language model prompted with a fixed rubric and asked to score each answer — and accept its noise.

The gold labels themselves do not confound anything here, because they were attached to character spans in the source documents rather than to chunk identifiers. That decision from chapter 1 is what makes chunk size testable at all: labels resolve to whatever chunks the current configuration produced, so an experiment that changes chunking does not invalidate its own ground truth. It is worth separating the two kinds of choice in that sentence. The encoder's input ceiling and the fact that prose states an antecedent once are constraints nobody picked. The 128-token budget, the absence of overlap, k=20, the ordering by document position and the span-based labels are all decisions somebody made, and every one of them can be revisited without asking permission from the physics.

Reading an inconclusive result honestly#

The conclusion available from a result of that shape is that the query set was never large enough to resolve an effect of this size, which is a statement about the experiment rather than about chunking. Distinguishing that from "chunk size is not the cause" matters, because the two lead to different next steps: the first says gather more evidence, the second says look elsewhere, and reporting the second when you have the first sends the team off to rewrite prompts.

Two things follow. The first is to enlarge the procedural set to fifty or sixty queries by mining search logs and support tickets for questions containing "how do I", "steps to" and their kin, which is a day of labelling and makes a twenty-point effect detectable. Those two numbers are related, and the relation is the reason for picking them. The width of the interval around a difference in proportions shrinks with the square root of the number of queries, so multiplying the set by four halves the error bar and multiplying it by nine cuts it to a third. At fifty to sixty queries per arm the bar comes down to a little under twenty points, which is what puts a twenty-point effect just outside it. Fifty to sixty is not a round number someone liked. It is the smallest set on which the effect you are hunting clears its own noise.

The second is to stop grading answers and measure the mechanism directly, because answer grading spends nine units of evidence where the retrieval layer offers ninety-nine. That pair of numbers closes exactly: nine procedural queries, eleven gold chunks behind each of them, and every one of those gold chunks either entered the candidate set or did not. The observations were collected already — they are sitting in the retrieval log — so the eleven-fold increase in sample costs nothing but the query to extract it.

The mechanism predicts something specific: among the eleven gold chunks per procedural query, the ones that fail to enter the candidate set are the ones with low lexical overlap with the query, and that is a correlation you can compute over every gold chunk in the set without generating a single answer. If gold chunks missing from the candidates share almost no content words with the query while the retrieved ones do, the mechanism is confirmed with far more statistical weight than the answer-level experiment could ever provide.

Where a contiguity fix buys nothing#

The ninety-nine observations are not ninety-nine independent ones, and the honest version of the claim has to say so. Eleven of them come from the same query and the same document, and chunks from one runbook resemble each other in exactly the way that makes them fail together. The effective sample sits somewhere between nine and ninety-nine, closer to ninety-nine the more the eleven differ from one another, and there is no way to know where it sits without looking. It is still a large improvement on nine, and it is not the eleven-fold one the raw count suggests.

Neighbour expansion is scoped to a shape rather than to a corpus, and the shape was established from the listing: the eleven gold chunks are contiguous, running #0 to #10 with no gaps. Expansion works by pulling document-adjacent chunks, so it can only recover material that is adjacent in a document. On a question whose answer needs one paragraph from a policy and one from a runbook, every neighbour it pulls is the wrong neighbour, and the fixed token budget means those wrong neighbours are displacing hits that would otherwise have arrived. The same experiment on that corpus would come back negative, and the negative would be correct about expansion while saying nothing at all about chunk size.

The oracle result carries a matching caveat. Nine of nine says the generator can use perfect context on this fixture, and it does not say the generator is reliable, because nine questions cannot say that about anything. What an hour bought is one hypothesis ruled out.

The mechanism test is the one that generalises. It says nothing about 128 versus 512 as numbers, and everything about whether the units in your index are self-sufficient for the questions your users ask, which is the property that actually matters. A unit is self-sufficient when everything needed to act on it is inside it, and a fixed token budget has no way of knowing where that boundary falls, because it is counting tokens and the boundary is a fact about meaning.

What none of this touches is a different class of complaint entirely, where the retrieved chunk is self-sufficient, well-ranked, precisely on topic, and describes a policy that stopped being true fourteen months ago.