Re-chunking moves every boundary in the corpus at once

A retrieval system does not search whole documents. Before anything is stored, a chunker cuts each document into passages — chunks — of a bounded size, and it is the chunks, not the documents, that are turned into vectors and searched. Two settings govern the cut. The window size is how much text one chunk may hold, and the overlap is how much of the previous chunk's tail is repeated at the head of the next. A corpus of support documents therefore reaches an index twice removed from what anybody wrote: once by whatever a parser managed to extract, and once by wherever a chunker decided to cut.

Those two settings look like numbers in a configuration file, and a number in a configuration file looks like something you can change for one failing query and leave alone otherwise. The reason it is not is mechanical rather than a matter of taste, and it is worth walking. A chunk identifier is derived from a document identifier plus a position inside that document — an ordinal, or a character span. A change to the splitting rule changes every position. So the second run produces a different identifier for the same sentence, and the vector stored under any given identifier is now the embedding of a different span of text than it was before. Nothing that was measured against the first index is a measurement of the second one.

That derivation has a second consequence, which the arrangement described below sidesteps rather than solves. In a store updated in place, an upsert keyed on chunk identifier can only overwrite an identifier the new run also produces, so chunks whose identifiers the new run never produces are not superseded by anything and two generations coexist. Rebuilding into a fresh index file instead of writing into the old one avoids that entirely. Keeping the old file is a design decision rather than a requirement, and the requirement it serves — that a change can be compared against what preceded it — turns out to be the one this chapter is about.

The failing query is how many included seats does the Enterprise tier come with? and the document that answers it is pricing/enterprise-tiers.md, a nine-row markdown table with a header row naming the columns. At 400 tokens with no overlap the first boundary lands between the fourth and fifth rows, so one chunk holds the header and four tiers, and the next holds five rows of bare numbers with no column names attached to them. The Enterprise row is in the second chunk. Retrieved on its own it reads | Enterprise | 250 | 50 | unlimited | 99.95% |, which tells a model nothing, and it does not retrieve anyway because a row of digits and pipes has almost no semantic surface for a query about seats.

Two separate failures are stacked there, and separating them is what makes the rest of the chapter predictable. The first is a retrieval failure: a dense retriever compares an embedding of the question against an embedding of each chunk, each chunk's embedding is computed from that chunk's text alone, and the text of the second chunk contains neither the word the question uses nor any word naming what its digits are. There is nothing in that vector for a question about seats to be near. The second is a comprehension failure, and it survives the first being fixed: hand that chunk to the model at rank 1 and it still cannot say which column is seats, because the column names are in a different chunk. One boundary produced both, and only one of them is about ranking.

Changing the chunker to 1,000 tokens with 200 of overlap puts the whole table in one chunk with its header. Re-embedding the corpus took 4 minutes and 11 seconds and about 9 cents, the query passes, and the full eval set moved from 16 correct to 14.

Every vector is a different vector now#

The two lost queries are not mysterious, and both explanations are sitting in the retrieval block of their trace records.

The e12 mechanism is dilution, and it is the mechanical heart of why widening a window is not free. An embedding of a passage is a single point summarising the whole window, so a chunk whose 400 tokens are almost entirely an on-call table sits close to queries about paging, while the same content inside a 1,000-token window that also contains a revision history and two paragraphs about rollback procedure produces a vector pulled towards the average of those three subjects. The old index had 9,842 chunks; the new one has 4,610, and every chunk that used to be tightly on one topic is now a blend of that topic with whatever was next to it in the file.

An analogy helps with dilution, and its limit is worth stating before it is used. A chunk's embedding behaves a little like the centre of mass of the subjects inside the window: add a second heavy subject and the centre shifts towards it, so it no longer sits on top of either one. Where that picture breaks is that a centre of mass is a straight average of positions, and an embedding model is not an average of its parts — it reads the whole window in one pass, and the point it returns need not lie anywhere between the points the two subjects would have produced alone. The analogy understates the problem for exactly that reason. A diluted chunk is not necessarily a compromise between paging and rollback; it can land somewhere neither question looks.

The e04 mechanism is redundancy from overlap. With 200 tokens of overlap, adjacent chunks share a fifth of their content, which makes them near neighbours of each other as well as of the query, so when a long document is relevant its consecutive windows tend to occupy consecutive ranks. On e04 the top 5 came from three distinct locations in the corpus rather than five, because ranks 1, 2 and 3 were overlapping windows of the same handbook section. Top-5 stopped meaning five independent looks at the corpus and started meaning three, and the document that actually answers the residency question was the fourth distinct location.

The overlap fraction is one of the few things in this chapter that can be checked without running anything: two hundred tokens repeated out of a thousand-token window is a fifth, and a fifth is what the previous paragraph claims. It matters because it sets how similar two adjacent chunks are to each other. Ranking does not know that two chunks came from the same paragraph; it only knows that both are near the query, and near-duplicates are near the same things. So the top-k budget is a budget over ranks, not over places, and overlap spends part of it on repeats. That is the step to hold on to: raising overlap does not add evidence, it lowers the number of distinct locations a fixed k can reach.

Both changes also moved every number the pipeline reports. Average prompt size went from 2,098 to 5,240 tokens because five chunks of 1,000 replaced five of 400, which raises input cost per query from $0.0063 to $0.0157 and re-opens the position effects from the fourth chapter, since the correct chunk is now further from the question in character terms even at the same rank. Index size fell from 9,842 to 4,610 vectors, so the matrix multiply got faster, which is the least interesting consequence on the list.

Two of those four figures are not independent measurements, and noticing that is worth more than either of them. Input cost is a token count multiplied by a price per token, and the price per token did not change when the chunker did, so the cost figures have to be the token figures times a constant. They are: the new prompt is about two and a half times the old one, and the new cost is about two and a half times the old cost, closing to every digit both figures carry. What that licenses is a piece of scepticism about reports — the cost line is not a second piece of evidence about the change, it is the token line restated in dollars, and a report offering both is offering one. It also licenses a check in the other direction. If prompt size and input cost ever stop moving together between two builds, either the price moved underneath you or the two numbers came from different runs.

A third figure has a genuine residual, and naming it is cheaper than rediscovering it. Five chunks of a thousand tokens replacing five of four hundred predicts a rise of three thousand tokens per prompt. The measured rise is larger than that, by a bit over a hundred tokens. The residual is everything in a prompt that is not chunk text — the question, the instructions, whatever separators the renderer puts between passages — together with the difference between a chunker's nominal window and what the prompt's own tokeniser counts when the same text arrives at the far end. It is small, and the reason to write it down rather than round it away is that recomputing it on a later build is the cheapest available check that the renderer has not quietly started adding something.

Two out of twenty is not noise here#

The instinct to wave this through — twenty questions, two moved, that is within sampling error — imports a statistical model that does not apply to this measurement. At temperature 0 against a fixed index nothing in the pipeline is sampled. The query embedding is a deterministic function of the question, the ranking is a deterministic function of that vector and the index, the prompt is a deterministic function of the ranking, and the completion at temperature 0 is as close to deterministic as the provider gives you. Rerunning the eval three times against the same build produced the identical 14 every time, and each of the two regressions has a mechanism visible in its own trace record. Nothing here is a random draw that a larger sample would smooth out.

The check most teams reach for at this point is to run the eval again, and it is worth being exact about why that check cannot settle anything. Its job would be to tell a real regression apart from an unlucky draw. An unlucky draw would give a different score on the second run; a real regression gives the same score every time. Here the second and third runs returned the same figure as the first, which is what a real regression looks like — and also what a stable score looks like when nothing is wrong. Rerunning is a defence whose output does not move when the thing it is supposed to discriminate moves, and a signal that reads the same in both worlds carries no information about which world you are in. The check that does discriminate is the one the trace records already support: write a mechanism against each moved rank and see whether it survives being written down.

The second half of the argument is about what the eval set is. Twenty questions is not a sample from the distribution of user questions; nobody drew it randomly and it has no confidence interval. It is a floor — a set of cases that must not break — which means each failure stands for a class rather than for itself. Counting the corpus makes the class concrete: 1,240 of the old 9,842 chunks were more than eighty percent table or list content, spread over 96 documents, and every one of those documents has just had its boundaries redrawn the same way db-failover.md did. The two visible regressions are the two members of that class that happen to have an eval question written against them.

There is a second reason two is not really two, and it is a product rather than a count. The number of queries a change breaks is the number of exposures multiplied by the chance that any one exposure goes badly, and re-chunking sets the exposures term to the whole set — every document at once, ninety-six of them table- or list-heavy out of four hundred and twelve. A per-document risk small enough to accept when one file is re-split is a different quantity when every file is re-split on the same afternoon, because only the second factor stayed still. Twenty questions sample the exposures term very thinly, so the count they return is a lower bound on the breakage rather than an estimate of it.

Three ways to ship that are not blind#

The framing that helps is that 1,000/200 is one point in a space, chosen because it fixed one query, and the space has cheaper points in it.

Read that table across the rows rather than down the columns. The rebuild-cost column varies by under a minute and by a single penny across every option, which means cost decides nothing here; the eval column varies by four points, and the risk column varies between a named regression and an unmeasured class. The decision is being made in the last two columns, and a discussion that spends its time on rebuild time is discussing the one column with no information in it.

It is also worth separating the requirement from the decisions. The requirement is that a question whose answer lives in a table gets answered, and nothing in that sentence mentions chunk size. Widening the window everywhere, widening it for one class of document, and refusing to split tables at all are three decisions that each satisfy it. A decision that meets a local requirement by moving a global setting acquires a blast radius the requirement never asked for, and that is the whole distinction here: the requirement is local to one query, and the window is not local to anything.

The eval column can also be read as arithmetic, and doing so is how one row turns out to close exactly and another not to. Start from sixteen, the score before anything moved. The parse-time option leaves every prose chunk at four hundred tokens, so e12 and e04 keep the ranks they had; it recovers the pricing query, and it recovers the glossary query described below. Sixteen plus those two is eighteen, which is what the row says, with nothing left over — every point of that gain is attached to a named query with a named mechanism, and that is what makes the row trustworthy. Now do the same for the wider window everywhere. Sixteen, plus the pricing query it recovers, minus the two regressions the trace records explain, predicts fifteen. The row says fourteen.

The residual is one question, and it is worth saying what it can be rather than rounding it off. Either a third question regressed and went unreported, or the pricing query is not one of the twenty and its recovery never entered the score at all. Those two readings are not equally comfortable, because the second one breaks the row that did close: if the pricing query is outside the set, the parse-time option recovers only the glossary and lands on seventeen rather than eighteen. Taking both rows at face value forces the first reading — a third regression, unreported. Nothing in this section changes a figure, and the point of the arithmetic is that it takes about a minute and tells you which questions the report has stories for and which it does not.

The per-class option works because the chunker already sees the document type and the text, so a rule of "if this document contains a markdown table of more than three rows, use 1,000/200" is four lines and applies to 96 of 412 documents. The mixed index is not a problem for cosine similarity — vectors of the same dimension from the same model are comparable regardless of how much text produced them — though it does mean top-5 covers a variable amount of context, which the prompt-token counter in the trace will show as a wider spread.

The parse-time option scores best because it attacks the actual defect. A table is a structural unit that carries meaning only with its header, so the chunker should never be allowed to split it, which is what the option above means by treating tables as atomic — a unit the splitter must emit whole or not at all. The fix is to emit each table as its own chunk with the header repeated.

TABLE_RE = re.compile(r"(?:^\|.*\|\s*$\n)+", re.MULTILINE)

def split_structured(text: str, size: int = 400, overlap: int = 0):
    """Yield (kind, text) units; tables stay whole, prose is windowed."""
    cursor = 0
    for m in TABLE_RE.finditer(text):
        if m.start() > cursor:
            for _, prose, _ in windows(text[cursor:m.start()], size, overlap):
                yield "prose", prose
        table = m.group(0)
        if len(ENC.encode(table)) <= 1200:
            yield "table", table
        else:                                   # long tables: repeat the header
            header = "\n".join(table.splitlines()[:2])
            rows = table.splitlines()[2:]
            for i in range(0, len(rows), 20):
                yield "table", header + "\n" + "\n".join(rows[i:i + 20])
        cursor = m.end()
    for _, prose, _ in windows(text[cursor:], size, overlap):
        yield "prose", prose

Eighteen out of twenty comes from this keeping every prose chunk at the tight 400 tokens that e12 and e04 depended on while giving tables the treatment they needed, and it also fixed a query that neither of the other options touched, where a two-column glossary had been split across three chunks. The cost is that it is real parser code with its own failure modes, and it only handles markdown — the same problem in a PDF needs a table extractor, which is a much larger commitment.

Whichever option ships, the reporting requirement is the same and it is the point of the chapter: the whole eval set rerun against the new index, with the before and after ranks for every query that moved. A report that says "the pricing query now passes" describes one row of a table in which every row changed.

Producing that report is cheaper than it sounds because nothing forces you to throw the old index away. Two index.npz files at 58 MB each sit side by side happily, and a comparison script embeds each eval question once and searches both, which makes the rank delta for every query a single pass costing about four seconds and two cents of query embeddings.

def rank_report(cases, old, new, k=5):
    for c in cases:
        q = embed_query(c["q"])                       # embed once, search twice
        before = [h["chunk_id"] for h in search(old, q, k=50)]
        after  = [h["chunk_id"] for h in search(new, q, k=50)]
        r_old = before.index(c["gold"]) + 1 if c["gold"] in before else None
        r_new = after.index(c["gold"]) + 1 if c["gold"] in after else None
        yield c["qid"], r_old, r_new, (c["gold"] in after[:k]) - (c["gold"] in before[:k])

Searching to depth 50 rather than 5 is what makes the report diagnostic rather than binary, because a gold chunk that fell from rank 1 to rank 7 is a different story from one that fell to rank 400, and only the first is recoverable by anything short of re-chunking again. The report also covers queries the eval does not fail on: e02 stayed correct while its gold chunk moved from rank 1 to rank 4, which is a query one boundary change away from breaking and worth knowing about before it does.

When chunk size was never the knob#

Two checks settle whether this whole line of work is aimed at the right stage, and both take under fifteen minutes.

Both checks are asking which factor in a product is zero. A question gets answered only if the text that answers it survived extraction, and is retrievable as a unit, and is readable by the model once retrieved. Those three conditions multiply, so any one of them at zero takes the product to zero, and improving the other two compensates for nothing, because there is nothing left to compensate with. Chunk size only ever touches the middle factor. Each check below pins one of the other two at its best possible value, and between them they say whether the middle one is where the work belongs.

Put the entire table in one chunk by hand, embed it alone, and query it. If the gold chunk still does not reach the top 5 with the whole table and its header intact, the problem is not where the boundaries fell. On this corpus that check passed, but the same check on the SLA credits query from the fourth chapter failed: the whole credit table in one chunk still ranked 19th for what is the SLA credit for a four-hour outage on Business?, because the query says "SLA credit" and the table says "service credit", and a table of durations and percentages has very little prose for a dense retriever to match against. That is a vocabulary and representation problem whose fixes are lexical search alongside the dense index, a synonym expansion on the query, or a generated one-sentence summary embedded alongside each table — and no chunk size touches any of them.

Read the extracted text before the chunker sees it. If the table arrived from the parser as a run of concatenated cell values with the column structure gone, which is the normal outcome of extracting a table from a PDF without a table-aware extractor, then no chunking strategy can reassemble it, because the information was destroyed two stages earlier. The manifest from the second chapter will not flag this, since character density looks entirely healthy — 40,000 characters from a 900 KB PDF is unremarkable — and the only detection is opening build/text/ and looking.

That manifest fails here for the same reason rerunning the eval fails above. Character density reads the same on the extraction that preserved the table and on the extraction that flattened it, because the count is of characters and what was destroyed was their arrangement — the cells all survived, in order, with the column structure that made them mean anything thrown away. A measure that cannot vary with the fault it is meant to catch is not a weak check; it is not a check. Reading the extracted text is tedious and it is the only instrument in the building that moves when this fault is present.

Widening a window is also not the state of the art, and it is worth knowing what sits beyond it: splitting on semantic boundaries by embedding sentences and cutting where consecutive similarity drops, or letting a small model propose boundaries. Both are more expensive at ingest and neither escapes the property this chapter is about, which is that any change to the chunker moves every boundary and every vector, so the rerun-the-whole-eval discipline applies identically.

The window setting is the smallest-looking change in the whole system and it has the widest reach, because every identifier and every vector is derived from it. A report that says the pricing query now passes is true and it is one row. The report that lets somebody else audit the decision is the whole eval rerun against a pinned index, every moved rank listed, a mechanism written against each one, and the residual named where the mechanisms do not account for the score.

That discipline — hold the inputs fixed, diff the intermediates, explain each delta by a mechanism you can name — is now the thing worth carrying forward, because it generalises past your own chunker. The next chapter applies it to a change you did not make and cannot see, when a teammate reimplements the same pipeline in a framework and it beats yours on three queries.