Change the chunker and there is no affected subset to compute

Two tickets landed in the same sprint. The first was operational: a content team had spent a week rewriting 40,000 product pages, roughly half a percent of the 8 million document corpus, and those edits needed to reach the index. The second came out of a retrieval quality review, which had found that 512-token chunks were burying short factual answers inside long passages, and recommended moving to 256 tokens with 20% overlap. In the planning meeting someone put both on the same line of the sheet and called them "two incremental re-embeds, one small and one medium". The estimate attached to the pair was four hours of compute and "a few hundred dollars, mostly the second one".

The second half of that estimate happens to be close to right on the API spend and wrong about everything that matters, because the two tickets are not the same kind of operation. One of them has an affected subset. The other one does not, and no amount of cleverness will produce one.

Document-scoped changes and pipeline-scoped changes#

The 40,000 edited pages are a document-scoped change. Each edited document produces new text, the chunks derived from it produce new fingerprints, and every other document in the corpus produces exactly the fingerprints it produced yesterday. The work is bounded by the edits: 40,000 documents at a mean of 3.5 chunks each is about 139,000 candidate chunks, and because the fingerprint is computed per chunk, the ones whose text did not move are skipped. Editing a paragraph in the middle of a nine-chunk page changes two or three of those chunks. In practice about 45% of the candidates come back with a new fingerprint, which is 63,000 chunks, 28 million tokens, and $0.56 at $0.02 per million. The job finishes inside a coffee break.

Chunk size is not a property of any document. It is a property of the function that turns documents into chunks, so changing it changes the output for every document that the function is applied to, which is all of them. There is no query you can write that returns the affected subset, because the predicate "is this document affected?" evaluates to true everywhere by construction. A change with that property is a full reindex — every chunk in the corpus is recut, re-embedded and written again, and the index that results is a new population rather than an edited version of the old one — where the 40,000 edited pages are an incremental re-embed, bounded by what moved. That is the whole difference in kind, and it survives every attempt to argue around it.

The argument people reach for is subdivision. Halving 512 to 256 sounds like it should split each existing chunk into two, leaving the text of the corpus partitioned along a strictly finer version of the same boundaries, so a clever implementation could re-embed only the halves and reuse nothing else. It fails on two counts. Chunk boundaries are not placed at fixed token offsets in any chunker worth using; they are placed at the nearest structural break — a paragraph, a heading, a sentence — searching outward from the target size, so halving the target moves every boundary in the document rather than inserting a new one inside each old chunk. And overlap makes the point unconditional. With 20% overlap each chunk carries the final 51 tokens of its predecessor, so the text of every chunk that touches a boundary differs from anything that existed before, and with the boundaries redistributed across the document, that is all of them.

The third and fourth rows are the ones that make a re-chunk a different project from a re-embed. When the chunk ids all change, you are not updating an index in place; you are building a second population of vectors and deleting the first, which brings in every concern the previous chapter's fingerprints do not touch — where the new vectors live while the old ones are still serving, how the two are kept from mixing in the same namespace — the store's own partition of an index, within which point ids are unique and searches are answered — and what the peak footprint is while both exist.

The chunker change spares the short documents and nothing else#

The one genuine saving available on a re-chunk comes from documents short enough to be a single chunk under both configurations. A 180-token FAQ answer is one chunk at 512 tokens and one chunk at 256, and its text is byte-identical in both cases, so its vector is still correct. Capturing that requires a small refinement to what the fingerprint is keyed on. Chapter 1 folded the whole configuration into a single pipeline id, which is the safe default and the right one to start with. A chunker change is the case where that default costs money, because the chunker does not influence the vector once the text exists — it influences which text exists. Splitting the identity into a text-producing half (extractor, normaliser, chunker) and a vector-producing half (model id, snapshot, dimension, input prefix) and keying the fingerprint on chunk text plus the vector-producing half lets identical text keep its vector across a chunker change.

How much that spares depends entirely on the length distribution, which is why it has to be measured rather than assumed:

import math

def chunk_count(doc_tokens: int, size: int, overlap: float) -> int:
    if doc_tokens <= size:
        return 1
    stride = int(size * (1 - overlap))
    return 1 + math.ceil((doc_tokens - size) / stride)


def billed_tokens(doc_tokens: int, size: int, overlap: float) -> int:
    n = chunk_count(doc_tokens, size, overlap)
    return doc_tokens + (n - 1) * int(size * overlap)


BUCKETS = [           # (documents, representative length in tokens)
    (2_400_000, 180),
    (1_600_000, 380),
    (2_000_000, 900),
    (1_440_000, 2_400),
    (560_000, 9_000),
]

for docs, length in BUCKETS:
    old = chunk_count(length, 512, 0.10)
    new = chunk_count(length, 256, 0.20)
    print(docs, length, old, new,
          billed_tokens(length, 512, 0.10),
          billed_tokens(length, 256, 0.20))

Run over the real corpus, the buckets come out like this:

The index goes from 27.8 million vectors to 57.5 million, a factor of 2.07. Billable tokens go from 12.35 billion to 13.44 billion, a rise of 9%, because token volume is set by the length of the corpus plus overlap, and only the overlap term changed materially. At $0.02 per million that is $269 of API spend, and at a sustained 3.5 million tokens per minute it is 64 hours of wall clock.

So the number to put in front of a manager is not "about $270". It is: $270 of API spend, 64 hours of wall clock during which the corpus keeps changing underneath the job, and a permanent doubling of the vector population from 27.8 million to 57.5 million, with the peak during cutover higher still because both populations exist at once. The API line is the smallest of the three by two orders of magnitude, and the line that has to be approved is the one about the index.

That estimate is most sensitive to two inputs. The first is the document length distribution, specifically its tail: the 7% of documents longer than 4,000 tokens produce 24.6 million of the 57.5 million chunks, so a corpus with twice as many long documents as you assumed has roughly 40% more vectors than you quoted, and the mean document length tells you nothing about this because the mean is dominated by the 50% of documents under 512 tokens. The second is the overlap fraction, which is the multiplier on everything. At 0% overlap the same 256-token target produces 47.9 million chunks rather than 57.5 million; the 20 percentage points of overlap are worth 9.6 million vectors and 1.1 billion tokens on their own. Both inputs are cheap to measure and neither can be reasoned about from a description of the corpus.

Classifying a change is easier to do in continuous integration than in a planning meeting, because the classification is mechanical. Keep a fixture of two thousand documents spanning the length buckets, run the pipeline over it on every pull request, and record chunk count, total billed tokens and the set of chunk fingerprints. A pull request that changes the fingerprint set without changing the chunk count is text-affecting; one that changes the count is structural; one that changes neither is safe. Attach the projected corpus-wide numbers to the pull request — "57.5 M chunks, +107%, 13.4 B tokens to re-embed" — and the conversation about whether the change is worth it happens before it merges rather than after the invoice.

Where the distinction stops being useful#

Classifying a change as document-scoped or pipeline-scoped tells you the shape of the work and nothing about whether the work is worth doing. A re-chunk that doubles the index is justified by a retrieval quality argument or it is not justified at all, and the arithmetic above is silent on that question — it prices the change, and pricing is not evidence.

The distinction is also not a proxy for size. A document-scoped change can be enormous: if the content team had rewritten 6 million pages instead of 40,000, that would still be an incremental re-embed with a computable affected subset, and it would still cost more than the re-chunk. What "incremental" buys is the ability to name the subset before running, which is what makes a partial run resumable and a quote defensible. What a pipeline-scoped change costs is precisely that ability.

And a full reindex driven by chunking is a much smaller thing than a full reindex driven by a model change, even though both re-embed every chunk. The chunker change keeps the same model, the same vector width, the same distance metric — the arithmetic the index uses to turn a pair of vectors into a closeness score — and the same query-side encoding, so the new index can be evaluated against the old one on identical queries with nothing else varying. Chapter 7 takes up the case where the model itself moves, and almost none of that holds.

Both tickets now have a defensible shape and a defensible number, because someone had already measured the length distribution and the mean chunk size of that specific corpus. The harder version of the question arrives when the corpus is new, nobody has run a chunker over it, and someone wants a figure this week.