A heading prefix buys most of what late chunking promises

A heading prefix is a short string glued to the front of a chunk's text at the moment that text is handed to the embedding model. The string is usually the chunk's breadcrumb — the chain of headings standing above it, from the document's title down to its own section — and its purpose is to put words into the encoder's input that the chunk's own sentences never contain. Nothing else about the chunk changes. The boundaries are wherever the splitter put them, and the text served to a generator is the text the author wrote.

The problem it answers is anaphora, the habit a document has of naming something once and then referring back to it as "the driver" or "the unit" forever after. A chunk cut from the middle of such a document has no copy of the name in it, so a vector computed from that chunk alone cannot encode which product the chunk is about. Chunk 12 of a vendor datasheet, the one about dimming curves that ranked 912th in an index of vendor documentation, is 190 tokens of exactly that kind of text. Prepend eleven tokens to it before embedding — Digidim 458/DALI dimming driver > 6. Dimming behaviour — and its cosine against "Digidim 458 dimming curve" goes from 0.508 to 0.671, and its rank from 912 to 6.

The prefix goes into the embedding, not into the context#

Late chunking, on the same chunk, produced 0.686 and rank 4. The prefix is a string concatenation in the ingest loop. Those two facts sitting next to each other are the whole argument of this chapter, and the mechanism underneath them has no subtlety in it at all. The encoder is a function of the tokens handed to it. The tokens that would discriminate this chunk — the brand name, the model number, the name of the section it sits in — were absent from those tokens. So put them there. The vector then contains what the chunk is about because the input contained what the chunk is about.

That is the requirement, and it is worth separating from everything that follows, which is a design decision. The requirement is that the encoder's input carry the discriminating vocabulary. Which string supplies it, what format that string is written in, whether it is stored or recomputed, and what the splitter's ceiling becomes as a result are all choices, and each of them can be got wrong without violating the requirement. What makes prefixing a technique rather than a hack is the first of those choices: the separation it requires between the two things a chunk record holds.

def build_chunk(section, doc, ceiling=480):
    prefix = " > ".join([doc.title] + section.breadcrumb[1:])
    return {
        "text":       section.text,                    # what the generator reads
        "embed_text": f"{prefix}\n\n{section.text}",   # what the encoder reads
        "breadcrumb": section.breadcrumb,
        "n_tokens":   count(section.text),
        "n_embed_tokens": count(prefix) + 2 + count(section.text),
    }

Keeping text and embed_text as separate fields is the part that gets skipped, and skipping it costs twice, in opposite directions. If the prefix is written into the served text, every retrieved chunk arrives in the context window with a breadcrumb glued to its front, which is 10 to 40 tokens of overhead per chunk and, more annoyingly, shows up in citations and in any UI that renders the retrieved passage. If instead the prefix is applied only at embed time and thrown away, you lose the ability to reproduce a vector from a stored record, which makes debugging a bad retrieval impossible — you cannot tell whether a chunk scored oddly because of its text or because of a prefix you can no longer see. Storing both is a few hundred megabytes on a corpus this size and removes both problems.

The n_embed_tokens field is there because of chapter 3, where an encoder that accepts 512 tokens indexes only the first 512 tokens of whatever it is given and drops the rest without complaint. A prefix consumes budget against that accepted length. A 480-token ceiling with a 38-token breadcrumb produces 520 tokens of embedding input against a 512-token encoder, and the last eight tokens of every chunk are silently dropped.

Those two figures are worth checking rather than accepting, because the obvious arithmetic does not produce them. A ceiling of 480 plus a breadcrumb of 38 falls two short of 520, and would overflow 512 by six rather than by eight. The missing two tokens are in the code above: f"{prefix}\n\n{section.text}" joins the prefix to the text with a blank line, and n_embed_tokens accounts for it with the + 2 term. Add that separator and the total is 520 exactly, and the overflow is eight exactly. The prose and the code close on each other with nothing left over, which is what licenses treating the separator as budget rather than as punctuation: two characters of formatting cost the last two tokens of every chunk that sits near the ceiling.

So the ceiling the splitter enforces has to be the encoder limit minus the separator minus the longest prefix the document will produce, and the longest prefix is a property of the corpus rather than a constant. On deeply nested Confluence spaces the longest breadcrumb was 61 tokens, which is not a rounding error against 512 — it pushes the usable text budget below the 480 the splitter was configured with, and a splitter that does not know this produces chunks whose tails are never indexed on precisely the documents with the deepest structure.

Three ways to say what a chunk is about#

The breadcrumb is the cheapest prefix and depends entirely on chapter 5's routing rule, where a document is split on its headings only when the extraction step reports that a human marked those headings and the file format preserved the mark. Prefixing a chunk with a breadcrumb a layout model invented is worse than prefixing nothing. The mislabelled inverter chunk from that chapter would be embedded as Inverter Manual > 6. Firmware update > Do not power-cycle the unit during firmware transfer followed by steps 5 through 7, and that vector now matches queries about power-cycling with a text that does not discuss it.

It is tempting to treat that as a metadata problem and therefore a contained one, because a breadcrumb lives in a metadata field, and a wrong value in a metadata field is visible to anyone who looks at the record. That containment stops the moment the string is concatenated into embed_text. The encoder has no notion of metadata; it reads the prefix as text, exactly as it reads the rest. A prefix propagates the metadata's error into the vector, where it is much harder to see — there is no field to inspect, only a float array that is now wrong in a way no schema check will report.

Where the structure signal is not verified, a generated prefix is the alternative: give a small model the whole document and the chunk, and ask for one sentence situating the chunk within it.

PROMPT = """<document>{doc}</document>
Here is a chunk from that document:
<chunk>{chunk}</chunk>
Write one sentence, under 30 words, stating what this chunk is about and
naming any entity, product or section it refers to only indirectly.
Output the sentence alone."""

The cost is arithmetic worth doing before dismissing it, and it is short enough to do in full. On 738,000 chunks with the document cached across all chunks from the same document, each request reads roughly 4,200 cached input tokens plus a 480-token chunk and writes about 40 output tokens. The input side is therefore the document plus the chunk, and it is charged at cache-read rates — the reduced price a provider charges for input tokens it has already processed and kept, which is what makes re-sending the same document once per chunk affordable — around $0.03 per million input tokens. Multiply that rate by those tokens and the input side of one request costs a little over fourteen hundred-thousandths of a dollar. The output side is 40 tokens at $1.25 per million for a small model, which is five hundred-thousandths of a dollar. Adding the two gives about $0.00019 per chunk, of which roughly three quarters is the input side.

Two more multiplications finish the estimate, and both close on the figures quoted for them. That per-chunk price times 738,000 chunks is roughly $140 for the corpus, plus one cache write per document. And 738,000 requests at 40 requests per second is about five hours of wall-clock, which parallelises cleanly because chunks are independent. What that closure licenses is confidence about which term to argue with: the estimate is reproducible from four inputs, three of them rates you can look up, so a disagreement about the total is a disagreement about the chunk count or about the cache hit rate and nothing else. Compared against three weeks of engineering and an eleven-hour quadratic embedding pass, the generated prefix is cheap enough that the interesting question is whether it works rather than whether it is affordable.

The format of the prefix is worth one experiment of its own, because the encoder reads it as text rather than as metadata. A breadcrumb written with arrows — Expense Policy > 3. Submitting a claim > 3.1 Deadlines — tokenizes into a string that looks like nothing in the model's training distribution, whereas the same information as a sentence — From the Expense Policy, section 3.1 Deadlines, part of Submitting a claim: — is ordinary prose. On this corpus the sentence form was worth about a point and a half over the arrow form, which is small but free, and the arrow form was in turn clearly better than no prefix. Generated prefixes are prose by construction, and part of the two-point advantage they show in the table below may simply be that.

Verifying the generated prefixes is a step people skip because the output looks fine. Sample fifty, check that the entity named in the prefix is actually the entity the chunk discusses, and count the hallucinations; on this corpus the rate was around 3 percent, mostly on chunks from documents that discuss two products. Three percent of 738,000 chunks is 22,000 vectors pulled toward the wrong product name, which is small enough to ship and large enough to want a number for.

The numbers in the first column are from one corpus and one query class, and the ordering rather than the values is what transfers. Read the column as the distance from the baseline rather than as five scores. Late chunking is worth ten points over the baseline. The breadcrumb prefix, which costs a string concatenation, is worth eight of those ten — four fifths of the expensive answer's gain, for none of its cost. That is the sense in which a prefix buys most of what late chunking promises, and the residual is worth naming rather than rounding away: two points remain, and they are the part of a document's context that a line of heading text cannot express.

The rest of the column says something stronger. The generated prefix is worth eleven points, which is more than late chunking's ten, so the cheap route is not merely close to the expensive one on this corpus — it is ahead of it. And the combination is worth twelve, one point beyond the generated prefix alone and two beyond late chunking alone. If the two techniques were supplying different missing information, their gains would tend to add, and twelve is nowhere near the sum of ten and eleven. Two of the cheap approaches land within two points of the expensive one, and the combination adds almost nothing on top of either, which is the expected result if they are supplying the same missing information by different routes. That is what the table licenses concluding, and it is a conclusion about this corpus rather than about the techniques.

Dependence a single line cannot carry#

Deep dependence does exist and is worth being able to recognise, because it is the case where the two points of residual in the table become the whole question. A chunk reading "In the cases described above, the 14-day limit does not apply" needs to know which cases; no heading says. A table's data rows need column semantics established in prose two chunks earlier. A section using "the exception" to mean something defined on page 3 needs the definition, not the section title.

What those three have in common is the reason a prefix cannot reach them. A breadcrumb can only ever say what somebody wrote in a heading, and a heading names a section rather than restating the section's premises. Full attention over the document reaches all of them, since attention is not restricted to what any one line says. A generated prefix reaches them only if the generating model happened to notice, which is a probabilistic property of a summariser rather than a guarantee. Sampling twenty failing chunks and reading what each one is actually missing sorts a corpus into these two piles faster than any metric.

Sixty-one percent, and where the other thirty-nine points went#

Both of the last two chapters have been answers to a question that deserves examining before either is funded: whether recall@10 of 61 percent on a 200,000-page corpus is a chunking problem at all. A recall of 0.61 means thirty-nine points are being missed, and nothing about that number says where they went.

Missing recall decomposes into causes that have nothing to do with each other, and the decomposition is done by reading, not by computing. No aggregate metric can perform it, because a metric counts what was missed and the buckets are distinguished by why. Take twenty failing queries, and for each one find where the answer actually lives and what was retrieved instead.

An afternoon of this on twenty failures gives you proportions, and the proportions decide the budget. Four of twenty in the first row means the anaphora fix addresses a fifth of the gap, so a three-week migration is buying at most eight points of the missing thirty-nine and probably fewer — at most, because a fifth of thirty-nine is a little under eight, and probably fewer because that assumes every query in the bucket is recovered. Twelve of twenty in the bottom three rows means chunking is not where the corpus is losing, and every configuration in this topic will move the number by two or three points while the actual problem sits in extraction or in the query set.

A prefix is the cheapest way to test a hypothesis about what a chunk's vector is missing, and testing it is the part that matters more than the eight points. The hypothesis is that a chunk fails because its text never names its own subject. Concatenating the name onto the front of the embedding input either recovers the failing queries or it does not, and either answer is worth a day. The version that recovers them tells you the corpus's context-dependence is the shallow kind, and the expensive migration would have been buying whatever residual the concatenation left — two points, on the corpus in the table above, and a figure each corpus has to produce for itself. The version that does not recover them tells you the dependence is deep, and now the three weeks have an argument behind them.

That reading exercise is also the only reliable defence against the failure mode this whole topic keeps circling — a configuration adopted because it won a comparison, kept because nobody re-ran the comparison, and still in place three corpora later. What a starting configuration needs is not more justification but a stated condition under which it stops being the answer.