A templated heading path fails on a definable set of chunks

A header, in this topic, is a short line of context prepended to a chunk's text before that text is handed to an embedding model — a line naming the document the chunk came from and the section it sits in, so that a chunk which never says what it belongs to arrives at the index saying it. There are two mechanisms available for producing that line. A template composes it mechanically out of things a splitter already holds: the document's title and the stack of headings standing above the chunk. Generation asks a language model that has been shown the whole document to write the line itself.

The choice between them is usually argued as cost against quality, and that framing hides the question that actually decides it. A template is not slightly worse than generation across the board. On documents with real headings it is close to free and close to as good; on a nameable class of documents it produces nothing usable at all, and no amount of tuning a splitter changes that, because the information a template needs was never in those files. So the useful question for anyone running an ingestion pipeline is not which mechanism is better. It is how large a share of a corpus falls in the class the cheap mechanism cannot serve, and whether that class can be identified at write time rather than discovered from a bad answer months later.

Two ways to produce the line, and only one of them meters#

The template is the mechanism from chapter 1, where a splitter walks a document keeping a stack of the headings it has passed under and joins that stack to the document title. Beyond costing nothing per chunk, it has a property that matters more than it first appears, and the property is worth walking rather than asserting. The template is a pure function of the document text: the same document produces the same heading stack, which produces the same header string, byte for byte. The header is concatenated with the chunk text to form the embed input, so a byte-identical header over byte-identical chunk text is a byte-identical embed input. An embedding model is deterministic given its input, so a byte-identical input returns the same vector. Each link there is checkable on its own, and together they license something operational: a pipeline that re-ingests nightly can compare the newly composed header against the one it stored, skip the embedding call when the two match, and treat a vector that did change as evidence that a document actually changed. Reproducibility is what makes incremental re-ingestion possible at all.

Generation asks a model to write the line. The method comes from Anthropic's Contextual Retrieval in AI Systems, which reports the numbers that made the technique widely known and publishes the prompt: give the model the whole document, give it the chunk, and ask for a short line situating the chunk within the document. The cost structure only works because of prompt caching, a provider feature that stores the tokens at the front of a prompt after the first request and charges a fraction of the input price to reuse them on later requests that begin the same way. The document is identical across every chunk of that document, so it goes in the cached prefix and the chunk goes after the breakpoint that marks where the reusable part ends:

CONTEXT_PROMPT = """\
Here is the chunk we want to situate within the whole document:
<chunk>
{chunk}
</chunk>

Give a short, standalone line of context that says where this chunk sits in \
the document and what it is about, so that the chunk can be found by search. \
Answer with the context line only."""


def generate_header(client, doc_text: str, chunk_text: str) -> str:
    resp = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=100,
        system=[
            {
                "type": "text",
                "text": f"<document>\n{doc_text}\n</document>",
                "cache_control": {"type": "ephemeral"},
            }
        ],
        messages=[{"role": "user", "content": CONTEXT_PROMPT.format(chunk=chunk_text)}],
    )
    return resp.content[0].text.strip()

Two things about that call decide whether the cost model holds. The cache_control breakpoint sits on the document block, and the chunk goes into the message after it, so the document is written to cache once per document and read at roughly a tenth of input price for every chunk after the first. If you put the chunk in the system block, or interpolate anything per-chunk ahead of the breakpoint, every call pays full price for the whole document and the bill goes up by an order of magnitude with no error and no warning — check usage.cache_read_input_tokens on the second call of each document before trusting any estimate. The other constraint is the cache TTL, the time to live after which an unused entry is discarded, which is five minutes by default; a document's chunks have to be processed as a batch, close together, or the entries expire and each read becomes a write.

The order of magnitude follows from what sits on each side of the breakpoint. Before it is the document; after it is one chunk. A document is longer than one of its chunks by roughly the number of chunks it was cut into, so the prefix is nearly the whole prompt. A tenth of input price on nearly the whole prompt, against full price on nearly the whole prompt, is a tenfold difference on the dominant term — and it is invisible from outside, because a full-price call returns the same header a cached one does. Only a counter in the response separates them.

The model choice here is deliberate. Claude Haiku 4.5 is the cheapest model in the family and this is the task it exists for — a bounded transformation with the answer visible in the input. Its cache minimum is 4,096 tokens, which most whole documents clear comfortably and short ones do not, so very small documents get no caching benefit and are cheap anyway. A larger model produces marginally better lines and multiplies a per-chunk cost by five, which is the wrong direction on a corpus of this size.

Two constraints in that paragraph look alike and are not. The cache minimum is a requirement: the provider will not cache a prefix shorter than it, and no amount of pipeline design changes the number. Choosing the cheapest model in the family is a decision, made because the header task is bounded and its answer is present in the input, and it is a decision that could be revisited on a corpus where header quality mattered more than per-chunk cost. Confusing the two is how a pipeline ends up defending an arbitrary choice as though physics required it.

What generation gives you that the template cannot is a header for chunks whose context was never in a heading. Here is the same chunk from a support ticket under both schemes:

chunk text:
  "Tried that, same result. It's still returning 401 after the rotation,
   and the old key works fine, so it's not the token itself."

template header:
  Support ticket 44812

generated header:
  Support ticket 44812, an escalation about API key rotation on the
  payments integration failing authentication. This message is the
  customer confirming that a newly rotated key returns 401 while the
  previous key still authenticates.

The template header is not wrong. It is a correct document identifier that discriminates ticket 44812 from ticket 44813 and says nothing about what either contains, which — per chapter 2, where a header shared by every chunk of a document was shown to separate documents from each other and nothing else — is exactly the kind of header that helps at the document level and does nothing within it. The generated header puts "API key rotation", "authentication", and "401" into the embedded text, and a query about rotated keys returning 401 now has something to match.

Throughput is the other thing generation changes, and it is easy to leave out of a plan. A template runs at splitter speed, which is tens of thousands of chunks a second. Generation runs at whatever concurrency your rate limits allow, and the rate that concurrency implies is worth deriving rather than quoting. Suppose 200 requests are in flight at once and each takes roughly 700 ms end to end. One slot finishes a chunk and starts the next about one and a half times a second, so 200 slots together are 200 divided by seven tenths of a second, which is about 285 chunks a second.

Two figures follow from that rate by division, and they are the ones a plan needs. A routed population of 816,000 chunks at 285 a second is a little under three thousand seconds, which is just under fifty minutes — a little under an hour. The full 2.4 million at the same rate is a bit over eight thousand seconds, which is just over two hours. Notice that the ratio between those two running times, a shade under three to one, is the same ratio as between the two chunk counts, which it has to be: dividing both by one rate cancels the rate out. Two routes to the same factor of three, sharing no step beyond the division itself, and it is a useful check that a throughput estimate has not quietly changed its units halfway through.

An hour is fine for a backfill and it is a different operational shape from a pipeline that used to reindex a document in the time between two heartbeats. Documents must also be processed as contiguous batches so the cached prefix survives, which means the natural unit of parallelism becomes the document rather than the chunk, and a corpus with a few enormous documents will have stragglers.

Counting the chunks a template cannot serve#

The share of a corpus a template cannot serve is measurable before a single model call is made: run the template over everything, then count how many chunks came out with a header that said anything. Here is that count on a corpus of three sources — 1.6 million chunks of product documentation, 620,000 chunks of support tickets, and 180,000 chunks from PDFs of signed contracts.

source              chunks       with path   path informative
product docs     1,604,112   1,459,742 (91%)   1,371,905 (86%)
support tickets    620,388       6,204 ( 1%)         0 ( 0%)
scanned contracts  180,441     180,441 (100%)     7,218 ( 4%)

The middle column counts chunks where the heading stack was non-empty. The right column applies a crude filter for whether the path carries meaning: at least one segment that is not purely numeric, not a bare section marker, and not one of a small stop-list of structural words.

Two arithmetic facts in that block are worth doing by hand, because both license claims made later. The first is that the three source counts add to a little over 2.4 million, which is the whole corpus. There is no fourth row and no residual bucket, so every later figure quoted against "the corpus" is a figure against exactly these three sources and nothing else — a total that closes like that is what makes a share meaningful rather than approximate. The second is that the two non-product rows add to just over 800,000, and just over 800,000 against a little over 2.4 million is almost exactly one part in three. That is where the claim that a third of the corpus is unserved comes from, and it is a division of two counts on the page rather than an estimate.

The columns are more informative read as a gap than as levels. Take the difference between the middle column and the right one, source by source. Product docs lose about five points crossing from "has a path" to "the path means something", which is a small tax on a mechanism that is basically working. Support tickets lose one point, because there was almost nothing there to lose; their middle column is already near zero. Scanned contracts lose ninety-six points, from a perfect score to nearly nothing.

Those three gaps are three different situations, and only the third is dangerous. Product docs are a source that works. Support tickets are a source that fails honestly — the coverage check says it has no headings, which is true, and anybody looking at the number knows immediately what they are dealing with. Scanned contracts are a source that fails while reporting success. Every one of those chunks has a heading path, so a coverage check that asks "does this chunk have a header" returns yes for all of them.

That is the stumble in this chapter, and it is the tempting kind. A coverage metric that counts non-empty headers looks like a defence against exactly this failure, and on the sources where it is easy to check by eye it agrees with the eye. What it cannot do is discriminate a path made of words from a path made of numbering, because it holds still while precisely that property varies: the check tests whether a string exists, and the thing that decides whether the header is worth anything is what is inside the string. A source of scanned contracts scores a perfect hundred on it, and produces headers such as Agreement > 4.2 > (b) that put no retrievable word into the embedded text. A defence that returns the same answer for a path made of section titles and for Agreement > 4.2 > (b) is not a defence; it is a field presence check wearing one.

The right-hand column is the repair, and it is why the measurement has three columns rather than two. It applies a test to the contents of the path instead of to its existence, and it is the only column in the block whose contracts cell is honest.

The population that decides it#

Three classes of chunk cannot be served by the template, and they are identifiable at ingestion rather than after the fact.

Sources with no heading structure at all are the largest class. Support tickets, chat transcripts, email threads, meeting notes, comment threads, and anything else authored as a conversation have no headings because their format has no headings. This is not a parsing failure to be fixed; there is nothing to parse. The template will emit the document title for every chunk and stop there.

Chunks whose referent sits in the preceding narrative are subtler and appear inside well-structured documents. A twelve-page section under one ### heading produces forty chunks that all share a header, and a chunk on page eight that begins "the second approach avoids that problem" has its referent three chunks back, in prose, not in any heading. The heading path is correct and identical for all forty, so it separates the section from other sections and does nothing within it.

Documents where the section path was never extractable are the third class, and they are the reason the contracts column reads 100% and 4%. A scanned PDF that has been through OCR — optical character recognition, the step that reads characters off a page image and returns them as a stream of text — has text and no structure; whatever the layout parser inferred from font size produced paths like Agreement > 4.2 > (b), which is syntactically a heading path and semantically nothing. Signed contracts, standards documents, and regulatory filings all live here, and the numbered structure that makes them precise for lawyers makes them opaque to an embedding model.

Detecting these is a predicate over the chunk and its document, cheap enough to run inline:

STRUCTURAL = {"overview", "details", "notes", "introduction", "general",
              "appendix", "misc", "other", "summary", "background"}

NUMBERED = re.compile(r"^\(?[\dIVXivx]+[.)]?([.\-][\dA-Za-z]+)*\)?$")


def path_is_informative(section_path: list[str]) -> bool:
    for segment in section_path:
        s = segment.strip().lower()
        if not s or NUMBERED.match(s) or s in STRUCTURAL:
            continue
        if len(s.split()) >= 1 and any(c.isalpha() for c in s):
            return True
    return False


def needs_generated_header(chunk, doc) -> bool:
    if doc.source_type in NO_STRUCTURE_SOURCES:      # tickets, chat, email
        return True
    if not path_is_informative(chunk.section_path):  # numbered or empty paths
        return True
    if chunk.siblings_under_same_path > 12:          # long undifferentiated run
        return True
    return False

Reading path_is_informative as a piece of logic explains why it answers the question the coverage check could not. It walks the segments of a path and continues past every segment that is empty, that matches the NUMBERED pattern, or that appears in the STRUCTURAL stop-list of words like "overview" and "appendix". Those three tests name the three ways a segment can be present and carry nothing. Any segment that survives all three and contains an alphabetic character causes an immediate return True. So the predicate is an existence test over segments — one informative segment anywhere in the path is enough — and it returns False only when every segment has been skipped. On Agreement > 4.2 > (b), the word "Agreement" survives, which is the correct answer for a path whose first segment is a real word and the honest limit of the test: it judges paths segment by segment and cannot tell that the segments doing the discriminating are the numbered ones.

The third condition in needs_generated_header is the crude form of the preceding-narrative case: rather than trying to detect dangling references, count how many chunks share the path and treat a long run as evidence that the path is not discriminating within itself. On the corpus above, that predicate routed 34% of chunks to generation, and the composition was almost entirely tickets and contracts with a small tail from long product-doc sections.

The order of the tests in that diagram is a design decision rather than an accident. The source-type test comes first because it is the cheapest and the most certain — a ticket has no headings by construction, so its path is not worth examining. The sibling count comes last because it is the only one needing a count over the whole document. Each arrow reaching generate header is a separate reason, and a chunk needs only one of them.

Routing is what makes the numbers survivable. Generating for all 2.4 million chunks and generating for 816,000 differ by a factor of three on the ingestion bill, and by more than that on the nightly re-ingest, because the sources with structure are also the ones that churn most.

Weighing the two against the same criteria#

The rows split cleanly into two groups, and noticing the split is most of what the table is for. The coverage and discrimination rows all run the same direction: generation wins wherever the template has nothing to work with. The cost, throughput, determinism and auditability rows all run the other way. Not one row has both mechanisms good or both bad, which is what it means to say the two are complements rather than competitors — and it is the structural reason a router is the right shape for the answer, rather than a winner.

The determinism row is the one that bites teams late. A generated header is a sample from a model, so re-ingesting an unchanged document produces a different header, a different embed input, and a different vector — which means a nightly re-ingest rewrites the whole index for documents that did not change, and any before/after comparison you run is comparing two samples as well as two designs. The fix is a cache keyed on a hash of the chunk text plus the document version, so an unchanged chunk reuses its stored header and never calls the model again.

The size of that mistake is a single division, and it is worth doing because the result is exact rather than approximate. If 4% of documents change on a given night, a correct incremental re-ingest re-embeds 4% of the vectors. A non-deterministic header re-embeds 100% of them. One hundred divided by four is twenty-five, with nothing left over, so the recurring embedding volume is twenty-five times what the plan assumed. That cache is not an optimisation you add later; without it, "4% of documents changed" turns into "100% of vectors rewritten" and the recurring cost in chapter 8 is wrong by a factor of twenty-five.

Where a generated line goes wrong#

Generation is not a summary of the chunk. The prompt asks where the chunk sits and what it is about in one line, and a model given latitude will happily produce four sentences restating the chunk's content, which adds tokens, adds nothing the chunk did not already contain, and — where every chunk in a section gets a similar restatement — reintroduces the sibling-collapse problem from chapter 2 in a subtler form. Cap max_tokens and say "context line only" in the prompt for exactly this reason.

Its failure mode is also different in kind from the template's, and worse to detect. A template that fails produces an obviously empty or obviously numeric header, which the predicate above can spot. A generation that fails produces a fluent, plausible line describing something the chunk is not about — most often when the chunk is a table fragment, a code listing, or boilerplate, and the model reaches into the surrounding document for a topic to attach to it. A header reading "this section describes the refund window for enterprise customers" on a chunk that is actually a currency-code lookup table will retrieve confidently for refund-window queries and be useless every time.

The asymmetry there is worth naming, because it decides where review effort goes. A template's failures are detectable by a program, since "empty" and "purely numeric" are properties of a string. A generation's failures are detectable only by comparing a claim against a chunk, which is a judgement about meaning and has no cheap mechanical form. Sampling a few hundred generated headers and reading them against their chunks is the only cheap check, and it is worth doing once per source type rather than once overall — because the way a model goes wrong on a scanned contract has nothing to do with the way it goes wrong on a chat transcript, and a single pooled sample hides both.

Generation is not a repair for chunking either. Topic 9.5 set the boundaries, and a header describes a chunk; it cannot merge one with its neighbour or move a boundary. If the evidence for an answer straddles two chunks, generation produces two well-contextualised halves.

And selective generation is not free even where it applies. The predicate above is code that has to be maintained, its NO_STRUCTURE_SOURCES set has to be updated when a new connector lands, and a change to the predicate reclassifies chunks and therefore changes vectors. That last consequence is the one to hold on to: reclassifying a chunk swaps its header, which swaps its embed input, which swaps its vector — the same chain that made the template's determinism valuable, running in reverse. Treat the predicate as part of the index schema, versioned with the rest, rather than as a heuristic somebody tunes in place.

The template and the model are answers to two different questions that a corpus of mixed sources asks at once, and the work that makes the choice tractable is the counting: run the cheap mechanism over everything first, and separate the chunks where it produced words from the chunks where it produced an empty string or a section number. That measurement needs no model calls, and it converts an argument about price into a statement about how many chunks have no alternative.

Both mechanisms produce one thing: a string that goes into the embedded text. That is a narrower output than it sounds, and the next chapter is about what it excludes. A header can make a chunk retrievable for "what does the returns policy say about damaged goods" and can do nothing whatsoever for "what changed in the returns policy since January", no matter how good the line is, because the two queries consume different halves of the query path.