A chunk's subject often lives in the heading chunking threw away

A chunk cut out of a document keeps every word the author wrote inside it and loses every word standing above it. Most of the time that costs nothing. Where it costs something, the damage is invisible from the outside: the chunk is still fluent, still accurate, still a well-formed statement about something, and the something left with the heading. Two repairs answer that, both of them made in the same few lines of an ingestion pipeline, between the splitter and the call that embeds a batch. This chapter is about the one that changes the text.

Naming the machinery first makes the rest of the argument checkable. A retrieval-augmented generation system, or RAG, answers a question by searching a corpus of a company's own documents and handing the passages that come back to a language model, which writes the answer out of them. An embedding model turns a string into a vector, a few hundred to a few thousand numbers arranged so that texts about similar things land near one another; every chunk is turned into one at ingestion time, the query is turned into one when it is asked, and the store returns the chunks lying nearest to it. Nothing else about the document travels with the chunk. A contextual chunk header is the repair this chapter builds, and it is as literal as the name suggests: a short line naming the document and the section a chunk came from, written onto the front of that chunk's text before the text goes to the embedding model. The other half of the topic is metadata design — the document id, the revision a chunk came from, an effective date, a tenant, a reference to the access rules the source system holds, all stored as fields beside the vector rather than written into it, where a query can filter and sort on them and where no similarity computation ever reads them.

A paragraph that stops meaning anything when it is cut out#

Take a section of an employee handbook, written in markdown the way most internal documentation is:

# Employee Handbook 2024

## Leave

### Parental leave

Eligible employees with twelve months of continuous service may take up to
26 weeks of parental leave, of which the first 13 weeks are paid at full
salary and the remainder at statutory rate.

It must be requested at least 30 days in advance, and cannot be split
across more than two separate periods.

A recursive splitter walks that document, treats the blank-line-separated paragraphs as its finest split boundary, and emits the last paragraph as a chunk in its own right because it fits comfortably inside the window. Nothing in that sequence is a bug. The heading ### Parental leave was consumed three chunks earlier and attached to whichever piece of text happened to follow it, which is what a splitter does with a heading — it is a line of the document like any other, and it goes with the chunk it falls into. So the chunk that holds the notice period, in full, in the vector store, reads:

It must be requested at least 30 days in advance, and cannot be split
across more than two separate periods.

Read that with no surrounding context, which is the only way anything downstream will read it. Those twenty words are the entire input, and the pronoun that opens them has no referent anywhere inside them.

What the embedding model was given to work with#

An embedding model does not have access to the document. It is handed a string and returns a vector, and everything it can express about that chunk has to be recoverable from the string it was handed. That is the property to hold on to, because every consequence below follows from it mechanically rather than statistically.

Follow the chain one step at a time. The string names no subject, since the only noun phrase that would have supplied one is the heading and the heading is not in the string. What the string does contain is a complete and well-formed statement about scheduling: something, unnamed, requires thirty days of advance notice and may be divided into at most two periods. The model therefore encodes the thing it was actually given, which is a scheduling constraint. A vector encoding a scheduling constraint lands among other vectors encoding scheduling constraints. That is the whole causal path, and no step in it is a failure of the model — the vector is a perfectly good vector for the text it was built from. The text is about nothing in particular.

The neighbourhood that vector lands in is the part worth looking at. In the corpus this handbook belonged to — 41,000 chunks across HR policy, IT policy, procurement, and a large body of facilities documentation — there were a great many other chunks about advance notice. Room bookings for the largest meeting rooms required fourteen days. Contractor access badges required ten business days. Non-standard purchase orders over £25,000 required thirty days at the procurement committee and could be resubmitted once. The parental leave chunk sat in the middle of that neighbourhood, and it had nothing in its text to distinguish it from any of them.

The measurement bore that out. Embedding the query how far ahead do I have to request parental leave? and scoring it against the whole index gave the parental leave chunk a cosine similarity of 0.31 and rank 178. Cosine similarity is the score these stores rank on: it measures the angle between two vectors, reaching 1 when they point the same way and falling towards 0 as they stop having anything to do with each other. The top five results were, in order, two procurement chunks about the thirty-day committee window, a facilities chunk about advance booking of the auditorium, and two paragraphs from a different part of the handbook that did mention parental leave but said nothing about notice periods — those two scored on the word "parental", which is the only lexical purchase the query had anywhere in the index.

The word "parental" is the whole story. The query and the chunk share no vocabulary and no subject. Every piece of information that would have connected them was in the heading, and the heading was thrown away by a step that had no idea it was load-bearing.

Those five results are worth reading as two groups rather than as five disappointments, because they sort cleanly and the sort is the diagnosis. Three of them match the query on notice periods and have nothing to do with parental leave. Two of them match on parental leave and have nothing to do with notice periods. Not one of the five carries both halves of the question, and the one chunk in the corpus that carries both is the chunk sitting at rank 178 — it holds the notice period in its text and the subject in a heading that was thrown away. The ranking is behaving correctly on the axis it can see. Each candidate above the gold chunk beat it on one half of the query while the gold chunk was competing on one half too, having been stripped of the other. A rank in the hundreds out of 41,000 is not a near miss caused by a slightly wrong vector; it is the position a text earns when the query has two requirements and the text can only answer to one of them.

There is a reframe here that carries more weight than a metaphor would. The heading is the paragraph's grammatical subject, parked one line above it. English lets a writer do that — write a heading, then write three paragraphs whose subjects are all pronouns pointing back at it — and the convention is so ordinary that it is invisible until a machine reads one of those paragraphs alone.

An analogy, offered as an analogy rather than as a mechanism: the orphaned chunk is a line of transcribed dialogue with the speaker's name stripped off. The line is still grammatical, still true, still says something, and you cannot tell whose position it states. Where the analogy breaks is worth naming, because it breaks in the direction that matters. A human reading unattributed dialogue can often recover the speaker from style, vocabulary or what was said three lines earlier; an embedding model reading a chunk has no such channel, because the three lines earlier are in a different chunk with a different vector, and nothing at query time consults them. The recovery route the analogy suggests does not exist here.

Writing the heading path into the text that gets embedded#

The repair is to make the embedded text carry the context that the document structure was carrying implicitly. Before the chunk goes to the embedding model, prepend the document title and the heading path that the chunk sits under:

Employee Handbook 2024 > Leave > Parental leave

It must be requested at least 30 days in advance, and cannot be split
across more than two separate periods.

That string is what gets embedded. Trace the same causal chain through the new string and every link changes. The string now names a subject, because "Parental leave" is inside it. A model encoding that string encodes a statement about parental leave that also carries a notice period. A vector encoding both halves of the question lands near queries that ask about both halves. The chunk stops competing on one axis and starts competing on the axis the query actually occupies, and it is the only chunk in the corpus positioned to win on that axis.

The size of the shift follows from that rather than from any tuning. The old vector was not nearly right and slightly misplaced; it was answering a different question competently. A change that swaps one subject for another moves a vector by roughly the distance between two subjects, and that distance is large.

The chunk's stored text can stay as it was, or can include the header as well — the two are separate decisions, and chapter 4 of this topic, on which fields are embedded and which are only filtered, is where the separation matters.

Carrying the heading state through the splitter#

Nothing about this requires a model, a service, or an extra pass over the corpus. It requires the splitter to keep a heading stack as it walks the document and hand it to each chunk it emits, which is a fifteen-line change to code that already exists after topic 9.5's chunking work:

from dataclasses import dataclass, field


@dataclass
class Chunk:
    text: str                      # the chunk as it appeared in the source
    doc_id: str
    doc_title: str
    section_path: list[str] = field(default_factory=list)

    @property
    def header(self) -> str:
        return " > ".join([self.doc_title, *self.section_path])

    @property
    def embed_input(self) -> str:
        return f"{self.header}\n\n{self.text}"


def chunks_with_context(doc_id: str, doc_title: str, markdown: str) -> list[Chunk]:
    stack: list[str] = []
    out: list[Chunk] = []

    for block in parse_markdown_blocks(markdown):
        if block.kind == "heading":
            # a level-3 heading replaces everything from depth 3 down
            stack = stack[: block.level - 1] + [block.text]
            continue
        for window in split_to_window(block.text):
            out.append(Chunk(
                text=window,
                doc_id=doc_id,
                doc_title=doc_title,
                section_path=list(stack),
            ))
    return out

The single line doing the structural work is the reassignment of stack, and it is worth reading slowly because it encodes the whole of markdown's nesting rule in one slice. A heading at level n replaces everything from depth n downwards and leaves everything above it alone, which is what stack[: block.level - 1] takes — the segments shallower than the incoming heading — before appending the incoming heading's own text. The subtraction of one is the conversion between two counting systems: markdown numbers its heading levels from one, and a Python slice numbers positions from zero, so a level-3 heading keeps the first two segments and becomes the third.

Watch it run and the three cases it has to get right fall out of that one line. The chunks under ### Parental leave get ["Leave", "Parental leave"], since the ## Leave above them was written into the stack first and survives the slice. A chunk under a later ### Bereavement leave gets ["Leave", "Bereavement leave"], because a heading at the same level as the previous one keeps the same number of shallower segments and swaps its own slot. A chunk under a subsequent ## Expenses gets ["Expenses"] with the leave path correctly dropped, because a level-2 heading keeps only the first segment's worth of what came before, and there was none. Getting the third case wrong is the common failure, and it is the expensive one: a stack that only ever appends gives every chunk in the second half of a document a header claiming it is still about leave.

The embedding call then takes embed_input rather than text:

vectors = embed([c.embed_input for c in batch])

That single substitution is the entire retrieval change. Note what it does not touch. The stored text is unchanged, so citations, snippets and anything rendered to a user are byte-identical to before. The vectors are the only artefact that moves, which is what makes the change cheap to try and cheap to roll back — a re-embed of the corpus, with no migration of anything a person sees.

Three small decisions inside the line#

The shape of the header string looks like a matter of taste and has two parts that are not. The first is where it goes. Putting it before the chunk rather than after protects it from tail truncation — an embedding model given more tokens than it accepts drops the end of the input, so a header appended at the bottom is the first thing lost on exactly the long chunks that most need it. Leading position also gets weighted a little more heavily by most encoders, which is a smaller effect and points the same way.

That first one is a requirement rather than a preference, and the distinction is worth keeping straight because it decides what is open to argument later. The model's input limit is imposed from outside; given that limit, truncation removes the tail; given that it removes the tail, a header at the tail is lost on the longest inputs. Nothing about the corpus or the use case can change that chain, so leading position is forced. The two decisions below are genuinely decisions, made against this corpus, and a different corpus could reasonably decide them differently.

The second is depth. A wiki with six levels of nesting produces paths like Engineering Handbook > Platform > Services > Ingest > Configuration > Retries, which is 14 tokens of mostly redundant hierarchy, and the leading segments are the least useful within a document because every chunk in the document shares them. Truncating to the document title plus the last three headings keeps the discriminating end of the path and drops the part that is constant, and it bounds the header length so that a deeply nested document does not quietly get a header three times longer than a flat one.

That token figure is the first place in this topic where a habit of counting words instead of tokens gives a wrong answer, and it is worth stopping on. There are fewer words on that line than the figure quoted for it. The count is in tokens, and a tokenizer charges separately for the separators between the segments and splits any word it does not hold whole into pieces, so a path's token count runs ahead of its word count rather than tracking it. Every budget in this topic is denominated in tokens, and a header sized by eye against its word count will come out short of what it actually costs.

The depth rule trades something away, and naming the trade is what makes it a decision. Dropping the outer segments discards exactly the part of the path that would separate one document from another, and keeps the part that separates one section from another within a document. That is the right trade only because the document title is retained on its own, and it does the between-document job by itself. Remove the title from the rule and the same truncation would be a bad one.

The third genuinely is taste, within limits. Employee Handbook 2024 > Leave > Parental leave and Section "Parental leave", under "Leave", in the Employee Handbook 2024 both work, and prose phrasing scores marginally better with models trained mostly on prose, at the cost of a few more tokens. What matters more than the choice is applying it uniformly: two documents ingested by two connectors with two header formats introduce a difference between their chunks that has nothing to do with their content, and that difference is in the vector.

Uniformity is a requirement hiding among the preferences. Any string prepended to a chunk is read by the model as part of the chunk, so a formatting difference between two connectors is a content difference as far as the vectors are concerned. The store then has a dimension of variation in it that corresponds to which pipeline ingested a document, and every query is scored against that variation along with everything else.

Which chunks this moves and which it leaves alone#

Prepending helps chunks whose meaning depends on something outside their own boundaries, and does close to nothing for chunks that already name their subject. Sampling 300 chunks from that 41,000-chunk corpus and classifying them by hand gave a rough distribution that has held up on other corpora since:

The share column closes exactly. Add the six percentages and they come to one hundred with nothing left over, which means there is no seventh row hiding in an "other" bucket. That is worth more than it looks: an unclassified remainder is where a distinct failure mode usually lives, and its absence licenses reading the table as a complete partition of the sample rather than as the four or five patterns somebody happened to notice.

A second identity sits underneath the first. Turn each share back into a count of chunks out of the 300 sampled and every one of the six is a whole number of chunks, and the six counts add up to 300 exactly. Percentages that survive that round trip are counts re-expressed, not estimates rounded to the nearest point, so the table is reporting what was actually found rather than a smoothed version of it.

That licenses one reading and forbids another, and the difference matters. It licenses arithmetic on the rows, so the four rows where a header helps strongly or moderately can be added straight off the table: roughly a third of the sample was not self-describing, and that third is where retrieval was failing. It does not license treating those shares as properties of the corpus. Three hundred chunks were drawn from 41,000, and a share measured on a sample carries sampling error that this table does not report and that no amount of exactness in the counting removes. The chapter's own hedge — a rough distribution — is the honest reading, and the figure to trust is the order of magnitude of that third rather than its final digit.

Reading the rows as gaps rather than as levels puts the mechanism on display. The top three rows share one property: each is a chunk whose subject is grammatically absent, whether as a pronoun, an elided imperative subject, or a list item severed from its lead-in. All three are rated Strongly. The fourth row's subject is present but underspecified — "the limit" names a category and not which limit — and it drops to Moderately. The fifth row's subject is named outright, and it drops to Barely. The gradient runs with how much of the subject is missing from the string, which is exactly what the causal chain above predicts, and it is the strongest evidence in the chapter that the mechanism is the one being claimed rather than a general benefit of longer inputs.

The last row breaks that pattern and is the more interesting for it. Its chunks are self-contained by the same test as the fifth row, and the header still helps moderately. What the heading adds there is not a missing subject but a discriminator — a product tier that separates this chunk from a near-identical chunk under a different tier. Those are two different jobs a header can do, and only the first is about repairing broken reference.

The 58% that already name their subject still get a header, and the header is not free for them — chapter 8 of this topic accounts for what it costs to prepend forty tokens to a chunk that did not need them — but the cost is small and the alternative of deciding per chunk is a classifier you have to build, evaluate and maintain.

The second-order effect is worth knowing about. Prepending also helps lexical retrieval — matching on the literal words a query and a chunk share, scored by a function such as BM25 — if you run a hybrid index, meaning one that searches vectors and words in parallel and combines the two result lists. It helps because "parental leave" is now literally present in the chunk's term frequencies. That is a real gain and it arrives for free, but it is not the mechanism being taught here, and a system that gets its whole improvement from the BM25 side has probably learned that its embedding model was doing less work than assumed.

Split evidence, vocabulary gaps and empty headings#

A header does not rejoin evidence that chunking separated. If the eligibility rule is in chunk 7 and the notice period is in chunk 9, and answering the question requires both, then adding Employee Handbook 2024 > Leave > Parental leave to each of them makes both retrievable and leaves the answer split across two results. Whether that is fine depends on whether your top-k — the number of chunks a search is asked to return, k being the number you pass to the store — is wide enough, and on whether the generator, meaning the model that writes the answer out of the retrieved chunks, can combine them. It is a different problem with a different fix, and reaching for headers when the failure is actually a chunk-boundary failure is how teams end up spending real money to move recall by two points.

Telling the two apart before spending anything is easy and is the check to run first. A reference failure and a boundary failure leave different traces: in a reference failure the chunk that answers the question is a single chunk and it is ranked far down, while in a boundary failure the answer is not in any single chunk at any rank. Look for the gold chunk in the full ranking rather than in the top-k. If it exists and is buried, a header is the right tool. If no single chunk would satisfy a reader who read it alone, no amount of context on the front of it will help.

A header does not close a vocabulary gap. If the query says "mat leave" and neither the heading nor the chunk ever says "parental", the header adds no bridge, because it can only surface terms the document already used. The same holds for internal jargon, product codenames that users do not know, and translated corpora where the query language differs from the document language. Those are problems for the embedding model, for query expansion, or for a synonym list on the lexical side of a hybrid index, and a header improves none of them.

A header does not rescue an uninformative heading. A path of Vendor Agreement > Section 4.2 > Subsection (b) restores no subject at all, because "Section 4.2" means nothing to an embedding model and, unlike a human reader, the model cannot go and look it up. Legal documents, standards, and anything auto-numbered are full of these. So are documents where the author used headings as visual dividers — a page whose sections are titled "Overview", "Details" and "Notes" has a heading structure that is syntactically perfect and semantically empty.

That case is worth stating as a general property rather than as a list of document types, because the general property is what predicts the next case. The header carries whatever discriminating power the heading already had. Where the heading discriminates, the header does; where the heading is a serial number or a visual divider, the header inherits its emptiness exactly, and the ingestion pipeline reports success either way. Nothing in the code above can tell the difference, which is why chapter 3 of this topic is about identifying that population in advance.

And a header is not a summary of the chunk. The header says where the chunk sits, and deliberately says nothing about what the chunk contains, because what it contains is already in the chunk. That distinction is the one a reader is most likely to collapse, and collapsing it produces a sentence that sounds like the lesson of this chapter and is false: adding context to a chunk improves retrieval. What this chapter has shown is narrower. Adding the context that tells a chunk apart from the chunks it will be ranked against improves retrieval, and the heading path qualifies because it changes from section to section. Context that is identical on every chunk of a document cannot tell those chunks apart, no matter how accurate or relevant it is, since a quantity that does not vary cannot explain a difference. It is tempting to believe otherwise, because the added text is true, on topic, and written by a human — every quality check a person would think to apply comes back clean.

A header is a claim about position, then, and it earns its keep exactly where position is informative. Where a heading names a subject the prose below it stops repeating, prepending that heading puts the subject back into the only string the embedding model reads, and a chunk that was competing on half a query starts competing on all of it. The change is a line: build the string from the document title and the heading stack, embed that instead of the raw text, leave the stored text alone.

The measured shape of the benefit is the part to carry forward. Roughly a third of a documentation corpus is not self-describing, that third is where retrieval quietly fails, and the remaining chunks pay a small fixed token cost for a uniform rule that is cheaper than deciding per chunk. What the rule cannot do is supply discrimination the heading never had, and it does real harm when the prepended text is the same on every chunk in a document. That failure is the subject of the next chapter, where a team reasoned that if a little context helps, the document's own abstract must help more, prepended the full 400-token abstract to all 900 chunks of every document, stayed well inside the embedding model's token limit, and made retrieval measurably worse.