Heading-aware splitting

Heading-aware splitting — also called structure-aware splitting — cuts a document where the document says its own sections begin, instead of every 480 tokens. Each chunk then carries a breadcrumb: the chain of headings standing above it, from the document's title down to its own section. A chunk is no longer a window onto some text; it is a named part of a named document.

That naming is most of what the strategy is worth, and it is also the reason it can fail badly. A boundary without a label is just a cut. A boundary with a label is a cut plus a claim about what is inside it, and everything downstream consumes that claim: metadata filters match on it, citations display it, rerankers weigh it, and a heading prefix pastes it into the embedding input. A strategy that produces claims is a strategy that can lie, and this chapter is mostly about when it does.

The phrase "where the document says its own sections begin" is doing more work than it looks, and unpacking it is the whole chapter. A file can say that in two quite different ways. In the first, a person opened an editor, selected a line, and applied a heading style; the file format then stored that decision as a mark distinct from the surrounding text — an <h2> element, a ## prefix, a DOCX heading style. Reading the heading back out of such a file is not inference. It is retrieval of a decision somebody made. In the second, the file stores only appearance — this line is fourteen points, bold, and alone — and a layout model looks at the appearance and guesses that a person must have meant a heading. Those two situations produce identical-looking breadcrumbs and completely different error rates, and nothing about a chunk record tells them apart unless somebody chose to record which one it came from.

What the breadcrumb buys#

Take a page of an expense policy, exported from Confluence as HTML with real <h2> and <h3> elements that somebody typed as headings in an editor. Split on those headings and a chunk looks like this:

{
  "id": "conf-8841-s3",
  "text": "### 3.1 Deadlines\nClaims must be filed within 60 days of the transaction date…",
  "breadcrumb": ["Expense Policy", "3. Submitting a claim", "3.1 Deadlines"],
  "section_id": "3.1",
  "parent_section_id": "3",
  "n_tokens": 190,
  "structure_source": "html_heading_element"
}

The text field starts and ends where the author decided a subject started and ended. The breadcrumb is true, which means a metadata filter for breadcrumb contains "Submitting a claim" returns what a person would expect, and the parent-section expansion of chapter 4 — serving a chunk together with the section above it — has a correct parent to expand to.

Both of those benefits run through the same three-step chain. A heading element in the source is a mark a human applied to a specific line; the splitter copies that mark into the breadcrumb field of every chunk under it; downstream components then read breadcrumb as though it were a description of the chunk's content, because it is a short, high-signal string sitting where a description would sit and nothing in the record marks it as an inference. The breadcrumb therefore inherits, exactly, whatever confidence the original mark deserved. When the mark was typed by a person, the inheritance is sound. When the mark was guessed, the same three steps launder a guess into a field that reads like a fact.

What a wrong heading costs twice#

Now take a scanned manual: no markup at all, so a layout model infers the headings from font size and weight.

On page 214 of a scanned inverter manual, the line "Do not power-cycle the unit during firmware transfer" is set in 14-point bold on its own line, because the technical writer wanted it to stand out. Every real heading on that page is 14-point bold on its own line. The model classifies the warning as a heading and the splitter cuts there.

That misclassification is not carelessness on the model's part, and seeing why matters more than the individual error. The model's only input is appearance. On this page, appearance does not distinguish the two cases, because the writer deliberately gave an emphasised warning the same visual treatment as a heading — that is what emphasis is. A classifier cannot separate two classes that its input renders identical. So the error rate is set by how often a document's typography reuses heading-like appearance for something that is not a heading, which is a property of the document's author rather than of the model, and no amount of retraining removes it.

The chunk before the cut ends after step 4 of a seven-step procedure. The chunk after it contains steps 5 through 7 and carries the breadcrumb ["Inverter Manual", "6. Firmware update", "Do not power-cycle the unit during firmware transfer"]. Both halves of the damage matter, and they compound. The procedure is severed, so a query about step 6 retrieves a chunk with no idea what is being updated or why. And the label is a lie: the chunk now advertises itself as a section about not power-cycling the unit — which is exactly the text a filter matches on, a citation renders, and a heading prefix embeds. A false boundary in a labelled scheme does not degrade gracefully into a fixed-size cut. It produces a mislabelled cut, which is worse, because the label is trusted.

The tempting belief here is that heading-aware splitting has a floor: that when the headings are poor it stops being clever and quietly becomes something like fixed-size cutting, so the downside is bounded at whatever a blind splitter scores. That reading is natural, since a wrong boundary and an arbitrary boundary both sever text at an unhelpful point, and a fixed-size splitter severs text constantly without anybody calling it broken. What makes it false is the second output. A fixed-size splitter emits a cut and nothing else, so a bad cut costs one retrieval. A heading-aware splitter emits a cut and a string that every downstream consumer treats as a description, so a bad cut costs one retrieval plus every filter, citation and embedding that string touches. The strategy has no floor at the blind splitter's score, because the blind splitter never had the second failure mode available to it.

The inverse error is quieter and larger. When a genuine heading is set in the same size as body text, or the OCR loses the line break before it, the model misses it and two or three sections merge into one chunk. Merged chunks of 2,400 to 4,100 tokens sit well over the 512-token encoder ceiling of chapter 3, so they are indexed by their opening section alone while carrying two further sections of text that are served but never described in the vector. One missed heading reproduces the runbook failure exactly — on the strategy that was chosen because it looked like the careful option.

Those two errors are worth holding apart, because they are visible at different places in the system. A false boundary announces itself in the metadata: the breadcrumb contains a sentence, and sentences do not look like section titles. A missed boundary announces itself only in the token count, and only if somebody records one. The chunk with a merged body has a correct-looking breadcrumb, a correct-looking parent, and three sections of text behind a vector describing one of them.

Measuring the structure signal#

"Are the headings reliable" has an answer you can compute, and computing it is a day of work rather than a research project. Sample 200 pages stratified across source formats — a fixed quota from each format rather than a random draw, so the formats that are rare but badly handled are represented — have someone mark the true headings, and score the parser's inferred headings against them as a detection problem.

The stratification is a design decision and not a requirement, and the difference matters when someone asks why the sample was not simply random. A random draw of pages from a real corpus is dominated by whichever format supplies the most pages, so the formats with the worst signal can arrive as a handful of pages with no usable rate attached to them — which is precisely the information the exercise exists to produce. A fixed quota per format buys a usable per-format rate and gives up the ability to read a corpus-wide average straight off the sample, since the sample no longer has the corpus's format mix in it. That trade is the right way round here, and it is a choice rather than a consequence.

def heading_scores(gold: list[set[str]], pred: list[set[str]]) -> dict[str, float]:
    tp = sum(len(g & p) for g, p in zip(gold, pred))
    fp = sum(len(p - g) for g, p in zip(gold, pred))
    fn = sum(len(g - p) for g, p in zip(gold, pred))
    return {
        "precision": tp / (tp + fp),   # of the cuts we would make, how many are real
        "recall":    tp / (tp + fn),   # of the real sections, how many we would find
        "pages_clean": sum(g == p for g, p in zip(gold, pred)) / len(gold),
    }

Read that code as arithmetic on sets and it stops being boilerplate. gold is a list with one entry per sampled page, and each entry is the set of heading lines a person marked on that page; pred is the same shape, holding the set the parser would cut at. For one page, g & p is the intersection — the headings present in both sets — so tp, true positives, counts headings the parser found that really are headings. p - g is set difference, the predictions with no match in the gold set, so fp, false positives, counts cuts the parser would make at lines that are not headings. g - p is the difference the other way, so fn, false negatives, counts real headings the parser walked past. Summing each of those across pages with zip, which walks the two lists in step so page i of one is compared against page i of the other, gives three corpus totals.

Every marked line on every sampled page falls into exactly one of those three buckets, and that is what makes the two rates readable. A line present in both sets is a true positive; present only in the prediction, a false positive; present only in the gold set, a false negative; present in neither, it is not counted at all, because it is one of the many body-text lines nobody claimed was a heading. There is no fourth bucket and no residual. So tp + fp is exactly the number of cuts the parser would make, and tp + fn is exactly the number of real sections there were. Dividing the same numerator by those two different denominators is what produces two numbers rather than one, and it is why they answer different questions: precision is a statement about the cuts you would make, recall a statement about the sections that exist.

The two rates do different jobs, and conflating them is how a mediocre signal gets approved. Precision governs false boundaries: 0.68 means roughly a third of the cuts are not section boundaries, so a third of the chunks are severed mid-thought and mislabelled. Recall governs missed sections, which is the oversized-chunk failure. The third figure — the fraction of pages where the inferred set matches the gold set exactly — is the one to quote to anyone deciding whether to route a corpus this way, because a page with any error on it produces at least one bad chunk.

The third figure is not a restatement of the first two, and the reason is that it is computed with a different operator. Precision and recall average over headings and let a page's successes offset its failures. g == p averages over pages and applies a logical AND to everything on the page: one extra heading or one missing heading turns the whole page's contribution to zero, whatever else went right on it. So the same pair of per-heading rates is consistent with two very different corpora — one where the errors are spread thinly across almost every page, and one where most pages are perfect and a minority are a mess — and those two corpora need different decisions. The first argues for abandoning heading-aware splitting on that format. The second argues for detecting the bad pages and routing only those. Precision and recall cannot tell them apart, and the page-level rate can.

Two things in that table are worth taking seriously. The gap is not between formats but between derived and inferred structure: wherever a human explicitly marked something as a heading and the format preserved that mark, the signal is near-perfect; wherever the mark was visual and has to be reconstructed, it is somewhere between adequate and poor. And "PDF" is not one row. A born-digital PDF exported from a documentation toolchain carries an outline; the same-looking PDF produced by printing a Word document with manual bolding does not, and the two arrive in the same directory with the same extension.

The way to read that table is by its gaps rather than by its levels, and the second column is what the gaps sort on. Every row whose structure signal names an explicit mark — heading elements, ATX prefixes, a PDF structure tree, DOCX heading styles — sits in the high nineties on precision and no lower than the high eighties on pages fully correct. Every row whose signal is "font size and weight" sits in the sixties, seventies and low eighties, with pages fully correct between a third and two thirds. There is nothing in between, and that is literally true rather than loosely true: in each of the three score columns, the lowest derived row still sits above the highest inferred row, so the two groups do not overlap anywhere. The empty band between them is wider, in every column, than the entire spread of the derived group. A table that behaves that way is not measuring a continuum of format quality; it is measuring one binary property with noise on top, and the binary property is whether a human's decision survived into the file.

The two DOCX rows make that sharper than any average could. Same file format, same parser, opposite ends of the table, because one author applied heading styles and the other pressed the bold button. The extension is held constant while the signal quality moves by a wide margin, which is as close to a controlled comparison as a table like this offers, and it says file type is the wrong thing to condition on.

Routing by provenance, not by hope#

The extraction step knows where its structure came from, and that provenance is the field to route on. It is not a guess or a confidence score the layout model invents about itself; it is a fact about the source document that the extraction step already had in hand and usually throws away.

The distinction is worth being precise about, because "route on a confidence score" is the obvious alternative and it fails for a structural reason rather than an empirical one. A layout model's confidence is produced by the same model, from the same input, that made the classification. On the inverter page above, the warning line genuinely looks like a heading in every feature the model can see, so the model is confidently wrong, and confidence is high exactly where the error is. Provenance is produced by a different part of the system reading a different property of the file — did this line arrive carrying an explicit mark, or not — and its answer does not depend on how convincing the line looks. Two independent instruments beat one instrument grading its own work.

VERIFIED = {"html_heading_element", "markdown_atx", "docx_heading_style",
            "pdf_tagged_outline"}

def split(doc) -> list[Chunk]:
    if doc.structure_source in VERIFIED:
        return heading_aware(doc, token_ceiling=480)
    return recursive(doc, size=480, overlap=64)

The four strings in VERIFIED are exactly the four table rows whose structure signal names an explicit mark, and nothing about that set is a threshold anybody tuned. It is the membership test "did a person's decision survive into the file", written out. That is the whole decision, and it means the corpus is chunked two ways on purpose.

The bottom row of that table is not an extra measurement, and checking why is the fastest way to see what a mixed-corpus average is. Average the two halves with equal weight — they are halves of a 12,000-document corpus, so equal weight is the right weight — and the fixed column gives back its stated aggregate exactly, and the recursive column gives back its stated aggregate exactly. Two numbers per column, one operation, no residual. The third column is the interesting one: averaging its two halves lands half a hundredth below the stated 0.62, which is what rounding to two places does to a value sitting exactly between them. That single rounding step is the entire margin by which heading-aware appears to beat recursive on the bottom row. Before rounding, the corpus-wide gap between the two strategies is half a point; after rounding it displays as one; and a migration was going to be funded on it.

What the identity licenses is a rule about which numbers carry information. If the bottom row can be reconstructed from the two above it by one arithmetic operation, then it contains nothing the two rows do not, and every disagreement between the halves has been averaged out of it by construction. The aggregate is not a summary of the evidence. It is a lossy re-encoding of two numbers into one, and the loss is exactly the part under dispute.

The same arithmetic explains the 0.66 in the last line of that comparison, which appears in none of the table's cells. A corpus routed by provenance runs heading-aware on the Confluence half and recursive on the scanned half, so the two numbers it actually produces are 0.74 and 0.58. Averaged with equal weight those give 0.66 exactly, with no residual at all — the number a routed system would report as its single headline figure. The two halves sit sixteen hundredths apart underneath it, and a headline figure that is the midpoint of a sixteen-point spread cannot say so about itself.

The token_ceiling argument in that code is not decoration. Heading-aware splitting produces chunks whose length is the length of the section, and section lengths have a long right tail: on the Confluence half, the median section ran 210 tokens and the 97th percentile 1,900. Without a ceiling, the three percent of sections that are long are the three percent that get half-indexed — and those are the substantial ones, the detailed procedure and the full troubleshooting matrix. Heading-first with recursive subdivision of any section that overruns is the combination that survives contact with a real corpus, and the subdivided pieces should inherit the parent's breadcrumb so the metadata stays true.

Those two percentile figures are worth pausing on, because the shape of the distribution is what forces the ceiling rather than the size of the average. A median of 210 tokens sits comfortably inside a 512-token encoder limit, and the mean of a long-tailed distribution sits somewhere above the median and probably still inside it. Neither statistic can tell you anything about the top of the distribution, and the top is the only part that overruns. The 97th percentile is the figure that decides the question, and it exceeds the encoder limit by more than triple. A ceiling is therefore a requirement imposed by the encoder, not a tuning parameter: without it, the sections most worth indexing whole are the sections indexed by their first few hundred tokens.

Where structure is the wrong thing to trust#

Perfect headings do not make heading-aware splitting correct everywhere. Three cases defeat it even on the half where the signal is real.

Documents whose sections are one or two sentences produce chunks too small to stand alone, which is the failure of chapter 4 by another route: a FAQ page with thirty ### questions becomes thirty 40-token chunks, each of which needs its neighbours to make sense. Merging consecutive short sections up to a floor, under the breadcrumb of their common parent, is the fix — and it is a second parameter that heading-aware splitting quietly requires. The strategy is offered as the one that reads boundaries off the document rather than out of a config file, and it arrives needing a ceiling above and a floor below — two numbers somebody has to choose and nothing in the document supplies.

Documents whose headings are navigational rather than semantic — "Overview", "Notes", "Appendix" — give correct boundaries attached to labels that carry no information. The boundary benefit remains and the metadata benefit evaporates. That is a perfectly acceptable outcome, and it is worth knowing before somebody builds a filter UI on top of section titles. It is also the one failure the measurement above cannot see, since a parser that finds "Notes" where a human marked "Notes" scores a true positive on every rate in the table. Heading detection and heading usefulness are separate properties, and only the first one has a score.

And no amount of heading fidelity touches the cross-reference problem. Section 3.1 and section 4.2 can both be correctly bounded, correctly labelled and correctly separate, which is precisely the situation that produced the wrong deadline in chapter 4. Structure-aware splitting makes boundaries honest; it does not make a document that points at itself self-contained.

There is a fourth case that is not a property of any document, and it is the one most likely to be running in production right now. A structure signal is measured once, on a sample drawn at one moment, and the resulting per-format rates are then treated as a standing property of the pipeline. Meanwhile the format mix moves: a team migrates a wiki, a vendor starts shipping scans instead of tagged PDFs, an extractor upgrade changes which rows a document lands in. The measurement is a defence that stays fixed while the thing it is meant to discriminate varies underneath it, which is the shape that makes a defence feel strongest exactly when it has stopped working — nobody re-runs a check that has never failed. The structure_source field is what makes this tractable: because it is recorded per chunk, the format mix is a query rather than an assumption, and a shift in it is visible without re-annotating anything.

Heading-aware splitting is not a better chunker. It is a chunker that trades a guess about where a fact ends for a claim about what a chunk contains, and that trade is only good when the claim is true. Whether it is true is a property of the source format, not of the corpus, and it is one you can measure in a day: sample per format, score precision, recall and pages-fully-correct, and read the third number hardest. Then route on provenance rather than on the result. structure_source is already in hand at extraction time and is a fact rather than a confidence; a document whose headings a human marked gets heading-aware splitting with a token ceiling, and everything else gets recursive splitting. Chunk the corpus two ways on purpose, keep a strategy field on every chunk, and report retrieval split by it — because the aggregate over a mixed-strategy corpus is the average of two different systems, and it will happily recommend the one that is losing.

The half this chapter routes around is still unsolved. On scanned documents, where the page-level rate says two thirds of pages carry at least one heading error, recursive splitting on paragraph then sentence is a fallback rather than an answer, because paragraph breaks in OCR output are themselves an artefact of line-spacing heuristics. When a document will not tell you where its sections are, the remaining option is to ask the text itself where its subject changes — which is the next chapter.