Recall with and without headers at one fixed k, or no number

A contextual header is a short string naming where a chunk came from — the document title and the chain of headings above it, or a generated sentence saying the same thing — prepended to the chunk text before that text is handed to an embedding model. The question this chapter answers is how to find out whether adding one helped, in a form somebody can act on. That turns out to be mostly a question about what is held still while the header is added, and only secondarily a question about which metric is computed.

The metric itself is recall@k: replay a fixed set of queries, ask the index for its best k chunks for each, and count what fraction of the queries got at least one chunk that a human had already marked as containing the answer. Those pre-marked chunks are the gold chunks for that query, and k is the number of results the search is allowed to return — the cut-off. Both of those are choices a person makes, and a recall figure carries no meaning at all until both are stated. A number quoted as "recall went from 0.62 to 0.79" has not said which queries, which labels, or which k, and each of the three can move the figure further than the treatment did.

Four numbers and the one everybody reads#

Here is what such a result usually looks like when it reaches a room. It is a real shape rather than a caricature: four measurements from a two-million-chunk index, taken before and after contextual headers were added.

Contextual headers — 2M chunk index
  recall@5      0.62 → 0.79
  index size    +38%
  ingestion     ~3x
  p50 latency   unchanged

Those four are recall at k=5, the growth in bytes on disk, the change in the one-off cost of rebuilding the whole index, and p50 query latency, p50 meaning the median: the time half of all queries come in under. Three of the four are costs and one is a benefit, and the benefit is the one that gets read aloud. The careful answer starts with what the 0.79 would have to be for it to mean what a room takes it to mean.

What recall@k counts#

The metric is small enough to write down completely, and writing it down is what makes the rest of the chapter's cautions obvious rather than pedantic.

def recall_at_k(index, queries, k: int, filter_fn) -> float:
    hits = 0
    for q in queries:
        got = {p.id for p in index.search(q.text, k=k, query_filter=filter_fn(q))}
        if got & q.gold_chunk_ids:
            hits += 1
    return hits / len(queries)


baseline = recall_at_k(idx_plain,  QUERIES, k=5, filter_fn=tenant_filter)
treated  = recall_at_k(idx_header, QUERIES, k=5, filter_fn=tenant_filter)

Take one query through it. q.text goes to index.search, which returns a ranked list and is cut off after k entries; got is the set of ids of those k chunks. q.gold_chunk_ids is the set of ids a labeller marked as answering that query. The & between them is set intersection — the ids present in both — and if got & q.gold_chunk_ids is true whenever that intersection is non-empty, which is to say whenever at least one gold chunk landed inside the cut-off. So a query contributes exactly one to hits or exactly zero, whatever else happened. hits / len(queries) is then the fraction of queries that got at least one piece of their evidence back.

The one property that governs everything below follows from the cut-off being a prefix of a ranking. The search returns results in rank order, so the top five are the first five entries of the same list whose first ten are the top ten. That makes the top-five set a subset of the top-ten set. If a gold chunk sits inside the top five, it is still inside the top ten, so every query counted as a hit at five is also a hit at ten. The hit count can therefore only stay the same or rise as k rises, and since the denominator len(queries) does not depend on k at all, the rate can only stay the same or rise with it.

Note which quantity moved and which did not, because the loose version of this gets stated as "raising k changes the denominator" and that is not what happens here. The denominator is the number of queries, and it is fixed by the query file. What k moves is the cut-off: how much the index is permitted to return before it is judged. Two arms measured at different k are not two rankings being compared, they are one ranking being given two different amounts of room.

The procedure that produces a number you can act on#

Build two indexes from the same chunk set. Not two ingestion runs — one chunking pass, producing one list of chunks with stable ids, embedded twice: once from chunk.text and once from header + chunk.text. Everything else held identical, including the embedding model and its version, the chunker configuration, a frozen corpus snapshot, and the filters applied at query time. Then replay one frozen query set against both at the same k and compare.

The reason for one chunking pass and two embeddings is the line got & q.gold_chunk_ids. Relevance labels are attached to chunk ids, and if the two indexes were produced by separate ingestion runs with separate ids, the labels are not portable and you end up re-labelling for the second index — which is the door through which the largest of the measurement errors walks in. Sharing the chunk set makes the ids identical by construction and the labels comparable by construction, and it also guarantees that no other variable moved, because there is only one splitter run to disagree about.

It is worth separating what that paragraph forces from what it merely recommends. A single chunking pass with stable ids is a requirement: the metric is defined by an intersection over ids, so ids that do not survive from one arm to the other make the intersection meaningless rather than merely noisy. Holding the embedding model version still is the same kind of requirement, since two model versions produce two different rankings for reasons that have nothing to do with headers. Choosing k=5 rather than k=10, and choosing two hundred queries rather than two thousand, are design decisions — defensible, arguable, and reversible, provided the same choice is made in both arms and written down before either is run.

Report the whole row rather than the headline. Recall alone answers "did it help" and cannot answer "was it worth it":

Read that table by its gaps rather than by its levels, and the two gaps of zero are the most informative entries in it. The chunk count is identical across the arms, which is the visible evidence that the single-chunking-pass requirement actually held — two ingestion runs would almost certainly have produced two different counts. The latency gap is also zero, which says the header changes what is stored and what is embedded and does not change what a query costs to answer; every price the headers charge is therefore paid at ingest or in the generator, and none of it at search time.

Two of the remaining gaps close against each other, and the closure is worth following because it converts a table row into a fact you can rely on elsewhere. The mean embed input rises from 812 tokens to 874, a gap of sixty-two tokens. The mean header on this corpus is separately reported as sixty-two tokens. Those two figures come from instruments that share no step — one is an average over the strings fed to the embedding model, the other an average over the header strings themselves — and they agree exactly. What that licenses is reading the entire +7.6% as header and nothing else: no re-tokenisation artefact, no truncation, no silently dropped text. The same sixty-two then multiplies by k to give the last row, since five retrieved chunks each carry their own header, and five times sixty-two is the 310 tokens printed there.

That last row is the one that was missing from the slide, and it is the cost that recurs on every query for the life of the system. At k=5 and a 62-token mean header, every answered question carries 310 tokens of header into the model's context — a per-query tax, paid whether or not the headers helped that particular query, and paid again on every turn of a multi-turn conversation that keeps the retrieved context. Chapter 8 does the arithmetic on what that comes to annually. Its presence here is to make the point that "38% more storage, and storage is cheap" is an answer to the wrong question, because storage is the one cost on that list that genuinely is cheap.

Three ways 0.79 gets inflated#

The measurement above is clean if and only if the query set and the labels were fixed before the treatment existed. Each of the following breaks that, and none of them announces itself.

Labels regenerated against the new chunk text. The most common way an evaluation set gets built is to run a model over candidate chunks and ask which of them answer the query, then have a human spot-check. If that generation is re-run after headers are added, the judge is reading Employee Handbook 2024 > Leave > Parental leave above the chunk and marking it relevant partly on the strength of the header — the same signal the treatment added. The labels then move in the same direction as the treatment, and the measured gain includes an unknown component of that circularity. The test is procedural rather than statistical: ask when the label file was last written, and if the answer is after the header work started, the number is not usable. Labels must be produced from the plain chunk text, once, and frozen.

A query set written by the people who designed the headers. Someone who has spent a week looking at heading paths writes queries in the vocabulary of heading paths. "What is the parental leave notice period" is a query whose lexical and semantic overlap with Leave > Parental leave is not accidental, and a set of two hundred such queries measures how well the headers match the header authors' own phrasing. Real queries look different: "how much warning do I need to give before mat leave", "can I take my leave in two chunks", "do i need 30 days notice". The fix is to sample from production query logs, stratify across sources and query types so the mix reflects traffic, and freeze the sample before anyone sees a header. Where there is no production traffic yet, have the queries written by people who have not seen the ingestion pipeline, from the user-facing documents rather than from the chunks.

Tuning and evaluating on the same documents. The header template, the informativeness predicate from chapter 3, and the stop-list of structural words all get tuned by looking at examples. If those examples came from documents that the evaluation queries also target, the tuning is fitted to the test. Hold out a slice of the corpus at the start — a fifth of documents, chosen by document id hash so the split is reproducible — and keep every tuning decision on the other four fifths.

There is a fourth, less discussed: the query set ages. A frozen set is what makes runs comparable, and six months of new documents and new user behaviour make it progressively less representative of live traffic. The reconciliation is to keep the frozen set for comparability and refresh it on a schedule that is deliberate and announced, reporting both the old and new set for one release across the changeover so the discontinuity is visible.

How many queries, and report the flips#

Two hundred queries is the size most evaluation sets end up at, because that is roughly what one person can label in a week. It is enough to detect a change of 0.17 and nowhere near enough to detect a change of 0.02, which matters because most subsequent tuning produces changes of 0.02. The rough shape: the standard error on a proportion near 0.7 with 200 samples is about 0.032, so a difference under about 0.06 between two independent measurements is inside the noise.

Where that 0.06 comes from is worth saying, because the number is a rule of thumb and behaves like one. A proportion measured on a sample has a standard error found by multiplying the proportion by one minus itself, dividing by the sample size, and taking the square root; near seven tenths on two hundred samples that arithmetic gives the 0.032 quoted above. The 0.06 is then roughly twice that figure, which is the usual informal band for "could plausibly be the same number measured twice". It is a band built from one arm's standard error rather than a formal test on two, so treat it as the threshold below which a difference is not worth arguing about, and not as a significance calculation.

The comparison here is not independent, though, and that helps. Both arms answer the same queries, so report the paired counts rather than only the two rates:

n = 200
  hit in both arms              118
  miss in both arms              28
  miss without -> hit with       40      <- what the headers won
  hit without  -> miss with       6      <- what they cost

Three of those four cells are pinned exactly by the two headline rates, and checking that is the fastest way to confirm the paired table and the summary table describe one measurement rather than two. A query that hit without headers either hit with them too or flipped to a miss, so the without-headers hit count is the both-hit cell plus the hit-to-miss cell — 118 and 6 — and that sum, taken over the 200 queries, is exactly the 0.62 in the summary table. A query that hit with headers either hit without them or was won by them, so the with-headers hit count is 118 and 40, and that sum over the same 200 queries is exactly 0.79. Neither rate needs rounding to land.

The fourth cell does not join that closure, and saying so is more useful than smoothing it over. The four printed cells add to eight fewer than n, so the miss-in-both entry is the only one the two recall rates do not independently pin, and it should be read as the remainder left after the other three rather than as a fourth measurement. Nothing else in the chapter depends on it: the downside bound, the flip counts and both recall rates are all built from the three cells that do reconcile.

Forty flips one way and six the other is a result that survives any reasonable test — McNemar's, the standard test for paired counts of this shape, which asks whether the two disagreement cells are too lopsided to be chance, is decisive on 40 versus 6 — and it is more informative than 0.62 → 0.79 in two ways. It bounds the downside explicitly, since a change that improves the aggregate while breaking a handful of previously-working queries is a change somebody should look at. And those six regressions are readable: pull them, and you usually find either a chunk whose header now dominates a short body, or a document whose heading path is misleading. Six queries is twenty minutes of reading and it is the highest-signal twenty minutes in the whole evaluation.

Why the top-k change cannot ride along#

Raising k from 5 to 10 in the same release destroys the ability to attribute either change, and it does so in a direction that guarantees a pleasing result. Recall@10 is greater than or equal to recall@5 on the same index by construction, because the top 5 are a subset of the top 10; a bundled release therefore cannot show a decrease, and a number that cannot go down is not a measurement.

This is the chapter's stumble, and it is tempting for a good reason rather than a careless one. Recall is the right family of metric here, the measurement was taken honestly, and the figure genuinely did rise — so "we measured recall and it went up" feels like a defence that has done its job. What that defence cannot do is discriminate. It returns the same verdict, it went up, when the headers carried the gain and when a wider cut-off carried it, because the property derived above means the wider cut-off alone cannot produce any other verdict. A check that answers identically across the cases it is supposed to tell apart is not a weak check; it is not a check.

Four cells are needed and the release proposes to measure one:

Filling in the top-right is a single evaluation run against an index that already exists and costs nothing but time, and it is the number that says how much of the 0.17 was available without paying for headers at all. If plain chunks at k=10 reach 0.76, the headers are buying 0.03 for a tripled ingestion bill and 38% more storage, and the honest recommendation reverses. That is not a hypothetical shape; wider k and better per-chunk context both address the same failure mode, which is the gold chunk being ranked below its competitors, so they partially substitute for each other.

Raising k also has costs of its own that the bundle would hide. Doubling k doubles the retrieved text in the generator's context, which doubles the header tax in the last row of that table from 310 tokens to 620, adds latency in the generation step, and lowers precision — more irrelevant chunks in context measurably increases the rate at which a model answers from the wrong one. If there is a reranker, it now scores twice as many candidates. None of those show up in recall@k at all, which is precisely why a release justified by recall@k should change one thing.

Answer accuracy, corpus mix and rank#

Recall@k measures whether the evidence reached the model's context, and stops there. Everything downstream — whether the model used the right chunk, whether it hedged, whether it hallucinated a number that was not in any chunk — is unmeasured by this procedure. A retrieval change that moves recall@5 by 0.17 and end-to-end answer accuracy by 0.02 is entirely possible and is worth knowing about before the roadmap gets rewritten around it. Run an end-to-end evaluation alongside, on the same query set, even if it is coarser and noisier.

It does not generalise across corpus mix. The 0.79 was measured on a particular blend of product docs, tickets and contracts; a corpus that shifts toward ticket-heavy content will behave like the population chapter 3 identified as template-resistant, and the headline number will drift without anyone changing anything. Report recall broken down by source type as well as in aggregate, because the aggregate hides exactly the variation that predicts how the number will move.

And a binary hit-or-miss recall is a coarse instrument. The definition above counts a query as a hit if any gold chunk appears anywhere in the top k, which treats rank 1 and rank 5 as identical and ignores the second gold chunk entirely. That is usually the right first metric because it is easy to reason about and hard to game; it is worth reporting a rank-sensitive metric beside it once the basic comparison is trustworthy — mean reciprocal rank (MRR), which scores each query as one divided by the position of its first gold chunk, or normalised discounted cumulative gain (nDCG), which credits every gold chunk found with a weight that decays down the list — since a change that moves gold chunks from rank 4 to rank 1 shows up as zero improvement under hit-rate recall and matters a great deal to a generator that weights early context more heavily.

Suppose all of that is done and the 0.79 survives. It says the headers work. It says nothing about whether to fund them, because the full bill has not been written down — the generation calls are the term everyone quotes and the smallest one in the accounting — and it says nothing about whether a recall problem is a header problem in the first place.

What makes the figure worth quoting at all is the discipline around it rather than the arithmetic inside it, and that discipline reduces to holding still everything except the one thing under test. One chunking pass so the labels stay attached to the same ids. One frozen query set and one frozen label file, both written before the treatment existed. One k, chosen in advance, because the cut-off is the one dial that can move the number without any part of the system having improved. Everything else on the slide — the disk, the rebuild, the latency, the per-query header tax — belongs beside the recall figure rather than underneath it, since three of those four are what the 0.17 costs.