A summary node answers what no single leaf chunk covers

A retrieval system answers a question by selecting passages from documents that somebody wrote. That is the whole of what every strategy in the preceding chapters does — expanding a matched chunk into its parent, merging adjacent siblings, diversifying a result list — and it is why one shape of question defeats all of them at once. Ask what changed in Q3 over a corpus of release notes, amended policies and pricing changelogs, and the material is present while the answer is not. Nobody wrote a paragraph called "what changed in Q3". The answer exists only as a property of that corpus, and selection cannot manufacture a sentence that was never written down.

A hierarchical summary index is the response to that gap. It manufactures the missing units by having a model write them: a summary per document, a summary for each cluster of related documents, sometimes a summary per cluster of clusters, all indexed alongside the leaf chunks as retrievable objects in their own right. A leaf chunk here is one of the slices of source text an ordinary index already holds; a summary node is a piece of text no author produced, generated from the nodes beneath it and stored with its own embedding so that a query can match it directly. When an aggregative query arrives — one whose answer requires synthesising across documents rather than locating a passage — the thing it matches is a summary node whose text actually says "during Q3, retry behaviour, notification thresholds and Pro-tier pricing were amended", and that node's children give the generator the specifics.

Why no leaf chunk carries the answer#

The reason is worth taking one step at a time, because it decides which repairs can possibly work. A retriever stores units of text and returns the units whose embeddings — the lists of numbers standing in for their meaning — lie closest to the query's. A unit therefore carries exactly the propositions somebody wrote into it, and no others. An aggregative question asks for a proposition quantified over a set: these were the changes, across all of these documents, in that quarter. If no author wrote that quantified sentence, no unit contains it. A top-k list is a selection of units, so for any k at all the list contains no unit that states the answer. Raising k relocates the failure without touching it. More evidence arrives, and the sentence the generator would need to quote still does not.

This is why the failure reads as a good result on a retrieval metric and a bad one to the person who asked. Each individual change was written down, in the document that made it, so the judged-relevant passages are found and recall@k — the share of judged-relevant passages appearing in the top k, from chapter 1 — is high. What is missing is not evidence but the conjunction of it. No metric that scores passages one at a time can see a conjunction going unserved, because every passage it scores is individually fine.

A three-unit case makes the geometry visible. Suppose three leaves: one saying webhook retry behaviour changed, one saying a notification threshold was amended, one saying Pro-tier pricing moved. A query naming all three subjects at once has to be scored against each leaf separately, since scoring one unit at a time is the only comparison an index performs. Each leaf matches on a third of what was asked and misses the rest, so all three score moderately, and whichever subject the query happens to word most heavily wins the ranking. The budget then fills with more passages about that one subject. The passages most similar to a passage about retries are other passages about retries.

A summary node breaks that pattern because it is a single unit whose text mentions all three subjects. One embedding now sits near a query about the conjunction, where previously the nearest thing was a unit about one conjunct. The union of the three leaves contains every fact the summary contains and cannot be retrieved as a union, since retrieval scores units and never scores sets. That distinction is the entire justification for the strategy. The summary adds no information; it adds an addressable unit.

The tempting response, and the one usually tried first, is a better embedding model. It is tempting for a sound reason: every earlier ranking problem in this topic did yield to better ranking, and a model that understood the question properly would surely put the right thing first. It does not work here, and the reason is structural. Ranking quality is a property of the ordering over a set of units. The defect is in the set. A perfect ranker over units none of which states the answer still returns units none of which states the answer, and its score is computed identically whether or not the answer exists anywhere in the index. A defence that cannot vary with the thing it is meant to detect is not a defence.

Levels, and two ways to search them#

The three node counts in that diagram are not independent, and reading them as ratios rather than as levels is what makes the shape of the strategy clear. Level 1 holds one node per document, so its size is the document count. Level 2 holds one node per group, and the size of a group is a parameter — the fan-out, meaning how many children a node has. Level 2 is therefore the document count divided by the fan-out, a level above it would be that divided by the fan-out again, and so on. Adding levels adds a shrinking geometric series on top of the document count, and such a series sums to a small multiple of it however deep the tree goes. The index barely notices: 8,041 document summaries plus 412 cluster summaries is 8,453 extra vectors, and that extra-vector figure is those two rows added together and nothing else.

The token bill does not shrink the same way, and the asymmetry is the part to carry. Node count falls by a factor of the fan-out at every level. The cost of building a level is the cost of reading what sits beneath it, which does not fall by the same factor: level 1 reads every document in full, while level 2 reads only the level-1 summaries. That is why the two levels differ by roughly two orders of magnitude in input tokens while differing by a factor of about twenty in node count. Depth is nearly free. The first level above the leaves is where the money is.

The traversal mode matches at the top level, then searches only within the matched node's subtree, which keeps each search small and makes the cost of a query roughly logarithmic in corpus size. It also means a routing mistake at level 2 is unrecoverable: if the query's evidence lives under a cluster the first hop did not select, no amount of good matching below can find it. Measured on an internal set, top-1 cluster routing put the gold evidence outside the selected subtree on 14% of queries, and widening the first hop to the top-3 clusters brought that to 4% at roughly three times the level-1 search cost.

Those two routing figures are worth reading as a gap rather than as levels. One query in seven lost outright at the first hop is a failure nothing downstream can repair, and widening the hop to three clusters leaves under a third of that loss standing. The price of the widening is exactly three subtree searches instead of one, which is where "roughly three times" comes from — it is a count of searches rather than a measurement. What the trade buys is a bound on the unrecoverable case. What it does not buy is any improvement in matching, since the same ranker is doing the same job inside a wider net.

Collapsed search puts every node from every level into one index and runs a single query against all of them, letting the ranking decide whether a leaf, a document summary or a cluster summary is the best match. RAPTOR, the 2024 paper that popularised recursive clustering-and-summarising, reported the collapsed variant matching or beating traversal on most of its benchmarks, and it is the mode to start with because it has no routing decision to get wrong. What it has instead is a competition between levels — summaries and leaves ranked against each other by cosine similarity, on a scale that has no idea one of them is a lossy paraphrase of the other.

def build_hierarchy(docs, chunk, summarise, embed, cluster, index):
    leaves = []
    for doc in docs:
        doc_leaves = [Node(text=c, level=0, doc_id=doc.id) for c in chunk(doc.text)]
        leaves.extend(doc_leaves)
        index.add_all(doc_leaves, level=0)

    doc_nodes = []
    for doc in docs:
        text = summarise(doc.text, target_tokens=220)        # 1 LLM call per doc
        n = Node(text=text, level=1, doc_id=doc.id,
                 child_hash=hash_of([l.text for l in leaves if l.doc_id == doc.id]))
        doc_nodes.append(n)
    index.add_all(doc_nodes, level=1)

    vecs = embed([n.text for n in doc_nodes])
    for members in cluster(vecs, target_size=20):            # ~400 clusters
        joined = "\n\n".join(doc_nodes[i].text for i in members)
        s = Node(text=summarise(joined, target_tokens=380), level=2,
                 children=[doc_nodes[i].id for i in members],
                 child_hash=hash_of([doc_nodes[i].text for i in members]))
        index.add(s, level=2)

The child_hash field is not decoration; it is the whole staleness mechanism, and the rest of this chapter turns on it.

Two choices decide whether the summaries are worth retrieving#

Two of the things that function fixes are requirements and two are design decisions, and confusing the categories is how implementations go wrong. That every document gets a summary is a requirement: without it the level does not cover the corpus, and a query can miss a whole document's material at the first hop. That the summaries are written by a language model is a requirement too, since the missing sentence has to be produced by something. The target cluster size and the summarisation prompt are decisions. Nothing forces a value, both admit a wide range, and each has a distinct failure mode at either end.

The clustering is the first, and its target size controls what a level-2 node can possibly say. At twenty members a cluster summary can describe a coherent theme in 380 tokens with room for specifics. At sixty members the summary degenerates into a topic list — "this collection covers pricing changes, security reviews, onboarding updates and eleven other subjects" — which is topically similar to every query about any of those subjects and informative about none of them. Measured on the corpus above, cluster summaries built at target size 60 mentioned an average of 11 distinct topics in 380 tokens, about 34 tokens each, and their retrieval precision was worse than the document summaries below them.

That figure of 34 is the arithmetic of a constraint rather than an observation. The summary length is fixed by the prompt, so the tokens available per topic are the length divided by the number of topics mentioned. A specific fact — a price, a date, a section number, with enough context to be usable — does not fit in 34 tokens, so a summary that mentions eleven topics cannot state one. That is precisely what makes it match every query about any of them and satisfy none. At the other extreme, target size 3 produces 2,700 clusters whose summaries are barely distinguishable from their members, so the level adds cost and no new capability. Those 2,700 are the document count divided by three, which is the same fan-out arithmetic read from the other end.

Soft clustering is worth the extra machinery here. RAPTOR uses a Gaussian mixture — a clustering method that gives each document a membership weight in every cluster instead of assigning it to exactly one — over dimensionality-reduced embeddings, specifically so a document can belong to more than one cluster, which matters because a Q3 pricing amendment genuinely belongs to both the pricing theme and the Q3 theme, and hard assignment forces a choice that will be wrong for half the queries. Allowing membership in two or three clusters roughly doubles level-2 node count and costs almost nothing relative to the level-1 build.

The second choice is the summarisation prompt, and it is where most implementations lose the gain. A summary written as a literary abstract — fluent, general, stripped of specifics — is close to unretrievable, because the handles a query matches on are exactly the proper nouns, dates, product names and numbers that abstracting removes. Summaries for retrieval want the opposite instruction.

SUMMARY_PROMPT = """Summarise the material below for a search index.

Rules:
- Preserve every product name, entity, date, quarter label and numeric
  threshold that appears. Do not generalise "$49/seat" to "a price change".
- Use the vocabulary of the source, not synonyms.
- State what the material is ABOUT and what CHANGED or was DECIDED.
- No preamble, no "this document discusses". Target {target_tokens} tokens.

MATERIAL:
{text}
"""

Each rule in that prompt follows from what a query is actually scored against. A query matches a unit through its embedding, and the parts of a summary that pull an embedding toward a specific question are the proper nouns, dates, quarter labels and numbers, since those are the tokens the query itself is likely to contain. Generalising $49/seat to "a price change" deletes exactly the handle. Instructing the model to use the vocabulary of the source rather than synonyms is the same argument one step further along: a synonym moves the summary's embedding away from the query's wording and buys nothing in return.

Building level 2 from the level-1 summaries rather than from raw leaves is a cost decision with a fidelity consequence. Summarising summaries costs 1.8 million input tokens instead of 201 million, which is the difference between six minutes and several hours, and it compounds the losses — a number dropped at level 1 cannot reappear at level 2. The prompt above mitigates that by making level-1 summaries specific enough to survive a second pass, and the check is mechanical: sample thirty documents, list the dates and numeric values in each, and measure what fraction survives into the document summary. Below about 70% survival, the tree will fail on precisely the aggregative queries it was built for, since "what changed in Q3" depends on the quarter labels making it through.

Both figures in that comparison are arithmetic on the corpus rather than measurements. Summarising every document from its raw text means reading 8,041 documents of about 25,000 tokens each, and that product is the 201 million. Summarising the level-1 summaries instead means reading roughly four hundred clusters of twenty summaries at 220 tokens apiece, and that product is the 1.8 million. The ratio between the two is the compression the first level already performed, which is the whole reason the second level costs minutes.

The bill, which is knowable before any evaluation#

Every number below comes from arithmetic on the corpus, not from a benchmark, which is what makes this strategy unusual to evaluate — you can price it before you know whether it works.

Those figures assume 8,041 documents averaging 25,000 tokens, a small summarisation model at $0.25 per million input and $1.25 per million output tokens, and 64 concurrent requests. A full rebuild is therefore about $94 and most of a working day, which is affordable annually and annoying weekly.

The build-cost column closes exactly, and that is worth stating because it is the property that makes the estimate usable. $38 at level 0, $54 at level 1 and $2 at level 2 add to the $94 quoted for a full rebuild, with nothing left over. No fourth line item sits outside that total, so quoting $94 as the complete build bill is legitimate under the stated assumptions, and a real build coming in materially above it means one of those assumptions is wrong rather than that something was forgotten. The identity does not license treating the individual cells as derivable. At $0.25 per million input tokens the 201 million input tokens account for most of level 1's $54, the 220-token outputs at five times the input rate account for part of the remainder, and what is left is prompt overhead the table does not itemise.

The incremental path is what you actually operate: when a document's leaves change, its child_hash no longer matches, so its level-1 summary is regenerated — about 25,000 input tokens, $0.008, two seconds — and that in turn dirties its cluster's summary. A corpus with 300 document edits a day costs roughly $2.50 and forty minutes of background work daily, which is a real pipeline with real on-call surface rather than an index build. That daily figure is 300 regenerations at $0.008 each, which comes to a little under two and a half dollars, plus the cluster summaries those edits dirty, and the two together are the $2.50.

Query-time cost is smaller and less avoidable. Collapsed search adds one index of 8,453 extra vectors, which is noise next to 1.9 million leaves. Traversal adds two sequential index calls at 15 to 30 ms each, so 30 to 60 ms of added p50 latency before the generator starts, on top of whatever reranking already costs.

Staleness is the price that does not appear on either line. A summary is a cached derived value with no invalidation protocol of its own, so between a leaf edit and the regeneration of its ancestors, the index contains a confident sentence that is wrong — a cluster summary asserting that the notification window is 30 days, three hours after an amendment made it 72. Nightly regeneration means a worst case of 24 hours of that. The child_hash comparison is what makes the window bounded and auditable.

Calling a summary a cached derived value is an analogy, and it is worth being exact about where it stops. A materialised view in a database is a cache with the same guarantee as the query it caches: recompute it and you get the value it was supposed to hold, exactly. A summary carries the invalidation problem without the guarantee. Regenerating it produces a different summary from the one it replaces, since a language model is not a deterministic function of its input in the way a sum over rows is, so "stale" and "fresh" are not two states of one value. The machinery transfers — hash the children, compare, regenerate on mismatch. The equality does not.

The belief to name here is that a nightly regeneration turns staleness into a solved problem, and it is tempting because the window is bounded and the job is monitored. What the job cannot tell you is whether anybody was served a wrong answer out of it. A stale summary does not fail. It returns a fluent, confident sentence with the superseded number in it, and the retrieval logs, the latency graph and the error rate are identical in the world where every summary is current and the world where a third of them are a day behind. child_hash is what converts that silence into something checkable: the fraction of nodes whose stored hash disagrees with their children's is a number you can put on a dashboard, and a system without it has summaries whose accuracy nobody can state.

Where a paraphrase outranks the primary text#

A summary is lossy by construction, so the specific facts it dropped are unrecoverable from it. A document summary reading "several pricing changes took effect in July" cannot answer "what is the new price of the Pro plan", and the danger is not that it fails to answer but that it ranks highly for the query and displaces the leaf chunk containing Pro: $49/seat/month, effective 2025-07-01. In collapsed search this happens routinely, because summary nodes are topically dense — they mention every subject in their subtree — which makes them broadly similar to almost any query about that subtree.

A better embedding model does not fix this either, and the reason has the same shape as before, one layer down. Cosine similarity between a query and a unit is computed from the two vectors and nothing else. Nothing in that arithmetic knows whether the unit is primary text or a paraphrase of primary text, since being derived leaves no trace in the embedding — a summary node's vector comes out of the same encoder, from the same kind of input, as a leaf's. The score is constant in exactly the property it would have to discriminate. That is why the mitigation is a quota rather than a threshold or a better model: a rule that reserves slots by level uses information the ranker does not have.

The mitigation is a level quota rather than a better embedding: reserve slots in the final context for level-0 nodes, so a prompt is never composed entirely of summaries. Something like six leaves plus two summary nodes, with the summaries permitted to bring their highest-scoring children along, keeps the aggregative capability without letting paraphrase crowd out primary text. Measuring the composition of served prompts by level is the diagnostic that tells you whether this is happening at all; if 70% of your top-8 is level 1 and 2, specific-fact queries are being answered from paraphrase. Those two numbers pair: six plus two is the top-8 the quota governs, and a top-8 that is 70% summaries has fewer than three slots left carrying primary text.

The questions a summary tree is the wrong instrument for#

The strategy does not create information that the corpus lacks. A cluster summary can state what its members have in common because a model read them; it cannot state a fact that appears in none of them, and asking "how many customers were affected in total" of a summary index produces a fluent number that no document supports. Aggregation over structured facts is a database's job, and a summary tree is a poor substitute for one — the correct architecture for counting is to extract fields into a table and query it.

It is also a strategy with a specific query shape as its justification, which means its value depends on how much of your traffic has that shape. If aggregative questions are 4% of queries, a $94 build, a daily regeneration pipeline and a staleness surface are being adopted for one query in twenty-five, and the per-corpus fan-out from chapter 6, where each corpus shape gets its own retriever and its own fixed k, is likely a better use of the same engineering time. If they are 30%, as they are for a corpus of meeting notes or incident reports where users ask about patterns more than facts, the calculus reverses.

The direction this is heading is worth knowing even if it is out of scope for most systems. Graph-based variants replace embedding-proximity clusters with communities in an entity graph: extract entities and relations from every leaf, detect communities over the resulting graph, and summarise each community. That yields clusters organised by what things are connected to rather than by what text they resemble, which handles a query like "everything touching the notification obligation" better than cosine clustering can. It also moves the build cost up by an order of magnitude, since extraction runs an LLM over every leaf rather than every document — 1.9 million leaves at roughly 250 input tokens each is 475 million tokens per full build against this chapter's 201 million, and the extraction output has to be parsed, deduplicated and stored as a graph.

That order-of-magnitude claim is about cost rather than about the two token counts quoted beside it, and the gap between them is worth spelling out, because at first reading the figures look as though they contradict the claim. 475 million input tokens against 201 million is a factor of about two and a half, which is not an order of magnitude. The order of magnitude lives in the call count and on the output side. Extraction runs once per leaf instead of once per document, which is more than two hundred times as many model calls, each with its own request overhead and its own retry surface. Every one of those calls emits entities and relations, so the output tokens — priced here at five times the input rate — scale with the leaf count as well. The token ratio is the cheapest part of the difference, which is why quoting it on its own understates the bill. The evaluation discipline is identical whichever variant you pick, and it is the part that decides the outcome.

A hierarchical summary index is the one strategy in this topic that adds units rather than rearranging them, and that is simultaneously its justification and its entire cost profile. It exists for questions whose answers are properties of a collection, which retrieval cannot select because selection cannot write a sentence. What it charges is a summarisation pipeline with its own schedule, alerting and backfill, a staleness window that becomes a property of the product, and a ranking in which paraphrase competes against primary text on a scale that cannot tell the two apart. Every one of those prices is knowable from the document count and the published token rates before a single query is run, and the share of your traffic that asks aggregative questions is what decides whether they are worth paying.

None of which is settled by the number a prototype reports. The proposal that arrives with this design usually arrives with a benchmark attached — a hierarchical index showing several points of nDCG@10, the ranking metric that rewards putting judged-relevant passages near the top of the first ten results and discounts them the further down they sit, over the current production pipeline — and that comparison is where the interesting problem is, because building the hierarchy almost always meant rebuilding the chunker, the embedder and the retrieval path in the same commit.