Retrieval strategies for RAG

Read these first

• Chunking strategies
• Reranking

What you will know

• Tune top-k with a measured curve rather than a guess

  Larger k can improve recall while diluting the prompt, increasing latency, and surfacing contradictory passages; select the useful region from labeled queries rather than an arbitrary default.

• Implement parent-document or small-to-big retrieval

  The index can use precise child matches without forcing the generator to answer from fragments, but oversized parents can reintroduce irrelevant context and token cost.

• Measure what auto-merging buys over flat chunks

  Merging matched siblings can restore section-level coherence, while weak thresholds may over-expand unrelated content or fail to merge evidence split across boundaries.

• Choose a strategy for a specific corpus shape and defend it

  Long structured documents, repetitive short notes, and mixed-quality sources expose different boundary, duplication, and metadata failures; the choice should follow query-level evaluation results.

• Show diversity mattering on a query with redundant sources

  A relevance-only retriever can spend its context budget repeating one viewpoint, so compare answer coverage or evidence recall with and without a diversity constraint on a deliberately redundant set.

Chapters

1. 1

   Recall@k and answer accuracy are two different curves

   Raising k from 5 to 20 lifted recall from 0.72 to 0.89 and cost four points of answer accuracy, because extra passages both raise the odds the evidence is present and stuff the prompt with near-misses the generator has to arbitrate. Sweep both curves on labelled queries and plot them together; the k you want is where the second peaks. Run rerank depth as a second axis, and retire the number whenever the embedder, chunker or reranker changes.

   Read chapter
2. 2

   How much of a parent is relevant decides small-to-big

   Indexing 200-token children and expanding each hit to its 2,000-token parent improves an API reference corpus and degrades an internal wiki, and one property accounts for both outcomes: how much of that parent is relevant given that one child matched. A tightly scoped section restores exactly the context the fragment was missing. A sprawling one hands the generator nine unrelated subsections, plus their token bill, and asks it to locate the answer inside.

   Read chapter
3. 3

   A fixed neighbourhood expands predictably; a section does not

   Sentence-window and parent-document retrieval both match small and return big, and the resemblance ends there. A window returns a fixed span around the hit regardless of document structure, so expansion is bounded and identical everywhere; a parent returns a structural unit whose size and internal coherence vary enormously across a corpus. That contrast is the whole risk profile, and it is what decides which one a given body of documents should use.

   Read chapter
4. 4

   An unchanged average may mean the merge rule never fired

   Auto-merging swaps three or more matched siblings for their parent, and your eval comes back identical to flat chunks. Before drawing any conclusion, count how often the condition actually triggered: at 4% of queries an unchanged aggregate is arithmetic over mostly identical prompts, not evidence about merging. From there you tune the two threshold failures — over-expanding into unrelated material, and failing to fire on evidence split across a boundary.

   Read chapter
5. 5

   Five accounts of one cause spend the budget on one viewpoint

   Asked what causes checkout failures, a relevance-only top-5 returns five write-ups of the most-documented cause while the other two causes never surface. An MMR-style term reallocates part of the budget to lower-scoring but different passages, raising coverage and lowering average relevance — worth it on multi-cause questions, a distractor swap on single-answer ones. It is also the first reach for 40,000 repetitive 120-token tickets, where each unit is already whole and nothing is worth expanding.

   Read chapter
6. 6

   Regulations and 120-token tickets fail in opposite directions

   Eight thousand 60-page regulatory PDFs with strict numbered sections alongside 40,000 short repetitive tickets means one flat 512-token setting damages both: it severs the clause hierarchy that makes a regulation interpretable, and staples four unrelated tickets into one chunk so a match on one drags in three as noise. Sharing a single index and top-k compounds it. You assign a strategy per corpus shape and name the query-level measurement that confirms the split.

   Read chapter
7. 7

   A summary node answers what no single leaf chunk covers

   The last strategy on the menu builds levels: an LLM summary per document and per cluster, indexed as retrievable units of their own beside the leaf chunks, with queries either routed down the tree or searched across every level at once. It exists for a query shape the earlier strategies cannot serve — 'what changed in Q3' spans forty documents, and no leaf chunk states the answer while a cluster summary can. The price is visible before you measure anything: a summarisation pipeline to run and re-run, a traversal on every query, and summaries that quietly go stale the moment their leaves change.

   Read chapter
8. 8

   A rebuild that changed four things cannot credit the hierarchy

   A hierarchical summary index reports +6 nDCG@10 against production, and the prototype rebuilt the chunker, the embedder and the retrieval path alongside the tree, so the delta absorbs every incidental fix that shipped with it. The comparison that isolates the variable is flat chunks rebuilt with the same chunker, embedder, reranker and k. Should a smaller gain survive, what you are adopting is a permanent summarisation pipeline, per-query traversal, and a new staleness surface. The claim that summaries pay off on broad questions is also testable on its own: stratify the query set into aggregative and specific-fact questions and compare within each stratum, since a gain spread evenly, or one living in the specific-fact half, falsifies the story.

   Read chapter