RAG architecture

Read these first

• Reranking
• Anatomy of a prompt

What you will know

• Draw the full pipeline from ingest to answer without prompting

  Mark the data and identifier contracts between stages, including where versioning, metadata, and access controls can be lost.

• Build a naive end-to-end RAG path over a small representative corpus

  The implementation should produce traceable answers from a known document set before optimization hides whether the baseline is actually useful.

• Instrument each stage and measure its latency, errors, and output counts

  Stage-level spans, token counts, cache hits, and p50/p95 timings reveal whether slow answers come from parsing, search, prompt construction, or decoding.

• Reproduce failures by separating corpus gaps, retrieval misses, context problems, and generation errors

  A labeled test set should make it possible to tell an absent fact from a wrongly ranked chunk or a model that ignored correct evidence.

• Compare two RAG designs with a stated quality, latency, and cost method, then choose when not to use RAG

  A larger model or more retrieved context can improve recall while increasing distraction, token spend, and tail latency; exact lookup may be safer for a small stable corpus.

Chapters

1. 1

   The eight stages are contracts, and every one can drop a field

   Ingest, parse, chunk, embed, index, retrieve, augment, generate: each boundary hands the next stage an identifier and a payload, and each is a place where a document version, a metadata field, or an access label silently stops travelling. You will draw the chain from memory and mark, at every arrow, what has to survive it and what breaks downstream the first time it does not.

   Read chapter
2. 2

   An index only forgets what ingest tells it to forget

   Upserting chunks keyed by page_id:chunk_ix can add and overwrite, but it has no way to express a page that vanished at the source, so a deleted document keeps answering at rank 1 indefinitely. Reconciliation is the contract instead: each run publishes the full set of live identifiers, the job diffs that set against what the index holds, and deletes the difference. Re-chunked revisions leave the same orphans.

   Read chapter
3. 3

   Permission filters belong in the retrieval query, not after the answer

   A citation check that drops unauthorised sources can pass while restricted content still reaches the reader, because the model paraphrases evidence and the sentence carrying the leak cites nothing. Follow the mechanism: the chunk was in the prompt, at the provider, and in your prompt archive before any check ran. Here the decision moves to where it works, as access labels in the payload and a pre-filter on the search, and you price what that costs.

   Read chapter
4. 4

   A baseline you can trace beats four optimisations you cannot attribute

   The plain path is small, and this is what it consists of: embed the question with the same model that embedded the chunks, take top-k by cosine, template the returned chunks and the question into a single prompt, make one model call. Landing hybrid search, a cross-encoder, a query rewriter and a semantic chunker on top of that in one release leaves no number saying which earned its latency, or whether the rewriter quietly deleted a constraint from the question. Stand the plain path up over a small representative corpus first, keep it as the control, and change one thing at a time against it.

   Read chapter
5. 5

   Per-stage spans turn a slow answer into a named stage

   Total request duration can tell you p95 moved from 3.1s to 7.4s while p50 barely shifted, and nothing beyond that. Emit one span per stage on a shared trace id, each carrying its own output count, token count and cache-hit flag, so opening a single slow trace shows which stage grew and whether it grew in time or in size. A tail-only regression usually means an input got bigger, not that the code got slower.

   Read chapter
6. 6

   A labelled set is an instrument you build before you trust its numbers

   Two to three hundred questions sampled from traffic you actually get, each tagged with the paragraph that answers it, plus a stated rule that turns a free-text answer into right or wrong: span match, a written rubric, or a judge model whose agreement with a human you have measured rather than assumed. Then the comparison itself is paired — the same questions through both systems, with the interval computed on the per-question difference rather than on each system's score separately, because two binomial intervals are the wrong instrument and are far wider than the comparison supports. Every pass rate, every eight-point gap and every do-not-use-RAG call in the rest of this topic is read off this one artefact.

   Read chapter
7. 7

   A corpus gap is not a retrieval miss, and no reranker closes it

   Four places a failure can live, and one field per stage that separates it from its neighbour: was the gold paragraph indexed at all, what rank did it return at, does the archived prompt contain it, and did the answer use it. A paragraph that was never indexed is unreachable at every retriever setting, so filing it as a ranking problem sends the team to tune search against a question that cannot pass at any value.

   Read chapter
8. 8

   Higher recall of the gold passage can lower answer accuracy

   Top-25 into a 40k prompt puts the right passage in the window more often than top-5, and seats twenty plausible neighbours beside it, so answers that were correct now ground themselves in something adjacent and wrong. Keep recall and accuracy apart, then build the comparison: quality with a paired interval, latency as a share of the user's session rather than a flat tax, and cost per query. When the ask is the expensive design's quality at the cheap one's latency, the levers are concrete and each is priced — cache the rewrite so repeat phrasings skip it, run it concurrently with first-stage retrieval instead of ahead of it, rerank 20 candidates rather than 50, swap in a smaller cross-encoder, skip the rewrite below a query-complexity threshold — and each hands back a slice of the eight points.

   Read chapter
9. 9

   Ninety structured records are a lookup, not a search problem

   Where a key already determines the answer, vector search inserts an approximate step whose signature failure is returning the neighbouring policy with total confidence. The same decision arrives from the other side with an 800-page manual that now fits in one window: long context is genuinely available, but priced per query and dependent on cache hit rates and traffic shape nobody has measured yet. Both calls get settled on the labelled set, not on sunk weeks.

   Read chapter