The stages of a RAG pipeline

Retrieval-augmented generation names an arrangement rather than a fixed architecture, and the arrangement most production systems settle on has four stages. A corpus of documents is cut into retrievable units offline and indexed. A question is embedded and searched against that index, producing a ranked list of candidates. A reranker may reorder the shortlist. The top few land in a language model's prompt, and the model writes an answer. Every one of those steps emits something, and the things they emit are the only places a measurement can be taken.

Scoring stages separately is impossible while the stages are a shape on a whiteboard. Two configuration choices account for most of the variance in the retrieval numbers you will be reading for the rest of the topic, and one of them silently destroys a result. A RAG evaluation without labelled relevant passages is only end-to-end evaluation with extra steps.

Four stages and the artifacts between them#

The chunker runs offline, on ingest, and it is the only stage whose output is shared by every question. It takes documents and cuts them into retrievable units, then hands each unit to an embedding model — which turns a passage into a fixed-length list of a few hundred numbers, positioned so that texts about similar things land near each other — and writes that vector plus the text into an index. Its output artifact is the index, and the unit of everything downstream — the thing retrieval returns, the thing labels are attached to, the thing that lands in the context window — is a chunk that this stage defined. A chunk that splits a table from its header, or that ends halfway through the sentence stating a retention period, is a defect created here and observed three stages later.

The retriever takes a question, embeds it with the same model used at ingest, and runs an approximate nearest-neighbour search against the index, meaning it finds the stored vectors closest to the question's vector without comparing against all 27,900 of them, usually returning far more candidates than will ever reach the model — 50 is typical. Its output artifact is a ranked list of chunk IDs with scores. Hybrid setups run a lexical search such as BM25 — which ranks by exact term overlap rather than by meaning — in parallel and fuse the two rankings, which matters for questions carrying exact identifiers like /v1/exports that a dense embedding smooths over.

The reranker takes the question and each candidate together and scores the pair with a cross-encoder — a model that reads both texts jointly rather than comparing two independently computed vectors. It is slower per item by orders of magnitude, which is why it never sees the whole corpus and only ever reorders a shortlist. Its output artifact is the same kind of list, shorter and in a different order. Some pipelines have no reranker, in which case the retriever's top k goes straight through.

The generator takes the top k chunks, assembles them into a prompt with a system instruction and the question, calls the model, and returns an answer. Its input artifact — the assembled context — is the single most important object in this topic, because it is what makes "the model ignored the evidence" and "the model never had the evidence" into distinguishable claims.

def answer(question: str, k: int = 5) -> Answer:
    qvec       = embed(question)
    candidates = index.search(qvec, k=50)          # [(chunk_id, score), ...]
    ranked     = reranker.rank(question, candidates)[:k]
    context    = [store.text(cid) for cid, _ in ranked]
    completion = llm.complete(PROMPT.format(context=context, question=question))
    return Answer(
        text=completion.text,
        retrieved_ids=[cid for cid, _ in candidates],   # all 50
        context_ids=[cid for cid, _ in ranked],         # the 5 that reached the model
        context_text=context,
    )

The three fields at the bottom are the difference between a pipeline you can evaluate per stage and one you cannot. retrieved_ids lets you ask whether the right chunk was ever found; context_ids lets you ask whether it survived reranking; context_text lets you ask whether the answer is consistent with what the model actually read. None of them cost anything to record and all of them are irrecoverable after the fact.

The two knobs that move retrieval most#

Chunking and reranking dominate because they act on the whole candidate set rather than on the margins, and each of the changes below has a known direction of effect.

The reranker row has a property worth stating plainly, because it constrains what you can conclude from a single number. A reranker cannot introduce a chunk that the retriever did not return, so recall@50 — measured on the candidate list before reranking — is a ceiling on recall@5 after it.

That ceiling follows from what reranking is, in three steps. The reranker receives a set of fifty candidates and returns the same fifty in a different order; it has no access to the index and no way to fetch a fifty-first. Recall@5 after reranking asks whether a gold chunk appears among the first five of that reordering. A chunk absent from the fifty is absent from every reordering of the fifty, and therefore absent from every opening stretch of one. So the gold chunks available to recall@5 are a subset of those available to recall@50, and a subset is never larger than the set containing it. Nothing about the reranker's quality enters that argument anywhere, which is why the bound holds for a good reranker and a broken one alike.

When recall@5 drops, checking recall@50 immediately separates "the right chunk was never found" from "the right chunk was found and then ranked out". On the pipeline above, a cross-encoder over 50 candidates costs roughly 60–120 ms and around $0.0004 per question at current small-model rates, which is why the shortlist is 50 and not 500.

Read against that table, "recall@5 fell from 0.81 to 0.66" stops being a number that dropped and becomes a question with four or five named suspects, each of which is a line in a config file with a timestamp and an author. That is what naming the stages buys.

What a boundary does to a single fact#

The overlap_tokens: 0 row is worth making concrete, because the mechanism by which a chunking parameter destroys a retrieval result is not obvious from the parameter's name. Here is the relevant paragraph of docs/security/audit-logs.md as it sits in the source document, and then as it survives a split at 1,500 tokens with no overlap:

source:
  "...Audit log retention varies by plan. Enterprise customers retain
   90 days of audit history. Team and Starter plans retain 30 days.
   Logs older than the retention window are deleted nightly..."

chunk 41 ends:   "...Audit log retention varies by plan. Enterprise
                  customers retain 90 days of audit history."
chunk 42 begins: "Team and Starter plans retain 30 days. Logs older
                  than the retention window are deleted nightly..."

Chunk 42 contains the fact the question needs and does not contain the words "audit log retention", so its embedding sits in a region of the space nearer to generic plan-comparison text than to a question about audit logs. Chunk 41 contains the topic vocabulary and the wrong number. A dense retriever asked "how long are audit logs kept on the Team plan" scores chunk 41 highly and chunk 42 poorly, which produces exactly the escalation from chapter 1, on end-to-end and component metrics — a confident answer of 90 days, faithfully drawn from a passage the retriever was right to consider topical.

The links in that argument are each worth checking on their own, because the step people skip is the first one. An embedding is computed from the text of the chunk and from nothing else — not from the document it came from, not from the heading above it, not from the chunk before it. So the only evidence the embedding model has about what chunk 42 is about is the words inside chunk 42. The words that name the topic, "audit log retention", are in chunk 41. What remains in chunk 42 is a sentence about plan tiers and a sentence about nightly deletion, and a vector computed from those words lands where text about plan tiers and nightly deletion lands. The retriever then ranks by closeness between the question's vector and each stored vector, and the question's vector is dominated by the words "audit logs". Chunk 42 is therefore ranked as though it were about something else, which in the only terms the retriever has access to, it is.

Overlap repairs this by making the boundary stop being a boundary for any short fact. With 128 tokens of overlap, the first tokens of chunk 42 are also written into the tail of chunk 41, so chunk 41 would have carried "Team and Starter plans retain 30 days" alongside the phrase "audit log retention" and the question would have been answerable from a single unit. The general form of the repair is that overlap buys a guarantee about span: a fact stated inside a window shorter than the overlap is guaranteed to sit whole inside at least one chunk, whatever the split points happen to be. A fact whose statement is longer than the overlap gets no such guarantee, which is why overlap is a mitigation and not a fix.

That failure is invisible at every stage except the one where it is measured. The chunker did what it was told, the retriever ranked correctly given the embeddings it had, and the generator was faithful to its context. The defect belongs to a stage that produced no error, and it is only nameable because the artifacts between stages were written down separately.

The table's first row says a larger chunk "matches any single question less sharply", and that phrase hides the mechanism that matters most in this chapter. An embedding model returns one vector per chunk, of a fixed width, regardless of whether the chunk is a sentence or three pages. A chunk covering one topic gets a vector that can sit squarely in that topic's region of the space. A chunk covering four topics gets one vector that has to stand for all four, so it settles somewhere between them and is not particularly close to a question about any one of them. Doubling max_tokens roughly doubles the number of topics an average chunk covers, which is the same thing as saying it moves every chunk's vector further from every specific question. That is why the direction column reads "recall down" without qualification: the effect does not depend on the corpus or the model, only on there being one vector per unit however large the unit is.

Two further consequences of chunk size are easy to conflate. Larger chunks reduce the number of retrieval slots the answer has to compete for, so a top-5 at 1,500 tokens carries 7,500 tokens of context against 4,000 at 800 tokens — more raw evidence per question, more input cost per call, and more opportunity for the distraction failures that appear later in this topic. Smaller chunks raise the chance that any single chunk is precisely on topic and the chance that a multi-part answer needs two or three at once. There is no setting right in general, which is why the parameter belongs in an eval loop rather than in a discussion.

Where the labels for RAG evaluation come from#

Every retrieval number in this topic is scored against a set of labelled relevant passages: for each eval question, the chunk IDs that actually contain the answer. It is worth being concrete about how the set gets built and what each source distorts.

Most working sets are a blend: mined tickets for realism, generated questions for coverage of documentation nobody has asked about yet, and a human pass over both to remove items where the "gold" chunk does not really contain the answer. Two rules keep the labels usable. Labels attach to chunk IDs, so re-chunking invalidates them. Storing labels as document-plus-character-offset spans and resolving them to chunk IDs at eval time survives re-chunking; storing raw chunk IDs does not. And a question can have more than one gold chunk, so recall has to be defined over a set rather than a single ID.

{
  "qid": "q-0143",
  "text": "How long are audit logs kept on the Team plan?",
  "gold": [
    {"doc": "docs/security/audit-logs.md", "start": 4120, "end": 4610},
    {"doc": "docs/plans/team.md",          "start": 880,  "end": 1090}
  ],
  "source": "mined-ticket-88214"
}

That record is the offset form, and the rule it embodies — spans rather than chunk IDs — is the one that costs money when it is skipped. The reason is a chain worth following once. A chunk ID is minted by the chunker at ingest time; it names one of the units that particular run of the chunker produced. Change max_tokens and the chunker produces a different set of units, so the identifiers it mints name different text, or name nothing at all. A gold label recorded as a chunk ID is then pointing at something that no longer exists. The recall function looks for that ID in the retrieved list, does not find it, and scores the question as a miss — which is arithmetically correct and says nothing whatever about the retriever, because the retriever was never given a chance to return an identifier the index no longer contains. Recall falls, and the fall is bookkeeping.

A document offset survives the same change because it names a position in the source document rather than a position in the index. docs/security/audit-logs.md characters 4120 to 4610 refer to the same sentences before and after a re-chunk, since the chunker rewrites the units and not the documents. Resolving that span to "whichever chunks overlap it" is done at eval time against the index that is actually being scored, so the label and the index are always contemporaries. The cost is one lookup per gold span per run, and what it buys is that a recall movement can only ever be about retrieval.

Where this decomposition stops being the right one#

Four stages is a description of one common shape, not a law. Plenty of production systems have no reranker, index whole documents rather than chunks, or run several retrieval passes with a query rewritten between them. Agentic systems, where the model decides when to search and with what, do not have a single retrieval boundary at all — they have one per tool call, and the interesting metric becomes whether the model chose to search, which the chunker-retriever-generator vocabulary cannot express.

The decomposition also flatters itself in a specific way: every box in that diagram looks equally measurable, and they are not. A chunker has an output but no defensible notion of a correct one, since "the right way to cut this page" is not a thing anyone can label. Chapter 4, on which stage boundaries can carry a probe, turns that observation into a test you can apply to any stage before you spend a sprint instrumenting it.

What the naming has produced is a set of places a measurement can go and a vocabulary for the causes of a movement. A chunker emits an index, a retriever a candidate list, a reranker a shorter list, a generator an answer with citations; record those four artifacts per request and the difference between a passage that was never found and a passage that was found and ignored becomes a lookup rather than a debate. Skip them and every retrieval number you compute later is uncomputable after the fact, which is the position the eleven-line diff left its team in.

The pair of metrics that make those two failure modes tell themselves apart is next. Both stages are now named, both artifacts are now recorded, and the labels exist to score against.