Retrieval-augmented generation

Every chapter after this one manipulates the same object, so the object is worth building explicitly rather than invoking as a slogan. A corpus here means the collection of documents a system is allowed to answer from — policies, help-centre articles, internal FAQs. A chunk is a passage that collection has been cut into, a few hundred tokens long, which is the unit a search actually returns. A retriever is the component that takes a question and returns a ranked list of chunks, and the context is what those chunks become once they are pasted into the prompt.

The boundary between the search half and the generation half matters because postmortems routinely turn on a sentence like "retrieval returned the correct document, ranked first", which people read as a clearing statement for the whole system. It is a real finding and it covers exactly one stage of a three-stage pipeline, and the stage boundaries are worth drawing before any experiment gets run across them.

From a corpus to five paragraphs in a prompt#

Offline, the corpus is prepared once and re-prepared whenever it changes. A support knowledge base of 812 documents is split into 4,180 chunks of roughly 600 tokens with 100 tokens of overlap, so that a passage spanning a chunk boundary is still wholly present in one of them. Each chunk is passed through an embedding model that maps it to a 1,024-dimension vector, an embedding model being one trained so that passages with similar meaning land close together in that space and unrelated ones land far apart. The vectors go into an approximate-nearest- neighbour index alongside the chunk text and its metadata, approximate because scanning every one of the vectors exactly would cost more per query than the answer is worth.

The step that makes any of this work is worth being explicit about, because it is the one a reader is likeliest to take on faith. An embedding model is trained so that two passages a person would call related come out as lists of numbers pointing in similar directions, and two unrelated passages come out pointing differently. That converts a question about meaning, which no algorithm can compute directly, into a question about geometry, which is a distance calculation. Searching is then finding the stored vectors closest to the question's vector, and the reason a question and its answer land close together is that the model was fitted on text where they co-occur. Nothing in that procedure inspects whether a passage is true, current, or applicable. It reports proximity in a space built out of what tends to be discussed alongside what.

Online, per request, the question takes the same path and the index is queried.

The dotted edge is the part that happens on a schedule rather than per request, and the two solid paths through the diagram are what "supplying evidence" means operationally. Everything to the left of the Assemble prompt node is retrieval. Everything to the right of it is generation. The two halves communicate through exactly one channel — a string — and neither can see inside the other.

The chunk is the unit of evidence, and its boundaries are a design decision#

The 600-token figure is not a neutral implementation detail; it decides what a piece of evidence can say. A chunk short enough to be precisely retrievable is often too short to carry the qualifier that makes it correct — split "Customers may request a refund within 14 days of the original purchase date" from "Annual plans purchased during a promotional period are subject to Section 6" and you have a chunk that reads as complete and is not. Overlap of 100 tokens exists to reduce that failure at the seams, and it works for short qualifiers sitting adjacent to the fact. It does nothing for a condition four paragraphs away.

The reason that failure is systematic rather than occasional falls out of what a chunk boundary does. Cutting a document into passages discards the relationships that ran across the cuts, and a legal or procedural document is largely relationships running across cuts: a rule in one section, its exceptions in another, its definitions in a third. A chunk is then a self-contained-looking piece of text that is not self-contained, and nothing downstream can tell the difference, because the qualifier's absence leaves no trace. The passage reads complete. The model treats it as complete. A verifier later in this topic will check a claim against it and find it entailed, correctly, by text that does not actually license the claim in context.

The opposite failure appears at the other end of the range. Chunks of 2,000 tokens carry their qualifiers reliably and retrieve badly, because a single embedding has to represent everything in the chunk, so a passage covering four topics has a vector that is close to none of them. Retrieval quality tends to peak somewhere in the 300-to-800-token band for prose corpora and then decline, and the peak is corpus-specific enough that the only honest way to pick is to sweep it against a labelled question set.

Two structural choices sit alongside the size. Splitting on document structure — section headings, list boundaries, table rows kept whole — rather than on a fixed token count keeps the chunk semantically self-contained, at the cost of variable-length chunks that some vector stores handle awkwardly. And prepending a small header to each chunk with the document title and section path ("refund-policy-v4.2 › Section 2 · Eligibility") gives the model something to cite and gives the retriever lexical anchors, for about twenty tokens of overhead per chunk.

Those decisions come back later. When a downstream check needs to verify that a claim rests on a specific span of a specific passage, the chunk is what the offsets are relative to, and a chunking scheme that severs a fact from its condition guarantees that some claims will verify cleanly against text that does not actually license them.

The ranked list is a table with scores in it#

Here is what came back for a question about a 19-day-old purchase, at k = 5, with cosine similarities between the question vector and each chunk vector.

Two observations follow immediately. The correct document came back at rank one, and the 14-day sentence was inside that chunk, so a model that then answered "30 days" was holding the sentence it contradicted. And the chunk at rank two is a help-centre article about the refund process, written for a different audience and full of general language about typical timelines — the most likely origin of an invented processing-time figure, not as a copied string but as material that made a processing-time claim feel like the expected next sentence.

The score is a similarity between two vectors and nothing more. A backend engineer already has the right intuition for this from search relevance: ORDER BY ts_rank(...) DESC LIMIT 5 returns the five rows the ranking function liked best, whether or not any of them answers the question, and the top score of 0.841 is not 84% confidence that the chunk is correct. It is a distance measurement in an embedding space, comparable across results for the same query and barely interpretable in absolute terms. Nothing in the retrieval stage has evaluated whether the chunk is true, current, applicable to this customer, or sufficient to answer the question.

The reason there is no absolute reading available is worth naming, because teams reach for a score threshold every time. A similarity is computed between the question's vector and a chunk's vector, and both of those depend on how the embedding model happens to place text of that kind — which varies with the model, with how long the passages are, and with how narrow the corpus's subject matter is. On a corpus about one product, everything is somewhat about the same thing, so scores bunch high and the 0.702 at rank five is irrelevant, as it is here. On a broad corpus that same figure could be an excellent match. The number orders the five results against each other, which is what it was computed for. It does not measure how good the best of them is, and there is no calibration step anywhere in the pipeline that would give it that meaning.

The call that produces the table is small, which is part of why the stage boundary gets forgotten:

def retrieve(question: str, k: int = 5) -> list[Chunk]:
    qvec = embed(question)                       # 1024-dim, ~11 ms p50
    hits = index.query(qvec, top_k=k)            # ANN search, ~18 ms p50
    return [store.get(h.id, score=h.score) for h in hits]

Most of the engineering effort that gets called "improving RAG" lives in this stage and changes this table. Hybrid search runs a lexical query — BM25 over an inverted index — alongside the vector query and fuses the two ranked lists, which mainly rescues exact-match cases that embeddings blur: product codes, error strings, version numbers, surnames. A cross-encoder reranker takes the top 20 or 50 candidates and scores each one jointly with the question rather than through independent embeddings, which is far more accurate and far more expensive, so it runs over a shortlist rather than the corpus. On this deployment, adding BM25 fusion moved recall@5 from 0.87 to 0.91 and cost about 8 ms; adding a reranker over the top 30 moved it to 0.94 and cost 140 ms. Every one of those numbers is a statement about which chunks appear in the table, and none of them is a statement about the answer.

The prompt is a string, and that is all the model gets#

Assembly turns the ranked list into text. There is no privileged channel: the chunks arrive as ordinary tokens, in the order the retriever put them, wrapped in whatever delimiters the prompt template uses.

from anthropic import Anthropic

client = Anthropic()

def answer(question: str) -> str:
    chunks = retrieve(question, k=5)
    context = "\n\n".join(
        f"[{i + 1}] source={c.source} score={c.score:.3f}\n{c.text}"
        for i, c in enumerate(chunks)
    )
    message = client.messages.create(
        model="claude-opus-5",
        max_tokens=1024,
        system=(
            "You are a support assistant. Answer using the numbered context "
            "passages below. Cite the passage number for each factual claim."
        ),
        messages=[{
            "role": "user",
            "content": f"<context>\n{context}\n</context>\n\nQuestion: {question}",
        }],
    )
    return "".join(b.text for b in message.content if b.type == "text")

The five chunks come to about 2,970 tokens; with the system prompt and the question the whole request is roughly 3,100 input tokens, against a 1M-token context window, so this is a small prompt by current standards and none of the failures in this topic have anything to do with running out of room. End to end, retrieval costs about 29 ms and generation costs a couple of seconds, so the retrieval stage is also not where the latency budget goes.

The numbering in that template is doing more work than it appears to. Passage indices give the model a handle it can cite, they give downstream tooling a key to resolve a citation against, and they impose an order. Order is not free: material at the start and end of a long context is weighted more heavily than material in the middle, an effect strong enough to have its own literature, and here the ordering is inherited from the similarity ranking rather than chosen. Feeding the ranked list in score order puts the best chunk first, which is usually what you want; some teams reverse it so the best chunk sits last, closest to the question, and report a small improvement. With five chunks and 3,000 tokens the effect is minor. At k = 20 and 12,000 tokens it is not.

Notice what the assembly step does not do. It does not check that the chunks agree with each other. Rank one says the window is 14 days; rank two, written before a policy revision, describes a "generous refund window" without a number; rank four is about cancellations, which have a different window entirely. All three go into the prompt with equal standing, distinguished only by their position in the list and a score the model has no calibrated interpretation for. If the model resolves that mixture the wrong way, the retriever did its job and the answer is still wrong.

That last sentence is the whole reason the stage boundary is worth drawing, and it is worth restating as a causal chain rather than as a conclusion. Retrieval's output is a list of chunks. The assembly step turns that list into a string. Generation reads the string as tokens and produces a distribution over the next token at each position. So the only thing retrieval can influence is which text is present, and the only thing that text can do is shift a distribution. There is no step at which "the rank-one chunk is the authoritative one" gets communicated, because there is nothing in the interface that could carry it. A metric computed on the list is therefore a metric about the input to a process whose output it does not constrain, which is exactly why rank@1 exonerates one stage and settles nothing about the other.

What rank@1 exonerates, and what it does not#

Retrieval quality has its own metrics, and they are metrics of the ranked list, computed without ever running the model. Recall@k asks whether the chunk containing the answer appears anywhere in the top k. Precision@k asks what fraction of the returned chunks are relevant. MRR and nDCG score how high the right one sits. On the support corpus above, recall@5 measured 0.91 against a hand-labelled set of 300 questions, meaning that for 91% of questions the answer-bearing chunk was somewhere in the five passed to the model.

That 0.91 is an upper bound on how well the system can possibly do, and nothing more. It says that in 91% of cases the model had what it needed. It says nothing about what the model then did with it, and the 9% failure mode is genuinely different from the failures inside the 91%: when the answer-bearing chunk never arrives, the model is being asked to answer from nothing, and inventing is the expected outcome; when it does arrive, any error is a misreading. Those two need separate fixes and separate owners, and the ranked list is the artefact that tells you which one you are looking at.

So a postmortem line reporting rank@1 is a real and useful finding, correctly scoped: the search half of the system did what it was asked. It does not follow that the system is fine, and it very specifically does not follow that a better retriever would help, because a better retriever's only possible contribution is to put the right chunk higher in a list where it was already first. Reading rank@1 as absolution for the whole pipeline is a category error — the retrieval metric answers a question about the list, and the failure happened after the list.

There is one more thing the ranked list is not, and it comes up whenever a team starts putting scores into prompts. The similarity figure is not a confidence signal the model can reason with. Passing score=0.841 into the context, as the assembly code above does, is useful for human debugging and for downstream tooling; treating it as an instruction the model will weight correctly is optimistic. Chunk five, at 0.702, is about chargebacks and is irrelevant to the question, and there is no threshold at which irrelevance reliably becomes visible in the number.

Supplying evidence, then, is a string operation with a search in front of it, and the string is the entire interface between the two halves of the system. That is what makes the stage boundary useful: everything upstream of it is measured by which chunks appear and in what order, everything downstream by what the model did with them, and no metric from one side answers a question about the other. Knowing which side a failure is on is most of the diagnosis, and the ranked list plus the assembled prompt are the two artefacts that establish it.

With the object defined, the experiment becomes readable. Run the same questions with the retrieved context and without it, label both sets claim by claim using the procedure from chapter 3, and compare the two rates rather than one. The result of that comparison is not the one most teams predict.