Measuring the effect of retrieval

The experiment is the obvious one and the design has a trap in it. Two arms, same questions, same system prompt, same model; one arm gets retrieved passages in its context and the other gets none. What makes the comparison hard to read is that intrinsic hallucination is defined against supplied context, so an arm with no supplied context has no intrinsic errors by construction — everything it says is unsupported, and the two arms end up measured on incompatible scales.

It is worth seeing why that is a trap rather than an inconvenience. A definition that refers to "the supplied context" produces a different yardstick for every arm that supplies something different, so a table comparing the arms is comparing two quantities that share a name and not a meaning. The failure is silent, because both columns are populated, both are computed by the same code, and the arithmetic is correct in each. What has gone wrong sits one level up, in what the numbers are of, and no amount of checking the calculation surfaces it. Any measurement whose definition mentions an input has this hazard whenever the experiment varies that input.

The fix is to separate the evidence the model receives from the evidence the annotator judges against. Both arms are labelled against the same hand-picked gold passages — gold in the sense the word carries throughout evaluation work, meaning a reference a human has certified as correct — and only the no-retrieval arm never sees them. This chapter runs that design over 400 real questions from an engineering-documentation assistant that had been running without retrieval at all — a system prompt, the user's question, and nothing else — and that had invented runbook PAY-114 section 3.2 in the course of doing so. Adding retrieval over a runbook corpus was obviously going to help, and the team wanted a number before rolling it out. The chapter reports what moved and what did not with intervals around both, explains the asymmetry from what generation actually does, and then sweeps the number of retrieved passages to find where more evidence stops helping.

Holding the evidence fixed while varying what the model sees#

def run_arm(questions: list[Question], *, with_retrieval: bool) -> list[dict]:
    records = []
    for q in questions:
        chunks = retrieve(q.text, k=5) if with_retrieval else []
        answer = generate(q.text, chunks)          # identical system prompt in both arms
        records.append({
            "id": q.id,
            "shown": [c.text for c in chunks],     # what the model got
            "gold": q.gold_passages,               # what the annotator judges against
            "answer": answer,
        })
    return records

baseline = run_arm(EVAL_400, with_retrieval=False)
treatment = run_arm(EVAL_400, with_retrieval=True)

Under that rule, a claim in the baseline arm that flatly contradicts the gold passage counts as intrinsic even though the model never saw the passage. That may feel unfair to the baseline, and it is deliberately so: the point of the measurement is not to grade the model's effort but to ask which class of error the system commits, on a scale that stays fixed while the intervention changes. The alternative — recomputing the yardstick per arm — makes the two columns non-comparable and produces exactly the kind of number that survives a review meeting without meaning anything.

What moved and what did not#

The extrinsic column collapses. Unsupported claims fall by a factor of six and a half, which is the whole of the improvement — the intrinsic column does not fall at all, and drifts slightly upward. Retrieval, on this corpus, did one thing: it stopped the model inventing. It did nothing whatever about the model getting supplied things wrong.

Two of those rows are worth reading as arithmetic rather than as findings. The extrinsic figure falling from 15.9% to 2.4% is a drop of 13.5 points, and dividing that drop by the 15.9 it started from is the −85% in the change column — a proportional statement, not a subtraction. The intrinsic figure moving from 6.1% to 6.6% is a rise of half a point, and the same division against 6.1 gives the +8%, which is a large-looking percentage sitting on a very small absolute move. Reading the two change figures side by side without noticing that they have different denominators is the commonest way this table gets misquoted, and the defence is to keep the absolute rates in view alongside the ratios.

The last row is the one to carry forward. Before retrieval, roughly three quarters of the defects were inventions and one quarter were contradictions. Afterwards the proportions have inverted, and 73% of what remains is the model producing claims that conflict with text sitting in its own context. The residual 9% is not a smaller sample of the same problem; it is a different problem that was always there, previously buried under a much larger pile of invention.

The small rise in the intrinsic column has two contributions and neither is alarming on its own. Retrieval makes the model more specific: the treatment arm produced 8% more claims per response, and the extra claims are the concrete ones — figures, conditions, named sections — which are precisely the claims that can conflict with a source. A vaguer answer has fewer opportunities to contradict anything. On top of that, four of the five retrieved chunks are usually about the topic without answering the question, and a chunk that discusses an adjacent policy is a source of confusion the no-retrieval arm never had.

Because the two arms ran over the same 400 questions, the comparison can be read as paired rather than as two independent samples, and pairing is worth the small amount of extra work. The quantity of interest is the per-question difference in defective claims, and a bootstrap over questions — resample the 400 question-level differences with replacement, 10,000 times, and read the 2.5th and 97.5th percentiles — put the drop in the extrinsic rate at 13.5 percentage points with a 95% interval of 11.9 to 15.1, and the change in the intrinsic rate at +0.5 points with an interval of −0.4 to +1.4.

Resampling is used here rather than a formula because the quantity of interest has no tidy distribution behind it. A per-question difference in defective claims is a count divided by a count, taken over responses of varying length, with claims inside a response correlated with one another — none of which satisfies the assumptions that make a closed-form interval valid. The bootstrap replaces those assumptions with an experiment. Build ten thousand alternative samples by drawing with replacement from the one you actually have, compute the statistic on each, and read the spread directly off the results. What it estimates is how much the answer would have moved had a different 400 questions come in, which is precisely the uncertainty a reader of the table needs to know about. The first interval is nowhere near zero and the second straddles it comfortably, which is the whole finding stated properly: one column moved, the other did not.

Resampling over questions rather than over claims matters here, because claims within a response are not independent — one wrong premise produces a cluster of related defects, and treating each as an independent draw shrinks the intervals to something like half their honest width. It is the same reason the inherited-claim flag from chapter 3 exists, applied to the statistics instead of the counting.

Evidence removes the need to invent, and nothing more#

The mechanism behind the asymmetry follows from what generation does. Asked about a retry budget with nothing in context, the model has one source for the next token — the patterns in its parameters — and those patterns will happily produce a small integer, an exponential backoff and a runbook citation, because that is what such answers look like. Put the actual runbook in the prompt and the local statistics change: three is now the token that fits the immediate context, because the surrounding text says three, and the model's continuation follows the strongest available pattern. Retrieval works by making the truth also be the likeliest continuation.

Where it fails is exactly where that alignment breaks. If the supplied passage is long and the relevant sentence is buried in the middle; if two retrieved chunks disagree; if the passage phrases the answer in a way that conflicts with a strong pretraining prior; if answering requires combining a figure in chunk one with a condition in chunk three — in all of those cases the highest-probability continuation is no longer the faithful one, and nothing in the pipeline is checking. The model is not instructed to follow the evidence in any enforceable sense. It is given text, and text influences the distribution over next tokens. Influence is not compliance.

That is the mechanism behind every number in the table, so it repays being stated as a chain. Before retrieval, at the position where a retry count is about to be emitted, nothing in the prompt bears on retry counts, so the distribution is whatever the weights alone favour — a small integer, chosen by prior frequency. Fabrication is not a malfunction at that position; it is the only behaviour available. After retrieval, the surrounding text now says three, and a token that matches nearby text is strongly favoured by a model fitted on human writing, where consistency within a passage is overwhelming. So the faithful token becomes the likeliest one and the invention stops.

The second half of the chain is where the residual lives. That mechanism moves the odds; it does not impose a constraint. Anything that weakens the local signal — distance between the fact and the position that needs it, a phrasing that fights a strong prior, two passages saying different things — leaves the faithful token merely competitive rather than dominant, and a competitive token loses some of the time. Nothing in the pipeline notices when it does, because losing produces a fluent sentence exactly like winning does.

This is why the interventions that reliably move the extrinsic number do nothing for the intrinsic one. Better embeddings, a reranker, a bigger k, hybrid keyword-plus-vector search, more aggressive chunk overlap — every one of these changes which text arrives, and none of them changes what happens after it arrives. The instinct when 22% becomes 9% is to read the remaining 9% as the part retrieval has not reached yet, and to conclude that a better retriever finishes the job. The measured composition says otherwise: retrieval already reached almost everything it can reach, because what it addresses is the absence of material, and material is now present.

More context is not monotonically better#

Sweeping k on the same evaluation set shows the trade directly.

The last column of each row is the sum of the two before it, and noticing that is what makes the optimum readable. Extrinsic and intrinsic are disjoint by construction, since a claim the context bears on cannot also be a claim it does not bear on, so the defect rate is their total. That total is a falling curve added to a rising one, and a sum of that shape has a minimum somewhere in the middle, positioned only by how fast each of the two is moving. At k = 3 the extrinsic curve is still falling faster than the intrinsic curve is climbing. By k = 10 the first has flattened and the second has not, so the sum turns back up. The shape is general; only the location of the turn belongs to this corpus.

Extrinsic errors fall monotonically, with sharply diminishing returns after k = 3. Intrinsic errors rise once k passes 3, and by k = 10 the increase is large enough to make the total worse than at k = 5. Each additional chunk is another passage that may be stale, may cover an adjacent policy, or may state a superficially similar fact about a different subject, and the model resolves conflicts between them with no reliable rule. The overall optimum on this corpus sits at k = 5 and is shallow; a team tuning k upward on the theory that more evidence is safer would have made the system worse while watching a retrieval metric improve.

The token column matters too. Going from k = 5 to k = 10 doubles input cost per request for a 0.4-point improvement in one column and a 1.7-point regression in the other. Retrieval tuning has a floor, and the floor is reached quickly.

Two cautions about transferring these numbers. The location of the optimum is a property of a particular corpus, a particular chunk size and a particular question distribution, so the shape of the curve generalises and the value of k does not; the sweep costs one evaluation run per setting and is worth doing locally. And the arms here were labelled against gold passages that a human picked, which is a deliberately generous yardstick for the retriever — measured against what the retriever actually returned, the intrinsic rate would move, because some of what the model contradicted was never on its own screen.

Retrieval is a control on one failure type and a non-control on the other, and seeing that in a table changes what a team does next. It stops the reflexive retrieval sprint after an incident where the right document was already at rank one; it sets an expectation that the intrinsic column will not fall when the embeddings improve; and it says where the ceiling is, which is around 9% defects per claim on this system with the retriever performing well. Anything below that ceiling has to come from somewhere other than the search half of the pipeline.

Two negative results are now on the table: the failure is not a decoding problem, and it is not, past a low threshold, a retrieval problem. The intervention almost every team tries next is a sentence in the system prompt asking the model not to make things up. That has been measured too.