Answer coverage in retrieved passages

Coverage is the property that the material handed to the model actually states the fact the answer needed. Relevance is the weaker property that the material is about the same subject as the question. The two come apart constantly, and the distinction matters because a retrieval system reports the second and nothing at all about the first. In a retrieval-augmented setup — a knowledge base cut into chunks of a few hundred tokens, each stored as a vector positioning it by meaning, with the chunks nearest the question pasted into the prompt — every request comes back with its top k passages and a confident-looking score beside each one. Those scores describe how the candidates ranked against each other. Nothing in them describes whether any candidate answers the question.

That gap is where a large share of single-trace investigations go wrong, because the observable failure is a model stating something false and the available evidence is a log full of high numbers. The material to work from is a frozen trace (chapter 1, where one request and its response are captured whole — rendered prompt, retrieval log, response and metadata — so the same bytes can be re-read without the live system moving underneath). This chapter is about closing the gap with a check you can run in a few minutes against such a trace, and about the three distinct causes hiding behind "the answer was not in the context", each with a different fix and none of them a prompt edit.

What a similarity score measures#

Cosine similarity between a query embedding and a chunk embedding is a statement about how close two vectors are, produced by a model trained so that text on similar subjects lands nearby. It ranks the candidates in your index against each other. It is not a probability, it is not calibrated across queries, and there is no threshold above which "this passage answers the question" becomes true.

The reason no such threshold exists is in how the two vectors were made. An embedding model is a function from a piece of text to a fixed-length list of numbers, trained on pairs of texts that belong together — a heading and the body under it, a question and a passage somebody judged a good result for it — with the training pushing the two members of each pair to point in similar directions. What that objective rewards is landing on the same subject. A page explaining how to start a return and a question asking how long you have to return something are a pair of that kind, and the model is doing what it was built to do when it puts them close together. Models tuned on question-and-passage pairs do pick up something about answering, and that is an average tendency across a benchmark rather than a guarantee about one query in your corpus.

The second half of the mechanism is when each vector gets computed. A chunk's vector is produced at ingestion, from the chunk alone, possibly months before anybody asks anything. No question exists at that moment, so those numbers cannot encode which questions the chunk would answer; they can only summarise what the chunk is about. At query time the question is embedded on its own by the same model, and the two summaries are compared. The comparison is therefore between two descriptions of subject matter, and the thing you needed is a fact rather than a subject.

One kind of scorer does see both texts at once. A cross-encoder re-ranker reads query and passage together, which puts it in a position to notice that a passage states what was asked for. Whether it does depends on what its training data labelled, and the usual label is a human relevance judgement — was this a good result for this query — rather than answer containment. Make anyone treating a re-ranker score as a coverage signal say which of the two their labels recorded.

A further reason the score is insensitive to exactly the property under investigation is dilution. A chunk vector is one summary pooled over every token in a few hundred tokens of text, and the clause carrying the answer is a dozen words. Deleting it from a chunk about returns moves the pooled vector correspondingly little, so two chunks differing only by the sentence you needed land close together, and the ranking has almost nothing there to separate.

The number that makes this concrete is the floor rather than the ceiling. On a typical support corpus with a normalised embedding model, two documents about completely different subjects still score around 0.72 to 0.78 against each other, because they share the same house style, the same navigation boilerplate, and the same closing line about contacting support. Those elements sit in every document, so every document vector carries a large common component, and a component present in both vectors of a pair pushes their similarity up before subject matter is consulted at all. The mechanism is the dot product, from the chapter of that name in the topic on vectors: the score is built by multiplying the matching positions of two vectors and summing the products, so agreement at a position raises the total whatever the rest of the vector is doing. The corpus's own furniture sets a floor that no pair of its documents falls below.

Anything genuinely on-topic sits somewhere from 0.84 up. Put the two figures together and the working range is narrow: from the start of on-topic to the top of the scale is roughly a sixth of a point, and every discrimination the retriever makes happens inside that. The difference between a passage that states a 14-day return clause and a passage that merely discusses returns is frequently smaller than 0.02, which is about an eighth of the working range — well inside the amount that chunk length, heading text and boilerplate density move a score around on their own. A quantity whose signal is smaller than its own noise cannot be thresholded, and that is before asking what the quantity measures.

Both figures belong to one corpus and one embedding model rather than to the technique, and a reviewer should make anybody quoting them measure the equivalent for the index in front of them: score some deliberately unrelated pairs from your own store to find the floor, then some on-topic ones to find where the band starts. The shape of the argument survives a change of numbers, and the numbers do not travel.

Retrieval also has no null result. A top-5 query returns five chunks whether or not the index contains anything relevant, because ranking is relative. Sorting a list and taking its head is defined for any list with items in it, and no step in that operation can express "none of these is good enough", since it only ever compares candidates against each other. If the answer was never ingested, the five best-ranked passages are simply the five closest pieces of adjacent prose, and they arrive with respectable scores attached. That combination — always five results, always confident-looking numbers, no signal for absence — is what makes the score field so misleading as a proxy for coverage. A similarity floor bolted on top is worth having, since it suppresses the case where a query is off-topic for the whole corpus, and it cannot establish coverage for the reason above: the quantity being thresholded never saw the answer. Nor is a floor portable between corpora, the score being uncalibrated across queries.

What replaces it is reading. The trace to read here is one where a shop assistant told a customer that returns are accepted within 45 days, against a real policy of 14, and two of its fields come before the passages. The rendered request — the single string the messages were expanded into before the model saw them (chapter 2) — passed its template check, so the blocks below did reach the model in the form shown. And the response's envelope carries finish_reason: end_turn. The finish_reason field records why generation stopped, and end_turn is the value meaning the model stopped because it had finished its turn, rather than the value reporting that generation hit a length limit. That field takes a second to read and removes a competing explanation: a truncated answer and an unsupported answer look alike from the output text alone, and a wrong number is what a longer, properly hedged answer cut off mid-flow would leave behind. end_turn says the model wrote what it meant to write, and ruling out truncation is what leaves coverage as the next thing to check.

Here are the five retrieved passages, with the retriever's scores.

[0] 0.913  kb/returns/how-to-return.md
    "To start a return, open Orders, select the item and choose Return.
     Print the label and drop the parcel at any partner location..."

[1] 0.887  kb/shipping/delivery-times.md
    "Standard delivery arrives in 3-5 working days. Returns are collected
     on the same schedule..."

[2] 0.881  blog/2025-holiday-returns.md
    "This holiday season we're extending our returns window. Shop with
     confidence through January..."

[3] 0.874  kb/returns/refund-timing.md
    "Refunds are issued to the original payment method within 5 working
     days of the parcel being scanned at our warehouse..."

[4] 0.869  kb/orders/cancelling.md
    "Orders can be cancelled before dispatch from the Orders page..."

Not one of those five states a return window. They are all unmistakably about returns, shipping and refunds, which is exactly what a similarity search over that query should surface, and it is why the scores are high.

Two beliefs compete at this point and both are tempting. The first is that a high retrieval score means the evidence was present, which is tempting because the score is bounded, reads like a confidence, is the only number in the log, and because in the ordinary case where retrieval works a high score does coincide with the answer being there. That correlation is real and it runs the wrong way to be used as evidence. Presence of the answer usually produces a high top score; observing a high top score does not run the implication backwards, because the same observation has other sufficient causes, and five topical passages that state nothing are one of them.

The second belief is that the model invented the number, with a string search as its supporting evidence: the wrong figure appears nowhere in the knowledge base. That search establishes what it says and no more. It tells you where the wrong answer did not come from, and the question under investigation is where the right answer went, so the search to run is for the fact the answer needed rather than for the one the model produced. Reading the retrieved passages for that fact settles between the two beliefs, and its two outcomes point at different systems. If no retrieved passage states a return window, invention was the only behaviour available to the model and the fault sits upstream of it. If a passage does state 14 days and the reply says 45, the evidence was in front of the model and went unused, which is the harder problem the rest of this topic takes up.

The model was asked a question, given five passages that discuss the subject and answer something adjacent to it, and produced the most plausible number for an e-commerce return window that its pretraining supports. Forty-five is not a random number; it is a common policy at large retailers. The system behaved the way an instruction-following model behaves when the evidence is missing and nothing licenses it to say so.

Three places the evidence goes missing#

"The answer was not in the context" has three distinct causes with three distinct fixes, and separating them takes one query each.

They sit at different stages of one pipeline, which is what makes them separable at all. A document has to be ingested before it can be indexed, indexed before it can be retrieved, and retrieved before it can be assembled into a prompt, each stage reading the previous stage's output and writing its own. A fact absent at the end failed to survive one of those handovers, and "at which handover" is a question with a definite answer.

The clause was never indexed. Ingestion walks a source of documents, extracts text from each, splits it and writes the pieces to the index, and every one of those steps has conditions under which it quietly produces nothing: a size limit that skips a large file, an unsupported file type, a scanned PDF whose pages are images with no text layer, a nightly run that failed and left yesterday's index in place. All of them leave the same downstream picture, an index with no such sentence in it, and none of them raises anything at query time, because the query path cannot know that a document it never received exists. Checking this means searching the source — the object store, the CMS export, the repository — for the string, rather than searching the index. If the phrase exists in the source and not in the index, the bug is in ingestion, and no amount of prompt work touches it.

The clause was indexed but not retrieved. This is the chunking-and-vocabulary case. A fixed 512-token splitter can cut a terms document between the heading "Returns and refunds" and the sentence "You may return any unworn item within 14 days of delivery", so the chunk carrying the sentence has no topical anchor and embeds as generic legal prose. The mechanism from the previous section then does the damage twice: the chunk is summarised by what it is about, what it is about has become legal boilerplate rather than returns, and the single clause that would have tied it to the query is diluted across a few hundred tokens of surrounding text. Or the documents say "return" where the user says "refund" and the embedding model does not bridge the gap for this domain. The test is to query the index directly for text you know exists and read its rank: if the right chunk is at position 19 for the user's query, retrieval is the problem and its shape is now specific. The rank is more informative than the hit, and a second query sharpens it further — a chunk that ranks badly for the user's phrasing and well for a phrasing lifted out of the document itself is a vocabulary mismatch, while a chunk that ranks badly for both is a chunking or embedding problem.

The clause was retrieved and then lost on the way into the prompt. This is the one that produces the most confusing traces, because the retrieval log and the prompt disagree and only one of them is usually inspected.

Only the third of these leaves a mark inside the trace, and that asymmetry decides the order the checks run in. It announces itself as a disagreement between two records the trace already holds: a document id in the retrieval log with no matching block in the rendered request, or a block whose body stops mid-sentence. The first two leave the trace looking identical to one another — a consistent retrieval log, every retrieved block present and intact, no clause anywhere — because in both worlds the retriever ranked what it had and the assembler delivered it faithfully. Telling those two apart needs a query the trace cannot answer, and that blind spot is why the wrong one gets blamed.

Run the queries from the source outwards. A negative result early makes every later reading meaningless, while a positive one never clears what comes after it. If the sentence is not in the source at all, the rank query returns whatever legal prose the index holds nearest the query, and reading that rank as evidence about retrieval is how a team spends a weekend on the splitter. A hit in the source establishes only that ingestion had something to work with, and the index query is still owed.

The retrieval log is not the prompt#

Between the retriever and the model sits an assembly step that most systems grow organically: a token budget, a per-block character cap, a deduplication pass, a re-ranker that rescores the shortlist with a slower and more accurate model, a formatter that joins blocks with a separator. Every one of those can remove or damage evidence, silently, after the scores were recorded.

Follow one block through it. The ranker's output is a list of document ids with scores attached, and that is the whole of it — no text has moved yet. The assembler takes each id and fetches a body from a document store, a second lookup that can return a revision other than the one that was embedded. It re-ranks or trims the shortlist, drops near-duplicates, cuts each body down to a character cap, and accumulates bodies until a token budget is exhausted, discarding whatever is left over. It escapes, wraps and joins the survivors into one string, and the template step of chapter 2 places that string into the prompt the model receives.

Every one of those steps happens after the ranker has returned, and information flows one way across that boundary: the assembler reads the ranker's output, and nothing writes back. The retrieval log is usually emitted at the moment of ranking, which puts it on disk before any later step has run. It records what the retriever decided to send, and the rendered request records what arrived. Reading the first as a description of the second is reading a decision as though it were a delivery, and they are kept as separate records precisely because they can disagree.

The eviction is also not neutral about which block it takes. A budget loop popping from the tail removes the lowest-ranked block, and the section above says what tends to sit there: a passage stating one fact in one legalistic sentence looks less like the query than a passage whose whole subject is the query's subject. The property that makes coverage invisible to the score is the same one that makes covering passages the first thing a budget loop throws away.

The check is a direct comparison between the two records already in your frozen trace.

import re

rendered = trace["rendered_request"]
retrieved = trace["retrieval"]["doc_ids"]          # what the retriever returned
present = re.findall(r"<document id=\"([^\"]+)\">", rendered)  # what got sent

print("retrieved:", len(retrieved), "present:", len(present))
print("dropped  :", [d for d in retrieved if d not in present])

for doc_id in present:
    body = re.search(
        rf"<document id=\"{re.escape(doc_id)}\">(.*?)</document>", rendered, re.S
    ).group(1)
    print(f"{doc_id:36s} chars={len(body):5d} ends_mid_sentence={not body.rstrip().endswith(('.', '!', '?'))}")

On one investigation this printed retrieved: 6 present: 5, with legal/terms-of-sale#c12 in the dropped list. The block had been ranked sixth, the assembler's budget was 8,000 tokens for context, the first five blocks consumed 7,840 of it, and the sixth was discarded without a log line. The evidence was found, ranked, scored, and then thrown away by a while total > budget: blocks.pop() loop written eight months earlier for a different prompt. Redis's write-up "Context Window Overflow: Fix LLM Errors Fast" is a good short treatment of why this class of eviction degrades answers with no error raised anywhere in the stack.

Subtracting the two budget figures is worth the ten seconds, because the remainder names the kind of failure. The space left over was a hundred and sixty tokens, and a chunk of a few hundred tokens does not fit into a hundred and sixty of them. Nothing about the sixth block was marginal and no part of it survived, because a loop popping whole blocks makes an all-or-nothing decision and leaves no partial delivery to notice. Whether it survives on any given request depends on how long five unrelated passages happen to be, so the same question can be answered correctly one day and wrongly the next with nothing in the index having changed.

The ends_mid_sentence column catches the softer version. A per-block cap of 1,200 characters applied after retrieval will happily cut "You may return any unworn item within 14 days of deliv" and leave the model with a truncated clause that is worse than no clause, because it looks like evidence. Note which check catches it and which does not: comparing the two id lists reports the block as delivered, its id being present in both records, and only reading the body shows the sentence stopping before the number. Formatting damage belongs in the same family: a markdown table of plan limits flattened into a single line of pipes, HTML entities left as <, or blocks concatenated with "\n" so that the last line of one passage reads as the first line of the next. All of these are visible in the rendered request and invisible everywhere else.

Reading the passages by hand is the right method at this scale, and it is worth saying why rather than reaching straight for a model-graded coverage check. Five passages is four minutes of human attention and produces a certain answer. A judge model asked "does this passage contain the return window" over the same five passages will usually agree with you, and when it disagrees you have acquired a second unexplained output to debug on top of the first. The trade turns over at volume, four minutes apiece across a thousand traces being more than sixty hours of reading. Automated coverage judging earns its place there; for one trace it adds a component whose reliability you would then have to establish.

Reading what each observation is worth#

Because the same investigation touches four different records — the source documents, the index, the retrieval log, the rendered prompt — it helps to be explicit about which claim each one supports.

Each row below is an implication that runs in one direction only, and the mistake the table exists to prevent is running one of them backwards. The right-hand column is not a list of caveats: it is the set of statements still open after the observation has been made, and therefore the set of things still worth a query.

The last row holds only once the same rendered request has been checked for truncation and formatting damage, since a block carrying the number in a mangled form is present to a regex and noise to the model. Given that condition, each check that comes back clean is worth more than "no bug found here": the three sites partition the space of coverage failures between them, so a clean result removes a branch rather than merely failing to find something.

That is where this chapter stops being sufficient. When the coverage check comes back clean — the clause is in the source, was retrieved, is present and intact in the rendered request, and the model still said 45 days — you have eliminated an entire class of cause and arrived at a harder question, which is which part of a 6,000-token context produced the wrong number.

A coverage check is not retrieval evaluation#

Recall@k over a labelled query set answers "how often does our retriever surface the right passage", which is an aggregate property measured over hundreds of queries and reported as a percentage. What this chapter describes is a single-trace check with a binary outcome for one request, and the two are complementary rather than substitutable: the aggregate number tells you whether to invest in retrieval this quarter, and the single-trace check tells you whether this particular bug is a retrieval bug at all.

A healthy aggregate is also weaker cover than it looks, for a reason built into how its labels are made. A labelled query set is assembled by taking documents you have and writing questions they answer, so every query in it has its answer in the corpus by construction, and nobody writes a question against a PDF the pipeline never ingested. Recall@k measures the retriever against the documents that got in, leaving the first of the three sites invisible however high it climbs.

It also does not tell you which of the three sites failed until you run the three separate queries, and the temptation is to skip that. A team that finds the answer missing from the prompt and immediately concludes "our chunking is bad" will rewrite the splitter, redeploy the index over a weekend, and discover that the document was never ingested. The queries against source, index and rendered request take a few minutes each and they discriminate between three fixes that share no code.

Coverage also says nothing about conflict. A context that contains the 14-day clause and a holiday marketing page promising an extended window contains the answer by any coverage test you can write, and can still produce a wrong output. That situation is the subject of the ablation work later in this topic, ablation being the practice of removing or replacing one piece of the input at a time and watching whether the output changes.

The habit worth taking from all of this is small: when a model states a wrong fact, find out whether the right one was in front of it before deciding what kind of failure you are looking at. The scores in the retrieval log cannot answer that question, having been computed by comparing two summaries of subject matter, neither of which was ever shown the answer. The passages can, and reading them takes minutes.

Before any of that, there is a problem with the way the last few paragraphs have been reasoning. Every check so far has been a comparison between records, which is safe. The next set of moves involves changing something and re-running, and a re-run only means something if you know what a re-run is worth — whether your stack reproduces the failure at all, and how often the bad output appears when nothing has been changed.