Retrieved at rank 1 and answered wrong is not a retrieval bug

Two complaints landed in the same morning, and both looked identical from the outside: a confident answer, a real document in a support corpus that says otherwise, an annoyed account manager. That identity is the problem. The symptom carries no information at all about which stage produced it, so anything triaged on the symptom is triaged on a constant — and a value that is the same in both of the cases you need to tell apart cannot tell them apart.

Four terms are needed to read what follows. A chunk is a passage of a few hundred words cut out of a document and stored under its own identifier. Top-k is how many chunks the retriever returns for a query, and a chunk's rank is its position in that returned list, rank 1 being the highest-scoring. The gold chunk for a question is the chunk that actually contains the answer. And a trace record is the one JSON object per query described in the previous chapter, holding the retrieved chunk ids and scores, the exact string rendered and sent to the model, and the raw output. Pulling the two records out of queries.jsonl took one jq filter each, and they turned out to have nothing in common except the shape of the symptom.

Two records, read as gaps#

The first, call it A, asked what is the SLA credit for a four-hour outage on the Business plan? — an SLA credit being the percentage of the monthly fee a service-level agreement promises back when the service misses its uptime commitment. The answer was "10% of the monthly fee", and policies/sla-credits.md says 25% for outages between two and eight hours. The retrieval block of the record explains itself:

"question": "what is the SLA credit for a four-hour outage on the Business plan?",
"retrieved": [
  {"rank": 1, "chunk_id": "marketing/uptime.md#0",        "score": 0.558},
  {"rank": 2, "chunk_id": "handbook/incident-comms.md#400","score": 0.541},
  {"rank": 3, "chunk_id": "faq/support.md#800",           "score": 0.529},
  {"rank": 4, "chunk_id": "marketing/uptime.md#400",      "score": 0.522},
  {"rank": 5, "chunk_id": "legal/msa-2023.md#3200",       "score": 0.511}
]

policies/sla-credits.md#800 is not there at any rank, and re-running retrieval against the same pinned index puts it at rank 34 with a score of 0.402. The chunk itself is a markdown table of four rows: duration bands in one column, percentages in the other, under a heading that reads Service credits. The query says "SLA credit", the document says "service credit", and the chunk is about seventy percent digits and pipe characters, so there is very little prose for an embedding to latch onto. Nothing downstream of retrieval had a chance, since the fact was never in the room.

That score column is more useful read as gaps than as levels. From rank 1 to rank 5 the scores fall by a little under five hundredths — that is the entire spread of the returned set. The gold chunk sits more than a tenth below rank 5, which is more than twice the spread of the whole top five. It is not a near miss that a slightly better ranking would have caught, and the arithmetic says so precisely: whatever separates the gold chunk from the returned set is larger than everything that separates the returned chunks from each other. That is what justifies reaching for k in the forties rather than nudging it to six.

The mechanism behind that gap is two things compounding rather than one. An embedding model turns text into a vector according to what the text is about, so a chunk that is mostly digits and pipe characters gives it almost nothing to work with, and the resulting vector is dominated by the few words of prose present — here, a heading. That heading then uses a different term from the one the query uses. Low signal on its own would leave the chunk retrievable by a well-matched query; wrong vocabulary on its own would leave it retrievable through the surrounding prose. Together there is neither.

The second, B, asked how many seats are included in the Team plan? The answer was five. The correct figure is ten, and the record shows the correct chunk at rank 1:

"retrieved": [
  {"rank": 1, "chunk_id": "pricing/plans-current.md#0",  "score": 0.791},
  {"rank": 2, "chunk_id": "faq/billing.md#400",          "score": 0.744},
  {"rank": 3, "chunk_id": "pricing/plans-2023.md#0",     "score": 0.739},
  {"rank": 4, "chunk_id": "partners/reseller-faq.md#800","score": 0.702},
  {"rank": 5, "chunk_id": "handbook/billing.md#1200",    "score": 0.688}
],
"prompt_tokens": 2098,
"output": "The Team plan includes 5 seats."

Reading the persisted prompt string shows the sentence The Team plan includes 10 seats; additional seats are billed at €12 per seat per month sitting in the first context block, in full, unmangled. It also shows that the block at rank 3 is the 2023 pricing page, which says five seats and was never removed from the corpus, and that the reseller FAQ at rank 4 repeats the five-seat figure because it was copied from that page. The model was handed one statement of ten and two statements of five, with no instruction about how to resolve a conflict and no signal about which document was current, and it answered with the majority.

B's score column read as gaps says the opposite of A's. Rank 1 leads rank 2 by nearly five hundredths, while rank 2 and rank 3 are separated by half a hundredth — the gap at the top of the list is about ten times the gap immediately below it. The correct page stands apart from everything under it, and the 2023 pricing page sits almost exactly on top of the chunk above it. Retrieval did an unambiguous job here, and on this query the separation at the top is wide enough that a score cut placed inside it would have left the correct chunk alone in the prompt. That the lever for a generation failure can sit in the retrieval config is the first sign that "which stage owns it" and "where the lever is" are different questions.

First divergence, not first suspicion#

The two failures differ in exactly one place: the first point in the pipeline where the artifact stops containing what it should. Running down the record in stage order turns diagnosis into a lookup with a defined answer.

def attribute(trace: dict, gold_chunk_id: str, gold_claim: str) -> str:
    if gold_chunk_id not in index_chunk_ids(trace["index_version"]):
        return "ingest"                       # never indexed; see the manifest
    ranks = {h["chunk_id"]: h["rank"] for h in trace["retrieved"]}
    if gold_chunk_id not in ranks:
        return "retrieval"                    # in the index, outside top-k
    if f"[{gold_chunk_id}]" not in trace["prompt"]:
        return "prompt-assembly"              # retrieved, then dropped or truncated
    if gold_claim not in trace["prompt"]:
        return "chunking"                     # right chunk, claim split across boundary
    return "generation"                       # the model saw it and said otherwise

The order of those four guards is the argument, and it is worth spelling out why it is the right order rather than one of the twenty-three others. A correct answer requires five things to hold at once: the fact is indexed, retrieval returns the chunk holding it, assembly puts that chunk into the string, the claim survives the chunk boundary intact, and the model uses what it was given. Treat each as a factor that is one when it holds and zero when it does not, and the answer is right only when the product is one. Each factor is also a precondition for the next being meaningful — a chunk that was never indexed cannot be retrieved, and a chunk that was never rendered cannot be read — so the factors are ordered by dependency, and the function tests them in that order and returns at the first zero.

Testing them in any other order gives a wrong answer with a straight face. Ask "did the model use it" first and a chunk that was never indexed comes back as a generation failure, because the model demonstrably did not use something that was demonstrably not in front of it. That is the failure mode the word suspicion names: suspicion runs on whichever stage you distrust most, and the record lets you run on dependency order instead.

Five outcomes, each with a different owner and a disjoint set of fixes. A returns retrieval. B returns generation, and the reason that verdict is trustworthy rather than a shrug is that the two preceding checks passed against a persisted artifact — the chunk id appears in the prompt, and the literal string "10 seats" appears in the prompt. Without the prompt string those two checks are inferences, and the function collapses to a guess about whether the model or the retriever is at fault.

generation is the label the function reaches by falling through every other test, which makes it the residual bucket, and a residual bucket is only meaningful when every test above it can actually run. If trace["prompt"] is missing, the two checks that reference it cannot be evaluated at all, and every failure that would have been prompt-assembly or chunking lands in generation instead. The residual then contains three different failures wearing one name — which is precisely how "the model hallucinated" becomes the standing explanation in a pipeline that keeps no prompts.

There is a second property of those early returns worth naming, because it bounds what any count built on this function can claim. The function reports the first divergence and stops. A query where the gold chunk was outside top-k and the template was broken returns retrieval, and the broken template is never looked at. So the label is the first zero in the product rather than the only zero, and a bucket count made from these labels counts first divergences, not causes. The upside is that fixing the first divergence is always the right next move, since nothing downstream of a zero can be evaluated until that zero is lifted.

The chunking branch is worth keeping even though it fires rarely, because it catches the case where retrieval and assembly both did their jobs and the fact was cut in half at a token boundary, so the prompt contains the heading and the first two rows of a table and not the row that answers the question. That failure reads exactly like a generation failure until you compare the chunk text against the claim.

The function needs a gold chunk id, which nobody has at the moment a complaint arrives, and finding it is a grep rather than a research project. The chunk file is line-delimited JSON containing every chunk's text, so searching it for the fact you expected — grep -n '25%' build/chunks.jsonl filtered to documents under policies/ — returns the chunk id in a second, or returns nothing, which is itself the answer and sends you to the ingest manifest. Recording that id in the eval case afterwards is what makes the same failure diagnosable automatically the next time it appears.

What the two columns are doing to each other#

Read the four "moves with" rows as one column negated by the other. Every lever that helps A is inert on B, and the one lever that helps B — prompt changes — is inert on A. That near-perfect opposition is what makes the diagnosis worth doing at all: with two failures pointing opposite ways, a single change chosen without attributing them first has one chance in two of being spent on the wrong one, and no chance at all of helping both.

The row that breaks the pattern is the first one, where both columns say yes, and the break is the informative part. That row is what rules ingest out for both cases, and it is a precondition rather than a finding — if either column read no, every row below it would be moot, because a chunk absent from the index cannot be retrieved, rendered, or reasoned over. The table only means anything because its top row is uninformative.

Two cells carry hedges, and the hedges are load-bearing. "Plausibly" on top-k for A is supported by the arithmetic above: the gold chunk is at rank 34, so k has to reach into the forties, and the gap analysis says a smaller bump lands nowhere. "Likely" on hybrid lexical search is weaker on purpose, because the exact-match route depends on the words the user happens to type — the query said "SLA credit" and the document says "service credit", so a literal-token search is reachable through the shared word and the band labels rather than through the phrase itself. A cell that said "yes" there would be claiming more than the record supports.

The row that does the most work is the top-k one, because it is the row that the instinctive fix gets backwards.

Why raising top-k makes B strictly worse#

The reasoning behind "the model can only work with what it is handed, so hand it more" is sound for A and inverted for B. In B the correct chunk is already first, so every additional chunk is by construction lower-scoring than the one that was already right, and each one carries three distinct costs.

More chunks means more distractors. Going from k=5 to k=20 on B added fifteen chunks, of which two more repeated the five-seat figure — a 2023 changelog entry and a support macro — so the prompt went from one correct statement against two wrong ones to one against four. With a 9,000-token budget so nothing was dropped, the answer at temperature 0 was still five seats, and the model now had a numerical majority of four to one behind it.

More chunks means more chance of two versions of the same document in one prompt. At k=5 the corpus's superseded pricing pages had a 3-in-9,842 chance each of appearing; at k=20 they are near-certain, because they are lexically and semantically almost identical to the correct page and therefore sit immediately behind it in every ranking. The specific pathology of a document corpus is that the nearest neighbour of the right answer is usually the previous version of the right answer.

The two clauses either side of that semicolon are measuring different things, and the jump between them is the argument. The chance figure is a null-model number — the odds if chunks were being drawn blind out of an index — and the clause after it is the statement that chunks are not being drawn blind. A ranked retriever exists precisely so that selection is not random, and what it selects on is textual closeness to the query. A superseded version of the right page is about as textually close to that query as a document can get. Ranking quality and version contamination therefore rise together, which is why this particular cost cannot be tuned away by improving the retriever: the better it gets, the more reliably it surfaces the wrong version alongside the right one.

And more chunks means distance. Long-context models do not weigh positions uniformly, and content in the middle of a long context is used less reliably than content at the beginning or the end — the effect is well documented and reproducible on your own pipeline in about twenty minutes. The controlled version on B: hold the retrieved set at twenty chunks and vary only where the correct chunk sits in the rendered block.

The middle column is the control, and its constancy is what the first three rows are for. Position moved and the token count did not, which is the evidence that nothing else moved either — same chunks, same template, same budget, same string length, different order. Where the token count does change, in the last two rows, the set changed too, and those rows are answering a different question.

Read the whole table as one rule and it closes with no exception. The answer is correct in exactly the rows where the correct chunk is the last thing before the question, and wrong in every row where something else follows it. Row three has it last inside a twenty-chunk set and gets ten seats; row five has it last because it is the only chunk, and gets ten seats. Rows one, two and four have it first or in the middle, and all three get five seats. Five rows, one predicate, no residual — which is a great deal more than "position matters", because it names which position.

Two comparisons inside the table isolate the variables cleanly, and one row refuses to be isolated. Rows one to three hold the set fixed and move position: the answer flips, so position alone is sufficient to change it. Rows one and four hold position fixed at first and change the set size from twenty to five: the answer does not move, so set size alone is not sufficient. Row five changes both at once — the correct chunk is simultaneously last and alone — so it is over-determined, and the table cannot say which of the two mechanisms produced its result. That does not weaken it as a control, because a control does not need to isolate a mechanism.

The last row is the control that makes the diagnosis final: given only the correct chunk, the model answers correctly every time, so no property of the model or the chunk explains the failure. The failure is a property of the set of chunks assembled next to it, which is a decision your render stage makes.

There is a second-order cost that shows up on the bill rather than in the answer. Going from k=5 to k=20 took the input from 2,098 to 8,240 tokens per query, which at $3 per million input tokens is a move from $0.0063 to $0.0247 per query, or from $19 to $74 a day at 3,000 queries a day, and it added roughly 600 ms of time-to-first-token. Paying four times as much to make an answer worse is a specific kind of expensive.

Three pairs of figures in that sentence stand in one ratio, and the agreement is the check that the arithmetic holds together. The token counts stand in a ratio of about four to one; the per-query costs stand in the same ratio, because a fixed price per token cannot change it; and the daily bills stand in it again, because a fixed query rate cannot change it either. A fourth reading confirms the two rows describe one pipeline with only k moved: the extra tokens divided by the fifteen extra chunks lands almost exactly on the chunk size implied by those #400-strided identifiers. And the previous chapter's arithmetic for a full top-five render at that chunk size — a prediction, multiplied out rather than measured — lands on B's measured prompt_tokens, which is what a query with nothing dropped is supposed to look like.

What you could diagnose with only the question and the answer#

Neither A nor B, strictly. With a question and a final answer you have the symptom and no artifact from any stage, so every statement about which stage failed is a hypothesis. What varies is how much of the evidence can be reconstructed afterwards, and the two cases differ sharply.

A is partially recoverable. Retrieval is deterministic given a fixed index, so if the build that served Tuesday's query still exists and nothing has been re-embedded, re-running the query reproduces the same top-5 and the same scores, and the record you never wrote can be regenerated. Every one of those conditions is a condition about ingest, and a corpus that receives eleven new documents on a Wednesday breaks all of them at once.

B is not recoverable at all. Its diagnosis rests entirely on the claim that the correct text was in the prompt, and the prompt is a function of the retrieved chunks, the template, the chunk boundaries and the budget clamp as they stood at that moment. Any of those four having changed makes the reconstruction a different string, and a template edit leaves no version marker at all unless you put one there. This asymmetry is what makes the ordering in the previous chapter concrete: the prompt string is the artifact whose absence permanently destroys a class of diagnosis, and it is also the cheapest one to keep.

The asymmetry runs the same way through the function above. Both of B's decisive checks read trace["prompt"], and neither has any fallback, so B's diagnosis is available exactly when that field was written and unavailable otherwise. A's decisive check reads trace["retrieved"], which is the one part of the record a deterministic index can regenerate. The cases that survive losing the prompt string are the cases you were least likely to misdiagnose in the first place.

"Generation failure" is a sourcing conflict, not a hallucination#

Attributing B to generation is not a synonym for hallucination, and using the word ends the investigation one step too early. The model did not fabricate five seats out of its weights; it read five seats from a document you retrieved and put in front of it, which is a sourcing conflict created by your corpus and your assembly, and it would have been resolved by any of several instructions the prompt did not contain. Calling that a hallucination assigns the failure to a component you cannot change and skips the four you can.

The word has the same defect as the symptom this chapter opened on, one level up. "The model hallucinated" is available as an explanation for every one of the five outcomes the function can return — a chunk that was never indexed, one that was outranked, one that was truncated away, one cut at a boundary, and one the model genuinely overrode. It is constant across the whole taxonomy, so it discriminates nothing, and an explanation that fits every case supplies no information about which case you have.

It also does not mean the fix is a better model. Both failures here reproduce on frontier models, because A is an information-retrieval problem in which the needed text is absent, and B is a conflict-resolution problem the prompt never posed. A larger model helps at the margins with B and does nothing at all for A, at several times the cost per query.

What B leaves is a specific, bounded set of remaining moves, all of them in the block you render: the order the chunks appear in, the delimiters that stop one document bleeding into the next, an instruction that confines the answer to the supplied context, a requirement to cite the chunk each claim came from, and an explicit rule for which chunk gets dropped when the budget overflows instead of leaving that to a break statement.

Every one of those is a decision about the string, and the string is the one factor in the product that neither the retriever nor the model owns. That is the whole content of "not a retrieval bug": rank 1 established that the second factor was one, the prompt string established that the third was one, and what remains is a stage nobody had been treating as a stage at all. The next chapter works through those moves, and through the discipline that has to accompany any of them, because a prompt change that fixes B and quietly breaks four queries you were not looking at is not a fix.