Triaging a wrong answer

Two things have to be fixed before any of this works, and both are cheap. The gold passage is the passage you have confirmed does contain the correct answer. The answer span is the short piece of its text that states the fact — "21 calendar days", next_cursor, the number itself. With those two written down, a wrong answer stops being a mystery and becomes a lookup in a retrieval-augmented pipeline's own output. Establishing them takes a full-text search of your corpus for the fact and a minute of reading, and it has to happen before any triage, because a complaint whose gold passage has not been identified cannot be distinguished from a question the corpus does not answer — which is a different category with a different response, and the subject of chapter 7.

The states are worth naming before the tests, because the names are what a triage result gets recorded as. State 1 is a retrieval miss: the span never appeared among the candidates. State 2a is losing it in the cut: it was retrieved and never reached the prompt. State 2b is distraction: it reached the prompt and lost to other retrieved passages that discuss the topic and state something else. State 3 is misuse: it reached the prompt, nothing competed with it, and the model still got the answer wrong.

Three states, one cheap test each#

The order matters, and it is cheapest-first for a reason beyond cost. The span check requires no model calls, no judge — a second model asked to rule on whether an answer is correct — and no labelled data beyond knowing what the right answer is, and it eliminates whole branches of the tree in one pass, whereas the ablation in the last branch, which re-runs the case with everything except the gold passage removed so that the change in the answer says what the removed material was doing, costs two model calls per case and needs a way to score correctness. Starting at the expensive end, which is what "let us try a stronger model and see if it helps" amounts to, produces an experiment whose result is uninterpretable: if the answers improve you do not know whether the model reads better or merely guesses the same fact from parametric memory, and if they do not improve you have learned nothing about which of the three states you are in.

Citations are the reason the answer text cannot be read for this. A system that cites its sources will cite the passage it used, and in state 2b the passage it used is a real retrieved document that genuinely discusses the topic, so the citation renders as a legitimate internal link to a legitimate internal page. Support reads a fluent answer with a plausible link and concludes the model made something up, which sends the investigation towards hallucination mitigation — the body of work aimed at models stating things no source supports — when the actual event was a ranking accident.

The first test is a string search. Dump what the retriever returned, dump what the assembler sent, and look for the answer span in both.

def triage(query: str, gold_span: str, pipeline) -> str:
    candidates = pipeline.retrieve(query, k=10)        # pre-cut
    sent = pipeline.build_prompt_passages(query)       # post-rerank, post-budget

    in_candidates = any(gold_span in c.text for c in candidates)
    in_prompt = any(gold_span in p.text for p in sent)

    print(f"query: {query!r}   span: {gold_span!r}")
    for rank, c in enumerate(candidates, 1):
        marker = "*" if gold_span in c.text else " "
        used = "SENT" if any(p.doc_id == c.doc_id for p in sent) else "    "
        print(f"{marker} {rank:>2} {c.score:.3f} {used} {c.doc_id}")

    if not in_candidates:
        return "state_1_retrieval_miss"
    if not in_prompt:
        return "state_2a_lost_in_the_cut"
    return "state_2b_or_3_needs_ablation"

The output shape settles the first branch immediately.

query: 'how long do I have to return a hardware unit?'   span: '21 calendar days'
   1 0.412 SENT policies/returns-overview.md#intro
   2 0.388 SENT policies/refunds-current.md#window
   3 0.377 SENT hardware/rma-process.md#shipping
   4 0.361      hardware/warranty.md#coverage
*  5 0.359      hardware/rma-process.md#timing
   6 0.344      policies/returns-2022.md#window

Rank 5, not sent, because the three passages above it averaged 1,900 tokens each and consumed the budget. The correct span was retrieved, ranked reasonably, and discarded by a truncation rule that nothing in the evaluation was measuring.

The budget arithmetic is worth doing, because it explains the cut-off exactly and it explains why raising k would not have moved it. Three passages averaging 1,900 tokens is five thousand seven hundred tokens against a six-thousand-token budget, which leaves about three hundred. Nothing in this corpus's chunk sizes fits in three hundred tokens, so the fourth candidate is refused whatever it contains, and so is the fifth. The SENT column in the dump reproduces that: the marker stops after rank three, and the star sits two rows below the last thing that was sent. Printing the two lists side by side is the whole diagnostic — one column is what the ranker chose, the other is what the assembler could afford, and the gap between them is a fact about token accounting rather than about relevance.

The score column says something else worth noticing. The top result scores 0.412 and the gold passage at rank 5 scores 0.359, a spread of about five hundredths across the whole visible list. A ranking that flat is not a ranking that got the query badly wrong; it is a ranking with almost no opinion, in which a cross-encoder pass or any small change in chunking would reorder the top five freely. Treating this as a retrieval quality problem would mean chasing five hundredths of similarity, while the actual gap between success and failure is a truncation rule that no similarity score participates in.

That distinction is the single most common surprise in this exercise, and it is why the second branch of the tree exists at all. The shallow reading of "the answer was in the top ten" is that retrieval is fine and the problem is generation, so the fix is a stronger model or a tighter system prompt. "In the top ten" and "in the prompt" are different claims, and on most pipelines they are different facts: an eval harness retrieves ten passages because k was set to ten, while a production request assembles a prompt from the top five after the reranker and then drops any passage that would push it past a 6,000-token budget, which on long chunks means three or four make it. A dashboard reporting recall@10 for a system that sends four passages is describing a system nobody is running, and every downstream conclusion drawn from it inherits the error.

The failure of that dashboard as a defence has a precise shape, and recognising the shape is more useful than recognising this instance of it. Ten is roughly three times four, so between a third and a half of every ranked list the metric inspects is material the assembler will never send. Work that improves the ranking inside that discarded region — promoting a passage from rank nine to rank six, say — raises recall@10 and changes nothing a user experiences. Work that damages the region the assembler does read can be offset in the metric by gains further down the list. So the dashboard can rise while the answers get worse, and it can rise while the answers stay identical, and there is no reading of the dashboard alone that distinguishes those cases from a genuine improvement. Measuring at the k the assembler actually sends — the recall@k_prompt in the decision tree above — removes the ambiguity by making the metric's list and the prompt's list the same list.

The measurement that separates distraction from misuse#

Cases with the correct span inside the prompt and a wrong answer anyway leave two candidate explanations that the triage script cannot tell apart. The passage may have been present but outcompeted by distractors — other retrieved passages that discuss the topic and state a different number, which the model resolves in favour of the wrong one. Or the model may have had the evidence in a clean position and used it incorrectly, misreading a conditional clause or ignoring an instruction.

The gold-only ablation settles it. Build the same prompt with the gold passage as the sole context, no distractors and nothing else, and re-ask.

def gold_only_ablation(cases, pipeline, judge) -> dict:
    full_correct = gold_correct = 0
    for c in cases:
        full = pipeline.answer(c.query)                       # production context
        gold = pipeline.answer_with(c.query, [c.gold_passage])  # gold only
        full_correct += judge(c, full)
        gold_correct += judge(c, gold)
    return {
        "full_context_accuracy": full_correct / len(cases),
        "gold_only_accuracy": gold_correct / len(cases),
        "distraction_delta": (gold_correct - full_correct) / len(cases),
    }

On the eight such cases in one investigation, gold-only produced the right answer six times. Those six are distraction: the model can read the passage correctly when nothing competes with it, and fails when the same passage arrives alongside three plausible neighbours. The two that stayed wrong with clean evidence are genuine misuse, and reading them showed why — both involved a passage stating "14 days for standard plans, 30 days for Enterprise with an active support contract", and the model returned 30 for a customer on a standard plan.

What makes the ablation decisive rather than suggestive is that it changes exactly one thing. The query is the same, the model is the same, the gold passage is the same text in the same wording, and the only variable removed is the company the passage was keeping. So a case that flips from wrong to right has had its explanation isolated: the model could always read that passage, and something else in the prompt was outweighing it. A case that stays wrong has had the opposite isolated, and the two Enterprise cases show what that looks like in practice. The passage states two numbers under two conditions, the prompt does not say which condition the customer is in, and no amount of removing distractors supplies a fact the passage never carried. That is a content problem wearing a retrieval problem's clothes, and the ablation is what undresses it.

Position within the prompt matters enough to measure separately, and it is measurable with the same harness by re-running each case with the gold passage inserted at different ranks among fixed distractors.

The comparison between the 3 of 5 row and the 3 of 10 row is the one that changes decisions: the same relative position with twice the context costs seventeen points. Retrieving more is not free, and a pipeline tuned to maximise recall@20 can produce worse answers than one at recall@5, because the extra fifteen passages are distractors by construction.

Reading the two rows against each other is what isolates the cause, and it is worth being explicit about what is held fixed. The gold passage sits third from the top in both, so its position is not what changed. What changed is the number of passages around it, from four others to nine. Thirty-four correct out of forty falls to twenty-seven out of forty, which is seven cases lost to five extra passages that were never relevant to begin with. The phrase distractors by construction is doing real work there: on a query whose answer sits in one passage, every additional passage retrieved is by definition not the answer, so raising k adds one relevant document at most and adds distractors without limit. That is why a recall metric and an answer-quality metric can point in opposite directions on the same change, and why the k in the retriever is a quantity to tune against answers rather than against recall.

The reranker's placement deserves its own note because state 2a is so common and so mechanical. A pipeline that retrieves 10, truncates to a token budget, then reranks whatever survived, is reranking a list that has already lost its best member — the ordering work happens after the information has been thrown away. The ordering argument is about information rather than about scores. Truncation is a lossy step: it removes passages from the list and there is no later stage that can bring one back. Reranking is a reordering step: it changes positions and removes nothing. Run the lossy step first and the reordering step inherits whatever survived, so a gold passage cut by the budget is not in the list the reranker is about to fix. Run the reordering step first and the gold passage is at or near the top when the budget applies, where the budget cannot reach it. Nothing about the reranker's quality enters that argument, which is why the fix works on a small reranker and why it is a pipeline change rather than a model change. Reranking the full candidate list first and truncating afterwards costs one cross-encoder pass over ten passages, about 40 ms on a small reranker — roughly four milliseconds a passage, which is the number to multiply if the candidate list ever grows. Capping each passage at 500 tokens with a mid-passage trim recovers more, at the cost of occasionally cutting a passage where the answer sat in the final paragraph, which is a trade worth measuring rather than assuming.

Making the triage cheap enough to run on every complaint, rather than as a fortnightly investigation, is a logging decision taken once. Each request records the query, the ordered candidate document identifiers with scores, the identifiers that survived reranking and truncation, the assembled prompt's token count, and the model and index versions in force. That is a few hundred bytes per request, and it turns triage from "reproduce the query against today's index and hope the corpus has not changed" into a lookup. Reproducing after the fact is unreliable in exactly the cases you care about, since a complaint from eleven days ago ran against a corpus and an index that have both moved, and the query that failed then will often succeed now for reasons unrelated to anything you changed.

Where the span check misleads#

The string search is triage rather than ground truth, and it has false negatives. A passage stating "three weeks" answers a question whose gold span is "21 calendar days", and a literal search will file it as a retrieval miss when the evidence was present in a different form. Questions requiring arithmetic or unit conversion fail the same way. Normalising both sides, or falling back to a model-based check on the cases the string search rejects, recovers most of it, at the cost of introducing a judge whose own errors then need checking.

It also has false positives. A passage may contain the string 14 days inside an unrelated sentence about trial periods, and the check will happily declare the evidence present. In one review of a batch this size, two queries turned out on reading to hold the span in a passage that did not actually answer the question, and both moved from state 2 back into state 1. The direction of that move is what makes it worth catching rather than tolerating, since state 1 and state 2 have disjoint fixes: a case misfiled as state 2 sends an engineer into the prompt assembler for a problem the ranker owns, and the assembler will look fine, because it is fine.

Read as a whole, the triage is a way of keeping the fix proportionate to the failure. Prompt and model changes are the most visible lever and the most frequently pulled, and they are the correct lever for the smallest of the three states in most systems. The three checks cost a string search, a comparison of two lists and a pair of model calls, and what they buy is knowing which of four rows in the table above you are in before spending a sprint on the wrong one.

None of these measurements say anything about the questions where no ranking, no reranking and no prompt assembly could have helped, because the corpus does not contain the answer or because answering requires reading more documents than any k admits. Those survive every fix in this chapter, and they need to be handled as a separate category rather than as a residue of noise.