Unanswerable queries and abstention

A retrieval-augmented assistant answers out of a corpus, and every metric in this topic so far has measured how well it finds the right part of one. Abstention is the opposite output: the system declining to answer and saying plainly that the corpus does not record this. It is the correct output for a class of question that no amount of retrieval work reaches, and it is not the default: a model handed four relevant-but-insufficient passages will usually synthesise something, because the passages are on topic, the question is answerable-sounding, and nothing in the context announces its own incompleteness.

Two shapes of question fall into that class, and they have nothing in common except that no ranking improvement addresses either. The first is a corpus gap: the fact was never written down anywhere the system can read. The second is an aggregation: the corpus contains the answer, distributed across more documents than any top-k result set will ever hold. They are worth separating because they call for different behaviour from the system and different treatment in the eval set.

A fact that is not there cannot be ranked into existence#

The corpus gap is the simpler of the two and the one people find hardest to accept, because the instinct after any wrong answer is to reach for the retriever. Ranking is a permutation of the corpus. If no document contains the fact, every permutation is equally without it, and pushing recall from 0.86 to 0.95 changes what happens on other queries while leaving this one exactly as it was.

That sentence about permutations is worth making exact, because the exact version is what closes the argument rather than merely making it sound closed. A retriever's output is always a subset of the corpus, chosen and ordered by a score. Choosing and ordering can change which documents come back and in what sequence; neither operation can produce a document that was not in the collection to begin with. So any fact absent from the corpus is absent from every possible output, of every possible ranker, at every value of k, permanently. That is not a statement about how good the current ranker is. It is a statement about what class of thing a ranker is, and it is why the correct response to this failure is a decision about behaviour rather than a ticket against the retriever.

The consequential decision is what the system should do instead. Abstention has to be asked for explicitly and then verified, since a system that abstains only when the retrieved set is empty will never abstain on this class — the retrieved set is full of good passages, all genuinely about the subject, none of them stating the fact.

The shape of that failure is the one to memorise, because it recurs. An empty-set check is a defence, and it produces the identical verdict — results were returned, so answer — whether the corpus holds the fact or not. The thing it is supposed to discriminate varies completely across the two cases; its output does not vary at all. A check like that cannot be tuned into usefulness, because there is no threshold on a quantity that does not move. What does move is whether any retrieved passage states the fact, and that is a different question from whether any passage was retrieved, which is why abstention has to be a requested behaviour with its own test rather than a fallback the pipeline reaches when something is missing.

What to do with such a question when you find it during eval construction is to keep it, with an empty relevant-document list. Deleting it is tempting because it looks like a broken row: no relevant documents means it cannot contribute to recall, and a labelling tool may even refuse to store it. Keeping it is what creates measurable pressure on the only behaviour that protects the user.

{
  "id": "unanswerable-bergman-outcome",
  "query": "what was the outcome of the Bergman escalation?",
  "relevant_docs": [],
  "expected_behaviour": "abstain",
  "insufficiency_reason": "corpus_gap",
  "abstention_must_mention": ["not recorded", "no record", "cannot find"],
  "answer_must_not_contain": ["resolved", "credited", "renewed"],
  "note": "Outcome agreed on a call, never written up. Verified 2026-03-02 by full-text search of the corpus for 'Bergman' (14 hits, none stating an outcome).",
  "recheck_when": "docs matching 'Bergman' change"
}

The note field carries the evidence that the gap is real, because an unanswerable case that has quietly become answerable is a test asserting the wrong thing. Somebody eventually writes the escalation up, and from that day the eval set is penalising the system for correctly answering a question it can now answer. Recording how the gap was verified, and under what condition to re-check, is what keeps that from taking a year to notice.

Finding these cases in the first place takes a deliberate pass, because they do not arrive labelled. Sampling questions from real traffic surfaces some, but the fastest source is the support queue's own unresolved tickets: questions a human could not answer from the documentation either are, by construction, candidates for corpus gaps. Verifying each one is a full-text search over the corpus for the entities involved plus five minutes of reading, and the outcome is binary and worth recording either way — a question you assumed was unanswerable and that turns out to have an answer buried in an oddly-titled page is a retrieval finding, not an eval case. Ten to fifteen verified unanswerable cases in a hundred-query set is a reasonable proportion for an internal corpus, and a set with none of them is a set that will never measure the behaviour that matters most when the system meets a question outside its coverage.

Because these cases carry no relevant documents, they are excluded from the recall denominator — the scored filter in the recall function from chapter 4, on the metric that counts queries rather than positions — and scored on a separate axis instead.

Read the table's arithmetic before its argument. The two query counts are 88 and 12, which is the hundred-query set from the paragraph above with the unanswerable stratum carved out of it, and neither group appears in the other's denominator. Both abstention rows close against twelve: System A abstained once and answered wrongly eleven times, System B abstained ten times and answered wrongly twice, and one plus eleven and ten plus two are both the whole stratum. That closure is what makes the second and third rows a single measurement viewed twice rather than two independent observations — on an unanswerable query there are only two outcomes, so knowing one row determines the other, and a table where they did not close would be reporting a third outcome nobody had named.

Converting the headline gap into queries is what settles the comparison. Two points of recall across 88 answerable queries is under two queries. System B pays that, and pays three more in false abstentions, so it serves about five fewer answerable questions than System A. In exchange it removes nine confident inventions, since eleven wrong answers become two. Five against nine is the whole trade, and it is a trade nobody can settle from the recall row alone — which is the point. The comparison is not close on any product where a confident invention is more expensive than a refusal, and it is genuinely arguable on a product where a refusal is a dead end for the user.

System A wins the headline metric and is worse to deploy, because eleven of its twelve responses to questions the company cannot answer are confident inventions, delivered to people who will act on them. The false-abstention row is what keeps this honest: a system that answers "I cannot find that" to everything scores 12 out of 12 on abstention and is useless, so the pair of numbers has to be reported together, in the same way precision and recall are only meaningful jointly. System B's three false abstentions on answerable queries are the price of its ten true ones, and whether that trade is right depends on what the answers are used for — three unnecessary escalations to a human are cheap in support and expensive in a self-service product with no human behind it.

The question that needs more than the top k#

The aggregation case fails for an unrelated reason. Suppose the answer is distributed across 34 records in a CRM, the customer-relationship-management system where accounts live, plus a pricing-change announcement with a date, plus 34 exit-survey responses. Retrieval returns the top five passages by relevance, and the true answer is a function over a set of about 70 documents. No value of k fixes this in the way it looks like it should: raising k to 70 puts the documents in the context window and hands the model a counting-and-grouping task over 70 passages, which it will perform unreliably, and the position measurements in chapter 6, on triaging a wrong answer, point the same way — with the caveat that they were taken at five and ten passages, so what they establish for seventy is the direction and not the magnitude.

The seventy is worth adding up, because it shows where the number comes from and therefore how it would change. Thirty-four CRM records, one pricing-change announcement and thirty-four exit-survey responses is sixty-nine documents, which is the "about 70" above. Two of those three terms scale with the number of churned accounts and one does not. So the size of this question's evidence set is roughly twice the answer it is asking for, and a system that grew to a few hundred churned accounts would need a few hundred documents in context to answer the identical question. Nothing about that is a ranking parameter to tune. It is the question's own shape.

The mismatch is between the operator retrieval implements and the operator the question needs.

Similarity search is a ranking operator with a limit clause. Asking it for an aggregate is asking ORDER BY similarity LIMIT 5 to behave like GROUP BY region HAVING ..., and the reason this is dangerous rather than merely wrong is the silence: the top-5 result of an aggregation query is indistinguishable in form from the top-5 result of a lookup query, so the answer arrives with the same confident shape whether it covers 5 of 5 or 5 of 34.

There is a second reason "just raise k" fails, and it is the more interesting one because it is circular rather than merely expensive. A LIMIT is a prefix operator: it returns the first n items of an ordering. An aggregate is a function of a whole set. A prefix determines an aggregate only when nothing outside the prefix could have contributed to it, which for a count over a filtered set means the prefix must already contain every matching record. So the smallest k that answers how many EU customers churned is the number of EU customers who churned — the very quantity being asked for. Choosing k correctly requires knowing the answer first. That is why no amount of turning k up converges on correctness: the parameter you would need to set is the output, and the last row of the table above names the consequence, since the twenty-nine records outside the top five leave no trace of themselves in a result set that looks complete.

Three responses are available, and choosing between them is a product decision rather than a retrieval one. The system can route the query to a mechanism that can actually compute it — a generated SQL query against the CRM, a saved report, an aggregation endpoint — which is correct and requires structured data to exist. It can run an agentic loop, in which the model itself drives repeated rounds of retrieval rather than being handed one result set: it decomposes the question, retrieves per sub-question, and accumulates, which handles the cases where the records are genuinely unstructured text and costs many model calls with error compounding at each step. Or it can answer within a stated scope: five accounts, named, with an explicit statement that this is a similarity-ranked sample and not the full set, plus a pointer to where the full set lives.

The scoped answer is underrated and is the only one of the three available on a Tuesday afternoon. Its whole content is a sentence the model is required to emit when the query carries an aggregation shape, stating what the result covers and what it does not: five accounts ranked by relevance out of an unknown total, with a link to the report that can count them. That converts a wrong answer into an incomplete one, which is a different category of harm — the analyst who reads "five of an unknown number" goes to the CRM, and the analyst who reads a confident list of five does not. Detecting the shape well enough to trigger the sentence is a rules problem rather than a modelling one, since aggregation queries in practice announce themselves with quantifiers and plurals: which of our, how many, all, list every, across, compare.

The eval case records which of those you consider correct, so that the question stops being a mystery failure.

{
  "id": "aggregation-eu-churn-post-pricing",
  "query": "which of our EU customers churned after the pricing change, and what did they say?",
  "relevant_docs": [],
  "expected_behaviour": "route_or_scope",
  "insufficiency_reason": "requires_aggregation",
  "accepted": [
    {"kind": "route", "target": "crm.churn_report", "params": {"region": "EU", "since": "2026-07-01"}},
    {"kind": "scoped_answer", "must_state": "similarity-ranked sample, not exhaustive"}
  ],
  "answer_must_not_contain": ["all EU customers", "the EU customers who churned"],
  "note": "Ground truth 34 accounts as of 2026-08-01; top-5 retrieval covers 5."
}

Keeping insufficiency_reason as a field with a small fixed vocabulary — corpus_gap, requires_aggregation, requires_multi_hop, requires_computation — makes the unanswerable stratum readable as a map of what the product cannot do. Nineteen aggregation cases and two corpus gaps is an argument for building a query router. Nineteen corpus gaps in the same subject area is an argument for a documentation project, and it is the kind of finding that would otherwise be distributed across a hundred support tickets and never counted.

Where the boundary is genuinely unclear#

Multi-hop questions sit between the two shapes and do not resolve cleanly into either. "Which of our enterprise customers are on a contract that predates the current DPA, the data processing agreement?" needs two retrievals where the second depends on the output of the first, and iterative retrieve-then-read loops handle a useful fraction of these — published results on multi-hop benchmarks have improved substantially with decomposition and repeated retrieval, and the technique is worth reaching for. What has not been solved is knowing in advance which questions need it, since a single-pass system will answer the first hop confidently and stop, producing an answer that is a correct response to a question nobody asked. Classifying incoming queries by the mechanism they require is an open problem, and the honest current practice is to detect the common shapes with rules, route those, and accept that the tail arrives as complaints.

The classification is also not permanent. A corpus gap closes when someone writes the document. An aggregation question becomes a lookup when a nightly job materialises the churn report as a page in the corpus, which is often the cheapest fix available and is invisible if you have filed the question as a retrieval failure. Reviewing the unanswerable stratum quarterly against those two changes is what stops a growing pile of cases from calcifying into a list of things everyone has agreed the product cannot do.

The rule to carry out of this chapter is that a question retrieval cannot serve is a specification rather than a defect. Write it down with what the system should do instead, keep it out of the recall denominator so it cannot flatter or punish the ranker, and score it on abstentions earned against abstentions wasted. That pair of numbers is the only thing standing between a user and a fluent answer to a question nobody in the company can answer.

Everything from the last seven chapters now exists as a set of files rather than as a set of opinions: a rubric naming the dimensions your task depends on, a hundred labelled queries with recorded agreement, constraint cases pinned by document identifier, a stratified table, a triage script, and an unanswerable stratum with its own pair of numbers. That is the harness. Running it against a real index, before and after each change, and letting the rows rather than the averages decide what to work on, is the work the rest of this material builds on.