Telling the model to say it does not know supplies no observation

A refusal instruction is a sentence added to a system prompt telling the model to decline when the retrieved passages do not contain an answer — some phrasing of "if the context does not contain the answer, say you don't know". It costs almost nothing, deploys in minutes, and reliably moves the number teams measure it with. Every part of that is true and it is not the same as the instruction working.

The reason to be careful is that the instruction and the gate from the previous two chapters are usually presented as alternatives at different price points, as though one were a cheap approximation of the other. They are different kinds of thing. A gate compares two objects — a claim and the retrieved passages — and returns a score somebody chose a cutoff on. An instruction changes how a single forward pass behaves, and whatever the model then reports about whether the passages support its answer is a continuation of a prompt rather than a comparison.

A one-sided evaluation#

Start with the evaluation rather than the instruction, because the shape of the evaluation is what makes the result unreadable.

Suppose a set is assembled entirely of questions whose answers are absent from a corpus, with each one checked to confirm the absence. That care is what makes the set feel rigorous, and it is exactly the property that makes it uninterpretable. On such a set the hallucination rate — the share of answers asserting something the retrieved passages do not support, which for this set means any answer at all — is the only quantity that can move. And it is minimised by a system that refuses everything.

That is worth stating as a construction rather than as a worry, because the construction is decisive. Replace the assistant with a three-line function that returns the constant string "I don't know". It scores zero on that set: no answer, therefore no unsupported assertion, therefore no hallucination. Zero is better than any real system will achieve. So the metric ranks a function that has been deleted above every candidate under consideration, which means it is not ranking the thing anybody cares about. The evaluation cannot distinguish an improvement in judgement from an increase in evasiveness, because it contains no question whose correct handling is to answer.

The reflex when a number looks shaky is to collect more of it, and here that reflex fails in an instructive way. At two hundred items and a rate of a few percent, a Wilson interval — an interval for a proportion that behaves at small counts, where the textbook normal approximation does not — spans several percentage points, which is wide. Running the same set ten times larger narrows it to roughly a point and a half. That is a genuine improvement in precision and it changes nothing, because the defect is in what is being measured rather than in how precisely. Ten times the data measures the same one-sided quantity ten times more confidently, and would confirm with great confidence that a system refusing every question hallucinates on none of them.

Composing the set out of strata#

Composition is the fix, and it means building the evaluation out of strata that are handled correctly in different ways.

The four shares add to a hundred, so the strata partition the set: every question has exactly one correct handling, and none is left to be handled either way. That is what makes an overall figure computable at all — a question that could correctly be answered or refused would have to be scored twice or dropped, and either choice would make the total mean something different.

The middle two strata are usually merged, and separating them is worth the effort because they diagnose different components. A hallucination on a question whose answer is sitting in the corpus unretrieved is a retrieval defect showing up in the generator, and no amount of prompt work fixes it; a hallucination on a question the corpus never covered is a scope problem that a refusal genuinely solves. Answerability has to be judged against the passages the retriever actually returned for that question, logged at the time, since judging it against the whole corpus measures the retriever and the abstention behaviour together and attributes the sum to whichever one was changed last.

Building the set is cheaper than it sounds if the strata are sourced differently. The answerable stratum comes from production traffic that the system already answered correctly, confirmed by a grader against the logged passages, which is a labelling pass over data you have. The unretrieved stratum can be manufactured from the same items by dropping the gold passage from the logged retrieval before the shadow run, which gives a controlled version of the failure that matters most and costs no new question writing. Only the genuinely-out-of-corpus stratum needs authoring, and the useful source is the support queue rather than anyone's imagination — questions users actually asked that the documentation has never covered. A set assembled that way took about three days for 600 items, most of it grading.

What the mixed set shows#

Running four prompt variants over 600 questions built to that composition, with two graders per item, produces a picture the one-sided set could not.

Read the first three rows as gaps and they trace a curve. Going from no instruction to the soft phrasing buys seventeen points of hallucination for seven points of over-refusal. Going from the soft phrasing to the strict one buys thirty-two more points for thirteen more. Each step pays in the same currency and the useful-answer column falls monotonically across the three, which is the signature of an operating point being moved rather than a system being improved. The soft phrasing sits at one place on that curve and the strict phrasing at another, and neither dominates the baseline on the fraction of questions handled correctly.

The fourth row is not on the curve, and one row off the curve is the whole finding. It improves hallucination against the strict variant, improves refusal on answerable questions against it, and improves the useful-answer rate. Nothing was traded. The reason is in the instruction: the quoting requirement asks the model to name the supporting sentence, and that produces an artefact that can be checked mechanically — the quote either appears in the retrieved passages or it does not — and a string comparison is an observation about the passages rather than a judgement about them. That is the only variant in the table that adds an observation, and it is the only one that moves both columns the same way.

The observation it adds is weak and it is real, which is the combination that matters. In the run above it caught 6% of responses whose quotes were not present verbatim in any passage, all of which were fabrications — a small share, but one that no amount of rewording the other three variants would have surfaced, because the other three have nothing to compare against. That check is a cheap relative of the entailment gate, differing from it in that a string comparison asks whether the words are there rather than whether they establish anything, and it composes with the attribution machinery from topic 9.12 rather than duplicating it.

The reason the other three behave like a threshold rather than an improvement is that they supply no observation. Asking the model whether the context contains the answer asks for a judgement produced by the same forward pass, conditioned on the same context, by the same process that would otherwise have produced the answer. It is the verbalized confidence of chapter 1 with a binary output and no number attached, and it inherits the same blindness: the model's assessment of whether the passages support a claim is not a comparison between two objects, it is a continuation of a prompt. What the sentence changes is the prior — how readily the model reaches for a refusal — which is exactly the knob a threshold turns, with the difference that a threshold is a float you can sweep and this one is prose.

The two outcomes at the ends of that curve are the ones teams actually report, and which one you get is mostly a function of how forceful the wording is. A mild instruction leaves behaviour close to the baseline, so the hallucination rate barely moves and everyone concludes the sentence did nothing; a forceful one produces blanket hedging, where the assistant qualifies half its answers and refuses questions it handled correctly the week before, and the complaint that reaches the team is that the product has become useless rather than that its abstention rate is miscalibrated. Both results come from the same mechanism and both are usually diagnosed as a prompt-writing failure, which leads to a few weeks of rewording in search of a phrasing that separates the cases.

The prose knob has poor resolution and moves unpredictably, and the size of the movement is what makes that concrete. Replacing "if you are not sure" with "if you cannot verify" in the soft variant, changing nothing else, took refusal-on-answerable from 9% to 19% and hallucination from 44% to 31%. Those are ten and thirteen points, which is larger than the step from the first row of the table to the second — a synonym moved the operating point further than the deliberate change did. And there is no way to interpolate between two wordings: a numeric threshold always has values lying between any two you might compare, while a pair of phrasings has nothing between them at all. This is the resolution argument of chapter 1 arriving from the other direction. There it was four round values on a numeric score; here it is a handful of wordings with no ordering and no midpoint.

That is also why the third column is the one to rank by. The first two columns are measured on disjoint strata and neither of them requires an answer to be correct — a variant could answer every answerable question wrongly and still post a refusal-on-answerable rate of zero. The useful-answer rate is the share of all 600 questions handled correctly on their own terms: answered right where the passages supported an answer, refused where they did not. It is not a function of the other two, which is precisely why it can separate a variant that got better from one that only moved.

Detecting a refusal is its own problem#

Both of the first two columns depend on classifying each response as an answer or a refusal, and the classification is less obvious than it sounds because models produce a middle category in quantity.

"I couldn't find the exact retry budget in the provided documents, but typically
 webhook consumers use three retries with exponential backoff."

That response refuses and then answers, and the answer is fabricated. Counted as a refusal it lowers the hallucination rate; counted as an answer it raises it. In the 600-question run, 11% of responses to unanswerable questions had this shape, and eleven points is larger than several of the gaps between adjacent rows in the variant table — which is why the choice of rule can reverse the ordering of two of the four variants. A category that big, assigned wholesale to either side, moves a column by more than the differences the table is being read for. The workable definition is behavioural: a response counts as an answer if it asserts anything the user could act on, regardless of how it is hedged.

CLASSIFY = """You will see a user question and an assistant response.
Answer with exactly one token: ANSWER or REFUSAL.

REFUSAL means the response asserts no fact the user could act on.
A response that declines and then offers a general or typical value is ANSWER.
A response that names only where to look, without stating the value, is REFUSAL."""

def classify(question, response):
    out = judge(CLASSIFY, f"Q: {question}\n\nA: {response}")
    return out.strip()

# hand-audit 100 of these before trusting the rest:
# judge vs two humans agreed on 94/100; the six disagreements were all
# responses that hedged a correct value ("possibly five, per the runbook")

Applying that definition across six hundred items means a second model call doing the labelling, which is the LLM-as-judge arrangement — a model prompted to classify another model's output against a written rule. Auditing the classifier against human labels on a hundred items is not optional, and the reason is directional rather than statistical: a systematic bias in the classifier moves hallucination and refusal in opposite directions on every variant at once, which is indistinguishable in the table from a real effect. Agreement on 94 of 100, with all six disagreements falling in the hedged-value category, is a claim about the classifier on one distribution of responses; it is evidence that the rule is applied consistently, not that the rule is the right one.

Where the sentence belongs#

None of this is an argument for removing the instruction. It costs about 20 tokens per request, it is deployable in minutes, and on the strict variant it took hallucination on unanswerable questions from 61% to 12%, which is a change no measured gate produced on its own. It belongs in the prompt, wired to the version-controlled prompt file, with its effect measured on the mixed set whenever it changes.

What it does not do is replace the gate from chapters 5 and 6, and the honest reading of a 22% refusal rate on answerable questions is that the strict variant is operating at a point nobody chose. That number is the false-abstention rate under a different name, and the response to it is the same as in the previous chapter: put it next to the hallucination rate, compute the exchange rate between the two, and hand both to whoever owns the decision. If the support lead's 3:1 stands, the strict variant is buying 49 points of hallucination reduction for 20 points of over-refusal, which is a bad trade at that price and a good one at the clinical lead's 40:1. The instruction is a choice on the same curve, so it deserves the same conversation rather than a merge on the strength of one number.

One consequence of that composition needs to be in the release checklist, and it is a causal chain worth having in full rather than as an observation. Changing the refusal instruction changes the wording of the answers. Hedged answers decompose into different claims than assertive ones, because a decomposer turns "possibly five, per the runbook" into a different proposition than "five". Hedged claims entail less readily against the same passages, since a qualifier is an extra condition the premise has to establish. So the entailment gate sees a different distribution of claims after a prompt edit, and its abstention rate moves without its threshold moving. That is why the combined rate is neither the sum nor the maximum of the two changes: when the strict variant shipped alongside a 0.7 gate on one support assistant, the combined abstention rate came out at 31% rather than the 12% the gate alone had produced, and neither change was individually surprising. Prompt edits are threshold changes, and the operating-point table from chapter 6 has to be re-measured after them.

Where this is heading is a different construction entirely. Training a model to abstain — with a proper scoring rule on its own stated confidence, a proper scoring rule being one whose expected penalty is lowest when the stated probability equals the true one, so that overclaiming is penalised during training rather than corrected afterwards — is what the Training Large Language Models to Express Their Confidence Verbally line of work pursues. It addresses the defect this chapter describes at the source, since a model trained against a scoring rule is receiving an observation about correctness during training, which is exactly what no runtime instruction can supply. It does not remove the need for the mixed evaluation, and a model that abstains well on its training distribution still needs the retrieval-gap stratum measured on yours.

The order to build in follows from all of this. Assemble a few hundred production questions with strata labelled, grade them, and compute separation for whatever signal you already have; add the entailment gate and score it on the same set with the strata kept separate; produce the operating-point table with the exchange-rate column; get a number out of whoever owns the cost; then wire the refusal instruction, re-measure, and put the whole table in the release checklist.

The first item is the one everything else depends on, and it is the only one that cannot be substituted. Every argument in these seven chapters — that a mean hides ordering, that recalibration cannot create separation, that a fluent fabrication outscores an awkward quotation, that five agreeing samples estimate a peak, that entailment survives the retrieval gap, that a complaint count only turns one way — is an argument about what a graded set would show. Without one, each of them is a story, and it is a week of unglamorous grading that turns them into measurements of your own system — work that no amount of signal engineering substitutes for.