Anti-hallucination prompt instructions

The instruction is the third intervention a team reaches for, after decoding settings and retrieval, and it is the cheapest: four lines in a system prompt, no infrastructure, no cost. It is also the one whose effect is hardest to read from outputs, because what it reliably changes is how the answers sound. Measuring it therefore needs a probe that can distinguish a change in disposition from a change in accuracy, and a grading scheme that separates the questions on which those two come apart.

This chapter runs that measurement over a 400-question evaluation set split three ways by what the supplied passages can support, reports what moved across five different phrasings of the instruction, prices the trade in agent-hours, and then explains why the sentence cannot do the thing it asks for — which is a fact about what the generation process computes rather than about the model's willingness to comply.

A probe that can actually fail#

The evaluation set is split three ways, and the split is what makes the result legible. Roughly half the questions are answerable: the supplied passages contain the answer outright. A quarter are ambiguous: the passages bear on the question but do not settle it, typically because the policy covers the general case and the customer's situation is a variant. The last quarter are absent: nothing in the supplied passages addresses the question at all, and the only correct behaviour is to say so.

That third slice is what makes the instruction testable, because it is where an honest system and a fluent one produce visibly different outputs; on answerable questions a model that ignores the instruction entirely still looks fine. Grading was done per slice, and outcomes were coded four ways: correct, meaning the answer is right and supported; wrong, meaning it contains at least one contradicted or unsupported claim; abstained, meaning it declines and asserts nothing; and, cutting across the first two, hedged, meaning the answer carries an explicit uncertainty marker.

The reason the design needs a slice where nothing supports an answer is worth being explicit about, because it is the difference between a probe that can fail and one that cannot. On answerable questions the two possible dispositions — try to answer, or decline when unsure — produce nearly the same behaviour, since the model is not unsure. Any measurement taken there is dominated by whether the model happens to know the answer, which the instruction does not touch. Only on questions the evidence cannot settle do the two dispositions diverge visibly: one produces a claim, the other produces a refusal, and the difference is readable from the output. A probe that omits that slice will show a change in tone and no change in any number, and it will be reported as "the instruction did nothing", which is not what the data would have said.

CONTROL = (
    "You are a support assistant. Answer using the numbered context passages "
    "below. Cite the passage number for each factual claim."
)

TREATMENT = CONTROL + (
    "\nDo not hallucinate. Only state facts you are certain of."
    "\nIf the context does not contain the answer, say so."
    "\nNever invent policy numbers, clause references, or timelines."
)

HEDGES = ("i cannot be certain", "i'm not sure", "may vary", "please verify",
          "i believe", "typically", "generally", "as far as i can tell")

def grade(record: dict) -> dict:
    claims = annotate(record["answer"], record["gold"])   # chapter 3 procedure
    return {
        "slice": record["slice"],
        "abstained": is_abstention(record["answer"]),
        "wrong": any(c["label"] != "supported" for c in claims),
        "hedged": any(h in record["answer"].lower() for h in HEDGES),
        "n_claims": len(claims),
    }

What changed#

The register moved a very long way. Hedge markers per response went up almost seven-fold, answers got 31% longer, and the phrase "I cannot be certain" appeared in 44% of responses against 6% in the control. Anyone reading a handful of outputs would conclude the system had become more careful.

The accuracy did not move. On the answerable slice, conditional accuracy — correct divided by correct-plus-wrong, which asks how often the model is right when it commits — went from 89.5% to 87.8%, slightly the wrong way. On the ambiguous slice it went from 45.1% to 45.7%, inside the noise of 90 questions. On the absent slice, of the answers the model still produced, 97.1% contained an unsupported claim against 98.7% before. In every case where the model chose to assert something, it was as likely to be wrong as it had been.

Conditional accuracy is the column that makes the finding legible, and it is worth saying what it is and why it is not the same as the correct row. The correct row mixes two things: how often the model gets an answer right, and how often it attempts one at all. Push abstention up and the correct row falls even if nothing about the model's judgement has changed, because some of what used to be correct is now a decline. Dividing correct by correct-plus-wrong removes the declines from both parts of the fraction, which leaves a quantity that specifies one thing only — accuracy among the cases where a commitment was made. It is the number to watch precisely because it is the one an instruction that only moves a threshold cannot move.

What the instruction did buy is on the absent slice: correct declines went from 12.2% to 61.1%, a genuine improvement worth having. It came with a bill. On the answerable slice, abstentions went from 1.4% to 9.5% — one question in eleven that the model could have answered correctly from evidence in front of it now got "the provided policy does not specify this". The instruction did not teach the model to tell answerable from unanswerable; it shifted a threshold, trading correct answers for declines at roughly one lost answer per five gained declines.

That exchange rate is the whole quantitative result, and it comes from putting the two movements side by side rather than reading either alone. Correct declines on the absent slice rose from 12% to 61%, a gain of forty-nine points. Abstentions on the answerable slice rose from 1.4% to 9.5%, a cost of eight points. Dividing the second by the first gives the ratio quoted, and it is a ratio of rates on two slices of different sizes, so converting it into tickets — which the section on the over-refusal bill does — needs the slice weights as well. The reason the trade is not obviously good or obviously bad is that the two sides are denominated in different things: unanswered questions on one side, and wrong answers avoided on the other.

Five phrasings, one shape of result#

The obvious objection is that the wording was clumsy, and it deserves testing rather than dismissing. Four alternatives went through the same harness: a hard protocol specifying the exact refusal string, a demand to quote the supporting sentence, two worked examples of correct abstention prepended to the prompt, and a threat.

Every variant does the same thing to different degrees: it moves the abstention threshold. The hard protocol moves it furthest and pays with the highest over-refusal apart from the threat, which is the worst trade in the table — it declines one answerable question in five while remaining slightly less accurate than the control on the ones it attempts. The conditional-accuracy column barely moves anywhere, which is the finding restated five times.

The quoting variant is the one anomaly, and the anomaly is instructive. Requiring the model to quote its supporting sentence raised conditional accuracy by 1.5 points, the only change in that column outside noise, while producing the lowest over-refusal of any variant. It is doing something structurally different: rather than asking for a disposition, it changes the shape of the output so that each claim has to be accompanied by an artefact, and producing that artefact constrains what the claim can say. It is still only a request — the model can and does quote a sentence that does not contain the fact — but the artefact exists afterwards, and something other than the model can look at it.

The confident caveat#

The most instructive outputs are the ones that hedge and are wrong at the same time, a category the instruction created rather than removed. From the absent slice, the same question before and after:

Q: How long does finance take to process an approved refund?

Control:
  Refunds are processed within 3 to 5 business days. [2]

With instruction:
  I cannot be fully certain from the provided context, but refunds are
  typically processed within 3 to 5 business days. Please verify with the
  finance team if the exact timing matters.

The false claim survives intact, the citation is dropped, and the sentence acquires two hedges and a referral. Under the per-claim procedure this is still one extrinsic hallucination, because "typically processed within 3 to 5 business days" asserts a processing time and the context asserts nothing about processing times. Under a reviewer's eye it reads as appropriate caution. Twenty-two of the 90 absent-slice questions produced answers of this shape, so the instruction converted roughly a quarter of that slice from obvious errors into errors that are harder to spot.

That effect was measured on people directly. Twelve support agents were shown forty responses each, twenty from each arm, shuffled and unlabelled, and asked to rate how much they would trust each draft on a five-point scale. Treatment responses scored a mean of 3.9 against 3.2 for control, a clear and consistent preference, on a sample where the two arms had statistically indistinguishable accuracy. Hedging reads as calibration to a human reader, because in human writing it usually is: a person who says "I believe, though I'd check" is signalling something real about their own state. The signal is genuine in the human case and imitated in the model case, and there is no way to tell them apart from the text.

The belief that fails here is worth writing out as a reviewer would hold it: this system sounds more careful, so it is being more careful. It is tempting because in every other setting where the inference is drawn — reading a colleague's memo, a doctor's note, an incident report — the hedge really was produced by somebody weighing their own confidence, so the correlation the reviewer is relying on genuinely exists among humans. What breaks it here is that the model produces the hedge by the same mechanism it produces the claim. The disclaimers cluster in the training data around uncertain writing, so asking for caution moves the text toward that cluster, and it does so whether or not the underlying claim is one of the weakly supported ones. The correlation the reviewer imported from human writing is exactly the thing the imitation destroys.

The over-refusal bill, and why the sentence cannot do more#

The abstention trade has a denominated cost on the operations side, and until it is quantified the argument stays qualitative and therefore stays about taste. The assistant drafts a reply that an agent approves, edits, or writes from scratch. An approved draft takes the agent 40 seconds; a rejected one costs the 40 seconds plus the four and a half minutes of writing it from scratch, and an abstention is always a from-scratch reply. At 2,100 refund-adjacent tickets a week, the instruction's extra 8.1 points of abstention on answerable questions is about 170 additional from-scratch replies, or roughly 13 agent-hours a week. Against that, the 49 additional points of correct declining removes some number of wrong answers that would have generated a follow-up ticket, a refund the company did not owe, or an escalation. Whether the trade is worth taking is a comparison once both sides are in hours and pounds, and it is not a question the model or the prompt can answer.

The reason the sentence cannot do better is not that the model disregards it. It follows it, and the following is the problem. "Only state facts you are certain of" is an instruction to condition generation on a quantity — the model's certainty about a proposition — that the generation process does not compute. What the model has at each step is a distribution over next tokens, a different thing measured on a different object, and one that was concentrated and confident at exactly the step where chapter 1's trace committed to a fabricated section number. There is no internal certainty score for "the refund window is 30 days" to be thresholded against. What the instruction can condition on is style, because style is fully determined by the tokens and the tokens are what the model models. "Do not hallucinate" appears in training data adjacent to careful, qualified, disclaimer-heavy prose, so adding it shifts the distribution towards careful, qualified, disclaimer-heavy prose.

That gap between what the instruction specifies and what the process computes is the whole of the result, and it can be stated without reference to any measurement. An instruction can only condition on something the generation process represents. Style is represented, because style is a property of token sequences and token sequences are what the model models. A proposition's truth is not represented anywhere, because nothing in the process ever assembles a proposition and holds it separately from the words expressing it. So the sentence asks for a filter over propositions, which are not there to be filtered, and gets a shift in a quantity that is. Every row of the five-phrasing table is that same substitution, performed with different wording and to a different degree.

It is worth adding that the disposition the instruction is fighting was put there deliberately. A model tuned against benchmarks that award nothing for abstention has been selected to answer, since a guess beats a certain zero on any leaderboard scored that way, and that pressure is applied over a whole training run while the instruction is applied in a sentence at inference time. A threshold shifted by four lines of prompt is working against a threshold shaped by the entire tuning process, which is why the effect is real, moderate, and especially unreliable at the extremes of the trade.

There is a second-order effect worth naming, which is that the instruction is read by everyone except the model. A system prompt containing "Never invent policy numbers" reassures the engineer who wrote it, the reviewer who approves the PR and the stakeholder who asks what you have done about hallucination. It functions as documentation of an intention, and it is very easy to mistake for a control, in the specific sense that a control prevents an outcome rather than requesting it.

None of this makes prompt text useless, and overcorrecting into a blanket dismissal of prompting is its own error. Instructions are the right tool for setting an operating point on a trade you have already measured: if you know that declining costs 0.2 correct answers per decline gained, and that a wrong refund figure costs more than an unanswered ticket, then dialling abstention up with a sentence is a cheap way to move along that curve. They also do real work on the parts of the output that are stylistic — citation format, response structure, prose versus table, what to do when the customer is angry. Fine-tuning and preference optimisation sit in a different position: training on examples where abstention is rewarded and confident error is penalised does change behaviour more durably than a sentence, at the cost of its own labelled data, and it moves the same trade further rather than replacing it with a guarantee.

Three levers have now been ruled out as fixes. Decoding does not reach the problem, retrieval stops reaching it past a low threshold, and asking the model to behave reaches only the wording. What is left is the observation sitting in the middle of every measurement in this topic: the labelling procedure works because each claim is checked against a specific span of supplied text. If the answer carried those spans itself, that check would not need a human.