Verifying claims against cited spans

A regular expression that checks clause identifiers against the supplied context is the narrowest possible version of this idea, and it works: on one support deployment it ran for four months without a false positive, because string equality does not have a false-positive rate. It is also almost useless in general, since it applies only to claims whose truth is settled by whether a short string appears somewhere in the prompt. "The refund window is 30 days" contains no identifier to check. Neither does "finance processes refunds in 3 to 5 business days".

The generalisation is in the annotation procedure from chapter 3, where a human labeller decides a claim's status by finding the span of supplied context that settles it and reading the two side by side. Every part of that is mechanical apart from the finding and the reading. The finding is something the model is in a better position to do than any post-hoc aligner, because the model produced the claim and knows which passage it came from; the reading is a task a much smaller model can do. This chapter puts both into the response path: an output contract that carries evidence, three checks over that evidence in increasing order of cost, and a policy for what happens when one fires.

Make the answer carry its own evidence#

The change is to the output contract. Instead of returning prose, the model returns a structure: a list of claims, each with the text of the claim, the passage index it rests on, and the exact sentence from that passage, plus the assembled answer text. Structured outputs — where the request carries a JSON schema and the API constrains the response to conform to it — make the shape enforceable at the API level rather than hoped for in a prompt.

from anthropic import Anthropic

client = Anthropic()

GROUNDED_ANSWER = {
    "type": "object",
    "additionalProperties": False,
    "required": ["claims", "answer"],
    "properties": {
        "claims": {
            "type": "array",
            "items": {
                "type": "object",
                "additionalProperties": False,
                "required": ["text", "passage", "quote"],
                "properties": {
                    "text": {"type": "string"},
                    "passage": {"type": "integer"},
                    "quote": {"type": "string"},
                },
            },
        },
        "answer": {"type": "string"},
    },
}

message = client.messages.create(
    model="claude-opus-5",
    max_tokens=2048,
    output_config={"format": {"type": "json_schema", "schema": GROUNDED_ANSWER}},
    system=(
        "Answer only from the numbered passages. For every factual claim, give "
        "the passage number and the exact sentence, copied verbatim, that the "
        "claim rests on. Omit any claim you cannot point at."
    ),
    messages=[{"role": "user", "content": f"<context>\n{context}\n</context>\n\n{question}"}],
)

The instruction in that system prompt has exactly the standing chapter 6 gave it: a request the model may ignore. The difference is that ignoring it is now detectable. A quote either appears in the passage or it does not, and the located text either supports the claim or it does not. The prompt asks; the code checks.

Asking for the quoted text rather than character offsets moves the hard part into code. The model emits quote, a verbatim string it claims to be copying, and the application locates that string in the passage itself.

def locate(quote: str, passage: str) -> tuple[int, int] | None:
    i = passage.find(quote)
    if i >= 0:
        return (i, i + len(quote))
    # tolerate whitespace and smart-quote drift, reject genuine divergence
    norm = lambda s: re.sub(r"\s+", " ", s.replace("’", "'")).strip()
    m = difflib.SequenceMatcher(None, norm(passage), norm(quote))
    block = m.find_longest_match(0, len(norm(passage)), 0, len(norm(quote)))
    return None if block.size < 0.9 * len(norm(quote)) else remap(block, passage)

Exact matching succeeded on 88% of quotes; the normalised fuzzy fallback took it to 96%; the remaining 4% failed to locate, which is itself a useful signal, because a quote that cannot be found in the passage was not copied from it. Failure-to-locate became the cheapest and highest-precision detector in the stack, and it costs a string search.

Why the fuzzy fallback is bounded rather than generous is the one real design choice in that function. Two kinds of difference turn up between what the model emits and what the passage contains. The first is transcription drift — a collapsed run of whitespace, a curly apostrophe where the source had a straight one, a stripped line break — none of which changes what the sentence says. The second is the model producing a sentence that is nearly the source's and is not it, which is where a fabricated quotation lives. Requiring the matched block to cover almost the whole quoted string separates them, because drift leaves the characters intact and rearranges nothing while a rewritten quote fails to align across its full length. Set that tolerance loose enough to accept a paraphrase and the check stops being a check.

That bound is also what makes failure to locate the most trustworthy signal in the stack. A quote that cannot be found in the passage was not copied from it, and once drift has been allowed for there is no benign explanation left. The check contains no model, no threshold anybody can argue about, and no dependence on what the claim means. It is right every time it fires, which is a property nothing else in this chapter has.

The claims list also has to be checked for coverage, because the model decides the decomposition and under-decomposes when left alone. On a 200-response sample, human annotators found 11.6 claims per answer against the 6.2 the model volunteered, and the omissions were not random — the claims it left out skewed towards the incidental operational details that are the most common extrinsic errors. Comparing the concatenated claim texts against the answer text and rejecting responses where a substantial fraction of the answer's sentences map to no claim closed most of the gap, at the cost of a second generation on about 9% of requests. Anything the model does not list is not checked, so coverage is not a refinement; it is the difference between verifying the answer and verifying a selected part of it.

The skew in what gets omitted is not bad luck and follows from the same mechanism as everything else here. A claim the model can point at is one where the supplied text was doing the work at that position, and those are exactly the claims that were unlikely to be wrong. A claim that came out of the weights alone has no span to point at, so listing it means listing something the instruction told the model to omit. The omission is not a rhetorical manoeuvre; it is the instruction being followed. The decomposition therefore selects against precisely the claims the check exists to catch, which is why coverage has to be enforced from outside rather than requested.

Three checks, in increasing order of cost#

Each check is a filter over the claims list, and they are worth running in order because the cheap ones eliminate work for the expensive ones.

The third check is where the intrinsic errors live. What runs there is a cross-encoder trained on natural-language-inference data — NLI, the task of deciding whether one sentence follows from another, contradicts it, or neither — at roughly 400M parameters, small enough to sit on the same host as the application. It takes the span as premise and the claim as hypothesis and returns three probabilities. On a T4 GPU it costs about 35 ms per pair at batch size one, and claims within a response batch cleanly, so a six-claim answer costs around 60 ms wall-clock rather than 210 ms. It is a much smaller and much cheaper model than the one that generated the answer, which is the point: verification is an easier task than generation, and it does not need the same capability.

Why it is easier is worth spelling out rather than asserting, because the claim sounds like special pleading. Generation is open-ended: at every position the model chooses among a vocabulary of tens of thousands of entries, and the space of things it could produce over a paragraph is unbounded. Verification here is a three-way classification over two short pieces of text that are both handed to it, with no search and nothing to invent. The hard part of the original task — knowing what to say — has already been done and thrown away, and what is left is a comparison. That asymmetry is why the checker can be two orders of magnitude smaller and still be useful, and it is the reason a verification layer is affordable at all.

def verify(claims: list[Claim], passages: list[str]) -> list[Verdict]:
    verdicts = []
    for c in claims:
        if not (0 <= c.passage < len(passages)):
            verdicts.append(Verdict(c, "unresolvable")); continue
        offsets = locate(c.quote, passages[c.passage])
        if offsets is None:
            verdicts.append(Verdict(c, "unresolvable")); continue
        start, end = offsets
        span = passages[c.passage][start:end]
        scores = nli(premise=span, hypothesis=c.text)
        if scores.entailment >= 0.70:
            verdicts.append(Verdict(c, "supported"))
        elif scores.contradiction >= 0.50:
            verdicts.append(Verdict(c, "contradicted"))
        else:
            verdicts.append(Verdict(c, "unsupported"))
    return verdicts

The thresholds are the tuning surface, and they are a precision-recall trade like any other. At 0.70 entailment the checker agreed with human labels on 84% of claims on a 600-claim validation sample, catching 71% of the intrinsic errors the humans found, at the cost of flagging 8% of genuinely supported claims. Loosening to 0.55 caught 81% of intrinsic errors and flagged 14% of good ones. Which point you want depends entirely on what happens after a flag.

The direction of that trade is not a quirk of this checker and is worth being able to predict without measuring. The threshold is a line drawn across a continuous score, and lowering it moves the line so that more claims fall on the flagged side. Some of those newly flagged claims are genuine intrinsic errors that had been scoring just above the line, which is the gain. The rest are supported claims that also score modestly, usually because they are paraphrased rather than copied, and those become false positives. There is no setting at which only the first group crosses, because the score does not separate the two groups cleanly — if it did, the threshold would not be a decision. Every threshold in this chapter buys recall with precision at a rate the validation sample measures, and the only question a team gets to answer is which of the two errors is cheaper for them.

That is also why the flag policy has to be decided before the threshold. If a flag withholds the response and routes it to a person, a false positive costs one escalation and the threshold should sit tight. If a flag only annotates the response for a reviewer, a false positive costs a glance and the threshold can sit loose. Choosing the number first and the consequence afterwards is how teams end up with a checker tuned for a policy they did not adopt.

The guarded path#

What happens after a flag is a policy decision, and it belongs in the pipeline rather than in the prompt.

The third branch is the one that changes the system's character. Abstention stops being a behaviour you hope the model produces and becomes a state the application enters when a defined condition holds: a claim that survives one repair attempt without a span that entails it does not reach the user. The response the customer sees is either verified against text or is a handoff, and which one they got is recorded.

The difference between the two kinds of abstention is worth stating precisely, because it is what everything in this chapter has been building toward. An instruction asks the model to decline under a condition the model has to judge, using a process that does not compute the quantity the condition refers to, so whether it declines is a property of the output distribution and varies from request to request. A gate declines under a condition the application evaluates in code, over an artefact the model produced, using a comparison whose result is the same every time it is run on the same inputs. The first is a disposition and the second is an invariant. Only the second is verifiable, only the second can be tested, and only the second is the sort of thing that can go into a written commitment — which is the subject of the next chapter.

The repair attempt in the middle branch is worth having because it is cheap and often works. Quoting the offending span back — "your claim the refund window is 30 days cites text reading within 14 days of the original purchase date; revise or drop the claim" — resolved 63% of flagged responses on the first retry in one support deployment. It costs one additional generation on the 12% of requests that flag at all, which came to roughly 1.4 seconds and about 3,400 extra tokens on those requests, and moved overall p95 latency — the figure the slowest 5% of requests exceed — from 2.9 to 3.6 seconds.

The token cost is smaller than it looks. Structured output makes the response longer — the claims array roughly doubles output tokens against plain prose, from about 190 to 400 on a typical answer — while the input side, where the 3,100 tokens of retrieved context live, is unchanged and can be cached across requests that share a prefix. Since output tokens are the expensive ones, the honest figure on that deployment was about an 18% increase in per-request cost before the repair loop and 24% after it, which is small enough that it has never been the deciding argument.

Anthropic's API also offers a built-in path to spans that avoids asking the model for offsets at all: setting citations: {enabled: true} on a document content block splits the response into text blocks, each carrying a citations array with cited_text and a char_location giving start and end indices into the document. The spans come back verified by construction, since the API produces them from the document rather than from the model's guess. The trade is that citations and output_config.format are mutually exclusive — sending both returns a 400 — so you get the spans and give up the enforced output shape, which means writing your own parser over the returned blocks. On a system that mainly needs spans, that is a good deal; on one that needs a strict claims array for downstream tooling, it is not.

The limits of a faithfulness check#

It does not make the answer true. Every check here is a faithfulness check against the supplied passage, so a claim entailed by a stale policy document passes cleanly, and a system running on a corpus that has drifted will verify its way confidently to the wrong answer. Correcting that is a content-freshness problem living entirely outside this pipeline.

It does not catch claims that are technically entailed and materially misleading. A span reading "requests outside this window may be escalated to a manager at the manager's discretion" entails "you can escalate this to a manager", and the inference model will pass it, while the customer reads a right where the policy grants a discretion. Over-reading a span is a real and common failure that survives every layer above, and the only reliable detector for it is a human on a sample.

It does not eliminate the checker's own errors. An 8% false-positive rate means roughly one supported claim in twelve gets flagged, and if your failure policy is to abstain, that is a meaningful volume of good answers converted into escalations. The checker's misses matter more: 29% of intrinsic errors passed, mostly ones requiring the reader to combine two facts or do arithmetic — "you purchased 19 days ago and the window is 14 days, so you are eligible" is contradicted only after subtracting, and a sentence-pair inference model does not subtract.

And it does not remove the manual pass from chapter 3. Every threshold quoted above came from comparing the checker against human labels, and those comparisons have to be repeated when the model changes, the corpus changes, or the question distribution shifts. The automation is a way of applying a measurement continuously; the measurement itself still comes from people.

What the arrangement produces is a residual you can put a number on, an escalation path that fires on a defined condition, and a set of thresholds that move that number in a known direction. Each of those is available because the answer now carries its evidence in a form code can resolve — which is the whole of the idea, and the reason it reaches errors that decoding settings, retrieval tuning and prompt instructions all left untouched.

That is a different kind of object from a sentence in a system prompt, and it turns out to be the only kind of object you can put in writing. The next chapter is about what writing it down actually looks like when somebody outside engineering asks for a guarantee.