Claim-level verification

Every comparator so far has been examined for what it cannot see. Exact match cannot see a format the consumer treats as identical; a normalizer that fixes that can invent equalities the system does not have; overlap counts shared wording, which is neither necessary nor sufficient; similarity counts topical proximity, which is blind to a dropped negation. This chapter builds the check they have been implicitly compared against, which is the only one on the list that reads the material the answer was supposed to be drawn from.

Claim-level verification breaks a model's answer into individual assertions and settles each one against a source document. It is not a similarity measure with a better model behind it; it is a different shape of check, producing a table of verdicts rather than a number, and it needs no reference answer at all — which is what lets it run on live traffic where no reference exists. It also costs about four orders of magnitude more per example than a cosine, so the interesting design question is not whether to use it but where in a week's work it runs.

One answer, six claims, four verdicts that matter#

A customer asks a claims assistant whether burst-pipe damage in a basement is covered and what the deadline is. The assistant answers:

Yes — sudden water damage from a burst pipe is covered under your policy,
subject to the $500 excess. Gradual seepage is excluded. You must notify us
within 30 days of discovering the damage, per § 14.2(c). Claims filed after
that window are assessed at the adjuster's discretion.

That answer scores 0.58 on ROUGE-L against the human reference and 0.95 on embedding cosine, because it uses the policy's own vocabulary and is unmistakably about the right subject. It contains a wrong excess, a section number that does not exist in the document, and a sentence about late claims that the policy never makes. The sentence-level comparisons cannot see any of that, because none of them ever looks at the policy.

Checking it means abandoning the idea that scoring is one comparison. The answer makes six separate assertions, and they are not right or wrong together:

Four out of six is not the interesting output. The interesting output is that claim 2 will cost the customer $500 and claim 5 will survive being checked by anybody who trusts the citation, and that both were invisible to every metric in the previous three chapters.

Where those six rows came from is worth tracing, because the decomposition is where this check succeeds or fails and it is the one step in the pipeline that returns no verdict of its own. The answer runs to four sentences and produced six claims, so two sentences were split in half. The first asserts two separate things: that burst-pipe damage is covered, and that the excess is $500. Those became rows 1 and 2, and they had to be separate rows because different parts of the policy settle them — § 6.1 decides the first and § 6.3 decides the second. Left welded together as one claim, they would have received one verdict, and a verdict on a compound assertion tends to follow whichever half is more prominent. The coverage half is the prominent one here, so the compound would very likely have come back supported and the $500 error would have vanished inside it. The third sentence split the same way and for the same reason: the notification deadline is one assertion and the section number attached to it is another, and only splitting them lets § 11.2 support the rule while the citation is separately found not to exist.

That gives the working rule for how far to split. A claim is atomic enough when a single span of the source is sufficient to settle it, and it is not atomic enough for as long as settling it would mean quoting two spans that could disagree with each other. Splitting past that point is harmless and mostly wasteful. Stopping short of it is how a wrong assertion rides through the check on the back of a right one, and no amount of care in the verification step recovers from it, because by then the two assertions are one string.

Three verdicts, not two#

The natural instinct is to label each claim true or false, and the third label is what makes the check usable. Supported means the source states it, and the verifier should be made to produce the span it relied on. Contradicted means the source states something incompatible. Not found means the source is silent — the claim may well be true in the world, it simply is not grounded in the material the answer was supposed to be drawn from.

Collapsing "not found" into "false" makes the check punish an assistant for correct general knowledge, and it will fire constantly on the ordinary connective tissue of a written answer. Collapsing it into "true" makes the check blind to exactly the failure everyone is worried about, because a fabrication is a claim the source does not contain. Keeping it separate lets the same pipeline serve two policies: a strict one for regulated answers, where anything not in the source is a defect, and a lenient one for a general assistant, where only contradictions block.

The distinction also carries the weighting. Claim 5 above — the invented citation — is a contradiction, since the document demonstrably has no § 14.2. Claim 6 is merely unsupported. A team that treats those identically will spend its attention on whichever is more numerous rather than on which does more damage, and in most corpora the unsupported claims are far more numerous.

The same check settles the cases the earlier chapters could not. A paraphrase that scored 0.31 on overlap decomposes into four claims that are all supported by its source; a verbatim splice that scored 0.62 has three supported claims and one — that the incident was caused by insufficient test coverage — contradicted by the postmortem's own finding that the change passed its tests. An answer whose only defect is a dropped negation comes back with its single claim contradicted against the decision record. One check, both failure modes, no thresholds.

A table of verdicts still has to become a number, and there are two obvious candidates that mean very different things. One is the share of claims that came back clean. The other is the share of answers containing no bad claim at all. These are not two views of one quantity, and what separates them is a conjunction: an answer is clean only if every claim in it is clean, so the share of clean answers falls away faster than the share of clean claims does, and it falls away faster the longer answers get. Take an answer with five claims of which four are supported and one is contradicted, alongside an answer whose five claims are all contradicted. On the claim ratio the first scores four-fifths and the second scores nothing, which is a real distinction and the right one if the question is how much of the answer was sound. On the per-answer verdict both score nothing, because both contain something a reader would have to catch.

Neither is wrong, and which one to publish depends on what a bad claim actually does. A fabricated section number is not diluted by the correct sentences surrounding it — the customer acts on it either way — so for this assistant the per-answer verdict is the one that tracks the consequence. The claim ratio also hides the shape of the errors altogether. A suite in which every answer carries one bad claim and a suite in which a tenth of the answers are bad from end to end can report an identical ratio, and they are completely different products with completely different fixes.

Decompose, then verify each claim#

Decomposition is one call; verification is one call per claim, and the calls are independent, so they run concurrently. Two properties of the verification call matter more than the prompt wording. The source goes in the system block with a cache breakpoint, which is the provider's prompt-caching mechanism: the encoded prefix is stored on the first call and re-read by the rest at a fraction of the input price, and every claim from the same answer shares that prefix. And the output is schema-constrained, so the aggregation step never has to parse prose.

import anthropic, json

client = anthropic.Anthropic()

VERDICT_SCHEMA = {
    "type": "object",
    "properties": {
        "verdict": {"type": "string",
                    "enum": ["supported", "contradicted", "not_found"]},
        "evidence": {"type": "string",
                     "description": "Verbatim span from the source; empty when not_found."},
    },
    "required": ["verdict", "evidence"],
    "additionalProperties": False,
}

def verify(claim: str, source: str) -> dict:
    msg = client.messages.create(
        model="claude-opus-5",
        max_tokens=1024,
        system=[{
            "type": "text",
            "text": "You judge single claims against the source document below.\n\n" + source,
            "cache_control": {"type": "ephemeral"},   # shared by every claim in this answer
        }],
        output_config={
            "effort": "low",
            "format": {"type": "json_schema", "schema": VERDICT_SCHEMA},
        },
        messages=[{"role": "user", "content": f"Claim: {claim}"}],
    )
    text = next(b.text for b in msg.content if b.type == "text")
    return json.loads(text)

Requiring a verbatim evidence span is not decoration. It is the cheapest available guard against a verifier that agrees with a claim because the claim sounded plausible: a span that does not appear in the source is caught by a substring test in the aggregation step, and in practice that test fires on a small but non-zero fraction of supported verdicts, all of which are wrong.

Why a failed substring test condemns the whole verdict rather than just the quotation is the step worth having. A supported verdict is meant to have been caused by something in the document: the verifier read a passage, and the passage entailed the claim. If the span it hands back is not in the document, then whatever produced the verdict was not a passage of the document, and the only other material in front of the verifier was the claim itself. So the verdict was reached from how plausible the claim sounded, which is precisely the failure the check exists to catch. The guard costs one string search per verdict and needs no second model to run it.

What it costs#

The reason this is not the default check everywhere is arithmetic. With a 2,000-token source document and an answer that decomposes into six claims, one example is seven model calls: roughly 15,500 input tokens and 1,400 output tokens. At $5 per million input and $25 per million output, that is about $0.11 per example, or $55 for one pass over a 500-example suite. Caching the source across the six verification calls — a write at 1.25× and five reads at roughly 0.1× instead of six full-price passes — takes it to about $0.07 and $35. Wall clock is around three seconds per example with the claim calls issued in parallel, so the suite finishes in about two and a half minutes at a concurrency of ten.

Every figure in that paragraph is reconstructible, and it is worth doing once, because the shape of the bill decides which lever is worth pulling. Seven calls is one decomposition plus one per claim, and this answer decomposed into six. The source document goes into all seven, so seven passes over a two-thousand-token document is fourteen thousand tokens on its own; the fifteen hundred or so remaining are the answer, the six claims and the instructions. Input at $5 per million turns fifteen and a half thousand tokens into a little under eight cents, output at $25 per million turns fourteen hundred tokens into three and a half, and the two together are the $0.11.

The caching saving falls out of the same breakdown, and it is larger than it looks because it lands on the part that dominates. Uncached, each of the six verification calls pays full price for the source, which is six passes over two thousand tokens, or six cents. Cached, the first of them writes the prefix at a quarter above full price and the other five read it at about a tenth, so the six passes together cost less than two full ones — a saving of about four cents, which is what takes $0.11 to $0.07 and $55 to $35. Nothing else in the bill moved. The lever that matters is the source document, not the model and not the number of claims.

The wall-clock figure is the same kind of division. Three seconds an example across 500 examples is fifteen hundred seconds of work, and running ten at a time finishes that in a tenth of the elapsed time, which is the two and a half minutes. That figure is therefore a property of the concurrency limit rather than of the check, which is worth remembering before anyone treats 2.5 min as an intrinsic cost of the technique.

Four orders of magnitude separate the last row from the first, and that gap is the whole reason the cheap metrics stay in the pipeline. Nobody runs claim verification on every commit against every example. The cost per example is also not fixed — a shorter source, a smaller model for the verifier, and batch processing at half price all move it, and each of those moves needs its own agreement measurement.

The size of that separation is worth reading off the table rather than taking as a slogan. Dividing seven hundredths of a dollar by the cosine's hundred-thousandth gives seven thousand, which is four orders of magnitude once rounded to the nearest power of ten, and the latency column gives a comparable ratio between three seconds and 0.4 ms. Against ROUGE-L there is no ratio to compute, because a local string comparison costs nothing to run at all — so the claim holds in the strict form against the cosine and in a looser one against the free check, where the honest statement is simply that one costs money and the other does not.

One property of the check changes where it can run, and it is easy to miss in a table of costs. Nothing in the pipeline uses a reference answer. The verdicts come from the answer and the source passages, both of which exist for every request the system serves, so the same check runs on live production traffic — where there is no reference and never will be. A 2% nightly sample of real conversations, scored the same way, catches a fabrication rate that is climbing on a document type nobody thought to put in the suite, which is the failure mode a fixed 500-example set is structurally unable to see. Every metric in the previous three chapters needs a reference and is therefore confined to the suite.

The asymmetry underneath that is worth naming, because it is what makes the production case possible rather than merely convenient. A reference answer is an artefact somebody wrote by hand, so it exists exactly where a person sat down and made one, which is the suite and nowhere else. The source passages are not an artefact at all. They are an input the system had to fetch in order to produce the answer, so they exist for every request by construction, and on most systems they are already in the logs. A check that consumes only the inputs and the outputs can run anywhere the system runs. A check that consumes a hand-written third thing is confined to wherever somebody wrote one.

The checker has its own error rate#

A verifier is a model, so it is wrong sometimes, and a correctness check whose own accuracy is unmeasured is just a more expensive opinion. The measurement is small: pull 200 claims stratified across the three verdicts, have two people label each independently, adjudicate the disagreements, and score the verifier against the adjudicated labels.

On one such suite the verifier agreed with the humans on 183 of 200. Fourteen of the seventeen disagreements sat on the supported / not-found boundary, where the claim was a mild paraphrase or a summary of two separate sentences and the humans split as well; the two annotators agreed with each other on 188 of 200, so the verifier is running close to the noise floor of its own labels. Three disagreements were real misses, all contradicted claims scored as supported, and all three involved a numeric comparison — a figure in the answer that differed from the source in a way that reads naturally.

Those counts are worth subtracting rather than converting into percentages, because what matters is the comparison between two small numbers. Agreeing on 183 of 200 leaves seventeen disagreements, and the paragraph above accounts for all of them: fourteen on the boundary where the humans split as well, and three real misses. The two annotators agreeing with each other on 188 of 200 leaves twelve disagreements between them, over the same claims. So the verifier disagrees with the adjudicated label five more times in two hundred than two trained people disagree with each other, and that difference is what "close to the noise floor" means here. Quoting the raw agreement rate as the verifier's accuracy would be the wrong move for the same reason: the stratified sample was built to contain hard claims, and on an unstratified draw the figure would be higher and would mean less.

Those three numeric misses are the honest limitation of the technique. Claim-level verification of numbers is materially weaker than verification of prose, which is why fields with types belong in typed comparisons rather than in this pipeline whenever the option exists.

Grounding is not truth#

Claim verification says nothing about whether the answer is any good. An answer consisting of three trivially supported sentences that does not address the customer's question passes cleanly, because every check here runs on the claims that were made. Usefulness, tone, structure and appropriate length are separate properties needing separate instruments, and a human preference test remains the right one for most of them.

It says nothing about omission either, and this is the gap most teams discover late. The pipeline inspects the output and asks whether the source supports it; nothing in that direction can detect a required element that the answer left out. Checking completeness means running the arrow the other way — enumerating what the answer was obliged to contain and checking coverage of that list — which is a different pipeline with different failure modes. Regulatory summaries that drop a mandated section fail on exactly this axis, and every claim they do make is supported.

Decomposition has failure modes of its own, and they are quieter because there is no verdict attached. A compound sentence that becomes one claim gets a single verdict for two assertions, and the verdict usually follows the more prominent one. A claim carrying a pronoun — "it does not apply to rented properties" — is unverifiable in isolation and tends to come back not_found, which reads as a fabrication when it is really a dereferencing failure; passing the surrounding sentence as context fixes most of this. And an implication that the answer conveys without stating — the reassuring tone of "you must notify us within 30 days" implying that notification is sufficient — never becomes a claim at all, so it is never checked.

Those failure modes add up to a belief worth removing, which is that the per-claim table is a faithful re-representation of the answer. It is not. It is the output of a model call with an error rate of its own, and it is the one step in the pipeline that returns no verdict, so nothing downstream is in a position to report that it went wrong. The consequence that bites hardest is that the decomposer chooses the denominator of every per-claim rate. Split the same answer more finely and the supported claims multiply while a single contradicted claim stays one row, so the supported percentage rises without the assistant having changed at all; split it more coarsely and the percentage falls. A per-claim rate is a joint property of the assistant and the decomposer, and comparing one across a change to the decomposition prompt compares nothing.

That is the strongest argument for the per-answer headline, and it is a different argument from the one about consequences. "Zero bad claims" is close to invariant under resplitting: cutting an answer's claims more finely does not turn a contradicted claim into a supported one, so an answer with something wrong in it remains an answer with something wrong in it. The share of answers that are clean survives a change of decomposer in a way the share of claims that are clean does not, which is why it is the number to put on a chart that has to stay comparable over months.

And the check inherits its source. supported means the source says so, which is a statement about the document, not about the world. If the retrieved passage is stale, if the policy was amended last quarter, or if the retriever fetched the wrong customer's policy entirely, the check will certify a wrong answer as grounded. Grounding and truth are different properties, and a claim-level pipeline measures the first.

The two research lines behind this are worth knowing by name. FActScore and the SAFE line of work formalised decompose-and-verify for long-form factual text; the faithfulness metric in the RAG evaluation frameworks is the same idea packaged for retrieval pipelines, and the RAG Evaluation Guide from DeepEval is the fastest way to point one of them at your own outputs and see the per-claim table for your data rather than for a benchmark.

What the check gives back for its price is a different kind of number. A mean similarity is a single figure whose relationship to correctness is an assumption; a per-answer verdict is a count of answers with nothing wrong in them, with an evidence span behind every judgement that a human can audit in seconds. That auditability is why the expensive check run on a sample is worth more than a cheap one run on everything.

All of this assumes an output that is prose — several sentences, each carrying an assertion, with no schema. Extraction outputs are the opposite shape: a dozen small typed fields where every value has its own consumer and its own comparator, already settled by the first two chapters of this topic. That case has a scoring problem of its own, and it starts with the fact that a record either matches or it does not.