Claim-level annotation

A response-level binary throws away three things at once. It loses the type, so intrinsic errors — claims contradicting text in the prompt — and extrinsic ones — claims the prompt never bears on — collapse into one bucket that moves for reasons nobody can attribute. It loses the density, so a reply with one wrong digit among nine good claims scores identically to a reply that is wrong in every sentence. And it loses the location, so nobody can go back and see which sentence failed or what in the source contradicted it.

Per-claim labelling recovers all three at the cost of a slower annotation pass. This chapter is the procedure: four steps in a fixed order, a storage format that survives in version control, scoring code that forces the denominator decisions into the open, and a sampling strategy, since a rate is only as meaningful as the sample it came from. It closes on what the pass actually costs — seven to nine minutes per response, and an inter-annotator agreement figure you should quote alongside every rate you publish.

The unit of measurement is a claim, not a response#

Decompose the answer into atomic claims. A claim is one assertion that could independently be true or false against the source. Split conjunctions ("refunds go to the original method and take 3 to 5 days" is two claims). Split a sentence carrying a number and an attribution ("the window is 30 days, per Section 2" asserts the window and asserts where it says so — the second half is separately checkable and is often where fabricated citations hide). Do not split a single assertion across the clauses that express it. Drop pure discourse — greetings, "happy to help", restatements of the question — since they assert nothing and inflate the denominator.

Working one sentence through those rules makes the boundary concrete. Take "the window is 30 days, per Section 2, and requests outside it may be escalated." The coordinating conjunction splits off the escalation clause, which is one claim. The attribution splits off "per Section 2", which asserts where the figure is stated and is separately checkable — and wrong here for its own reason, since Section 2 says something else. "Outside it" is a subordinate reference back to the window rather than a fresh assertion, so it does not split. Three claims, not two and not five, arrived at by applying three rules in order.

Granularity matters more than it appears to because every rate in this chapter is a fraction with the claim count underneath. An annotator who splits more finely finds the same errors and divides them by a larger number, so both rates fall with nothing about the system having changed; an annotator who splits coarsely inflates them the same way. That is why the guideline fixes the splitting rule rather than the target rate, and why two annotation passes run under different guidelines cannot be compared at all, however carefully each was executed.

Align each claim to the source. For every claim, find the span of the supplied context that settles it, and record the character offsets, not a paraphrase. The offsets are what make the label auditable later and what a downstream automated checker can be validated against.

Two things go wrong when that alignment is stored as a paraphrase instead. The first is that a paraphrase is written by the annotator, so it already carries their reading of the passage, and the label stops being independently re-checkable — a reviewer who disagrees ends up arguing with a summary rather than with the text. The second is that a paraphrase cannot be resolved back to a location, so nothing downstream can consume it. An automated checker that proposes a span has to be scored against a span, and a rate measured against paraphrases leaves the automation with no reference to be validated on. Recording offsets costs a drag of the mouse, and it is the difference between an annotation that supports a later engineering decision and one that supports only the meeting it was produced for.

Assign one of three labels. Supported if the aligned span entails the claim. Contradicted if the aligned span asserts something incompatible with it. Unsupported if no span in the supplied context bears on it either way. Contradicted claims are intrinsic; unsupported claims are extrinsic.

Mark inheritance. A claim that is a valid deduction from an earlier defective claim gets flagged as inherited rather than counted as a fresh failure. "At 19 days you are still eligible" follows correctly from a 30-day window; the reasoning is sound and the premise is not. Counting it as a second contradiction double-charges one error and distorts the density figure.

Applied to a support reply that answered a 19-day-old purchase against a policy stating a 14-day window, the whole annotation is one small table.

Four claims, one intrinsic error, one extrinsic error, one supported claim, one inherited. That is four numbers where the checkbox gave one bit, and each of them points at a different piece of work.

Storing it so a machine can score it#

The annotation needs to survive as data, not as a highlighted PDF. A flat JSONL record per response — one JSON object per line, so the file appends cleanly and reads a record at a time — with claims as a list, is enough and stays diffable in git.

{
  "id": "sup-48213",
  "context": "REFUND POLICY v4.2 — Section 2. Customers may request a refund within 14 days of the original purchase date. Section 3. Requests outside this window may be escalated to a manager at the manager's discretion. Section 4. Refunds are issued to the original payment method.",
  "question": "I bought this 19 days ago and want a refund. Am I eligible?",
  "claims": [
    {"text": "The refund window is 30 days from purchase",
     "label": "contradicted", "span": [41, 108], "inherited": false},
    {"text": "At 19 days the customer is eligible",
     "label": "contradicted", "span": null, "inherited": true},
    {"text": "Refunds are issued to the original payment method",
     "label": "supported", "span": [212, 268], "inherited": false},
    {"text": "Finance processes refunds in 3 to 5 business days",
     "label": "unsupported", "span": null, "inherited": false}
  ]
}

Scoring is then a few lines, and writing it out forces the denominator decisions into the open rather than leaving them implicit in a spreadsheet formula.

import json
from dataclasses import dataclass

@dataclass
class Rates:
    n_responses: int
    n_claims: int
    intrinsic_per_claim: float
    extrinsic_per_claim: float
    responses_with_any: float

def score(path: str) -> Rates:
    responses = [json.loads(line) for line in open(path)]
    fresh = [c for r in responses for c in r["claims"] if not c["inherited"]]
    intrinsic = [c for c in fresh if c["label"] == "contradicted"]
    extrinsic = [c for c in fresh if c["label"] == "unsupported"]
    dirty = [r for r in responses
             if any(c["label"] != "supported" for c in r["claims"])]
    return Rates(
        n_responses=len(responses),
        n_claims=len(fresh),
        intrinsic_per_claim=len(intrinsic) / len(fresh),
        extrinsic_per_claim=len(extrinsic) / len(fresh),
        responses_with_any=len(dirty) / len(responses),
    )

print(score("annotations/support-v4.jsonl"))
# Rates(n_responses=200, n_claims=2314, intrinsic_per_claim=0.0255,
#       extrinsic_per_claim=0.0138, responses_with_any=0.31)

Both denominators earn their place and they answer different questions. The per-claim rates — 2.55% intrinsic, 1.38% extrinsic — are the engineering signal, comparable across model versions and prompt changes, and stable even when average answer length shifts. The per-response figure of 31% is the user-facing signal, because a customer does not experience a rate, they experience one reply that either was or was not wrong somewhere. Report both and never let one stand in for the other. A change that shortens answers will improve the per-response number while leaving per-claim accuracy untouched, and a team watching only the response rate will bank that as a win.

Choosing which responses to label#

The default way teams pick the sample is the worst one available: annotate the responses somebody complained about. Those are the responses that were wrong and noticed, which selects hard for visible, high-salience errors and against exactly the fluent, plausible ones this topic is about. A set assembled from escalations will show an intrinsic rate near 100% and tell you nothing, because the denominator is missing.

Draw from production logs instead, uniformly at random over a fixed window, and record the window. Two hundred responses is enough to see the shape of the problem and to settle an argument about where a quarter should go; it is not enough to detect a two-point improvement, which is a distinction chapter 8 makes precise when it works out sample sizes. Sample from a single window rather than accumulating opportunistically, because question mix drifts — a pricing change or a product launch shifts what customers ask within days, and a set stitched together across three months is measuring the drift as much as the model.

Alongside the rolling sample, freeze a regression set: 100 or so responses whose annotations are checked in and never re-drawn. The rolling sample answers "how are we doing now"; the frozen set answers "did this change help", because it holds the questions constant while the system underneath changes. Re-running the frozen questions against a new prompt and re-annotating only the responses that differ from the stored ones cuts the annotation cost of a routine change by roughly two thirds, since most answers to most questions do not move.

One caution about the frozen set: it decays. Every time somebody looks at a failing example and edits the prompt to fix it, that example stops being an independent test and becomes part of the training signal. Rotate a quarter of it each quarter, and treat a suspiciously large gap between frozen-set performance and rolling-sample performance as evidence that the frozen set has been fitted rather than as good news.

What it costs, and where annotators disagree#

Claim-level annotation is slow. On a support corpus with a 1,400-word policy in context and answers averaging 11.6 claims, an experienced annotator took 7 to 9 minutes per response once the guidelines had settled, and 15 to 20 minutes before they had. Four hundred responses is therefore roughly a person-week for a single pass, or two person-weeks if you double-annotate a subset to measure agreement, which you should.

The first agreement measurement on that corpus came out at Cohen's kappa 0.58 across two annotators — kappa being the standard score for how often two people assign the same label once the agreement you would get from guessing has been discounted, so that 1.0 is perfect and 0.0 is chance.

Discounting chance is what makes that figure worth publishing rather than a formality. Most claims in most responses are supported, so two annotators who never read the text and simply wrote "supported" every time would agree with each other on the great majority of claims. A raw agreement percentage rewards exactly that, and would report a high number for a process carrying no information at all. Kappa subtracts the agreement the label distribution alone would have produced and reports what is left over, which is why a figure that looks unimpressive beside a raw percentage is the one to quote.

A kappa of 0.58 is poor, and the disagreements clustered in three places. Decomposition granularity produced most of them: one annotator split "the manager may escalate at their discretion" into two claims and the other kept it as one, which changes every denominator downstream. The supported-versus-unsupported boundary produced the next tranche, over how much inference from a span is allowed before "the policy implies this" becomes "the policy does not say this". And inheritance was applied inconsistently, with one annotator marking downstream conclusions fresh.

Writing those three decisions down as rules moved kappa to 0.79 on the next batch: split on coordinating conjunctions and on attributions but not on subordinate clauses; require the span to entail the claim without an intervening assumption; mark inheritance whenever the claim is false only because a premise was. Two things are worth copying from how those rules ended up written. Each one is stated as a decision procedure rather than a principle — "split on coordinating conjunctions" tells an annotator what to do at 4pm on a Thursday, while "split into atomic claims" starts an argument. And each is paired with two worked examples from the actual corpus, one on each side of the line, because the boundary cases are what the guidelines exist for and an abstract rule with no examples gets re-interpreted by every reader.

The annotation tooling matters more than it seems, too. Nobody should be typing character offsets. Annotators work in an interface that shows the context on the left and the answer on the right, splits the answer into candidate claims they can merge or divide, and records offsets from a drag-selection over the source, so that highlighting the sentence is what stores the span. Building that took about three days and roughly halved the per-response time, most of the saving coming from eliminating tab-switching between a document and a spreadsheet. It also removed a whole category of silent data errors, since a hand-typed offset that is off by forty characters produces an annotation that looks fine and points at the wrong sentence.

That agreement figure is the honest ceiling to quote alongside your rates: a measured 2.55% intrinsic rate produced by annotators who agree 79% of the time above chance is not a figure to defend to three decimal places. Two further limits are worth stating plainly. The procedure measures faithfulness against the supplied passage, so a claim marked supported is supported by a document that may itself be wrong or out of date, and nothing here catches that. And it is manual: automating it is possible and is what chapter 7 builds towards, but an automated checker has to be validated against human labels, so the manual pass comes first and does not go away — it becomes the smaller, periodic thing you check the automation against.

Why that ordering cannot be reversed is worth stating, because the tempting shortcut is to have a model produce the first pass and have people spot-check it. A checker's output is interpretable only against something known to be right, and if the known-right set is itself produced by a checker, the measurement has no anchor at all: a systematic error in the checker becomes a systematic error in the standard it is judged against, and nothing in the resulting numbers will ever surface it. Human labels are not better because people are more careful. They are better because they come from a process whose errors are uncorrelated with the automation's, which is the only property that makes the comparison mean anything.

What the procedure produces is two rates that respond to different work, attached to spans that make every label re-checkable by somebody who was not in the room. That is the difference between a quality argument and a measurement, and the cost of it is a person-week per pass, which is why the sampling decisions above matter as much as the labelling ones.

Two rates now exist and can be moved. The obvious lever is to give the model better evidence, which raises a question the numbers cannot answer on their own: what exactly is "the evidence", how does it get into the prompt, and what does it mean when someone says the right document came back first? That object has to be pinned down before the experiment that uses it can be read.