The judge and the labelled set are instruments with their own error

The previous chapter ended with a citation accuracy of 0.55 and two stories that both explain it: the pipeline attributes claims wrongly, or the checker that grades attributions does. Those lead to entirely different quarters of work, and nothing in the aggregate distinguishes them. That is not a defect of citation accuracy in particular. It is the standing condition of every metric built from a model verdict, and the way out of it is not a better metric but a measurement of the one you have.

There is a second instrument in the same position and it attracts less suspicion. Retrieval metrics compare what came back against a labelled set: a list of questions, each with the passage IDs somebody judged capable of supporting the answer. That set is not read out of the pipeline, it is constructed, and the construction has choices in it that move the resulting score by more than most pipeline changes do. This chapter measures both — the judge, and the set — and both measurements are cheap relative to the work they redirect.

A verdict is a reading, and readings have error#

Everything in this topic that produces a score for a single answer — faithfulness, answer relevance, citation accuracy — is an LLM or NLI verdict, natural language inference being the older and cheaper of the two, a small model trained to label whether one text entails, contradicts or says nothing about another. That is a reading from an instrument, and an uncalibrated instrument is not evidence, however precise its output looks when averaged to three decimal places over 400 samples. Calibration here means the ordinary thing: sample, label by hand, compare, and characterise the disagreement. What decides whether it is useful is three details that are easy to get wrong.

Sample stratified by the verdict rather than uniformly. Faithfulness failures typically run at 8–15% of claims, so 40 uniformly sampled claims contain four or five failures and tell you almost nothing about the failure class, which is the class you care about. Sampling half from each side gives the false-positive and false-negative rates separately, at the cost of needing to weight back when you estimate the population rate.

Forty claims sampled that way from the citation run, half from claims the checker had marked mis-attributed and half from claims it had passed, and labelled independently by two engineers who saw the claim and the passage it cited with the checker's verdict hidden, produced this:

The two engineers agreed with each other on 37 of the 40; the three disputes were resolved by reading the passage together. Raw agreement between the checker and those labels is the two cells on the diagonal over the total, 17 plus 13 over 40, which is 0.75. Every one of the ten disagreements is in the same direction: the checker calls a citation wrong when a human reading the same two texts calls it right.

Report agreement on the minority class, not only overall agreement. A checker that marks every claim supported scores 0.88 raw agreement on a corpus where 88% of claims are genuinely supported, while carrying no information at all. That is the belief this chapter exists to break, and it is worth seeing why raw agreement fails so completely: agreement rewards a checker for landing on the common answer, and on a skewed corpus the common answer is available without looking at anything. The number stays at 0.88 whether the checker reads the claim, reads the passage, or reads neither — it is constant in the one property that makes a checker useful. Cohen's kappa, which corrects for agreement expected by chance, is 0.00 for that checker, and it was 0.51 for the citation checker above — moderate, and low enough that a twenty-point movement in the metric could plausibly be the instrument rather than the pipeline.

def calibration_report(pairs: list[tuple[bool, bool]]) -> dict[str, float]:
    """pairs: (judge_says_pass, human_says_pass)"""
    tp = sum(j and h for j, h in pairs)
    fp = sum(j and not h for j, h in pairs)
    fn = sum((not j) and h for j, h in pairs)
    tn = sum((not j) and (not h) for j, h in pairs)
    n = len(pairs)
    agree = (tp + tn) / n
    p_chance = ((tp + fp) * (tp + fn) + (fn + tn) * (fp + tn)) / (n * n)
    return {
        "agreement": agree,
        "kappa": (agree - p_chance) / (1 - p_chance),
        "false_alarm_rate": fn / (fn + tp) if fn + tp else 0.0,   # judge fails a good claim
        "miss_rate": fp / (fp + tn) if fp + tn else 0.0,          # judge passes a bad claim
    }

The p_chance line is the whole of the correction and it is worth reading rather than trusting. It multiplies each margin of the table by the corresponding margin of the other axis and divides by the square of the total, which is the probability that two raters agree by landing on the same label independently. On the table above the checker passed 18 claims and the humans passed 26, and the four margins give a chance-agreement a little under a half. Subtract that from the observed 0.75, divide by what was left available above chance, and the result is the 0.51 quoted — a figure you can reproduce from the four cells with the arithmetic in the function and nothing else.

Read the disagreements rather than only counting them, because their distribution names the fix. Of the ten in the table, nine came from two mechanical causes. Six were overlapping chunks: the chunker used a 200-token stride over 512-token windows, so the sentence supporting a claim appeared in both p-3310 and p-3311, and the checker, shown only the cited passage, found a truncated version of the sentence and refused it. Three came from decomposition producing a claim that fused material from two sentences, which no single cited passage entails. The tenth was a genuine judgement call about whether "contact your account manager" entails "extensions are available on request".

Both mechanical causes have concrete repairs. Deduplicating overlapping chunks before the citation check, by mapping each passage to a canonical span, removed the six. Marking fused claims as unscoreable rather than failed removed the three, at the cost of dropping about 4% of claims from the denominator. After both, the checker reported 0.73 on the same run against a human estimate of 0.75, with kappa at 0.79.

That 0.73 is worth arriving at a second way, because two routes landing on one figure is what makes the repair believable rather than merely welcome. Take the original run and correct it statistically instead of fixing the checker. The checker failed 45 in every hundred cited claims, and of the 22 claims it failed in the sample, 9 were actually correct — a little over two fifths. Adding that share of the failed population back gives roughly eighteen points to return, which lifts 0.55 to a shade over 0.73. One route repairs the instrument and re-measures; the other leaves the instrument broken and corrects its output using the human labels. They share no step and they agree to a hundredth, which is what licenses acting on either. Extrapolating the same sample the other way, the false-positive rate of 9/26 puts the true citation accuracy somewhere near 0.75 rather than 0.55, so roughly twenty points of the alarming number belonged to the measuring device. The remaining twenty-five points of loss are real mis-attribution, which means both of the proposals below had something to fix and neither was going to move the metric until the checker stopped generating false alarms on top of it.

The labelled retrieval set flatters you in four specific ways#

Judges are the visible instrument. The labelled retrieval set — the questions, and the passage IDs marked relevant for each — is the invisible one, and it produces the numbers that redirect the most work, because a retrieval score of 0.94 sounds like a component that has been ruled out. Before a number like that moves a quarter of engineering, four properties of the set are worth checking, and each has a cheap test.

The first row is the one that produces the largest number and the least suspicion, and its mechanism is a loop rather than a bias. If the labelling process only ever shows an annotator passages the current retriever returned, then a passage the retriever misses cannot be marked relevant, so it never enters the gold set, so its absence from the results is never counted as a miss. Recall computed against that set is measuring whether the retriever returns what the retriever returned. The check in the third column is the loop stated as a number: in a set built by pooling, some gold passages will sit outside any single retriever's reach, and a healthy 10–20% is the evidence that labelling saw more than one system's output.

On a pipeline reporting 0.94, three of the four applied. The gold sets had been built by running the then-current retriever at k=10 and marking which of the returned passages a support engineer judged relevant, which meant the labelling process could not represent a passage the retriever had never surfaced — 0 of 180 gold IDs fell outside the retriever's top 50, where a pooled set on the same corpus produced 14%. That zero is the circularity showing up exactly where the table predicts. The questions had been written by a model prompted with each gold chunk, so their mean BM25 score against the gold passage was 18.4 against 7.9 for logged questions on the same corpus — a query more than twice as lexically close to its answer as a real one, on a channel that rewards lexical closeness. And the eval ran at k=20 while serving retrieved 20 and reranked to 5.

Rebuilding the set to fix those three took four days: pooling candidates across three retrievers, replacing 180 generated questions with 180 sampled from six weeks of logs and hand-labelled, and pinning the eval k to the serving k. Recall on the rebuilt set came back at 0.79. That is a fifteen-point correction to a number that had already been used to justify a hiring plan, and none of the pipeline changed.

Four days is the honest cost, and it is the reason these sets get built the cheap circular way in the first place. Labelling a pooled candidate list is slower than labelling a retriever's top 10, because pooling three retrievers at depth 50 produces 90 to 120 distinct passages per question before deduplication, and a support engineer reads a passage and decides relevance in about fifteen seconds. There is a defensible shortcut: have a judge model pre-label the pool, then have a human adjudicate only the passages the judge marked relevant plus a 10% sample of the ones it rejected. On the rebuild above that cut human reading from an estimated 34 hours to 9, and the 10% sample of rejections turned up two missed gold passages in 180 questions, which is a rate low enough to accept and high enough to justify continuing to sample.

Both of those hour figures are worth checking against the per-passage rate, since one of them holds and one needs an assumption stated. Thirty-four hours at fifteen seconds a passage is a little over eight thousand passage judgements, which across 180 questions is around forty-five passages each — well under the 90 to 120 the pool holds before deduplication, so the estimate is already assuming that deduplication removes more than half of it. The drop from thirty-four hours to nine, on the other hand, is exactly what the shortcut predicts: the human reads the judge's positives plus a tenth of its negatives, which on a pool where a small minority are relevant is roughly a quarter of the list, and a quarter of thirty-four hours is nine. The shortcut carries the obvious caveat that a judge pre-labelling the pool is another instrument in the chain, so its false-negative rate on the rejected sample is a number worth writing down rather than glancing at.

The limits of a calibrated judge#

Calibrating a judge does not make the metric correct in any absolute sense; it makes the judge's disagreement with a specific set of human labels known. If the humans labelled inconsistently — and two engineers who agreed on 37 of 40 claims here were working from a written rubric, which is why they agreed that often — the calibration inherits their error and reports it as truth. Writing the rubric first, having two people label independently, and reporting their agreement with each other alongside the judge's agreement with them is what keeps this from being a single opinion dressed as a ground truth.

It is also not a substitute for a sample-size argument. A checker whose kappa is 0.79 on 40 claims still produces a number with a confidence interval, and on 200 answers a citation accuracy of 0.73 carries a standard error near 0.03, so a run-to-run movement of two points is nothing. That standard error is computed over answers rather than over the several hundred claims inside them, which is the conservative choice and the right one: claims from one answer share a generator call, a passage set and a topic, so they are not independent draws, and treating each claim as its own sample would report an interval about half as wide as the data supports. Teams that calibrate carefully and then treat every dashboard wobble as a signal have swapped one error for another.

And calibration is expensive enough that it does not happen on every build. The realistic cadence is once when a metric is introduced, again whenever the judge model or the prompt changes, and again when the corpus changes character — because a judge validated on product documentation has no established error rate on legal contracts, and the assumption that it transfers is exactly the assumption that produced 0.55.

Two instruments, then, and the same discipline for both. A judge's verdict becomes evidence when it has been compared against human labels on this task and this corpus, and reported as a false-alarm rate and a false-negative rate rather than as a single agreement figure that a constant answer could achieve. A labelled set becomes evidence when its questions came from somewhere other than the passages, its gold passages came from somewhere other than the retriever being graded, and its k matches the one you ship. Neither check costs more than a few days, and both are cheaper than the quarter that a number nobody questioned is about to redirect.

With an instrument you have measured and a labelled set that is not built out of your own retriever's output, the retrieval-side numbers become worth computing. Both of them rest on the same labelling decision this chapter has been treating as given: somebody, for each retrieved passage, said relevant or not relevant.