A labelled set is an instrument you build before you trust its numbers

Every comparison in the rest of this topic is read off one artefact. That a design scores eight points higher, that a failure belongs to generation rather than retrieval, that a corpus does not warrant retrieval at all — each of those is a number produced by running a set of questions through a system and grading the answers, and none of them means anything until the set and the grading rule are things somebody built on purpose. A labelled set here is a list of records, each pairing a question with the identifier of the passage that answers it, together with a stated procedure for deciding whether an answer is right.

Calling that an instrument is more than a turn of phrase, and it is worth saying where the comparison stops. A thermometer is trusted because somebody characterised its error before anybody read a temperature off it; a labelled set earns the same trust the same way, by having its agreement with human judgement and its run-to-run variability measured before its readings are quoted. Where the analogy breaks is that a thermometer's error is a property of the thermometer alone, while a question set's error depends on what it is being asked to measure — one set can be accurate about questions on prose and useless about questions on tables at the same time. The analogy also understates the problem, because a thermometer does not silently become wrong when somebody redecorates the room, and a labelled set does exactly that when a passage it points at is edited.

A score is a share multiplied by a pass rate#

The arithmetic that governs everything else here is a weighted sum, and it is worth writing out because the consequence is easy to state and easy to disbelieve. Sort the questions in a set into kinds — lookups, procedures, comparisons, and so on. The overall score is the sum, across those kinds, of the fraction of the set that each kind occupies multiplied by the fraction of that kind the system gets right. Each kind's contribution is a product of exactly two factors: how much of the set it is, and how well the system does on it.

A product with a zero factor is zero whatever the other factor is. So a kind of question that occupies none of the set contributes nothing to the score, and it contributes nothing whether the system answers it perfectly or fails it every single time. The score is not merely uninformative about the missing kind; it is numerically identical in the best case and the worst case. Two systems that differ only in how they handle a kind of question nobody sampled will produce the same score to the last decimal place.

That is the whole reason a small demonstration cannot become a measurement by being repeated. The number of questions of a given kind that a sample of a given size is expected to contain is another product — the size of the sample multiplied by that kind's share of real traffic — and for a small sample and a modest share the expected count rounds to none at all. A dozen questions chosen by the person who built the system is not a small measurement. It is a demonstration, and a demonstration reports on the kinds of question the demonstrator had in mind, which is a different set from the kinds the system will meet.

Nothing in that requires anybody to have been careless. The person choosing the questions picks the ones they can judge, and the ones they can judge are the ones drawn from material they know. The bias is in the sampling procedure rather than in the person, which is why the repair is procedural too.

Two to three hundred questions, each tagged with the paragraph that answers it#

The set is a list of records, and each record carries the question, the identifier of the passage that answers it, and enough of that passage's text to survive the corpus being re-chunked.

{
  "qid": "q-0184",
  "question": "how long is the enterprise trial on the legacy plan",
  "asked_by": "support",
  "source": "conversation_log:2026-05-12",
  "stratum": "policy_lookup",
  "gold_doc_id": "confluence:PRICE/8821",
  "gold_chunk_id": "confluence:PRICE/8821#0003",
  "gold_text": "Legacy Enterprise trials run for thirty (30) days from activation and do not auto-convert.",
  "expected": "30 days, and it does not auto-convert",
  "grading": "rubric",
  "labelled_by": "j.okafor",
  "labelled_at": "2026-05-20"
}

Three of those fields are the same fact recorded at three levels of durability, and the redundancy is deliberate. gold_chunk_id names the exact unit the retriever would have to return; gold_doc_id names the document that unit came from and survives the chunk boundaries moving; gold_text is the sentence itself and survives the document being re-identified. A set that records only the chunk identifier stops being usable the first time anybody changes the chunker, which is a change this topic expects teams to make. Recording labelled_by and labelled_at costs two strings and is what makes it possible to ask later whether a run of disputed labels came from one person or one week.

Where the questions come from decides what the set can tell you. Questions written by the team building the system inherit the team's model of the corpus, which is the sampling bias above arriving in its most common form, and the case worked at the end of this chapter illustrates how far it can go. Questions sampled from real traffic — conversation logs, the search box the assistant replaced, support tickets, the internal help channel — carry the actual distribution, including the malformed ones, the ones with three constraints in a sentence, and the ones about documents nobody on the team has read. Before there is any traffic, the closest substitute is the search logs of whatever the assistant is replacing, which is usually available and usually nobody thinks to ask for it.

Stratification is what keeps a sampled set from being dominated by its most common shape. A realistic breakdown over a support corpus might be 40% single-fact lookups, 20% procedural how-do-I questions, 15% questions with a qualifying constraint such as a plan tier or a region, 10% comparisons across two documents, 10% questions whose answer is genuinely not in the corpus, and 5% ambiguous questions where the correct behaviour is to ask back. The 10% with no answer in the corpus are the stratum most often left out and the one that catches the most damaging behaviour, which is a system that produces a confident answer when it should decline. Recording the stratum on each record also means every result can be broken down by it, and a per-stratum breakdown is the only thing that shows a change helping one kind of question while ruining another.

Those six shares sum to exactly a hundred per cent, with nothing left over, and the closure is worth checking rather than assuming. It says the six strata are exhaustive: every question in the set falls into one of them, so there is no unclassified remainder quietly holding a seventh kind. That is what licenses reading the overall score as the weighted sum described above, since a sum over categories only equals the whole when the categories cover the whole. It also makes the shares a budget rather than a wish list. Adding a seventh stratum means taking share from the six that exist, and deciding which one to shrink is a decision about what this instrument is for.

Tagging the gold passage is the part that takes real hours — roughly four to six hours of a domain expert's time for 250 questions, working from the corpus with a search tool — and it is what makes the set usable for anything beyond a pass rate. That works out at a minute or so a question, which is the right order for a task that is mostly searching and occasionally reading, and it is worth quoting to whoever is being asked for the time. With the gold chunk identifier recorded, the same run that produces the answer also produces whether that chunk was retrieved, at what rank, and whether it appeared in the rendered prompt, which is the raw material for the failure attribution in the next chapter. Without it, the set produces one bit per question and no way to ask why.

A stated rule that turns free text into right or wrong#

An answer is a paragraph and the score has to be a number, so a rule has to convert one into the other, chosen before the run rather than argued about afterwards.

Read that table by the gaps in its third column rather than by the entries. The two mechanical rules run in seconds, a judge model takes minutes, and a person takes hours — each step a factor of tens, and the two steps together putting a human rubric something like three orders of magnitude away from a string comparison. That distance is the entire reason judge models are used at all. A grading step measured in hours cannot sit in front of a merge and one measured in minutes can, and no argument about grading quality changes which of those is true.

The fourth column breaks the pattern of the first three, and the break is the informative part. Span match, extraction and the human rubric each fail for a reason that is a property of the rule itself — a paraphrase defeats string matching whatever you do, and a human grader drifts over months whoever they are. The judge model's entry names something else: it fails to the extent that you have not measured it. That is a statement about your knowledge rather than about the instrument, and it is the only one of the four failures you can shrink without changing the rule.

Span match is the only rule that is genuinely objective, and it is the one that most often disagrees with a human, because an assistant that answers "the legacy tier gives you a month, and it won't roll over automatically" is correct and contains none of the gold string. Most sets end up mixed: exact rules for the questions that have a single value, a rubric for the rest.

A judge model is the practical way to run 300 rubric gradings on every change, and its number is only usable to the extent that its agreement with a human has been measured rather than assumed. The measurement is not elaborate. Have a person grade a stratified 100 of the questions, run the judge over the same 100, and compute agreement — raw agreement plus Cohen's kappa, a coefficient that discounts the agreement two graders would have reached by chance alone, since raw agreement flatters any judge on a set where most answers are correct. A judge at 91% raw agreement and κ = 0.74 is usable for tracking a difference between two systems; a judge at 82% raw and κ = 0.41 is producing noise with a decimal point on it, and the usual fix is that the rubric is underspecified rather than that the judge model is weak. Re-measure agreement whenever the judge model version changes, because a provider upgrading a model silently re-calibrates your instrument.

Kappa is worth unpacking rather than accepting, because the shape of the calculation is what makes it the right number. Take how often the two graders actually agreed, subtract how often two graders would agree by coincidence given how often each of them says "correct", and divide by how much room there was above that coincidence level. The subtraction removes a constant that has nothing to do with skill, and the division rescales what is left so that agreeing no better than chance scores zero and agreeing perfectly scores one.

The gap between the two judges quoted above is where that matters. Raw agreement falls by nine points between them, while kappa falls by about a third of its entire range. Those two descriptions of the same pair of judges cannot both be proportionate, and raw agreement is the inflated one — inflated by exactly the constant that kappa subtracts.

This is the mistake worth naming in this chapter, and it is not an arithmetic slip. It is a belief that a defence works. Raw agreement is being trusted to tell a judge that discriminates apart from a judge that does not, and on a set where most answers are correct it reads high for both — a judge that marks every answer correct without reading it agrees with the human on every question the human passed. The signal is very nearly constant across the two cases it exists to separate, and a near-constant carries almost no information about a variable. Kappa is not a refinement of raw agreement; it is the repair for that specific defect, and quoting raw agreement without it is quoting the part of the number that the base rate produced.

Judges have known biases that matter for exactly the comparison this topic ends with. They prefer longer answers, they prefer answers that restate the question, and they are more generous to an answer produced by a model from the same family. Giving the judge the gold passage and asking it to decide whether the answer is consistent with that passage — rather than asking it which of two answers is better — removes most of the length bias and makes the grading independent of which system produced the text.

The comparison is paired, and the interval goes on the difference#

Two systems run over the same 300 questions, A scoring 71% and B scoring 79%. The tempting way to describe the uncertainty is to put an interval on each: with n = 300, the normal-approximation 95% interval on 71% is about ±5.1 points and on 79% about ±4.6, so the top of the interval around the lower score sits above the bottom of the interval around the higher one — they overlap, if only just — and the honest-sounding conclusion is that the difference may not be real. That reasoning is wrong, and it is wrong in the direction of throwing away a real improvement, because it treats the two runs as independent samples when they are the same questions.

Almost every question is answered the same way by both systems. Those contribute nothing to the difference and everything to the width of each separate interval. What carries the difference is the discordant questions — the ones where exactly one system got it right — and the interval belongs on that quantity.

The cancellation is exact, which is what makes the argument a proof rather than an intuition. Score each question as one if the system got it right and zero if it did not, and take B's score minus A's score question by question. A question both systems answered correctly gives one minus one, which is zero. A question both got wrong gives zero minus zero, which is also zero. Every concordant question therefore contributes exactly nothing, with no residue to worry about, and the average of those per-question differences is decided entirely by the questions where the two systems disagreed. The residual left after the cancellation is signed rather than symmetric: it is the count where B alone was right, less the count where A alone was right.

import numpy as np


def paired_diff(a: dict[str, bool], b: dict[str, bool], iters: int = 10_000) -> dict:
    """a, b: {qid: correct?} over the SAME question set."""
    qids = sorted(a)
    d = np.array([int(b[q]) - int(a[q]) for q in qids])   # +1, 0 or -1 per question

    b_only = int((d == 1).sum())    # B right, A wrong
    a_only = int((d == -1).sum())   # A right, B wrong

    rng = np.random.default_rng(0)
    boot = np.array([d[rng.integers(0, len(d), len(d))].mean() for _ in range(iters)])
    lo, hi = np.percentile(boot, [2.5, 97.5])

    return {
        "diff_points": round(100 * d.mean(), 1),
        "ci95": (round(100 * lo, 1), round(100 * hi, 1)),
        "b_only": b_only,
        "a_only": a_only,
        "regressions": [q for q in qids if a[q] and not b[q]],
    }

The bootstrap in that function is a resampling procedure: draw 300 per-question differences at random from the 300 you have, allowing repeats, take their mean, do that ten thousand times, and read the middle 95% of those means as the interval. It makes no assumption about the shape of the distribution, which is convenient here because the per-question difference only takes the values −1, 0 and +1. On the run behind those two scores, b_only was 38 and a_only was 14, a net of 24 questions out of 300, which is the 8.0-point gap. The paired bootstrap interval came out at 3.4 to 12.6 points; the naive interval on the difference of two independent proportions is roughly ±7.3 points, wider than the comparison supports and wide enough to make a real effect look marginal. The paired interval excludes zero, so the improvement is not a labelling artefact, and its width still says the true gap could be three points or could be twelve — which is the number that matters when the eight points are being traded against 2.7 seconds of latency.

Two properties of that interval are worth checking rather than taking on trust, because both of them are things a bootstrap is not obliged to produce. Its two ends sit the same distance either side of the observed gap, so the resampled distribution came out symmetric and the interval can be read as a plain plus-or-minus. And that half-width — a shade over four and a half points — is what a closed-form standard error on the same per-question differences gives, computed with no resampling at all. Two procedures that share no machinery landing on the same number is the licence to use the simpler one for a rough answer and the bootstrap when the distribution is less obliging.

The counts also say how big this comparison really is. Fifty-two questions disagreed and the rest did not, so the difference between the two systems rests on those fifty-two rather than on 300. That is the honest sample size for the claim, and it is why the interval is as wide as it is despite the set being three hundred questions long.

The regressions list is the field that gets used most and is easiest to omit. Fourteen questions that A answered and B did not are not noise to be averaged away; they are fourteen concrete cases, each with a gold passage, and reading five of them usually explains a mechanism — a query rewriter dropping a qualifier, a reranker preferring a fluent summary over the specific paragraph. A net gain with an unexamined regression set is a change that improved the average by making a specific class of question reliably worse. That list is the same length as a_only by construction, which means the count is never a figure anybody has to trust: it is a set of identifiers somebody can open.

Running the set is cheap enough to do on every change once it exists. Three hundred questions through the pipeline is 300 retrievals and 300 generations, which at a few cents each is under ten dollars and, with modest concurrency, under four minutes of wall clock; the judge grading adds about a dollar. That puts it comfortably inside a pre-merge check for anything touching retrieval, and the discipline that matters more than the automation is running it on the change alone rather than on a branch carrying three changes, for the same attribution reason the baseline chapter gave.

Set size follows from the smallest difference worth resolving. At 300 questions a paired comparison resolves gaps of roughly five points; resolving two points takes something closer to 1,500, which is why teams that need to detect small changes invest in growing the set rather than in more clever statistics. Two to three hundred is the size at which the labelling is affordable and the instrument still distinguishes the differences anyone is actually arguing about.

The reason those two numbers sit so far apart is that precision improves with the square root of the count rather than with the count itself. Halving the resolvable gap costs four times the questions, so the step from a five-point resolution to a two-point one is a step from a few hundred questions to well over a thousand, rather than to a few hundred more. That is also a check on the first of the two figures: the paired interval worked out above has a half-width just under five points, on a set of exactly this size, which is the same rule of thumb reached from the other direction.

How a set decays, and what it never covered#

It is not a benchmark. A published retrieval benchmark measures a different corpus with different documents and different question shapes, and a system's score on it predicts very little about the corpus you have. It is also not a regression suite in the software sense, because answers are not deterministic strings; running it twice on an unchanged system produces slightly different scores, and knowing that run-to-run variance — typically one to two points with temperature at zero, more if any component samples — is what stops anyone chasing a three-point move that is noise.

Those two figures are separate bars and a difference has to clear both, which is the part most easily run together. Run-to-run variance is the instrument reading differently on the same system, and it is fixed by repeating the run. The paired interval is the instrument reading a sample rather than the whole population, and it is fixed by adding questions. A three-point move sits above the first bar and below the second: repeating the run would not make it go away, and it still cannot be called, because a set of this size cannot resolve it. When an instrument's own error is the same size as the effect, the reading is not a small result. It is not a result.

The set decays, and the decay is invisible. A gold passage gets edited, a document is deleted, a policy changes, and the recorded answer is now wrong while the label still says it is right, so a correct system starts scoring as incorrect. Storing gold_text alongside gold_chunk_id gives a cheap detector: before each run, check that the recorded text still exists in the corpus, and flag questions where it does not. On a corpus with normal editing traffic, five to eight per cent of labels need revisiting after six months.

Note which direction that decay pushes. A stale label marks a correct answer wrong, so it depresses scores rather than inflating them, and in a paired comparison it depresses both systems on the same questions. On every stale question where the two systems behave alike it therefore cancels out of the difference exactly, by the argument above, leaving the absolute number damaged and the comparison largely intact. That protection is conditional, and the condition is worth making somebody demonstrate: it holds only where the stale label does not fall on a question the two systems answer differently. A team quoting the difference is largely insulated from label rot; a team quoting the pass rate is not, and a six-month drift of that size is comparable to the effects people argue about.

Finally, the set answers only the questions in it. A system can score 79% on 300 sampled questions and fail entirely on a category that appears twice a week in production and never made it into the sample, which is an argument for periodically re-sampling from traffic rather than treating the first set as permanent.

The whole of this comes back to the product at the start. A score is a share multiplied by a pass rate, summed over kinds of question, and the two factors fail in different ways: a wrong share makes the score answer a question nobody asked, and an unmeasured grading rule makes the pass rate mean something other than what it says. Sampling from traffic and stratifying fixes the first. Measuring agreement, and quoting kappa rather than raw agreement, fixes the second. Pairing the comparison and putting the interval on the per-question difference is what stops the honest width of the remaining uncertainty being mistaken for an absence of effect.

With the instrument in place, a failing question is no longer an anecdote. It is a record carrying a gold chunk identifier, a trace id, a rank, an archived prompt and a graded answer, and those fields between them locate the failure at one specific stage — which is what the next chapter uses them for.