Stratified sampling for judge validation

Every probe in the two preceding chapters compares a judge against itself, which establishes that it is consistent and says nothing about whether it is right. Validation is the step that puts an external standard in: a person applies the same written rubric to the same outputs, blind to what the judge said, and the two sets of verdicts are compared. Fifty hand labels is what most teams can actually afford — two people, an afternoon each, one argument about a borderline case — and it is a small enough budget that how it is spent decides what the resulting number is capable of saying.

Which is why a stratified sample, one drawn in fixed quotas from groups you define in advance rather than uniformly from whatever traffic happens to supply, is worth the extra half hour of setup over a random draw from production. The groups are chosen so that the fifty contain the cases where the judge could plausibly be wrong, in numbers large enough to see. A stratified sample is deliberately not a miniature of production, and that is the source of both its usefulness and the one way it is routinely misread.

What a random draw actually measures#

A support-reply judge — one grading whether a customer-support reply satisfies a written standard — passes 92% of production outputs, and it is roughly right to do so, because most tickets are routine and most replies are fine. Draw fifty at random and you get about forty-six the judge passed and four it failed, because 92% of fifty is forty-six. Now imagine a broken judge that has learned nothing at all and simply outputs pass on everything. On that same sample it agrees with your labels on every output you also passed, which is around forty-six of fifty, for 92% agreement. A real judge that scores 88% on such a sample has just been beaten by a constant.

Following the constant judge's arithmetic all the way through is what makes the point stick, because the trick is that it never has to be right about anything. It passes all fifty. The humans passed about forty-six. It therefore matches on those forty-six and misses only the four the humans failed, and forty-six over fifty is the 92%. Nothing in that calculation required the constant judge to read a single reply. Its score is set entirely by how lopsided the sample is, and the same instrument scored on a sample where half the replies were bad would agree with the humans half the time.

The agreement percentage on an imbalanced sample is dominated by the majority class, and the majority class is the part nobody was ever worried about. An unbiased estimate of a quantity whose value is set almost entirely by easy cases is a precise measurement of the wrong thing. That is the specific way a random draw misleads: it is not biased, it is not too small, and every step of drawing it was done correctly. What it measures is the agreement rate over production, which is a real quantity, and the question anybody asks a validation round is whether the judge can be trusted on the cases where a verdict is in doubt.

Building the stratified sample#

Draw the fifty from strata chosen to concentrate the budget on the region where the answer is in doubt, and reweight to production at the end if you need a population estimate.

The quotas sum to fifty — eight, eight, twelve, ten and twelve — and the shape of them is the argument. Sixteen of the fifty, the two confident strata, go to cases the judge was sure about, which is enough to catch a judge that is confidently passing things nobody would and not so many that the budget drains into cases where everyone already agrees. The remaining thirty-four go where verdicts are contested, unstable or expensive. Adjust the split if your rubric has a criterion you are specifically worried about — a stratum of ten outputs that exercise it is a better use of the budget than ten more borderline cases — but keep the composition written down, because next quarter's round is only comparable to this one if it was drawn the same way.

Two of those strata are only available if your harness records more than a verdict. The borderline stratum needs some per-criterion detail in the judge's output, which a structured verdict schema gives you for free if you ask for criterion-level fields rather than a single label. The self-disagreement stratum is the per-pair flip rate from chapter 1, where each comparison is run in both orders, applied to single scoring instead: it needs a second judging pass on some slice of traffic, which is worth running on a few hundred outputs a week for exactly this purpose.

Budget the labelling honestly, because the schedule is what kills this exercise rather than the cost. Fifty outputs at two to four minutes each is a couple of hours per labeller, and with two labellers plus an adjudication pass on the items they disagree about, the whole round is most of a working day spread across two people. Adjudicate by having both labellers reread the disputed items together and agree a final label; that adjudicated set is your gold standard, and the pre-adjudication agreement between the two of them is what you use for the ceiling later, so record both. A quarterly cadence is achievable at that price. A monthly one is not, and a team that promises monthly will quietly stop after the second round.

Label all fifty by hand against the rubric, blind to the judge's verdict. Blindness matters more than it sounds like it should: knowing the judge passed something makes a human labeller find reasons it passes, and the resulting agreement rate is contaminated in the direction that flatters the judge. Hide the verdict in the labelling interface, and record each labeller's reasoning in one sentence, because those sentences are what you will read when the numbers disagree.

Why 88% is worth either 0.14 or 0.76#

Suppose the judge agrees with your labels on 44 of the 50 — 88%. That number by itself is uninterpretable, because agreement includes agreement you would have got by chance from the base rates alone. The chance-corrected version is Cohen's kappa, which subtracts the agreement two independent raters would reach if each kept its own pass rate and answered at random.

The subtraction is the whole idea and it is worth stating before the code. Two graders match on an output either by both passing it or by both failing it. If each of them were simply flipping a weighted coin — passing at its own habitual rate and taking no notice of the reply — they would both pass an output with a probability equal to one rate times the other, and both fail it with a probability equal to one fail rate times the other. Adding those two products gives the matches available for free. What kappa reports is how much of the remaining room the graders actually took: observed agreement minus free agreement, divided by one minus free agreement, so that a grader taking all of the remaining room scores 1 and one taking none of it scores 0.

def cohens_kappa(judge: list[bool], human: list[bool]) -> float:
    n = len(judge)
    observed = sum(j == h for j, h in zip(judge, human)) / n
    judge_pass, human_pass = sum(judge) / n, sum(human) / n
    expected = judge_pass * human_pass + (1 - judge_pass) * (1 - human_pass)
    return (observed - expected) / (1 - expected)

Run that on the stratified fifty, where the judge passed 26 and the humans passed 28, and chance agreement comes to 0.502, so kappa is (0.88 - 0.502) / (1 - 0.502) = 0.76. Every step of that is arithmetic you can do on the numbers just given. The judge passed 26 of 50, which is a little over half, and the humans passed 28 of 50, a little over half again; multiplying those two gives the chance of both passing, a bit under three tenths. The two fail rates are a little under half each, and multiplying those gives the chance of both failing, a bit over a fifth. Adding the two comes to 0.502, so almost exactly half the matches were free. Observed agreement of 0.88 exceeds that by a little under four tenths, the room above chance was a little under a half, and dividing the first by the second lands on 0.76.

Run it on a random production fifty with the same 88% observed agreement, where the judge passed 93% and the humans 92%, and chance agreement is 0.861, so kappa is (0.88 - 0.861) / 0.139 = 0.14. The same two multiplications do it: 93% of the time times 92% of the time for both passing, which is most of it, and the two small fail rates multiplied together for both failing, which is almost nothing. Chance agreement of 0.861 leaves only 0.139 of room above it, and the observed 0.88 sits less than two hundredths above the 0.861, which is about a seventh of that small room and is the 0.14. Identical headline, and one of them describes a judge you can build a release gate on while the other describes a judge that is barely distinguishable from a rubber stamp.

The reason the stratified version scores so much higher is that its 88% was earned on contested cases, where chance agreement is low and getting it right means something. Notice which way the correction runs. It is not that the stratified sample was measured more carefully; both rounds observed the same 88%, and both observations are correct. What differs is how much of that 88% was available without reading anything, and on a sample built so that the judge passes about half and the humans pass about half, that share is at its smallest.

This is also the reason the kappa from a stratified sample is not an estimate of the kappa you would see in production — you built the sample to be hard. If somebody needs a production-level number, compute agreement within each stratum and reweight by each stratum's actual share of traffic, and report both figures with their labels attached. The stratified kappa answers "can this judge tell hard cases apart"; the reweighted figure answers "how often does this judge agree with us on a random output", and confusing them in a slide deck is how a 0.76 becomes an overclaim.

Two limits of kappa are worth knowing before you quote it. It is defined for categorical verdicts, so a judge emitting 1–5 scores needs a quadratic-weighted variant or a rank correlation instead of the binary form above; treating a 4-versus-5 disagreement as equivalent to a 1-versus-5 disagreement throws away most of what the scale was for. And kappa itself is sensitive to prevalence in ways that make cross-project comparisons unreliable, so a kappa of 0.76 on your rubric and a kappa of 0.76 on somebody else's are not the same achievement. Use it to compare judge versions against each other on a fixed sample, which is the comparison it handles well.

The ceiling you are measuring against#

A judge is not competing against perfection. Have a second person label the same fifty independently, and compare the two humans to each other. On the stratified fifty above, the two labellers agreed on 46 of 50, giving a human-human kappa of about 0.84 — the same chance correction applied to two people rather than to a person and a machine, on a sample built to make chance agreement small, so 46 out of 50 lands well above the 44 the judge managed and the correction treats them comparably. That is the ceiling — a judge at 0.76 is not far off the level of consistency two competent people reach on the same rubric, and pushing it to 0.90 would mean pushing it past the agreement humans manage with each other, which is a sign of a judge that has learned one labeller's idiosyncrasies rather than the rubric.

Reading the ceiling as a limit on the rubric rather than on the judge is what makes it actionable. Two competent people applying the same written standard to the same fifty replies disagree on four of them, and each of those four is a place where the standard admits two readings. A judge cannot be pushed past that without being taught which of the two readings one particular labeller prefers, which is learning a person rather than a specification. So a large gap between the judge and the ceiling is a judge problem, and a low ceiling is a rubric problem, and only one of those is repairable by anything done to the prompt.

Measure the ceiling once per rubric, and re-measure when the rubric changes materially. Without it, both possible reactions to a judge kappa of 0.76 are unfounded: you cannot tell whether it is nearly as good as this task allows, or whether two humans would have agreed 98% of the time and your judge is leaving a great deal on the floor.

Before a disagreement becomes a few-shot example#

Pasting disagreements into the judge prompt as few-shot examples showing the correct verdicts does work, and it comes with four conditions that are easy to skip.

Fix the rubric first where the rubric is the problem. If the judge got a case wrong because the criterion was ambiguous, an example teaches the judge to resolve that one instance your way while leaving the ambiguity in place for every case you did not label. Examples patch instances; rubric text patches classes.

Retire those examples from every future validation set. An example in the prompt is training data, and measuring agreement on an item the judge has been shown the answer to is not a measurement. Keep an explicit held-out pool that no example has ever come from.

Balance the directions. Six disagreements that are all "the judge was too lenient" will, pasted in as a block, teach the judge that failing is what a careful judge does, and the next validation round will find it failing things it used to pass correctly. Include examples of correct passes alongside the corrections, even though those feel redundant.

Re-validate on a fresh stratified sample afterwards, and treat the whole thing as a change to the metric rather than a tune-up. Adding few-shot examples changes what the judge scores, which means every number produced before the change and every number produced after it were produced by different instruments.

Fifty labels is a small budget and it buys more than it looks like it should, provided the fifty are drawn from where the answer is contested rather than from where it is obvious. The percentage those labels produce is not the deliverable; the chance-corrected figure is, read against the ceiling two humans reach on the same rubric, and read alongside the direction the disagreements point. A judge at 0.76 against a human ceiling of 0.84 is a usable instrument with a known limitation. The same judge reported as "88% agreement" is a number that could equally describe a rubber stamp.

The last of the four conditions above is the one with teeth, and it is not specific to few-shot examples. Everything in this chapter validates the judge as it exists on the day you run it. It says nothing about whether today's judge and last quarter's judge are the same instrument, which matters the moment anybody draws a line through a series of monthly scores and calls the slope progress.