Judge agreement and base rates

The base rate is the share of items already carrying the majority label before any judgment is applied, and therefore the score any instrument gets for free by guessing that label every time. It is the number that turns an impressive agreement figure into an ordinary one, and it is almost never quoted alongside the agreement figure it deflates.

This chapter is about what to measure once a judge is being proposed as a gate. Agreement with human labels is the natural thing to compute and the least informative; underneath it sit three quantities that decide whether the arrangement works — recall on failures, which is the only cell a gate acts on; the direction of the errors, which decides whether the metric bends or merely blurs; and the roll-up rule that converts criterion verdicts into a pass, which turns out to be a dial rather than a fact.

Most of the agreement was free#

Here is a calibration sample: fifty outputs drawn at random from a week of production traffic and labelled by hand, with the judge run over the same fifty.

Four numbers in a two-by-two grid is a confusion matrix, and reading one slowly the first time pays for itself. Each of the fifty outputs carries two labels, what a human said and what the judge said, and every output falls into exactly one of the four combinations those two labels can make. The top-left cell, 42, is the outputs both called a pass. The bottom-right cell, 4, is the outputs both called a fail. The other two are the disagreements: 3 outputs the judge passed and a human failed, and 1 the judge failed and a human passed. Add the four and 42 + 3 + 1 + 4 = 50, which is the whole sample, and that sum is the check that no output was counted twice or left out.

The totals down the edges are each instrument's own verdicts, read off on its own. The column headers give the humans': a human passed 43, which is the 42 in the top-left plus the 1 below it, and failed 7, which is 3 plus 4. Reading across the rows instead gives the judge's: it passed 42 + 3 = 45 and failed 1 + 4 = 5. Those four totals are the marginals, and they are worth having in hand because two raters can produce very different matrices while agreeing exactly on how many passes there were in total.

Agreement is the two cells on the diagonal running from top left to bottom right, because those are the cells where both labels carry the same word. It comes to 42 + 4 = 46 items, and 46 divided by 50 is 0.92 — the judge matched the human label on 46 of them, which is 92%. Two humans grading the same fifty had matched each other on 44, and 44 out of 50 is 88%, so the judge agreed with the humans better than the humans agreed with each other.

The belief that forms at exactly this point is that a high agreement rate validates a judge, and it is tempting for three separate and respectable reasons. It is the number a careful person computes first. It is comparable across teams and workloads in a way that most evaluation numbers are not. And it survived the obvious control: held against a second human it came out ahead rather than behind. What is true instead is narrower. Agreement measures the population the sample was drawn from, and this sample was drawn from a population in which almost everything passes.

Roughly 85% of production outputs on this workload pass, and this sample came in at 43 of 50, close enough. Work out what that concedes before reading the 92% again. A judge implemented as return "pass" — no model, no rubric, no cost — labels all fifty a pass, so it lands on the human label for every one of the 43 the humans passed and misses all 7 they failed. Its agreement is 43 out of 50, which is 86%, and no step in that arithmetic consulted an output. The judge's 92% is six points above a constant. Measured against the workload's underlying 85% rather than against this sample's slightly luckier 86%, it is seven points. Either subtraction says the same thing: the free part of the score is the large part and the earned part is the small one.

That subtraction worked because the pass rate on this workload was known and could be looked up. On a workload where nobody has measured it, or when two teams want to compare judges over populations with different pass rates, the subtraction has to come out of the matrix itself rather than out of a figure somebody remembers. Chance-corrected agreement makes the same point with a number that travels. Cohen's kappa asks how much of the observed agreement exceeds what two raters with these marginal rates would hit by coincidence:

def kappa(a: int, b: int, c: int, d: int) -> float:
    """a=both pass, b=judge pass/human fail, c=judge fail/human pass, d=both fail."""
    n = a + b + c + d
    observed = (a + d) / n
    judge_pass, human_pass = (a + b) / n, (a + c) / n
    expected = judge_pass * human_pass + (1 - judge_pass) * (1 - human_pass)
    return (observed - expected) / (1 - expected)


kappa(42, 3, 1, 4)   # 0.623

Every line of that function is doing something nameable, and the names are worth attaching before the result is quoted. n is the sample size, 42 + 3 + 1 + 4, which is 50. observed is the diagonal over the total, (42 + 4) / 50, which is 0.92 — the same agreement computed a second time. The two marginal shares follow: judge_pass is the judge's own pass rate, (42 + 3) / 50, which is 45 of 50 or nine-tenths, and human_pass is the human pass rate, (42 + 1) / 50, which is 43 of 50 or 86%.

expected is the line to slow down on, because it carries the whole idea. Hold both raters' pass rates exactly where they are, and suppose they assigned those verdicts at random, without reference to the output in front of them or to each other. That supposition is a construction rather than a claim about what these raters did. The question it answers is how often two raters like that would land on the same word anyway. They both say pass when two independent events coincide, and the chance of two independent events both happening is the product of their separate chances, so the both-pass corner arrives nine-tenths of the time times 86% of the time, which is a little over three-quarters. They both say fail when the two complements coincide: the judge fails one time in ten, the humans fail 7 of 50 or fourteen times in a hundred, and those multiply to 0.014. Adding the two corners gives 0.788, the agreement two raters with these marginals get for free.

Independence is what makes the multiplication legal, and it is also the assumption doing the work. Two raters who genuinely read the output are not independent of each other, and that is exactly the point. The amount by which their observed agreement exceeds the independent figure is the part that reading the output bought, and everything below it is coincidence they would have collected while asleep.

The last line turns that into a score. Observed agreement is 0.92 and the free part is 0.788, so the agreement above chance is 0.92 − 0.788, a shade over thirteen hundredths. The most that was available above chance is 1 − 0.788, because perfect agreement is 1 and the free part cannot be earned a second time; that headroom comes to a little over twenty-one hundredths. Kappa divides the first by the second — what share of the available headroom the two raters actually took — and the division lands on 0.62.

Two anchors make that readable. A kappa of 0 means the raters agreed exactly as often as two independent raters with those marginals would have, so nothing at all was earned. A kappa of 1 means they agreed on every item and took all of the headroom. Everything in between is the fraction of the headroom claimed, and because the base rate has been subtracted out of both the numerator and the denominator, the fraction is comparable across workloads whose base rates differ. That is what "a number that travels" means.

So the line to quote is observed agreement 0.92, expected agreement 0.788, kappa 0.62. That is a respectable number and a much smaller one than 92% sounded, because it is measured against the right baseline. The 92% is dominated by the 42 items in the top-left corner — outputs that obviously pass, which nobody would have disputed and no rubric was needed for. The comparison with human-human agreement does not rescue it either, because the humans were graded on the same easy sample. Two raters agreeing on 44 of 50 is 88% observed with a kappa around 0.53, so both instruments are mediocre on the same axis and the judge being marginally better tells you nothing about whether either is good enough.

A gate is a decision about failures#

A gate does one thing: it stops a release when the judge says fail. On the items the judge passes it takes no action at all, because shipping is what happens by default and a judge that agrees with the default has changed nothing. So "is this judge good enough to gate" is not a question about all fifty items. It is a question about the items where the gate would have to act. Blocking is an action taken on failures, so the only cells that matter to a gate are the right-hand column, and there are seven items in it.

Of seven human-labelled failures, the judge caught four. Recall on failures — the share of real failures the judge flags — is 4/7, or 57%. Both parts of that fraction come straight off the matrix. The denominator 7 is the right-hand column total, every output a human called a failure. The numerator 4 is the single cell in that column where the judge said fail as well. The other 3 in the column are failures the judge waved through, and a gate would have shipped every one of them.

Precision runs along a row rather than down a column. The judge said fail 5 times, which is the bottom row, 1 + 4. Four of those 5 were real failures and 1 was a good output flagged wrongly, so precision — the share of its flags that turn out to be real failures — is 4/5, or 80%. The two fractions share a numerator and differ only in what sits underneath it, and that difference is the whole distinction: recall asks what share of the real failures were caught, precision asks what share of the catches were real.

A gate operating at 57% recall lets nearly half of genuinely bad releases through, and it does so while reporting 92% agreement on the dashboard. Those two numbers are not in conflict and neither is miscomputed. They are fractions over different denominators, one over all 50 items and one over the 7 failures, and the 42 easy passes that inflate the first are not in the second at all.

The seven-item denominator is the second half of the problem. A Wilson interval is the range of true recall values that could plausibly have thrown up 4 out of 7, given how far a proportion measured on seven items wanders by luck alone; its arithmetic is standard and not worth reproducing, but its answer is. A Wilson interval on 4 of 7 runs from about 25% to 84%, which is to say the sample supports "the judge catches between a quarter and five-sixths of failures" and nothing more precise. That range spans the difference between a gate worth having and one worth switching off, so the measurement has not decided anything.

Fifty items drawn at random from an 85%-pass distribution simply does not contain enough failures to measure the quantity the gate depends on, and the reason is a division. If 85% of outputs pass, 15 in every hundred fail, so a random draw turns up failures at a rate of 0.15 per item pulled. Getting the recall interval down to ±10 points needs roughly 100 labelled failures, and 100 divided by 0.15 is about 670 — which at an 85% pass rate means labelling about 670 random outputs to harvest the hundred that carry information. The remaining five hundred and seventy are hand-labelled passes, paid for at the same rate per item and telling you nothing the base rate had not already told you. Or, far more cheaply, not sampling at random.

Sampling where the disagreement lives#

The fix is to stop drawing calibration samples uniformly and start drawing them where the judge is likely to be wrong. Three strata are available without any extra machinery:

• Criterion splits. Items where the judge passed some criteria and failed others sit near its decision boundary by construction.
• Repeat disagreement. Running the judge five times over the same item and keeping the items whose verdicts are not unanimous surfaces its own instability, at five times the inference cost on a small sample.
• Swap disagreement. For pairwise setups, the split verdicts from a swapped protocol are already a list of items the judge could not decide.

Drawing 50 items from those strata and labelling them by hand gives a very different picture:

Read the last column before the first one. The random sample held 7 human-labelled failures; the criterion-split sample held 22, out of the same 50 items and the same afternoon of hand-labelling. Nothing about the judge changed between those two rows. The sampling changed. Selecting items where the judge was internally split concentrates real failures, because an item that trips one criterion and clears another is far more likely to be a genuine partial failure than an item pulled blind from a stream where almost everything is fine. The recall figure that was 4 of 7 is now 14 of 22, and a fraction with 22 underneath it survives much more weight than one with 7.

The agreement column moves the other way for the same reason. It falls from 92% to 71% to 64% as the sample gets harder, and that fall is not the judge deteriorating between rows — it is the same judge with the free part of its score taken away, since the easy passes that made up most of the 92% are precisely what these strata exclude. Kappa falls alongside it, from 0.62 to 0.41 to 0.28, which matters because kappa has already had the base rate divided out. Even after chance correction the judge claims much less of the available headroom on hard items than on easy ones, so the drop is a fact about the judge and not an artefact of a harder mix.

The random sample says the judge is good. The borderline samples say the judge is barely better than a coin on the cases where a verdict is actually being decided, and they say it with denominators large enough to mean something. Neither picture is wrong — they measure agreement on different populations — but only one of them is about the items a gate would act on. The stratified numbers are not the judge's accuracy on production traffic, and quoting them as such would be the mirror-image error. They are conditional: agreement given that the item is hard. The honest headline is the pair rather than either number alone.

Which way the errors lean#

Three of the four errors in the sample were the judge passing something a human failed; one was the reverse. Small numbers, but the direction is consistent with what probe sets typically find — the judge likes answers that read as competent — and direction matters more than magnitude, because a symmetric error rate mostly adds noise while an asymmetric one bends the metric.

The difference between adding noise and bending is mechanical, and it decides whether a metric can still be used at all. Errors that run both ways in equal measure move some items out of the pass column and roughly the same number back into it, so the two movements cancel in the total and the reported rate stays near the true one even though individual verdicts have become unreliable. Errors that run predominantly one way have nothing to cancel against, so every one of them displaces the reported rate in the same direction, by about the size of the imbalance. A blurred metric is a noisy reading of the right quantity, and averaging over more items sharpens it. A bent one is a sharp reading of the wrong quantity, and averaging over more items makes it more confidently wrong.

Work through what a lenient judge reports as quality falls. Take a judge with 57% recall on failures and 98% specificity, which is to say it wrongly flags two good answers in every hundred, and let the true pass rate drop from 85% to 70%:

true pass 85%:  flagged = 0.15×0.57 + 0.85×0.02 = 0.086 + 0.017 = 0.103
                reported pass rate = 89.7%

true pass 70%:  flagged = 0.30×0.57 + 0.70×0.02 = 0.171 + 0.014 = 0.185
                reported pass rate = 81.5%

Both lines are the same two-term sum, and each term counts one of the two ways an item can end up flagged. An item is flagged if it is genuinely a failure and the judge catches it, which happens at the true failure rate times the judge's recall: on the first line 0.15 of items are true failures and 0.57 of those get caught, and the product is what the line rounds to 0.086. An item is also flagged if it is genuinely fine and the judge flags it anyway, which happens at the true pass rate times the false-flag rate: 0.85 of items are true passes and 0.02 of those get wrongly flagged, giving 0.017. The two groups cannot overlap, because an item either is a real failure or is not, so adding them double-counts nothing and the flagged fraction is 0.103.

The reported pass rate is then whatever was not flagged, because the pipeline reports a pass for exactly the items the judge did not flag. One minus 0.103 is the reported rate, and shifting the decimal point two places to read it as a percentage gives the 89.7% on the second line of the block. Running the same sum with the true pass rate at 0.70 gives 0.171 from the failure term and 0.014 from the false-flag term, a flagged fraction of 0.185, and a reported rate of 81.5%. Notice which term moved: the failure term roughly doubled because there were twice as many failures to catch, while the false-flag term barely shifted, so almost all of the reported movement came from the failures the judge does catch — and it catches only 57% of them.

A 15-point real drop shows up as an 8-point reported drop. The metric still moves in the right direction, compressed by roughly half, which is survivable if slow.

The worse case is when the regression's failure mode is one the judge is specifically bad at. Suppose a prompt change makes answers longer and more confidently phrased, and the new failures are the polished-and-wrong kind — recall on those is near zero rather than 57%, while the extra fluency also nudges the judge toward passing borderline items it used to flag. Now the flagged fraction can stay flat or fall as true quality drops, and the reported pass rate rises. That is the same mechanism that produces cheerful win rates during a decline in a chained pairwise series, arriving through the pointwise door instead of the pairwise one. A judge whose blind spot is correlated with the change you are shipping does not merely fail to detect the regression; it reports the regression as an improvement.

Where the threshold goes#

The 57% recall figure is not a fixed property of the judge. It is a property of the judge plus the rule that turns three criterion verdicts into a pass or a fail, and that rule is a dial. Moving it trades recall against precision and against how much human review the arrangement consumes:

Each row is the same judge reading the same three criterion verdicts, with a different threshold on how many of them have to fail before the item does, and the trade runs one way by construction. Loosening what counts as a failure — from "any criterion fails" down to "only invented navigation" — necessarily flags fewer items. Some of the items that stop being flagged were real failures, so recall falls. The flags that survive the stricter bar are the ones the judge was most confident about, so precision rises. No setting improves both, because both fractions are computed from the same set of flags, and shrinking that set moves one numerator relative to a fixed denominator and the other denominator relative to a fixed numerator.

The last column prices the same trade in somebody's time. At four minutes a review, 170 flagged items is a shade under seven hundred minutes, and dividing by the sixty minutes in an hour gives the 11.3. The middle row flags fewer than a third as many items and therefore costs fewer than a third as many hours, which is the 3.3, and the strictest row flags 21 for 1.4. That column is what turns an abstract preference for catching failures into a number somebody has to find room for in a week.

The middle row is what this judge has been running, and it was chosen as a rubric-design question with no numbers attached. The numbers in the table above change the character of the choice. The strict rule catches four in five real failures at the cost of flagging 170 items per release, half of which are fine. The lenient rule is nearly always right when it flags something and misses three failures in five.

Nothing in the data says which is correct, because that depends on the cost ratio, which is a product question. If a missed failure means a customer follows a fabricated menu path and files a ticket, and a false flag means four minutes of somebody's afternoon, the ratio favours recall by a wide margin and the strict rule is obviously right — until you notice that 11.3 hours per release is more review than the team has, at which point the rule gets loosened to fit the calendar rather than the cost ratio. Making that trade explicit is most of the value of computing the table. A recall of 57% chosen deliberately, because review capacity is three hours and that is the operating point three hours buys, is a defensible engineering decision. The same 57% arrived at because someone wrote if failures >= 2 while drafting the rubric is the same number with none of the defensibility, and it will never be revisited because nobody knows it is a choice.

Agreement with humans is not accuracy#

The humans are a standard, not ground truth. They carry their own rubric-application problems, they are frequently the same people who wrote the rubric, and on a text-to-SQL workload the row set returned by executing a query is a better arbiter than any rater. Where a deterministic check exists, agreement with humans is the wrong target and execution is the right one. Agreement is for the properties that genuinely need a reader, and reaching for it on properties that do not is how a suite ends up paying a judge to approximate a substring test.

The reason a deterministic check outranks agreement is not that machines read more carefully than people. It is that agreement compares one instrument against another instrument, and two instruments can be wrong in the same direction — especially when the raters are the people who wrote the rubric, since a rubric that is ambiguous in some particular way gets read that same ambiguous way by its authors and by a judge prompted with its text. Executing a query consults no rubric. It returns the rows or it does not, and two queries returning the same rows are the same answer for the purpose the queries exist to serve, whatever any reader thinks of how they are written.

None of this argues for throwing the judge away, and the arrangement that survives the numbers is not a smaller version of gating — it is a different shape of one. A judge with 57% recall cannot decide, and the same judge is very good at sorting a release into the part nobody needs to read and the part somebody does. Using it the second way asks a question it can answer — is this item near my decision boundary — rather than the one it cannot, which is whether this particular item is wrong. The number that makes that arrangement work is not the 92%; it is the size of the band and the recall inside it, and both have to be re-measured rather than assumed.

Six months of running an arrangement like this produced a judge-reported 91% pass rate, a blinded human review at 89% two months ago that matched it well — blinded meaning the raters saw the outputs without the judge's verdict, the release label, or each other's scores — and a fresh blinded review this week that came back at 74% while the judge still says 91%. Explaining a fifteen-point gap requires knowing whether the judge that produced this week's 91% is the same instrument that produced the one two months ago, and in most pipelines nothing recorded that.