Judge bias in best-of-N

Best-of-N replaces counting with selecting, and the selector is a model with preferences. Those preferences are not neutral with respect to the thing being selected, because the generator and the judge share a great deal about what a good answer looks like: fluency, confident phrasing, canonical structure, hedges in the places a well-written document usually hedges. None of those is correlated with factual correctness in any dependable way. When the generator produces a wrong answer that is wrong because it followed a plausible-sounding pattern, the judge is being asked to detect exactly the pattern it also finds plausible.

That does not make judges useless, and the useful version of the claim is quantitative rather than rhetorical. A judge contributes something; the question is how much, over what baseline, on which cases, and at what price. All four are measurable in an afternoon on labelled data, and the measurements are the part that usually goes missing when a reranker ships.

Measuring a judge requires cases where it can be wrong#

Judge accuracy quoted over all traffic is close to meaningless, because on most cases all N candidates are correct and any selection rule scores 100%. The measurement that carries information is restricted to the discriminative subset: cases where at least one candidate is correct and at least one is not, so a choice can be graded. On one underwriting set that subset was 31% of cases, and it is where every point of the judge's value has to come from.

The reason an all-traffic figure carries so little is visible if the number is taken apart. Traffic splits three ways: cases where every candidate is correct, cases where none is, and the discriminative remainder. On the first group any selector scores full marks, because there is nothing to get wrong. On the second group every selector scores zero, because there is nothing to get right. Only the third group responds to how good the judge is. A headline accuracy is therefore the first group's share, plus the third group's share multiplied by the judge's real skill — which means that with a discriminative subset of 31%, roughly two thirds of the published number was fixed before the judge read anything.

def judge_report(cases, candidates, correct_fn, judge_pick) -> dict:
    """Score a selector only where the choice could have gone either way."""
    hits = trials = 0
    oracle_available = random_baseline = 0.0
    for case in cases:
        cands = candidates[case.id]
        flags = [correct_fn(case, c) for c in cands]
        if not (any(flags) and not all(flags)):
            continue                                  # nothing to discriminate
        trials += 1
        hits += flags[judge_pick(case, cands)]
        oracle_available += 1
        random_baseline += sum(flags) / len(flags)
    return {
        "discriminative_cases": trials,
        "judge_accuracy": hits / trials,
        "random_pick_baseline": random_baseline / trials,
        "oracle": oracle_available / trials,
    }

The random_pick_baseline line is the one most reports omit and the one that decides whether the judge is doing anything. If three of five candidates are correct on a typical discriminative case, picking uniformly at random scores 60%, so a judge scoring 64% has contributed four points, not sixty-four.

That baseline is one division, and the code performs it per case: sum(flags) / len(flags) is the number of correct candidates over the number of candidates. It is the right quantity because a uniform random pick lands on each candidate equally often, so its chance of landing on a correct one is exactly the share of candidates that are correct. Three of five correct gives three fifths, or 60%, and averaging that fraction across the discriminative cases gives the baseline the judge has to beat. Subtracting the baseline from the judge's score is then the lift, and on the illustrative case above that subtraction leaves four points.

The oracle line is worth reading the same way, because it is not a measurement. The loop adds one to oracle_available for every case it keeps, and it keeps only cases where at least one candidate is correct, so the oracle score is one by construction. It is not the best any model could do. It is the arithmetic ceiling that follows from how the subset was defined, and it exists in the report to make the gap between the judge and that ceiling readable.

Building the subset is cheaper than labelling a fresh eval, because the labels you need are per-candidate rather than per-case: for each stored fan-out, mark which of the N candidates are acceptable. Cases where all N are acceptable and cases where none are get discarded from the measurement, which usually removes most of them, so the annotation concentrates on the minority of cases that can discriminate. The discarded majority is not wasted either — the share of cases where no candidate was acceptable is the coverage ceiling, and it caps the judge and every future selector alongside it.

Sixty-four percent against a fifty-eight percent baseline, at four tenths of a cent per request, is a small effect bought at a real price. The oracle in the last row is not a selector anyone can build; it is the score a selector would get if it always chose a correct candidate whenever one was present, and it is there to mark the ceiling every row above it is failing to reach.

The measured lift is that subtraction on the first and fourth rows: 63.9% less the 58.1% a random pick achieves, which is just under six points rather than the four the illustrative case gave, because the measured baseline sits a little lower than the three-in-five the illustration assumed. Two rows down from the baseline is the reading that does the most damage. Picking the longest candidate involves no model, no rubric and no money, and it scores 60.4%; the same-family judge with a four-round rubric scores 63.9%. The rubric is buying about three and a half points over a rule that counts characters.

The last row is where the arithmetic turns. It scores 79.3% against the same-family judge's 63.9%, a gain of more than fifteen points, and it costs $0.0044 against $0.0041 — a difference of three hundredths of a cent per request. Nothing about the model changed between those two rows. What changed is that some candidates were eliminated by something that is not a model at all before the judge was asked to choose.

Two ways to see the correlation#

Correlated preferences leave traces that need no labelled dataset to read. Both of the checks below describe the judge's behaviour rather than its accuracy — what it reaches for, and whether it reaches for the same thing when the list is shuffled — which is why both can be run against production traffic with no gold answers anywhere in sight.

The first is what the chosen candidates look like. The same-family judge picked candidates averaging 214 tokens against 176 for the ones it rejected, and among candidates that hedged explicitly — "the income evidence appears insufficient, though the payslips may be partial" — it selected them 18% of the time even though hedged candidates were correct 44% of the time. It was systematically discarding accurate, appropriately uncertain answers in favour of confident ones. That measurement needs nothing but the candidates and the judge's picks; it does not need labels, which makes it the cheapest bias check available.

Length on its own proves nothing, and this is where the reading is tempting and incomplete. Longer answers could simply be better answers, and a judge preferring them would then be doing its job. The table settles that objection rather than the token counts doing it: picking the longest candidate every time scores 60.4% on the discriminative subset against a random pick's 58.1%, so length is worth barely more than nothing as a correctness signal on this task. A judge whose picks run about a fifth longer than its rejects is therefore tracking something that is very nearly uninformative, which is what makes the token profile evidence rather than trivia.

The hedging half of that same measurement needs one quantity nobody has supplied yet, and a reviewer should ask for it before repeating the conclusion. Knowing that the judge selected hedged candidates 18% of the time says nothing on its own until it is set against how often hedged candidates appeared at all: if hedged answers were rare in the candidate pool, an 18% selection rate could be neutral or even generous. The claim of systematic discarding holds when the share of candidates that hedge is meaningfully above the share the judge picks, and that share is the number to demand. The 44% correctness rate is doing the second half of the argument — it establishes that hedged answers were worth having — but it cannot do the first half.

The second is the position-swap flip rate in the last column of the table. Present the same candidates in reversed order and see how often the judge's choice changes. A judge reading only the content would flip near zero; the same-family judge flipped on 22% of cases, which means roughly a fifth of its decisions were determined by list position rather than by the candidates. That is cheap to measure and cheap to partially fix — randomise the order per request so the bias does not align with anything systematic, or run both orders and treat a flip as an abstention — and the flip rate is a useful ongoing health metric, since it moves when the model version changes.

That reading of the flip rate is a floor rather than an exact count, and the difference matters if the number is going into a report. A decision made purely on content survives a reversal, so it never contributes a flip. A decision made purely on position always flips. In between sits a decision that position merely nudged, which may survive the reversal and be recorded as stable. Flips therefore under-count position influence, and the honest statement is that at least a fifth of the same-family judge's decisions were position-driven — with the true share somewhere above that and no way to pin it down from this measurement alone.

Rubrics that make the judge look before it scores#

JUDGE_PROMPT = """You are checking five candidate explanations against an application file.

For each candidate, answer three questions before scoring it:
1. Which specific facts in the file does it cite? Quote them.
2. Is every quoted fact actually present in the file? Mark any that is not.
3. Does the cited rule match the decision recorded in the file?

Then score each candidate 0-3 on factual support only. Ignore fluency,
length and tone. Return JSON: [{"id": .., "quotes": [..], "unsupported": [..],
"score": ..}]"""

Rewriting the rubric to make the judge extract quotable evidence before scoring, and to score every candidate independently rather than ranking them against each other, moved discriminative accuracy from 63.9% to 68.2% and dropped the flip rate to 15%. Independent scoring matters more than the wording: when a judge ranks a list, its answer for one candidate depends on the others, which is what makes position effects possible in the first place. Scoring one at a time costs N judge calls instead of one, so it is a real trade, and the middle path — one call, but a per-candidate structured score with evidence quoted — captures most of the benefit at the original price.

The chain behind the position effect is short enough to follow all the way down, and following it explains why independent scoring is the part that matters. Ranking asks a single question whose answer is a whole ordering, so every candidate's placement is computed in the presence of the others. A model producing that ordering is generating text conditioned on the list as it was written down, and the list has a first item and a last item. Those slots are features of the input in exactly the way the candidates' words are, so a preference attached to them is available for the model to act on. Score each candidate against the rubric on its own and the other candidates are no longer in the context, so there is no slot for the preference to attach to. The channel closes rather than the bias being resisted.

The two figures in that rubric change are one subtraction each: 68.2% less 63.9% is the 4.3 points, and 22% less 15% is the seven points of flip rate. The second number is the one to watch, because a falling flip rate is evidence that the accuracy gain came from the mechanism claimed rather than from the wording happening to suit this eval. An accuracy gain with an unchanged flip rate would be the same size of number carrying a much weaker argument.

Consistency is not accuracy#

The metric most teams reach for when they want to trust a judge is agreement, and it does not support the weight put on it. Run the same judge twice on the same five candidates at temperature 0 and it agrees with itself 91% of the time. Run two judges from the same family with the same rubric and they agree 84% of the time. Both numbers look like evidence of a well-behaved evaluator, and neither one touches correctness, because a judge with a stable preference for confident prose will express that preference consistently, run after run, in perfect agreement with itself, while getting the same third of cases wrong every time.

No route leads from either of those numbers to correctness, and the reason is structural rather than statistical. Computing an agreement rate means comparing two selections against each other. The correct answer appears nowhere in that computation at any point. A quantity that never touches the labels cannot report how often the labels were matched, however high it climbs — and it climbs highest exactly when a preference is strongest, which is the condition a stable fluency bias produces.

The distinction is the ordinary one between a reliable instrument and a valid one. A scale that reads 400 grams whatever you put on it is perfectly repeatable. The only measurement that establishes validity is agreement with something outside the system — human labels on the discriminative subset, an executable check, a downstream outcome — and that measurement costs labelling effort, which is why agreement gets reported instead. When a judge is described as "highly consistent", the useful follow-up is which of the two properties was measured.

The stuck scale is an analogy, and its limit is worth naming because the limit is where the practical difficulty sits. A stuck scale is trivial to expose: put one known weight on it and the fault is visible in a second. A judge's bias does not misread every input. It agrees with the truth on the cases where fluency and correctness happen to point the same way, which is most of them, and departs from it on the cases where they diverge — which are the discriminative cases, and the only ones anybody bought a judge for. There is no single known weight that exposes it, which is why the detection method has to be a labelled subset rather than a probe.

Inter-family agreement is a partial exception worth noting: when a judge from a different family agrees with a same-family judge, the agreement is more informative than self-agreement, because the two selectors share less. That is the reasoning behind using a different family as the judge at all, and the 7.7-point jump in the table is the size of the effect on this task. It is an improvement in the correlation rather than a removal of it, since two strong instruction-tuned models — models put through a post-training stage that teaches them to follow written instructions and to produce answers people rate well — still share a great deal about what a good answer looks like.

That jump is the difference between two rows of the table: 71.6% for the different-family judge against 63.9% for the same-family one, and 71.6 − 63.9 = 7.7. The flip rate moves the same way across the same pair, from 22% down to 14%, and the corroboration is the useful part. A selector sharing less with the generator turns out to be both more accurate on the discriminative subset and less swayed by where a candidate sits in the list, which is what a reduction in shared preferences should look like if that is really what happened.

Where the bias bites is predictable from the task#

The severity tracks how much of the correctness is visible on the surface of the text. A judge asked to pick the better of two Python functions can be given the test suite output, and correctness becomes an observable; a judge asked to pick the more faithful summary of a 40-page report has to hold the report and the summary in mind simultaneously and check each claim, and it will approximate that job with a fluency assessment. Between those extremes sit the cases that cause the most trouble in production: claims that cite a source, arithmetic embedded in prose, and any output whose errors are plausible-sounding substitutions rather than obvious breakages. Those are precisely the outputs teams reach for best-of-N to improve, which is why the technique so often measures well on an eval built from clearly-good and clearly-bad candidates and then underperforms on real traffic, where the wrong candidates are wrong in the same well-written way the right ones are right.

Being predictable from the task rather than discoverable only after deployment is what makes the exposure worth estimating in advance, and the chain that makes it predictable is short. A judge can score only what is in front of it. A claim whose correctness depends on a document the judge was never given therefore cannot be scored on correctness. What is left for it to score is how the claim reads. And how a claim reads is precisely the property the generator was optimising when it wrote the claim, which is why the judge's residual criterion lines up with the generator's. Every link there can be checked against a specific system before any reranking code exists, by asking what a judge would need in its context to verify the output and whether it is being handed that.

The property that makes a deterministic check valuable is that its errors are uncorrelated with the generator's. It does not find confident prose persuasive, it has access to the record rather than to a summary of it, and when it says a figure does not match, that is a fact rather than a preference. Any signal with those properties will outperform a same-family judge in the region where it applies: unit tests for generated code, a schema validator, a database lookup, a recomputation of stated arithmetic, a search that either finds the cited passage or does not.

The corresponding limitation is coverage. A check that settles two claims cannot select among candidates that all pass it, and a decline paragraph making five claims leaves three unchecked. That is why the winning row in the table is a check and a judge rather than a check alone — the check eliminates, the judge selects among survivors, and its job has shrunk to the part that genuinely requires reading.

What a judge call costs as a share of the pipeline is routinely misread, and the misreading comes from counting the wrong unit. Counting calls, a five-way fan-out plus one judge is six calls, so the judge looks like a sixth of the work. Billing follows tokens rather than calls, and a judge's prompt has to carry the original input, the rubric and all five candidates at once, which makes it several times the length of any single generation call. Counted in tokens the judge sits nearer a quarter of the spend than a sixth, and the gap widens as N rises.

Best-of-N with a judge remains the right tool when the output is free-form and no verifier exists, which is a large and common situation. Be clear about what you are buying: a few points on the discriminative subset, a selector whose preferences you should measure rather than assume, and a call whose cost is a quarter of the pipeline rather than a sixth. Be equally clear about what you are not buying, which is an independent opinion — and about how to tell, which is the discriminative-subset accuracy against a random-pick baseline, the token-length and hedging profile of the picks, and the flip rate under a reversed candidate order. Three measurements, no new infrastructure, and they are the difference between a reranker that is known to contribute four points and one that is believed to contribute sixty-four.

That leads to the other job people give these samples. Once you have five answers and a measure of how much they agree, the agreement looks like a confidence score, and a confidence score looks like something you can route on.