Mid · 6 chapters
Judge failure modes and calibration
Position bias (and position-swapping as the fix), verbosity bias, self-preference, leniency drift; validating the judge against human labels
A judge is a prompt and rots like one — version it.
Read these first
What you will know
Demonstrate position bias, then fix it by swapping and re-running
Judges prefer whichever answer came first. Swap the order, run both, and average.
Run controlled probes for verbosity bias and self-preference, and quantify each effect
Hold task quality constant while changing answer length or author identity; a score shift exposes the unwanted preference.
Validate judge scores against human labels on a sample
Sample fifty examples, label them by hand, and check the judge agrees. If it does not, the metric is fiction.
Detect a judge that has drifted more lenient over time
Leniency drifts as the judge prompt is edited, so scores across months are not comparable unless it is versioned.
Version a judge prompt and treat a change to it as a change to the metric
A judge is a prompt. It rots, needs review, and belongs in version control like any other.
Chapters
- 1
Position bias in pairwise judging
A pairwise judge's verdict moves with the slot an answer sits in, so a harness that writes the incumbent into the first field every time measures its own template. This chapter defines position bias, gives the identical-answer check that exposes it, works through the both-orders procedure and what the second pass costs, and argues that the per-pair flip rate belongs beside every margin published. Also settled: why temperature 0 makes the bias invisible rather than absent, and the two things a swap cannot control for.
- 2
Probing verbosity bias and self-preference
Suspecting that a judge rewards length is not the same as measuring it. This chapter builds the controlled probes for the two preferences that travel inside an answer rather than beside it: a padded variant that adds words and no information, and a cross-family generation design anchored to human labels. It sets out what each probe holds fixed, what its number means, why a byline test measures something else entirely, and why measuring a bias is not correcting one.
- 3
Stratified sampling for judge validation
Fifty hand labels are what turn a judge score into a measurement, and how they are drawn decides what the number can say: taken at random from traffic where nearly everything passes, high agreement is mostly a constant predictor scoring well on undisputed cases. Covered here: the strata that concentrate the budget where the answer is in doubt, the chance-corrected statistic that turns 88% into a defensible figure once the base rate is subtracted, reading disagreements for direction rather than counting them, the human-human ceiling, and the four conditions before a disagreement becomes a few-shot example.
- 4
The frozen anchor set
A monthly pass rate collapses the product, the judge and the traffic into one scalar, and a rising line is consistent with any of them moving. This chapter builds the experiment that separates them: a fixed set of archived outputs, re-scored under today's judge, so that any movement on them belongs to the instrument. It works through what the paired comparison supports, how to standardise away a shifting input mix, what the whole thing costs to run, and where a frozen distribution stops being representative.
- 5
Versioning the judge prompt
A two-sentence edit titled 'clarify wording' moves a release gate from 84% to 90% overnight, because a judge prompt is the definition of the number a team steers by and a metric change in a markdown file gets reviewed as prose. This chapter reads a rubric diff as a specification, stamps an assembled-prompt fingerprint onto every score written, extends the same treatment to the harness settings that move verdicts without touching the rubric, and puts an anchor re-score on the pull request so the delta is reviewed rather than discovered.
- 6
Diagnosing judge drift
A re-score compares two instruments and returns their difference; the sign of that difference says which is more permissive, not which is right. This chapter lays out the five explanations that fit a large re-score gap equally well, shows what hand labels applied to both rubric texts contribute that no amount of re-scoring can, separates an escalation from the drift investigation it gets attached to, and sets out how to restate a series of past gate decisions without turning them into accusations.
External resources
- Self-Preference Bias in LLM-as-a-Judgearxiv.org
Measures how much a judge favours its own family's outputs.
- Lesson 2.4: Common judge failure modes | buildevalsbuildevals.com
This is a substantive, mechanics-focused explainer that catalogs judge failure modes, gives concrete detection and mitigation tests, and describes calibration and drift monitoring protocols.
- MM-JudgeBias: Evaluating Compositional Biases in MLLM-as-a-Judgearxiv.org
Quantifies position and verbosity bias magnitudes you should expect.
- What Is LLM-as-a-Judge Calibration? Power & Limitsdeepchecks.com
How to validate a judge against human labels and catch leniency drift.
- LLM-as-a-Judge: Bias Metrics in Evaluationemergentmind.com
Survey of judge bias metrics and agreement rates across many studies.
- A pragmatic guide to LLM evals for devsnewsletter.pragmaticengineer.com
Shows position, verbosity, self-preference, and leniency biases, plus validation against human labels.
- LLM-as-a-judge: a complete guide to using LLMs for evaluationsevidentlyai.com
Explains judge failure modes—position, verbosity, self-preference, leniency—and validates judges against human labels.
- Pop Goes the Stack | LLM-as-a-Judge: Bias, Preference Leakage, and Reliability | AI BiasF5, Inc.
Walks through self-preference, position and verbosity bias, and judge reliability
- LLM as a Judge: Scaling AI Evaluation StrategiesIBM Technology
Shows position, verbosity, self-preference, and leniency biases, plus position-swapping and human-label validation.