Position bias in pairwise judging

A judge is a prompt, and like every prompt it carries preferences that have nothing to do with the thing it was pointed at. This topic is about finding those preferences, putting a number on each one, and keeping that number meaningful across the months in which the judge itself is edited. The first of them is the cheapest to measure and the one most likely to invert a conclusion somebody has already acted on.

Two moving parts need naming before anything is argued from them. A pairwise judge is a large language model that receives a task, a rubric, and two candidate answers at once, and returns which of the two better satisfies the rubric. It exists because absolute scoring is hard to hold steady — asked whether one reply on its own is a 3 or a 4 out of 5, a model answers differently across days and batches, while the relative question stays anchored by the two texts sitting side by side — and because the decision a team actually faces is usually itself a comparison: keep the current prompt, or ship the new one. You meet pairwise judging wherever two versions are weighed against each other without a human reading every output, which covers one prompt against another, one model against another, two retrieval configurations, and the public arena leaderboards that rank chat models on which of two responses a rater preferred.

Position bias is the failure mode a fixed presentation order invites: the judge's verdict moves with the slot an answer occupies rather than with what the answer says, and in most layouts that means a standing preference for whichever candidate it read first. It arises because the judge reads one linear sequence of tokens in which one reply necessarily arrives before the other, so the two slots are not symmetric conditions in an experiment. Any harness that loops over pairs and writes the incumbent into the first slot every time — which is the natural way to write the loop — carries the effect into every margin it reports. Order is the first thing to check about any pairwise judge, because it is cheap to measure, common enough to assume by default, and large enough to invert conclusions. Topic 3.10, on writing the judge prompt itself, covered how to ask a well-posed question; a well-posed question asked in a biased frame still produces a biased answer, and the frame here is the order the candidates arrive in.

What the first answer does to the second#

The asymmetry of those two slots works on the judge in a specific way. The first reply arrives with nothing to compare against, so the model builds its reading of what a good answer to this task looks like while processing it; by the time the second reply arrives, there is already a standard in context, and the second reply is read as a departure from that standard rather than as an independent candidate. Departures are easy to notice and easy to score against, which pushes verdicts toward the first answer in most formats.

That asymmetry is not a quirk of the prompt; it follows from what a model is given. A prompt is one sequence of tokens, processed in order, and the internal state the model has built at any position depends on every token before that position and none after it. So when the tokens of the first reply are being processed, the only material available to interpret them against is the ticket and the rubric, and the reading that gets built is a reading of what an answer to this ticket looks like. When the tokens of the second reply are processed, the first reply is already part of the state, and the second is interpreted against it. Two things that a designer intended to be symmetric conditions in an experiment are therefore not symmetric at all, and no wording in the rubric can make them so, because the asymmetry is in the order of the bytes rather than in the instructions. This is why the fix later in this chapter is to run the experiment twice rather than to write a better prompt.

Several smaller effects stack on top of that. Labels carry priors: models have seen enormous quantities of multiple-choice text in which the first option is correct more often than chance, and a judge asked to output A or B inherits some of that. Distance to the verdict matters too, because the second reply sits closer in the context to the tokens where the judge commits to an answer, which in some formats pulls the other way and produces a recency bias instead. Which direction you get depends on the prompt layout, the length of the answers, and the model, which is precisely why you measure it on your own harness instead of reading a direction off a paper. Reported magnitudes vary widely by setup; MM-JudgeBias puts the position and verbosity effects on a common scale across several judge models and is worth reading for a sense of the range you should expect rather than for a number to copy.

The important structural point is that the effect is a property of the pair of positions rather than of the pair of answers. Nothing about a candidate makes it win more often; being first does. A judge with a strong enough first-position preference would hand a lopsided win rate to two identical answers, which is a sanity check worth running once — feed the judge the same answer twice, in both slots, on a hundred inputs, and see how far from a clean sweep of ties you land.

On a support-reply harness of the kind this chapter works with, that check returned 58 first-slot wins, 21 second-slot wins and 21 ties. Those three counts are the whole of a hundred comparisons, so the judge declared a winner between two byte-identical texts on 58 plus 21 of them, which is 79% of the time. That figure is worth sitting with, because whatever it is doing on those comparisons has nothing to do with content: the two texts are the same text, so the only difference between the thing that won and the thing that lost is which field it was pasted into.

Read the split rather than the total and it says two separate things. That any winner was declared at all — on nearly four comparisons in five — measures how reluctant the judge is to say the two are equivalent. That the winners are not evenly divided between the slots, 58 against 21, measures the direction and size of the positional preference: with identical texts an unbiased judge that insisted on picking would produce something close to half and half. A judge that will not call a tie between an answer and itself is not going to call one between two answers that differ only in phrasing, and that reframes the tie column in every pairwise report you have read: a low tie rate is usually taken as evidence of a decisive judge, and it is equally consistent with a judge that has been given no comfortable way to express indifference.

Ties deserve a decision of their own rather than a default. Dropping them from the denominator inflates whatever margin survives; counting each as half a win treats indifference as an observation about the candidates; and forbidding them in the output schema, which is the most common choice, converts every genuine indifference into a coin flip that the position bias then decides. Report the three-way split — A, B, tie — alongside the flip rate, and let whoever reads the number see how much of the population the comparison actually resolved.

Running each comparison twice#

The fix is mechanical. Every comparison runs twice, once in each order, and the reported number is the average of the two. Position is now a controlled variable rather than a confound, in the same way that you would not benchmark two servers by always running one of them first on a cold cache.

import json
from anthropic import Anthropic

client = Anthropic()

VERDICT_SCHEMA = {
    "type": "object",
    "properties": {
        "winner": {"type": "string", "enum": ["first", "second", "tie"]},
        "reason": {"type": "string"},
    },
    "required": ["winner", "reason"],
    "additionalProperties": False,
}


def judge(rubric: str, ticket: str, first: str, second: str) -> str:
    """Return 'first', 'second' or 'tie' — positional, not identity-based."""
    response = client.messages.create(
        model="claude-opus-5",
        max_tokens=8192,
        thinking={"type": "adaptive"},
        system=rubric,
        output_config={"format": {"type": "json_schema", "schema": VERDICT_SCHEMA}},
        messages=[{
            "role": "user",
            "content": (
                f"<ticket>{ticket}</ticket>\n"
                f"<answer_1>{first}</answer_1>\n"
                f"<answer_2>{second}</answer_2>\n"
                "Which answer better satisfies the rubric?"
            ),
        }],
    )
    return json.loads(response.content[-1].text)["winner"]


def compare(rubric: str, ticket: str, answer_a: str, answer_b: str) -> tuple[str, str]:
    """Both orders. Returns (verdict when A is first, verdict when B is first)."""
    a_first = judge(rubric, ticket, answer_a, answer_b)
    b_first = judge(rubric, ticket, answer_b, answer_a)
    as_read = {"first": "A", "second": "B", "tie": "tie"}
    as_swapped = {"first": "B", "second": "A", "tie": "tie"}
    return as_read[a_first], as_swapped[b_first]

Two details in that snippet are load-bearing. The judge is asked to name a position rather than a candidate, so that nothing in the prompt ever says which reply belongs to which prompt version — the mapping back to A and B happens in the harness. And the swap is a genuine swap of the answer texts inside identical scaffolding, so the two runs differ in exactly one thing.

The cost is that you now make two judge calls per comparison instead of one. For a set of 400 pairs that is 800 calls at roughly 1,600 input and 400 output tokens each, so about 1.3M input and 320k output tokens — 800 times 1,600 is a little over a million and a quarter, and 800 times 400 is 320 thousand exactly; at the published rates for the model in that snippet, somewhere near $14 for the full swapped run. The input side is cheaper than that arithmetic suggests, because the rubric and instructions sit in the system block ahead of everything that varies, so a cache breakpoint at the end of it turns most of those input tokens into cache reads at roughly a tenth of the price across all 800 calls. What the second pass really costs is output tokens and wall-clock, and at twenty concurrent requests the whole thing finishes over a coffee break.

The flip rate describes the harness#

Averaging gives you a less wrong margin. It does not tell you how much of the original margin was ever real, and that is a separate question with its own answer. Cross-tabulate the two runs by pair rather than summing them:

The four cells add to 400, which is the check to run first on any table of this shape. The two on the leading diagonal, 154 and 138, are pairs where the judge said the same thing in both orders, and they come to 292. The two off it, 90 and 18, are pairs where it did not, and they come to 108. So of 400 pairs, 292 produced the same verdict under both orders and 108 did not, a flip rate of 27%.

Reading the off-diagonal cells separately is where the diagnosis is. Those 108 pairs are ones on which the judge has no stable opinion at all; its verdict there is determined by the slot the answer landed in. Of them, 90 resolve toward whichever answer was shown first and 18 toward whichever was shown second, so the positional preference on the unstable pairs runs five to one in favour of the first slot — which is where a first-slot advantage in the headline comes from. An average taken over that population is an average in which more than a quarter of the members are coin flips with a weighted coin.

from collections import Counter


def summarise(pairs: list[tuple[str, str]]) -> dict[str, float]:
    counts = Counter(pairs)
    n = sum(counts.values())
    consistent_a = counts[("A", "A")]
    consistent_b = counts[("B", "B")]
    prefers_first = counts[("A", "B")]
    prefers_second = counts[("B", "A")]
    consistent = consistent_a + consistent_b
    return {
        "a_win_rate_averaged": (2 * consistent_a + prefers_first + prefers_second) / (2 * n),
        "flip_rate": (prefers_first + prefers_second) / n,
        "consistent_pairs": consistent,
        "a_win_rate_among_consistent": consistent_a / consistent,
    }

# {'a_win_rate_averaged': 0.52, 'flip_rate': 0.27,
#  'consistent_pairs': 292, 'a_win_rate_among_consistent': 0.527}

The averaged win rate in that dictionary is worth deriving by hand once, because the formula in the code looks arbitrary and is not. Every pair was judged twice, so 400 pairs produced 800 verdicts, which is the denominator. A pair where A won in both orders contributes two A wins, which is why the consistent-A count is doubled; a pair that flipped contributes exactly one A win whichever way it flipped, since one of the two runs went to A and the other to B. So A's total is twice 154, plus 90, plus 18 — four hundred and sixteen A wins out of 800 verdicts, which is 52%. The scenario below arrives at the same 52% by a different route, averaging a 61% run with a 43% run, and the two agreeing is a useful check that the cross-tabulation and the headline describe the same experiment.

Restricting attention to the 292 pairs where the judge holds the same view under both orders, A wins 154 of them, which is 52.7%, with a standard error of 2.9 points. That standard error comes from the usual three steps: multiply the win rate by the loss rate, divide by the 292 pairs, take the square root. Two of those standard errors either side of 52.7% comfortably covers an even split, so the honest reading of the whole experiment is that on the pairs where the judge has an opinion, the two prompts are indistinguishable at this sample size, and on 27% of pairs the judge has no opinion. Both halves of that sentence are actionable. The first says stop arguing about which prompt is better and go find a comparison the judge can actually resolve. The second says the rubric is underspecified for a quarter of your traffic, and reading twenty of those flipped pairs by hand will usually show you exactly which criterion is doing no work.

Report the flip rate next to every pairwise margin you publish. A 52% average with a 4% flip rate and a 52% average with a 27% flip rate are the same headline over completely different underlying facts, and only one of them is worth a follow-up experiment.

Determinism is not neutrality#

There is a tempting argument that a judge running at temperature 0 cannot have position bias, because a deterministic function has no room for the randomness that bias implies. The argument confuses reproducibility with correctness. A deterministic judge that prefers the first answer prefers it on every call, forever, with no variance at all. Run the same comparison ten times and you get the same verdict ten times, which is exactly what the check was looking for and exactly what a perfectly stable systematic error looks like.

Following what temperature does to the numbers is what makes that obvious rather than merely plausible. The model computes a score for every candidate next token; temperature divides those scores by a constant before they are turned into probabilities and one is drawn. A high temperature flattens the differences between candidates, and a temperature of zero removes the draw entirely and takes whichever candidate scored highest. In every case the ranking of the candidates is untouched, because dividing a set of numbers by the same positive constant cannot reorder them. Position bias lives in that ranking: it is the fact that first outscores second when the two answers are identical. Turning temperature down does not narrow the gap between those two scores, it makes the higher one win every single time. The setting controls whether the judge is repeatable, and the bias is a property of what it repeats.

Determinism makes a bias harder to notice rather than less likely to exist. Sampling noise at least produces disagreement between repeat runs, and disagreement is a signal somebody eventually investigates. A stable bias produces clean, repeatable, confidently wrong numbers that pass every consistency check anyone thinks to run. The only check that detects it is the one that varies the suspected cause, which here means varying the order, and that check is unaffected by temperature.

It is worth adding that on several current model versions the sampling parameters have been removed from the API altogether — temperature, top_p and top_k are rejected outright — so "we set temperature to 0" is not even a configuration you can point at any more. Whatever reassurance the setting used to provide was never about bias in the first place.

Length and rubric correctness travel with the answer#

Swapping controls for one variable. If one prompt's answers average 190 words and the other's average 340, the swapped-and-averaged margin still contains whatever the judge gives to length, and no amount of reordering will remove it, because length travels with the answer into both slots. The same holds for anything else that is a property of the answer rather than of its position: formatting, tone, apparent confidence, and which model wrote it.

Swapping also does nothing about the correctness of the rubric. A judge that faithfully applies a criterion nobody agrees with will apply it identically in both orders, produce a 0% flip rate, and give you a stable, position-neutral measurement of the wrong thing. Order control is a prerequisite for the number meaning anything; it is not evidence that the number means what you want.

Position bias is the one judge failure mode you can remove rather than merely account for, because position is the one property of a comparison that lives entirely in your harness. Two calls instead of one, the answers named by slot rather than by identity, and the two verdicts averaged — that is the whole control, and it costs a second pass that finishes over a coffee break. What it buys is that a margin you publish is a fact about the candidates rather than a fact about which field the harness happened to fill first.

The flip rate is the part that people leave out, and it carries more information than the margin does. A margin averaged over a population where a quarter of the pairs flip is an average over coin tosses, and knowing that is what tells you whether the next experiment should be a bigger sample or a better rubric. Both numbers belong in the report. Neither of them says anything about the properties that ride along inside the answers themselves — length, style, authorship — which cannot be swapped and so need probes built on a different principle.