Position bias and tie policy

Running each pairwise comparison in both presentation orders leaves two verdicts per item rather than one, and those two verdicts do not always agree. A split — a pair whose winner reverses when the answers change slots — is the swap protocol working as designed: it is the mechanism by which a judge's position preference becomes visible instead of riding silently inside a headline number. What the protocol does not decide is what a split is worth.

That decision is called the tie policy, and it is a piece of accounting with no default. Half credit to each side, exclusion from the denominator, no credit to the challenger, and three rates reported separately are all defensible readings of the same verdicts, and they produce different numbers from identical data. A related and separate decision is whether the judge is offered a tie option at all, which changes what the judge does rather than what you do with what it returns. The cheaper alternatives to swapping give up more than they appear to.

What a flipped pair actually is#

An item where the judge picks whichever answer appears in the first slot is an item where the judge could not tell the two answers apart on the rubric's terms. Position preference is a tiebreaker the model reaches for when the criteria do not decide the question — that is precisely why running both orders exposes it. A set of flips is not a set of failed measurements to be repaired; it is a set of successful measurements whose result is "these two answers are equivalent under this rubric."

Why the slot should make any difference at all is worth working out rather than assuming, because the answer is what licenses reading a flip as equivalence. A judge is a language model, and a language model produces its verdict by continuing the text it was handed: the rubric, the question, the two candidate answers, and whatever wording asks for a decision. The two answers do not arrive as two arguments to a function. They are stretches of text at particular places in one string, and the model's score for every candidate continuation is conditioned on the whole of that string.

Swapping the answers therefore does not present the same input twice. It presents two different strings, and two different strings are entitled to produce two different distributions over the possible verdicts. Nothing in the arrangement guarantees otherwise, and there is no step at which the model holds the two answers up against each other as objects and then renders the comparison into words — the words are the whole computation. Once the two orders are seen as two different inputs rather than one input viewed twice, a disagreement between them stops looking like a malfunction and becomes the ordinary consequence of a function being evaluated at two points.

Which slot a given judge favours is a property of that judge and that prompt rather than a law, and the swap does not need to know the direction in advance, because a disagreement is symmetric. What the protocol rests on is the weaker and more robust claim: that the slot carries information it has no business carrying, and that the size of that effect can be read off the rate at which reversing the slots reverses the verdict.

The reason a disagreement can be attributed to the slot at all is that the two runs differ in exactly one respect. Same judge, same rubric, same question, same two answers, same wording around them; the only thing that moved is which answer went where. When one variable is the only thing that changed and the output changed, the output changed because of that variable, and that is what makes the swap a measurement rather than two opinions. It also carries a condition worth attaching, because it is the one that gets dropped: a judge whose picking step is left free to sample can disagree with itself in a fixed order too, so part of every measured flip would then be the sampler rather than the slot. The protocol is clean only when the verdict is taken deterministically, and pinning that is a prerequisite rather than a refinement.

The step that is easiest to get wrong here is the one that feels most obviously right. Running both orders looks like a controlled experiment, and a controlled experiment is the standard way of removing a nuisance variable, so the natural conclusion is that the swap controls for position bias — that having measured each answer in each slot, the effect has been cancelled and what is left is the judgement. The reading is tempting because the design really is symmetric: every answer occupies every slot exactly once, and there is no residual asymmetry anywhere in the protocol to point at.

What the swap actually does is narrower. On a pair whose two verdicts agree, position did not decide the outcome and there was nothing to cancel. On a pair whose verdicts disagree, position did decide the outcome, and the swap has not decided it instead — it has produced two contradictory answers and no rule for choosing between them. The protocol converts an undetected bias into a detected disagreement, which is a measurement rather than a correction. A pair that flips has not been resolved by the swap; it has been revealed as a tie, and what a tie is worth is a separate decision that the arithmetic below is entirely about.

Reading them as equivalence rather than as noise settles most of the policy question. Here is the whole spread, on a 300-item comparison in which 241 pairs agreed with themselves across the swap — 148 for system B, 93 for system A — and 59 did not:

Those four rows are one set of verdicts scored under four rules, and working the arithmetic through in order makes their ordering predictable rather than surprising. Start with what the figures are made of. The 241 self-consistent pairs are the 148 that went to B plus the 93 that went to A, and the 59 splits are what is left when those 241 come out of 300. B's share of the pairs that came back decided is 148 out of 241, a shade over three-fifths, and that is the 61.4% in the second row.

Every other row is that same figure diluted by whatever credit the policy hands the splits. Half credit gives each split half a win, so 59 of them contribute 29.5 wins — half of 59, which is where the half in that numerator comes from — and the numerator becomes 148 plus 29.5 while the denominator returns to the full 300. Read as a weighted average, that is B winning three fifths of the decided pairs and exactly half of the split ones, and mixing a rate of one half into a rate above one half pulls the result down. It lands at 59.2%. Awarding the splits no credit at all is the same weighted average with the splits entered at nothing rather than at a half, which pulls further in the same direction, and it lands at 49.3%.

Excluding the splits is the third position on that one dial, and it is the position that reads as neutral. Removing them from the denominator does not score them at nothing or at a half; it scores them, silently, at whatever rate the remaining items produced, because an average over a set with part of the set deleted is simply the average of what is left. So exclusion returns the undiluted 61.4%, and it is the highest of the three for a reason that has nothing to do with this particular data: whenever the leading system is ahead among the decided pairs, the splits are sitting below its decided rate, and taking them out can only move the number up.

One of the four readings says B wins comfortably, one says B loses, and the sentence they go into — "B wins X% of head-to-head comparisons" — is the same sentence. The spread is the distance between the highest row and the lowest, which is exclusion against no-credit, and it is fixed entirely by how large the split bucket is relative to the set. Twelve points of spread is a large number to be produced by a choice nobody has made yet.

Excluding splits from the denominator is the option that quietly does the most damage, because the flips are not randomly distributed across the item set — they concentrate on the items where the two systems are genuinely close. Dropping them removes exactly the hard cases and reports the win rate over the easy ones, which is why exclusion produced the highest headline in the table. On this data it inflates the margin by 2.2 points, which is the second row of the table less the first. On a comparison between two systems that differ only in a small prompt edit, where most items are near-ties, the same policy can turn a 51% result into a 70% one.

The size of that inflation is not a fixed penalty, and the arithmetic says which way it moves. What exclusion removes is the drag that the split bucket exerts on the average, and the size of a drag depends on both how far the dropped items sit from the rest and how many of them there are. Here the splits sit about a tenth of a rate below the decided pairs and make up under a fifth of the set, so the drag is small and removing it moves the headline by a couple of points. Between two systems that barely differ, the same two quantities both grow: the split bucket swells because more items are indeterminate, and the decided pairs are the minority on which one system happened to look better. Excluding the splits then reports that minority as though it were the whole comparison, which is how the same policy walks a result that is barely above even into one that looks decisive.

Counting a split as half a win to each side keeps the denominator honest and lets equivalence pull the headline toward 50%, which is what equivalence should do. It is the least distorting of the three collapsing policies. But every collapsing policy throws away the fact that a fifth of the comparisons were indeterminate, and that fact is worth as much as the win rate itself. Reporting the three rates separately preserves it, at the cost of no longer having a single number to put in a Slack message.

Whether to offer the judge a tie option at all#

A separate decision, and one that changes the judge's behaviour rather than the arithmetic: a schema that forces a choice between first and second gives the judge nowhere to record indifference. Adding tie to that enum lets it decline.

Re-running the same 300 items with three options produced a different distribution. The judge returned tie in both orders on 88 items, a consistent winner in both orders on 152 items — B 118, A 34 — and a mixed result on the remaining 60, where one order gave a winner and the other gave a tie or the opposite winner. Excluding declared ties and mixed results, B's win rate is 118 of 152, or 77.6%.

That is the same underlying pair of systems reporting a headline of 77.6% instead of 59.2%, and neither number is a mistake. Offering an explicit tie changes what the judge does with a marginal item: instead of guessing and producing a flip that the swap would catch, it declares indifference, which concentrates the remaining verdicts on the items where the difference is large. The win rate over confidently-decided items is genuinely higher than the win rate over all items, and reporting the former as though it were the latter is the error.

Setting the two runs side by side as two ways of deciding which items count makes the size of that gap arithmetic rather than atmosphere. Under forced choice, 241 of the 300 came back decided and 59 did not. Under the tie-allowed schema the declared ties and the mixed results together account for 88 plus 60 items, leaving 152 decided where forced choice had left 241. B's share of that smaller set has risen from 148 in 241 to 118 in 152 — from a shade over three-fifths to a shade over three-quarters.

Nothing about the two systems changed between those runs, so the rise has to come from which items left. The items that leave are the ones the judge declined, and a judge declines the items it cannot separate; those are the items that would otherwise have divided close to evenly between the two systems. Removing a population that divides near a half from a population that divides above a half raises the average, by precisely the mechanism the exclusion row of the earlier table ran on. The tie-allowed headline is the exclusion policy again, with the judge rather than the person writing the report choosing which items to drop — which is why it deserves the same suspicion, not less of it for arriving through the schema.

The forced-choice version surfaces indeterminacy through disagreement, which requires the swap to detect. The tie-allowed version surfaces it through an explicit label, which is more legible but shifts the boundary: judges given a tie option tend to use it generously, and 148 of 300 items landing outside the decided set means the headline describes half the golden set. Which is preferable depends on whether you would rather have a noisier number over everything or a cleaner number over a subset you did not choose.

Deciding before the number exists#

The resolution is procedural. Every one of these choices is defensible in isolation, and the reason to fix them before the run is that afterwards you are choosing among numbers you have already seen, which is indistinguishable from choosing the answer you wanted — including to yourself.

# eval/pairwise.yaml — settled before the first judge call
protocol: pairwise
orders: [ab, ba]              # every pair judged twice
schema: forced_choice          # tie is not an option offered to the judge
tie_policy: half_credit        # a split counts 0.5 to each system
report:
  - win_rate                   # headline, over all 300 items
  - split_rate                 # published alongside, never omitted
  - self_consistency           # 1 - split_rate; gate below
gates:
  min_self_consistency: 0.70   # below this the comparison is not reported

The resolution logic is four lines and carries the whole policy:

def resolve(forward: str, reverse: str) -> dict[str, float]:
    """Two verdicts from the swapped runs -> credit for each system."""
    if forward == reverse:
        return {forward: 1.0, other(forward): 0.0}
    return {"A": 0.5, "B": 0.5}

The gate at the bottom is the part most configurations omit and the part that does the most work. Self-consistency — the fraction of pairs whose verdict survives the swap — is a property of the judge on this comparison, and it has an interpretable floor: a judge flipping half its verdicts is answering at chance, and any win rate computed from it is arithmetic on coin flips. The 80.3% observed here is comfortably above the 70% gate, so the comparison is reportable. Had a rubric change dropped it to 55%, the correct output of the pipeline is not a smaller win rate but a refusal to produce one, and the fix is to sharpen the criteria until the judge can tell the answers apart rather than to add examples until the number stabilises.

The 80.3% is not a separate measurement, and seeing where it comes from is what makes the gate readable. It is the 241 agreeing pairs over the 300 items. It is also one minus the 19.7% split rate, because a pair either survives the swap or does not, so the two shares exhaust the set and have to add to the whole. That identity is the comment in the config file above, and it is the reason the gate can be written against either quantity without changing what it means.

The floor is the part that has to be argued rather than computed, and the argument is short. Suppose a judge with no view at all, picking a slot by coin flip on every call. The two runs then agree whenever the same answer happens to win twice, which is half the time, so pure noise sits at a self-consistency of one half. A judge with a pure first-slot habit sits lower still: it names the first answer in both runs, the first answer is a different answer each time, and the two verdicts contradict each other on every single item. Anything a discriminating judge produces has to sit well above that noise floor, which is what makes a threshold at 0.70 a statement rather than a round number. It is also why a fall to 55% is disqualifying whichever explanation turns out to be right: a blunted rubric and two systems that have genuinely converged both leave the verdicts too close to coin flips to carry a headline, and the pipeline cannot tell those two apart from the number alone.

The cheaper alternatives to swapping#

Doubling the calls is not the only way to deal with position preference, and the alternatives are worth understanding because two of them are commonly used and one of them is a trap.

Randomising the order — a coin flip per item, one call each — costs the same as a single-order run and does cancel position bias in aggregate, because roughly half the items present A first. What it cannot do is tell you which items were indeterminate, because each item has exactly one verdict and there is nothing to disagree with. The win rate comes out unbiased in expectation and carries no split rate, so the fifth of comparisons that were coin flips are silently folded into the headline as though they were decisions, and no gate on self-consistency is possible.

The cancellation randomisation achieves is real and narrower than the phrase suggests, and being precise about it is what separates this option from the swap. It works on the total and on no individual item. If the judge tilts toward one slot by some fixed amount, and the coin puts each system into that slot about equally often, then the tilt helps A on about half the items and B on about half, and the two contributions subtract from each other inside the sum. Every single verdict is still tilted. What is unbiased is the average of a large number of them, which is a weaker property than it sounds like when stated as "the bias cancels".

That is enough for a headline and not enough for anything underneath one. A verdict produced under randomisation carries no indication of whether it was decided on the rubric or settled by the slot, because there is no second verdict for it to disagree with, and a quantity with nothing to disagree with cannot be gated on self-consistency at all. The swap buys per-item information at exactly twice the price; randomisation buys an unbiased average at the single-run price and forfeits the per-item information entirely.

Estimating the position effect once and subtracting it is the trap. Run a calibration where the same answer is presented in both slots against itself, measure the tilt, and correct the headline by it. The arithmetic is fine and the assumption underneath is not: position preference is not a constant, it varies with how similar the two answers are and how long they are, and it is largest exactly on the items where it matters. A global correction applies an average tilt to a population where the tilt is concentrated.

Judging both orders in a single call — presenting the pair twice within one prompt and asking for two verdicts — halves the call count while keeping both verdicts. It also lets the model see its own first answer before giving its second, which makes the two verdicts correlated by construction: self-consistency measured this way runs several points higher than the two-call version on the same items, because the model is being consistent with its own visible prior output rather than independently re-deciding. The number it produces is not the number the swap was designed to produce.

A preference that does not depend on position flips nothing#

The swap cancels the component of position bias that reverses when you reverse the order, and that is a narrower guarantee than "position bias is handled". A judge that prefers the second answer by a fixed margin will produce a flip on every genuinely close item, which the half-credit policy absorbs correctly. A judge with a preference that does not depend on position at all — a preference for longer answers, say — flips nothing. Both orders agree, self-consistency looks excellent, and the win rate is confidently, reproducibly wrong in favour of whichever system writes longer answers. High self-consistency is evidence that the judge is deciding on something stable; it is not evidence that the something is the rubric.

The two cases behave differently because of where the property being responded to lives. A slot preference is attached to the position, so reversing the positions reverses which answer benefits from it, and the two runs are pulled in opposite directions — which is exactly what produces a flip on an item where nothing else decides the question. A length preference is attached to the answer, and it travels with the answer into whichever slot the answer is put. Reversing the order moves the longer answer from one slot to the other and leaves it just as long, so both runs are pulled the same way and both agree.

Stated in plain terms, the swap is a test for asymmetries under one specific transformation, and it is silent about every property that transformation leaves untouched. That is a genuine guarantee and a narrow one, and the narrowness is why a probe set is a separate instrument rather than a redundant one: probes work by changing the answers, and the swap works by changing only where they sit.

None of this is statistical significance, either. A half-credit win rate of 59.2% over 300 items has a sampling interval around it — roughly ±5.6 points at 95% confidence, meaning an interval built this way covers the true rate in 95 of every 100 repetitions of the experiment — and the tie policy question is entirely separate from and prior to that calculation. Settling the policy does not tell you whether 59.2% is distinguishable from 50%; it tells you which quantity the interval is around.

And a split verdict is not the same object as a human rater's tie. A human who marks two answers equivalent has considered them and concluded they are equally good. A judge that flips under swapping has not concluded anything — the flip is an artefact you inferred equivalence from, and the inference is reasonable but it is an inference. On items where the stakes are high enough to matter, the flips are the right sample to send to a person.

A tie policy, then, is not a detail of the reporting layer. It is the thing that decides which of four numbers spanning twelve points goes into the sentence, and the only way to choose it honestly is to choose it before the numbers exist. What comes out the other side is a single defensible figure with a documented policy behind it — one system winning 59.2% of head-to-head comparisons against another, with a 19.7% split rate, on a 300-item golden set — plus a review queue that is more informative than the figure.

The obvious next move is to keep the harness and run it every release against the previous one, so that quality has a permanent metric and nobody ever has to argue about a 1-to-5 scale again. That move has a failure that takes about four releases to become visible.