Pointwise and pairwise judging

Two judging protocols cover almost every use of a model as a grader. Pointwise judging shows the judge one output and asks for a verdict on it in isolation — a score from 1 to 5, or a pass against a rubric. Pairwise judging shows the judge two candidate answers to the same input and asks which one better satisfies the rubric. The first produces a level, the second produces a direction, and the choice between them is usually made by whichever the harness already implements rather than by which question is being asked.

The distinction matters because the two protocols fail differently. A pointwise scale has to mean the same thing across calls, across days and across rubric edits, and it does not. A pairwise comparison has no scale to drift, and pays for that with a doubled inference cost and an answer that is relative to whatever it was compared against.

Why a pointwise scale slides underneath you#

Ask a judge to rate a single answer from 1 to 5 and it has to locate that answer on an internal scale it reconstructs from the rubric on every call. That reconstruction is sensitive to things that have nothing to do with the answer. A batch of easy questions pulls the whole distribution up, because the judge calibrates partly against what it has decided a good answer to this kind of question looks like. A rubric edit that clarifies one criterion nudges the meaning of every score. A model version change moves the anchors wholesale.

The word doing the work there is reconstructs, and the three failures above are consequences of it rather than three separate problems. A model call carries nothing over from the call before it, so whatever the judge settled on as the meaning of a 4 while grading the previous item is not stored anywhere this call can reach. The meaning of a 4 is not something the judge holds. It is re-derived from the rubric text and from whatever else the call happens to contain, once per item, from the beginning. Nothing ties one derivation to the next. That is why batch composition, rubric wording and model version all move the number: each of them changes the material the anchor is derived from, and the anchor is the only thing standing between an answer and its score.

The effect is measurable and it is not small. Re-scoring 300 stored outputs from one support assistant a week later, with the same judge prompt and the same model, moved the mean from 3.41 to 3.48 — a shift of 0.07 with nothing changed but the day. On a comparison between two prompt variants that differed by 0.11 points, that drift is the same size as the difference the experiment was trying to detect, which is the whole problem: a measurement is only useful when the thing it varies with is the thing you are asking about.

Setting the two numbers beside each other is what shows how bad this is. The signal was 0.11 points and a week's delay moved the instrument 0.07, which is nearly two thirds of the effect. Worse, it is not the kind of error that averages away when more items are added. The re-score ran over the same 300 stored outputs, so not one thing being graded had changed, and any movement in the mean was movement in the instrument rather than scatter in the sample. More items shrink the sampling error around each mean and do nothing whatever to a shift that has moved both means together.

Head-to-head judging removes the scale entirely. The judge sees the question and both answers in one context and answers a different question — which of these two better satisfies the rubric — and that question has no anchor to drift, because the comparison is internal to a single call. The judge does not have to remember what a 4 means; it has to decide which of two concrete texts is better, with both in front of it. Discriminating between two nearby items is an easier task than placing one item on an absolute scale, for people and for models alike, which is why a pairwise protocol detects differences that pointwise scoring buries in noise.

The protocol, and the swap that makes it honest#

The naive version presents A first and B second on every item, and it is broken in a specific way: judges have a position preference. Presented with two answers of genuinely equal quality, a judge will not split 50/50 — it will favour whichever slot it reads first, or last, at rates that in published measurements run from a few points to well over twenty depending on the model and the prompt. That preference is invisible in the results, because a 58% win rate for B looks like a win for B regardless of how much of it came from B being in slot two.

The preference has a mechanism, and knowing it is what makes the swap look necessary rather than superstitious. The two answers are symmetric in the question being asked and not symmetric in the request that carries it. One of them occupies the first slot and one the second, a model produces its answer conditioned on everything in front of it, and the answer in the first slot was read into a context holding only the question while the answer in the second was read into a context that already held the first answer. Those are two positions in a sequence rather than two equivalent inputs, and no wording of an instruction makes a sequence positionless. Telling the judge that the order carries no meaning, which the system prompt below does in its last line, is worth doing and does not remove the asymmetry, because the asymmetry is in the arrangement of the text rather than in the judge's intentions.

The fix is to run every pair twice with the answers swapped and treat the two verdicts as one measurement:

import json
from anthropic import Anthropic

client = Anthropic()

SCHEMA = {
    "type": "object",
    "properties": {
        "winner": {"type": "string", "enum": ["first", "second"]},
        "reason": {"type": "string"},
    },
    "required": ["winner", "reason"],
    "additionalProperties": False,
}

PAIRWISE_SYSTEM = """You compare two candidate answers to the same support
question against a fixed rubric.

{rubric}

Decide which answer better satisfies the rubric. Judge only the rubric's
criteria. Do not prefer an answer for being longer, better formatted or more
confident in tone. The order in which the answers appear carries no meaning."""


def compare_once(question: str, first: str, second: str) -> str:
    response = client.messages.create(
        model="claude-opus-5",
        max_tokens=600,
        system=PAIRWISE_SYSTEM.format(rubric=RUBRIC),
        output_config={
            "format": {"type": "json_schema", "schema": SCHEMA},
            "effort": "low",
        },
        messages=[
            {
                "role": "user",
                "content": (
                    f"<question>{question}</question>\n"
                    f"<answer_1>{first}</answer_1>\n"
                    f"<answer_2>{second}</answer_2>"
                ),
            }
        ],
    )
    return json.loads(response.content[0].text)["winner"]


def compare(question: str, a: str, b: str) -> tuple[str, str]:
    """Returns which system won in each presentation order: (a-first, b-first)."""
    forward = compare_once(question, a, b)
    reverse = compare_once(question, b, a)
    return (
        "A" if forward == "first" else "B",
        "B" if reverse == "first" else "A",
    )

What the swap does is give each answer the first slot exactly once. Whatever advantage that slot confers is handed to A on one call and to B on the other, so across the two calls it lands on both sides equally and cancels in the total. The cancellation is a property of the aggregate and of nothing smaller. On an item where the slot advantage was large enough to decide the outcome, the two calls come back with opposite winners, and that is not a cancelled measurement — it is a measurement that came out split. Those items are still sitting there individually after the totals have been taken.

The compare function returns two verdicts per item rather than one, and that is deliberate — the pair of verdicts is the raw measurement, and collapsing it to a single winner is a decision with consequences that the next chapter deals with.

What the extra inference cost buys#

Running each pair twice doubles the judge calls for a fixed comparison, which is the headline cost, and the prompts are larger too because both answers ride in every call. Measured on a 300-item comparison between two support-assistant prompts:

Every money figure in this chapter comes out of one calculation, and doing it once makes the rest readable at a glance. Take a single pairwise call. It sends about 1,700 input tokens and gets back about 150 output ones. At $5 per million input tokens, a thousand input tokens costs half a cent, so 1,700 of them cost a little over eight tenths of a cent. At $25 per million output tokens, a thousand output tokens costs two and a half cents, so 150 of them cost a little under four tenths of a cent. The two add to about one and a quarter cents for the call. Multiplying by the 300 calls of a single-order run gives the $3.68 in the table, and multiplying by 600 instead, because the swap runs every pair twice, gives the $7.35.

That multiplication is allowed only because the token counts are averages, which is what the tildes in the table are admitting. Answers vary in length and no individual call costs exactly one and a quarter cents. What makes the total trustworthy anyway is that the error runs in both directions across hundreds of calls, so the long answers and the short ones pull the average back toward the middle. The figures are quoted to the cent because the arithmetic lands on cents, not because the estimate is good to a cent.

Reading the two halves of that call separately gives a shape most people do not expect. Input cost about eight tenths of a cent and output about four tenths, so the input side is more than twice the output side — despite output being priced five times higher per token. The whole of the reason is in the counts. There are more than eleven times as many input tokens as output ones, and a factor of eleven in the counts beats a factor of five in the price. A judge call is a great deal of reading and a very small amount of writing: a rubric, a question and two full answers go up, and a winner with a sentence of reasoning comes back. That is a property of the shape of the request rather than of the price list, which is why an estimate built by picturing what the judge writes comes in under every time.

The pointwise column has a different shape for the same reason, and it is worth pricing to see the contrast. A pointwise call carries one answer instead of two, so its input falls to about 1,100 tokens, and it returns a score with a justification rather than a single word, so its output rises to about 200. That is a little over half a cent of input and half a cent of output, near enough one cent a call, with the two sides close to balanced. Six hundred such calls — the 300 items scored once for each of the two systems — is the $6.30. The same price list produced a differently shaped bill because a differently shaped request went up.

The phrase about a doubled inference cost is worth checking against those three figures, because the obvious reading of it is wrong. The doubling is measured against pairwise run in one order only, where it is exact: $3.68 becomes $7.35. It is not a doubling against pointwise. Pointwise had to score both systems, so it was already making 600 calls, and $6.30 against $7.35 is a difference of about a sixth. The sentence a reader is likely to write after meeting the swap — that head-to-head judging costs about twice what scoring costs — has correct arithmetic attached to the wrong pair of columns. Relative to the protocol it would replace, the swap is close to free, and it is close to free because scoring two systems separately was never a single-pass operation either.

Seven dollars and four minutes is not a budget conversation for a single comparison. It becomes one along two axes. The first is frequency: a nightly run over 800 examples is 1,600 swapped calls a night, about $39 a night and $1,200 a month, which is a real line item for a metric nobody looks at most mornings. The second is the number of systems, and this is the axis that bites. Pointwise scoring is linear — five prompt variants over 300 examples is 1,500 calls. All-pairs comparison is quadratic: five variants is ten pairs, 6,000 swapped calls, four times the cost to answer a question about the same five systems. Comparing against a single fixed baseline instead of all pairs restores linearity, at the price of only ever learning how each variant does against that one baseline.

The quadratic claim is worth counting out rather than taking on trust. With five variants the pairs are A against B, A against C, A against D and A against E, then B against C, B against D and B against E, then C against D and C against E, and finally D against E. That is four, then three, then two, then one, which comes to ten. Each pair is counted once rather than twice, because comparing A with B and comparing B with A is one comparison run in two orders, and running it in two orders is what the swap already does. Ten pairs over 300 examples in both orders is 6,000 calls against the 1,500 pointwise needs for the same five systems, so the ratio of four is exact in calls. In money it is somewhat worse than four, because a pairwise call carries two answers and costs more than a pointwise one.

What makes that quadratic rather than merely large is what happens when a sixth variant arrives. It does not add one pair. It adds five, one against each variant already present, and the seventh would add six. The number of pairs grows roughly as the square of the number of systems, so doubling the variants roughly quadruples the work and a comparison that is comfortable at five is not at ten. That is also what makes the fixed-baseline arrangement above linear rather than merely cheaper: with a baseline, each new variant adds exactly one pair however many are already in the room, so the work grows by a constant each time instead of by the number of systems already there.

How many items the decision needs#

Cost is one constraint and statistical power is the other, and they pull in the same direction, so it is worth knowing where the floor is before paying for anything above it.

A win rate is a proportion, so the interval around it is the usual binomial one — the range of true win rates that could plausibly have produced the count you observed. At 300 items and an observed 55%, the standard error is about 2.9 points and the 95% interval runs from 49.4% to 60.6%, which includes 50%. A measurement like that does not, at conventional confidence, establish that B is better than A. It establishes that B is probably better.

Both of those numbers come out of one formula, and watching them do it is what turns the item counts further down from quoted figures into predictable ones. The standard error of a proportion is the square root of the proportion times its complement, divided by the number of items. Here that is 55% times 45%, which is a shade under a quarter, divided by 300; the square root of what is left is just under three hundredths, and three hundredths of a proportion written as a percentage is the 2.9 points above. A 95% interval reaches a little under two standard errors either side of the estimate, so it extends a little over five and a half points each way from 55%. That is the 49.4% at the bottom and the 60.6% at the top, and neither endpoint was measured — both were computed from the count.

What the interval means is the part worth being careful about. It is the range of true win rates that could plausibly have produced 55 wins in every hundred observed here. Fifty is inside that range, so two systems of exactly equal quality could have produced this result with nothing unusual happening. That is not the same as concluding the systems are equal. An interval containing 50% fails to rule out a coin flip and does not establish one, and 55% remains the single most likely value in the range. The data lean toward B and do not settle the question, which is why the decision in the scenario below is recorded as a balance of evidence rather than as a result.

Making a five-point edge distinguishable from a coin flip needs roughly 400 items. A three-point edge needs about 1,070, and a one-point edge needs nearly 10,000, because the requirement scales with the inverse square of the effect:

The cost column of that table is not four separate measurements. A swapped comparison spends two calls on one item, and two calls at a little over one and a quarter cents each is about two and a half cents an item — so the column is that one price repeated. A hundred items at two and a half cents is the $2.45, four hundred is the $9.80, and 9,600 is the $235. The useful consequence is that nothing in the cost of resolving a smaller difference is driven by the price of anything. It is driven entirely by how many items the statistics demand, and a cheaper judge moves every figure in that column by the same factor without changing a single item count.

The inverse-square rule is visible in the item counts themselves. Halve the edge, from 10 points to 5, and the count goes from about 100 to about 400, which is four times as many. Divide the edge by ten, from 10 points to 1, and the count goes from about 100 to about 9,600, which is close to a hundred times as many. The middle row obeys the same rule: moving from a five-point edge to a three-point one multiplies the edge by three fifths, so the count multiplies by the square of five thirds, a little under three, and four hundred items becomes a little over a thousand.

The reason is the standard error again, and specifically the square root in it. The item count sits under that root, so the noise around a win rate shrinks with the square root of the number of items rather than with the number of items. To resolve an edge half the size you need the noise half the size, and halving something that shrinks like a square root takes four times as much of it. Everything unpleasant about the last row of that table follows: measuring twice as finely costs four times as much, and measuring ten times as finely costs a hundred times as much.

The intuition to carry away is that precision is bought in a bad currency. Doubling the budget does not double what you can see — it improves the resolution by about forty per cent. That is why reporting a difference as too small to see is often the honest output rather than an admission of defeat: the alternative is an amount of judging nobody is going to authorise.

Two corrections make the real numbers worse than that table. Split verdicts carry no directional information, so a 20% split rate means the effective sample for the direction is about 80% of the item count. And a golden set — the fixed, curated collection of inputs a suite replays — is fixed, so you cannot conjure 1,070 items to detect a three-point edge if you have 300, which means for small differences the honest output is "below our resolution" rather than a longer run.

That 20% is not a hypothetical rate on this workload. The run described below produced 241 pairs that agreed with themselves across the swap and 59 that did not, and 59 out of 300 is a shade under a fifth. The 241 that agreed are the only ones carrying a direction, and 241 out of 300 is about four fifths, so a comparison budgeted at 300 items is doing the statistical work of the 241. Fed back through the square-root rule that costs roughly a tenth of the precision, which is small — and it is small only because this judge split on a fifth of its items. A judge that split on half of them would be running a 300-item comparison at the resolution of 150, and the split rate is therefore something to measure rather than assume when sizing a run.

The instinct at this point is to reach for pointwise scoring on the grounds that a continuous score carries more information per item than a binary win. Scored on the same items, the paired difference in pointwise scores has a smaller variance than the raw scores do, and the item count needed to detect a 0.11-point mean difference works out around 320 — competitive with pairwise. So the case for pairwise is not primarily about sample efficiency. It is that the paired pointwise calculation assumes the scale meant the same thing for both systems, which holds when they are scored in the same run with the same judge and stops holding the moment the two runs are separated in time. Pairwise buys immunity to that assumption, and the doubled inference cost is the price of not having to defend it.

Why pairing helps at all is the same principle the swap runs on: a nuisance factor stops distorting a comparison once both sides are made to carry it equally. A score on an item is part item and part system. Some questions are harder than others, and both systems are handed the same hard ones. Comparing two whole-set means leaves all of that question-to-question variation sitting in the noise, and the per-item standard deviation of 0.94 is mostly measuring how much the questions differ from each other rather than how much the systems do. Subtracting the two scores item by item removes the shared part, because whatever the question contributed it contributed to both sides. What survives the subtraction is the difference the experiment is about plus a smaller quantity of noise, and less noise is exactly what the square-root rule converts into fewer items.

Deciding between the two protocols is therefore a question about what the answer will be used for rather than about which instrument is the more accurate, and the practical rule that falls out is short: use pairwise when the two systems exist simultaneously and the question is which to ship. Use paired pointwise when they exist simultaneously and you also need the level. Use neither when the systems were measured weeks apart, because in that case the only defensible move is to re-score both with today's judge, and once you are re-scoring you are back to a simultaneous comparison anyway.

The question pairwise cannot answer#

A win rate is a direction and a magnitude of preference, and it is not a statement about how good either system is. Consider what 55% is compatible with. Both systems could be excellent, differing on stylistic margins the rubric happens to care about. Both could be poor, with B slightly less poor. B could be dramatically better on the 55% and catastrophically worse on the 45%, which a win rate cannot distinguish from B being marginally better everywhere. The comparison is silent on all of this, because it never asked how good anything was — only which of two was better.

That silence matters wherever a threshold is involved. Questions like whether an assistant is good enough to enable for the enterprise tier, whether quality has recovered since last quarter's regression, whether a feature clears the bar the team set in the design doc, and whether an individual response should be blocked before it reaches a customer are all questions about a level. A win rate has no level in it. Pointwise scoring, with all its drift, is the only one of the two that produces a number you can compare to a threshold, and that is why it survives despite being the noisier instrument.

The two protocols are therefore not competitors so much as answers to different questions, and the practical arrangement most teams land on uses both: pairwise for the ship-or-not decision between two candidates, where sensitivity matters and the comparison is a one-off; pointwise for the standing metric, where the drift has to be managed rather than avoided. Managing it is the harder half, and the next-to-obvious shortcut — making the standing metric a win rate against the previous release, so you never need an absolute scale at all — has a failure mode worth understanding before you adopt it.

Neither of those is the immediate problem, though. The immediate problem is the 59 items where the two swapped verdicts disagreed. Those are 59 items where the judge preferred whichever answer it read in a particular slot, and the run produced a headline of 55% without yet saying what was done with them.