Anchoring a win-rate series

Comparing each release against the one before it is the obvious way to turn a one-off pairwise comparison into a standing metric. It is cheap, it needs no absolute scale, and it produces a number every time you ship. It is also a chained index — a series in which every measurement is taken against a different reference — and chained indices have a known pathology that quality metrics inherit wholesale.

An anchor is anything held fixed across releases so that successive measurements are commensurable: a stored set of outputs, a set of reference answers, a pinned pointwise rubric, or a human control set. What follows is why a chain without one cannot express absolute movement, which anchor to pick, the cheap per-comparison signal that catches the same problem before the trend does, and what to do when the anchor itself goes stale.

A chain of comparisons has no fixed point#

Here is a chained series over four releases of one support assistant, each release judged against its immediate predecessor with the swapped protocol and a pinned tie policy:

Every number there is a measurement of a difference, and the baseline moves forward every time. There is no point in the series at which anything is compared to something that stays still, so the series cannot express absolute movement. Four win rates of 55% are equally consistent with quality rising steadily, quality staying flat while stylistic preferences shuffle, and quality falling — because each measurement was taken against a different reference, and nothing in the arithmetic connects them. Statistical agencies handle the same problem in price indices by periodically rebasing to a fixed reference year and publishing both series over an overlap window. A quality metric built from consecutive win rates is a chained index with no rebasing at all.

One number decides how every row of the win-rate table above is read, and it does not appear anywhere in it: the point at which a win rate says "no difference". Under the half-credit tie policy, a pair the judge cannot separate contributes half a win to each side, so two systems that are genuinely indistinguishable come out at one half. Every number in the win rate column is therefore read as a distance above or below even, and four releases sitting five or six points above it are four small claimed improvements. Each claim is only as good as the thing it was measured against, and no two of them were measured against the same thing.

Why those four claims cannot be combined is a property of the quantity rather than a caution about it. A win rate is a count over a denominator: the share of items on which one system was preferred to another particular system. It is not a reading taken on a scale, and both systems in it are arguments — change either one and what you have is a different number answering a different question. Two win rates taken against two different references are therefore not two readings of one quantity, and there is no operation that combines them. Adding 56.1% to 53.8% produces a number. It does not produce a fact about v3 and v1.

Chaining is not a blunder that price statisticians fell into, which is worth saying before the price-index comparison above is used against them. A chained index measures each period against the last, and that is what lets it handle a changing basket gracefully: goods enter and leave, and each link only ever has to compare two adjacent periods that mostly overlap. The cost it pays for that is the other half of the same property — any small systematic error enters at every link and accumulates — and rebasing is the standing repair. So the technique is sound and comes with a known maintenance obligation, and a win-rate series has adopted the technique while skipping the obligation.

Where the comparison stops holding is in what makes the chaining legitimate, and the limit is worth naming because the analogy is a good one right up to it. A price relative chains because each link is a ratio of two prices measured in the same units on the same scale, and ratios of a common quantity genuinely multiply: this year over last year, times last year over the year before, cancels the middle term and leaves this year over the year before that. The cancellation is the whole trick, and it needs a shared middle term to cancel. A win rate has none. It is not v3's quality divided by v2's quality; it is a count of preferences expressed about one specific pair, and the pair is the entirety of what the number is about. Nothing inside it survives to be cancelled against the next link. The series has the shape of a chained index without the arithmetic that would let one be recomposed, which is why anchoring it is not a refinement on the metric but the step that makes it a metric.

The specific mechanism by which this particular series went wrong is in the last column of the table. Answer length grew about 15% per release, from 214 tokens to 331, which nobody planned — it is what happens when four rounds of prompt edits each add a clarifying instruction. This judge, per its probe set, has a residual preference for the longer of two answers worth roughly eleven points of win rate at that size difference. So a release that was genuinely slightly worse, and would have lost 44-56 to its predecessor under a length-blind judge, came back at 55%. Four times.

Three figures in that paragraph are one addition, and it is worth doing slowly because it is the whole mechanism of the chapter on one line. Start with the release that was genuinely a little worse than the one before it. A judge with no length preference registers that, and the pairwise result comes back a few points below even — the 44-56 split above, with the newer release on the losing side. The judge actually in use is not length-blind. Its preference is worth about eleven points of win rate at the size difference these releases happen to have, and those eleven points are added to whichever side is longer, which is the newer release every time. Adding eleven to 44 lands on 55, and 55 is a number a dashboard reads as a win.

Nothing in that addition is specific to one release. It holds wherever the true gap is smaller than the bias, which is another way of saying this chain can absorb a genuine decline of up to about eleven points per link before the reported number crosses below even. That figure is not a constant of anything. It is this judge's sensitivity at this size difference, measured on a probe set, and the part worth keeping is not the number but the fact that a number exists, has been measured, and can be compared against the size of the effect being claimed.

The composition of the links is what makes that decline invisible. Each link is contaminated by the same bias in the same direction, and each link's contamination is small enough to look like an ordinary result. The bias does not cancel across the chain; it accumulates, because the thing driving it moves monotonically. Run v5 against v1 directly and the story changes: the length ratio is 1.55×, well past where the judge's preference saturates, the true quality gap is now four releases wide, and v5 wins 38.4% of the head-to-head. The number that detects the decline is the one comparison the pipeline never made.

Cancellation would need the bias to change sides, and it never does. An error that favours a different system on each link does average out over enough links, because the contributions carry opposite signs and a sum of opposing terms stays near zero. Here the newer release is the longer one on every link without exception, so the judge's preference favours the challenger every time. A run of same-signed errors does not average out; it sums, and the sum grows with the number of links. A judge that mildly prefers the longer or the newer of two answers therefore keeps the numbers reassuring the whole way down. That is the difference between an inaccurate series and a misleading one, and it is why four ordinary-looking results can sit on top of a decline that a single direct comparison finds in one run.

Diagram

Non-transitivity makes this worse than a bookkeeping problem, because even a bias-free chain would not license the inference. Pairwise preferences from a judge are not guaranteed to be transitive: A beating B and B beating C is entirely compatible with C beating A, and cyclic preferences show up in measured judge data often enough that they are a known property rather than a curiosity.

Seeing why a judge is under no obligation to be transitive removes most of the surprise from that. A transitive ordering exists when every candidate holds a position on one underlying scale and each comparison does nothing but read off which position is higher; the scale is what forces consistency between comparisons that were never run together. The judge is not consulting a scale. It is handed one specific pair, and it decides on whatever rubric-relevant differences that particular pair happens to exhibit — one pair may turn on completeness, the next on accuracy, the next on tone, because those are the differences present in the two texts in front of it. Different pairs surface different criteria, so different comparisons are effectively answering different questions, and nothing anywhere forbids the answers from forming a cycle.

So there is no arithmetic that reconstructs "v5 versus v1" from four consecutive win rates, however clean each one is. The composite comparison has to be run.

Which anchor to pick#

Four things can play the role, and they detect different failures:

The call counts in that table are the swapped protocol's arithmetic rather than a fresh quote. A swapped run puts every item through the judge twice, once in each order, so a 300-item golden set is 600 calls. A pointwise rubric asks one question per item and is 300. At a shade over a cent a call, the gap between those two rows is a few dollars a release, which is small enough that the choice between them should turn on what each one detects rather than on what each one costs — and the fourth row, priced in rater-hours rather than calls, is the only one where cost is a real constraint.

The cheapest useful arrangement is the first. A release's stored outputs cost nothing to keep and are not a hypothesis about quality — they are a record of what the system did on a specific date. Every release then runs two comparisons instead of one: against its predecessor, which stays sensitive to small changes and answers "did this release help", and against the frozen anchor, which answers "where are we relative to where we started". The anchored series over the four releases above is 54%, 49%, 44%, 38% — visibly monotone, visibly a problem, and visible from release three.

"Frozen" is carrying the weight in that arrangement, and it is worth separating from "specified". An anchor works because two measurements taken months apart are against the same object, and the only way to guarantee sameness is to keep the object. Storing v1's prompt and model name instead — a recipe for reproducing the outputs rather than the outputs themselves — looks equivalent and is not. Re-running that recipe next quarter goes through a provider whose model behind the name may have moved, a picking step that is allowed to be random, and a retrieval corpus that has gained documents since. What comes back is a different set of answers wearing the same label.

The failure that follows is this chapter's own failure in disguise. If the anchor is regenerated each release, every anchored measurement is taken against a slightly different reference, which is a chained index again — this time with the chain hidden behind a fixed name, which is worse, because the dashboard now carries a column that looks anchored and is not. Storing the outputs removes the question rather than managing it. A stored answer is a record of what the system emitted on a date, and a record does not drift.

What the anchored series buys is permission to subtract. All four of its numbers were taken against one unmoving reference, so the difference between any two of them is a difference in the same quantity: how each release fares against one fixed opponent. The sixteen-point fall from the first of those numbers to the last is a real statement, and it is not the same statement as a direct v5-against-v2 comparison — non-transitivity is precisely why it is not — but it is a common yardstick, and a common yardstick is the minimum a trend line requires.

No subtraction of that kind is available in the chained series. The gap between the second and fourth rows of the win-rate table is the difference between a fact about v3 and v2 and a fact about v5 and v4, and the difference between two facts about different things is not a fact about anything. That is why the two series are not two views of one measurement with one of them clearer. Only one of them is measuring a quantity that a trend can be taken of at all.

The frozen baseline is silent about whether the anchored release was good, which is the standing limitation of every relative measure and the reason a pointwise rubric usually sits alongside it. Anchoring converts an uninterpretable series into an interpretable trend; it does not convert either into a threshold. "We are 12 points below where we started" is that last anchored figure read against the one half that parity would have given, and it is actionable in a way that "we won 55% again" is not. It still does not answer whether shipping is acceptable, and no relative measure ever will, because every one of them reports a distance and none of them reports a position.

Catching the confound while it is still small#

The anchored series is a detector for accumulated drift, which means it works by waiting. Three releases went by above before the trend was unambiguous, and there is a cheaper signal available on every individual comparison.

A probe set already tells you what a judge is sensitive to. Any property the judge is known to respond to and that also moves monotonically across releases is a candidate confound — something that varies in step with what you are measuring and could account for the result on its own — and the list is short and enumerable: answer length, formatting density, hedging rate, whether the answer contains a code block. Recording those alongside every comparison costs nothing, since they are computed from text you already have, and turns a confound from something you discover into something you watch.

The diagnostic that follows is a length-matched subset. Of the 300 items in the v4-to-v5 comparison, 96 had answers within 10% of each other in token count, and on those 96 the win rate was 47.9% rather than 54.6%. That number was available on the day v5 shipped. It says the entire measured advantage lived in the items where v5's answer was substantially longer, which is either a real effect of length on quality or the judge's known preference, and the comparison cannot tell you which — but it is enough to stop the release from being recorded as an unambiguous win.

Reading those two rates as counts over different denominators sharpens what is being claimed. The 54.6% is over all 300 items. The 47.9% is over the 96 whose answers came within a tenth of each other in length, which is under a third of the set, and on that third the newer release is behind rather than ahead. The whole of the difference between the two rates therefore has to be sitting in the other two thirds — the items where v5's answer was substantially the longer one. That is a conditional statement rather than a corrected headline. It says what the win rate looks like given comparable length, and it is silent on what the rate would have been had length never moved, because no items exist in which it did not.

A guardrail makes it automatic. If the mean length ratio between the two systems exceeds 1.10, the pipeline reports the length-matched win rate next to the headline and marks the comparison as confounded. On this history that condition fires on all four releases, which is the correct outcome: all four were confounded, and the dashboard said so on none of them.

The subset shrinks as the systems diverge, which is the limitation. By v5 against v1 the length ratio is 1.55 and there are 34 matched items left, too few to say much. The shrinkage is not an incidental nuisance; it is what a matched subset is. The subset is the overlap between two length distributions, and two distributions whose centres pull apart share progressively less of their mass. At a ratio near a tenth the two still sit largely on top of each other, which is why 96 items qualify; at 1.55 they barely meet, and 34 is what is left where they still do. The matched-subset check is a tool for the adjacent comparison, where the systems are still similar enough to have overlapping distributions; the anchor is the tool for the distant one. They fail in opposite regimes, which is why both are worth running.

Anchors age, and the fix is an overlap window#

A frozen anchor rots. Stored outputs were produced against a product that then gains features, changes its plan structure, and retires settings the answers refer to. Several releases later, a comparison against that anchor is partly measuring whether the current system still matches a world that no longer exists, and the anchored win rate starts rising for reasons that have nothing to do with quality.

The discipline that handles this is the one the price-index analogy suggests. Rotate the anchor deliberately, on a schedule, and run an overlap period in which both anchors are measured — a release judged against the old anchor and the new one in the same run. The overlap gives the conversion factor between the two series, so the historical trend can be spliced rather than restarted, and it makes the rotation an explicit event with a number attached instead of a silent change in what the dashboard means. It costs one extra comparison during the overlap window and nothing afterwards.

What the overlap actually supplies is one release measured on both scales at once. Its win rate against the retiring anchor and its win rate against the incoming one are two readings of the same system taken on the same day, and the relationship between that pair is the only thing available to carry the old series and the new one onto a common footing. It is an estimate taken at a single point, which is the honest limit of the technique: splicing assumes the relationship between the two anchors is the same for the releases before the overlap as it is during it, and nothing measures that assumption. The alternative is not a better estimate but no estimate, which is a chart that restarts and a history that cannot be read across the break.

Two rules keep this honest. Rotate on a calendar, not when the anchored number looks bad, because rotating in response to a decline is choosing a baseline that flatters the present. And never rotate silently: the anchor identity belongs on the chart, next to the number, because a trend line whose reference changed halfway is a different kind of lie from the one this chapter started with.

The first of those follows from the arithmetic rather than from etiquette, which is worth spelling out because it is what makes the rule enforceable rather than merely sensible. Rotating the anchor onto the current release sets the next anchored measurement against a system the current one matches exactly, so the series returns to parity by construction, whatever was happening to quality underneath. A rotation timed to a decline therefore does not fail to detect the decline. It erases it, and it erases it in a way that leaves a plausible-looking chart behind, which is why the timing has to be taken out of anyone's hands before there is a number to react to.

The reference point is not the judge#

A frozen baseline fixes the reference point. It does not fix the judge. A judge with a length preference still has that preference when comparing v5 to v1 — the anchored series showed the decline because the length gap grew past where the preference could cover it, not because the preference went away. On a comparison where the bias and the true difference happen to be the same size, an anchored measurement is as wrong as a chained one. Anchoring buys a trend you can read; probe sets and human calibration are what make the individual readings trustworthy, and they are separate work.

Anchoring also cannot help where the comparison is not decidable from the two answers alone. Everything above assumes a judge can read two candidate responses and say which better satisfies the rubric — plausible for support answers, where the criteria are about what the text says, and implausible the moment correctness depends on something outside the text.

A chained series, then, is not a weak trend line. It is a set of local measurements that never composes into a trend, and the amount of decline it can absorb is set by how much of each link the judge's biases can cover. One extra comparison per release against something that never moves is what converts it into a series a person can read. It is not expensive; it was skipped because the chained number looked fine.

Two SQL queries against the same question can be compared for style all day without either judge or reader knowing which one returns the right rows, and no amount of holding the baseline still makes that comparison mean anything. For those tasks the anchor has to move down a level: not one fixed system to compare against, but one fixed correct answer per item.