The frozen anchor set

A judge validated against human labels this week is a judge you know something about this week. It is a separate property from comparability across time, which is the question of whether the number your scoring pipeline emitted in January and the number it emits in May were produced by the same instrument, and only one of the two properties is about the past. A quarterly chart is a claim about the past.

An anchor set is the artifact that makes that claim checkable: a fixed collection of archived product outputs — the input, the output, and every field the judge saw — frozen at a moment in time and never added to or drawn down. Its whole value comes from being files. Re-score it with a judge that has since been edited and any change in the pass rate is attributable to the judge, because nothing else about those items can have changed.

That last sentence is the entire experiment and it rests on a single step worth making explicit. Two runs differ in their result for some reason. If everything fed into the second run is identical to what was fed into the first except for one thing, then whatever differs in the output is caused by that one thing, since nothing else was available to cause it. Freezing the outputs as files is what buys the "everything else identical" clause: a stored answer cannot be regenerated slightly differently, cannot be re-retrieved against an updated index, and cannot drift with the product prompt. Regenerate the answers instead of storing them and the clause fails, along with every conclusion that depends on it — which is why the archive has to hold the text rather than the recipe for producing it.

Three inputs, one output, no separation#

Everything upstream of a monthly pass rate collapses into a single scalar, and the collapse is where the attribution is lost:

Two of those arrows belong to the thing you are trying to measure and four belong to the instrument measuring it. A rising pass rate is consistent with the product prompt getting better, with the judge prompt getting more forgiving, with the served model changing under a rolling alias, with the input mix shifting toward easier traffic as a product's user base changes, and with any mixture of the five. The series contains one measurement per month rather than one measurement per thing that moved, so no amount of care in reading it separates them.

Freezing three hundred outputs#

Three hundred is a reasonable size: large enough that a shift of five points is outside the noise, small enough to re-score for the price of a coffee. Sample it the way you would sample anything you intend to trend, meaning stratified across the segments you care about and frozen at that composition forever, since an anchor set whose membership drifts is not an anchor.

-- Freeze 300 month-one outputs as an immutable anchor set.
CREATE TABLE anchor_2026_01 AS
SELECT
  o.id            AS output_id,
  o.input_text,
  o.output_text,
  o.segment,
  s.verdict       AS original_verdict,
  s.scored_at     AS original_scored_at
FROM outputs o
JOIN judge_scores s ON s.output_id = o.id
WHERE o.created_at >= '2026-01-01'
  AND o.created_at <  '2026-02-01'
  AND s.is_latest
ORDER BY hashtext(o.id::text)   -- deterministic pseudo-random draw
LIMIT 300;

If you have no such archive, this is the single highest-value thing to fix today, and it costs a table. Store the input, the output, and every field the judge saw, not just the verdict — a re-score needs to reconstruct the judge's input exactly, and an archive that kept only "pass" is an archive of a conclusion rather than of evidence.

Reading a re-score#

Re-scoring 300 archived January outputs with a judge that has since been edited four times returned 228 passes, or 76%. They had originally scored 204, or 68%.

Read across the top row and you get the instrument's contribution: eight points appear when the same outputs are graded by the edited judge, with nothing about those outputs having changed. Both of those percentages are counts over the same denominator — 228 of 300 is 76%, 204 of 300 is 68% — so the eight points are 24 outputs out of 300, and that count is the thing to reason about rather than the percentage. Read down the right column and you get what remains for the product: three points, measured with one judge held fixed, which is the only comparison in the table that is apples to apples on the instrument.

Because the same items were scored twice, the comparison is paired and you should read it that way rather than as two independent proportions. Of the 300, 34 flipped from fail to pass and 10 flipped from pass to fail; 256 were unchanged. Thirty-four minus ten is the net movement of 24 items, which over 300 is the eight points, and the 256 that did not move contributed nothing to it in either direction.

That last clause is the reason the standard error looks unfamiliar. Written out it is about sqrt(34 + 10) / 300, roughly 2.2 points, and the striking thing is that 256 of the 300 items appear nowhere in it. They are excluded because they carry no information about the difference: an output that passed under both judges tells you the two judges agree about it and says nothing about how far apart they are. All the evidence about the size of the shift lives in the 44 items that moved, and the uncertainty is therefore the uncertainty in how those 44 split between the two directions. Treating the two rates as independent proportions would ignore that the same items were scored twice and would report a much larger error, which is how a real instrument shift gets dismissed as noise.

So the eight-point instrument shift is comfortably real — four times its own standard error — and the three-point product shift is not. That second comparison is between January's outputs and May's, which are different items, so nothing pairs and the two proportions each carry their own error; combining them puts the error near 3 points, which is as large as the shift itself.

The fourth mover is the traffic#

Three of the points survived the instrument check, and they are still not the product's until one more confound is ruled out. The input mix is an arrow into the output node of that diagram, and it moves on its own schedule: marketing campaigns change who arrives, a new self-serve flow diverts the easy questions or the hard ones, seasonality shifts the ratio of billing disputes to password resets. A pass rate computed over a changing mix of segments moves when the mix moves, with no change in how well any individual case is handled.

Checking it costs nothing beyond a GROUP BY, provided your scores carry a segment label. Score January's and May's outputs with today's judge, break both out by segment, and then recompute May's overall rate holding January's segment shares fixed:

The bottom two rows are the ones that do the work, and both are the same calculation with different weights. To get the observed May figure, multiply each segment's May pass rate by its May share and add the four products: 70% at just over a quarter of the traffic, 89% at not quite a quarter, 82% at about a quarter, 78% at a fifth — which comes to the 79%. To get the standardised figure, run the identical sum with May's pass rates and January's shares instead, so billing disputes are weighted at 41% rather than 28% and password requests at 12% rather than 24%. Nothing about how well any segment is handled has been altered; only the weights have. The answer falls to 77%.

Two of the three remaining points came from serving an easier mix of questions. Billing disputes, the hardest segment, fell from 41% of traffic to 28%, while password and access requests — which the assistant handles at 89% — doubled their share. The direction follows mechanically from those two moves: traffic shifted out of the segment with the lowest pass rate and into the segment with the highest, and an average over a changing mix moves when the mix moves even where every component is frozen. The shipping and other segments barely shifted in share, which is why they contribute almost nothing to the difference between the two bottom rows.

Nothing about that is dishonest; the product genuinely serves a different population than it did in January, and if a self-serve flow removed the hard tickets on purpose, that is a real improvement to something. It is not an improvement in the quality of the answers, which is what a quality chart claims. The mix-adjusted comparison is the one to report, with the raw one beside it and the difference explained in a sentence.

Standardising like this requires only that every score row carries the segment. If yours do not, add the field before you need it, because retrofitting segment labels onto six months of archived outputs is an afternoon of classification work you can avoid with one column.

What it costs to run#

Three hundred re-scores at roughly 1,600 input and 400 output tokens each is about 480,000 input and 120,000 output tokens — 300 times 1,600 and 300 times 400. At the published rates for a frontier-tier judge model that is under six dollars, and at twenty concurrent requests it finishes in about ten minutes. Setting up the archive table and the re-score job is most of a day the first time and nothing thereafter.

Those numbers are worth quoting whenever somebody objects that this is a lot of process for an eval. The recurring cost of knowing whether your quality series means anything is a few dollars a month and a cron job. The cost of not knowing is a quarterly review built on a chart that inverts under a ten-minute experiment.

Which judge is right is not in the table#

The re-score establishes that the edited judge is eight points more permissive than January's on this data. It says nothing about whether January's was too strict, today's is too lenient, or the rubric legitimately changed to reflect a standard the business actually revised. All three produce the same eight points, and choosing between them needs people applying both rubric texts to the same outputs rather than more scoring.

The three-point product estimate has a second limit, which is that it is measured on January's input mix. The anchor set is frozen, which is what makes it useful, and it is frozen at a distribution that is five months old. If the product now serves a different mix of tickets, "three points better on January's traffic" is the honest label and "three points better" is not. Refresh the anchor by adding a second frozen set from a recent month rather than replacing the first, so that you keep the long baseline and gain a current one, and re-score both whenever the judge changes.

The fourth cell of that 2×2 is usually unrecoverable, and only partly for the obvious reason. Scoring May's outputs with January's judge would complete it and let you check the two deltas against each other. If the judge prompt lives in version control you can retrieve January's text and try; you will not, however, retrieve January's model, because the alias that request was addressed to now resolves to different weights. A reconstructed January judge is an approximation whose error is exactly the quantity you were trying to measure. Where your provider publishes dated snapshot identifiers, pin to one and record it with every score; where it publishes only version aliases, record the alias together with the run date and treat every provider release note as a version boundary in your series.

Running the previous judge in shadow#

There is a complementary technique that closes the gap the anchor set leaves open at the moment of change. When a new judge version goes live, keep the previous one running in shadow over the same traffic for a week or two. Both verdicts get written, only the new one is used for anything, and the overlap window gives you a direct old-versus-new comparison on current data rather than on frozen data. It costs a second judge call on whatever slice you shadow — a 10% sample is plenty — and it answers the question the anchor set answers on January's distribution, on today's. The two together let you say "the judge moved by six points, on both old and current traffic", which is a much stronger statement than either produces alone.

What neither of them does is stop the drift. An anchor set is a detector of that drift — a judge's effective standard shifting from one month to the next while its scores continue to be reported as one series — and it runs after the fact. It told you in May about edits made in February, March and April, by which point four months of decisions had already been taken on a moving number.

The experiment at the centre of this chapter is almost embarrassingly simple: score the same files twice and see whether the answer changes. What makes it worth the standing cost of an archive is that no other evidence available to you can distinguish an instrument that moved from a product that improved, and the default reading of a rising line is always the flattering one. Three hundred frozen outputs, a paired comparison, and a segment column turn eleven points into eight, two and one, each with a name attached.

Doing it in May for edits made in February is still doing it late. Making the drift visible at the moment it is introduced means treating the judge prompt as what it structurally is — the definition of the metric — and putting the same review around a two-sentence change to it as around a change to the code that computes a revenue figure.