Judge versioning

A score produced by a judge is a joint fact about the output and the instrument that read it. That is uncontroversial and almost universally ignored, because the instrument is a prompt in a repository rather than a device on a bench, and prompts get improved by people who are right to improve them. The improvement is real; what it also does is make every score recorded before it incomparable with every score recorded after.

The chain from an edit to a broken number has four links, and each one is checkable on its own. A judge is a model call, so the verdict it returns is a continuation of a piece of text, and that text is the system prompt, the rubric, the output schema and the output under review, concatenated. The model scores candidates for what comes next conditioned on the whole of that text rather than on some designated part of it, so an edit anywhere in it changes the input the verdict was computed from. A function handed a different input is answering a different question, which means the mapping from output-under-review to verdict is a different mapping after the edit than before it. Two pass rates recorded under the two mappings are readings taken by two instruments, and their difference contains both the change in the product and the change in the instrument with nothing in the data saying how much of it is which.

The belief that keeps this invisible is that a judge prompt is configuration. Configuration is a thing you tune until it is right, and improving it is unambiguously good in the way that fixing a bug is unambiguously good. That reading is tempting for a solid reason: the edit usually is an improvement, and the person making it is right to make it. What the framing misses is that accuracy and comparability are separate properties of an instrument, and a change that raises the first destroys the second. A prompt is not a setting the judge consults. It is the judge.

There is a comparison worth making here, and it is offered as an analogy rather than as a description of the mechanism. Reading a room's temperature in Celsius one month and in Fahrenheit the next is not evidence about the weather, because the two numbers came off two scales and most of their difference is a fact about the scales. Editing the judge prompt does that to a quality series. The analogy breaks at the place that matters most: Celsius and Fahrenheit are related by a formula anybody can apply in either direction, so a mismatched pair of readings is recoverable on paper. Two judge prompts are related by no formula at all, which is why the repair below is not a conversion but a re-run.

The discipline that survives this has three parts. Store the raw model outputs, not just the scores, so a historical number can be recomputed rather than argued about. Attach a fingerprint — a hash over every input that can change a verdict — to every scored row, so that the boundary between two instruments is visible in the data rather than in somebody's memory of a commit. And run a procedure when the fingerprint moves, rather than trying to avoid moving it.

Re-score, don't reason#

Take a quality series that closed one quarter at a mean answer-quality score of 3.8 and the next at 4.1, with a judge-prompt edit somewhere in between. The raw model outputs from the earlier quarter were still in object storage — 4,100 of them, question and answer text, one JSON line each. Running the current judge prompt over all of them cost about $9 and eleven minutes:

Q3 scored with the Q3 judge (v4)     3.81
Q3 scored with the Q4 judge (v5)     4.05      <- the comparable number
Q4 scored with the Q4 judge (v5)     4.12

Three rows, and their arrangement is what makes them readable: each row names both what was scored and what did the scoring, so any two rows differ in exactly one of those two things. The first two rows hold the outputs fixed — the same 4,100 pieces of Q3 text, unchanged — and swap the judge, so their difference is the instrument and nothing else. The last two rows hold the judge fixed at v5 and change the quarter, so their difference is the product and nothing else. The quarter-end pair of 3.8 and 4.1 that started the argument changes both at once, which is why it cannot attribute its own movement to either.

Subtracting along each axis separately gives the two facts. The product moved 0.07, which is 4.12 minus 4.05: same instrument, two quarters. The judge moved 0.24, which is 4.05 minus 3.81: same text, two instruments. The raw quarter-end gap is about three-tenths of a point and the part of it the product earned is 0.07. Any slide built on the raw comparison is wrong by roughly a factor of four, and the edit that caused it was a good change — it made the judge more accurate, and being more accurate meant scoring the same outputs differently, which is the whole point and also exactly what breaks the time series.

There is no way to derive 4.05 from 3.81 and a description of the prompt edit. You cannot reason about how a wording change propagates through a judge's verdicts; you can only run it.

That deserves a mechanism rather than a warning, because "you can only run it" sounds like caution and is in fact a structural fact about the edit. A judge's verdict on any single item is decided by which side of a boundary that item falls on, and a wording change moves the boundary by an amount that depends on the item. Items well inside the pass region do not move. Items well inside the fail region do not move. Items sitting near the boundary flip, and which of them flip depends on how the new wording interacts with the specific text of each output. So the aggregate effect of an edit is a sum over an unknown subset of items, and the only way to learn which items are in that subset is to score all of them and look. A description of the edit can tell you which direction the boundary moved. It cannot tell you how many items were standing in the strip it swept across, and that count is the entire size of the effect.

That is what makes storing the raw outputs the precondition for everything else here — a pipeline that keeps scores and discards the text it scored has thrown away its ability to correct itself, and the loss is invisible until the day someone asks a question like this one.

What a judge version is#

The judge is not the prompt. It is every input that can change a verdict, and each of them belongs in a fingerprint that rides on every score you record:

import hashlib, json

def judge_fingerprint(cfg: dict) -> str:
    material = {
        "system_prompt": cfg["system_prompt"],
        "rubric": cfg["rubric_text"],          # the file contents, not its path
        "schema": cfg["output_schema"],        # enum changes alter the verdict space
        "model": cfg["model"],                 # exact id, never a floating alias
        "effort": cfg["effort"],
        "protocol": cfg["protocol"],           # pointwise | pairwise
        "tie_policy": cfg.get("tie_policy"),
        "aggregation": cfg["aggregation"],     # how criteria roll up to a score
    }
    blob = json.dumps(material, sort_keys=True).encode()
    return hashlib.sha256(blob).hexdigest()[:12]

Two choices inside that function carry most of its value. The material is hashed rather than version-stamped, because a version string is a claim a person makes and a hash is derived from the thing itself. Someone who edits the rubric and forgets to bump a version leaves two numbers agreeing while the instruments behind them differ, and a hash over the rubric's bytes cannot make that mistake because there is no step at which anybody could forget. The rubric goes in by contents and not by path for the same reason: a path is stable across exactly the edits that matter, so hashing it yields a fingerprint that stays put while the instrument moves underneath it. The comment on that line is the whole argument.

sort_keys=True is doing a third job, and it is the sort of detail that only announces itself once. A hash is taken over bytes, so two dictionaries holding identical material would produce different digests if their keys happened to serialise in a different order, and the same judge would then look like two instruments — breaking a series that never actually broke. Sorting the keys makes the byte sequence a function of the material alone, which is the property the whole scheme rests on.

Every scored row carries that fingerprint:

{"item_id": "q3-01881", "score": 4, "criteria": {"names_setting": "pass",
 "states_limitation": "pass", "no_invented_nav": "fail"},
 "judge": "a3f19c2b74e0", "scored_at": "2026-11-14T09:22:03Z",
 "output_ref": "s3://evals/outputs/q3-01881.json"}

With that field present, the dashboard query becomes mechanical: group by fingerprint, and refuse to plot two groups on the same axis. A trend line that spans a fingerprint boundary either gets re-scored or gets a visible break in it. This is the same discipline as rotating a frozen pairwise anchor, applied to the instrument rather than to the baseline, and it fails the same way when it is done silently.

The list of things that invalidate history is longer than most teams assume:

The row without a checkmark is the one that bites, and its lack of a checkmark is structural rather than an oversight in the code above. A fingerprint can only hash material the pipeline is able to read, and every item in that function lives in your repository. A floating alias is a name the provider resolves to a set of weights at request time, so the string your config holds — the only part of the arrangement available to be hashed — is by definition unchanged when the resolution behind it changes. Pointing the judge at a floating alias means the weights underneath can change without anything in your repository changing, and the fingerprint stays identical while the verdicts move. Hashing more of your own configuration cannot reach it, because the thing that moved was never yours.

Pinning the exact model id closes most of it, by making the name resolve to one set of weights rather than to whichever set is current. What closes the rest is a canary: 100 items with frozen expected verdicts, re-scored nightly with the current fingerprint, alerting when the verdicts move while the fingerprint has not. The two halves of that alert are what make it readable. Because every input you control is pinned and hashed, a moved verdict has exactly one candidate cause left, and it is on the provider's side of the wire. The same canary run against an unfingerprinted judge would fire on your own edits as well, and an alert that fires for two unrelated reasons gets muted within a week. That is about a dollar a night and it is the only mechanism that catches drift you did not cause.

Changing the judge without losing the history#

The judge will change. The rubric accumulates rulings from every disagreement session, a probe set catches a bias worth fixing, the model you pinned gets deprecated on a schedule you do not control. The discipline is not avoidance; it is having a procedure that runs when the fingerprint moves.

The procedure has three steps. Before the change lands, score a fixed re-score set with both the outgoing and incoming judge. Publish the difference between the two as an instrument change, with its own row in the changelog, separate from any product movement. Then either re-score the full history and restate it, or — when that is not affordable — leave the history intact, mark the boundary, and carry the measured offset alongside it.

Whether to restate everything is arithmetic. Four quarters of this workload is about 16,000 stored outputs; re-scoring all of them costs $35 and forty-five minutes, so there is no argument for not doing it.

Those two costs are the same cost seen at two scales, and checking that they line up is a cheap way to know the estimate was measured rather than guessed. Re-scoring one quarter's 4,100 outputs came to $9. Four quarters is about 16,000 outputs, near enough four times as many, and $35 is near enough four times $9. The cost of a re-score is dominated by the tokens the judge has to read, and the tokens scale with the number of outputs, so a total that grows in step with the count is exactly the shape to expect. A quoted figure that did not grow in step would mean something other than reading was being charged for, and would be worth querying before it was budgeted.

A pipeline scoring two million outputs a year is a different case from these sixteen thousand, and there the re-score set is the whole mechanism: 400 items, stratified across quarters and across the criteria most likely to be affected by the edit — drawn in fixed quotas from each group rather than at random from the whole, so that every group is represented — scored twice, for about two dollars.

The tempting shortcut at that point is to take the mean difference on the re-score set and add it to every historical point. Do not. A worked example added to a judge prompt to fix its handling of partial answers affects partial answers specifically, and its effect on items with no plan restriction in play is approximately zero — so the offset is 0.61 on one stratum and 0.02 on another, and a single scalar applied to the whole series would misstate both.

Setting those two figures side by side shows what the shortcut actually does. A mean over the re-score set lands somewhere between 0.61 and 0.02, at a position decided by how many items of each kind happened to be drawn. Added to every historical point, that mean overstates the correction everywhere the edit had no purchase and understates it everywhere the edit did the work, and the size of the error on each group is the distance from the mean to that group's own offset. Neither group ends up carrying a number that was measured for it. The mean is also a fact about the composition of the re-score set rather than about the edit, so a set drawn in different proportions would produce a different correction for the same unchanged history. The shortcut is the same error as correcting a whole win rate by an averaged position tilt: an effect concentrated in a subpopulation, smeared uniformly over a population where it does not apply. Report the offset per stratum, and if the strata are not stable enough to do that, report that the series is not restatable and break it visually instead.

Two naming rules keep the boundary legible afterwards. Version identifiers are appended, never reused — a judge v5 that gets a small fix becomes v6, because "v5" already labels numbers sitting in a database. And restated points are marked as restated on the chart, because a historical value that changed between two viewings of the same dashboard, with no annotation, destroys more trust than the original error did.

Fifteen points, three explanations#

Versioning earns most of its keep on a situation it does not itself resolve. Suppose a judge has gated releases for six months and reports 91% pass, a blinded human review of 60 sampled outputs two months ago came out at 89%, and a fresh blinded review this week, on 60 different outputs, comes out at 74% while the judge still says 91%. Three explanations fit, and they are not equally likely to be checked.

The product regressed and the judge cannot see it. Something shipped whose failure mode sits in the judge's blind spot — longer, more confident answers whose errors read as competence. Real quality fell 15 points; the judge's reported rate barely moved because its recall on this failure type is near zero.

The judge drifted lenient. A prompt edit, a rubric tweak, or a silent model update made it pass things it used to fail. Real quality is roughly flat; the judge's 91% is now measuring something looser than it was, and the humans are correct in both reviews.

The human measurement changed. Different raters, a rater who reinterpreted a criterion, or a sample drawn from a different slice of traffic — a week containing a marketing push that skewed the question mix harder. The judge is fine, the product is fine, and the 74% is an artefact of how it was produced.

Ordering the checks by plausibility is the natural instinct and it is the expensive one. The most plausible explanation is usually the regression, and checking a regression means diffing everything that shipped across two months — days of work whose honest first step is to rule out the other two explanations anyway. Ordering by cost and decisiveness inverts that. A check earns an early slot when it is cheap and when its answer eliminates a branch outright rather than nudging a belief, and the first two checks below have both properties because the evidence they need was recorded before anybody thought to ask the question.

The order to check them in is by cost and decisiveness, not by plausibility:

The first two steps are free and take minutes, because the fingerprint log and the canary set already exist if the earlier discipline was followed. The third costs a few rater-hours and is the one that distinguishes "the world changed" from "our measurement of it changed" — re-labelling old outputs with this week's raters, shuffled in with this week's sample so nobody knows which is which. Only after all three come back clean does the expensive investigation start, and by then it has a real target: something shipped between two dates and the outputs differ.

Note what the diagram refuses to do. It never treats "the humans are unreliable" as a conclusion you can reach by asserting it. That explanation requires the same evidence as the others — the re-labelling step — and a fifteen-point gap on a 60-item sample is far outside what rater noise produces on its own. A binomial interval is the range of true pass rates that could plausibly have thrown up an observed figure from a draw of a given size, which is the standard way of asking how much of a number is sampling luck; the interval on 74% with n=60 is roughly ±11 points. Even the widest reading of sampling error therefore leaves most of the gap unexplained, which is what makes "the humans are unreliable" an explanation that still owes evidence rather than one the arithmetic has already supplied.

Refitting the judge to the reading you wanted#

Whichever explanation holds, an arrangement that let a gap like that run for two months undetected has to change. The judge stops being the sole gate for the category where the divergence appeared; a standing human sample becomes a recurring control with an alert on divergence rather than an occasional exercise; and the judge version is frozen between calibrations, so that any movement in the metric has exactly one candidate cause at a time.

Three things do not change, and holding them is harder than making the changes. The human review does not get relaxed to bring it closer to the judge, because the judge is the approximation and the humans are the standard it approximates. The control does not get retired for being expensive, because its expense is the price of knowing whether any of the other numbers mean anything. And the judge prompt does not get re-tuned until it reproduces 91% — that is fitting the instrument to the reading you wanted, and it is the failure this whole topic is about, arriving in its most reasonable-sounding form.

A judge version, then, is not metadata. It is the second half of every score you have ever recorded, and a series that does not carry it is a series in which a correct improvement to the instrument is indistinguishable from a change in the product — a distinction that cost one quarterly slide a factor of four and would have cost nothing to preserve. Storing the outputs, hashing the inputs and running a canary are between them perhaps a day of work and a dollar a night.

The operating position all of this points at is narrow and worth stating as a stance. A judge is an instrument you have characterised: you know what it agrees with people about, where it disagrees, which direction its errors lean, and which version produced every number in your history. Run one you have not characterised and you have not automated evaluation — you have automated the production of numbers, which is a different and much easier thing, and the numbers will be stable, cheap, and confidently wrong in a direction nobody has measured.