When the score moves and the system did not

At 08:40 the quality panel reads 91.3%. It read 96.1% yesterday and has not been below 95.4% in seven weeks. The deploy log shows nothing since Thursday afternoon, the prompt registry has no revision newer than the one that has been live for eleven days, the provider status page is green, and the latency and error objectives are exactly where they were. Someone pulls twenty of the failing traces and reads them, and this is the part that makes the morning strange: the questions are ordinary, the retrieved context looks right, and the answers are the sort of thing you would accept from a colleague. The judge scored them fail.

The instinct at that point is to go looking for what changed in the system, because a five-point drop feels like it must have a cause in the thing being measured. The more productive move is to spend the first twenty minutes proving that the measurement did not change, because there are only three places a measurement can change and all three are cheap to check, whereas the search space in the system is unbounded. An evaluator is a dependency like any other, except that it ships behaviour changes without a changelog and nothing in your CI will notice.

Three suspects, checked in order of cost#

Volume first, because it is a single query and it discriminates a failure mode that looks nothing like itself. If the evaluator timed out on a fraction of samples, the SLI is computed over the survivors, and survivors are not a random subset — a judge call times out more often on long answers with large retrieved context, which are exactly the answers most likely to fail groundedness. Dropping them raises the score. Dropping them the other way round, when a schema-validation error discards every response the judge returned without a failure_reason, removes passes and lowers it. Either way the number moves without any answer changing, and the only tell is that yesterday scored 3,140 samples and today scored 1,890.

Second is the judge itself, which has two versions and both of them matter. The model version is the one people remember to pin and forget to enforce. A pinned snapshot is a dated build of the model that never changes, whereas an alias is a pointer that stands for whichever snapshot the provider currently serves under that name, so an alias resolves to a new snapshot on the provider's schedule rather than yours, and the new snapshot is stricter about whether an uncited paraphrase counts as grounded. The template version is the one nobody thinks of as a version at all. The rubric lives in a markdown file in the repo, someone tightened the wording of one criterion in a pull request about documentation, and the reviewers read it as a docs change because that is what it looked like.

Third is the sampling filter, which selects which requests become samples. A filter written as a query over trace attributes changes meaning whenever the attributes change. Adding a new route that inherits the eligibility predicate, removing a language = 'en' clause during a translation project, or fixing a bug where cancelled streams were excluded and are now included — each of these swaps in a different population under a constant objective, and the score moves for the reason chapter 2 described.

The record that makes this diagnosable#

None of that triage is possible unless each scored sample carries the identity of the thing that scored it. The scored-sample row is the artefact that makes the whole objective auditable, and it wants roughly this shape.

CREATE TABLE quality_samples (
  sample_id       uuid PRIMARY KEY,
  trace_id        text NOT NULL,
  scored_at       timestamptz NOT NULL,
  -- what was scored
  question        text NOT NULL,
  retrieved_ctx   jsonb NOT NULL,
  answer          text NOT NULL,
  -- what did the scoring
  evaluator_model text NOT NULL,   -- full snapshot id, never an alias
  evaluator_temp  real NOT NULL,
  template_path   text NOT NULL,
  template_sha    text NOT NULL,   -- content hash of the rubric file
  sampler_version text NOT NULL,
  -- the result
  verdict         text NOT NULL,   -- pass | fail | error
  failure_reason  text,
  rationale       text,
  judge_latency_ms integer,
  judge_error     text
);

Storing question, retrieved_ctx and answer is what makes a sample replayable, and replay is the only test that settles the argument. Storing verdict = 'error' as a real verdict rather than dropping the row is what makes the volume check work, because a table that silently omits failures cannot report that it omitted them.

With that table, the morning's triage is three queries. The first counts rows per day and per verdict, including errors. The second groups by the evaluator identity:

SELECT date_trunc('hour', scored_at) AS hr,
       evaluator_model, template_sha,
       count(*) AS n,
       avg((verdict = 'pass')::int)::numeric(4,3) AS pass_rate
FROM quality_samples
WHERE scored_at > now() - interval '36 hours'
GROUP BY 1, 2, 3
ORDER BY 1;

On the morning in question that query returns two distinct evaluator_model values, with the changeover at 02:00 UTC and pass rates of 0.958 before and 0.912 after. The evaluator service had been redeployed overnight as part of a routine dependency bump, and the client library's default model constant moved from a pinned snapshot to the provider's current one.

Replaying separates a changed judge from changed traffic#

Knowing the judge changed is not the same as knowing the judge caused the drop, because a judge change and a traffic change can land on the same night. The experiment that separates them holds one factor constant at a time, and it is possible because the samples table stores inputs.

Reading down a column: the same judge gives the same answer on both days' traffic, so the traffic did not change. Reading across a row: the same traffic scores 4.6 points lower under the new judge, so the judge did. If instead the left column had read 95.8% and 91.5%, the old judge would be reporting a genuine regression in today's answers and the judge change would be a coincidence you still have to unwind before you can trust any number for the rest of the week.

Re-scoring 3,140 stored samples through a judge that costs about $0.0024 a call is roughly $7.50 and forty minutes of queue time, which is the entire budget of the investigation. Doing it in the other order — pulling the retrieval index, checking the embedding model, diffing the vector store, interviewing the team about Thursday's deploy — costs a morning of several people and ends in the same place.

Reading the rationales confirms the mechanism rather than just the magnitude. Among the 143 samples that passed under the old judge and failed under the new one, 118 carry failure_reason = "unsupported_claim" and rationales that object to a sentence paraphrasing the source document without quoting it. The old snapshot treated a faithful paraphrase as supported; the new one wants the span. Neither reading is wrong. The rubric said "every claim must be supported by the retrieved context" and left the word "supported" to the judge, which is the same unanchored-criterion problem that makes any rubric sensitive to who is reading it.

The judge has a noise floor even when nothing changes#

A useful thing to establish once, before any of this happens, is how much the same judge disagrees with itself. Re-scoring 500 stored samples through the identical pinned snapshot at temperature 0 produced 9 verdict flips on this workload, or 1.8%. Temperature 0 is not determinism — it selects the highest-probability token, and floating-point non-associativity across differently sized batches on the provider's side can change which token that is when two candidates are close — so a fraction of samples sit near enough to the rubric's boundary to land on either side depending on nothing at all.

That 1.8% is the resolution limit of the instrument, and it is worth knowing because it bounds what a replay can prove. A replay that shows 95.8% against 91.2% is far outside it and settles the question. A replay that shows 95.8% against 95.1% is inside the range the same judge produces against itself, which means the judge change is not the explanation and the twenty minutes have correctly sent you back to the system. Teams that have never measured the self-agreement figure tend to treat any difference in a replay as evidence, which turns a clean diagnostic into a coin flip.

Treating the evaluator as a versioned dependency#

Once the cause is understood the fix is not to roll back and forget it, because the old snapshot has a deprecation date and this will happen again on the provider's schedule rather than yours. The practices that make it survivable are the ones you would apply to any dependency whose behaviour is load-bearing.

Pin the judge to a dated snapshot and fail the evaluator's startup if the configured model is an alias, so that the pin cannot be lost by a well-meaning dependency bump. Hash the rubric template and store the hash on every sample, so that an edit to the file is visible in the data rather than only in git. Keep a calibration set of a few hundred human-labelled examples and require any candidate judge to be measured against it before promotion — on the set behind the numbers above, the old snapshot agreed with the human labels on 79% of examples and the new one on 71%, which says the "stricter" judge is not more accurate, only more severe. And run a judge migration the way you would a database migration: dual-score every sample with both judges for a week, publish both series, choose the changeover point deliberately, and annotate the dashboard at the boundary so that nobody in a quarterly review compares a figure from before to a figure from after.

The dashboard needs the same discipline as the data. Every evaluator promotion, template edit and sampler change should land as an annotation on the quality chart at the timestamp it took effect, drawn from the same version fields the samples carry rather than typed in by hand. Without it, the first question in any review — did something change around here — is answered from memory by whoever was on call, and the answer is wrong often enough to send an investigation down the wrong path for a day.

There is a limit to how well this can be made to work. A hosted judge is a model you do not control on a lifecycle you do not set, and snapshots are retired on twelve-month timescales, so a quality time series longer than about a year will contain at least one discontinuity that is an artefact of the instrument. Self-hosting an open-weights judge removes the drift and hands you a serving problem, a GPU bill and a model that is generally weaker at the task; teams with long-running quality series and enough volume do make that trade, and most do not. The honest position is that a quality SLI is comparable within a judge era and needs an explicit bridge across one — which is a re-score of a retained sample set, published as such.

Where this triage does not apply#

Checking the measurement first is a heuristic about cost, not a claim that the measurement is usually at fault. Roughly, it earns its place because measurement changes are cheap to rule out and produce step changes, which is what a five-point overnight move looks like. A quality regression from a real cause more often arrives as a slope over days as a traffic mix shifts, or as a step confined to one slice rather than the aggregate. When the volume, the versions and the sampler all check out, the twenty minutes were still well spent, because every subsequent hypothesis about the system is now being tested against a number you have reason to trust.

The triage also cannot see the failure that never reaches the aggregate at all. Everything in this chapter assumes the score moved; the measurement was intact, the versions were pinned, the sampler was stable, and the number went down. There is a worse case in which every component of the objective is working exactly as specified, the reported figure is correct to two decimal places, and a customer has been receiving unusable answers for six weeks. That is what the next chapter is about.