The evaluation audit trail

An evaluation harness that prints a percentage answers one question, and it is the question nobody asks the morning after the number moves. What gets asked then is which items changed, whether the judge is the same judge, and whether yesterday's answers would score differently today — and each of those needs something the harness either wrote down or did not.

This is the least interesting infrastructure in the topic and the one that decides how the rest of it plays out. Tracing an injection means asking which source documents the affected items shared. Reward hacking is traced by watching output length and rubric vocabulary move across generator versions. A rubric edit is caught by re-scoring a frozen set under two configurations. None of those are possible against an aggregate, and all of them are a query against a per-item record.

What one row per run cannot answer#

Here is the schema most harnesses start with, and it is the schema they keep until an incident.

CREATE TABLE eval_runs (
  run_id      uuid PRIMARY KEY,
  suite       text,
  started_at  timestamptz,
  n_items     int,
  pass_rate   numeric
);

Which items flipped? Unknown, because per-item results were aggregated at write time. Did the judge change? Unknown, because the judge's identity was never recorded, only implied by whatever was in the repository at the time. Could yesterday's outputs be re-scored under today's judge? No, because yesterday's outputs were discarded after scoring. Every question worth asking is unanswerable from that table, and the answer a team ships instead is a footnote.

The reason those are permanent rather than inconvenient is worth stating once, because a team that has not stated it will keep proposing to recover the information later. pass_rate is a function of the per-item verdicts, and it is a function that throws information away: a very large number of different verdict patterns produce the same percentage, and nothing about the percentage says which of them occurred. That direction cannot be run backwards, not with more compute and not with a cleverer query, because the distinguishing information was never written. The same goes for the judge's identity, which was not lost but never captured — the run inferred it from whatever the repository happened to contain and recorded nothing, so there is no artefact anywhere that says which judge produced the number. Aggregation at write time is therefore not a storage decision that can be revisited. It is a decision about which questions the system will be able to answer for the rest of its life, taken by whoever wrote the first version of the schema, usually without noticing that they were taking it.

One row per graded item#

The alternative carries everything needed to reconstruct both sides of the comparison.

{
  "run_id": "0b6f…", "item_id": "sup-1183", "graded_at": "2026-02-10T02:14:07Z",
  "harness_sha": "9c41a77",
  "dataset_version": "support-eval-2026-01-b",

  "generator": {
    "model": "claude-sonnet-4-5-20250929",
    "prompt_sha256": "3f2a…", "prompt_ref": "prompts/gen/support@v11",
    "temperature": 0.2, "max_tokens": 800, "seed": 4471
  },
  "judge": {
    "model": "claude-sonnet-4-5-20250929",
    "prompt_sha256": "c81d…", "prompt_ref": "prompts/judge/faith@v4",
    "rubric_version": "faith-v4", "temperature": 0.0
  },

  "input_sha256": "aa71…",
  "output_text": "## Summary\nYour refund window for orders placed…",
  "output_tokens": 312,
  "retrieved_doc_ids": ["kb-441", "kb-98"],

  "verdict": "PASS",
  "evidence": "refunds may be requested within 30 days of delivery",
  "tool_calls": [],
  "injection_probe": {"family": null, "flipped": null}
}

Two fields do most of the work and are the two most often omitted. The full output_text, stored rather than hashed, is what makes re-scoring possible and is what a human reads when the automated evidence runs out. The judge.model field holding a dated snapshot rather than an alias is what makes "did the judge change?" a lookup instead of an investigation; pinning the alias in config and recording the alias in the log records that you did not know.

The rest of the record earns its place in specific investigations. retrieved_doc_ids is what lets you ask whether the flipped items share a source document, which is the query that would have found the logistics customer from chapter one — the account whose contract template carried an evaluator note into every summary of it — in seconds instead of a quarter. tool_calls, stored with their arguments and reason codes, is where a fetch to an unexpected host shows up attached to the item that caused it, rather than in an egress log with no way back to a grading call. input_sha256 distinguishes "the dataset changed" from "the answers changed", which is a surprisingly common confusion when several people can add fixtures. And injection_probe carries the fixture family for the adversarial rows, so the flip rates from the earlier chapters live in the same table as the ordinary scores and can be reported next to them instead of in a separate spreadsheet nobody opens.

The volume argument against storing outputs does not survive arithmetic. At 3,100 items a night and roughly 4 KB per record including the answer text, a run is 12 MB, a year is 4.4 GB, and object storage prices that at cents per month. Both steps are worth doing in the open, because the objection is always raised as an order-of-magnitude worry rather than a calculation and an order-of-magnitude worry is answered by a calculation. Three thousand-odd records at four kilobytes each is a bit over twelve thousand kilobytes, which is twelve megabytes — one photograph. Repeat that every night for a year and you have twelve megabytes taken about three hundred and sixty-five times, which is a little over four gigabytes: less than a laptop's spare capacity, for the entire evidentiary record of a year of evaluation. Teams discard outputs because the first version of the harness printed a percentage and nobody revisited it, not because anyone weighed the cost.

Narrowing a jump to a list of items#

With per-item records the morning after becomes a procedure rather than a debate.

The re-score is the discriminating experiment, and it is worth being precise about why it discriminates, because that is the transferable part. A measured number here is produced by two things: the outputs, and the judge that scored them. When both change between Monday and Tuesday, a difference in the number is consistent with either having moved, and no amount of staring at the two percentages separates them. Re-scoring Monday's outputs under Tuesday's judge constructs a third measurement in which one of the two is deliberately held still. Monday's outputs are fixed — they are bytes on disk, and bytes do not drift — so any difference between what they scored on Monday and what they score now has exactly one thing left that could have caused it. That is the whole logic, and it is the reason storing output_text rather than a hash is the field the schema turns on: a hash proves Monday's outputs were what you think they were, and only the text itself can be fed to a different judge.

The experiment is also cheap: 3,100 stored outputs through today's judge is about twenty minutes and a few dollars, with no generator calls at all because the outputs already exist. If the older answers score 86% under the newer judge, the answers did not change and the judge did. If they still score 78%, the judge is stable and the change is in what the generator produced, which the second branch resolves by reading. Forty flipped items is enough to tell "this answer now cites the right policy section" from "this answer says the same thing under three headings with the hedges removed", and reading them takes half an hour.

Before either branch, the cheapest thing per-item records give you is the set of items that moved. Joining one night's rows to the next on item_id and filtering to where the verdict differs turns nine aggregate points into a concrete list — on one such run, 291 items flipped FAIL to PASS and 21 flipped the other way — and the list is immediately informative. Flips concentrated in one dataset slice point at retrieval or at a document; flips spread evenly across every slice point at the judge, because a shift in the measuring device does not respect topic boundaries. Two minutes of SQL narrows the search space before any model is called.

Running the re-score against the older judge as well gives you the noise floor, meaning how far the aggregate moves when nothing has changed at all, which matters because none of this is exactly reproducible. Even at temperature 0.0, provider-side nondeterminism moves individual verdicts on borderline items; on this suite, re-scoring the same outputs with the same judge configuration twice disagreed on 2.1% of items and moved the aggregate by 0.4 points. A nine-point jump is twenty times that, so the comparison is safe, and a 1.5-point jump would not have been.

Temperature at zero not producing identical output is the belief this measurement exists to correct, and it is a reasonable belief to hold, because temperature zero does remove the sampling step: the model takes the highest-scoring next token rather than drawing from a distribution. What it does not remove is variation in the scores themselves. The provider batches your request with other traffic, and how the arithmetic is grouped across a batch depends on what else is in it, and floating-point addition performed in a different order does not give bit-identical results. On most tokens the gap between the leading candidate and the next one is far larger than that difference and the choice is unaffected. On a borderline item it is not, and the verdict flips. Which is why "deterministic" is the wrong word for this configuration and "stable except on ties" is the right one — and why a noise floor has to be measured rather than assumed to be zero.

The two figures in that measurement look inconsistent and are not, and the reconciliation is the useful part. Two point one percent of 3,100 items is around sixty-five items whose verdict differed between the two runs, yet the aggregate moved by 0.4 points, which on the same suite is around a dozen items' worth. The difference is direction. A flip from fail to pass raises the percentage and a flip from pass to fail lowers it, so sixty-five flips split roughly evenly cancel down to a small net, and the net is what an aggregate reports. That has a consequence for monitoring worth carrying: the aggregate understates how much churn there is underneath it, so a suite whose pass rate has not moved may still be disagreeing with itself on dozens of items, and only the per-item join finds them.

No single column separates all three, which is why the record has to support all four checks. The held-out judge column is the one that distinguishes the first row from the third, and it costs another twenty minutes of scoring against a model family the harness never selects on.

A rubric edit is a change to the metric#

The incident has a fix, and it is a process one. A rubric lives in a text file in the same repository as the harness, so it is reviewed like documentation — someone reads the wording, agrees it is clearer, and approves. Nothing in that review surfaces that the file is the definition of the number the team steers by, and that editing it re-bases every comparison against every previous run.

Treating the rubric and judge prompt as instruments means a change to either has to come with a measurement of its effect, and the measurement is mechanical: keep a frozen set of stored outputs — a few hundred items, spanning passes, fails and the adversarial fixtures — and re-score that set under both the old and the new configuration whenever either changes.

$ judgectl rescore --frozen-set frozen-2026-01 \
      --from prompts/judge/faith@v4 --to prompts/judge/faith@v5

  items                600
  pass rate  v4       76.2%
  pass rate  v5       85.5%   (+9.3 pts)
  flipped FAIL->PASS   61
  flipped PASS->FAIL    5
  injection flip rate  11.7% -> 14.2%   (+2.5 pts, evaluator_note family)

  gate: FAIL — metric shift exceeds 2.0 pts without a recorded rationale

Read the flip counts before the pass rates, because they say something the percentages do not. Sixty-one items moved from fail to pass and five moved the other way, on a frozen set of 600. The edit is not adding scatter in both directions — it is pushing verdicts one way, twelve times as often as the other, which is the signature of a threshold that has moved rather than of an instrument that has become noisier. Net, that is fifty-six items of 600, a little over nine in a hundred, which is exactly the 9.3-point shift the tool reports. A gate set at two points is therefore not a tight tolerance; it is five times the noise floor measured earlier in this chapter, chosen to sit above ordinary churn and well below anything a wording change plausibly does by accident.

The gate does not forbid the edit. It requires that the shift be stated, so that the graph carries an annotation at that date and the nine points are attributed to the rubric rather than to the product. The same run reports what the edit did to the injection flip rate, which is how a wording change that softened the rubric from "supported by" to "broadly consistent with" gets caught making the judge more suggestible as well as more generous. Two-tenths of a percent of the effort of the original investigation, spent before the change lands rather than after.

Version the judge model the same way, with the wrinkle that you do not control when it changes. Pinning a dated snapshot rather than an alias converts an unannounced shift into a deliberate upgrade, and the upgrade then goes through the same frozen-set comparison — which is the only way anyone finds out that a new snapshot scores your suite four points differently before it has already been reported as product progress.

The recorder is not the investigation#

An audit trail is the flight data recorder of an evaluation harness: it does not prevent anything, it does not explain anything on its own, and its entire value is that the questions asked afterwards have answers. The distinction matters because teams that add per-item logging sometimes stop there, satisfied that they now have observability, and observability of a metric is not a defence of it. Stored records do not stop a rubric edit from shifting nine points of measured quality, and they would not stop a customer document from flipping verdicts. What they do is make the difference between an afternoon and a fortnight.

The trail also has to be written by something outside the loop it observes. If the run that produces the scores is also the process that decides which fields to store, then a harness change can quietly stop recording the field that would have exposed it — the alias example is the mild version of this, where the config recorded a name that pointed at different weights on different days. Recording the resolved snapshot at request time, from the provider's response rather than from your config, is the fix, and it generalises: prefer the value the system actually used over the value you asked for.

There are limits worth being honest about. Storing raw outputs means the eval store now holds model answers derived from customer documents, which puts it inside the same retention and access rules as production data; a store that was casually readable by the whole engineering organisation needs revisiting before it becomes a secondary copy of the corpus it was built from. Reproducibility is partial, because a snapshot you pinned can be deprecated out from under you on a provider's timeline, typically with six to twelve months of notice, after which yesterday's judge cannot be re-run at all — which is an argument for storing the judge's per-item verdicts and justifications rather than assuming you can regenerate them.

The cost of this record is a few gigabytes a year and a schema nobody enjoys writing. What it buys is that every failure in this topic becomes a query instead of an argument: which documents the flipped items shared, whether output length and rubric vocabulary moved with the score, whether the judge that produced Monday's number is the judge that produced Tuesday's. None of those questions can be reconstructed later from an aggregate, and all of them are asked at the worst possible moment, when a chart is already in a deck.

Two of the three explanations in the table above have now appeared in this topic in their own right, and the third is ordinary progress. The stored evidence separates them because each leaves a different pattern in the records, and those patterns are worth stating directly: what injection looks like in an eval store, what reward hacking looks like, and why a fix aimed at one does nothing for the other.