A study nobody shipped is worth nothing and decays

The support-triage comparison finished in June with a 5.4-point advantage to the loop, a cost per ticket of $0.031 against $0.142, and a p95 of 8.4 seconds against 31.2. The write-up was circulated, three people replied approvingly, and the five-stage pipeline kept serving every ticket. In October the engineer who ran it had moved to the payments team, evals/runs/ had no entry newer than 12 June, and the loop arm's last commit was two days before the study finished. Somebody asked in a planning session whether the team should just do the migration now, given that the work of deciding had already been done.

What has already been done is a measurement of two systems that no longer exist. Asked what the study is worth, the answer is nothing operationally, because no ticket has been handled differently as a result of it, and its evidentiary value has been decaying on both sides for four months.

Both arms moved, in opposite directions#

The pipeline moved because it is in production and production systems get worked on. Twenty-three commits touched its five prompts over the four months, mostly narrowing the policy-check stage after a refund incident in August. The classify stage gained a sixth category when a new product line launched. Two of the four tools changed their response schemas, and the retrieval tool's default result count rose, which the pipeline's stages absorbed because they are exercised daily.

The loop moved by standing still while everything around it changed, which is the worse kind of movement because it is invisible. Running the June loop arm against today's tools in October produced hard errors on 12% of tickets — one tool now returns a nested entitlements object where it previously returned a flat list, and the loop's prompt describes the old shape. Nobody noticed, because nothing runs it.

Two of those rows favour the pipeline, one plausibly favours the loop, and three have unknown sign. Netting them out by argument is not possible, and that is the whole content of the claim that the study has decayed: it is not that the loop probably still wins or probably does not, it is that nobody can say, and the only instrument that could say has been switched off since June. Migrating production on the June numbers would be a change justified by evidence about a system that has been superseded, which is the same standing as the pipeline's own justification, and swapping one unevidenced architecture for another is not progress even if the new one happens to be right.

A quarter, in the order that makes each step meaningful#

The ordering matters more than the contents, because several of these steps are worthless if taken after one of the others.

Week one contains the step people are most tempted to skip, which is writing the decision rule before running anything. Skipping it in a re-run is more dangerous than in a first study, because everyone already believes they know the answer, and a threshold chosen in week three will be chosen to agree with June.

The last two phases are the standard pair for putting a changed system in front of real traffic without betting on it. Shadowing sends live requests to both the production system and the candidate, keeps the production answer and throws the candidate's away after scoring it, so the candidate is measured on real traffic while affecting nobody. A canary release then routes a small share of traffic to the candidate for real, growing the share in steps and watching the same metrics at each one, so a problem is discovered by a few per cent of users rather than all of them. The shadow phase in weeks four to six is the part that a purely fixture-based process leaves out, and it is where the fixture set gets checked against reality. Held-out tickets are held out from tuning; they are not held out from the fixture set's own sampling assumptions, and a 30-day sample taken in week one still misses whatever arrives in week five. Shadowing 10% of live tickets for three weeks produced roughly 4,100 scored comparisons at $0.031 each, about $130, and it caught something the harness had not: the loop's success on the new product line's tickets was 6 points below its overall rate, because those tickets reference a knowledge base the loop's single prompt does not mention. That is a fixable gap found before any customer saw it, and it would have been found by the canary too, at the cost of a worse week for whoever was on call.

Every dotted edge in that diagram returns to the harness rather than to a debate, which is only possible because the tripwire thresholds were written into the ADR in week three and wired to metrics before the first canary percent. A rollback that has to be argued for is a rollback that happens two days late.

Making the check something other than a person's intention#

The failure that produced this situation was not analytical. Nobody misread a number; the study simply had no next step attached to it and no owner after the author changed teams. The structural fix is that the comparison runs on a schedule, and that the reversal conditions in the ADR are assertions in that job rather than sentences in a document.

name: architecture-rebaseline
on:
  schedule: [{ cron: "0 3 1 */3 *" }]   # 03:00, first of the month, quarterly
  workflow_dispatch:

jobs:
  rebaseline:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: evals run --manifest evals/manifests/triage-current.json --trials 5
      - run: evals assert --adr docs/adr/046.md --conditions R1,R5
      - if: failure()
        run: |
          gh issue create \
            --title "ADR-046 reversal condition fired" \
            --body-file evals/runs/latest/summary.md \
            --assignee "$(yq '.owner' docs/adr/046.md)" \
            --label "adr-review,p1"

The assert step reads the thresholds out of the decision record itself, so the document and the check cannot drift apart. Cutting the issue with an owner and a priority label is what converts a fired tripwire into work that appears in a planning session, and the fifteen-day deadline from the record is what stops that issue from ageing. Running quarterly rather than nightly is a deliberate economy: the re-baseline costs about $190 and twenty minutes, the full architecture study with the boundary ladder cost roughly $1,100 and a day of analysis, and the cheap check exists to tell you when the expensive one is worth repeating.

Two things are worth not changing. Trace reading stays manual, because attributing a failure to a boundary requires judgement that no scorer reproduces, and the value of chapters four and five came from someone opening traces rather than from a metric. And the bar stays where it is: the temptation after a study stalls is to make the next one lighter by dropping the held-out set or the audit round, which produces faster studies whose conclusions cannot be trusted, and a stalled study that was rigorous is a better position than a shipped one that was not.

What continuous evaluation does not fix#

A scheduled job keeps the comparison current and lets the fixture set go stale on its own schedule. Eighty tickets sampled in January describe January's traffic, and by July the product mix has moved again; the job will keep producing green results against a set that no longer resembles what arrives. Refreshing the fixtures automatically from recent production traffic solves that and quietly destroys the held-out property from chapter three, unless the refresh is blind — sampled and labelled by someone who is not reading failure traces, on a cadence unrelated to whether anyone is unhappy with the current numbers.

Scheduling also does nothing about the case where both arms drift toward the same failure because they share a tool that has degraded, since the paired design cancels exactly that. A comparison job and a production regression suite answer different questions, and chapter two's argument about pinning applies here as its practical consequence: keep both jobs, and do not let the pinned comparison stand in for a suite that tracks whether today's system still works.

The last unsolved piece is the cost of the whole apparatus relative to the decision it informs. A quarter of shadow and canary work, a maintained losing arm, a labelled fixture set refreshed twice a year and a day of trace reading per study is a real ongoing expense, and it is obviously worth paying when the two candidate architectures differ by 4.6× in cost per task at production volume. It is much less obviously worth paying for a system handling forty requests a day. The judgement about when to run this process at all is one the material does not settle, and the honest heuristic is to compare the annual difference between the arms — in spend, in latency budget, in incident load — against the cost of the harness, and to run the light version until that comparison justifies the full one.

The place to start on the next comparison is week one of that table, before either arm exists: the decision rule, the fixture set and the manifest come first, because they are the parts that everyone intends to write afterwards and nobody does. Anthropic's Demystifying evals for AI agents is a good companion for the harness itself, and Evaluating AI Agents: A Production Blueprint with Strands and AgentCore carries end-to-end code for scoring runs on success, cost and latency if you would rather adapt something than start from an empty file.