Faster diagnosis and more confident misdiagnosis arrive together

The five-agent migration was approved on one sentence in a planning document: moving coordination onto a shared artifact will substantially reduce time to diagnose failed runs. That is a falsifiable claim about human performance, and it was made to a director who will ask about it in a quarter. Nothing in the last six chapters constitutes evidence for it. The artifact is durable, correct under concurrency, and renderable, and none of those properties entail that a person looking at it reaches the right conclusion faster than a person reading a message transcript.

The material for a real test already exists, which is what makes this worth doing rather than arguing about. The archive holds 24 failed runs from the previous nine months, each with a complete message transcript, and each replayable against the new design because the inputs and the agent versions are pinned.

Two representations of the same 24 failures#

The experiment compares representations, so everything except the representation has to be held still. Each of the 24 archived failures is replayed under the shared-artifact design until it fails in the same way, verified by comparing the final error and the set of outputs produced. That produces 24 matched pairs: one message transcript and one artifact, describing the same run failing for the same reason.

Ground truth comes first and comes from somewhere else. Two engineers who will not be observers work through the complete record of each failure — every trace, both representations, the model calls, the storage logs — and write down the blocking decision:

{
  "case": "run_7b31",
  "blocking_decision": {
    "task": "clause",
    "at_version": 34,
    "what": "clause agent marked task done with 12 of 61 outputs written",
    "category": "premature_completion"
  },
  "accept_as_correct": ["premature_completion"],
  "near_miss_not_accepted": ["lease_expiry", "downstream_report_bug"],
  "adjudicated_by": ["e.okonkwo", "s.varga"],
  "adjudicated_at": "2026-04-14"
}

Writing near_miss_not_accepted before anyone is observed is the part that keeps the study honest. Lease expiry is present in that run and is a consequence of the premature completion rather than its cause, and deciding after the fact whether it counts is how an experiment gets talked into the answer its designer wanted.

The crossover design matters more than the observer count. With eight people, half of any difference between two independent groups would be the people rather than the representation; having each person see both arms means the comparison is within a person, and each observer contributes their own baseline.

One honest limitation is worth writing into the protocol rather than discovering later. An observer reading an artifact for a failure they have already reasoned about in transcript form is contaminated, which is why no case appears twice for the same observer. What cannot be removed is that all eight observers work on this system and have seen some of these failures in production, so absolute times will be optimistic for both arms. The comparison between arms survives it; a claim about how long diagnosis takes for a stranger does not.

The result that would stop the migration#

The stopping rule was written as two conditions, either of which halts the rollout pending a redesign. The first is no reliable time improvement in the outsider stratum, which is the four observers from adjacent teams counted on their own — if only the people who built the artifact are faster with it, the design has produced a representation that requires its authors, which is the opposite of the argument for it. The second is accuracy on the artifact arm falling more than eight percentage points below the transcript arm, whatever happens to time, because a faster wrong answer at 2am is worse than a slower right one and the entire justification was about incident response.

Committing to that in advance is what makes the six-week result interpretable rather than negotiable.

Six weeks in#

Time halved, and it halved most for the people who did not build the system, which is the strongest possible version of the result the migration was promised on. Accuracy fell twelve points, which trips the stopping rule. In the same six weeks, two production incidents were misdiagnosed in ways that had not happened before: in one, an engineer concluded the report agent had a bug when the actual fault was a clause task that had completed against a truncated input, and the fix went to the wrong component and shipped.

The tempting reading is that this is early-days unfamiliarity with a new representation and will fade. It is tempting because it is sometimes true of new tools, because the metric everyone promised did move, and because it costs nothing to wait a quarter. It also explains away the one result that contradicts the design while accepting the one that confirms it, and the mechanism points the other way.

Why the artifact invites a confident story#

The artifact shows current state. Chapter 1 established that as its defining property and the reason it reads in one screen, and chapter 6 built a render that makes the reading faster still. A reader looking at run_7b31 sees a clause task marked done with an output pointer, a risk task whose lease expired, and a report that produced a document missing most of its content. That is a coherent picture, and the most natural story it supports is that the report agent mishandled its inputs.

The transitions that produced the picture are absent. Version 34, where the clause agent marked the task done with 12 of 61 outputs written, is not visible in version 88; the count field was later overwritten by a retry that wrote 61 pointers to files that mostly do not exist. The transcript arm contains that moment as a discrete message with a timestamp, which is why observers reading transcripts found it more often and took longer doing it. They were reconstructing a sequence, slowly, from something that recorded a sequence. Artifact readers were reading a state, quickly, and then constructing a sequence that would explain it.

Familiarity makes this worse rather than better. A practised reader of the render recognises the shape of the picture faster and reaches a story sooner, and the story is generated from the same incomplete evidence. The confidence column in the results table is the leading indicator: artifact observers were both less accurate and more sure, and every month of practice moves both of those in the direction that hurts. Waiting for the effect to disappear is waiting for the mechanism to reverse itself, and nothing in the mechanism suggests it will.

What changes, and what does not#

The conclusion is not that the migration was wrong. Chapter 8 will show that the artifact buys things transcripts cannot, and a 53% reduction in time to a hypothesis is real value that does not vanish because the hypothesis is sometimes wrong. The conclusion is that the representation is missing the evidence people need in order to be right, and that the missing evidence is the thing chapter 1 named as the blackboard's weakest property. Speed was never the whole target; it was the proxy the planning document happened to write down.

Three changes follow. The first is to put transitions back into the representation, using the bounded appended event list from chapter 5, and specifically to append every status change with the values that changed, so that "clause: in_progress → done, outputs 12" survives the later write that made it 61. That is roughly 40 events per run at 120 bytes, about 5 KB, and it does not turn the artifact into a log because it is bounded, structured and rendered separately from the state.

The second is to re-run the study on the new representation with the same 24 cases and observers who have not seen them, because a fix asserted is not a fix measured. The prediction is specific: artifact-arm accuracy should recover toward the transcript arm's 83% while the time advantage shrinks somewhat, since reading transitions costs something. If time regresses all the way to the transcript arm's 13.8 minutes, the added events have simply reproduced a transcript inside the artifact and the render needs work.

The third change is to the metric itself, and it applies whether or not the redesign works. Reporting time and accuracy as two numbers lets an organisation quote the flattering one, which is exactly what the planning sentence did. The measure that matches what anyone actually wants from incident response is time to a correct diagnosis, with incorrect diagnoses counted at the cost they impose rather than dropped from the average — in the two production cases that cost was a wrong fix shipped and roughly a day and a half of engineering. Computed that way, the artifact arm at six weeks is not ahead, which is a considerably more useful sentence to bring to the director than either raw number.

Diagnosis is only one of the axes the migration was supposed to move, and it is the one that turned out to be complicated. The others — whether the work gets done as well, what coordination costs, and what happens when a worker dies — are more tractable to measure, and they are where the honest case for this architecture is made.