Zero deadlock incidents can mean the label moved

The reliability review has one slide about the intake pipeline. Deadlock incidents by month: 7, 5, then a vertical line marking the mitigation, then 0, 0, 0, 0, 0, 0. The proposal on the slide is to close the coordination workstream and move the two engineers to the retrieval rewrite. Someone asks what changed in the coordination code, and the answer is nothing — git log on the reconciler and the lock manager shows the last functional commit seven months ago, three weeks before the outage. The only thing that shipped was a ten-minute cancellation applied to every run.

That is the whole evidence base, and it is consistent with two quite different states of the world.

Two stories that fit the same graph#

The first is that the deadlocks stopped happening. Something removed the condition — a dependency upgrade that changed lock acquisition order, a traffic mix that no longer produces the interleaving, a customer who stopped uploading the bundle shape that triggered it. Under this story the cancellation is a safety net that has never caught anything, and closing the workstream is correct.

The second is that the deadlocks happen at the same rate and now end as timeouts. The cap was installed precisely to convert an unbounded hang into a bounded one, and it succeeded at that; what it also did was replace a distinctive terminal outcome with a generic one. Under this story the incident count went to zero because the incident label stopped being applied, and roughly the same number of customer workflows are being killed every month.

The two stories make different predictions, and the observables that separate them are all already in the run table.

Only one of these requires anything to have been instrumented in advance, and none of them requires a reproduction.

The query that takes twenty minutes#

Start with the shape of the duration distribution at termination, restricted to runs that ended by cancellation.

SELECT width_bucket(EXTRACT(EPOCH FROM (ended_at - started_at)),
                    540, 660, 12) AS bucket,
       count(*) AS runs
  FROM runs
 WHERE ended_at > now() - interval '30 days'
   AND terminal_state = 'cancelled'
 GROUP BY 1 ORDER BY 1;

 duration band   runs
 540–550 s          1
 550–560 s          0
 560–570 s          2
 570–580 s          1
 580–590 s          3
 590–600 s          6
 600–610 s         49        <-- the deadline
 610–620 s          4
 620–630 s          1
 ...
 total cancelled month-to-date: 67 of 9,140 runs (0.73%)

A spike of 49 runs inside ten seconds of the deadline is not a distribution of slow work; slow work does not cluster on a round number that exists only in the configuration. The pre-mitigation deadlock incident rate was 5 to 7 reported incidents a month, and reported incidents were always a fraction of occurrences because only the ones that blocked a paying customer's urgent bundle generated a ticket. Sixty-odd cancellations a month against 5 to 7 tickets is the same phenomenon with a better counter, not a different one.

Then establish what those runs were doing. If blocked_on from chapter 2 is being recorded, this is a join and the answer is unambiguous: for each cancelled run, take the last step per agent and resolve the edges. In this pipeline 38 of the 49 resolve into a two-cycle between a reconciler and an extractor, 6 point at an external OCR sink, and 5 have no edges at all. The 38 are deadlocks.

If blocked_on was never recorded — the common case, since a team that had it would probably not be in this meeting — the check degrades but does not disappear. Count tool calls and model calls per agent in the final eight minutes of each cancelled run:

SELECT r.run_id, count(*) FILTER (WHERE s.kind IN ('tool','model')) AS activity,
       count(DISTINCT s.agent) AS agents
  FROM runs r JOIN steps s USING (run_id)
 WHERE r.terminal_state = 'cancelled'
   AND s.at > r.ended_at - interval '8 minutes'
 GROUP BY 1 HAVING count(*) FILTER (WHERE s.kind IN ('tool','model')) = 0;

Runs with two or more agents and zero activity for eight minutes are the signature chapter 1 established: not slow, not spending, stable. That is enough to tell the meeting which story is true. It is not enough to say which two agents were waiting on what, which is the thing you need to fix it, and that gap is the argument for recording the field going forward.

What the other result would have looked like#

It is worth writing down in advance what would have supported the first story, because a check whose every possible outcome confirms your prior is not a check. If the cancellations had been 3 a month rather than 49, spread across the 540–660 second range rather than piled on 600, with tool calls still arriving in the final minute and one agent per run rather than two or more, the honest reading would be that the deadline is catching genuinely slow bundles and the deadlocks are gone. Closing the workstream would then be the right call, and the residual risk — a condition that is dormant rather than removed — would be worth accepting given nobody can point at a code change that removed it.

Two confounders are worth eliminating before drawing either conclusion, a confounder being a third cause that would move the numbers the same way and leave you crediting the wrong thing. Traffic mix is the first: if the customer whose bundles triggered the original outage churned four months ago, the deadlock rate could have dropped for reasons that have nothing to do with the mitigation, and the cancellations you are looking at could be a different failure entirely. Segmenting the cancelled runs by customer and by bundle shape answers that in one more query. The second is the deadline itself changing behaviour upstream: a ten-minute cap makes some agents give up and take a shortcut path, so a fraction of runs that would once have deadlocked now fail earlier for an unrelated reason and never reach the deadline at all. Both confounders push the measured rate down, which means 49 is a floor on the occurrence rate rather than an estimate of it.

The mitigation was worth keeping and is not a fix#

None of this makes the ten-minute cancellation a mistake. Before it, a deadlocked run held two worker slots and a bundle's locks indefinitely, and the outage that prompted it happened because eleven such runs accumulated over a weekend and starved the pool to the point where healthy uploads stopped being scheduled. The cap converted an unbounded resource leak into a bounded one of ten minutes per occurrence, which at 49 occurrences a month is about eight hours of held slots instead of an ever-growing amount. That is a real improvement and it should stay.

What it did not do is reduce the number of customers affected. Sixty-seven cancelled runs a month for six months is roughly 400 workflows that ended with a generic failure, and the customer-side behaviour is visible in the same database: 71 percent of cancelled runs are followed by a re-upload of the same bundle within an hour, which is a customer doing the retry the system should have done, and about a fifth of those hit the same cycle again. The support queue has been carrying the cost the whole time under a label — "processing stuck, advised re-upload" — that nobody mapped back to coordination.

The general pattern is worth naming because it is not specific to deadlock. A mitigation that converts failure mode A into failure mode B removes A from your metrics by construction, and if B is a bucket that already contains unrelated things, the conversion is a form of data loss. Timeouts are the most common such bucket in agent systems, since almost every distinct failure eventually presents as a run that did not finish in time. A team that treats "cancelled" as a terminal state rather than as an unclassified one will keep discovering that its most common failure has no cause attached.

Classify before you kill#

The change that follows is small and it goes in the cancellation path, not in the coordination code. Before a run is torn down, run the detectors from chapters 2 and 6 against its final state and record what they say in the terminal outcome.

def cancel_run(run_id, reason, ledger, steps, store):
    cause = "unclassified"
    edges = build_wait_for(run_id, ledger, steps)          # chapter 2
    cycle = find_cycle(edges)
    if cycle:
        cause = "deadlock"
    elif run_signature_repeated(run_id, store, k=3):       # chapter 6
        cause = "livelock"
    elif any(e.kind == "tool" for e in blocked_edges(run_id, steps)):
        cause = "slow_tool"
    ledger.write_terminal(run_id, state="cancelled", reason=reason,
                          cause=cause, participants=cycle or [],
                          evidence_uri=store.dump_steps(run_id))
    propagate_cancellation(run_id)                          # chapter 5

Classification happens before teardown because the wait-for graph is built from live state that teardown destroys, and the step dump is written to durable storage because a deadlock you classified and then could not reproduce is only marginally better than one you never classified. The alert then hangs off cause, so the on-call page says "deadlock rate 38/month, participants reconciler↔extractor on lock pages:*" instead of "cancellation rate normal".

With participants named, the underlying fix becomes tractable and it is ordinary concurrency work. Both agents are acquiring an artifact dependency and a page-range lock in whichever order they happen to need them, which is Coffman's circular-wait condition holding; imposing a global order on lock acquisition, or requiring an agent to acquire everything it needs before it starts any of it, removes the cycle at the source. That is a two-week change to the lock manager, and it is a change nobody could justify while the metric said zero.

Saying it in the meeting#

The thing to say to the person who considers this fixed is the number, followed by what it costs, followed by a proposal that keeps their mitigation. "Deadlock incidents are zero because we stopped labelling them. Forty-nine runs a month are cancelled within ten seconds of the deadline, thirty-eight of them have two agents idle for the full eight minutes before the cut, and seven in ten of those customers re-upload. The cap stays — it turned an unbounded leak into a bounded one. What I want is the cause recorded at cancellation, which is a day of work, and two weeks of data before we decide whether the lock ordering change is worth doing."

That framing survives contact with a room because it concedes the part that is correct and disputes only the inference. The failure mode to avoid is arguing that the mitigation was wrong, which invites a defence of a decision that was, in fact, a good one taken under pressure.

There is a loose end in the code above, and it is the kind that only shows up when it matters. cancel_run runs on a path that executes a few dozen times a month, always during a failure, often while the very agents it is inspecting are themselves partly dead. Nothing in normal operation exercises it, no test in the suite covers the case where build_wait_for is called against a run whose ledger connection is timing out, and a cancellation path that throws while cancelling leaves exactly the orphaned run it exists to prevent. Confidence in any of this requires breaking it deliberately.