Termination you have not broken on purpose is untested

The classify-before-you-kill change shipped on a Tuesday and met its first real deadlock on the Thursday. It did not go well.

16:41:02.000  supervisor  run_9701 wall_clock exhausted -> cancel_run(reason=deadline)
16:41:02.004  supervisor  build_wait_for: SELECT owner_of_task_producing(...)
16:41:32.004  supervisor  ledger query timeout (30s); retry 1
16:42:02.010  supervisor  ledger query timeout (30s); retry 2
16:42:07.550  supervisor  killed by its own 65s watchdog
16:42:07.550  (no terminal record written; propagate_cancellation never called)
16:47:00.000  operator    notices 3 of 5 agents still holding leases on run_9701

The ledger was slow because two hundred runs had hit the deadline in the same thirty seconds, which happens when an upstream dependency degrades and every run in flight starts waiting on it at once. The cancellation path did its diagnostic work before its cleanup work, took an unbounded dependency on the store it was diagnosing, and left the run in exactly the orphaned state it was written to prevent. Every unit test for cancel_run passed, because every one of them ran against a healthy ledger.

Ordering the operations differently — propagate cancellation first, classify afterward, and put a 2-second budget on the classification with cause="unclassified" as the fallback — is a five-line fix. Knowing that it was needed required the path to run while something else was broken, and the only reliable way to arrange that is to break things on purpose.

The faults worth injecting#

The scenarios are not open-ended. Termination in a multi-agent run has a small number of moving parts, and each one has a small number of ways to fail.

Each row is a claim made by an earlier chapter, restated as something a machine can check. The epoch fence from chapter 3, the obligation accounting from chapter 4, the cancellation procedure from chapter 5 — none of those were ever exercised by a run that worked, because a run that works never has a zombie — an agent that was stalled past the expiry of its claim, had the work reassigned underneath it, and then woke up still believing the item was its own — never has an unclosed obligation, and never cancels anything.

Five invariants, checked after every scenario#

The assertions in that table are scenario-specific. Underneath them sit five properties that must hold at the end of every run, whatever was injected, and the harness is mostly a loop that injects one fault and then checks the same five things.

FAULTS = ["kill_mid_task", "stop_past_lease", "drop_message", "duplicate_message",
          "tool_hang", "late_tool_result", "ledger_blackhole", "host_loss",
          "budget_exhausted_in_spawn"]

@pytest.mark.parametrize("fault", FAULTS)
async def test_termination_under_fault(cluster, clock, fault):
    run = await cluster.start_run(BUNDLE_3_VENDORS, budget=BudgetSpec())
    await cluster.inject(fault, at=FaultPoint.random_within(run))
    await clock.advance(run.budget.wall_clock_s + GRACE_S)

    rec = cluster.ledger.terminal_record(run.id)
    assert rec is not None, "no durable terminal outcome"           # 1
    assert cluster.ledger.terminal_record_count(run.id) == 1        # 1
    assert rec.cause in KNOWN_CAUSES

    assert cluster.live_agents(run.id) == []                        # 2
    assert cluster.open_leases(run.id) == []                        # 3
    assert cluster.held_locks(run.id) == []                         # 3
    assert cluster.worker_slots_in_use(run.id) == 0                 # 3
    assert cluster.open_obligations(run.id) == []                   # 4

    for item in cluster.ledger.items(run.id):                       # 5
        assert item.status in TERMINAL_STATUSES
        if item.status == "done":
            assert cluster.store.head(item.output["uri"]) is not None

The five are: a durable terminal outcome exists and there is exactly one of it; no descendant of the run is still executing; everything the run held is released; every obligation is closed; and every ledger item is terminal with a verifiable artifact behind each done. Nothing in that list depends on the run having succeeded. A run cancelled at three minutes with two of fifteen sections extracted satisfies all five, and that is the point — the invariants are about termination being clean, not about the answer being good.

The terminal record has to carry enough for the assertions to be meaningful, which constrains its schema more than teams expect. state alone is not enough, because chapter 7's whole argument was that cancelled without a cause is an unclassified bucket. The record this pipeline writes has state, cause, the counter that was exhausted if any, the participants of a detected cycle, a pointer to the dumped step trace, and a summary of what exists — items done, items failed permanently, items never started, with artifact URIs. That last part is what makes a cancelled run resumable rather than merely finished, and it is also what a test can check without knowing which fault was injected.

The terminal_record_count(run.id) == 1 assertion earns its place more often than the others. Two terminal records appear when the supervisor cancels a run at the same moment the last agent discharges its final obligation and declares completion, which is a race between chapter 4's edge-triggered completion and chapter 5's cancellation. The conditional UPDATE on state = 'running' is what makes one of them lose, and it is easy to reintroduce the bug by adding a second write path that does not carry the predicate.

A virtual clock is what makes this affordable#

Nine faults against a run whose wall-clock budget is 900 seconds is over two hours of waiting if the tests run in real time, and a suite that takes two hours does not run on every commit. The enabling piece is that nothing in the system reads the system clock directly — every deadline, lease expiry, heartbeat interval and grace period comes from an injected clock, so the test can advance time by 920 seconds in a single call and every timer fires in the correct order in about 40 milliseconds.

That constraint is worth accepting early because retrofitting it is painful. A single time.time() inside a lease renewal loop means that test hangs for the full duration, and the failure presents as a flaky suite rather than as an obvious violation. Grepping for direct clock reads is the same kind of mechanical audit as grepping for unwrapped waits in chapter 2, and both are worth a lint rule.

With the virtual clock, this pipeline's 41 termination scenarios — nine faults across several run shapes, plus the fault points placed by hand — run in about 90 seconds in CI against in-process brokers and a real Postgres. That suite is the fast tier. The second tier runs weekly in staging against the real broker and the real object store, with faults injected by a proxy that can drop, delay and duplicate, and it takes twenty minutes because it uses real time for the parts where real network behaviour is the thing under test. The third tier is a monthly game day — a scheduled hour in which someone breaks a real thing on purpose, during working hours, with the team watching — run in production against an internal tenant, where an engineer terminates an instance during a live run and the on-call practises reading the terminal record.

Killing a process is not killing a machine#

The host_loss row is separated from kill_mid_task because the two exercise different code. A SIGKILL leaves the operating system alive, so every TCP connection the agent held is closed by the kernel with a FIN — the packet that tells the other end the connection is finished — the broker sees the consumer disconnect immediately, and the ledger's connection is released. The system learns about the failure through connection teardown, in milliseconds.

A lost machine sends nothing. The broker holds an open consumer for its heartbeat interval, the ledger holds a connection until its TCP keepalive gives up, and the only thing that reclaims the item is the lease expiry from chapter 3 firing on time. Testing the fast path and not the slow one leaves you believing that recovery takes 200 milliseconds when it actually takes the lease duration, and the gap shows up in production as an item that is not reclaimed for ninety seconds while a customer waits.

Injecting real host loss without a real host is a matter of removing the network rather than the process: blackhole the instance's traffic with a firewall rule, or in the simulated cluster, stop delivering to and from that node without closing anything. The other half is placing the cut precisely. Random timing will eventually hit the 106-millisecond window between the artifact write and the ledger transition that produced the customer-facing defect in chapter 3, and "eventually" at that width is roughly one run in several thousand. A named fault point is deterministic:

# in finish_item, between put_object and ledger.transition
fault_point("after_artifact_before_transition")   # no-op unless armed

# in the test
await cluster.inject("host_loss", at="after_artifact_before_transition")

The barrier is a no-op in production and a rendezvous under test — a point where the agent stops and waits until the harness says to continue — so the scenario runs in milliseconds and hits the exact window every time. Every window an earlier chapter identified deserves one of these, which for this topic is about a dozen.

What the suite does not prove#

Coverage here is over faults somebody thought of, and the injection is one fault at a time. The Thursday incident was two things together — a deadlock and a degraded ledger — and no single-fault scenario in the table produces it. Extending to pairs turns nine scenarios into thirty-six, which is affordable; extending to the timing interleavings that matter is not, and that is where deterministic simulation testing comes in. Running the entire system on a simulated network and a simulated clock with a seeded scheduler, so that a failing run can be replayed exactly, is how FoundationDB and a handful of others get real confidence, and it costs a rewrite of every I/O boundary behind an interface the simulator can control. Whether that is worth it depends on what a wrong answer costs, and for most agent pipelines the honest answer is that the fault table plus named fault points plus a monthly game day is where the returns flatten.

What the suite does establish is enough to act on. Termination in this system is now something with evidence behind it: a stuck run gets classified rather than aggregated into a timeout bucket, a run that stops has a durable record saying why, a budget bounds what any run can consume, and the paths that only execute during failure have been executed on purpose. The next time the intake pipeline goes quiet at 09:12 on a Tuesday, the first question — is anything actually wrong — is answerable from a dashboard in under a minute, which is what all of this was for.