Termination and convergence

Read these first

• Blackboard / shared-artifact architectures
• Error recovery

What you will know

• Detect two agents waiting on each other

  A wait-for graph should expose cycles even when both agents remain healthy and emit plausible status updates.

• Bound total steps across a whole multi-agent run, not per agent

  Count retries, tool calls, messages, and wall-clock time globally, then cancel descendants when the shared budget expires.

• Define what "done" means for a system with no single controller

  Distinguish quiescence from silence by accounting for in-flight work, delayed messages, unresolved obligations, and partial results.

• Show a run terminating correctly under failure

  Inject crashes, timeouts, and lost messages, then verify cleanup, cancellation propagation, and a durable terminal outcome.

• Instrument termination decisions and classify repeated states as loop, deadlock, or legitimate waiting

  Without correlated state-transition traces, operators cannot tell productive progress from livelock or a budget that merely masked the fault.

Chapters

1. 1

   A step limit never fires on two agents waiting

   One run has two agents reporting that they are waiting for input every thirty seconds, with no tool call in eight minutes; another has a single agent issuing the same search eleven times in four minutes. Calling both loops and shipping a step counter catches the second and never the first, because a deadlock consumes no steps and stays stable indefinitely. A quiet dashboard then reads as evidence the class does not occur.

   Read chapter
2. 2

   Cycles in the wait-for graph, not liveness checks

   Both processes are healthy, both sit inside their token budgets, both emit plausible status updates, and nothing has been produced for twelve minutes. Health checks cannot separate two agents waiting on each other from one waiting on something still in flight. Build the graph from what each agent reports it is blocked on and look for a cycle, which requires agents to report the object of the wait rather than the fact of it — the first of the two per-step fields a dashboard needs to classify a stuck run on its own.

   Read chapter
3. 3

   Silence is not completion while an obligation is open

   No agent executing and no message in flight does not mean the run finished: a reply may still be in transit and obligations may be unmet. Reading completion off a ledger with nothing pending is worse, since a crash mid-task leaves an item in progress forever while a crash just after marking one complete leaves an item whose output never landed. Durable transitions and expiring claims are what make that check sound.

   Read chapter
4. 4

   Quiescence is an obligation count reaching zero, not a quiet log

   Silence is what you observe; quiescence is what you have to establish, and the ledger of the previous chapter only accounts for work somebody already claimed. Give every dispatch an obligation created when a message or task is sent and discharged only when its receipt is acknowledged, so that a run is quiescent when no agent is executing and that count is zero. Then face the part that makes distributed termination genuinely hard: a snapshot taken agent by agent can find every one of them idle while a message sits on the wire between two of them, which is why the count must be maintained jointly by senders and receivers rather than sampled from the outside. Partial results and unresolved obligations fall out of the same accounting.

   Read chapter
5. 5

   One budget for the run, decremented at every hop

   A run burned $46 and fourteen minutes with every agent inside its 25k-token, 12-call cap, and halving those caps only buys more, cheaper laps. Count steps, tool calls, messages, retries and wall-clock against the whole run; on exhaustion, cancel every descendant, release what they hold, and write a durable terminal outcome. A wall-clock deadline alone is a lagging indicator that spends the money first and orphans whatever was running.

   Read chapter
6. 6

   A repeated state with changed inputs is progress

   An agent has arrived at what looks like the same state four times in nine minutes. Before capping it you need to know whether anything changed between visits — new evidence, a different tool result, a decremented budget — since identical inputs mean livelock and changing ones mean work. Recording that per step is what lets a dashboard classify loop, deadlock and legitimate waiting without a human reading traces.

   Read chapter
7. 7

   Zero deadlock incidents can mean the label moved

   After a deadlock outage the team added a ten-minute cancellation and the incident count went to zero, with the coordination code untouched. Two explanations fit, and one cheap check separates them: how many runs are being cancelled at exactly ten minutes, and what were they doing. If that rate matches the old deadlock rate, customers have had cancelled workflows for six months behind a generic outcome.

   Read chapter
8. 8

   Termination you have not broken on purpose is untested

   Runs that end cleanly when everything works tell you nothing about the case you care about. Inject crashes, tool timeouts and dropped messages into real runs, then verify that cancellation reaches every descendant, that claims and resources are released, and that a terminal outcome is recorded durably. This is also the test that shows whether a definition of done survives a machine dying mid-run.

   Read chapter