A step limit never fires on two agents waiting

At 09:12 on a Tuesday the support queue has two tickets from the same customer, both saying a contract bundle has been "processing" for far too long. The system behind them is a document intake pipeline: each uploaded bundle starts a run with a router, two extractor agents, a reconciler that merges their output, and a writer that produces the final summary. The agents coordinate through a shared task ledger of the kind covered in 11.12, and some of their tools — OCR, or optical character recognition, on scanned pages, and a case-law lookup — routinely take two or three minutes to return, so slowness on its own is not a signal.

The question the engineer is about to work on is a termination question, and the subject is worth naming plainly before any log gets read. A run is one whole job the system performs, from the upload that starts it to the summary it returns, and it is carried out by several agents: separate processes, each one a model deciding what to do next together with the tools it is allowed to call. Termination is the property that a run eventually stops, where stopping means arriving at a state the system treats as final — an answer produced, a budget spent, a failure recorded and closed. Convergence is the stronger property that the run is getting somewhere while it runs, so that each step leaves it closer to one of those final states than the step before. A run that converges reaches an answer on its own, and a run that terminates without converging stops because something outside it decided to cut it off.

Neither property is free, because nothing about the arrangement supplies it. A single program stops when its last statement runs, and a run made of several agents that call each other, wait on each other and choose their own next actions has no last statement, so somebody has to decide what finished means for it and then arrange evidence that the condition has been met. Where that decision is missing the run does not crash; it sits, holding worker slots, ledger leases and money, answering every health check truthfully while producing nothing. In practice the subject reaches you as an incident — a job that normally takes ninety seconds is still going an hour later — and as a set of settings whose values nobody can defend: the step limit in an agent framework, the deadline in an orchestration config, the stop condition in a design document that was never made precise. The two runs on the engineer's screen are the reason a single one of those settings cannot do the job.

The engineer on call pulls the step logs for both runs. They do not look the same at all.

run_8842   window 08:47:00 → 08:55:00
08:47:02  reconciler   status   blocked, waiting for input
08:47:04  extractor-2  status   blocked, waiting for input
08:47:32  reconciler   status   blocked, waiting for input
08:47:34  extractor-2  status   blocked, waiting for input
        … 14 more identical pairs, every 30s …
08:55:02  reconciler   status   blocked, waiting for input
08:55:04  extractor-2  status   blocked, waiting for input

tool calls in window: 0
model calls in window: 0
prompt tokens in window: 0

run_8817   window 09:03:00 → 09:07:00
09:03:11  extractor-1  tool  search_clauses(q="indemnity cap", doc="b7") → 0 hits
09:03:33  extractor-1  model call, 2,140 prompt / 180 completion
09:03:35  extractor-1  tool  search_clauses(q="indemnity cap", doc="b7") → 0 hits
09:03:58  extractor-1  model call, 2,330 prompt / 176 completion
        … same pair repeating …
09:06:52  extractor-1  tool  search_clauses(q="indemnity cap", doc="b7") → 0 hits

tool calls in window: 11
model calls in window: 11
prompt tokens in window: 27,400

Both runs are stuck in the sense the customer means. The mechanisms have nothing in common, and the difference shows up in the one column that a naive fix depends on.

Two failures that share a symptom and not a mechanism#

run_8817 is a loop: one agent keeps issuing an action that does not change the world or its own knowledge, and the model keeps deciding to issue it again because nothing in its context has changed to suggest otherwise. Topic 10.6 dealt with this shape for a single agent, and the detection there — normalise the call, hash it, notice the repeat, ask whether anything new entered the context — applies here unchanged. The loop is expensive while it runs: eleven model calls at roughly 2,200 prompt tokens each is about 27,400 tokens in four minutes, and on a mid-tier model at $3 per million input tokens that is around eight cents per four minutes, or a bit over a dollar an hour per looping agent. Left alone overnight it produces a bill and a very large trace file.

run_8842 is a deadlock: the reconciler is waiting for extractor-2 to publish its clause table, and extractor-2 is waiting for the reconciler to release the page-range lock it holds on bundle b7. Neither will act until the other does, and neither is going to. The word "waiting" in both status lines is accurate. Both processes are alive, both are answering their health endpoints, both are well inside their token budgets, and both are reporting truthfully every thirty seconds.

The property that matters for detection is in the last three lines of each trace. The deadlocked run consumed zero tool calls, zero model calls and zero tokens in eight minutes. It is not merely slow — it is not spending anything from the resources a run-limiting mechanism usually meters. It will continue to spend nothing for as long as you leave it running, which means it is perfectly stable. A run that is stable and cheap is a run that no counter, threshold or rate alarm will ever trip.

Why the counter is the tempting fix, and what it actually covers#

The obvious response to "the agents are looping" is to bound the loop. Every agent framework offers the hook: the OpenAI Agents SDK exposes max_turns on the runner and raises MaxTurnsExceeded when the loop exceeds it, and every other framework has an equivalent under some other name. The fix is three lines and it demonstrably works on run_8817.

from agents import Runner
from agents.exceptions import MaxTurnsExceeded

try:
    result = await Runner.run(extractor, bundle_prompt, max_turns=20)
except MaxTurnsExceeded:
    ledger.mark_failed(run_id, agent="extractor-1", reason="max_turns")

Ship that with max_turns=20 and run_8817 dies after about seven minutes and 20 model calls instead of running until someone notices. The dashboard grows a max_turns counter, it goes up a handful of times a week, and the loop problem is visibly under control.

run_8842 never reaches that code path. max_turns is decremented when the agent takes a turn, and a blocked agent is not taking turns. The reconciler's turn counter stopped at 6 the moment it blocked and it will still read 6 tomorrow. The same is true of the token budget, the tool-call budget and the retry budget — all four of the budgets from 10.6 are counters over activity, and a deadlock is precisely the absence of activity. The one budget that does fire is wall-clock, because time passes whether or not anyone is working.

The row about resources is the one people skip. A deadlocked run costs nothing in tokens and is not free. In this pipeline each blocked agent holds a worker slot on a pool of 32, an exclusive lease on the ledger rows for its bundle, and a page-range lock on the shared OCR cache. Two deadlocked runs sitting there for six hours are four worker slots and two bundles' worth of locks that nobody can reclaim, which is how a customer discovers the problem — some other bundle they uploaded is queued behind a lock held by a run that stopped doing anything at 08:47.

What a wall-clock timeout does to each of them#

Since wall-clock is the only limit that fires on a deadlock, the natural next move is to put a deadline on every run. It is worth being precise about how well that serves each case, because the answer is different in the two directions and the asymmetry is easy to get backwards.

For the deadlock, a wall-clock deadline is an acceptable stopgap. The run is spending nothing per minute, so the cost of letting the deadline run its full ten minutes is a bounded amount of held resources and no money. And the action the deadline takes — stop the run — is the correct action, because a genuine cycle will never resolve on its own, so nothing is lost by cutting it. What the deadline does not do is tell you that a cycle existed. It records "run exceeded 600s", which is the same outcome you get from a slow OCR job on a 400-page scanned bundle, so the cause disappears into a generic bucket. That substitution has consequences of its own, and chapter 7 is about what it does to your incident data over six months.

For the loop, the same deadline is a poor instrument. It fires late, after roughly 68,000 tokens have gone through the model at the rate run_8817 was burning them, when a turn limit would have stopped it at 20 turns for a third of that. It also cannot tell a loop from legitimate long work, so the first time it kills a real 400-page bundle the pressure is to raise the deadline to twenty minutes, which doubles the cost of every future loop. A limit whose threshold gets tuned by its false positives drifts in exactly the direction that makes its true positives more expensive.

The mistake this chapter exists to prevent is collapsing the two runs into one phenomenon — treating run_8842 as a slow loop and run_8817 as a fast one, and concluding that a single step limit covers both. Under that view you ship the turn cap, the loop terminations show up in the log, the deadlock terminations show up nowhere, and the reasonable inference from a quiet dashboard is that mutual waiting is not a class of failure your system has. It is a class of failure your instrument cannot see. Six months of that and the belief is well supported by data, which is what makes it hard to dislodge.

Three neighbours that wear the same symptom#

Mutual waiting and repeating action are not the only ways a run stops producing output, and the three states nearest to them are worth naming now, because each one is a different bug with a different fix and all four look identical from the outside.

The first is a legitimate long wait. A 400-page scanned bundle takes the OCR service between six and nine minutes, and while it is out there the extractor's trace looks exactly like run_8842: no tool calls, no model calls, no tokens, a status line every thirty seconds. The activity counters cannot separate a run that is waiting correctly from one that is waiting forever, so any rule of the form "zero tool calls for N minutes means broken" is a rule that kills healthy runs at whatever rate your slowest tool exceeds N. The pipeline here has a p99 OCR latency of 8.4 minutes, which means an eight-minute silence threshold would abort roughly one legitimate bundle in a hundred.

The second is starvation. When run_8842 and two other deadlocked runs hold six of the pipeline's 32 worker slots, a fourth run's extractor sits in the dispatch queue and never starts. It consumes no steps for the same reason a deadlocked agent does, but there is no cycle anywhere near it and no bug in its own coordination — it resolves the instant a slot frees. Diagnosing starvation as deadlock sends you to read the coordination code of a run that is behaving perfectly.

The third is oscillation, where two agents alternately claim and release the same ledger item, each seeing the other's claim, backing off, and retrying. That one does burn steps, so a turn limit will eventually stop it, and the log it leaves looks like a loop in a single agent. The cause is coordination rather than a model repeating itself, so the fix that works on run_8817 — change the prompt, add the tool result to context, dedupe the call — does nothing at all here.

What you cannot get from one agent's log#

Look again at what the on-call engineer had to do to classify run_8842. They read both agents' status lines, knew that the reconciler consumes what extractor-2 produces, knew that the OCR cache lock is held per page range, and put those three facts together into a cycle. None of that is in either agent's own record. The reconciler's log says it is waiting; the interesting part is what it is waiting for, and whose behaviour has to change before it can proceed.

A step record that says status: waiting is not classifiable by a machine, because a healthy agent waiting on a three-minute OCR call emits exactly the same line as a deadlocked one. The record has to name the object of the wait — the specific artifact, lock or agent — and it has to be emitted even on steps where the agent does nothing, since those are the only steps a blocked agent produces. Once every blocked agent names what it is blocked on, the relation between agents becomes data rather than tribal knowledge, and the cycle that the engineer found by hand becomes a graph query. Building that graph, and being honest about what it can and cannot separate, is the next chapter.