Breaking a tool-call retry loop

An agent that receives a tool error decides what to do next the same way it decides everything else, by continuing from its context. Nothing in that process counts attempts, notices that it has been here before, or distinguishes a fault worth retrying from one that will fail identically forever. Topic 2.11, on error handling, covers the retry policy a client should apply to a provider's own errors — backoff, jitter, budgets, idempotency — and all of it assumes a caller that repeats a request. This is a caller that regenerates one.

That distinction is the whole of the problem. Two calls a model makes for the same purpose are two independently generated pieces of text, so every protection built on comparing them byte for byte is checking a property they do not have, and the loop those protections were installed to stop runs anyway.

Byte equality is the wrong identity#

Here are the first three calls of a turn that produced fourteen, trimmed from the trace:

[
  {"n": 1,  "tool": "create_ticket", "args": {"customer_id": "c_40912", "order_id": "88214",
    "summary": "Customer has not received refund for returned boots"},
   "outcome": "timeout", "elapsed_ms": 30000},
  {"n": 2,  "tool": "create_ticket", "args": {"customer_id": "c_40912", "order_id": "88214",
    "summary": "Refund not received for returned boots - customer following up"},
   "outcome": "timeout", "elapsed_ms": 30001},
  {"n": 3,  "tool": "create_ticket", "args": {"customer_id": "c_40912", "order_id": "88214",
    "summary": "Missing refund on order 88214 (returned boots), customer awaiting resolution"},
   "outcome": "timeout", "elapsed_ms": 30002}
]

A deduplication check that hashes the serialised arguments is checking whether two calls are the same string. What matters is whether two calls will produce the same side effect, and those come apart immediately once a free-text field is in the argument list.

>>> sha256(json.dumps(calls[0]["args"], sort_keys=True).encode()).hexdigest()[:12]
'a3f19c0b7e42'
>>> sha256(json.dumps(calls[1]["args"], sort_keys=True).encode()).hexdigest()[:12]
'6d80b4ff1c95'

Two different hashes, two different tickets, and a dedupe cache that saw fourteen distinct keys and let all fourteen through. The fix is to key on the fields that determine the side effect rather than on the whole argument object. For create_ticket, a ticket is about one customer and one order, so (customer_id, order_id) is the identity and summary is decoration — expensive decoration, since it is the field that defeated the protection. Which fields are identity is a per-tool judgement that has to be declared alongside the tool, because nothing in a JSON Schema distinguishes a field that names the thing being acted on from a field that describes it.

Identity is only half of the key. The other half is the error class, because two calls that fail the same way are a loop and two calls that fail differently are progress. A model that calls create_ticket, gets a validation error about component_id, fixes it, and gets a timeout has made a legitimate second attempt at a different problem; a model that gets a timeout fourteen times has not.

IDENTITY_FIELDS = {
    "create_ticket": ("customer_id", "order_id"),
    "refund_order":  ("order_id",),
    "transfer_stock": ("sku", "from_warehouse", "to_warehouse"),
}

def attempt_key(tool, args, error_class):
    fields = IDENTITY_FIELDS.get(tool)
    identity = (tuple(args.get(f) for f in fields) if fields
                else canonical_json(args))          # fall back to full args
    return (tool, identity, error_class)

class AttemptTracker:
    def __init__(self, cap=2):
        self.counts, self.cap = {}, cap

    def record(self, tool, args, error_class):
        key = attempt_key(tool, args, error_class)
        self.counts[key] = self.counts.get(key, 0) + 1
        return self.counts[key]

    def exhausted(self, tool, args, error_class):
        return self.record(tool, args, error_class) > self.cap

With cap=2, the fourteen-ticket turn stops at three tickets and ends 5m30s earlier.

That saving is a multiplication, and doing it is also a check on the trace. Each attempt in the excerpt above ran for about thirty seconds, which is the client timeout, because every one of them reached it rather than returning; the elapsed_ms values differ by a millisecond or two and nothing more. Fourteen attempts at that price is a little over seven minutes, which is where the turn length in the postmortem comes from, and it is arrived at from the trace without consulting the postmortem at all. Stopping after the third ticket removes everything from the fourth attempt onward, which is eleven attempts that never happen at thirty seconds each, and that is the five and a half minutes. The saving is linear in the number of attempts prevented, so the cap is the entire lever, and nothing about making each attempt cheaper would have helped.

Stopping is an action, not the absence of one#

Counting attempts is the easy half. The harder half is what the agent does when the count is exceeded, and getting this wrong produces an outcome that is worse than the loop in a different way. If the dispatcher simply refuses the call and returns another error, the model is in the same position it was in before, holding a request it cannot serve, with tools still available and every incentive to reach for a neighbouring one — which is the selection failure from chapter 1 arriving as a consequence of a badly designed stopping condition.

The terminal action has to do two things: give the model a result it can report to the user, and remove its ability to keep calling tools.

def dispatch(tool, args, tracker, client):
    try:
        return client.call(tool, args)
    except ToolTimeout as e:
        if tracker.exhausted(tool, args, "timeout"):
            return terminal(
                f"Stopped after 3 attempts at {tool}. The request may have "
                f"succeeded server-side; do not retry. Tell the user their "
                f"request is being processed and that a support agent will "
                f"confirm, then end your reply."
            )
        raise e

# terminal() results are tagged; the turn's next model call is issued with
# tool_choice="none", so the model can only produce prose.
if any(r.terminal for r in results):
    reply = model.respond(messages, tools=[], tool_choice="none")

Forcing tool_choice="none" for the final turn is what actually ends it. A message in the tool result asking the model to stop is a suggestion that competes with everything else in the context, and under a strong task instruction models frequently keep trying. Removing the tools removes the option.

The wording of the terminal result matters as much as the mechanism. "The request may have succeeded server-side; do not retry" is accurate about a timeout, which reports that the outcome is unknown rather than that the operation failed. A message saying "ticket creation failed" would be a lie that the model then passes to the customer, who is told nothing happened while fourteen — or three — tickets sit in a queue.

A timeout is structurally an unknown rather than a failure, and the chain that makes it one is worth following because everything downstream depends on it. The clock a client runs is a clock on the response. The response is produced after the operation, since the service has to do the work before it can describe what it did. So a timer expiring tells you the response did not arrive in the time allowed and nothing whatever about which side of the write the clock ran out on. Both worlds are consistent with the same silence: nothing happened, or everything happened and the confirmation is still being assembled. Any wording that picks one of those two is inventing information the client does not have, which is why the honest terminal message reports the ambiguity and instructs against a retry, and why removing the ambiguity at source needs something on the write itself rather than better wording on the way out.

The two fixes that get proposed instead#

Exponential backoff is the standard reflex, and it comes from HTTP clients and message consumers where the fault being smoothed over is transient contention: a saturated connection pool, a rate limit, a leader election in progress. Waiting helps there because the next attempt encounters a different world. Nothing in this failure was transient — the ticket service was committing every write and taking 34 seconds to render a response it was always going to take 34 seconds to render — so backoff produces the same fourteen tickets over eleven minutes instead of seven, and the request budget still runs out.

The general statement is that backoff is a control on spacing and the damage here is a function of count. Waiting longer between attempts changes when each duplicate is written and not whether it is, and the number of attempts is set by whatever eventually stops the agent — which in this incident was the request budget, not the wait. Stretching the same fourteen writes over a longer wall clock makes the incident last longer while producing exactly the same queue at the end of it. A timing lever cannot fix a counting problem, and the reason backoff feels like it should is that in the systems it comes from, waiting changes the outcome of the next attempt rather than just its timestamp.

Capping total tool calls per turn at ten is the other proposal, and it does bound the damage, at ten tickets rather than fourteen. It is a budget, which is a different mechanism from a stopping condition: it does not distinguish a loop from legitimate work, so a bulk operation that genuinely needs to call twelve tools is now broken, and it terminates the turn by starvation, leaving the model with no result and no explanation at precisely the moment it most needs to say something to the user. Both are worth having — a budget is a good outer bound and backoff is right for the faults it was designed for — and neither is the thing that stops this.

Loops that never error#

The tracker keys on an error class, which means it is blind to the loop that never errors. An agent that calls search_orders with "boots refund", gets an empty list, rewords the query to "boots return refund status", gets an empty list, and does that nine more times has burned the same budget with eleven successful 200s and no error class to key on. This is the chapter 1 misroute running in a loop rather than once, and it is common enough that a loop breaker built only on exceptions will miss a meaningful share of real failures. Extending the key with a coarse fingerprint of the result — empty versus non-empty for search-shaped tools, or a hash of the returned identifier set — catches it, at the cost of a per-tool decision about what "the same result" means. A blunter version that works well enough as a backstop is a per-turn counter on calls to the same tool that returned no rows, capped at three, terminating the same way.

Detecting either in production takes one metric rather than a system: record the maximum attempt count reached per key per turn, and watch its upper percentiles. A healthy agent sits at 1 for p99 with occasional 2s, and a turn that reaches 6 is worth investigating whether or not anything threw. That histogram is also what tells you a new loop has appeared after a model upgrade or a description change, on the same principle as the argument-mismatch trend line from chapter 3 — a silent failure becomes manageable at the moment it acquires a number that moves.

Recording the maximum per turn rather than a rate over calls is the choice that makes that metric work, and it is the same lesson the eval chapters teach in a different costume. A retry rate averaged over calls divides the fourteen bad calls by every call the fleet made that day, which buries them; the incident is not a small elevation spread across traffic but the whole of one turn. Taking the maximum per turn keeps the concentration intact, so a single turn that reached fourteen shows up as a single observation at fourteen and cannot be diluted by however many healthy turns surround it. Whenever the failure being watched for is a concentration, the statistic has to be one that concentration survives.

The loop breaker bounds the number of duplicates and does not prevent the first one. Three tickets is better than fourteen and it is still two tickets too many, and the instrument that gets it to one is an idempotency key on the write — a caller-supplied identifier the service stores with the result, so that a second request arriving under the same key returns the first outcome instead of performing the operation again — derived by the dispatcher from the same identity fields so that every retry of the same intent carries the same key. Letting the model generate the key defeats it for the same reason the argument hash failed, since the model will produce a different key each time.

IDENTITY_FIELDS is also a hand-maintained mapping, which is the kind of thing that is correct on the day it is written and quietly wrong eighteen months later. When create_ticket gains a queue parameter and two tickets on the same order in different queues become legitimate, the identity is now three fields and nobody will remember to update the dictionary, so the tracker starts collapsing calls that were not duplicates and blocking work that should have proceeded. Declaring the identity fields next to the tool definition rather than in a central table, and failing CI when a tool has parameters but no declaration, keeps the drift visible; it does not eliminate the judgement call, which genuinely requires knowing what the operation does.

The four instruments in this topic fit together in a specific order. The argument value check from chapter 3 is hours of work, runs on live traffic, and needs no labels, so it goes first. The ladder from chapter 5 needs a labelled set, which is the expensive part and the one that unlocks everything downstream — the pair table it produces is what tells you whether your selection problem is six confusable pairs or a catalogue too large to attend to, and that answer determines whether you spend the next fortnight rewriting descriptions or building a retriever.

The loop breaker is independent of both, and it is the one to build first if you have ever had a turn like the one above, because it is the only failure in the topic that can create fourteen of anything before a human notices.