Bounding the tool-calling loop

Chapter 7 established that the number of iterations in a tool-calling loop is chosen by the model, one pass at a time, against a history in which nothing counts anything. That is a resource commitment your code has made in advance, and this chapter is about where to put the ceiling on it, what to send back on the iteration that exceeds it, and which of the other things people reach for — a wall-clock deadline, an HTTP timeout, a token budget — bind the same resource.

The distinction that organises all of it is between a bound and a tendency: a tendency reduces how often something happens and a bound makes it impossible. The difference shows up in the tail rather than in the average, which is why a loop with a tendency and no bound looks fine for months.

A prompt instruction is a tendency, not a ceiling#

The natural first fix for a loop that ran away is a line in the system prompt: do not call the same tool more than three times in one turn. The failure looks like a behaviour problem, and behaviour is what prompts govern.

The reason it does not work is structural rather than empirical. A prompt instruction changes the probability distribution over the model's next action, so it reduces the chance of a fourth call on each iteration without eliminating it, and a bound that holds with probability 0.98 per iteration is not a bound. Two per cent per iteration is a run of twenty with probability about 2 in 3, and one long conversation a day is enough to keep the incident alive. The instruction is also being enforced by the component least able to enforce it — the model on iteration 63 does not experience itself as having called a tool sixty-two times; it sees a history in which several searches did not find the answer and a user question still outstanding.

The sum behind those numbers is short and worth doing, because it is the reason a high compliance rate reads as safety and is not. If the instruction holds independently on each pass with probability nought point nine eight, the chance it holds on all twenty passes of a twenty-iteration run is nought point nine eight multiplied by itself twenty times. Multiplication is the right operation because the run only survives if every pass survives, and that comes out at roughly two thirds. Read the other way, about one run in three of that length breaks the instruction at least once. The compliance rate did not change; what changed is the number of opportunities, and a probability applied repeatedly approaches certainty.

That is the general shape of a tendency: multiply the per-iteration success probability by itself once per iteration, and any figure below one falls towards zero as the count grows. A bound does not participate in that arithmetic at all, because it is not a probability. The counter in your loop is either below the cap or it is not, and no number of iterations makes that comparison come out differently.

This is the same argument that decided, some years ago, that clients do not enforce rate limits. A well-behaved client backs off, and the server still has a limiter, because the guarantee has to live somewhere that a misbehaving client cannot influence. A prompt is a client-side politeness convention. The ceiling belongs in the loop, which is the only component in the system that can observe the iteration count directly and act on it unconditionally.

Keeping the prompt instruction alongside the hard bound is still worth doing, since a model told to search at most twice before answering with what it has will usually stop at two, and the cap then only fires on the pathological tail. What must not happen is the prompt being the thing you rely on.

What a runaway looks like from inside#

The reason a cap is not obviously necessary is that a runaway does not look like one from any single iteration. Here is a sample of the queries from a documentation assistant that issued 214 calls to search_docs in one turn:

iter   1  search_docs("webhook retries")
iter   2  search_docs("webhook retry policy")
iter   3  search_docs("does the API retry webhooks")
iter   7  search_docs("webhook delivery guarantees")
iter  19  search_docs("retry backoff configuration webhooks")
iter  63  search_docs("webhook retry attempts maximum")
iter 147  search_docs("what happens when a webhook endpoint is down")
iter 214  search_docs("webhook redelivery")

They are not identical, and that is the whole difficulty. Every result came back with three passages, none of which mentioned retries, because the documentation for that feature had never been written. The model read three plausible passages about webhooks that did not answer the question, concluded the phrasing was wrong, and tried again — a defensible move each time, given the previous one.

The aggregate is a different object from any of its steps. That turn held one Python worker at 3.1 GB, ran for nine minutes, assembled a final request of 178,000 tokens across 215 assistant turns, and consumed roughly 19 million cumulative input tokens on the way, since every earlier call and result is resent on every later request. The user, who had asked whether the API supported webhook retries, received a 504 when the ingress — the load balancer sitting in front of the workers — killed the connection.

Two of those figures check each other, and reconstructing the link is how the quadratic growth from chapter 7 stops being an abstraction. The final request held 178,000 tokens across 215 assistant turns, so each round was adding something in the region of eight hundred tokens to the history — divide the one by the other. Because the history only grows, the first request was near the prefix size and the last was the full 178,000, and a quantity growing steadily from almost nothing to a final value averages about half that value over the run. So the typical request in this turn was somewhere near ninety thousand tokens, and 215 requests at that average is around 19 million, which is the cumulative figure. Nothing was billed twice by mistake. Each of the 215 invocations was charged for the whole history it carried, and the history it carried was on average half of the largest one.

The other number to sit with is the one that never appears in a dashboard. None of this was a malfunction. There was no exception, no retry storm, no failing dependency; every component did precisely what it was asked to do, 215 times. A loop with no ceiling does not break when it runs away, which is why the symptom that reaches an on-call engineer is a resource figure rather than an error, and why the shape of the problem is not visible from any one of its steps.

The cap, and what you send when it trips#

Adding a counter is trivial. The part that takes thought is what happens on the iteration that exceeds it, because stopping badly leaves the user with nothing and leaves a broken conversation in your database.

The wrong version raises an exception at the top of the loop. Doing that discards a conversation containing real findings, returns an error to a user whose question may well be answerable from what has already been retrieved, and — if you persist messages as you go — stores an assistant turn whose tool-use blocks have no matching results, which is a conversation that cannot be resumed because the provider will reject it.

Ending cleanly takes two things. Every outstanding call still gets a result, because the structural requirement from chapter 5, that a turn with a dangling request is rejected, does not relax just because you have run out of budget, and the result says plainly that the budget is gone. Then one final request goes out with tool calling switched off, so the model has no option but to answer with what it has.

Setting tool_choice to none is the one place in this topic where removing the model's freedom is the right move, and it is worth seeing why this case is different from the forcing case in chapter 6. Forcing a named tool removes the option to decline and then demands arguments the conversation may not contain, which is how a required field gets an invented value. Switching tools off removes options and demands nothing: the only thing left is prose, and prose about a history the model has in front of it. Every input the final request needs is already present. That is the test for whether constraining the model is safe — not how strong the constraint is, but whether what remains permitted can be satisfied from what the conversation contains.

MAX_ITERATIONS = 8

for i in range(MAX_ITERATIONS + 1):
    exhausted = i == MAX_ITERATIONS
    resp = client.messages.create(
        model="claude-sonnet-4-5", max_tokens=1024,
        tools=TOOLS,
        tool_choice={"type": "none"} if exhausted else {"type": "auto"},
        messages=messages,
    )
    messages.append({"role": "assistant", "content": resp.content})

    if resp.stop_reason != "tool_use":
        return text_of(resp)

    calls = [b for b in resp.content if b.type == "tool_use"]
    if exhausted:                      # cannot happen with tool_choice none,
        break                          # but the branch keeps the loop total

    messages.append({"role": "user", "content": dispatch(calls)})

    if i == MAX_ITERATIONS - 1:        # last chance to use a tool
        messages.append({"role": "user", "content": [{
            "type": "text",
            "text": "Tool budget for this turn is exhausted. Answer the "
                    "customer with what you have retrieved, and say clearly "
                    "what you could not determine.",
        }]})

The wording of that final message decides the quality of the answer you get. Telling the model that the budget is exhausted and asking it to state what it could not determine produces "I couldn't find documentation on webhook retry behaviour — I've opened a ticket with the docs team so you get a definitive answer." Saying nothing and simply disabling tools produces a model that answers from memory as though the searches had succeeded, which is worse than the timeout, because it is a confident answer about a feature nobody has documented.

Iteration count and wall clock measure different things#

A deadline in the HTTP handler is the obvious alternative, on the grounds that what actually hurt was nine minutes of a held worker and a 30-second timeout would have ended it. The deadline is worth having and it does not do the job of the cap. A timeout in the handler terminates the request from outside the loop, so the user gets a 504 rather than the partial answer the model could have written, and whatever the loop was holding — an in-flight write, a half-persisted conversation — is abandoned mid-step. It also bounds the wrong quantity. The 214 calls averaged 2.5 seconds each because search_docs hits a vector store over the network; the same runaway against a cached, in-process tool returning in 40 ms would fit roughly 2,000 iterations into the same nine minutes, and a 30-second deadline would still permit around 400. Cost and context growth track iterations, not seconds.

The relationship those illustrative figures are pointing at is a division: the number of iterations a deadline permits is the deadline divided by the duration of one iteration, and the duration of one iteration is the model round trip plus the tool execution. Which of those two terms dominates is what decides whether a clock behaves anything like a counter. When the tool takes seconds, the tool term dominates and a deadline is a rough proxy for an iteration cap. When the tool returns in milliseconds, the model round trip dominates and the same deadline permits a completely different number of rounds, without a line of your code changing. A bound whose effective value moves by an order of magnitude when somebody adds a cache in front of a tool is not a bound on the thing you meant to bound.

Checking the clock inside the loop is what makes the deadline usable, since a deadline = now + 25s compared at the top of each iteration lets you take the same clean exit as the iteration cap — outstanding results returned, one tool-free request, a real answer to the user — while still protecting the worker. The infrastructure timeout stays at 60 seconds behind both of them, doing what backstops do.

None of these bounds knows whether the loop is progressing#

Every mechanism here counts something, and none of them knows whether the work is going anywhere. Iteration 63 and iteration 2 are indistinguishable to a counter, though one was worth running and the other was not, and the signal that separates them — that the last four results contained nothing the earlier ones did not — is available in principle and hard to compute reliably.

The available approximations each fail on a different case. Deduplicating results by hash catches the exact-repeat case and misses the reformulation case, which is what actually happened here. Embedding successive results, meaning turning each one into a vector so that near-duplicates score as similar, and stopping when similarity crosses a threshold works in demos and produces false stops on legitimate refinement, where each search genuinely does return something new. Asking a second model call to judge whether the loop is converging costs a model call per iteration on top of the one you are already making.

So the state of the practice is a blunt cap, tuned from your own distribution, with a graceful exit. That is unsatisfying in the way that a retry limit is unsatisfying — it will occasionally cut off work that would have succeeded on the ninth attempt — and it is the version that keeps a worker alive at 3 a.m. The more interesting long-term answer is that the model should be given the budget as information: telling it in the system prompt that it has eight tool calls for this turn, and including the count remaining in each tool result, converts an invisible constraint into something it can plan against, and in replay it reduces the number of turns that hit the cap without changing what the cap guarantees.

The ceiling, then, is a small integer living in your loop, chosen from a distribution you measured rather than from a number that sounded safe, paired with a clean exit that returns a result for every outstanding call and asks the model one last time with tools switched off. Nothing about the arrangement is clever, and its whole value is that the guarantee does not depend on the judgement of the component that produced the problem.

That is the protocol entire: a definition the model can read, a call it emits, a result you compute and return, an iteration that repeats while the model asks for more, and bounds you own on how far it goes. The parts that remain are the ones that only appear when you build with it — deciding which of your existing endpoints deserve to become tools, writing the descriptions that make them selectable, choosing what each one returns, and watching a real distribution of calls to find out which of those decisions were wrong.