The tool-calling loop

The tool-calling loop is what the round trip from chapter 1 becomes once the model is allowed to react to what came back. You send the conversation, the model asks for a tool, you run it and append the result, you send the conversation again — and the response to that second request is not necessarily an answer. It may be another tool call, made in light of something the model did not know when it made the first one.

A handler written for a single call reads the response after sending the results, finds text, and returns it to the user. That handler produces a correct answer only when the model happens to be finished, and produces a truncated or empty reply the moment it is not. The condition being tested is not whether you have already run a tool but whether this response contains a tool-use block, evaluated fresh every time, and that one change is the difference between a request-response exchange and a loop whose length the model decides.

A result is an input, not an ending#

A customer wrote in with a single sentence: "why have I been charged twice for AB-40912?" Answering it took five requests to the model, and the trace reads as a chain in which each link is the model reacting to something it did not know when it made the previous decision:

iter 1  → model: tool_use lookup_order(AB-40912)
        ← result: {status: "shipped", total: 89.90, customer: cus_8841}
iter 2  → model: tool_use list_charges(order_id=AB-40912)
        ← result: [{id: ch_71, 89.90, "2025-03-02", captured},
                   {id: ch_92, 89.90, "2025-03-02", captured}]
iter 3  → model: tool_use get_charge(ch_92)
        ← result: {idempotency_key: null, source: "retry", parent: null}
iter 4  → model: tool_use search_docs("duplicate charge policy")
        ← result: "Duplicate captures are refunded within 5 working days…"
iter 5  → model: text, stop_reason end_turn
        "You were charged twice on 2 March — a retry captured a second
         €89.90. I've flagged ch_92 for refund; it clears in 5 working days."

Nothing in the first request predicted that shape. The model asked for the order, and only the order's existence made the charge list worth requesting; only two identical captures made the second charge's provenance worth checking; only a confirmed duplicate made the refund policy relevant. Each result was an input to a decision that had not been made yet.

The loop that implements it is about fifteen lines, and its shape is the same across every agent framework you will meet, because there is only one shape available:

messages = [{"role": "user", "content": user_text}]

while True:
    resp = client.messages.create(
        model="claude-sonnet-4-5", max_tokens=1024,
        tools=TOOLS, messages=messages,
    )
    messages.append({"role": "assistant", "content": resp.content})

    if resp.stop_reason != "tool_use":
        return text_of(resp)                     # the only exit

    calls = [b for b in resp.content if b.type == "tool_use"]
    results = dispatch(calls)                    # concurrency classes as before
    messages.append({"role": "user", "content": results})

The while True is not laziness in the example. It is a faithful rendering of what the protocol specifies, which is that the exchange continues for as long as the model keeps emitting tool-use blocks and ends when it stops. Read that line as a resource commitment rather than as control flow: the number of model invocations, tool executions and tokens this handler will consume is a value the model chooses at run time, one iteration at a time, and your code has agreed in advance to whatever it picks.

Within a single iteration the model can still emit several calls at once, so the two axes are independent. The duplicate-charge conversation was five iterations of one call each, which is a narrow chain. A different conversation might be two iterations of four calls each: a fan-out of independent lookups, then a decision, then a fan-out of writes. Iterations are sequential because each one depends on the last, and the calls inside an iteration are concurrent because the model committed to all of them without seeing any answers.

What the model is reading on each pass#

It is worth being precise about what a fresh iteration contains, because the intuition that the model "remembers" the previous step is wrong in a way that matters for cost. Each request is built from scratch and carries the entire history: system prompt, tool definitions, the user's message, every assistant turn including its tool-use blocks, and every result you have returned. Nothing is retained on the provider's side between iterations, so the fifth request contains four rounds of calls and results in full.

The intuition that the model remembers is tempting because the behaviour is indistinguishable from memory. The model does refer back to the order record it saw three iterations ago, it does not ask for it twice, and it draws a conclusion that depends on holding all four results at once. Every one of those observations is true. What produces them is not a retained state on the provider's side but the fact that you re-sent the whole transcript, so the material was in front of the model again. The distinction is invisible in the behaviour and decisive in the bill, and it is the reason a conversation that feels like it is getting cheaper — the model asking for less each round — is getting more expensive.

The token arithmetic for the duplicate-charge conversation, with a 900-token system prompt and six tools at about 110 tokens each, runs like this:

Every figure in that table can be rebuilt from two numbers and the rule about resending, which is the point of putting it there. Start with what is in every request regardless: the 900-token system prompt plus six definitions at about 110 tokens each — six times a hundred and ten is six hundred and sixty — for a fixed prefix of roughly 1,560 tokens, which is the same 1,560 the caching paragraph below refers to. Add the customer's one-sentence question and the first request is 1,640. From there each row's input is the previous row's input plus whatever that iteration appended, because nothing is ever taken away: the increments are two hundred and seventy, then three hundred and thirty, then two hundred and thirty, then four hundred and ninety. The last increment is the largest because iteration four returned the 380-token documentation passage, and that passage is then part of the fifth request's input.

The cumulative column is the running total of the input column, and it is the column the invoice is computed from. Adding the five input figures in order gives 1,640, then 3,550, then 5,790, then 8,260, then 11,220. Addition is the right operation because each of the five was a separate billed invocation, exactly as in chapter 1 — the provider does not know the fifth request is a continuation of the first, and charges it as though it had arrived on its own.

The reply cost 11,220 input tokens rather than the 1,640 a tool-free answer would have used, because every result is paid for again on every subsequent iteration. Growth in cumulative cost is quadratic in the number of iterations for results of roughly constant size, which is the arithmetic behind the observation that a conversation going twenty rounds does not cost four times a conversation going five. It costs closer to sixteen times, and a single 700-token result returned on iteration three is 700 tokens on iterations four, five and six as well.

Quadratic is worth deriving rather than accepting, because it is the fact that decides where to spend effort. If each iteration adds roughly the same amount to the request, the first request carries one unit, the second two, the third three, and the n-th carries n. The total billed for the conversation is the sum of those, which grows with the square of the number of iterations rather than in proportion to it. Squaring is where the multiplier comes from: four times as many rounds is four squared — sixteen — times the tokens, not four times. The practical reading is that halving the size of an early result is worth more than halving a late one, because an early result appears in more of the terms being added.

Latency compounds along the same axis and is more visible to the user. The five model round trips at 2.4 to 3.1 seconds each cannot be overlapped at all, because iteration four cannot start until iteration three's result exists, and the tool executions sit between them, averaging 600 milliseconds across this handler's traffic. What took this particular conversation to the ninety seconds the latency dashboard flagged was the p95 tail on the vector-store search behind search_docs — the slowest call in twenty — landing inside a chain that could not proceed without it.

That last sentence is doing more work than it looks, and the arithmetic shows why. Five round trips at the stated range account for somewhere between a dozen and sixteen seconds. Five tool executions at the stated average add about three more. Everything else in the ninety — the large majority of it — was one call that happened to land in its own slow tail. A p95 figure is the value that only one call in twenty exceeds, so on any single request it is unlikely, and across a five-iteration chain there are five chances for it to occur. The chain has no way to absorb it: iteration five cannot begin until iteration four's result exists, so a slow call is not averaged away against fast ones, it is added to them in full. This is the reverse of the parallel case in chapter 5, where the turn's duration was the maximum of its calls. Here the durations are in a sum, and a sum is exactly where a long tail does the most damage.

Two mitigations exist and neither removes the growth. Prompt caching covers the stable prefix — system prompt and tool definitions, roughly 1,560 tokens here — and on providers that support caching the growing message list too, cached reads bill at a fraction of the input rate, which flattens the money without touching the clock. Trimming what tools return, as with the 700-token ORM row from chapter 1, reduces the per-iteration increment and pays off on every later iteration, which makes it the highest-leverage thing you can do to a loop that goes more than three rounds.

The one place the exchange can stop#

The protocol offers exactly one stopping condition, which is the model declining to emit a tool-use block, and it is worth sitting with the fact that this is the component whose behaviour you are least able to predict. There is no field in the request that says "at most four rounds". There is no error the provider returns when a conversation has been going round for a while. There is no signal in the response distinguishing an iteration that is making progress from one that is repeating itself. The API's view is that each request is independent and complete, and the fact that this is the fifth request in a chain is knowledge that exists only in your process.

Everything that behaves like a bound in practice binds something adjacent. The context window, which is the hard ceiling on how many tokens a single request may contain, terminates a runaway loop eventually by making the request too large to send — for a 200,000-token window and results averaging 500 tokens, that is somewhere near 250 iterations, which is a long way past the point at which anyone wanted the conversation to stop and it arrives as an exception rather than as an answer. max_tokens bounds the output of one iteration and has nothing to say about how many there are. A rate limit slows the loop without ending it. None of these is a stopping condition you designed, and each of them ends the conversation in a state where the user has waited a long time and received nothing.

The context-window figure is worth reconstructing, partly because it is the one people reach for as a safety argument and partly because the obvious way of computing it gives the wrong answer. Dividing the 200,000-token window by the 500-token average result suggests something nearer four hundred iterations. That division is wrong because a result is not the only thing an iteration adds. Each pass appends the assistant turn as well — its text, its tool-use block, the serialised arguments — and both halves stay in the history. The per-iteration increment is therefore the result plus the assistant turn, which is what brings the count down to the region of 250. The general form is the window divided by what one round adds, and one round adds everything the round produced on both sides.

Even at the right figure, treating the window as a bound has the direction of the argument backwards. A limit that terminates a runaway after 250 rounds is not protecting the user, who left after the first minute; it is protecting the provider's memory allocator. It also arrives as a request-too-large error, which means the conversation ends with no answer and a stored history that is now too big to retry. A bound designed for the user has to trip long before this one, and it has to trip in a way that produces a reply.

Where a loop stops being the right shape#

Not every multi-step task should be a tool-calling loop. If the sequence is known in advance — validate the order, check the refund window, issue the refund, send the confirmation — then writing it as four steps in your code with the model filling in the parts that need judgement is cheaper, faster and far easier to test. The loop earns its cost when the branching is genuinely data-dependent, as it was above, where the third step existed only because two charges came back identical. The failure mode of using a loop where a pipeline belongs is quiet: it works, and it costs five model round trips to execute a fixed procedure that needed one. The failure mode of the opposite mistake is louder, and it produces a handler that can only ever do the thing you anticipated.

Persistence deserves a decision here rather than a default, because an iteration is a natural-looking place to write to the database and a mid-loop conversation is not a valid resting state. If the worker dies between emitting a tool-use block and returning its result, what is stored is a conversation the provider will refuse on the next request, and resuming it means either replaying the dispatch to fill in the missing results or truncating back to the last complete iteration. Writing the assistant turn and its results together, as one unit at the end of each pass, keeps every stored state resumable, and storing the iteration index alongside them means a resumed conversation continues counting from where it stopped rather than starting its budget again.

The loop is also not an agent framework, and the distinction is worth drawing because the two get conflated. What is described here is a transport-level iteration with no planning, no memory beyond the message list, no notion of a goal and no evaluation of progress. The model's decision on each pass is local: given this history, what is the next thing to ask for. That locality is exactly why it works well on the duplicate-charge conversation, where each step follows obviously from the previous result, and exactly why it has no defence against a chain of individually reasonable steps that goes nowhere. Frameworks that add planners, scratchpads or explicit task decomposition sit on top of this loop; they change what the model tends to do inside it and they do not change the fact that the loop's length is still the model's decision.

A result is an input, the test is whether this response contains a call, and the exchange ends when the model stops asking. Everything expensive about the arrangement follows from that last clause, because the model is choosing the iteration count without being able to see it — each pass is a local decision made against a history in which nothing counts anything.

What the loop does not have anywhere in it is a number. One production conversation on this same handler issued 214 calls to search_docs over nine minutes before something else killed it, and every one of those iterations passed through the same fifteen lines, each looking locally reasonable to the model that chose it. Where the ceiling belongs, what you send back when it trips, and why an iteration count and a wall-clock deadline are not substitutes for one another, are the subject of the next chapter.