The assistant turn, then one result block per call id

A tool call is a block inside a model's reply that names a tool and supplies arguments for it, and the provider stamps each one with an identifier so that the output it produces can be attributed back to it later. A model may emit several of those blocks in a single turn, which is what lets three independent lookups run at the same time instead of over three round trips. What that costs you is a bookkeeping obligation, and the obligation is stricter than it looks: the next request you send has to reproduce the assistant turn exactly as it arrived and then answer every one of its calls, in one message, with nothing missing and nothing extra.

Getting it wrong is not a rare or exotic failure. It is the most common way a working agent produces a broken request, and its distinguishing feature is that the expensive machinery is innocent — the network calls, the database queries and the money movements all succeed, and the request is rejected anyway by a rule about list membership. The rule those three lines of bookkeeping have to satisfy is short enough to state in one sentence, and the sentence has more consequences than it looks like it has.

The rule, and the request it produces#

An assistant turn that contains N tool_use blocks obliges the next request to contain that assistant turn unchanged, immediately followed by a single user message containing N tool_result blocks, one per call, each carrying the tool_use_id the model assigned. Nothing between them, nothing missing, nothing duplicated.

That sentence is easier to use if it is restated as an equation, because an equation can be checked and a sentence has to be remembered. Collect the ids of the tool_use blocks in the assistant turn into one set. Collect the tool_use_id values of the tool_result blocks in the user message that follows it into a second set. The rule is that those two sets are equal. Equality of sets is a two-sided claim — every id in the first appears in the second, and every id in the second appears in the first — and each side rules out a different family of mistakes, which is why one statement does the work of a list of prohibitions.

Take the left-to-right side first. Every id in the assistant turn must appear among the results, so a call whose result was dropped breaks the equality, whatever the reason for dropping it. It does not matter whether the tool raised, whether a branch returned early, whether the code that gathers results skipped a failure, or whether somebody decided an empty answer was not worth sending; the id is in the turn, so a block carrying it must be in the following message. Now the right-to-left side. Every id among the results must appear in the assistant turn, so a duplicated result breaks the equality — the second copy carries an id that is already accounted for and there is no second call for it to answer — and so does an invented id, and so does a result left over from a previous turn. Those look like three different bugs on a dashboard and they are one violation seen from three angles.

Here is the correct shape for a three-call turn, with the tool outputs abbreviated:

[
  { "role": "user", "content": "Where is my order and can I still return it?" },
  { "role": "assistant", "content": [
      { "type": "text", "text": "I'll check the order, the shipment and the policy." },
      { "type": "tool_use", "id": "toolu_01Ab", "name": "search_orders",
        "input": { "customer_id": "cus_8123" } },
      { "type": "tool_use", "id": "toolu_01Cd", "name": "get_shipment",
        "input": { "order_id": "4471" } },
      { "type": "tool_use", "id": "toolu_01Ef", "name": "lookup_policy",
        "input": { "topic": "returns" } }
  ]},
  { "role": "user", "content": [
      { "type": "tool_result", "tool_use_id": "toolu_01Ab", "content": "[{\"id\":\"4471\",...}]" },
      { "type": "tool_result", "tool_use_id": "toolu_01Cd", "content": "{\"carrier\":\"DHL\",...}" },
      { "type": "tool_result", "tool_use_id": "toolu_01Ef", "content": "Returns accepted within 30 days..." }
  ]}
]

The pairing is by id, not by position. You may build the results list in whatever order your dispatch finishes — which matters, because concurrent dispatch is the reason the model asked for three at once — and the model will still attribute each output to the call that produced it. What you may not do is leave one out, invent an id, send four results for three calls, or split the three across two user messages.

Pairing by id rather than by position is what makes the equality the right description in the first place, and it is worth seeing why the alternative would not have worked. If results were matched by their order in the list, then the results message would have to be an ordered sequence rather than a set, and building it would mean holding a slot open for every call until that call returned. Concurrent dispatch would then be pointless: the slowest tool would gate the assembly of the message, and any code that reordered results for readability would silently reattribute outputs to the wrong calls. Because the id travels with the result, the message is a set and the order inside it carries no meaning, which is exactly the property that lets three tools run at once.

The code that satisfies this is not complicated once the invariant is stated, and it is worth writing so that the failure is structural rather than incidental:

def append_turn(messages: list, response, dispatch) -> list:
    """Append the assistant turn and, if it asked for tools, exactly one
    result block per call id in a single user message."""
    messages.append({"role": "assistant", "content": response.content})

    calls = [b for b in response.content if b.type == "tool_use"]
    if not calls:
        return calls

    results = run_concurrently(calls, dispatch)          # dict: call id -> str
    assert set(results) == {c.id for c in calls}, "one result per call, no extras"

    messages.append({
        "role": "user",
        "content": [
            {"type": "tool_result", "tool_use_id": c.id, "content": results[c.id]}
            for c in calls
        ],
    })
    return calls

The assertion earns its place. It converts the two most common mistakes — a result dropped because a dispatch branch returned early, and a result added twice because a retry re-appended — into a failure at the line that caused them rather than a 400 with a message index in it.

Read that line against the equation and it is the equation, written in Python rather than in prose. set(results) is the set of ids your dispatch produced; {c.id for c in calls} is the set of ids the model asked for; == is the two-sided claim. That is why a single assertion catches both a dropped result and a duplicated one without a second check for either: the same property is doing both jobs, because a set equality has two sides and each side is one of the failures. It is also why the assertion fires at the moment of the mistake and the API's rejection does not — your code is holding both sets, and the provider is holding only the serialised outcome.

Four ways to break it while every tool succeeds#

The failures cluster, and they are worth separating because they have different symptoms and different fixes.

Three of those four rows are the equality failing, and naming which side fails is faster than memorising the row. Dropping the assistant turn empties the left-hand set entirely, so every id among the results is unmatched — which is why the error names an unexpected id rather than a missing one, and it is the same violation as a duplicate seen at a larger scale. A missing result leaves an id in the turn with nothing answering it, which is the left-to-right side failing. A duplicate leaves an id among the results with nothing to answer, which is the right-to-left side failing. The fourth row is the one the equality has nothing to say about, and it is treated on its own below.

The first row is the one that shows up most often in real code, and the reason it is tempting is that it reads like the right thing. The mental model is "the tool produced something, so append it", and appending the output is the part that feels like work. The assistant turn feels like bookkeeping — it is a message the model produced, it is already in the response object, nothing about it needs computing. So it is the line that gets lost when someone refactors dispatch into a helper, or adds a retry that re-enters the append path, or writes an error branch that skips the happy-path code. The result is a request that is internally inconsistent: it asserts an answer to a question it does not contain.

The fourth row is different from the other three, and more dangerous, because nothing fails. Splitting parallel results across two user messages produces a legal request, the model reads all the outputs, and the run completes. What changes is subtler: the transcript no longer shows a single turn's worth of parallel calls being answered together, and over a conversation the model becomes measurably less willing to request several tools at once. On a task where three lookups can run concurrently in 128 ms, losing parallel calling turns one step into three, which on a typical loop means roughly 2.4 extra seconds of model latency and around 5,000 extra input tokens from the two additional round trips that resend the whole history.

Both of those figures are gaps rather than levels, and reading them as levels makes them look implausible. Neither is the cost of the run. Each is the cost of the two round trips the run did not need to make: the latency figure is two model calls at a little over a second each, and the token figure is two extra resends of a history that had already been sent, at a couple of thousand tokens apiece. Whether the run as a whole is slow or cheap does not enter into it, which is why the penalty is the same on a short conversation and easy to miss on a long one, where it is a small fraction of a large total.

The dispatch timings quoted later in this chapter read the same way, and the obvious arithmetic on them is wrong. Three tools dispatched concurrently do not cost the sum of their durations. They cost the slowest of the three plus the overhead of starting them and gathering what they return, which is why the total wall time quoted for a three-lookup turn sits just above the largest of the three individual timings rather than anywhere near their total. The number to compare against a serial design is therefore the maximum, and the gap between that maximum and the sum is precisely what the fourth row of the table quietly hands back.

The stumble: a green test suite is not the invariant holding#

The belief worth naming here is that the API is checking your work. It is a reasonable belief and it is nearly true: three of the four rows above are caught by the provider, immediately, with an error message that names the problem. A team that has been taught by those three rejections learns that a malformed message list is rejected on the spot, and draws the obvious conclusion that a request which is not rejected is well formed.

What makes this the dangerous kind of mistake is that the quantity being watched genuinely stays constant while the property that matters varies underneath it. The request validates. The run completes. The answer is correct. The tests are green, and they stay green through the rest of the conversation while the transcript degrades, because there is no assertion anywhere in the system whose subject is the shape of the conversation rather than the shape of the request. The provider validates structure — are the ids matched, are the blocks in the right containers — and the thing that has changed is not structural. It is that the transcript no longer contains an example of three calls being answered together, and the model's next decision is conditioned on the transcript.

That is the general shape to carry forward, and the specific instance is worth stating plainly. A structural validator can only reject what its schema forbids, and a legal-but-degraded transcript is legal by construction. So the defence has to come from somewhere else: from a check you write, or from the arithmetic in the previous section, or from the side-by-side comparison in the next one. The API is not going to tell you.

Where the invariant gets broken in practice#

The bugs above are rarely written on purpose, and they enter through four recognisable doors. Knowing the doors is more useful than knowing the bugs, because you can check them during review.

The first is a retry wrapper. Somebody wraps the model call in a decorator that retries on 429 and 529, which is correct and necessary, and the retry re-enters a function that has already appended to messages. The second attempt then appends the assistant turn twice, or appends results for a turn that is now duplicated. The defence is to make the append path idempotent, meaning that running it twice leaves the message list in the state it would have been in after running it once, or, more simply, to retry only the HTTP call and keep every mutation of the message list outside the retried region.

The second is concurrent dispatch with a partial failure. Three tools run in a thread pool, one raises, and the code that gathers results either skips the failed one or lets the exception escape the gather. Both produce a results message with two blocks for three calls. asyncio.gather without return_exceptions=True does the second by default, which is why this shows up disproportionately in async loops.

The third is persistence. A loop that stores its trajectory — the accumulated message list for one run, in the order the next request would present it — so that the run can be resumed after a restart has to serialise the assistant turn's content blocks, including tool_use blocks with their ids, and reload them in a form the API accepts. Truncating the stored content to the text blocks — which is tempting, because text is what a support agent wants to display in an admin UI — produces a resumed conversation whose results refer to calls that no longer exist. Store the whole turn; render a subset of it.

The fourth is a mid-run change to the message list. Trimming old messages to control context growth, dropping large tool outputs, or summarising the first half of a conversation are all reasonable things to do on a long run, and each of them can orphan a result block or an assistant turn. Whatever the trimming policy is, its unit has to be the pair: an assistant turn and the user message answering it are removed together or not at all.

All four doors are the same door seen from different sides, which is the useful thing to notice about them. Each is a place where code other than the append path modifies messages, and the equality is a claim about two adjacent messages, so any edit that touches one of a pair without touching the other breaks it. The trimming rule follows from that directly rather than being an extra thing to remember: removing an assistant turn removes ids from the left-hand set, so the results answering them must go too, and removing a results message strands the turn above it.

All four have the same tell in review. Any code path that touches messages outside the one function that appends turns is suspect, and the cheapest structural defence is to have exactly one such function — the append_turn above — and to assert the invariant inside it rather than trusting every caller to preserve it.

The flattening shortcut, and what it costs later#

There is a version of this that skips the structure entirely. Instead of result blocks, the loop joins the tool outputs into prose and sends them as an ordinary user message:

# The version that passes every test you have written so far.
summary = "\n\n".join(f"Result of {c.name}: {results[c.id]}" for c in calls)
messages.append({"role": "user", "content": summary})

It works. The model reads text either way, it can tell from the wording which output came from which tool, and on a single-tool agent the difference is invisible. Teams arrive at it honestly, usually while porting between providers or while working around a serialization bug, and the tests stay green.

What it forfeits is everything that is keyed on the call id. Parallel tool calls stop being expressible: with three outputs concatenated into one blob, the association between a call and its result is a naming convention in a string, and if two calls to the same tool differ only in arguments the model has to infer the pairing from the content. Per-call error signalling goes with it: is_error is a field on a result block, so a flattened transcript cannot say "this one call failed and the other two are fine" without you inventing prose to say it, and the model's ability to retry exactly the failed call degrades accordingly. And the whole provider-side layer built on top of the structure becomes unavailable — context editing that clears old tool results by block type, server-side compaction, strict tool-result accounting in usage, and the trace tooling that reconstructs a run by joining tool_use and tool_result blocks on their ids. Replaying a stored trajectory through a different model, which is how most evaluation harnesses work, needs the same structure.

Every item on that list is the same loss, which is worth saying once rather than treating them as five separate arguments. Flattening deletes the id, and the id is what makes the association between a call and its output a fact in the data rather than an inference from the words. Everything above is something that was reading that fact: the model, when it decides which call to retry; the provider, when it prunes old results by block type; your trace viewer, when it joins two blocks on a shared key; your evaluation harness, when it replays a stored run. Once the association is prose, each of those has to guess, and most of them do not try.

If you have to pick one of those to make the case with, pick parallel calls, because it is the one you can demonstrate in a single run rather than argue about. Take a task that needs three independent lookups, run it once with paired result blocks and once flattened, and put the two traces side by side: one has a single assistant turn with three tool_use blocks and a 128 ms dispatch, the other has three round trips. The others are real but arrive later, and "later" is easy to discount in a review. A stopwatch is not.

That experiment is also the answer to a fair objection, and it is worth naming what the objection is. The claim that the model becomes less willing to request parallel calls is a claim about behaviour over a conversation, and it holds where the transcript is the model's only evidence about what parallel calling looks like — which is the normal case, and is not guaranteed for every model, every system prompt or every task. It is not something a reviewer should accept on assertion. What a reviewer should ask for is the pair of traces: the same task, the same tools, the two message shapes, and the step count and wall time of each. That is a measurement anybody can run in an afternoon, and it is the step to insist on before either believing the claim or dismissing it.

Two neighbouring ideas, and one thing the rule does not promise#

Three ideas get confused with this one often enough to be worth separating, and the first two are near neighbours that people reach for as substitutes for the rule.

The first is ordering within the user message. The rule is about which messages exist and in what order, not about the sequence of blocks inside the results message. Sorting results to match the order of the calls is a courtesy to whoever reads the transcript; it is not what makes the request valid, because the matching is done on ids. Code that carefully reorders results while forgetting to assert that the sets of ids are equal has optimised the cosmetic half of the problem.

The second is portability. This is the shape of Anthropic's Messages API, the HTTP endpoint the loop posts a conversation to and receives a turn back from, and it is not universal. An OpenAI-style endpoint expresses the same relationship with one message per result, each carrying role: "tool" and a tool_call_id, rather than one user message carrying N blocks. The underlying protocol is the same — an assistant turn requests actions, each action's output is returned tagged with the identifier of the request — and the transport differs. A loop written directly against one provider's message shapes will need a translation layer to move, and the thing worth keeping stable across that move is the pairing, not the container.

Those two containers share no machinery and arrive at the same relationship, which is the strongest available evidence that the relationship is the protocol and the container is not. One puts N results in one message; the other puts one result in each of N messages. Neither can be derived from the other, and both satisfy the same set equality on the same identifiers. What that licenses is a specific and useful conclusion about a port: a translation layer rewrites the container, and it must never touch the pairing, because the pairing is the part both providers already agree on. A port that starts by inventing its own correlation scheme has replaced the one thing that did not need replacing.

The third thing this rule is not: a guarantee that the run is correct. A perfectly formed request can carry a tool result that says [] when the model expected data, or an error string that leaks a connection URI, or the output of a tool that should never have been called with those arguments. The structure is what makes those cases expressible; it does not make them right.

The case the rule does not cover yet#

Every result block in this chapter carried the output of a tool that ran and returned. The dispatch code assumed that, and so did the assertion: results[c.id] presumes there is something to put in the slot. A tool that raises has no output, and the slot still has to be filled, because a missing result is the second row of the table above and produces a 400 regardless of how good your reason for omitting it was.

That constraint follows from the equality rather than from a separate rule, and seeing that it does is what stops the error-handling chapter from reading as a new set of obligations. The id is in the assistant turn the moment the model emits it, and nothing your code does afterwards can take it out. A tool that raises does not remove an id from the left-hand set; it only removes your ability to fill the matching slot with an output. So the slot gets filled with something else, and the question is what.

So the question the loop cannot yet answer is what a result block contains when the call it answers did not succeed — what belongs in that text, what must never be put into it, and why the version that catches the exception and appends nothing at all is worse than the version that lets it crash.