Parallel tool calls

Parallel tool calls are what you have when one assistant turn asks for more than one thing at once. The response is a list of content blocks, there is no rule that at most one of them is a tool call, and a model that can tell from the conversation that it needs four separate pieces of information, none of which depends on another, emits four blocks in a single turn. Both major providers do this by default and both expose a switch to turn it off.

What that changes on your side is the dispatcher. Code written for one call per turn is wrong in two directions at once: it drops the calls it did not read, which the provider rejects, and where it does handle several it tends to run them one at a time, which turns the tool phase of a turn into the sum of its latencies rather than the longest of them.

One assistant turn, several requests#

An order-status turn on a support assistant with a broad toolset comes back like this:

{
  "stop_reason": "tool_use",
  "content": [
    { "type": "text", "text": "Let me pull the order and the delivery status." },
    { "type": "tool_use", "id": "toolu_01Ha", "name": "get_order",
      "input": { "order_id": "AB-40912" } },
    { "type": "tool_use", "id": "toolu_01Hb", "name": "get_customer",
      "input": { "customer_id": "cus_8841" } },
    { "type": "tool_use", "id": "toolu_01Hc", "name": "get_shipment_tracking",
      "input": { "order_id": "AB-40912" } },
    { "type": "tool_use", "id": "toolu_01Hd", "name": "search_docs",
      "input": { "query": "delivery delay compensation policy" } }
  ]
}

The identifiers stop being bookkeeping here. With one call the mapping from result to request is obvious; with four it is carried entirely by tool_use_id, and the order of the result blocks is not what matches them. Returning results in a different order than the calls arrived is fine as long as each is tagged, and returning the right content under the wrong id produces a model confidently describing a shipment using the customer record.

Those four calls measured 180 ms, 240 ms, 1,900 ms against a carrier's API and 820 ms against the vector store behind search_docs. Walked with a plain for loop that is three point one four seconds of tool time, spent one call at a time, sitting between two model round trips of roughly 2.7 seconds each. Running them concurrently takes the tool phase to 1,900 ms, the duration of the slowest one, and returns 1.24 seconds to every turn of that shape. That is the entire optimisation, and it is available because the model emitted all four requests before seeing any of the answers, which is a statement about their independence that you get for free.

Both of those totals are worth reconstructing, because which arithmetic applies is the whole point. Sequential work adds: the second call cannot start until the first has returned, so the elapsed time is 180 plus 240 plus 1,900 plus 820, which is the three-point-one-four figure. Concurrent work does not add, because all four requests are in flight from the same instant and the clock is not waiting on any one of them in particular; the turn is over when the last one lands, so the elapsed time is the largest of the four rather than their sum, and that is 1,900 ms. The saving is the difference between the two, three point one four seconds less one point nine, which is where 1.24 seconds comes from.

Two things follow from a maximum replacing a sum, and both are useful when you are deciding where to spend effort. Adding a fifth fast lookup to a concurrent turn is very nearly free, because it changes the maximum only if it is slower than 1,900 ms. And the only call worth optimising is the slowest one — halving the 180 ms lookup returns nothing at all here, while halving the carrier API returns almost a second. Under the sequential dispatcher the opposite was true: every call's saving landed in the total, because every call was in the sum.

An asyncio dispatch that preserves the tagging is short, and the one detail that matters is that failures are captured per call rather than allowed to cancel the group:

async def run_one(block) -> dict:
    try:
        payload = await TOOLS[block.name](**block.input)
        return {"type": "tool_result", "tool_use_id": block.id,
                "content": json.dumps(payload)}
    except Exception as exc:                     # captured, not raised
        return {"type": "tool_result", "tool_use_id": block.id,
                "is_error": True,
                "content": json.dumps({"error": type(exc).__name__,
                                       "detail": str(exc)[:200]})}

calls = [b for b in resp.content if b.type == "tool_use"]
results = await asyncio.gather(*(run_one(b) for b in calls))
messages.append({"role": "assistant", "content": resp.content})
messages.append({"role": "user", "content": list(results)})

asyncio.gather without return_exceptions=True would propagate the first exception and abandon the other three coroutines, leaving you holding one traceback and no results at all for a turn in which the model asked four questions. Catching inside run_one means every branch produces a result block, which is the property the rest of this chapter turns out to depend on.

Concurrency is a property of the tools, not of the turn#

The model emitting four calls together is evidence that it believes they are independent, and for read-only lookups that belief is usually right. It is not a guarantee, and it says nothing about the constraints your systems have. Consider a different turn from the same assistant, where the four calls are get_order, get_customer, charge_card and send_receipt_email.

The two lookups go together without argument. The two writes do not, and for two separate reasons. They have an order dependency, because a receipt quoting a charge that has not been made is a support ticket in its own right, and if the charge is declined the email should never be sent at all. They also have side effects that are not idempotent from the customer's point of view, so interleaving them unpredictably makes the failure modes timing-dependent. A parallel dispatch that treats the four calls as a homogeneous batch will send the receipt and the charge simultaneously and be right most of the time, which is the worst possible failure rate for something you have to debug at 3 a.m.

The rule that survives contact with production is to classify tools rather than turns. Each tool carries a declared concurrency class in your registry — safe to run with anything, safe to run with reads only, must run alone and in the order the model emitted — and the dispatcher groups the calls in a turn accordingly, running each group in sequence and the members of a group in parallel. A read-only lookup is parallel; anything that writes, charges, sends or cancels is serial. This is your decision to make in code, because the model has no view of your transactional boundaries and no way to acquire one.

Classifying the tool rather than the turn is the right unit for a reason that is easy to state and easy to skip past. A turn's composition is chosen fresh by the model on every request, so any property you attach to a turn has to be recomputed, by you, from a set of calls you have not seen before. A tool's properties are fixed at the moment somebody writes the tool: get_order reads and will always read, charge_card writes and will always write. Attaching the class to the fixed thing means the dispatcher's job on any turn reduces to grouping by a field it can look up, with no inference involved. It also puts the decision where a human can be asked for it — at code review, when the tool is added — rather than at runtime, when nobody is available.

The belief this displaces is that four blocks in one turn are the model's assurance that the four are independent. It is tempting because the model did commit to all four without seeing any answers, and that genuinely is evidence: it did not think it needed the first answer to write the second call. What it is not is a claim about your systems. The model knows nothing about the row locks the payments service takes, the ordering the mailer assumes, or the fact that a receipt is a customer-visible artefact and a charge is a bank-visible one. It is reporting on its own reasoning, and its own reasoning is not where the constraint lives.

Both major providers also let you turn multi-call turns off entirely, which is occasionally the correct blunt instrument. Anthropic takes "tool_choice": {"type": "auto", "disable_parallel_tool_use": true} and OpenAI takes a top-level "parallel_tool_calls": false; either way the model emits at most one call per turn, and a task that needed four now takes four round trips instead of one, costing roughly 8 seconds of extra latency and four times the input-token billing on the growing history. That trade is worth it for a toolset that is entirely writes and nothing else. For a mixed toolset, classifying tools costs less.

The latency figure there comes from the round-trip number above and not from anywhere else. One turn becomes four, so three model round trips are added, and each of those was measured at roughly 2.7 seconds — three times two point seven is a little over eight. It is worth being careful about which quantity is being multiplied: the extra cost is three round trips rather than four, because the turn was always going to make one. The same care applies to the billing clause, which says four times the input-token charge and not four times the bill. Each of the four requests re-sends the system prompt, the tool definitions and whatever history has accumulated, so the prefix is paid four times over; the output tokens are not multiplied, since the model emits the same four calls in total either way.

Every call needs a result, including the one that failed#

In one turn the model emits get_weather("Madrid"), get_weather("Lisbon") and book_flight(...); your code runs them, and Lisbon comes back 503 from the weather provider. The temptation is to return the two results you have and let the model work it out, or to abandon the turn and start again.

Neither works, and the first one does not even reach the model: the API rejects a request whose assistant turn contains three tool_use blocks followed by a user turn containing two tool_result blocks, with an error about a missing result. The requirement is structural. Every call the model made in that turn needs a block answering it before the conversation can continue, because the provider will not reconstruct a valid state from a turn with a dangling request. What goes in the block for Lisbon is an error payload, tagged with the same identifier as the call, and the model reads it exactly as it reads the other two.

The reason that requirement exists rather than being a quirk is worth one step down, because it explains several other rules in this topic at the same time. The provider holds no state between requests, so the message list you send is rendered into the token sequence the model was trained on, and that rendering has to produce a coherent transcript. A call with no answer beside it has no coherent rendering: it is a transcript in which a question was asked and the next thing that happens is a new question from the same speaker, which is not a shape the model was trained on and not a shape the provider is willing to construct. Rejecting the request at the API boundary is the provider declining to guess what the missing answer was. The alternative — filling the gap with silence — is the failure from the first chapter of this topic, and it produces a model writing around a hole rather than an error you can see.

That is what makes an isolated failure cheap. The model sees Madrid at 19 °C, a 503 against Lisbon, and a confirmed booking, so its next move is to retry the one leg that failed or to tell the user that Lisbon's forecast is temporarily unavailable while reporting the rest. Discarding the turn and re-prompting throws away the booking that succeeded and the Madrid lookup that cost you a request, and it puts the model back at a decision point it has already passed, from which it may well book a second flight. Discarding and restarting a turn is not a neutral operation once one of its calls has already changed the world.

What goes into the error content is worth being deliberate about, because the model's next action is decided by it and by nothing else. A bare {"error": true} is an invitation to retry immediately and indefinitely.

The last two rows carry most of the value. A retryable flag set honestly is the difference between a model that backs off and one that spends four iterations hammering a dead dependency.

A timeout on a write is not a failure#

charge_card goes out and eight seconds later your HTTP client raises a read timeout. What you know is that the request left your process. What you do not know is whether the payment processor received it, whether it committed a charge, and whether a response is at this moment being written to a socket nobody is reading. The customer may have been charged €89.90 or may not have been, and the timeout carries no information either way.

Writing {"error": "charge failed"} into the result asserts something you have not established, and the model's response to a failed charge is reasonable and dangerous: it re-issues charge_card with identical arguments. If the first charge did commit, that is a second charge on the same card for the same order, and it will surface as a chargeback rather than as an alert.

The result should say what is true, which is that the outcome is unknown, and it should point at the tool that can resolve it. Shown here with the content expanded rather than JSON-serialised into the string the field actually carries:

{
  "type": "tool_result",
  "tool_use_id": "toolu_01Hc",
  "content": {
    "status": "unknown",
    "detail": "charge_card did not respond within 8s. The charge may or may not have been taken.",
    "next": "call get_charge_status with order_id AB-40912 before re-issuing"
  }
}

What does not go in there is a guess, an apology, or an instruction phrased as a prohibition the model is expected to honour under all circumstances. It also should not carry the raw exception text, since a stack trace mentioning ConnectionResetError reads to the model like a definite failure and pushes it toward exactly the retry you are trying to avoid.

Content is evidence, not enforcement#

Suppose you write the unknown-outcome result perfectly and the model re-issues charge_card anyway, with the same arguments. Nothing in the result content prevents that, because content is evidence rather than enforcement, and a model that has been asked by the user to complete a purchase has a strong pull toward completing it. Whether the customer is charged twice is therefore not decided in the tool result at all. It is decided by whether your dispatcher derives a stable idempotency key for that intent and whether the payment processor has recorded that key against the committed charge, in which case the second request returns the first charge's identifier and no money moves.

This is the same at-least-once delivery problem as any queue consumer or webhook receiver, with the model in the role of the retrying client, and the same answer applies: the receiver deduplicates, because the sender cannot be trusted to send once. The key derivation from chapter 4, where the dispatcher hashes the conversation id, the order and the amount rather than accepting a key from the model, is what makes the retry safe, since a re-issue of the same intent within the same conversation produces the same key and collapses into the original charge. A key minted fresh per dispatch call, or taken from the model's arguments, gives you two distinct keys for one intent and no protection whatsoever.

That comparison to a queue consumer holds for the mechanism and breaks on one property worth knowing about before you rely on it. A retrying queue consumer replays a byte-identical message, so any deduplication scheme that keys on the message will catch every duplicate. A model re-issuing a call is not replaying; it is generating again, and it may generate slightly different arguments the second time — a rounded amount, a different order among several under discussion. The key derived above is a hash of the conversation, the order and the amount, so it collapses a re-issue only when all three are unchanged. A second attempt at €89.90 for the same order in the same conversation hashes to the same key and is safely absorbed. A second attempt that differs in the amount hashes differently, and the processor has no grounds to treat it as a duplicate because, as far as anything on that side can tell, it is not one. The protection is against repetition of an identical intent, which is the common case, and it should be described that way to whoever signs off on it rather than as protection against double charging in general.

Both halves of this chapter follow from one fact about a turn: the model committed to every call in it before it saw any answer. That is why the calls can be run at the same time, and it is why every one of them has to be answered — the model is holding a set of outstanding questions, not a queue it will re-ask. Classify the tools so concurrency follows your transactional boundaries rather than the model's guess about them, tag every result with the id of the call it answers, and describe a timed-out write as unknown rather than as failed.

Concurrency in a single turn is bounded by construction: however many calls the model emitted, that is how many you run. What is not bounded is what happens after the model reads the results, because a result is an input to the next request rather than the end of anything. The next chapter deals with a different question that arrives sooner — what to do when the model's freedom to choose no tool at all is the thing causing your problem.