Concurrent dispatch is a protocol feature, not a safety property

An agent, in the sense used throughout this topic, is the ordinary arrangement: a loop. A model reads a task, and where it needs something from the outside world it does not act itself but emits a tool call, which is the name of one of your functions together with the arguments to run it with. Your code executes the function, returns the result, and the loop goes round again. Providers now let a model name several functions in one assistant message — parallel tool calls — and the framework that receives that message hands them to a concurrent dispatcher, which starts all of them at the same instant rather than waiting for each to finish before beginning the next. Agent state and concurrency, the subject of this topic, is the question of what happens to the things those functions touch — a row in a database, a file on disk, a counter, a request to somebody else's API — when two of them are in flight together. The characteristic failure is a race condition, which is a result that depends on which of two overlapping operations reaches the shared data first, so the same code is right on almost every run and wrong on the run where the timing falls the other way.

This chapter is about the distance between what the layers above your handler have promised and what a safe overlap would require, and one afternoon on a billing operations agent measures that distance. The agent handled about two hundred account changes a day, and on a Tuesday in March it turned a 55.00 EUR credit into a 15.00 EUR credit. The customer had two disputed line items, a 40.00 EUR overage charge and a 15.00 EUR seat charge, and the agent did exactly the right thing: it read the case, decided both were valid, and emitted two apply_credit calls in a single assistant turn. Both calls returned success. The audit log showed two entries, one for each amount. The account balance showed 15.00.

Why the calls arrive together at all#

Concurrent dispatch exists because of arithmetic that has nothing to do with correctness, and the arithmetic is worth doing rather than asserting, because its shape explains why no provider or framework treated the feature as a safety change. The first generation of function-calling APIs allowed one call per assistant turn. Follow what that costs an agent that needs some number of independent lookups, calling that number n. Each lookup is its own round trip: the model is asked to produce the next call, which is one inference; your code runs the tool, which is one execution; the result is appended to the conversation and the model is asked again. So the wall-clock cost is n inference latencies plus n tool executions, and there is a third term that is easy to miss. The prompt is re-processed on every one of those inferences, and it is longer each time, because the previous call and its result are now inside it. The total therefore grows faster than n on its own.

Put the billing agent's own figures into that. A mid-sized model took roughly 900 milliseconds per inference, so an agent needing four independent lookups paid four full round trips: four inference latencies of roughly 900 milliseconds each, plus four tool executions, plus the growing prompt being re-processed each time.

Letting the model name all four in one message changes the shape of that sum rather than shaving it. There is one inference instead of four. There is one prompt processed once rather than four prompts each longer than the last. And the four executions no longer add up, because they run at the same instant, so the execution term collapses from the sum of the four durations to the largest of them. On the billing agent's own traces the four-lookup case went from 5.1 seconds to 1.4.

Two separate decisions are packed into that improvement, and only one of them was ever discussed. The requirement is that four lookups happen and that the agent does not spend most of a turn waiting. The design decision is that they happen at the same instant, which is one way among several to meet that requirement, and it was chosen by people who were measuring latency. Every provider added the feature for that reason, and every framework added gather behind it for the same reason, which is why most teams already have it whether or not they decided to: parallel emission is on by default in the common SDKs, and the dispatcher that exploits it is three lines long. The feature is a latency optimisation that happens to be implemented by running your code at the same time, and optimisations do not come with safety analyses of the code they speed up.

What each layer in the stack actually promised#

It helps to walk the call from the model down to the disk and ask, at each step, what was guaranteed. The model produced an assistant message whose content contained two tool_use blocks:

{
  "role": "assistant",
  "content": [
    {"type": "text", "text": "Both charges qualify. Applying two credits."},
    {"type": "tool_use", "id": "toolu_01A", "name": "apply_credit",
     "input": {"account_id": "acct_8831", "amount_eur": 40.00, "reason": "overage_dispute"}},
    {"type": "tool_use", "id": "toolu_01B", "name": "apply_credit",
     "input": {"account_id": "acct_8831", "amount_eur": 15.00, "reason": "seat_dispute"}}
  ]
}

The provider's contract around that message is about shape and sequencing of the conversation. Multiple tool_use blocks may appear in one assistant message, and the client must return a tool_result block for every one of them in the single user message that follows, matched by tool_use_id. Anthropic's Parallel tool use documentation describes the emission and the reply format, and there is a disable_parallel_tool_use flag in tool_choice for suppressing it.

That matching rule is the one hard guarantee in the whole exchange, so it is worth saying precisely what it licenses and what it leaves behind. It licenses the claim that no result is dropped and none is attributed to the wrong call: every block that went out gets exactly one block back, and the identifier says which. What it leaves behind is everything about the world those calls acted on. Two handlers can satisfy the matching rule perfectly while one of them overwrites the other's write, because the rule is a statement about the shape of a conversation. The protocol has no vocabulary for saying that two calls are independent, and no mechanism by which it could find out.

The model's own contribution is a judgement about the task, made from the text in its context. It decided that crediting the overage and crediting the seat charge were two separate things that both needed doing, which is true at the level of the business process. It has no information about whether apply_credit is implemented as an atomic increment or as a read, an addition in Python, and a write. That information is not in the tool description, and if it were, the model would be the wrong component to be enforcing it.

The framework's contribution is the smallest of the three, and the easiest to over-read:

async def dispatch(blocks: list[ToolUse]) -> list[ToolResult]:
    return await asyncio.gather(*(invoke(b) for b in blocks))

That line runs the coroutines concurrently, collects their results in the order the blocks arrived, and returns them. The ordering in that sentence is worth separating into its two halves, because the second half quietly borrows credibility from the first. Results come back in block order, which is a real and useful determinism. Execution order is not constrained at all, and the results being tidy afterwards says nothing about what happened while they were being produced. Beyond that the line is the same guarantee a thread pool gives you: the callables will be called. ThreadPoolExecutor.map has never claimed to make the mapped function thread-safe, and no one expects it to, but the same construct wearing the words "parallel tool calls" acquires an aura of having been thought about by somebody. Laid out against each other, the layers make plain how little of the safety question anyone is holding:

Read that table by the gap between its two columns rather than by how much authority each row appears to carry. Every entry in the left column is a statement about one call considered on its own: this block is well-formed, this handler was invoked, this row was updated. The property that would have saved the balance is a statement about a pair of calls — that the effect of running them together equals the effect of running them one after the other — and no row states it. The row at the bottom is the only one with real concurrency semantics in it, and it is the row nobody read.

It is worth noticing that nothing above is unusual or specific to agents. A web application that handles two concurrent requests to the same endpoint faces the identical problem, and backend engineers have a well-worn set of habits for it. That comparison is close enough to be useful and it breaks in one place, which is the place the habits stop transferring. In a web application the set of operations that might overlap is enumerable, because it is the set of routes, and a load test is written by picking pairs from that set. With an agent the concurrency is decided at inference time by a component with no view of your storage layer, so the set of pairs is not enumerable from your routing table. Two apply_credit calls on the same account in the same turn is a request pattern no load test would have generated, because no human client would ever send it.

The handler that made the arithmetic wrong#

async def apply_credit(account_id: str, amount_eur: float, reason: str) -> dict:
    account = await db.fetch_one(
        "SELECT credit_balance_eur FROM accounts WHERE id = $1", account_id
    )
    new_balance = account["credit_balance_eur"] + amount_eur
    await db.execute(
        "UPDATE accounts SET credit_balance_eur = $1 WHERE id = $2",
        new_balance, account_id,
    )
    await db.execute(
        "INSERT INTO credit_log (account_id, amount_eur, reason) VALUES ($1, $2, $3)",
        account_id, amount_eur, reason,
    )
    return {"applied_eur": amount_eur, "balance_eur": new_balance}

This function was correct for eighteen months, because for eighteen months it was only ever called once at a time from a single turn, and the two-worker deployment made a genuine collision on the same account unlikely enough never to happen. It reads a value, adds to it in the application process, and writes it back, with no transaction wrapping the read and the write and no condition on the update.

What that structure does to a pair of calls follows from three lines of algebra, and doing it in letters rather than in euros shows that the outcome does not depend on the amounts at all. Write the balance before the turn as b, the amount the first call is crediting as a, and the amount the second is crediting as c. The handler computes new_balance from the value it read, so what it writes is always (the value it read) + (its own amount).

Run them one after the other. The first call reads b and writes b + a. The second call reads what is now in the row, which is b + a, and writes b + a + c. Both credits survive, and they survive because the second read happened after the first write.

Now overlap them. Both calls read before either writes, so both read b. The first writes b + a. The second writes b + c, and it writes it over the top, because its UPDATE sets the column to a value it computed rather than adding to whatever is there. The final row holds b + c, and the credit a is gone. The general form of that is worth stating, because it is more alarming than the two-call case: however many calls overlap, the surviving total is b plus exactly one of the amounts, since every write carries a value derived from a read that predates every other write. Ten concurrent credits on a fresh account leave one credit's worth of money.

Each write, taken alone, is correct given what its own call read. That is the licence the code actually has, and its residual is the whole bug: correctness given a stale read is not correctness. Substituting the incident's figures, the account's balance before the turn was 0.00 — the first UPDATE in the query log below writes 40.00, which is that starting balance plus the overage credit — so b is zero, a is 40.00, c is 15.00, and b + c is 15.00, which is the number the account showed. The interleaving that produced it looks like this in the query log, with the timestamps as they were recorded:

14:02:11.402  toolu_01A  SELECT credit_balance_eur ...  -> 0.00
14:02:11.404  toolu_01B  SELECT credit_balance_eur ...  -> 0.00
14:02:11.419  toolu_01A  UPDATE accounts SET credit_balance_eur = 40.00
14:02:11.421  toolu_01B  UPDATE accounts SET credit_balance_eur = 15.00
14:02:11.427  toolu_01A  INSERT INTO credit_log (40.00, 'overage_dispute')
14:02:11.430  toolu_01B  INSERT INTO credit_log (15.00, 'seat_dispute')

The window between the two selects and the second update is seventeen milliseconds wide, measured from toolu_01B reading the stale value at 14:02:11.404 to toolu_01B overwriting the row at 14:02:11.421. That is why the sequential dispatcher never hit it and why the concurrent one hits it whenever both calls land on the same account: a sequential dispatcher does not start the second read until the first write has returned, so the window has no second occupant to catch.

Both handlers returned success and both told the truth as they knew it. The tool_result for toolu_01A said the balance was 40.00, which was accurate for as long as that value survived, and the log dates its survival from 14:02:11.419, where toolu_01A writes it, to 14:02:11.421, where toolu_01B replaces it — so it was already false by the time toolu_01A inserted its own log row at 14:02:11.427, and false before the result was returned at all. The fifteen milliseconds that matter here belong to the other call and to the earlier half of the window: toolu_01B read 0.00 at 14:02:11.404 and toolu_01A did not write 40.00 until 14:02:11.419, so toolu_01B spent that stretch holding a number that was about to stop being true and had no way to find out. The model then summarised for the customer that 55.00 EUR had been credited, because the sum of two successful results is the obvious reading, and no error was ever raised anywhere in the system.

There is a defence most teams believe they have here, and naming it is more useful than repeating that concurrency is hard. The belief is that a wrong answer of this kind would announce itself — that a run which loses money would show up as an exception, a failed call, or a missing row in the ledger. Check that belief against the log above. The number of successful tool_result blocks on the failing run is two, which is exactly what it is on a correct run. The number of rows in credit_log is two, which is also exactly what it is on a correct run. Both of those quantities are constant across the good case and the bad one, and the only thing that varies is the balance. Any monitor built on counting successes or counting log rows is watching a quantity that holds still while the dangerous one moves, which is the general shape of a defence that does not defend anything.

The credit log is the part that made this expensive rather than merely wrong. Two log rows exist for a balance that only reflects one of them, so the ledger and the account disagree, and the discrepancy is invisible until something reconciles them. The reconciliation job ran monthly.

Four different systems, four different answers#

The account balance is a database problem, and the database offers several ways to make it correct, which the next chapters get to. What matters here is that the same turn touched three other kinds of resource, and each one has its own answer to the question of what happens when two calls arrive together.

The invoice regeneration tool wrote a PDF to /tmp/invoice-render.pdf before uploading it, because a fixed path was simpler than threading a temporary name through the renderer. Two concurrent calls produce one file, and the upload that runs second may upload the other call's bytes under its own invoice number. The filesystem gave exactly what was asked of it and there is no error to catch, since both writes and both uploads succeed. The step that makes it silent is the same one as in the balance: a name that is fixed rather than derived from the caller turns two callers into one writer, and nothing in the write path can tell that it happened.

The receipt email is an external side effect with no undo. If the ledger write fails after the email has gone out, the correction has to be a second email, and there is no transaction that spans the SMTP handshake. This is the category that survives every fix applied inside your own process, and it is the reason the later chapters treat "no work silently dropped" and "no duplicate outbound action" as invariants to assert rather than properties to assume.

The partner tax API is the interesting one, because it does have concurrency semantics and they are documented: it accepts an Idempotency-Key header, treats two requests with the same key as one, and rejects concurrent requests carrying the same key with a 409 while the first is in flight. That is a closing identity, meaning a guarantee strong enough to settle a question rather than defer it, and it is worth reading in two halves. What it licenses is precise: any two of your calls that carry the same key produce at most one effect, whatever order they arrive in and however much they overlap, so a retry after a timeout is safe in a specific, checkable sense. What it does not cover is everything the key does not name. The guarantee is about duplicate submission of the same request, and two different requests — two different keys, two different amounts — racing to change the same underlying record on the partner's side are outside it entirely, and are governed by whatever the partner's storage layer happens to do. It is the only place in the whole turn where somebody actually answered the question, and even there the answer covers one of the two failures this chapter has shown.

Three fixes aimed at the dispatcher#

Running tool calls concurrently is worth doing, and the 3.4 seconds to 1.1 seconds is the reason the change shipped. Most tool pairs an agent emits are read-only or touch disjoint resources, and forcing them into a sequence buys nothing while making every multi-step turn feel slow. The problem is confined to the pairs where two calls reach the same mutable thing, which is a much smaller set than "all parallel calls" and a much larger set than "none of them", and the work is in knowing which pairs those are.

It is also not a failure of the model's reasoning. The plan was right, the amounts were right, and there is no prompt that would have produced a better decision, because the decision was not the problem. Blaming the model here leads to prompt engineering as a fix, which reduces the frequency of the emission and never removes the failure mode.

Most tempting of all, and worth naming because it is the fix people reach for first, is disable_parallel_tool_use. Setting it makes the model emit one tool_use block per turn, the two credits arrive in sequence, and this particular incident stops happening. What it does not touch is the fact that a second agent run, started by a different customer conversation forty milliseconds later and hitting the same account because a support engineer opened the same case, executes exactly the same interleaving against exactly the same row. That is the test to apply to any proposed fix in this area: name the property it establishes, then ask whether the property is about the turn or about the row. The window in the query log is a property of the handler, and the assistant turn is only one of several ways to arrive at it. Turning off parallel emission trades away the latency win and leaves the read-modify-write untouched, which is why chapter 3 goes after the handler rather than the dispatcher.

Concurrent dispatch, then, is a feature of the protocol and of your framework, and safety is a property of the things your handlers write to. The two are separate, they are supplied by different people, and only one of them is switched on by default. Everything above your handler has told you that your functions will be called; nothing above your handler has looked at what they do when two of them are inside the same row at once, and the balance that read 15.00 is what that gap costs when the amounts are small and the reconciliation is monthly.

There is one class of collision where the ordering is not incidental, and that is where the next chapter starts. When two calls in the same turn touch the same resource because the second one genuinely requires the first to have finished, no amount of atomicity in the handler helps, since the problem is not that the operations interleave but that one of them has no meaning until the other has returned. Nothing in a tool schema can express that requirement, and the dispatcher will start both regardless.