Keying the argument accumulator

An accumulator is the small piece of state that turns a run of argument fragments back into the JSON document the model wrote: a map from something that identifies a call to the string collected for it so far. Its whole job is bookkeeping, which is why the choice of key looks like a detail and is not. Get it wrong and the concatenation still runs, the parse still succeeds most of the time, and the failure arrives months later as a call that was announced by the model and never reached your dispatcher.

The tempting key is the tool's name. It arrives in the announcement, it reads well in a log line, and it is unique in almost every turn you will see during development. The key the stream actually offers is a positional index that appears on every fragment, alongside a stable identifier that appears once. This chapter is about the gap between those two choices, which is exactly one turn wide: a turn containing two calls to the same tool.

Where the two documents go#

Suppose a user asks a support assistant to show their refunded orders from March and their open ones. The model does the sensible thing and emits two calls to search_orders, one filtered on refunded, one on open. The stream announces both calls and ships both sets of fragments:

content_block_start  index=1  tool_use  id=toolu_01Ka  name=search_orders
  input_json_delta   index=1  {"status": "refu
  input_json_delta   index=1  nded", "from": "2025-03-01", "to": "2025-03-31"}
content_block_stop   index=1
content_block_start  index=2  tool_use  id=toolu_01Mb  name=search_orders
  input_json_delta   index=2  {"status": "op
  input_json_delta   index=2  en", "from": "2025-03-01", "to": "2025-03-31"}
content_block_stop   index=2

A dictionary keyed by name has one entry, search_orders, and every one of those four fragments lands in it. The completion event for the first call fires while the second call's fragments are still to come, so the parse either happens too early against a buffer that is about to grow, or happens at the end against a buffer holding both documents end to end:

{"status": "refunded", "from": "2025-03-01", "to": "2025-03-31"}{"status": "open", "from": "2025-03-01", "to": "2025-03-31"}

What that string does next depends entirely on which parser you reached for, and the three common choices fail three different ways. json.loads in Python raises JSONDecodeError: Extra data: line 1 column 65, which is the good outcome, because it is loud. JSONDecoder().raw_decode, which people use precisely because it tolerates trailing content, returns the first object and the offset where it stopped, so it succeeds and silently discards the second call. A repair-style tolerant parser — one that closes any unterminated string and any open brace so that an incomplete fragment still yields an object, of the kind chapter 5 will use for display — may merge the two objects into one, and because both documents share the keys status, from and to, the result is one call whose status is whichever document won the merge. That last case is the one that ends up in a postmortem, because the arguments look completely normal.

The column number in that first error is worth checking against the string, because it tells you precisely what the parser saw. Count the characters of the first document, from its opening brace to its closing one, and there are sixty-four of them. Column 65 is therefore the very next character, which is the opening brace of the second document. json.loads is not confused about the first object at all — it read it successfully — and the complaint is only that the input did not end where the object did. That is the difference between the three parsers in one sentence: all three read the same first object, and they disagree about what to do with the bytes that follow it. One refuses, one returns the object and the offset, and one keeps going.

The merge case deserves the extra step, because it is the one where the choice of tool made the failure invisible. A repair parser inserts whatever closing tokens are needed to make its input a single object, so it treats the second document's contents as further members of the first. Both documents have the same three keys, since they are calls to the same tool. A JSON object cannot hold a key twice, so the later occurrence overwrites the earlier one, and the surviving object has the second call's status with everything else identical. The result is not corrupt, not partial, and not detectably wrong: it is exactly the arguments of one of the two calls the model asked for. Schema validation passes it, because it satisfies the schema. It is the call the model never made that no component in the system has a record of.

Providers do not all emit blocks strictly one after another. Anthropic's content blocks are sequential, so the common failure is the back-to-back concatenation above. OpenAI's Chat Completions endpoint delivers tool calls as an array inside each delta, and a single chunk can carry entries for more than one array position, so fragments belonging to different calls genuinely arrive interleaved:

{"choices":[{"delta":{"tool_calls":[
  {"index":0,"id":"call_8Hs1","type":"function",
   "function":{"name":"search_orders","arguments":"{\"status\": \"refu"}},
  {"index":1,"id":"call_9Jt2","type":"function",
   "function":{"name":"search_orders","arguments":"{\"status\": \"op"}}
]},"finish_reason":null}]}

A name-keyed handler walks that array in order and appends both entries to the same string, so after this single chunk the buffer holds {"status": "refu{"status": "op, and the next chunk appends the two continuations the same way. The end state is not two documents you could split apart at a brace but two documents shuffled together at fragment granularity, which no parser recovers and no repair library improves. Its one useful property is that it fails loudly.

The key the stream already gave you#

Every streaming API for tool calls identifies calls twice: by a positional key that appears on every fragment, and by a stable identifier that appears once, in the announcement. You buffer on the positional key because it is the only thing the fragments carry, and you keep the identifier alongside because it is what you must return the result against, in the shape topic 5.8 established for the tool-calling loop.

Two identifiers for one call looks like redundancy and is not, because they are solving different problems and neither can do the other's job. The positional key has to be small, because it rides on every fragment and a stream carries a great many fragments; it is a counter within one response, which is all the information a fragment needs to say which buffer it belongs in. The stable identifier has to be long enough to be unique across every call the provider has ever issued, because it is what a tool_result will name in a later request, possibly minutes later, after your process has restarted. Sending that long identifier on every fragment would be waste, and sending only the short one would leave you unable to answer the call. So the announcement carries both, once, and binds them — which is why the announcement is the event you cannot afford to miss, and why the code above records it rather than merely noting that a call began. In traces printed in this topic the stable identifiers are abbreviated for readability; the real ones are longer and are opaque strings you never construct yourself.

Two details in that table are worth reading twice. The tool name arrives once, in the announcement, and never again — so if you key by name you are keying by a value you had to look up anyway, when a key that is unique per call was sitting in the same event. And OpenAI's Chat Completions endpoint has no per-call terminal event at all: finish_reason fires once for the whole choice, which means every call in the turn completes at the same moment and a per-call state machine has to synthesise the boundary. Responses and Anthropic both give you a real per-call marker, which is one concrete reason to prefer them for anything that dispatches during a turn.

Here is the accumulator with the key corrected, and with the announcement recorded so the id survives:

from dataclasses import dataclass, field

@dataclass
class CallBuffer:
    call_id: str
    name: str
    args: str = ""

class Accumulator:
    def __init__(self) -> None:
        self.buffers: dict[int, CallBuffer] = {}
        self.completed: list[CallBuffer] = []

    def on_block_start(self, index: int, call_id: str, name: str) -> None:
        if index in self.buffers:
            raise ProtocolError(f"duplicate announcement for index {index}")
        self.buffers[index] = CallBuffer(call_id=call_id, name=name)

    def on_argument_delta(self, index: int, fragment: str) -> None:
        buf = self.buffers.get(index)
        if buf is None:
            raise ProtocolError(f"fragment for unannounced index {index}")
        buf.args += fragment

    def on_block_stop(self, index: int) -> None:
        buf = self.buffers.pop(index)
        self.completed.append(buf)

The two raise statements are not defensive padding. A fragment arriving for an index you never saw announced means your event routing has a hole in it — most often that you subscribed to argument deltas before you subscribed to block starts, or that a reconnect replayed part of a stream. A duplicate announcement for a live index means the same. Both are conditions where the correct behaviour is to fail the turn rather than to guess which call the bytes belong to, and both are silent under a defaultdict.

The defaultdict point is the small one that decides the size of the outage. A missing key is the only evidence you will ever get that a fragment has arrived for a call you are not tracking. A container that manufactures an empty buffer on first access converts that evidence into an ordinary successful append, and the turn proceeds to parse a document assembled from a subset of its fragments. The convenience that reads as removing a special case is removing the detection. This is the general form of every silent failure in this topic: something that should have been impossible was given a plausible interpretation instead of an error.

One accumulator per stream#

The key is unique only within the stream that issued it. index restarts at 0 for every response, and so does OpenAI's tool_calls[].index, which makes an accumulator held at module scope — or on a long-lived client object, or in a cache keyed by user — the same collision with a wider blast radius. Two concurrent conversations both announce a call at index 1, the fragments of one land in the buffer of the other, and the resulting document usually fails to parse. Occasionally, when both calls are to the same tool with the same argument shape, it does not, and one user's query executes with another user's filter.

The same reasoning applies inside a single agent loop. Topic 5.8's loop runs several turns to answer one question, and every turn is a fresh response with its own indices, so an accumulator that lives for the loop rather than for the stream finds index 1 already occupied on turn two and trips the duplicate-announcement guard above, which is the good outcome and only happens because the guard is there. Constructing the accumulator where you construct the stream keeps the lifetime right:

async def run_turn(client, messages, tools) -> list[CallBuffer]:
    acc = Accumulator()                      # per stream, never shared
    async with client.messages.stream(messages=messages, tools=tools) as stream:
        async for event in stream:
            route(event, acc)
    return acc.completed

The lifetime rule reads as obvious written down and is easy to lose in a refactor, because an accumulator looks like a utility and utilities get hoisted to module scope or wrapped in a singleton for reuse. What holds the line cheaply is giving the accumulator no constructor argument it could sensibly share, and asserting when the stream ends that acc.buffers is empty — a non-empty map at that point is either an abandoned call, which chapter 6 handles, or a leak from a stream that is not yours.

Opaque, ordered, owned#

Ordered means you may only concatenate, never sort, never reassemble from a set, and never process argument deltas on a different task than the one preserving stream order — an asyncio.gather over per-event handlers will happily reorder two fragments that arrived 4ms apart and produce {"status": "refnuded"}. Owned means the key comes from the event, not from any property of the call's content. Opaque means the only operation is append.

The reason those rules are not negotiable is that the provider splits the document at byte offsets chosen by the tokenizer, and those offsets land wherever they land:

inside a quoted string      ...{"note": "shipped on the 3   |   rd of March"}...
inside an escape sequence   ...{"note": "line one\          |   nline two"}...
inside a unicode escape     ...{"city": "M\u00             |   e9rida"}...
inside a nested object      ...{"filter": {"status": "op    |   en"}}...
between key and colon       ...{"limit"                    |   : 20}...

Not one of those halves is valid JSON, and three of them are actively misleading: {"note": "line one\ ends in a dangling escape that a repair parser will resolve as a literal backslash, and "M\u00 will decode as the four characters M, \, u, 0, 0 under any decoder tolerant enough to accept it. Concatenate first, in arrival order, and parse exactly once when the call reports itself finished, and every one of those splits becomes invisible, because the document you parse is the document the model wrote.

Why a name key survives seven months#

The name key is tempting because it reads well and because it is almost always unique. Most turns contain one call. Turns with several calls usually call several different tools — get_order and get_customer, the parallel case from topic 5.8 — and those keep separate entries under a name key, so parallel dispatch works and the code looks proven. Two calls to the same tool in one turn is a narrower event: on a support assistant with eleven tools it showed up in roughly 0.4% of tool-using turns, which is once every few hundred conversations, which is a rate that will not appear in a fixture suite and will appear in production within a day of launch.

Turning that percentage into an interval is the step that makes it alarming rather than reassuring. Four tenths of a per cent is four in a thousand, so the reciprocal is one in two hundred and fifty: on average, one tool-using turn in that many contains the shape that breaks a name-keyed buffer. A fixture suite containing a few dozen recorded turns has a good chance of containing none, which is why the code looks proven. An assistant handling a few thousand turns a day meets it many times before lunch. The same number reads as "rare" and as "constant" depending only on how many draws you take, and the test suite and production take wildly different numbers of draws.

There is a further reason the rate understates the exposure. It was measured on a toolset of eleven tools, and the probability that a turn's several calls collide on one tool depends on how the traffic is distributed across those tools. Two calls to the same tool become likelier as a toolset narrows, as one tool becomes the workhorse, or as a schema stops being able to express a compound request in a single call — which is exactly the property exploited a paragraph below to reproduce the bug on demand. The figure is a measurement of one assistant's traffic, not a constant of the protocol.

Eliciting it deliberately takes one prompt and one property of the schema. A tool whose status parameter is a single enum value rather than an array leaves the model no way to express "refunded and open" in one call, so a request naming two statuses produces two calls with high reliability — on the support assistant that prompt reproduced the double call in 19 of 20 attempts at temperature 1. Both major providers also expose the switch that turns the behaviour off, OpenAI's parallel_tool_calls and Anthropic's tool_choice.disable_parallel_tool_use, and the first of those defaults to true. Turning it off is a reasonable stopgap while you fix the key and a poor permanent setting, because it costs a round trip for every genuinely parallel pair — two sequential turns at roughly 1.4 seconds each instead of one — and it does nothing about the reconnect and replay paths that can deliver the same index twice.

That last clause is the one to weigh, because the switch is otherwise a very attractive fix: one line, no refactor, and the failure genuinely stops. What it does is remove one of the ways two buffers can be live at once. It does not give the accumulator a correct key. A stream that drops and is resumed can replay events you have already seen, and a replay harness or a proxy can deliver a turn's events twice by construction; in both cases a name-keyed buffer appends the repeated fragments to the string it already holds, and the result is a document with a duplicated tail. Disabling parallel calls is a change to what the model emits. The bug is in what your code does with what arrives, and those are different layers.

There is a second reason the mistake persists, which is that the failure does not look like an assembly bug. The dispatcher receives arguments that parse, validate against the schema, and describe a real query. The user sees an answer to half their question and usually rephrases rather than complaining. The logs show one search_orders call, exactly as expected, because the second one was never constructed. Nothing in the system records the fact that a call was announced and then dropped — unless the accumulator counts announcements against completions, which is a cheap invariant worth asserting: the number of buffers created in a turn must equal the number of completed calls plus the number explicitly failed.

The whole of the accumulator's correctness is in three words. Fragments are owned, so they need a key that comes from the event rather than from their content. They are ordered, so they may only be appended. They are opaque, so nothing may look inside them before the call is finished. A buffer written to those three rules is a dozen lines long and is correct for any number of calls in a turn, on any provider in the table above.

What it does not yet handle is that those calls are not the only thing in the turn: the same stream is carrying prose that the user is reading in real time, and the ordering between the sentence and the call is information the interface has to keep.