Argument deltas and text deltas

Streaming tool calls are what you have when a model that can invoke your functions delivers its turn incrementally, in pieces, as it writes, rather than as one finished response at the end. Topic 5.8, on the tool-calling loop, established the invoking half: the model replies with a structured request to run lookup_order with particular arguments, your code runs it, and the result goes back into the conversation for the model to answer from. Topic 2.9, on streaming a response, established the delivery half: the server keeps the HTTP connection open and pushes a sequence of small events down it — server-sent events, abbreviated SSE, a plain-text protocol in which each event is a labelled line carrying a JSON payload — so the first words reach the screen a few hundred milliseconds after the request rather than several seconds later. Switch both halves on at once and the tool call has to travel down that same pipe in those same small pieces, which means the arguments to lookup_order reach your code as a dozen or more fragments of a JSON document instead of as an object you can read.

The reason to accept that complication is the one streaming always offers, which is the user's sense of the wait. A support assistant that answers order questions needs about 4.2 seconds to produce a complete turn, look the order up and answer; streamed, its first words land at around 290 milliseconds, and a person reading from 290 milliseconds experiences a system that replied at once and then kept talking. Turning streaming off would make every problem in this topic disappear and would cost that difference on every turn, which is why almost every chat product streams and almost every chat product therefore has to handle a tool call arriving in fragments. You meet those fragments in two places at once. In the client code, a request carrying stream: true hands you an iterator of events instead of a response object, and some branch inside that loop has to decide what each event means. On the screen, a user is watching those same events land in real time, so a mistake in the branch is visible to them within a second.

What the stream says about a call#

A streamed turn does not hand you a tool call. It hands you an announcement that a call is starting, a run of fragments, and a marker that the call is over. Here is a real Anthropic-style turn, with the SSE framing collapsed so the payloads are readable:

content_block_start   {"index":0,"content_block":{"type":"text","text":""}}
content_block_delta   {"index":0,"delta":{"type":"text_delta","text":"Let me pull"}}
content_block_delta   {"index":0,"delta":{"type":"text_delta","text":" that up for you."}}
content_block_stop    {"index":0}
content_block_start   {"index":1,"content_block":{"type":"tool_use",
                       "id":"toolu_01QxR","name":"lookup_order","input":{}}}
content_block_delta   {"index":1,"delta":{"type":"input_json_delta",
                       "partial_json":"{\"order_id\": \"A-44"}}
content_block_delta   {"index":1,"delta":{"type":"input_json_delta",
                       "partial_json":"71\", \"include_line_ite"}}
content_block_delta   {"index":1,"delta":{"type":"input_json_delta",
                       "partial_json":"ms\": true}"}}
content_block_stop    {"index":1}
message_delta         {"delta":{"stop_reason":"tool_use"}}

Three things in that trace matter. The content_block_start carries the call's identity — the toolu_01QxR id and the tool name — and the input field is an empty object, because nothing of the arguments has been sent yet. The deltas carry partial_json, which is a string, not an object: the provider has cut a JSON document into pieces at arbitrary byte offsets and is shipping the pieces. The content_block_stop is the only statement in the stream that the arguments are finished.

It is worth knowing why the splits land where they do, because every rule in this chapter follows from it. The model produces tokens, one at a time, and a token is a chunk of characters the model's vocabulary happens to contain. The server flushes whatever tokens have accumulated since the last flush into the next SSE frame. Neither of those steps consults the JSON structure being written: the tokeniser was built over text in general, not over this document, and the flush is a decision about network buffering. So a boundary falls between A-44 and 71 for the same reason it falls anywhere else, which is that a frame had to end somewhere and that is where the writing had got to. There is no sense in which the provider chose to break the order id in half. It never saw an order id.

That is also why the terminal marker has to exist as its own event rather than being inferred. If the splits were structural you could watch for a balanced closing brace and be finished. They are not, so the only thing that knows the arguments are complete is the process that finished writing them, and the only way it can tell you is to say so.

OpenAI's Responses API names the same three things differently — response.output_item.added for the announcement, response.function_call_arguments.delta for the fragments, and response.function_call_arguments.done for the marker, which conveniently repeats the full argument string — and its Chat Completions endpoint expresses all of it through a tool_calls array inside the delta, where the terminal signal is finish_reason: "tool_calls" on the choice. The event names differ. The shape does not, and neither does the constraint underneath it.

Two properties a text delta does not have#

The reason one renderer cannot serve both is that a text delta and an argument delta answer different questions.

"The order shipped on the" is a true, correct, renderable prefix of "The order shipped on the 3rd of March." Every prefix of a text stream is a valid smaller version of the same thing, which is why streaming prose works at all: the reader is looking at the answer, just less of it. {"order_id": "A-44 is not a smaller version of the arguments. It is not a JSON document, it does not describe an order, and there is no operation you can perform on it that yields a fact about the world. It becomes meaningful in one step, when the last fragment lands and the call reports itself finished, and until that moment it is a byte range.

The second property is ownership. Text deltas belong to the message, and the message is one thing, so the accumulator can be a single string. Argument deltas belong to a call, and a turn can contain several — topic 5.8 established that a model can emit four calls in one assistant turn — so the accumulator has to know which call each fragment is for. The trace above gives you that in index, and the content_block_start binds that index to the toolu_01QxR id you will need later when you return the result. Chapter 2, on the key that accumulator should use, is about what happens to code that ignores this.

Concatenate, and do nothing else#

The accumulator's contract is narrower than it looks, and the narrowness is the point. Fragments belonging to one call arrive in order — that is the one assumption the stream actually supports — and your job is to append them as opaque strings in exactly that order, parsing nothing, decoding nothing, trimming nothing, until the call reports itself finished.

type CallBuffer = { id: string; name: string; args: string };

const buffers = new Map<number, CallBuffer>();

function onBlockStart(index: number, block: { id: string; name: string }) {
  buffers.set(index, { id: block.id, name: block.name, args: "" });
}

function onArgumentDelta(index: number, fragment: string) {
  const buf = buffers.get(index);
  if (!buf) throw new Error(`argument delta for unannounced call ${index}`);
  buf.args += fragment; // no trim, no parse, no decode
}

Each of the three prohibitions in that comment corresponds to a bug that has shipped. Parsing per delta is tempting because you want to know what the call is going to say, and chapter 4, on dispatching before the model has finished, deals with why acting on that knowledge is unsafe. Decoding per delta breaks because the provider splits at byte offsets that have nothing to do with JSON structure: a split can land between the backslash and the n of an escape sequence, or between the \u00 and the e9 of an escaped character, and a decoder handed either half will either fail or silently produce the wrong character. Trimming is the one that surprises people, because whitespace in JSON is usually insignificant and stripping fragments looks like hygiene. Consider a call carrying a description:

fragment 7:  {"summary": "urgent request
fragment 8:   from ops about the March
fragment 9:   invoice run"}

Concatenated raw, that is {"summary": "urgent request from ops about the March invoice run"}. Concatenated after .strip() on each fragment, it is {"summary": "urgent requestfrom ops about theMarch invoice run"}, which parses cleanly, validates cleanly, dispatches cleanly, and creates a ticket with a mangled summary that no error anywhere in your system will ever mention. The leading space was inside a string literal, where whitespace is data.

Work through where each missing space went and the rule generalises past this one bug. Fragment eight opens with a space and fragment nine opens with a space, because the split happened to fall just before a word rather than just after one. Stripping removes the character at the join, and the join is inside the quoted value, which is the one place in a JSON document where whitespace is not formatting. The reason no error follows is that the damage respects the grammar: removing a space inside a string literal leaves a shorter string literal, and a shorter string literal is a perfectly valid document. Every check downstream — parser, schema validator, type coercion — is asking whether the document is well formed, and it still is. The only component that could notice is a human reading the ticket.

This is not a statement about JSON being hard to parse. Incremental JSON parsers exist and are good — streamjson is a compact reference implementation for exactly this stream shape — and chapter 5, on displaying a call as it forms, uses one for a real purpose. The constraint is about what the fragment is, not about how hard it is to read: it belongs to a call that has not declared itself finished, so any interpretation of it is provisional, including interpretations that happen to be syntactically valid. A buffer holding {"limit": 20} looks like complete arguments right up until the next fragment turns it into {"limit": 20, "status": "refunded"}.

That example is the one to keep, because it disposes of the defence most people reach for first. The instinct is that a parse in a try-catch is a safe completion test: attempt to read the buffer as JSON on every delta, treat failure as "not finished yet", and treat success as "finished". It is a tempting design because it needs no new events, works with any provider, and is right on most fragments — a buffer cut mid-key really does fail to parse. The reason it is wrong is that the test has no false-positive protection. A prefix of a JSON document can itself be a complete JSON document, and when it is, the parse succeeds and reports finished on arguments that are missing a field. The failure only occurs when a split lands on a closing brace, which is rare, so the design passes review, passes every fixture anyone thinks to write, and produces a wrong dispatch occasionally and forever. Syntactic validity is a statement about the bytes you have. Completion is a statement about the bytes still coming, and no examination of the former can establish the latter.

One layer below the fragments there is a second concatenation with the same rule. SSE frames arrive as bytes, and a chunk boundary can land in the middle of a multi-byte UTF-8 sequence — the two bytes of é, the four of an emoji — so a client that calls bytes.decode("utf-8") on each socket read raises UnicodeDecodeError on a stream that is perfectly well formed. Python's codecs.getincrementaldecoder("utf-8")() and Node's StringDecoder exist for this, and every mainstream SDK uses one internally, which is why the problem stays invisible until someone writes their own transport for a proxy, a gateway or a replay harness. Hold the bytes you cannot yet interpret, and interpret when the next piece arrives.

The two layers rhyme closely enough that it is worth naming the property they share, because recognising it saves you from meeting the same bug a third time. In both cases a producer has a unit of meaning — a character, a JSON document — and a transport that is allowed to cut anywhere. A consumer that assumes the cut respects the unit will be right most of the time, since most cuts do fall on boundaries by luck, and wrong at a rate set by how often the unit is longer than one chunk. The correct handling is identical at both layers: hold what you have, add the next piece, and interpret only when something outside the byte stream tells you the unit is complete. At the character layer that something is the decoder's own state machine. At the argument layer it is content_block_stop.

What the message surface does to a fragment#

Appending an argument fragment to the message is a lossy mistake as well as a visible one, because a chat bubble is rarely a plain-text sink. It runs its accumulated string through a markdown renderer and a sanitiser before anything reaches the screen, and argument objects are full of characters markdown treats as syntax. A note field reading see *both* attachments renders as both in italics with the asterisks gone. An odd number of backticks in a description opens a code span that swallows the rest of the bubble. A value beginning with # becomes a heading, a value beginning with > becomes a blockquote, and a lone \ at a fragment boundary is markdown's own escape character, so it disappears against whatever follows it.

The step that makes this a data-loss bug rather than a cosmetic one is that the message surface is not a display of the buffer. It is a transformation applied to the buffer, and the transformation is not reversible. Markdown consumes the characters it treats as syntax: once *both* has become italic text, the asterisks are not stored anywhere that a later reader can recover them from. So if the message string is the thing you persist, what you have written to the database is the output of the renderer rather than what the provider sent, and no amount of care later in the pipeline can reconstruct the difference.

A support assistant that made this mistake produced the specimen below, captured from a customer's screen while a lookup_order call was in flight:

Let me pull that up for you.{"order_id": "A-4471", "include_line_ite

The sentence is correct. The fragment after it is the first two thirds of the arguments, rendered straight into the message, and it stops mid-key because the third delta had not arrived when the screenshot was taken. A second later the bubble ended with ms": true} appended and no space, and then the real answer streamed in underneath.

None of that is recoverable from the message state, which matters whenever the message state is what you persist or what a copy-to-clipboard button reads. The visible half of the failure is the bubble. The invisible half is that the stored message for that turn contained a mangled copy of the arguments, two asterisks short, in a database row that a later replay would read back as the model's own words. Routing argument fragments away from the message surface keeps the buffer byte-identical to what the provider sent, which every later chapter depends on: chapter 4 compares a prefix against the finished document, chapter 6, on finalising a call, parses the buffer strictly, and neither works on bytes that a renderer has been through.

Fixtures that cannot fail#

Fragment counts scale roughly with argument length, and the interesting behaviour only appears once there are enough of them. In a sample of production turns, a search_orders call with a 24-character argument object arrived as 2 deltas; a create_ticket call with a 610-character object, most of it a description field, arrived as 47 deltas averaging 13 characters each. The short call cannot exercise a split inside a string, inside an escape, or between the two halves of a nested object, because there are no splits. The long one exercises all three by accident.

The relationship between those two observations is a division, and doing it once tells you how long a fixture has to be. Dividing the long call's argument length by its delta count gives the 13-character average stated above, and dividing the short call's length by its delta count gives very nearly the same figure. That is the useful finding: fragment size is roughly constant, set by how much the model produced between two flushes, and it does not care how long the document is going to be. Delta count is therefore argument length divided by that constant, which is what "scales roughly with argument length" means. A fixture's number of splits is something you control by choosing the length of its arguments, and a fixture with two deltas has exactly one interior boundary to land anywhere.

One thing to be careful of when counting characters against a recorded stream: the lengths on the wire are of the escaped form, the one visible in the trace earlier in this chapter, where every quote inside partial_json carries a backslash in front of it. An argument object counted from your logs after parsing is shorter than the same object counted as the provider shipped it, and it is the shipped form that the splits fall inside.

A fixture set that would have caught the screenshot needs five things a single-short-call fixture does not have. A turn with two calls in it, so the buffers have to be distinguished. A call whose arguments are long enough to fragment — 400 characters is plenty — so splits land in awkward places. A value containing an escape sequence and a non-ASCII character, with at least one fixture that splits inside the escape. A turn with prose before the call and more prose after the result, which is chapter 3's subject. And a stream that stops mid-arguments with no completion event, which is chapter 6's. Recording one real turn against a real provider and replaying its raw events gives you the first three almost free, and hand-editing that recording gives you the last two.

Recording it is one request with the raw bytes kept:

curl -sN https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d @turn.json | tee fixtures/two-calls-long-args.sse

The turn.json body carries "stream": true, the tool schemas, and a user message chosen to provoke two calls with long arguments. The -N flag disables curl's output buffering, so the frames land in the file at the sizes the server sent them rather than in 16KB blocks, which preserves exactly the split positions you are trying to test against. What you keep is the SSE text rather than a parsed structure, because the parsing is the code under test. A replay harness then feeds those frames to the same handler the production client uses, at whatever timing you choose: all at once for a fast unit test, and with the recorded inter-frame delays when you are exercising the display path.

An argument delta is a byte range belonging to a call that has not finished speaking, and every rule worth having about it follows from those two facts. It is owned, so it needs a key. It is provisional, so it cannot be parsed, decoded or trimmed. It is not prose, so it cannot go anywhere prose goes. The handler that gets this right is shorter than the one that gets it wrong, because the correct behaviour is to do less.

Which raises the question of what that key should be — and the obvious choice, the tool's name, is wrong in a way that takes months to surface.