Streaming response framing

Streaming means asking the provider to send the response as it is generated rather than as one document at the end. Both major providers do it over Server-Sent Events, usually written SSE: a plain-text HTTP response with content-type: text/event-stream, delivered as blocks of field: value lines separated by blank lines. That much is shared, and it is where the sharing stops. The vocabulary of events inside the stream, the path to the generated text, the marker that ends it and the location of the token counts are all chosen independently by each vendor.

Why there is anything to stream is worth one paragraph, because the rest of the chapter's numbers only mean something once it is clear. A model produces its reply one token at a time, each one conditioned on everything before it, so the last token of a long answer is not available until the generation has run to the end. A non-streaming response is assembled from all of those tokens and sent as one document, which means the first byte the caller sees leaves the provider after the last token is produced. Time to first byte and total generation time are the same number, by construction. Streaming changes only when the bytes are handed over: each piece is forwarded as it is produced, so the first characters arrive after the first token rather than after the last. Nothing about the generation is faster. The waiting has been moved to somewhere the user is already reading.

The consequence is that a streaming handler is not portable in the way a non-streaming one nearly is. A function that reads content[0].text from one provider and choices[0].message.content from another is a four-line rename; a handler written against one event framing and pointed at the other recognises nothing at all, and fails in two directions at once — no text arrives incrementally, and the usage numbers it records are whatever it initialised them to. This chapter sets the two framings side by side, gives a parser that reads either, and covers the case that breaks in production rather than in testing: tool calls, whose arguments arrive as a JSON string assembled across many events.

Two framings of the same stream#

The first framing gives every event a name and models the response as a document being assembled. A message opens, content blocks open and close inside it, each block accumulates deltas, and a terminal event carries the stop reason and the output token count.

event: message_start
data: {"type":"message_start","message":{"id":"msg_01CQ8k","model":"claude-sonnet-4-5",
       "content":[],"stop_reason":null,"usage":{"input_tokens":612,"output_tokens":1}}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"The invoice"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" for order"}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},
       "usage":{"output_tokens":87}}

event: message_stop
data: {"type":"message_stop"}

The second framing emits a sequence of chat-completion chunks, each one a partial copy of the non-streaming response object, with the new text in choices[0].delta.content. Nothing carries an event name; every block is a bare data: line. The end is marked by a sentinel line that is not JSON at all.

data: {"id":"chatcmpl-9xQ2","object":"chat.completion.chunk","choices":[
       {"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-9xQ2","object":"chat.completion.chunk","choices":[
       {"index":0,"delta":{"content":"The invoice"},"finish_reason":null}]}

data: {"id":"chatcmpl-9xQ2","object":"chat.completion.chunk","choices":[
       {"index":0,"delta":{"content":" for order"},"finish_reason":null}]}

data: {"id":"chatcmpl-9xQ2","object":"chat.completion.chunk","choices":[
       {"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-9xQ2","object":"chat.completion.chunk","choices":[],
       "usage":{"prompt_tokens":634,"completion_tokens":91,"total_tokens":725}}

data: [DONE]

Where the usage numbers live#

Two rows in that table deserve more than a row each, because they are the ones that silently zero a cost dashboard.

On the chunk framing, usage is not emitted by default at all. The final chunk carrying it appears only if the request asked for it with stream_options: {"include_usage": true}. Without that flag there is no usage anywhere in the stream, and code that initialises a counter to zero and never finds a value to overwrite it reports exactly what it was initialised to.

The corresponding trap on the typed framing is different in shape. Usage is split across two events, with input_tokens on message_start and output_tokens on message_delta, so a handler that reads usage from the first event it sees records the input count and an output count of 1 — which is the placeholder the opening event carries, not a measurement. Neither of these produces an error, and both produce a number, which is why they survive review.

Both traps come from one fact about when the numbers exist, and holding that fact makes them predictable on a provider this chapter has not covered. The input count is known before generation starts: the input is what you sent, and it has been read and counted before the first output token is produced. The output count is a tally of tokens that have not been produced yet, so it cannot be reported at the start of a stream by anybody, however the framing is designed. Every provider therefore has to solve the same problem, and there are only two solutions available: split the usage across the stream and send the output half at the end, which is the first framing; or send one usage record at the end and make it something the caller opts into, which is the second. Once you know that, the question to ask of any streaming API is not "does it report usage" but "where does it put the half that cannot be known yet".

Parsing either framing#

Server-Sent Events parsing is short enough to own: accumulate data: lines, remember an event: name if one is present, flush on a blank line, and treat the sentinel as a terminator rather than as JSON.

import httpx, json

def stream_chunks(url, headers, body):
    """Parse either framing: yield (event_name_or_None, parsed_json)."""
    with httpx.stream("POST", url, headers=headers, json=body, timeout=None) as r:
        r.raise_for_status()
        name, data = None, []
        for line in r.iter_lines():
            if line == "":                       # blank line ends a block
                if data:
                    payload = "\n".join(data)
                    if payload != "[DONE]":
                        yield name, json.loads(payload)
                name, data = None, []
            elif line.startswith(":"):           # comment / keep-alive
                continue
            elif line.startswith("event:"):
                name = line[6:].strip()
            elif line.startswith("data:"):
                data.append(line[5:].lstrip())

Everything provider-specific belongs above that function, in a small piece of code per framing that pulls text out of a payload and updates a usage record. Writing the split that way is what makes a provider's framing a forty-line adapter instead of a condition scattered through your handler.

The framings also differ in how much state a consumer has to keep. The typed-event framing tells you explicitly when a content block opens and closes, so a handler can maintain a list indexed by block and know when each one is complete. The chunk framing has no such markers; a chunk is a partial copy of the response object, and the consumer's job is to merge each delta into an accumulator keyed by choice index, inferring structure from which fields happen to be present. Neither is harder than the other once written, and both are easy to write incorrectly by assuming the first shape you saw is the general one.

Neither framing was invented for streaming, and noticing that makes both of them predictable rather than arbitrary. Each is the provider's own non-streaming response shape, projected onto a sequence. A response modelled as a list of several completions streams as partial copies of that list, with an index selecting which completion a fragment belongs to — so a consumer merges fragments into the object it would otherwise have received whole. A response modelled as one message containing a list of typed content blocks streams as announcements about those blocks: this one is opening, this one is text, here is more of it, this one is closed. The differences in the table above are not eight independent decisions. They are one decision, made before streaming existed, and read out eight times.

Tool calls arrive in fragments too#

Streaming text is the case everyone tests. The case that breaks in production is a tool call — the model asking your code to run a function it declared — because both framings send the arguments as a JSON string assembled across many events, and a partial JSON string does not parse.

On the typed framing the deltas carry a distinct type, so a handler that switches on delta.type sees input_json_delta instead of text_delta and can route the fragments into a per-block buffer:

event: content_block_start
data: {"type":"content_block_start","index":1,
       "content_block":{"type":"tool_use","id":"toolu_01A","name":"lookup_invoice","input":{}}}

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"{\"order"}}

event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"_id\": 88213}"}}

On the chunk framing the fragments arrive under choices[0].delta.tool_calls, as a list whose entries carry an index telling you which call they belong to, with the function name usually present only on the first fragment and the argument string split across the rest. The accumulation rule is therefore different in a way no amount of renaming fixes: one framing gives you a block index and a delta type, the other gives you a nested array index and requires you to remember the name you saw earlier.

tools = {}   # index -> {"name": str, "args": str}
for _, chunk in stream_chunks(url, headers, body):
    for tc in chunk.get("choices", [{}])[0].get("delta", {}).get("tool_calls", []):
        slot = tools.setdefault(tc["index"], {"name": "", "args": ""})
        slot["name"] += tc.get("function", {}).get("name", "") or ""
        slot["args"] += tc.get("function", {}).get("arguments", "") or ""

Only once the terminal marker has arrived is slot["args"] a complete JSON document. Code that tries to parse it early gets a JSONDecodeError on a string that is perfectly valid three events later, and the usual bad fix — wrapping the parse in a try/except and ignoring failures — turns a timing bug into a tool call that silently never fires.

What a stream does and does not buy you#

A stream is a delivery mechanism and nothing more. It does not reduce cost — you are billed for the same input and output tokens whether or not you asked for them incrementally — and it does not reduce total generation time, which is why the migration above showed no change in end-to-end latency. What it buys is time to first token, and it buys it only if every hop between the model and the user is willing to forward bytes as they arrive. A reverse proxy with response buffering enabled will hold the entire event stream and deliver it in one piece, producing exactly the symptom of a broken parser with a parser that is entirely correct, which is why the first check on a stream that arrives all at once is whether it arrives all at once at the edge of your own infrastructure too.

The chunks are also not tokens, despite the arithmetic frequently working out close enough to encourage the belief. A single delta may carry several tokens or a fragment of one, the boundaries depend on batching inside the provider's serving stack, and counting deltas is a bad estimator for output_tokens. The count on the terminal event is the billed number, and it is the only one worth logging.

The reason to distrust the estimator even when it looks accurate is that its error is not noise. A delta boundary is decided by when the serving stack flushed, and a serving stack flushes according to how loaded it is, how many requests it is batching together, and what the network between you and it is doing. So the ratio of deltas to tokens is a function of the provider's conditions, not of your text — which means it will be stable and close to right during the week you validate it, and quietly wrong on the day the provider is busy. An estimator whose error correlates with load is worse than no estimator, because it is most wrong exactly when somebody is reading the dashboard.

Timeouts need rethinking under streaming as well. A single read timeout applied to a non-streaming call is a sensible bound on the whole exchange, but on a stream the same setting bounds the gap between events instead, and the total duration is unbounded unless you impose one yourself. Both providers emit keep-alives — named ping events on one framing, bare comment lines beginning with a colon on the other — specifically so that a long pause before the first token does not look like a dead connection to intermediaries, and those keep-alives also reset your read timeout, which is exactly what you want and also means the timeout can no longer detect a generation that has stalled for two minutes. A wall-clock budget enforced by the consumer, separate from the socket timeout, is the piece that has to be added.

The most consequential thing a stream is not is a guarantee of success. Once the provider has sent response headers, the HTTP status is fixed at 200, so a failure that happens after generation has begun — an overloaded backend, a timeout inside the provider, an aborted generation — cannot be reported as a status code. It arrives inside the stream as an error event on the typed framing, or as a chunk carrying an error object on the chunk framing, and a parser that only knows about text deltas will treat a stream that ended early as a short but successful answer.

That constraint is not a provider choosing to be unhelpful, and the chain behind it is three steps long. A status code is part of the response's opening, and the opening has to be sent before any of the body can follow. A streamed body starts arriving almost immediately, which is the entire point of asking for one. So by the time anything can go wrong during generation, the status has been on the wire for seconds and there is no mechanism for revising it. Every streaming protocol that has ever existed reaches the same place, and the conclusion for your code is uncomfortable but unambiguous: on a streamed call, the status code tells you the request was accepted and tells you nothing whatsoever about whether it succeeded.

The rule that follows from all of this is to make the terminal event part of your success criterion rather than an incidental detail. A stream is complete when the event that closes it has arrived, carrying a stop reason and a usage record; anything else is a failure, however much text you accumulated. Written that way, the four decisions a framing makes — event names, text path, terminator, usage location — become four things you check per provider once, rather than four assumptions your handler carries silently until the day it meets the other shape.

Handling that properly means classifying what went wrong, which is where the two providers diverge again and more sharply than they do on framing. Before that, though, there is a class of failure your code may never see at all, because the client is already handling it: the attempt that failed and was quietly made again.