Streaming responses and buffering

Streaming is the mode in which a server writes each token into an open HTTP response as it is generated, rather than assembling the complete answer and sending one JSON document at the end. Every chat API implements it with server-sent events (SSE): one response is held open and the server writes a sequence of data: lines into it, so the client can read each one the moment it arrives rather than waiting for the body to end. The model's work is identical either way — the same prefill, the same number of decode steps, the same GPU occupancy — and the only thing that changes is when a token becomes visible.

That makes streaming unusual among latency changes: its benefit is delivered by the whole path rather than by the server, so what it is worth depends both on how long the answer is and on whether every hop between the model and the eye forwards a token as it arrives.

The same generation, twice#

Take one fixed request — 2,000 prompt tokens, 300 output tokens, temperature 0, so both runs do identical work — and issue it with stream: false and then stream: true.

The model did the same 300 decode steps in both runs. In the non-streaming run the server held every token until generation finished and then sent one JSON document, so the first useful output and the last coincide at 2,190 ms. In the streaming run the first token left at 96 ms and the last at 2,215 ms — 25 ms later than the non-streaming run, because 301 separate events carry framing overhead, force a socket flush each, and turn 1.2 KB of JSON into 41 KB of data: lines.

That 1.1% penalty on total time is the honest cost of streaming, and it is the correct trade almost everywhere a human is reading the output. The quantity that changed by 23× is the one the user experiences as responsiveness.

Both figures are a single division, and confirming them takes a minute that pays, because they are the two halves of the trade. The penalty is 25 ms of extra total time set against a 2,190 ms total, which is a little over one part in a hundred. The gain is 2,190 ms of waiting with nothing on screen set against 96 ms, which is the factor of 23. About one per cent of the total was spent to remove about ninety-five per cent of the wait.

Where those 25 ms come from is the event count rather than the tokens. Each of the 301 events carries its own framing: the data: prefix, a JSON envelope naming the model and the index and the delta, and a blank line to end the event, all wrapped around a few characters of text. The envelope is several times the size of its payload, which is how 1,180 bytes of answer becomes 41,600 bytes on the wire. Each event also forces a flush instead of accumulating in a socket buffer, and a flush is a system call and usually a packet. Three hundred and one small writes cost more than one large write. The penalty is therefore proportional to the number of events rather than to the length of the answer in words, which is also why providers that coalesce several tokens into one event pay less of it.

The size of the win depends entirely on how long the answer is, which is worth working out before promising anything. Streaming saves the user the decode time of every token after the first, so on a 300-token answer at 7 ms per token it converts a 2.19-second wait into a 96-millisecond one; on a 20-token answer it converts 230 ms into 96 ms, and nobody notices. Endpoints that produce short answers — a classifier returning a label, a router returning a tool name, a structured extraction of six fields — gain almost nothing from streaming and inherit all of its complications. The benefit scales with answer length, so the features that most need streaming are the ones already at the top of your total-latency chart.

Written against the composition formula the rule is exact rather than a rule of thumb. Without streaming a user waits for the whole composition: time to first token, plus one decode step for every token after the first. With streaming a user waits for time to first token alone. Subtract the second from the first and the saving is precisely the decode time of everything after the first token. That is why the win is proportional to answer length, why a classifier returning one label gains nothing worth having, and why neither the model nor the hardware enters into the size of it. Streaming is not a speed-up that happens to be larger on long answers. It is the deletion of one term from a sum, and the term it deletes is the one that grows.

The comparison also shows why inter-token latency is unmeasurable without streaming. In the non-streaming run there is exactly one timestamp, so the only derivable number is total time divided by token count, which averages prefill and decode together into a rate that describes neither. The 7.0 ms figure exists only because 301 timestamps do.

Five places a token can be held#

The server's emission timestamp and the user's perception timestamp are separated by a chain of components that each have their own opinion about buffering. Traced through a real path, with a reverse proxy configured the way it ships, the delay lands almost entirely in one hop:

model emits token          t = 96 ms
  → SSE framing + flush    t = 96 ms
  → nginx (proxy_buffering on)   held until t = 4,190 ms  ← the bug
  → CDN / edge             +12 ms
  → browser fetch reader   t = 4,210 ms
  → React state + paint    t = 4,260 ms   ← what the user sees

Reverse proxies buffer by default because for ordinary responses it is the right thing to do: it lets nginx free the upstream connection early and absorb slow clients. For an event stream it destroys the feature. The fix is a per-location override plus the header that lets the application opt out for a single response:

location /v1/chat/ {
    proxy_pass http://inference_backend;
    proxy_buffering off;            # forward each chunk as it arrives
    proxy_cache off;
    proxy_read_timeout 300s;        # generations outlive the 60s default
    gzip off;                       # gzip buffers to fill its window
    chunked_transfer_encoding on;
}

The application sends X-Accel-Buffering: no alongside Content-Type: text/event-stream so that any nginx in the path, including ones you do not control, disables buffering for that response specifically.

Why none of this ever raises an error needs stating plainly, because the expectation that it would is what keeps these bugs alive for weeks. HTTP has no notion of promptness. A response is a status line, a set of headers and a body, and a proxy that forwards the body in one piece at the end has produced exactly the same response as one that forwarded it in three hundred pieces as they arrived: same bytes, same order, same status, same length. Timing is not part of the contract any hop is checking, and nothing downstream can tell from the response it received how the response was paced. Every component in the path can therefore be behaving correctly while the feature is entirely absent. There is no exception to catch, no header to inspect at the far end, and nothing to alert on — only the difference between two timestamps taken at opposite ends of the path, which is why the diagnosis has to be a comparison rather than a check.

Compression is the subtler member of the family. A gzip encoder wants a full window of input before it emits anything, so a compression layer anywhere in the path will hold tokens until it has enough of them, producing a stream that arrives in bursts of two hundred tokens rather than one at a time. The symptom is a TTFT that looks acceptable and an inter-token latency histogram with a bimodal shape — long stalls separated by instantaneous bursts — and it is worth checking for explicitly, because a burst of two hundred tokens after a one-second stall is measurably worse to read than a steady stream at the same average rate.

That behaviour follows from what compression is rather than from a bad default. A gzip encoder finds repeated sequences inside a window of recent input, so it cannot emit the encoding of a byte until it has seen enough of what comes after that byte to know whether it begins a repeat. Flushing after every token would force it to close a block early each time, which both discards most of the compression it was installed to provide and adds framing per flush. Waiting is therefore the correct behaviour for a compressor and the fatal behaviour for a stream. The conflict is real rather than a misconfiguration: on a streaming response, compression trades latency for size, and there is no setting that delivers both. Disabling it for event streams while leaving it on for everything else is the resolution, which is why gzip off appears in the location block above rather than anywhere global.

Some infrastructure cannot stream at all. An API gateway configured for buffered request/response integration, a serverless function without a streaming response mode, or an application layer that parses the whole body before re-emitting it will each turn a streaming upstream into a non-streaming service, and no configuration flag on the model server changes that. This is worth checking before designing a product around time to first token, because the fix is architectural.

The last hop is the client's own rendering. A loop that appends each token to a string, sets state and lets the framework re-render can spend more time laying out text than receiving it once the answer is a few thousand characters long, and markdown re-parsed from scratch on every token is the usual culprit. Batching updates on an animation frame — accumulate tokens into a buffer, flush on requestAnimationFrame — decouples render cost from token rate and typically removes the problem entirely.

Measuring at the point where the promise is made#

If your service level objective — the SLO, the numeric promise a team writes down and is then held to, which chapter 8 is entirely about — says a user sees the first word within 800 ms, the measurement has to be taken where the user is. The browser-side version is short and does not require any special tooling:

const t0 = performance.now();
const res = await fetch("/api/chat", {
  method: "POST",
  body: JSON.stringify({ messages, stream: true }),
});
const reader = res.body.getReader();
let firstChunkAt = null;
const arrivals = [];
for (;;) {
  const { value, done } = await reader.read();
  if (done) break;
  const now = performance.now();
  if (firstChunkAt === null) firstChunkAt = now - t0;
  arrivals.push({ at: now - t0, bytes: value.length });
}

Two things about that measurement need care. It records chunk arrivals, and a chunk is not a token: providers frequently coalesce several tokens into one SSE event, and a client that computes inter-token latency as the gap between chunks will report a number several times too large. Parse each event and count the tokens inside it, then divide the gap by that count, or report the honest thing instead — a sustained tokens-per-second rate computed over a sliding window. And firstChunkAt is the first byte, which on many APIs is a role prelude or an empty delta rather than visible text. The number that matches the product promise is the timestamp of the first chunk containing non-empty content, and on providers that emit a preamble the two can differ by 30 to 100 ms.

A stall metric belongs alongside the averages, because the thing that makes streaming feel broken is rarely the mean rate. A stream delivering 140 tokens per second on average, with one 900-millisecond gap in the middle where the scheduler admitted a batch of new prefills, reads worse than a steady 90 tokens per second, and the p50 inter-token latency of the first is better than the p50 of the second. Recording the maximum gap per response, and the count of gaps above some threshold such as 500 ms, catches what the percentiles of a per-token distribution smooth away — a single 900 ms stall in a 300-token response is one sample in three hundred and does not reach the p99.

Once the numbers are at the client, the useful comparison is the pair: server-side TTFT from the engine metrics of chapter 4, and client-side first-visible-token from this measurement. Their difference is the pipeline's contribution, and it should be a network round trip plus a few milliseconds. A gap of 4.1 seconds is a buffering proxy; a gap of 300 ms is a compression layer; a gap that grows with response length is the client's rendering loop.

Capacity, batch jobs and half-delivered answers#

Streaming does not make anything faster in the sense of total work. The generation takes marginally longer, the token cost is identical, and a request that took 2.2 seconds still occupies the GPU for 2.2 seconds. Capacity planning is unaffected by whether you stream.

It does not help a workload with no human waiting. A nightly summarisation job that writes to a database gains nothing from receiving tokens incrementally and pays the framing overhead and the extra connection lifetime for it, which chapter 9 returns to when it works through choosing an endpoint for offline work.

And it complicates failure handling in a way worth knowing before you commit. Streaming responses send HTTP 200 and their headers before generation finishes, so a failure at token 180 arrives as a truncated body or an error event inside an already-successful response. Clients that only check status codes will treat a partial answer as a complete one. Retries become awkward, because the user has already seen half an answer, and any policy that inspects a complete response — a moderation pass, a schema validation on structured output, a guard that rewrites unsafe text — has either to buffer the stream and give up the benefit, or to operate incrementally and accept that it may act after the user has already read something.

Streaming, then, is a change to the delivery path rather than to the model, and that is what determines both its value and its failure modes. Its value is bounded by how long the answer is and realised only if every hop cooperates; its failures are silent, because a buffered stream is a correct HTTP response. Anything that promises a first-word latency has to measure at the point where the word appears, which is the browser, and anything that promises a smooth stream has to record the largest gap rather than the average one.

Streaming's other effect on this topic is that it makes inter-token latency a user-visible quantity rather than an implementation detail. Before streaming, a server could slow decode by 40% and only a total-time chart would notice; after streaming, the text visibly crawls. That matters because the most common way to raise aggregate throughput is to run more sequences through decode at once, and every one of them shares the same weight reads. The next chapter is about why that sharing is so effective, and what it does to the rate each individual reader sees.