Server-Sent Events

A model that needs eleven seconds to write 700 tokens has something worth showing at 340 milliseconds and, under the assumption that one request produces one complete response, no way to show it. Streaming is the arrangement that lifts the assumption: the server sends its answer in pieces as it produces them rather than holding the finished message and writing it once at the end. The pieces are deltas, each one a short run of text — often two or three characters — that the client appends to whatever it has already received, so the message on screen grows by accumulation. This topic is about what that costs and what breaks around it, and every problem in it comes from something sitting between the tokens and the person reading them: a buffer, a proxy, a retry, an abort that never travelled.

The transport carrying those deltas, on every major provider, is Server-Sent Events — abbreviated to SSE everywhere below — and there are two places in a normal system where you meet it. The first is between your server and the provider, switched on with a stream: true field in the request body; the provider's SDK may hide the framing behind a typed iterator, and the bytes underneath are SSE either way. The second is between your own server and the browser, and on that leg there is no SDK doing the hiding, because the endpoint is one you write. This chapter is about the first leg and about the framing itself, because none of the failures on the second leg — an event arriving halved, a stream that ends without saying so, a first token that appears nine seconds late — is legible while the format is a black box.

What comes back on the wire#

Server-Sent Events predates any of this. It was standardised as part of HTML5 around 2009 for pushing stock tickers and notification counts into browsers, and it exists because the alternative at the time was long-polling: open an XHR, have the server hold it until it had something to say, get one message, open another. SSE replaced that with one long-lived HTTP response whose body never ends until the server decides it does. Model providers reached for it because a token stream is exactly the shape it was designed for — many small messages, one direction, over ordinary HTTP.

Run a streaming call with curl -N, which disables curl's own output buffering, and you can watch the body arrive:

curl -N https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{
    "model": "claude-opus-5",
    "max_tokens": 256,
    "stream": true,
    "messages": [{"role": "user", "content": "Name the capital of Portugal."}]
  }'

The response headers say content-type: text/event-stream and carry no content-length, because the server does not know how long the body will be when it writes the headers. What follows, on the wire, looks like this — the blank lines are real and load-bearing:

event: message_start
data: {"type":"message_start","message":{"id":"msg_01Ab","role":"assistant","content":[],"usage":{"input_tokens":14,"output_tokens":1}}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"The capital"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" of Portugal is"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":" Lisbon."}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":9}}

event: message_stop
data: {"type":"message_stop"}

Before the field names are worth memorising, it is worth knowing why the format needs a terminator at all. An ordinary HTTP response announces its size in a content-length header, and a client that has been given that number knows where the body ends without inspecting a single byte of it. A server writing a token stream cannot supply the number. The length depends on how much the model decides to say, and the headers go out before the model has said anything, which is the same property that made streaming worth doing. What is left is to mark the boundaries inside the body, and marking a boundary inside a body requires a byte sequence that can be recognised on sight and that a payload can never produce by accident.

The format gets one for free from a rule it already had. Every line is field: value, so every line that belongs to an event has at least a field name on it, which leaves the empty line as the one sequence no event can contain. That is the whole reason the blank line carries the meaning, and the reason a client that looks for any other boundary is looking for something the format never promised it. It is also why the missing content-length costs you a second thing that is easy to want: there is no total to compare progress against, so a streaming answer cannot be given a progress bar, only a count of what has arrived so far.

The framing rules are short enough to state completely. The body is UTF-8 text, always — there is no binary mode. It is a series of lines of the form field: value, where the fields that exist are event, data, id and retry, and anything else is ignored. A blank line terminates the current event and dispatches it. If an event carries several data: lines they are concatenated with a newline between them, which providers do not generally use but which your parser must not break on. A line beginning with a colon is a comment, and its usual purpose is a heartbeat: some infrastructure sends : ping every fifteen seconds so that an idle connection does not look dead to an intermediary. Your parser has to skip those rather than hand them to JSON.parse.

The comment line deserves one more sentence, because it is a case of a format acquiring a feature without being given one. A line that begins with a colon has an empty field name, and the rule that unrecognised fields are ignored already covers it, so any server or intermediary can write bytes into a quiet stream without inventing a message type and without any client having to be taught about it. That is what makes a heartbeat cheap here: the bytes mean nothing to every conforming parser, and their entire job is to have travelled. A later chapter depends on that being free, when a relay has to keep an idle connection alive across a slow tool call by writing the ticks itself.

Two fields in that list exist for a use case that token streaming does not have. id: labels an event, and retry: tells a reconnecting client how long to wait; together they support resumption, where a client that drops reconnects with a Last-Event-ID header and the server replays from that point. That works for a notification feed, where the events already exist on the server and can be replayed. It does not work for generation, because the events do not exist until the model produces them and a reconnect would start a new, differently-sampled answer while being billed as a fresh request. Providers therefore do not send id: on deltas, and you should not build a resumption scheme on the assumption that they might.

The step in that argument worth slowing down on is why the events cannot simply be stored and replayed the way a notification feed's are. A feed's events exist before anybody subscribes to them: something else produced them, they were written down, and a subscription is a view onto a record that is already there. A generation has no such record. The text is produced by decoding, one forward pass per token, and the state that produced it is discarded when the request ends — so there is no position in it to seek back to and nothing to replay from. Asking a provider to resume is therefore asking it to run a new generation, which it will bill as one and which will not say the same words, because a fresh draw from the same distribution is a different draw. The id: field is not absent from these streams through oversight. There is nothing for it to name.

Providers differ in what they put inside the data payload. The trace above is Anthropic's, which gives every event a named type and separates the message envelope from the content deltas. OpenAI's Chat Completions stream omits the event: line entirely, ships one JSON object per data: line shaped like a chat completion with a choices[0].delta fragment instead of a message, and closes with a literal sentinel:

data: {"id":"chatcmpl-9x","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":" Lisbon."},"finish_reason":null}]}

data: [DONE]

That [DONE] is not JSON and will throw if you parse it blindly. It exists because HTTP gives the client no way to distinguish "the body ended because the answer finished" from "the body ended because a router dropped the connection" — in both cases the socket simply closes. A terminal sentinel is the stream's own statement that it completed, and a later chapter turns entirely on that distinction.

The decoding loop#

The mistake worth naming here is treating the transport's chunks as if they were the protocol's events. A ReadableStream reader, or a Python iterator over the raw body, hands you whatever bytes happened to arrive in one TCP segment. That boundary has nothing to do with where events begin and end. Most of the time an event fits inside one read and the naive version works, which is why it survives code review and reaches production; then a read ends at the path's maximum transmission unit — the MTU, the roughly 1,400 bytes a single network segment can carry — landing mid-payload, one data: line is split across two reads, and the parse throws on a truncated object. Under a local test against a fast provider connection you can send a few hundred requests without seeing it once.

The chain that produces the mismatch is short and worth following once, because the same chain reappears three chapters later with a relay standing in the middle of it. The provider serialises one event and writes it. The write hands those bytes to a kernel, which is free to send them when and how it likes, subject to the path's MTU and to whatever is already queued ahead of them. On the receiving side, a read returns whatever has arrived by the instant you ask for it. Not one step in that sequence takes the event's contents as an input, so where a read ends is determined by timing and packet size and by nothing else. Events and reads coincide constantly, and they coincide for an environmental reason rather than a protocol one.

That is also why the naive parser survives review rather than failing loudly on somebody's laptop. Against a provider on a fast, uncongested link, each small write leaves as one segment, arrives as one segment, and is handed back by one read, so events and reads line up on nearly every request — and "nearly every" is indistinguishable from "every" until the volume gets large or the path gets worse. The belief being carried into production is not that the parser is correct. It is that a green test run is evidence about framing, when it is evidence about the network the test ran on.

The correct shape is a buffer that accumulates bytes and only yields events when it has seen the terminator:

const res = await fetch("https://api.openai.com/v1/chat/completions", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "gpt-4.1",
    stream: true,
    messages: [{ role: "user", content: prompt }],
  }),
});

if (!res.ok || !res.body) throw new Error(`upstream ${res.status}`);

const reader = res.body.getReader();
const decoder = new TextDecoder("utf-8");
let buffer = "";
let text = "";

while (true) {
  const { done, value } = await reader.read();
  if (done) break;

  buffer += decoder.decode(value, { stream: true });

  let boundary: number;
  while ((boundary = buffer.indexOf("\n\n")) !== -1) {
    const raw = buffer.slice(0, boundary);
    buffer = buffer.slice(boundary + 2);

    for (const line of raw.split("\n")) {
      if (!line.startsWith("data:")) continue;
      const payload = line.slice(5).trim();
      if (payload === "[DONE]") return text;
      const delta = JSON.parse(payload).choices[0]?.delta?.content;
      if (delta) {
        text += delta;
        render(text);
      }
    }
  }
}

Three details in that loop are easy to skip and expensive to skip. The { stream: true } option on decoder.decode tells TextDecoder to hold back an incomplete multi-byte sequence rather than emit a replacement character; without it, a chunk that ends in the middle of an em dash or a Japanese character produces a permanent � in the stored answer, and the bug reproduces only in languages your tests do not use. The inner while loop matters because one read can contain several complete events — at a steady 40 to 60 events per second, and a network that delivers every 20 to 40 milliseconds, two or three events per read is normal, and an if instead of a while silently drops all but the first. Finally, the buffer is checked for a terminator before anything is parsed, so a partial line simply waits for the next read.

Why the decoder needs to be told about the split is worth putting in mechanical terms, because it explains why the bug hides where it does. UTF-8 spends one byte on the characters in the ASCII range and two, three or four bytes on everything else, and a read boundary is a boundary between bytes rather than between characters. A decoder handed the leading fragment of a multi-byte sequence has two things it can do: hold the fragment until the rest arrives, or emit the replacement character and move on. The default is the second, because the default assumes it has been given a whole document rather than a piece of one. An English test corpus never puts a read boundary in the middle of a character, because it contains no multi-byte characters to be in the middle of, which is why the first em dash or Japanese noun in production is where the flag stops being optional.

Real SSE also allows \r\n\r\n as a terminator, so a parser hardened for arbitrary servers splits on a small regex rather than a literal "\n\n". Against a known provider the literal is fine, and it is written that way here so the mechanism stays visible.

You may reasonably ask why this is hand-written at all when every provider ships an SDK whose stream helper does it for you. In application code, use the SDK: client.messages.stream(...) in Python or client.chat.completions.create({ stream: true }) in TypeScript hands you typed events, accumulates the final message, and handles the framing correctly, including the cases above that are easy to get wrong. The reason to have built the loop once is that the failures in the rest of this topic present as symptoms in your own infrastructure, where a working model of the framing is a prerequisite for diagnosing any of them. The other reason is that an SDK helper only exists between your server and the provider. Between your server and the browser there is no SDK, because the protocol on that leg is one you are about to define.

Painting a message that keeps growing#

The render(text) call in that loop is doing more work than it looks, and the amount depends entirely on what the text is. If the target is element.textContent, four hundred assignments over nine seconds cost nothing measurable. Chat answers are almost always markdown, though, and markdown has to be parsed before it can be shown.

Two things go wrong at once. The first is that partial syntax renders as itself: a delta that arrives as **Liab puts two literal asterisks on the screen until the closing pair arrives half a second later, a heading shows as ### for a beat, and a table renders as a column of pipe characters for the several seconds it takes the model to finish the rows. None of that is incorrect — the parser is faithfully rendering incomplete input — but it reads as the interface glitching.

The second is arithmetic. Re-parsing the accumulated text on every event is linear in the message length per event and therefore quadratic over the message, so a 3 KB answer delivered in 400 events involves 400 parses averaging 1.5 KB each. Add a virtual DOM reconciliation for each of those and, on a mid-range Android device, a full parse-and-diff cycle at that size measures 12 to 20 ms. At 50 events per second the work needed exceeds the time available, the main thread saturates, and the page stutters while the model is still generating.

The word "quadratic" there is carrying an argument that is easier to believe once the sum is written out. Each parse costs time in proportion to the length of the text handed to it, and the text handed to it grows by one delta each time, from nearly nothing on the first event to the whole answer on the last. Adding up a quantity that climbs evenly from nothing to its final value gives the same total as multiplying that final value by half the number of terms, which is why the average parse over a 3 KB answer delivered in 400 events runs over about 1.5 KB rather than over 3 KB. Both factors — how many parses there are and how big each one is — scale with the length of the answer, so the total work scales with the square of it. An answer twice as long does not cost twice as much to render. It costs four times as much.

Coalescing to the display refresh fixes the cost without changing anything about the protocol:

let pending = "";
let scheduled = false;

function onDelta(chunk: string) {
  pending += chunk;
  if (scheduled) return;
  scheduled = true;
  requestAnimationFrame(() => {
    scheduled = false;
    setMessage(pending);   // one parse+render per frame, not per event
  });
}

That caps rendering at the frame rate regardless of how fast events arrive, which at 60 events per second is roughly one render per frame instead of one per event, and it costs nothing in perceived smoothness because the display cannot show more than one state per frame anyway.

That last clause is the whole justification, and it is a claim about hardware rather than about taste. A display presents a fixed number of images per second, and every state the page passes through between two of those images is overwritten before any of it reaches an eye. Rendering a state that is never presented is therefore work with no observable result, and all the coalescer does is stop performing it. Nothing about the protocol changes: the same deltas arrive at the same moments and accumulate in the same order, and the only thing made less frequent is the parse. That is the difference between capping the render and capping the stream, and it is why the fix belongs in the client rather than in the wire format.

The partial-syntax problem is untouched by any of that, since coalescing does not make an unterminated ** less visible. The usual approach there is to hold back block-level constructs until their terminator arrives — show a fenced code block as plain text until the closing fence, and buffer table rows until the row ends — while letting inline text through immediately.

One more piece of behaviour belongs here because it is invariably filed as a bug later: scroll position. Appending to a container that is scrolled to the bottom should follow the text down, and appending to one the user has scrolled up in must not yank them back. Track whether the container is within a few pixels of the bottom before each render and only auto-scroll when it is.

Where SSE stops being the answer#

The browser has a built-in SSE client, EventSource, and it is close to useless for this. EventSource issues a GET with no request body and no custom headers, which means you cannot POST a conversation to it and cannot attach an Authorization header. Teams work around it by stuffing the prompt into a query string and the token into a cookie, and then discover that URLs have practical length limits around 2,000 characters and that a 40-message conversation does not fit. fetch with a manually decoded ReadableStream, as above, is what production clients actually use; the cost is that you give up EventSource's automatic reconnection with Last-Event-ID, which is not a feature you want for token generation anyway, since a reconnect would restart a paid generation without telling you.

It is tempting to read EventSource's limitations as an oversight, and they are closer to a consequence. It was built to reconnect on its own: when the connection drops it waits and re-issues the request with no involvement from the page that opened it. A request the browser has to be able to reproduce unattended must be fully described by something the browser kept, and what it keeps is a URL. A body is not part of a URL, and neither is an Authorization header, so an interface that promises unattended reconnection has already given up both before anyone writes a line of application code. The workarounds fail for exactly the reason they were reached for. A URL is the only thing available, and a URL is not a place to put a conversation or a credential.

SSE is also text-only, which is not a limitation you notice on a chat endpoint and is a hard wall on anything else. There is no binary framing, so audio or image bytes have to be base64-encoded into the data field at a 33% size penalty, and a stream that carries 10 MB of audio pays 3.3 MB for the privilege. Providers that stream audio do it over a different transport for exactly this reason, and the chapter after next weighs that transport against SSE.

The size penalty has a cause worth knowing, because knowing it bounds what any work-around can achieve. Base64 exists to express arbitrary bytes using only characters that survive a text channel, and it does that by taking three bytes at a time and writing them as four characters. Four characters carrying three bytes is a third more than you started with, and that ratio is a property of the encoding rather than of the payload, so it does not improve on larger files and there is no setting to tune. A text-only transport is not a mild inconvenience for binary data. It is a fixed tax on every byte, charged before anything else about the design is decided.

It is also worth being precise about what a delta is. A content_block_delta is not a token. The provider emits whatever text it has flushed at that moment, which may be part of a token, one token, or several; deltas near the start of a response are often two or three characters and deltas mid-paragraph are often a whole word plus its leading space. The relationship between decode steps and network events is loose, and treating event counts as token counts will give you a number that is close enough to be believed and wrong enough to matter.

SSE moves bytes earlier; it does not move work earlier. The model is doing the same decode described in the topic on inference internals, one forward pass per token, and the total generation time is unchanged. What you have bought is the ability to show the user each token at the moment it exists rather than at the moment the last one does.

That is the sentence to carry out of the chapter, because it is the one that gets streaming proposed for the wrong reasons. The model runs one forward pass per token whatever the socket is doing, the last token arrives no earlier than it would have, and a human cannot read the answer before the model has written it. The only quantity that moved is when the first token became visible, and the whole of the rest of this topic is the price of that move — because holding one connection open across the same eleven seconds, instead of closing a short one at the end, is what puts an accounting event out of reach, leaves a failure with no status code to travel in, and turns a Stop button into a signal somebody has to carry by hand.

Everything later in this topic is built on the two mechanics above, and both are worth being able to write from memory: an event is bytes up to a blank line, and a client accumulates reads until it sees one. The wire format has no other subtlety in it — four field names, a comment line, a terminal sentinel that some providers send and others express as a named event — which is fortunate, because the format is the only part of a production streaming path that behaves the same in every environment you will run it in.

What the format does not tell you is what any of it cost. The deltas carry text and nothing else, which is fine while a human is watching the words appear and becomes a problem the first time somebody asks the obvious follow-up question about the invoice.