In-band errors and terminal sentinels

The property that makes streaming useful — headers out immediately, body written over the following nine seconds — means the response has already committed to being a success while almost all of its content is still unwritten. The comparison that fits is a phone call: you can decline it before you pick up, but once you have said hello the only way to report a problem is to say something, and the only alternative is that the line goes dead.

Underneath the picture the mechanism is short, and it is the whole chapter. An HTTP response is a sequence of bytes and its first line is the status line. Streaming means the first bytes leave at 340 milliseconds, which is the entire point of the feature, so the status line is chosen and sent while almost none of the answer exists. Bytes already on the wire cannot be recalled. Any failure after that instant therefore has no status code available to it. It is not that HTTP forbids a second status; the slot a status occupies was written and spent nine seconds before the failure happened, and there is no second slot. What remains open is the body, and the only alternative to using the body is to stop writing it.

The phone call is a picture rather than a fact about HTTP, so it is worth saying where it stops being useful. It gets the sequence right: acceptance is committed before content, and after acceptance the content is the only channel. It gets one thing wrong that matters a great deal here. On a phone call silence is ambiguous for a moment and then somebody says "hello?", because the medium is symmetric and either party can probe it. A client reading an SSE body has nothing it can say back over that body, so silence stays silent for as long as whatever platform limits exist allow it to. That asymmetry is why the fourth state below needs a timer somebody writes by hand, and why nothing in HTTP will write it for them.

Two terms carry the rest of this chapter. An in-band error is a failure reported inside the stream's own body, as one more event alongside the text, because the channel for out-of-band signalling closed when the status line went out. A terminal sentinel is the event that says the message finished — data: [DONE] on OpenAI's stream, the named message_stop on Anthropic's, and whatever your own relay emits to a browser. Neither is decoration. Together they are the only two things that let a client distinguish a completed answer from a dead connection, and the chapter is about what your relay owes each of them.

The four ways a stream ends#

There are exactly four terminal conditions, and a client that handles fewer than four will mis-report at least one of them.

The stream completes and says so, with the provider's message_stop or [DONE] or the event: done your relay emits. The stream fails and says so, with an in-band error event. The connection closes with no terminal marker at all, because a router dropped it, a process restarted, a load balancer idled it out, or the upstream socket reset. Or nothing arrives and nothing closes, which is a stall: the sockets are healthy, the upstream has simply gone quiet, and nothing will notice unless a watchdog that resets on every forwarded event is running. fetch has no read-idle timeout by default, only a connection timeout, so a stalled provider will hold both legs open until whatever platform limit exists fires many seconds later.

The state worth staring at is Unknown. From the client's position, an EOF with no sentinel — end of file, the read loop reporting that the body has finished and no further bytes are coming — is indistinguishable from an EOF after a sentinel that never got written, and both look, to any code that simply awaits the end of the body, like the loop finishing normally. A for await over a ReadableStream exits cleanly in both cases; nothing throws; the accumulated text is whatever arrived. If you do not check for the sentinel explicitly, a truncated answer is stored and rendered as a complete one.

That is the whole reason a terminal sentinel exists. It is the only positive evidence that the message finished, because the transport gives you none.

"Positive evidence" is the precise phrase and it repays being unpacked, because what a client would prefer to rely on is the absence of trouble, and the absence of trouble is not available here. Nothing arrives to announce most of the ways a stream fails, so an inference of the form "no error appeared, therefore it went well" rests on a channel that disappears in exactly the cases it is being asked about. A sentinel has no such dependency. It can only have been written by the process that produced the last token, and it can only have been written after that token, so its presence is a fact about the producer's state rather than about the client's failure to notice anything. Everything else in this chapter — the persisted partial flag, the metric in the finally, the client flag checked after the loop rather than inside it — is machinery for making a system that has only positive evidence behave sensibly.

What the relay says when the upstream breaks#

Providers do emit in-band errors when they can. Anthropic sends a named event, which can appear at any point after the stream has begun:

event: error
data: {"type":"error","error":{"type":"overloaded_error","message":"Overloaded"}}

OpenAI's stream can carry an error object inside an otherwise ordinary chunk, and in either case the important property is that your HTTP client does not raise. response.ok was true nine seconds ago. The SDK's stream iterator may translate a recognised error event into a thrown exception, which is convenient and worth relying on where it happens, but a relay reading raw SSE sees an ordinary event with an unusual payload and will forward it as a delta unless it looks.

The reason your HTTP client stays quiet is the same reason the status code cannot help, arriving one layer up. An HTTP client raises when the response cannot be obtained or when the status says so, and both of those questions were settled at the headers. Once the headers are good, every subsequent byte is just body, and a body is a thing the client hands to you rather than a thing it judges. An error event is a run of field: value lines terminated by a blank line, exactly like every delta before it, and nothing about its shape marks it out. Only something that reads the payload can tell — which is why the check belongs in parseProviderEvent, alongside the code that already distinguishes a text delta from a stop, rather than in a try around the transport.

So the relay's event parser gains a third kind alongside text and stop, and the whole of the failure handling becomes a try around the loop plus a discipline about what leaves the process. Here the read-split-parse loop from the previous chapter is lifted into an async generator, providerEvents, so that the error handling can wrap it rather than live inside it:

const out = new ReadableStream({
  async start(controller) {
    const send = (event: string, payload: unknown) =>
      controller.enqueue(
        encoder.encode(`event: ${event}\ndata: ${JSON.stringify(payload)}\n\n`),
      );

    let full = "";
    let terminated = "unknown";

    try {
      for await (const event of providerEvents(upstream.body!)) {
        if (event.kind === "error") throw new UpstreamError(event.code);
        if (event.kind === "stop") { terminated = "complete"; break; }
        if (event.kind === "text") {
          full += event.text;
          send("delta", { text: event.text });
        }
      }

      if (terminated !== "complete") throw new UpstreamError("truncated");

      await persist(conversationId, full, { status: "complete" });
      send("done", { chars: full.length });
    } catch (err) {
      terminated = "error";
      await persist(conversationId, full, { status: "partial" });
      send("error", {
        code: err instanceof UpstreamError ? err.code : "relay_failure",
        retryable: true,
        chars: full.length,
      });
    } finally {
      metrics.increment("stream.terminated", { by: terminated });
      controller.close();
    }
  },
});

Three decisions in that block are the substance of the chapter. The upstream body ending without a stop event is treated as a failure rather than as a completion. The partial text is persisted with an explicit partial status instead of being dropped or stored as if it were an answer, so that anything reading the conversation later can tell the difference. And the error payload carries the character count already delivered, because the client needs to know how much of what it is showing came from a stream that did not finish.

That partial flag has more consumers than it first appears to. The next turn replays the conversation, and a truncated assistant message sent back as history reads to the model as a message that stopped mid-sentence, which it will sometimes try to explain or continue; a summarisation job over stored conversations will treat the half-answer as the answer; and an evaluation set built by sampling stored messages will quietly include truncations as model output, depressing scores for a reason nobody can locate. One boolean, checked in a few places, avoids all of that.

The client side of that contract is a single flag, and its placement matters more than its complexity:

let sawTerminal = false;

for await (const event of clientEvents(res.body!)) {
  if (event.type === "delta") append(event.text);
  if (event.type === "error") { fail(event.code); sawTerminal = true; break; }
  if (event.type === "done") { sawTerminal = true; break; }
}

if (!sawTerminal) fail("connection_lost");   // the read loop ended by itself

The last line is the one that gets omitted, because the loop above it exits cleanly and there is nothing to catch. Deciding on the stream's outcome after the loop rather than inside it is what makes the Unknown state observable at all.

What the relay must not put in that payload is the upstream's message verbatim. Provider error strings have been known to include request identifiers, model names you have not announced, and occasionally fragments of the prompt, and this one is going to a browser. Map to your own codes and log the original server-side.

The failure the relay never sees#

Everything above assumes the process gets to run its catch block, and the most common streaming failures do not give it that chance. If the pod is evicted mid-generation, if the connection is reset, if a network path disappears, the client observes the Unknown state and no error event is ever written — the code that would have written it is gone.

Four of the seven rows produce no error event, and three of those produce nothing at all. Silence is not the exceptional case; it is the majority of the failure modes by count, and typically 0.2% to 0.6% of streamed requests on a fixed-line population, higher on mobile. A client written on the assumption that failures announce themselves is correct about one row of that table.

That silence dominates is a consequence rather than an accident of how the table was drawn. An error event is written by your process, by the code path that is handling the failure. Any failure that removes the process, the container, the network path or the connection therefore also removes the thing that would have reported it — the reporter and its subject share a fate. The failures with the widest blast radius are the ones guaranteed to produce no report, and only a narrow class survives to write one: the cases where something upstream of you breaks while you carry on running. Designing a client around loud failures is designing it around the minority that happened to leave a witness behind.

The practical consequence is that the client's completion check is a positive test, not a negative one. Treat the stream as failed unless a sentinel arrived. In the browser that means a flag set only by the done handler, checked after the read loop exits, with the partial text left visibly marked as incomplete rather than silently committed.

An availability number that counts sentinels#

A dashboard computing success from the response status will report every one of those rows except the first two as a success, and the fix is available from the metric the relay already emits in its finally. Count streams started and streams that reached a terminal state; the ratio of stream.terminated{by=complete} to stream.started is the availability of the endpoint as a user experiences it.

The ratio is the right shape because of what its two ends are counted at. The numerator is incremented by code that ran only after a sentinel arrived, so it counts answers that finished. The denominator is incremented before anything can go wrong, so it counts attempts. Dividing one by the other asks the only question a user has, which is what fraction of the times they asked did they get a whole answer — and it asks it without consulting the status code at all, which is the point, since the status code is a fact about the first 340 milliseconds and the question is about the following nine seconds. A status-based dashboard is not measuring the wrong thing carelessly. It is measuring a different thing accurately.

Two supporting series make the number diagnosable rather than merely alarming. Splitting terminations by cause separates a provider problem from an infrastructure problem: a rise in by=error points upstream, a rise in by=unknown points at the network path, your own process lifecycle, or a deploy. And recording the delivered token count on non-complete terminations tells you whether users are losing a sentence or an entire answer, which is the difference between a degraded experience and an outage. A failure at token 850 of 900 is a rough edge; a failure at token 12 is nothing being delivered at all, and the same error rate can mean either.

Where the ratio misleads#

User-initiated cancellation also terminates without a complete, and on an interactive product that can be a third of long requests. Unless those are tagged separately at the point where you already know about them — the client disconnect handler — the availability metric tracks how often people press Stop, which is a product statistic wearing an outage's clothes. The next chapter is where that handler gets written, and it is worth wiring the tag at the same time rather than discovering the contamination in an incident review.

It is also worth being clear about what a 200 on a streaming route means, since a dashboard reading 100% through a degradation makes it tempting to conclude the status is useless. It means the request was accepted, authorised, and the body began, which is real information, and it is why pre-header failures should still be actual status codes rather than error events. A relay that cannot reach the provider at all should return a genuine 502, and that is correct precisely because nothing has been sent yet.

The stall state has no natural detector, which is the last gap. Nothing in HTTP notices that a healthy connection has stopped producing bytes, so it has to be a timer you own: a watchdog reset on every forwarded event, firing at some multiple of the longest legitimate quiet period — which, for a route that runs tools between deltas, may be a minute rather than twenty seconds. When it fires, the handling is the same as any other upstream failure: abort the upstream call, emit an error event, persist the partial.

Nothing lower down will do this for you, and the reason is worth one paragraph because teams look for a setting first. A stall is a connection on which everything is working: the sockets are open, the peers are reachable, the kernels have nothing queued. There is no failed operation for a library to report, because no operation was attempted and failed — the upstream simply has not written anything. A read timeout would notice, and fetch does not have one: its timeout governs establishing the connection, which succeeded nine seconds ago. So the only component in the whole path with an opinion about how long a quiet stream is allowed to stay quiet is the one you write, and the threshold has to be a judgement about your own workload rather than a default, because a route that runs tools between deltas is legitimately silent for far longer than one that does not.

So the rule is short enough to carry as a sentence — "Once the headers are sent, a failure is an event or it is silence" — and the long form adds only what follows from it. A failure is an HTTP status if it happens before the first byte and an event if it happens after, and because a large fraction of failures manage neither, the client decides on the outcome after its read loop rather than inside it and the server measures availability with sentinels rather than status codes. Everything else in this chapter is machinery for making those two sentences true.

The last row of the table is the one that costs money rather than trust. When the user closes the tab, the server sees the same silent close it would see from any network failure, and it is by a wide margin the most frequent one. Doing nothing about it is the default, and the default keeps the model generating.