Resuming from an acknowledged sequence

A durable chunk log — every emitted chunk written under a generation id with a monotonic sequence number, as the previous chapter built — tells you what was produced. A resume endpoint has to answer a different question: what does this particular reader still need. The two are not the same, and the gap between them is where reconnection bugs live.

The obvious resume endpoint gets it wrong in the safe direction. Given a generation id, send everything stored so far, then follow the live tail. For a client that has rendered nothing that is exactly right, and for a client that has rendered 311 chunks it re-sends 311 chunks the user is already looking at. Appending text to a DOM node has no idea whether that text was there before, so the visible result is a paragraph that repeats itself. This chapter is about where the boundary comes from instead, and about the delivery guarantee that boundary can and cannot buy.

Three places the boundary could come from#

The server can start replaying from position zero, from its own record of what it wrote, or from a number the client supplies.

Position zero is the version above. It is safe in the sense that nothing is ever lost, and wrong in the sense that the client's display doubles.

Starting from the server's own last write looks like the disciplined fix and it loses tokens. The durable append happens before the flush, which means the log is a superset of what the reader received: at any instant there is at least one chunk that is written and not delivered, and if the process died in that window there may be several. A server that resumes from last_seq is asserting that everything it wrote was delivered, which is precisely the assertion the write-first ordering was designed not to make. The reader ends up missing a fragment in the middle of a sentence, and because a missing fragment reads as slightly odd prose rather than as an error, nobody reports it.

That middle option is tempting for a reason worth naming, because it is the disciplined-looking answer and careful people choose it. It reads as the option that respects bookkeeping: the server has a record, the record is exact, and using a number you hold yourself feels safer than trusting one a client sent you. The trust half of that even points the right way — a cursor arriving from a client is attacker-controlled input and has to be validated against the range the log actually holds. What the reasoning misses is that the server's record answers a different question. last_seq is the answer to "what have I written", the question on the table is "what has been seen", and the write-before-flush ordering was chosen precisely to make those two numbers differ.

The boundary has to be the last sequence the client actually rendered, sent by the client, with the server transmitting strictly what follows. This is the same shape as a TCP cumulative acknowledgement, and for the same reason: the sender cannot know what arrived, only the receiver can, so the receiver names the frontier and the sender resumes after it. The difference is that a TCP ack is scoped to one connection and dies with it, while this ack is scoped to a generation id and survives the connection, the process, and a switch from a phone to a laptop.

Why the receiver is the only party able to name that frontier comes down to what each side observes. The server's evidence is that bytes left its process: a write returned, a buffer accepted them, a socket did not complain. Between that moment and a paragraph appearing on a screen sit a kernel send buffer, a network, possibly an intermediary holding the body, the browser's receive buffer, a stream parser and a state update, and any one of them can be the last place a chunk was seen. Each of those hops knows only that it handed the bytes onward. The reader is the single participant with no next hop, which is what makes its report a statement about arrival rather than about departure.

Server-sent events have carried this in the protocol since the beginning. An event may include an id: field; the browser's EventSource records the id of the last event it dispatched, and on automatic reconnect it sends that value in a Last-Event-ID request header. The mechanism predates streaming language models by more than a decade and was designed for exactly this — a long-lived feed whose reader occasionally falls off the network.

id: 311
event: delta
data: {"text":" no exceptions were noted."}

id: 312
event: delta
data: {"text":" Findings for Q1"}

Most clients here cannot use EventSource directly, because it only issues GET requests and a chat turn is a POST with a body, so the stream is read with fetch and a ReadableStream reader. That means writing the cursor tracking yourself, which turns out to be an advantage: EventSource updates its last-event-id when the event is dispatched, whereas your own reader can update it after the state update that puts the text on screen has committed.

let lastSeq = restoreCursor(generationId) ?? 0;

const res = await fetch(`/generations/${generationId}/stream`, {
  method: "POST",
  headers: { "Last-Event-ID": String(lastSeq) },
  body: JSON.stringify({ messages }),
});

for await (const ev of parseSSE(res.body!)) {
  if (ev.seq <= lastSeq) continue;      // idempotent apply
  chunks.set(ev.seq, ev.data.text);     // keyed, not concatenated
  render(chunks);
  lastSeq = ev.seq;
  persistCursor(generationId, lastSeq); // survives a reload
}

Two details in that loop do most of the work. The chunks go into a map keyed by sequence rather than being appended to a string, so applying the same chunk twice overwrites rather than duplicates and the render is a fold over the map in sequence order. And the cursor is persisted next to the generation id — in the URL, in sessionStorage, wherever page state lives — because a reconnect after a reload is a client with no chunks and a cursor of zero, which is a legitimate request for the whole log rather than a bug.

The reconnect, end to end#

The replay and the live tail have to be a single operation, or the seam between them leaks. If the handler runs a range query for everything after 311, then subscribes to new events, any chunk produced between the query returning and the subscription being established is dropped, and at 45 tokens per second with 80 ms coalescing that window is wide enough to lose a chunk on a regular basis. Redis Streams collapse the two: XREAD BLOCK 0 STREAMS gen:{id} 311-0 returns the backlog after that id and then blocks for new entries on the same cursor, so there is no gap to fall into. On a relational store you get the same guarantee by subscribing first, buffering whatever arrives, then running the range scan, then emitting the buffer with anything at or below the last emitted sequence discarded — which is the same idempotent filter the client applies, running on the server.

The order of those four steps is the whole of the guarantee, and it is easy to read as arbitrary. Subscribing first means the subscription is already live while the range scan runs, so a chunk produced during the scan is caught by the subscription instead of falling between the two operations. That is what creates an overlap where the naive version created a gap. An overlap is then a duplication problem, and the last step resolves it deterministically by discarding anything at or below the highest sequence already emitted — which is only possible because the numbers come from one writer and are therefore comparable. Reverse the first two steps and the overlap becomes a hole, and no amount of filtering afterwards can reconstruct what was never captured.

The load this adds is small enough to be worth stating. Take a desktop-heavy product, where disconnects are far rarer than the mobile figures quoted earlier in this topic: 40,000 generations a day, answers of about 900 tokens, and roughly 3% of clients dropping mid-generation. That is around 1,200 resume requests a day, each one a range scan over a mean of 55 rows, which is noise beside the traffic that produced it. The alternative is 1,200 users re-running a 900-token answer, so resumption saves a little over a million output tokens a day — about $16 at $15 per million — and saves each of those users a second twenty-second wait, which is the part they notice.

Each of those figures is one step from something already on the page. Three per cent of 40,000 generations is 1,200, and that is a count of resume requests a day rather than a rate. If every one of those readers would otherwise have re-run a 900-token answer, the tokens saved are 1,200 lots of 900, which is a little over a million a day, and a million output tokens at $15 per million is the $16. The twenty seconds is the same 900 tokens read through the decode rate instead of the price list: at 45 tokens per second, 900 tokens is twenty seconds of watching an answer reappear that had already been produced once. It is worth keeping both conversions in view, because $16 a day will never justify building this and twenty seconds multiplied by 1,200 people will.

The client's own view of "rendered" deserves one moment of scepticism. The cursor should advance at the latest point you can afford, because everything between the network and that point is a place where the chunk can be lost after being acknowledged. Advancing it when bytes arrive in the reader, before parsing and before the state update, means a crash during parse loses text that the server now believes was delivered. Advancing it after the state commit means the loss window is closed at the cost of a slightly staler cursor, which costs nothing except a few duplicate chunks on reconnect — and duplicates are free, given the keyed apply above.

Exactly-once is not on the table#

There is no ordering of ack and render that gives each chunk to the reader exactly once, and the reason is structural rather than a matter of engineering effort. The client can acknowledge before it renders, in which case a crash between the two loses a chunk the server considers delivered. It can render before it acknowledges, in which case a crash between the two causes that chunk to be replayed. Making them atomic would require the render and the server's record of the ack to commit together, across a network, which is the two-generals problem in its usual clothes — two parties on a link that can drop messages cannot both become certain that a given message arrived, however many acknowledgements they exchange. Every real system picks the second option and lives with duplicates.

That impossibility is worth one more turn, because the two-generals problem gets quoted as a name more often than it gets used as an argument. Suppose the client acknowledges chunk 412 and wants to be certain the server received the acknowledgement. The only evidence available is a message coming back, and that message can be lost — so certainty now requires acknowledging the acknowledgement, and the new message can be lost in turn. Each round eliminates one uncertain case and creates another, and no finite number of rounds terminates the sequence. The consequence for a resume endpoint is narrow and practical rather than philosophical: do not build anything whose correctness depends on both sides being certain at the same time. Build something that is correct whether or not the last message arrived, which is what applying a chunk into a position rather than appending it to an end achieves.

What makes that acceptable is that duplicates can be made invisible, and this is the property worth engineering for. Delivery is at-least-once, which means a chunk may reach the reader more than once and is guaranteed never to reach it fewer than once; application is idempotent, which means applying the same chunk a second time leaves the display in the state the first application produced; the composition behaves like exactly-once from the user's point of view without ever having achieved it on the wire. The keyed map is the whole trick, and it generalises: any chunk carrying a sequence number can be applied into a position rather than appended to an end, and once that is true, replaying an overlapping window is a no-op rather than a corruption. Ably's write-up on resumable token streaming names the four requirements as persistence, buffering, session tracking and ordering, and the fourth is the one that makes the third safe — without a per-generation order, a resumed session cannot tell a duplicate from new text.

The reverse failure has no such mitigation. A chunk that was never sent because both sides believed it had been is gone, and nothing downstream can detect it, which is why the write-before-flush asymmetry propagates all the way to the client: given the choice, arrange to duplicate.

The cold read, and a truncation delivered faithfully#

The cursor does not survive the client losing its state along with it. A hard reload with nothing persisted is a cold read, which is correct behaviour but means the user watches 400 chunks replay at whatever rate your replay loop runs. Replaying a backlog as fast as the socket accepts it produces a jarring flash of the whole answer; replaying it at the original pace makes a returning user wait through text that already exists. Emitting the backlog as a single coalesced payload and only streaming from the live tail onward is the usual compromise, and it means the resume endpoint has two output modes rather than one.

It also does nothing about the generation being wrong. The cursor is a delivery mechanism and a resumed stream faithfully delivers whatever the model produced, including the case where the model produced a truncated answer because it hit max_tokens. A user who reconnects to a complete-but-truncated generation sees a clean handoff to a bad answer, and the terminal record's finish_reason is the only thing that distinguishes them.

Both of those are contained problems, and the shape of the solution to the whole class is one number travelling in the unusual direction. Everything else in a streaming system moves from the provider toward the reader; the acknowledged sequence moves the other way, and it is the only thing in the design that knows what a human has actually seen. Give it to the server on reconnect, serve strictly what follows, apply chunks by position rather than by appending, and the reconnect becomes uninteresting — which is the correct outcome for a mechanism that fires on 3% of requests and must never be the reason one of them is wrong.

The one that is not contained is what happens on the server while a slow reader is being served. Every undelivered chunk between the provider and the client is sitting in your process, and the loop that writes to the subscriber without checking whether the write succeeded has no upper bound on how much of it accumulates.