Streaming structured output

Read these first

• Schema-constrained generation
• Streaming

What you will know

• Parse a partially complete JSON object as it streams

  A parser must preserve lexical state across chunk boundaries, including strings, escapes, nesting, and incomplete literals.

• Decide which fields are safe to render before the object closes

  Render only values whose schema and semantics are complete; a syntactically present string can still be a misleading prefix.

• Handle a stream that ends mid-object

  Treat unexpected end of input as an incomplete result rather than accepting defaults or presenting a partial object as final.

• Reproduce a failure caused by applying `JSON.parse` to each network chunk, then patch it with a boundary-aware parser

  Transport chunks can split UTF-8 sequences, quoted strings, or escape sequences, so chunk boundaries carry no JSON meaning.

• Build a UI that fills in progressively without flickering

  Stable identity, monotonic updates, and explicit pending states prevent late-arriving fields from reordering or erasing content.

Chapters

1. 1

   Two protocols share the name streaming JSON

   A provider sending token deltas that concatenate into one document and a protocol sending one complete object per event place different obligations on your client. Calling `JSON.parse` on every SSE `data:` line is correct for the second and a bug for the first, where each line carries a fragment. That is the mechanism behind the client that parses per chunk, swallows the failures, keeps the last success, and mangles text only under production load.

   Read chapter
2. 2

   The parser state that must survive a chunk boundary

   A transport chunk is an arbitrary slice of bytes with no relationship to JSON structure, so it can end inside a string, between the halves of a `\u` escape, or partway through a multi-byte character. Parsing each chunk on its own is parsing a random substring, and the parses that succeed are whichever slices happened to be self-contained. Writing a boundary-aware parser means carrying four things from one chunk to the next: whether you are inside a string, whether the previous byte was a backslash, the stack of containers still open, and any partial sequence not yet decoded.

   Read chapter
3. 3

   A finished string is not yet a finished decision

   Two rules decide what may go on screen before the object closes. The first is syntactic: a value is renderable once its own token terminates — a string at its closing quote, an array at its closing bracket. The second is semantic and per-field: `recommended_action: refund` is a complete string and not a complete decision until `blocking_conditions` arrives, and a half-rendered legal disclaimer is a legal document you did not write.

   Read chapter
4. 4

   Six of nine fields is not a partial success

   When a stream dies 80% of the way through, your parser holds a partial object that looks salvageable and is not: nothing downstream can distinguish a field that was legitimately absent from one that never arrived, so persisting it manufactures data. The chapter also separates the three causes — `max_tokens`, client disconnect, upstream reset — because retrying a truncation reproduces it exactly and only the reset is worth a retry.

   Read chapter
5. 5

   Everything shown before the close is provisional

   The previous chapter settles what the write path does with a dead stream; this one settles what the screen does, because at that moment the user is already reading values the system has just decided are not results. Progressive rendering is only honest if every field shown before the object closes is marked provisional, there is one defined terminal transition to committed or withdrawn, and withdrawal is a state the UI was built to enter rather than a bug it falls into. Design that transition before the failure happens, or the interface has no way to take anything back.

   Read chapter
6. 6

   Stable keys, monotonic updates, and no flicker

   Items that jump position and briefly show the wrong label are keyed by array index, so every insertion reassigns identities; keying by a stable field fixes it. Beyond that, a progressive UI needs updates that only ever add information and explicit pending states, so a late-arriving field fills a slot that was already reserved instead of reordering or erasing what the user just read. Monotonic applies inside the provisional phase defined in the previous chapter: it governs how slots fill, not whether the whole render can later be withdrawn.

   Read chapter
7. 7

   Streaming costs the consumer that cannot spend it

   A backend worker that enqueues once the object closes gains a time-to-first-token improvement it has no way to spend, in exchange for an incremental parser, partial-state handling and a fresh class of mid-object failure. Decide who the stream is for before building it. When a UI and a worker consume the same object, the design that survives hands the worker the completed value and the UI the increments.

   Read chapter