The parser state that must survive a chunk boundary

Reading a JSON document incrementally means running a lexer — the layer that walks the characters one at a time and decides what role each one is playing, without yet assembling any values out of them — whose state outlives any one chunk. The reason it has to is that the boundaries are not yours. A chunk can cut inside é, which is two bytes in UTF-8: 0xC3 0xA9, between the \ and the n of an escape, between \u00 and e9, or between the opening quote of a string and its content. Applying JSON.parse to a chunk is applying it to a random substring of a document, and the successes you observe are the slices that happened to be self-contained.

The belief worth naming before any of the machinery is that a try/except around that parse is a defence. It is tempting because everywhere else it is one: a parse that fails on input somebody else produced is a genuine error, and swallowing it keeps a service alive. What makes it the wrong instrument here is what the exception is reporting. The bytes are not malformed, the document is not malformed, and nothing upstream has misbehaved — the exception says only that the question was asked too early, of a substring rather than of a document. Catching it converts a fact about timing into a fact about data and then throws that away, and because the chunks that raise are exactly the ones carrying a boundary, what you are left with is not a degraded version of the right answer. It is the right answer minus whichever pieces the network chose to split.

That is the whole mechanism behind a client that parses per chunk, swallows the failures, keeps the last successful parse, behaves beautifully in dev, and mangles text under load: the boundaries move, and which text survives moves with them. On a developer's laptop a 2,847-byte response comes back in a single read, because loopback has a 65,536-byte MTU — the largest payload the interface will carry in one packet — and no proxy in the path, so there is no boundary to land badly. Over the internet the same payload does not fit in one segment at all: a TCP segment on an ordinary path is capped near 1,460 bytes, so a response of that size needs at least two of them, and every re-chunking on the way — a load balancer, a TLS record boundary, a compression layer — multiplies the count from there. The developer's machine is not a faster version of production. It is a path with no boundaries in it where production has several, which is why the class of bug in this chapter is invisible in precisely the environment it gets written in.

Four things the parser cannot forget#

The first piece of state is whether you are currently inside a string. Outside a string, {, }, [, ] and , are structure; inside one they are ordinary characters, and a chunk that ends between the quotes of "note": "the total is {1200}" will look, to a structure-counting parser with no memory, like an object opening that never closes.

The second is whether the previous byte was a backslash. "path": "C:\\logs\\" ends with an escaped backslash followed by the real closing quote, and a parser that resumes at the wrong point either terminates the string early or runs off the end of the document looking for a quote that already went past. This is one bit of state and it is the one most often dropped, because it is invisible in every test payload that contains no escapes.

The third is the stack of containers still open — which of { and [ you have entered and in what order. You need it to know that the document has not closed, and, as the rest of this chapter shows, to close it artificially when you want to look inside.

The fourth is any partial byte sequence you have not finished decoding. That covers the multi-byte UTF-8 case, and also the \uXXXX form, where the four hex digits can be split across a boundary and, worse, where a character outside the Basic Multilingual Plane, meaning the first 65,536 code points that a single \uXXXX escape can address, is written as a surrogate pair of two such escapes — 😀 is \ud83d\ude00, twelve characters that must be reassembled in order before they mean anything.

Both halves of that fourth item repay counting out. One \u escape is six characters — the backslash, the u, and four hex digits — so a cut can land at any of five interior positions and leave the parser holding something that is neither a character yet nor an error. A code point above the Basic Multilingual Plane needs two such escapes, because one escape addresses sixteen bits of code point and there are not enough of those to cover the whole range, so the pair comes to twelve characters with eleven places to cut between them. A decoder that hands back whatever it is holding at each boundary will therefore emit a lone surrogate, which is a legal sixteen-bit value and not a character, and what happens to it next depends entirely on how strict the layer above is: a replacement glyph in one runtime, an exception in another, and in the unlucky case a string that travels through several services before anyone notices.

Every arrow in that diagram is a transition that can be interrupted by end of chunk. The parser's job is to be resumable at any point, which in practice means it is a loop over characters with a small struct beside it rather than a function that takes a document.

A parser you can hold in your head#

The decoding half of the problem has a ready answer in every runtime, and reaching for it saves you writing a UTF-8 state machine by hand. Python's codecs.getincrementaldecoder("utf-8")() holds trailing bytes internally and emits them once the character completes; JavaScript's new TextDecoder("utf-8") does the same when you call decode(chunk, { stream: true }), and forgetting that option is how a chunk ending after 0xC3 becomes a dropped chunk and a company name rendered as Caf. Neither one knows anything about JSON — they solve the fourth piece of state and none of the other three.

import codecs, json

class PartialJson:
    def __init__(self):
        self._decode = codecs.getincrementaldecoder("utf-8")()
        self.buffer = ""
        self.in_string = False
        self.escaped = False
        self.hex_left = 0
        self.stack = []          # list of "}" and "]" waiting to be closed

    def feed(self, chunk: bytes) -> None:
        text = self._decode.decode(chunk)     # holds partial characters back
        for ch in text:
            self.buffer += ch
            if self.hex_left:
                self.hex_left -= 1
            elif self.escaped:
                self.escaped = False
                if ch == "u":
                    self.hex_left = 4
            elif self.in_string:
                if ch == "\\":
                    self.escaped = True
                elif ch == '"':
                    self.in_string = False
            else:
                if ch == '"':
                    self.in_string = True
                elif ch == "{":
                    self.stack.append("}")
                elif ch == "[":
                    self.stack.append("]")
                elif ch in "}]" and self.stack:
                    self.stack.pop()

    @property
    def complete(self) -> bool:
        return bool(self.buffer) and not self.stack and not self.in_string

That is the whole lexer, and it is deliberately not a validator: it will happily accept {{{ and tell you three containers are open. Validation is the job of the schema check you run on the finished document, which topic 5.2, on constraining and validating model output against a schema, already put in your pipeline, and duplicating it here buys nothing while doubling the code that has to be right on every chunk.

Closing the document you were given half of#

State alone tells you where you are. To get a value out before the stream finishes, you need a snapshot: take the buffer as it stands, make it into legal JSON, and parse that. The stack gives you most of it, because closing every open container in reverse order turns a prefix into a document. The rest is a short list of endings that cannot simply be closed.

Reverse order is the whole trick, and it is worth saying why it works rather than observing that it does. JSON's containers nest, which means the opens you have seen form a stack: the most recently opened container is the one that has to close first, and self.stack already holds the closers in the order they were pushed. Read it backwards and you have the exact string that finishes the document. That is also the reason the parser keeps a stack rather than a counter. A counter would tell you how many closers to append and not which ones, and {"a":[1 needs ]} while [{"a":1 needs }] — same count, different string, and appending the wrong one produces a syntax error instead of a snapshot.

The last two rows are the ones that produce wrong answers rather than exceptions, and they are the reason an invoice whose real amount is 1200 renders as 12. A number at the end of the buffer is a prefix of a number, and 12 is a perfectly valid JSON number, so a repair that keeps it hands the caller a plausible, parseable, wrong figure. There is no marker in the syntax that tells you a number has ended — a number terminates at the delimiter that follows it, which is precisely the character you do not have yet. The same holds for tru, fals and nul, though those at least fail loudly if you keep them. Any repair worth using drops an unterminated number rather than exposing it.

    def snapshot(self):
        text = self.buffer
        if self.hex_left or self.escaped:
            text = text[:text.rindex("\\")]
        if self.in_string:
            text += '"'
        else:
            text = _drop_unterminated_scalar(text)   # numbers, tru/fals/nul, trailing , or :
        return json.loads(text + "".join(reversed(self.stack)))

Run that after every chunk and you get a sequence of increasingly complete objects. Feeding the invoice stream from the previous chapter through it in 12-byte slices yields snapshots in this order: {}, {"customer_name": "Mar"}, {"customer_name": "Martinez SA"}, {"customer_name": "Martinez SA"} — the amount suppressed while it is still a bare number — and finally the complete object with amount at 1200 once the closing brace arrives.

Two costs are worth knowing before you put snapshot() on the hot path. It re-parses the entire buffer each time it runs, so producing a snapshot per chunk is quadratic in the document size: a 4 KB result arriving in roughly 120 chunks re-parses an average of 2 KB per chunk, about 240 KB of parsing in total, which measures in single-digit milliseconds and is free. A 200 KB result arriving in 6,000 chunks re-parses about 600 MB, which is not free — on a Node process that lands around 900 ms of CPU spread across the stream, all of it stealing from the event loop. The fix is to decouple snapshot frequency from chunk frequency and take one every 50 to 100 ms, which is faster than a screen refresh can usefully show anyway.

"Quadratic" is worth deriving rather than accepting, because the word gets attached to a great many costs that are really linear with a large constant. snapshot() re-parses the buffer from its first byte, so the work it does on any given call is proportional to how much of the document has arrived by then. Over a whole stream the buffer grows from nothing to the full document at a roughly even rate, which makes the average call parse about half the final size. Multiply that average by the number of calls and you have the total. Half of 4 KB is 2 KB, and 2 KB across roughly 120 chunks is about 240 KB — invisible. Half of 200 KB across 6,000 chunks is about 600 MB, which is not. The chunks in both cases are about the same size, so doubling the document doubles both the average buffer and the number of calls, and it is that pair of doublings, not either one alone, that puts a square in the cost. A figure nobody would think to measure at one document size is a stalled event loop at another.

The second cost is that the naive self.buffer += ch in feed is a string concatenation per character. It is fine in CPython for these sizes and it is a trap in languages where strings are immutable and the runtime does not optimise the append; a list of chunks joined on demand, or a byte array, is the ordinary remedy.

Making the bad boundary happen on purpose#

The property that matters is that the parser gives the same final object no matter where the cuts fall, and that is directly testable without a network. Take a payload containing the things that break naive code — a multi-byte character, an escaped quote, a \u escape, a surrogate pair, a brace inside a string — and feed it through at every possible slice point:

PAYLOAD = json.dumps({
    "name": "Café Rialto",
    "note": "he said \"total {1200}\" \u2014 twice 😀",
    "path": "C:\\logs\\",
    "amount": 1200,
}).encode("utf-8")

def test_every_boundary():
    for cut in range(1, len(PAYLOAD)):
        p = PartialJson()
        p.feed(PAYLOAD[:cut])
        p.feed(PAYLOAD[cut:])
        assert p.complete
        assert p.snapshot() == json.loads(PAYLOAD)

That is 180-odd cases from four lines of test, and it fails immediately against chunk.decode("utf-8") per chunk, against a lexer that forgets escaped, and against one that counts braces without tracking strings. Add a second loop with two cuts if you want the escape-plus-multibyte interaction, and randomise slice sizes between 1 and 64 bytes if you want something that runs in CI as a fuzz case.

The case count comes straight from the payload — one cut position between each adjacent pair of bytes, so a payload of a couple of hundred bytes yields a couple of hundred cases and the loop enumerates them for you. That is the shape worth copying whenever a defect depends on where something was cut: the space of cut points is small, finite and enumerable, so there is no reason to sample it. A test that tries every position is not a fuzz test that got lucky. For that payload it is a proof, and the reason it finds defects nobody reported is that production is running the same enumeration slowly, one boundary per request, against whatever payload happens to be in flight.

You do not have to write this yourself. The Vercel AI SDK ships parsePartialJson, the npm package partial-json exposes the same idea with flags for which incomplete types you are willing to accept, and Python has json-repair and several partial_json_parser implementations. Reading one of them is worth an hour, because the differences between them are all in the table above — chiefly whether they hand you truncated numbers, which some do by default.

Three things the lexer leaves to a layer above#

It is not a fix for the protocol confusion of the previous chapter. Running this parser over a stream of concatenated NDJSON documents produces a snapshot of the first document and then a syntax error at the second {, because a document boundary is a real boundary that the stack does not model. Pick the layer first, then parse it.

It also does not make partial data safe to use. Every snapshot in that sequence is a legal object, and {"customer_name": "Mar"} is a legal object describing a customer who does not exist. The parser's contract is that whatever it hands you is a faithful reading of the bytes received so far, which is a claim about syntax and carries no claim about meaning.

And it does not detect truncation. If the upstream dies after {"customer_name": "Martinez SA", "amo, the parser will produce the same clean snapshot it produced a moment earlier, with the same absence of amount, and nothing in its state distinguishes "not sent yet" from "never coming". complete returning False is the only signal there is, and acting on it belongs to a layer above the lexer.

Four pieces of state and a repair table are the entire contract a boundary-aware parser offers, and the contract is narrow on purpose: given the bytes that have arrived, here is the object they describe. It is worth having because the alternative — parse each chunk, catch what throws, keep the last thing that worked — is not a weaker version of the same guarantee but a different behaviour that happens to look identical on a fast local connection and produces silently wrong fields everywhere else.

Which leaves the question the snapshots pose on every chunk. You now have a stream of legal objects, each one a little fuller than the last, and something has to decide which of their fields is allowed on a screen.