Extracting the JSON object

A model asked for JSON often returns JSON with something else around it: a markdown code fence, a sentence of preamble, a paragraph of justification after the object. The bytes of the object are fine in every one of those cases, and the parse fails anyway, which makes extraction — deciding which part of the response is the document — a separate job from parsing it. Almost every service does that job with one line, and the line is a greedy regular expression that takes everything from the first { to the last }.

That rule is right about the responses it was written for and it cannot report when it is wrong, which is the property that matters. This chapter is about what it does on the three shapes of response it was not written for, and about the rule that handles all three by refusing rather than guessing.

Why it is tempting is worth saying plainly, because a reader who has just been told the rule is unsafe still has to explain to somebody why it survived fourteen months. It is tempting because it is correct on every response anyone has looked at. An object wrapped in a fence and an object with a sentence in front of it are both handled exactly right by first-brace-to-last-brace, and those two shapes account for nearly all the failures a service sees on the day the parser is written. The rule is not a guess that happened to work. It is a correct rule with an unstated precondition — that the response contains exactly one brace-delimited region — and the trouble begins on the day the prompt stops guaranteeing that, which is not a day anybody schedules.

What the greedy span is actually assuming#

Here is the whole of a parser of that kind. Eleven lines, no dependencies:

import re, json

def parse(raw: str) -> dict:
    match = re.search(r"\{.*\}", raw, re.DOTALL)
    if not match:
        raise ValueError("no object found")
    return json.loads(match.group(0))

The assumption inside \{.*\} is that the first brace in the response opens the document and the last brace in the response closes it. That holds while every response is an object, optionally wrapped in a fence, optionally preceded by a sentence containing no braces. It stops holding as soon as the prompt asks the model for anything alongside the object, and the three ways it stops are worth separating because they arrive at different times and with different volumes.

The first is a second brace in prose that talks about a field by name. Models that have just emitted {"status": "partial"} and are asked to explain themselves quite often write the {status} field was ambiguous or quote a fragment of the object back. The moment a brace appears after the object, the greedy span runs from the object's opening brace to a brace sitting in the middle of a sentence, and json.loads raises Expecting value on a substring that is object-plus-prose. This one is loud. It shows up in the parse-failure counter within an hour of a deploy.

The second is length. Any prose the model writes is generated from the same max_tokens budget as the object, since that ceiling covers everything the model writes in one response rather than the JSON alone, so a request that gains a justification paragraph needs perhaps sixty to ninety extra output tokens per document. Documents that previously finished at 1,180 tokens against a ceiling of 1,500 are fine; a forty-line-item invoice that previously finished at 1,460 now does not finish at all. What arrives is a response that stops mid-array, and the greedy span on a truncated object ends at the closing brace of the last complete line item, which produces an unbalanced substring and another parse error. Also loud, but arriving with a delay, because it only fires on the long tail of document sizes and correlates with month-end when the big invoices come in.

Those two example figures are picked to show where the boundary falls, and running them takes five seconds and turns a vague worry into a prediction. Add ninety tokens to 1,180 and there are still well over two hundred tokens of headroom under the 1,500 ceiling, so nothing changes for that document. Add sixty to 1,460 and the total is past the ceiling, so generation is cut off mid-array. Nothing about the model differs between the two cases. What differs is how close each document already sat to a fixed line — which is why the failure arrives sorted by document size rather than by anything a reviewer would have thought to vary, and why a staging set of ordinary invoices is on the safe side of that line by construction.

The third is ordering. Nothing in a request establishes that the prose comes second — an instruction saying after the JSON object is a request the model complies with most of the time. On inputs where the extraction is genuinely uncertain, the model has a strong prior toward reasoning first and concluding second, and it will sometimes lead with the paragraph. Combine that with the second failure and you get a response whose prose runs long, whose object begins at token 900 and is cut off at 1,500, and whose first { is the one in the {total} field was hard to read.

There is a quieter version of the third case, and it is the one that costs money rather than pages. On some documents the model declines outright, and a decline frequently comes with an illustration:

I can't reliably extract this invoice — the scan is cut off below the
line items. If you rescan it, the response would look like:

{"invoice_number": null, "issue_date": null, "total": null,
 "currency": null, "status": null, "line_items": []}

The greedy span finds exactly one balanced object here, json.loads accepts it, and if your schema permits nulls it validates. A row lands in the invoices table with every field empty and no counter anywhere fires, because from the parser's point of view this response was indistinguishable from a successful extraction of a blank invoice.

That case is the one to hold on to, because it is the only one here whose failure produces no error anywhere in the stack. The other three arrive as exceptions, which means somebody eventually sees a stack trace and a counter moves. This one arrives as a row. It is not that the check is weak — there is no check, because every component behaved correctly on its own terms. The model produced an illustration and said in plain English that it was one. The extractor found a balanced object, which is what it looks for. The parser parsed it, which is what it does. The validator saw fields it permits. Four correct steps in sequence turn an illustration into a record, and the shape is worth naming now because it recurs for the rest of this topic: the dangerous failures are the ones in which every stage succeeds.

One value, terminated, with nothing after it#

The rule that survives all three is a scan rather than a span. Look through the response for complete, balanced JSON values; require that exactly one of them exists; require that no value began and failed to close. Anything else is a rejection you count, not a substring you salvage.

Python's standard library already has the primitive for this. json.JSONDecoder().raw_decode(text, index) attempts to decode one value starting at index and returns the value together with the offset where it ended, which is precisely the "does this close, and where" question. Its exception carries a .pos, and whether that position is at the end of the string is what separates a brace in prose from an object that ran out of tokens.

Why those two failures land in different places is the step that makes the discriminator sound rather than lucky. A decoder consumes characters until it either completes a value or meets one that cannot continue the value it is building. Take a { sitting in the middle of a sentence: what follows it is a word, and a word is not a quoted key, so the decoder gives up within a character or two of where it started and its reported position is nowhere near the end of the response. Now take an object that max_tokens cut off: every character in it was a legal continuation, so the decoder consumes all of them happily, and the only thing that ever stops it is running out of input — which happens, by definition, at the last byte. The position of the failure is therefore not a heuristic about the response's contents. It is a direct consequence of which of the two things went wrong.

import json

_DECODER = json.JSONDecoder()

class ExtractionError(Exception):
    def __init__(self, reason: str) -> None:
        super().__init__(reason)
        self.reason = reason

def extract_one_json_value(text: str) -> object:
    """Return the single complete JSON value in `text`, or raise."""
    candidates: list[object] = []
    unterminated = False
    i, n = 0, len(text)

    while i < n:
        if text[i] not in "{[":
            i += 1
            continue
        try:
            value, end = _DECODER.raw_decode(text, i)
        except json.JSONDecodeError as exc:
            # Ran out of input mid-value: the response was cut short.
            if exc.pos >= n:
                unterminated = True
            i += 1
            continue
        candidates.append(value)
        i = end                      # skip nested values inside this one

    if unterminated:
        raise ExtractionError("truncated")
    if not candidates:
        raise ExtractionError("no_json")
    if len(candidates) > 1:
        raise ExtractionError("ambiguous")
    return candidates[0]

Three properties of that loop are doing the work. Advancing i to end after a successful decode means nested objects inside an accepted value are never counted as separate candidates, so a well-formed invoice with forty line items yields one candidate rather than forty-one. Checking exc.pos >= n separates the two ways a decode can fail: a brace in prose fails within a character or two of where it started, and a truncated object fails at the last byte of the response. And collecting into a list rather than returning on first success is what makes the two-candidate case detectable at all — the greedy span cannot notice that a second object exists, because it never had a concept of a first one ending.

Markdown fences need no special handling under this rule, which is a small but real payoff. Backticks are not JSON-shaped, so the scanner walks over ```json and finds the one object inside it; a response that is nothing but a fenced object and a response that is a bare object take the same path. The rejection reasons are worth carrying as separate strings, because truncated, no_json and ambiguous have three different causes and three different fixes, and chapter 5, on counting the failure classes apart, is about what you can conclude when they are counted separately.

The cost is negligible at the sizes involved. Each { in the response costs one decode attempt, most of which fail on the first character, and a 6 KB response with a few hundred brace characters resolves in well under a millisecond on CPython. The worst case is quadratic — a pathological response consisting of thousands of unclosed braces — so a length guard on the raw response before scanning is worth having, and it is the same guard you want anyway to stop a runaway generation from filling a log line.

Where that quadratic comes from is worth seeing, because it says what the length guard is actually guarding. Every { starts a decode attempt, and an attempt that never meets a character it cannot continue keeps scanning until it runs out of input. A response consisting of nothing but opening braces therefore has one attempt per brace and every attempt walks to the end, so the total work is the number of braces multiplied by the length of the response — which, when the braces are the response, is its length squared. Ordinary responses never come near that, because an attempt starting on a prose brace dies almost immediately and an attempt starting on the real object jumps i past everything it consumed. The pathological input is less likely to be malicious than to be a generation that has fallen into a loop, which is the same failure the log-line guard exists for.

What a one-line strip encodes#

Someone will suggest handling a trailing paragraph by stripping everything after the closing brace, usually as a one-line change: match up to the first } at column zero, or split on "}\n\n", or raw.rsplit("}", 1)[0] + "}". It fixes the failures you can see today, and its problem is that it encodes an assumption the model never agreed to — that the object comes first and the prose comes second. Nothing in the request enforces that order. The prompt asks for it, and asking is what the previous chapter, on the three request modes, established as the weakest of the three guarantees.

The more serious objection is that the strip cannot fail loudly. When the ordering flips, it runs on a response whose prose came first, silently removes part of it, and hands the parser whatever is left; when the response is truncated, it has nothing to remove and the parser fails with the same generic error it would have produced anyway. A rule whose incorrect application looks identical to its correct application is a rule you cannot instrument.

The scan is also worth being honest about, because it is a narrower tool than it first appears. It does not repair anything: a truncated response is still a lost response, and the scanner's contribution is that it says truncated instead of Expecting ',' delimiter: line 1 column 812. It does not tell you whether the object it found is the right one when two are present — it refuses, which is deliberate, and which means a prompt that reliably produces two objects will produce a 100% rejection rate until someone fixes the prompt. And on any endpoint where the schema-constrained mode from the previous chapter is available, none of this should be running in the hot path at all; the scan belongs on the fallback provider, on the older model you route to when the primary is unavailable, and on the day someone adds a line to the prompt without thinking about the parser.

What the scan gives you is a single object that is complete, balanced, and unaccompanied, together with a named reason on every response that is none of those things. That is the whole of its contract, and both halves matter: the object is what the next stage consumes, and the reason is what makes a bad week diagnosable instead of merely visible.

It is also a statement about bytes and nothing more. The fields inside the object the scan returns are still whatever the model chose to put there — which is where a total of "1,240.00" and a status of "overdue" are sitting right now, having passed every check this chapter describes.