Getting JSON out

Read these first

• Anatomy of a prompt

What you will know

• Run the same extraction prompt through unconstrained text, JSON mode, and a schema-constrained response format; compare parse and validation failures.

  Provider guarantees differ: JSON mode may prevent syntax errors without enforcing required fields, types, or semantic constraints.

• Parse defensively against markdown fences and preamble text

  Truncation, concatenated objects, and trailing commentary create ambiguous boundaries that should be rejected rather than guessed through.

• Instrument parse, schema, refusal, and truncation failures separately, and reproduce each from a captured raw response.

  Separate counters expose whether prompt changes, provider behavior, token limits, or validation rules are driving incidents; redact sensitive payloads in stored samples.

• Validate parsed output against a versioned schema and exercise wrong types, missing fields, extra fields, and nulls.

  A JSON parser accepts structurally valid data that can still violate contracts or break downstream consumers during schema evolution.

• Implement bounded retry or repair with an idempotent fallback, then measure recovery rate, added latency, and token cost.

  Unbounded retries amplify spend and tail latency, while regex repair can silently alter values; recovery must preserve a clear failure path.

Chapters

1. 1

   Three ways to ask for JSON

   A plain instruction, the provider's JSON mode and a schema-constrained response format travel on nearly identical requests and establish three different contracts. JSON mode buys bytes that parse and says nothing about which keys appear, what type each holds, or whether an enum value is one you defined. Constrained decoding narrows the tokens the decoder may emit, which still leaves truncation at max_tokens, refusals, and a value that is the right type and the wrong fact.

   Read chapter
2. 2

   Extracting the JSON object

   Fenced blocks, a sentence of preamble and a trailing justification paragraph all put the object somewhere inside a larger response. The rule that survives them replaces first-brace-to-last-brace with a scan: find one complete balanced JSON value, require that it terminates, and require that nothing else JSON-shaped follows. Two candidates, or one that never closes, is a rejection you count rather than a substring you salvage.

   Read chapter
3. 3

   Validating the parsed object

   Validation against a versioned schema is a second gate, separate from parsing, and it is where wrong types, missing fields, unexpected extras and explicit nulls get caught. A nightly job that wrote 300 invoices with an amount of zero had a parse-failure counter reading zero throughout, because every response parsed cleanly and `total` arrived as the string `1,240.00`. Schema evolution belongs at the same gate, since a contract that changes breaks consumers a parser never notices.

   Read chapter
4. 4

   Detecting a model refusal

   Truncation is readable from the stop reason, a schema failure is whatever the validator rejected, and a provider content block arrives as an HTTP error — but a model refusal is a successful response carrying prose, byte for byte the same situation as a parse failure. The discriminators that separate them are a provider refusal field where one exists, the stop reason, a check on the shape of the response, and a cheap secondary classification, and the refusal is the class that must be routed off the retry path.

   Read chapter
5. 5

   Splitting the failure counter

   One counter reading `structured_output_failure: 4.1%` tells you to change something without telling you what. Split into parse, schema, refusal and truncation it points at the prompt, the validator, the safety policy or `max_tokens` before you touch anything, and marks the refusal as the class that must never be retried. Capturing raw responses for reproduction, and redacting customer PII before storing them, is part of the same work.

   Read chapter
6. 6

   Bounded repair loops

   Raising a repair cap from three attempts to six looks free because it touches 0.4% of traffic; those retries are concentrated in the slowest responses, so they land on the tail they are inflating. Measure recovery rate, added latency and token cost per recovered document, choose an idempotent fallback for exhaustion, and see why closing an unbalanced bracket turns a truncated record into a complete-looking one that is quietly short several line items.

   Read chapter