Validating the parsed object

A parser answers one question: are these bytes a JSON document. A validator answers a second one: is this document the one your system agreed to accept. They have separate failure modes, and running only the first is the default state of most extraction code — json.loads returns a dict, a dict supports [], and nothing in the language forces the second question to be asked.

The gap between the two is where a well-formed response carrying the wrong data travels unremarked into a database. This chapter is about closing it: what a validator catches that a parser cannot, which coercions are safe to allow and which are guesses about a locale, and what happens when the contract itself changes underneath a producer nobody on your team controls.

The coercion that eats the error#

The characteristic shape of the problem is a helper that turns a model's value into a domain type and treats failure as a default:

from decimal import Decimal, InvalidOperation

def to_amount(value: object) -> Decimal:
    try:
        return Decimal(str(value))
    except InvalidOperation:
        return Decimal(0)

Two different inputs reach that except branch. Decimal("1,240.00") raises, because a comma is not a decimal separator in any locale Decimal accepts, and a model extracting from an invoice image will emit totals exactly as they appear on the page — thousands separator included. Decimal("None") also raises, which is what happens when the response carried an explicit "total": null and str() was applied to it before anything checked. Both paths land on Decimal(0), silently. A parse-failure counter reads zero throughout, because both responses parsed perfectly.

Following the null case one call at a time is what makes it obvious rather than surprising. The response carried "total": null, which json.loads turns into Python's None — correctly, since that is what a JSON null means. str(None) then produces the four characters None, and it produces them without complaint, because str is defined on everything and never fails. Decimal is handed those four characters, finds no numeral in them, and raises. The except catches the raise and returns zero. Nothing in that sequence violates its own contract, and what comes out the other end is a perfectly ordinary amount that a ledger will happily add up. The defect is entirely in the joins: str was applied to something that might not be a number, and a default was substituted for a value that had never been obtained.

A validator asks the second question instead of coercing past it. Here is the same contract expressed as a Pydantic model — a Python class whose type annotations are compiled into a runtime check, so that constructing it from a payload either returns a typed object or raises with the offending field path:

from decimal import Decimal
from typing import Literal
from pydantic import BaseModel, ConfigDict, Field

class LineItem(BaseModel):
    model_config = ConfigDict(extra="forbid")
    description: str
    quantity: Decimal
    unit_price: Decimal

class Invoice(BaseModel):
    model_config = ConfigDict(extra="forbid", strict=True)
    invoice_number: str = Field(min_length=1)
    issue_date: str = Field(pattern=r"^\d{4}-\d{2}-\d{2}$")
    total: Decimal = Field(ge=0)
    currency: Literal["USD", "EUR", "GBP"]
    status: Literal["paid", "unpaid", "partial"]
    line_items: list[LineItem]

Under that model, "1,240.00" is rejected and null is rejected, and neither of them reaches to_amount. Each such response becomes a counted validation failure with a field path and a reason attached, which is a worse-looking dashboard and a much better morning.

The strict=True in that config is a real decision rather than a default worth copying. Pydantic's lax mode, which is what you get without it, will coerce "1240.00" to Decimal("1240.00") quite happily. That is convenient, and it is a coercion you are performing on model output, which is the same category of thing to_amount was doing. The question to ask about each coercion is whether the two representations mean the same thing with certainty. A numeric string with no separators does; a string with a currency symbol or a thousands separator does not, because the moment you start stripping characters you are guessing at a locale.

Two adjacent rows in that table explain each other, and reading them together is more useful than memorising either. Lax mode accepts "1240.00" and rejects "1,240.00", and the reason is that lax coercion is not a general willingness to be helpful. It is one specific operation: running the string through the same numeric parser a numeric literal would go through. Digits and a single dot are what that parser accepts, a comma is not, so the coercion fails and the row rejects. Strict mode rejects both, for a different reason that happens to produce the same outcome on one of them — it never attempts the coercion at all, because the declared type is a decimal and what arrived was a string. The distinction shows up when you read an error rather than a table: decimal_parsing means something tried and failed, decimal_type means nothing tried. And the reason not to reach for a comma-stripping helper at this point is that swapping the roles of the comma and the dot is exactly what some locales do, which makes a stripper a locale assumption with no locale attached to it.

The last row is the one people argue about. An extra key is not, on its face, harmful — nothing downstream reads vat, so nothing downstream breaks. What the extra key tells you is that the model is answering a slightly different question from the one your schema describes, and that is information you want on the day it starts happening rather than three months later when someone asks why 40% of responses carry a field nobody defined. Where a prompt edit causes the model to start emitting subtotal alongside total and a later refactor reads the wrong one, extra="forbid" is the only thing in the stack that notices.

Explicit null and an absent key deserve separate treatment, and most schemas conflate them by declaring total: Decimal | None = None. That declaration accepts both, and by the time your code sees invoice.total is None it cannot tell whether the model said "this invoice has no total" or simply forgot the field. Pydantic exposes the difference through model_fields_set, and the cheap discipline is to make optionality explicit in the contract: a field that may legitimately be unknown is nullable and required, so the model has to say null on purpose, and an absent key is always a violation.

A contract that changes is a contract someone has to version#

A bad value against a fixed contract is half the work. The other half is what happens when the contract itself moves, and that failure looks nothing like a zero in a ledger.

Backend engineers already have the model for this from event streaming. An Avro or Protobuf schema in a registry is versioned, compatibility is checked before a producer is allowed to publish, and the rules about which changes are safe are written down: adding a field with a default is backward-compatible, removing a required field is not, changing a field's type is not, and renaming a field is a delete plus an add no matter how it looks in the diff. Structured model output has exactly the same problem with two differences that make it harder. There is no registry enforcing anything, and the producer is a model whose behaviour changes when the provider ships a new checkpoint — a checkpoint being the particular frozen set of weights served behind a model name, which vendors replace periodically without the name changing — which means the producer can change without anyone on your team writing a line of code.

One rule generates every row of that table, and holding the rule means you can classify a change the table does not list. A consumer accepts some set of documents; a producer emits some set; things work exactly while the producer's set fits inside the consumer's. Every row is one of those two sets moving. Widening what a consumer accepts is always safe for that consumer and never fixes anything on the producer's side. Narrowing it is what breaks. Adding a required field or a new enum member narrows nothing on the consumer but widens the producer, so documents start arriving outside the old consumer's set. Removing an enum member narrows the consumer while leaving the producer where it was, which is why that row reads safe for consumers and still spikes rejections — a model does not stop emitting a value because a schema file changed. And renaming a field is a delete plus an add because the two sets share no member at that key, whatever the diff looks like.

That last row has no technical signature at all. If total means "grand total including tax" in version 1 and someone rewrites the prompt so it means "amount due this period", every response validates, every row writes, and the only evidence is a discontinuity in a number that finance looks at monthly. The defence is a version tag carried with the contract and stored with the data:

SCHEMA_VERSION = 3

VALIDATORS: dict[int, type[BaseModel]] = {
    2: InvoiceV2,
    3: Invoice,      # v3 added `vat_total`, nullable
}

def validate(payload: dict, version: int = SCHEMA_VERSION) -> BaseModel:
    model = VALIDATORS.get(version)
    if model is None:
        raise ExtractionError("unknown_schema_version")
    return model.model_validate(payload)

Storing the version on the row is what makes a backfill possible later, a backfill being the job that goes back over rows already written and rewrites the ones a fix applies to. When someone discovers in March that version 2 was reading subtotals, the query that finds the affected rows is a WHERE schema_version = 2 rather than a date range that someone has to reconstruct from deploy logs.

Keeping two validators live also gives you a migration that does not require a flag day. Run the new one in shadow for a week — validating live responses alongside the current validator while only the current one decides what gets written — count how often it disagrees with the current one, and only then move the write path across. On one invoice pipeline, a version bump that added vat_total and tightened issue_date to an ISO pattern showed a 6.2% disagreement rate in shadow, all of it on issue_date, where the model was emitting 12/03/2024 on European invoices — which is a prompt fix that would otherwise have landed as an incident.

What that shadow run bought is worth being precise about, because "run it in shadow" is advice people follow without knowing what question it answers. The bump made two changes at once: a new nullable field and a tightened pattern. A disagreement rate on its own would have established only that something was off. That every disagreement carried the same field path is what isolates the cause to one of the two changes, with nobody having to reason in advance about which was riskier. The size of the rate then sets the price of proceeding: had the new validator gone straight onto the write path, that share of a day's documents would have failed at the moment of writing instead of in a log nothing depended on. A shadow run is a way of paying for the same information with a log line rather than an incident.

Where the validator stops being able to help#

A schema constrains form. It has nothing to say about whether the form is filled in correctly, and the audit in the chapter on the three request modes put a number on the gap: nine of a hundred and twenty perfectly valid invoices carried a total that was the subtotal from an earlier page. Every one of those passes strict=True, ge=0, and every pattern you could reasonably write, because a subtotal is a non-negative decimal and so is a grand total. Catching that class needs a check that knows something about invoices — that the line items should sum to the total within rounding, that the total should not be less than the largest line item, that a status of paid implies a payment record exists somewhere else. Those are business rules, they live downstream of validation, and they are the only thing standing between you and a well-typed wrong number.

Strictness also has a price that is easy to forget when you are writing the schema on a good day. Every constraint you add is a way for a response to be rejected, and a rejected response costs a retry, a dead-letter entry, or a human. When a provider shipped a new model checkpoint on a pipeline running extra="forbid", the validation-failure rate went from 0.3% to 2.1% overnight, entirely because the new checkpoint had started including a confidence field nobody asked for. The strict schema did its job — it noticed the producer had changed on a Tuesday — and it also rejected 1.8% of a day's traffic to tell you. Whether that trade is right depends on what the downstream write does, and it is a decision to make deliberately per field rather than by copying a config block.

The 1.8 there is one subtraction — the rate moved from 0.3% to 2.1% — and the share it names is documents that were being written before the checkpoint changed and were refused afterwards. Both halves of that are true at once, which is what makes the decision genuinely hard rather than merely unpleasant. Those documents were carrying a field nobody asked for. Every one of them was also carrying a total, an issue_date and a set of line items exactly as good as the week before. extra="forbid" is not weighing the value of that data against the risk of the unknown field, because it has no way to weigh either. It is a policy stating that an unrecognised change in the producer stops the pipeline — the right policy ahead of a payment run and the wrong one ahead of a reporting job, for reasons that have nothing to do with JSON.

A clean parse is a statement about bytes and a validator is the only thing that makes a statement about data. Installing it converts a class of failure that was arriving as a wrong number in a ledger into a class of failure that arrives as a counter with a field path on it, and versioning the contract is what keeps that counter meaningful across a producer nobody on your team controls.

There is now a second counter next to the parse counter, and between them they cover two of the four ways a response can fail. The other two were already visible in the three-mode comparison: six responses that stopped at max_tokens, and two where the model declined. The truncated ones are distinguishable from everything else by a field the provider hands you. The refusals are not, because a refusal arrives as a 200 with a body, and the body is prose.