Reading a rising repair rate before loosening the schema

The premise that makes this diagnosis possible is a fixed system. When the code, the prompt, the schema and the model are all pinned, the only remaining variable is what arrives at the endpoint, and a rate that moves is a statement about the input distribution rather than about quality. That is the same inference a database engineer makes when a query that ran in 40 milliseconds for a year starts taking 6 seconds with no schema migration and no new index: the plan changed because the statistics changed, and the statistics changed because the data did. Nobody responds to that by rewriting the query first. They look at what is in the table now that was not in it before.

The reflex in a model pipeline is different, and it is to loosen the schema — make the failing fields optional, and the failures stop being failures. This chapter is about the queries that come first: the mix-versus-rate reading, the error-type distribution that clears or convicts a recent change, and the mapping from each segmentation result to the thing it implicates. Only one of those results points at the schema.

The events already contain the answer#

The per-response event from the previous chapter carries input_class, error_types, tenant_id and page_count, and the first query worth running is the one that groups the rate by input class and shows both the rate and the share of volume:

select
  date_trunc('week', ts)                              as week,
  input_class,
  count(*)                                            as docs,
  round(100.0 * count(*) / sum(count(*)) over (partition by date_trunc('week', ts)), 1) as pct_of_volume,
  round(100.0 * avg((outcome <> 'accepted')::int), 1) as repair_rate
from extraction_result
where ts >= now() - interval '8 weeks'
group by 1, 2
order by 1, 2;

Two columns matter and only one of them usually gets looked at. On an extraction service whose repair incidence went from 4.0% to 11.8% over six weeks, the result was this:

Each figure in the last column is the weighted sum from the previous chapter, and it is worth rebuilding one row rather than taking the shape on trust. In the week of 25 March, digital PDFs were 92% of the volume failing at 2.0%, contributing a bit under two points, and scanned documents were 8% of the volume failing at 27%, contributing a bit over two. Those two add to the 4.0 in the table. Run the same sum on the last row, with the shares now 61% and 39% and both rates exactly where they started: the digital contribution slips a little because there is less digital traffic, the scanned contribution rises to almost five times what it was because there is almost five times as much scanned traffic, and the total is 11.8. Neither multiplication needed a rate to move. The only quantity that differs between the first row and the last is which share each rate is being multiplied by.

Neither cohort got worse. Digital PDFs failed at 2% throughout and scanned documents failed at 27% throughout, and the aggregate nearly tripled because the mix moved.

This is a different failure of aggregation from the one in the previous chapter. There, a small cohort's terrible rate was invisible because its weight was tiny. Here, every cohort's rate is perfectly stable and the aggregate still moves, because the weights themselves are time-varying. A rate over a mixed population is only interpretable as a quality signal if the mix is constant, and nobody's mix is constant, which is why the share column belongs on the panel next to the rate. Plotting them together makes a mix shift visible as two lines diverging rather than as one line rising for no stated reason.

The fix that follows from this diagnosis has nothing to do with the schema. Photographs of invoices lose the decimal separator in a total, cut off the right-hand column of a line-items table, and produce OCR text where 8 and B are interchangeable. That is a preprocessing and routing problem: run a better OCR pass on the scanned class, send that class to a model with stronger vision handling, or set a different generation configuration for it. Any of those attacks the 27%. Making tax_rate optional does not.

The Tuesday case, and clearing a suspect#

The same reasoning runs faster on a sharp jump than on a slow climb. Repair incidence goes from 3% to 11% on a Tuesday, and on Monday you tightened the schema — currency narrowed from a free string to the ISO-4217 enum, the international standard's list of three-letter currency codes, and tax_rate added as required. The obvious suspect is the obvious suspect, and the cheapest evidence that settles it is the distribution of validator error types, which is one group by away because error_types is already on every event.

Neither field you touched on Monday gained any share of the failures: currency held exactly where it was and tax_rate slipped a point, while issue_date went from under a third of the failures to nearly two thirds. Share is the right reading here rather than a count, because the two columns have different denominators — 1,140 failures on Monday against 4,220 on Tuesday — and the question being asked is which field the growth went into. A field driving an increase takes share from the others. A field merely carried along by one does not, and issue_date is the only field in the list that nobody had edited since v4. Cross-tabulating those failures by tenant puts 89% of them in one account, and pulling ten of that account's documents shows dates rendered as 03.11.24 where they used to arrive as 2024-11-03, because that customer's enterprise resource planning system — the ERP that runs their invoicing — had its export template changed on the Monday evening. The Monday deploy is exonerated in about fifteen minutes, and reverting it would have cost a day and fixed nothing.

One division makes that reading solid rather than merely plausible. Incidence went from 3% to 11%, a factor of a bit under four, and the failure counts went from 1,140 to 4,220, which is the same factor. The request volume was therefore flat across the two days, and none of the movement in the shares is an artefact of a denominator that grew. Had Tuesday simply been busier, every field's count would have risen together, every share would have stayed where it was, and the table would have looked like the Monday column twice. It does not.

The general form of the check: an error distribution that concentrates on fields you did not change, in requests that concentrate in one cohort, is evidence against your change. An error distribution that concentrates on the exact fields you tightened, spread evenly across every tenant and input class, is evidence for it. The second pattern is also the one that arrives with a clean explanation, so it is worth spending the query even when you are confident, because the confident answer here is wrong often enough to be expensive.

What each pattern in the segmentation means#

Segmenting is only useful if you know in advance what each result would imply, so it is worth writing the mapping down before running the queries rather than after seeing them.

Only one row in that table ends with the schema, and it is the row where the failures land on the fields the schema constrains, in every cohort, starting at the version bump. Six weeks of a straight climb with no version bump anywhere does not match it. The value of writing the mapping out is that it makes the schema hypothesis falsifiable, and a hypothesis nobody can falsify is the one that gets acted on by default because it is the only one anyone proposed.

Optionality moves the failure, it does not remove it#

Making a required field optional changes exactly one thing: where the absence of a value is detected. The model's behaviour is untouched, because the model is not conditioned on the JSON Schema's required array in any way that makes it try harder; it produces what it produces, and the validator either objects or does not.

Optional fields cost nothing is the sentence to argue with, and what is true in it should be granted first. Between two pieces of hand-written code an optional field really is close to free: the producer either has the value or does not, the consumer branches on it, and the same person usually writes both sides. The phrase fails here because the producer is a model, and a model does not experience the two cases differently. A required field asks it for a value it may not omit; an optional field asks it for a value it may omit. Neither instruction reaches the decoder at all unless the decoder is constrained by the schema, so the same request produces the same distribution of responses under both. What changes is only which of those responses the validator objects to — and therefore whether an absence is counted, owned and escalated, or accepted and turned into a null on its way out.

The repair rate falls to 4% on the Monday after a pull request like that merges, and none of the underlying documents became more extractable. What was a counted failure with an owner becomes an uncounted null travelling into whatever consumes the response. In that service the tax ledger summed tax_rate * net per invoice, and a null coerced to zero, so six weeks of merged-and-forgotten optionality would have produced a value-added-tax figure quietly short by the value of every photographed invoice — a number that surfaces at a quarterly filing rather than on a dashboard. The team would have been looking at a healthy graph the whole time.

The chain from a loosening merged in a pull request to a wrong tax filing has four links, and every one of them is individually reasonable. The schema stops requiring tax_rate, so a response omitting it validates. A validated response is accepted, and the missing field materialises as a null, because that is what an absent optional field becomes in every typed model anyone would write. The null is written to a column that is nullable, because the migration followed the schema and the schema said optional. And the ledger multiplies tax_rate by net, in a runtime where a null inside an arithmetic expression either raises in a batch job at four in the morning or evaluates as zero — and the behaviour that does not wake anybody is the one that survives. Not one of those four steps is a defect on its own terms. The defect is the first of them, and it is invisible from the first of them, because everything downstream is doing precisely what its own contract says.

This is the same failure as chapter 3's result.total if result else 0, arriving through the schema rather than through the exhaustion path. Both replace an explicit error with a plausible default, and both are attractive for the same reason: the graph improves immediately and the consequence lands somewhere the person making the change does not own. The test that catches it before merge is to ask which consumers read the field, and what each of them does with a null. If any answer is "treats it as zero", "skips the row silently", or "we would have to check", the field is not optional in any sense the business recognises.

When the contract really is the problem#

Loosening a schema is sometimes correct, and the cases have a shape. A field the source document genuinely does not always contain — a purchase order number on invoices from suppliers who do not use them — is not a model failure at all; the schema is asserting something false about the world, and the extraction is right to omit it. The honest fix there is still not a bare optional field, because a null then means both absent from the document and the model did not find it, and those need different handling downstream. A three-state representation costs one line and preserves the distinction:

class Field[T](BaseModel):
    value: T | None
    status: Literal["extracted", "absent_from_document", "unreadable"]

With that, the model can report an honest absence and satisfy the contract, the validator still fails on a field the model simply skipped, and the ledger can distinguish an invoice with no VAT from an invoice whose VAT could not be read. The repair rate falls for the same superficial reason as the naive change, and this time the information survives.

The other legitimate case is a constraint that is genuinely tighter than the domain. A regular expression constraining invoice_number, written from 200 German invoices, will reject the Portuguese ones, and no amount of repair will help, because the model is reading the number correctly and the pattern is wrong. That failure looks exactly like the sixth row of the table above — spread across cohorts, concentrated on the field you constrained — and it is the pattern that licenses changing the contract.

What the segmentation leaves to a human#

None of this identifies a cause. It identifies a cohort and a field, which narrows a search from the entire system to a handful of documents you can open and read, and the reading is still done by a person. The queries here told you that failures concentrated in one tenant's issue_date; they did not say that an ERP template had changed, and nothing in the data could have. Expect the instrumentation to buy an hour of triage rather than an answer.

Segmentation is also blind in the direction the previous chapter's audit covers. Every diagnosis here is built on validator output, so it can only see failures the validator produces, and the population of responses that pass every check while being wrong is exactly as invisible in this analysis as it was on the aggregate panel. A mix shift toward photographed invoices raises the repair rate, and it almost certainly also raises the rate of confidently misread totals that parse cleanly, which shows up in a labelled audit and nowhere else.

The order of operations is what this chapter is really asking for: read the mix, read the error distribution, read the cohort, and only then consider the contract. A schema change is the one repair that removes the evidence along with the symptom, so it is the one that has to be argued for rather than reached for.

The loop these five chapters have described is now complete enough to build: structured validator feedback with the original context, a business-rule check separating shape from sense, a bounded budget with a decided outcome at the ceiling, and per-response events that let you read the rate rather than just watch it. The order to build them in is roughly the reverse of the order they were taught. Instrument first, because you cannot tune a loop you cannot see, and the events are the one part that cannot be added retroactively. Then bound it, because an unbounded loop is an outage waiting for the right input. Then make the feedback specific, and only then start moving constraints into the schema and the decoder, where the failures you have been repairing stop being generated at all.