Malformed and wrong-but-valid need different machinery

Two failures can arrive in the same minute and share nothing but a timestamp. One is a response the calling service cannot read; the other is a response it reads perfectly and should not act on. The first is detected in microseconds by machinery that already exists. The second is detected by a phone call from somebody downstream, sometimes weeks later, and nothing in the request path knew anything was wrong.

This chapter is about the second one: what check catches it, why that check cannot be a stricter schema, and what happens to your metrics when both failures increment the same counter. The distinction matters most for the repair loop of the previous chapter, because sending the second kind of failure through it is worse than doing nothing.

Here is the first, out of a bug-triage service's logs:

{"severity": "P1", "component": "billing", "summary": "Invoice PDF fails to

The response was truncated at the max_tokens ceiling, json.loads raised on it, and the repair loop picked it up, sent the parser's position and message back, and got a complete object on the second attempt. That took 2.8 seconds and about 1,900 tokens, and it worked.

Here is the second:

{
  "severity": "P0",
  "component": "settings",
  "summary": "Save button is misaligned by approximately 3px on the account settings page",
  "affects_customers": "all",
  "requires_immediate_page": true
}

It parsed. It validated: severity is one of the four allowed literals — P0 through P3, the usual triage ladder on which P0 means wake somebody up tonight and P3 means put it in the backlog — component is a known string, affects_customers is one of three enum members, an enum being a fixed list of permitted values written into the contract, requires_immediate_page is a boolean, and no required field is missing. Every check in the pipeline passed and the record was written.

The two checks answer different questions#

A parser answers whether the bytes are a JSON document. A schema validator answers whether that document has the fields, types and enum members the contract requires. Both questions are about form, they are both decidable by looking only at the response, and a machine can answer them with certainty and no domain knowledge. That property is what makes them cheap and what makes them fast, and it is also exactly what bounds them.

Whether P0 is the right severity for a cosmetic alignment defect is a question about the world. Answering it requires a definition of P0 that lives in a runbook somewhere, and possibly a lookup — how many customers are actually on the settings page, whether the release is inside a change freeze. No amount of tightening the schema reaches it, because every candidate answer is inside the enum. The enum was designed so that every candidate answer would be inside it.

It is worth following the tightening instinct far enough to watch it stop, since the instinct is right about some rules and the boundary is not obvious from outside. A rule saying requires_immediate_page may only be true when severity is P0 or P1 is expressible, a validator enforces it, and it deletes a real class of nonsense. Now try to express P0 is proportionate to a three-pixel misalignment. That constraint has to compare the severity against the content of summary, which the schema can only describe as a string; against how many customers actually reach the settings page, which is nowhere in the response; and against a definition of P0 that exists as English in a runbook. A schema is a statement about which documents belong to a set, and membership has to be decidable by reading the document in front of you. Each of the three inputs this rule needs is somewhere the document is not. The chain is short and it terminates in the same place every time: a schema constrains form, form is decidable from the response alone, this constraint is not decidable from the response alone, and therefore no schema expresses it however strict anyone is willing to be.

The check that catches it is a business-rule check: ordinary code, written against the parsed object and whatever state the application has, that asks whether the values inside an already well-formed response are permitted here, for this customer, today. On the refund side of the same pipeline it reads like this:

from decimal import Decimal

class RuleViolation(Exception):
    def __init__(self, rule: str, detail: str, recoverable: bool):
        self.rule, self.detail, self.recoverable = rule, detail, recoverable

def check_refund(decision: RefundDecision, customer: Customer) -> None:
    if decision.amount > customer.remaining_balance:
        raise RuleViolation(
            rule="refund_exceeds_balance",
            detail=(
                f"approved {decision.amount} against remaining balance "
                f"{customer.remaining_balance} for {customer.id}"
            ),
            recoverable=False,
        )
    if decision.amount > Decimal("500.00") and not decision.manager_approval:
        raise RuleViolation(
            rule="refund_over_cap_without_approval",
            detail=f"approved {decision.amount} with no manager_approval flag",
            recoverable=False,
        )

There is nothing clever in that function, and that is what makes it worth having. It is deterministic, it runs in microseconds, it costs no tokens, and it has access to customer.remaining_balance, which the model was never given and could not have inferred. A pipeline whose only validator is a schema will happily emit {"approved": true, "amount": "1240.00"} against a customer with 90 euros left, because 1240.00 is a well-formed decimal and approved is a boolean, and both of those statements remain true regardless of the balance.

How much of this is flowing through#

The triage service handled about 30,000 reports a month, and its validity dashboard read 99.6% — the 0.4% were parse and schema failures, nearly all of which the repair loop recovered. That number is easy to read as "the pipeline is right 99.6% of the time", and reading it that way is the whole mistake.

Pulling 300 accepted responses at random and having two engineers assign severity independently gave a different picture. On 273 of them the model's severity matched the human consensus. On 18 it was one level off, which is inside the range where the humans also disagreed with each other. On 9 it was two or more levels off, and all 9 were inflations rather than deflations — cosmetic and single-tenant issues pushed to P0 or P1. Three percent of 30,000 is roughly 900 badly-mistriaged reports a month, and somewhere between a dozen and a hundred of those were paging someone at night.

Both of the numbers in that paragraph are worth deriving rather than accepting, because the step between them is where the argument actually lives. The 300 were pulled at random from responses the pipeline had already accepted, so the shares inside the sample are estimates of the shares inside the month's accepted traffic: nine bad ones in three hundred is three in a hundred, and three in a hundred of 30,000 is 900. The second figure, 97%, is the other two groups added together — the 273 that matched and the 18 that sat one level off, which are counted as acceptable because the two human reviewers also disagreed with each other at that distance, so there is no consensus for the model to have missed. A disagreement inside the range where the graders disagree is not evidence about the model. A disagreement of two levels or more is, because nothing in the human data produces one.

None of the 900 appeared anywhere in the error metrics, because every one of them was a valid document. The gap between 99.6% valid and 97% correct is not a rounding difference; it is two different measurements of two different things that happen to be reported in the same unit and on the same dashboard, which is why they get confused.

Part of why they get confused is that the habit is inherited from ordinary API work, where the two really do collapse into one. A request from a deterministic client that satisfies the schema is, in almost every case, a request that means what it says, because the client had no capacity to produce a well-formed lie. Validation earned its reputation as a sufficient gate in a world where the thing on the other end of the wire was a program. A model can produce a fluent, well-typed, entirely wrong document as easily as a correct one, and it costs the same number of tokens.

Where the boundary between the two actually sits#

Some rules can move into the schema and some cannot, and the line is whether the constraint is a constant at request-assembly time or a fact about state.

The rows that move into the schema move there because a constraint expressible before generation can be enforced during generation, which deletes the failure instead of catching it. The rows that cannot move share one feature: the model was not given the information the rule depends on. A refund of 1,240 euros is not a formatting mistake and it is not a mistake at all from the model's position, because nothing in its context said the balance was 90 euros. Handing that failure to a repair loop asks the model to correct an error it has no way to see.

There is a tempting way round the fourth row, and it deserves to be taken seriously because it works in a demonstration. Put the balance in the prompt. The model then holds the information the rule depends on, the constraint becomes checkable from inside the response after all, and the row moves up into the part of the table that the generation configuration can enforce. What that actually buys is a model that is usually right about an inequality, in place of a comparison that is always right about it. Three things break. The balance has to be fetched before the request is assembled, which is the lookup the move was supposed to avoid, and it is stale by the length of the generation. A comparison the model performs is a sample from a distribution rather than an evaluation, so it is wrong at some rate, and that rate is not zero for arithmetic on quantities written as text. And the failure is silent in the way this whole chapter is about: a model that concludes 1,240 fits inside a balance of 90 emits a well-formed approval, and the pipeline is back to having no check. Giving the model the balance is a reasonable way to improve its first draft. It is not a way to make the response self-checking, because the thing being checked would be the response.

The last row is the honest one. Nothing deterministic catches a disproportionate severity in the general case, and a rule like P0 requires more than one affected component is a heuristic that will misfire in both directions. Some judgements have no cheap oracle, and the correct engineering answer for those is sampling and review rather than a check in the request path.

Why response B must not enter the repair loop#

The temptation is to send both failures down the same path, because a retry is cheap and sometimes a re-roll produces a sensible severity. It does sometimes work, and that is what makes it durable as a habit.

What it is doing is resampling an opinion until the sample is one you like. The model, on that bug report with that prompt, put some fraction of its probability mass on P0 — say 22% — and that fraction is a property of the prompt, the severity definitions you wrote, and the report text. Retrying draws again from the same distribution, so with three attempts and a "reject P0 on cosmetic reports" filter, you will land on something acceptable about 99% of the time and you will have learned nothing. The 22% is still 22%. The next report of the same kind will do the same thing, and the retry will paper over it again.

The 99% is worth deriving, because the arithmetic is what makes the practice durable rather than what makes it wrong. Each attempt is an independent draw from the same distribution, since nothing about the prompt, the report or the model changed between them. Getting P0 on one draw has a probability of a little over a fifth. Getting it on all three requires that outcome three times running, and independent events multiply, so the chance of three P0s in a row is a fifth cubed — about one in a hundred. Ninety-nine times in a hundred, therefore, at least one of the three draws misses P0 and the filter finds something it will accept. The loop looks like it is working, and it looks that way for exactly the reason it is not: nothing in it moved the distribution, so the fraction of reports of this kind that come back inflated on the first attempt is untouched, and the only thing the retry changed is whether anyone finds out.

Meanwhile the actual information — that the severity definitions in the prompt do not distinguish affects all customers from is visible to all customers — never surfaces, because it is destroyed at exactly the point where it would have been visible. If both failures increment a counter called validation_failures, the graph shows a pipeline with a modest, steady error rate that the repair loop is handling well. Split them and the two series say completely different things:

REPAIRABLE.labels(kind="parse_error").inc()        # truncation, bad escaping
REPAIRABLE.labels(kind="schema_violation").inc()   # wrong type, missing field
RULE_VIOLATION.labels(rule=exc.rule).inc()         # never retried

A rising parse_error series points at generation configuration — output token limits, decoding settings, a prompt that invites prose around the JSON. A rising rule_violation series with rule="severity_disproportionate" points at the definitions you gave the model, and it is the only artefact in the system that would have told you the severity taxonomy was broken. Merged into one counter, that signal is arithmetically indistinguishable from a few more truncations.

"Arithmetically indistinguishable" is the exact claim and it is checkable rather than rhetorical. A counter is a sum, and a sum throws away the decomposition that produced it. Once validation_failures reads some value for the hour, no query against that series can tell a run of truncations from a smaller run of truncations plus a batch of rule violations, because both produce the same value and the series holds nothing else. Nor is the split recoverable afterwards, since the label that would have carried it was never written. That is the property that makes the decision a design decision rather than an operational one: the split has to exist before anybody needs it, and the same cheapness that makes a merged counter attractive makes its loss total rather than partial.

Routing the two populations#

The pipeline that keeps them apart has four exits rather than two:

The two arrows leaving the business-rule node are the substance of this chapter. A refund exceeding a balance stops and goes to a person, because the loop has no move that would improve it. A severity that looks disproportionate is recorded and passed on with a flag, because blocking on every judgement call turns an automated triage service into a queue with extra steps, and because the flags aggregate into the evidence you need to fix the prompt.

None of this is an evaluation#

A business-rule check is not an evaluation harness. Evals run offline against a labelled set and tell you how a prompt or model performs in aggregate; a business-rule check runs in the request path on one response and decides whether that response may proceed. They use different data and answer different questions, and a team that has one often assumes it has the other.

It is also not a second model grading the first. Model-as-judge has its uses, and it belongs in the row of the table where no deterministic oracle exists, but it is a different instrument with its own error rate, its own latency, and its own tendency to agree with whatever it is shown. The checks in this chapter are ordinary functions with ordinary test coverage, and their value comes from being boring: amount > balance is never wrong about itself.

The division to hold onto is that shape is decidable from the response and sense is not. Enforce everything decidable from the response as early as it can be enforced, and give everything that needs state or judgement its own code, its own counter and an exit that does not lead back to the model. A pipeline built that way still gets things wrong; it stops being unable to say so.

What remains open is the repair branch. The routing above sends malformed responses back for another attempt, and the REP --> R edge has no floor on it: a parse failure that repairs into another parse failure returns to the top of the diagram, and nothing drawn here says how many times that may happen or what becomes of a request that never satisfies the schema.