Feeding the validator's own words back to the model

A language model returns text, and the service around it needs an object with fields in it, so something in between has to check the text against the contract the application expects — the field names, their types, which of them are required, which values are allowed. That check is validation. A response that satisfies the contract can be handed downstream; a response that does not is invalid, and the service has nothing it can use.

What happens next is where the judgement lives, because an invalid response is a different kind of event from a failed database write. A model is a sampler, so the same request run twice produces different text, and a response that came back malformed once may come back well-formed on the next draw. Sending the request again, usually with something appended saying the last answer was rejected, is retry-on-invalid, and the surrounding cycle of validate, describe the failure, ask again, validate again is a repair loop. It is a different subject from retrying a failed HTTP call, which topic 2.11, on error handling, covers as classification, backoff and idempotency: there the call did not complete, and here it completed and returned something unusable. The repair loop exists because the alternative is discarding a request the model mostly got right, and because a large share of first-attempt failures are clerical rather than failures of comprehension — a response truncated mid-object, a number written with a European decimal separator, a category the model reworded on its way out.

Every structured-output stack has this layer somewhere, whether or not anyone named it. Instructor exposes it as max_retries, LangChain wraps it in an OutputFixingParser, and hand-rolled pipelines grow one during the first week that a downstream consumer chokes on a half-written object. It runs on a few percent of traffic and stays invisible while the dashboard is green, and how it is built decides whether that few percent becomes recovered requests or becomes latency, token spend and quietly wrong data. Here is the version almost everybody writes first, from an invoice-extraction service, and the whole of it fits on six lines:

resp = client.messages.create(**request)
try:
    return LineItems.model_validate_json(resp.content[0].text)
except ValidationError:
    request["messages"].append({
        "role": "user",
        "content": "The previous response was invalid. Please try again.",
    })
    resp = client.messages.create(**request)
    return LineItems.model_validate_json(resp.content[0].text)

model_validate_json parses the response as JSON (JavaScript Object Notation, the format almost every model is asked to emit structured data in) and confirms the result matches the LineItems class, a schema declared with Pydantic, the standard Python data-validation library. On that service, 31% of repairs passed on the second attempt.

There is a cheap way to find out what the appended sentence was worth. Delete it and re-run the same experiment, so that the second attempt is a straight resample of the identical request. On a replay of 5,000 logged failures that measured 29%. The instruction was worth two percentage points, which is inside the noise of the sampling temperature — the setting that governs how much randomness goes into each draw — so the honest description of that loop is that it rolled the dice twice and told itself a story about the second roll. Whatever fraction of the model's output distribution was already valid showed up again on the retry, and the sentence contributed nothing because it contained nothing the model could act on.

That comparison is the shape of every honest claim about a repair design, and it is worth stating as a procedure rather than as an anecdote. Any repair loop has a null version of itself — send the same request again, change nothing — and the null version is not idle. Temperature above zero means that at each step of generation the model scores every token it could emit next and one is sampled from those scores rather than the highest always being taken, so two runs of one request follow different paths and some share of those paths happen to end in text the schema accepts. That share is what the null version recovers, and it recovers it without knowing a first attempt ever happened. Any figure quoted for a repair design is therefore a sum of two quantities, the share the schema was going to get for free and the share the feedback earned, and only the second is attributable to the design. Each refinement of the message is worth what it adds over that floor, which means measuring the floor is the first experiment to run and it costs one replay with the appended sentence deleted.

What the validator already knew and threw away#

The except ValidationError branch catches an object that is far richer than the control flow suggests. Pydantic's ValidationError carries a list of individual failures, each with the path to the offending field, the constraint that was violated, and the input value that violated it. On one of the failing invoices it looked like this:

[
  {
    "type": "decimal_parsing",
    "loc": ["items", 3, "unit_price"],
    "msg": "Input should be a valid decimal",
    "input": "1.250,00"
  },
  {
    "type": "missing",
    "loc": ["items", 3, "currency"],
    "msg": "Field required"
  },
  {
    "type": "literal_error",
    "loc": ["status"],
    "msg": "Input should be 'draft', 'issued' or 'paid'",
    "input": "OUTSTANDING"
  }
]

Three failures, each one specific enough that a junior engineer could fix the response by hand in under a minute. The fourth line item used European decimal formatting, one field on that item was absent entirely, and the document-level status came back as a word outside the enum. None of that reached the model. It received "the previous response was invalid", which conveys that something is wrong somewhere in a 1,400-token JSON object and leaves the model to guess which part.

None of that list had to be computed specially. The validator produced it as a by-product of work it was already doing: to decide that the response was invalid it had to walk the object field by field, and at the instant it rejected unit_price it necessarily knew the path it had reached, the rule it was applying, and the value sitting in front of it. Those three facts are the whole substance of an error entry, which is why every validation library carries them and why exploiting them costs one method call — err.errors() — and zero extra tokens. The six-line version at the top of this chapter builds its repair message inside an except ValidationError clause that is holding that object and never opens it.

The comparison worth holding onto is a compiler. A type error that reads cannot assign str to int at line 214, column 9 is actionable because it names a location, a rule, and the offending value, and the fix follows from the message. A compiler that emitted compilation failed, try again would be useless in exactly the way this repair loop was useless, and nobody would defend it, because in that domain the expectation that errors are specific is old enough to be invisible.

Building the second request#

The repair message has to carry three things: what was originally asked, what came back, and what was wrong with it. The first of those is where most implementations quietly go astray, because it is tempting to build a fresh, minimal request containing only the broken JSON and the errors. That request asks the model to fix an object in isolation, and a model that cannot see the invoice can only make the object well-formed.

import json
from pydantic import ValidationError

def repair_messages(
    original: list[dict],
    invalid_text: str,
    err: ValidationError,
) -> list[dict]:
    return original + [
        {"role": "assistant", "content": invalid_text},
        {
            "role": "user",
            "content": (
                "Your previous response failed validation. "
                "Correct only the fields listed below and return the "
                "complete object again, using the invoice text above.\n\n"
                + json.dumps(err.errors(include_url=False), indent=2)
            ),
        },
    ]

original is the untouched message list, so the system prompt and the OCR text of the invoice — the characters an optical character recognition pass read off the scan — are still in place. The invalid output goes back as an assistant turn rather than being quoted inside a user message, which matches the shape of a conversation the model has seen a great deal of during training: it said something, and it is being corrected. The errors go in as a user turn, serialised rather than prose-summarised, because the JSON preserves the field paths exactly and prose tends to lose the array index that says which line item was the problem.

The instruction to return the complete object matters more than it looks. Asking for "just the corrected fields" produces a fragment that then has to be merged into the first response, and merging into a document that failed to parse is not always possible. Regenerating the whole object costs output tokens and gives you something you can validate with the same code path as the first attempt. This is the pattern the Instructor library calls a reask, and its documentation on retry mechanisms is worth reading if you want the version with max_retries and backoff already wired in.

What the numbers do when the feedback gets specific#

Replaying the same 5,000 logged failures through four repair message designs, holding the model, temperature and schema fixed, gives a shape worth memorising. Two columns are needed rather than one, because the obvious metric — did the second attempt validate — is exactly the metric the worst design wins on. The last column comes from a hand-audit of 200 repaired responses per row, where a reviewer compared every extracted field against the source invoice.

The first column is easier to compare as a failure rate than as a pass rate, because the interesting quantity is how much of the remaining failure each design deletes. With the appended sentence, a bit under seven responses in ten still fail. Showing the model its own broken output brings that to a bit under five in ten, which divides the failure rate by roughly one and a half. Adding the structured errors brings it to about one in ten, which divides what was left by a bit over four. Read as points of pass rate, the second step is worth about half as much again as the first; read as a factor on the failures, it is worth about three times as much, because a bit over four is roughly three times one and a half. The second reading is the one that transfers to another service with a different baseline, since a design that divides failures by four does so wherever it is deployed and a design that adds thirty-something points cannot add them to a service already passing at ninety.

The first three rows move the way you would expect, and the jump from 54% to 89% is the argument of this chapter: showing the model its own broken output helps, and telling it which constraint the output broke helps roughly three times as much again. The fourth row is where the interesting failure lives, and it is the reason the audit column exists at all. Dropping the source document holds validation pass at 88%, one point below the best design, and takes field agreement from 96% down to 71%. The validator confirms the repaired output conforms, and conformance is precisely what a model can produce without ever seeing the invoice. Given "unit_price": "1.250,00" and an error saying that is not a valid decimal, a model with no source available will emit 1250.00, or 1.25, or 125000, and all three satisfy the schema. Given Field required on currency, it will emit "EUR" because the surrounding text mentioned a German supplier, or "USD" because that is the modal currency in its training data. Given a status outside the enum, it will pick the enum member closest in meaning to OUTSTANDING, which is a reasonable guess and is wrong whenever the invoice actually said something else.

The belief that makes the fourth design attractive is not carelessness, and it is worth writing out in the form somebody would defend it in: the validator passed, so the output is correct. That is tempting because it holds almost everywhere else an engineer has met a validator. A schema check on a form submission, a foreign key constraint, a type checker — each of those runs against input produced by a program, and a program has no capacity to produce a well-typed falsehood on purpose, so conformance and correctness arrive together and the habit of treating them as one thing costs nothing. What breaks the habit here is where the input comes from. A model handed a schema and an error list, with the document removed, has been asked for a shape and has everything it needs to produce one. Conformance is precisely the property that survives deleting the evidence, and correctness is precisely the property that does not, which is why the two columns of that table separate on exactly the row where the evidence went missing.

The measurement that separates those two designs does not require new infrastructure. Take a sample of repaired responses — a few hundred is enough — and compare each extracted field against the source document, by hand or against a labelled set you already keep for regression testing. Score field-level agreement, not response-level pass. The prediction going in should be that the shape metrics are identical across the two designs and the value metrics separate, with the separation concentrated in exactly the fields the repair touched, because those are the fields the model regenerated blind.

That prediction is worth writing down before the sample is read, because the alternative reading of a 71% audit is that the model is generally unreliable and the whole extraction needs work. The two are distinguishable: a repair design that has lost the source damages the repaired fields specifically, and leaves the fields it never touched agreeing at whatever rate the first attempt achieved.

Which errors should have reached the loop at all#

Before tuning a repair message it is worth asking which of the three errors above had any business being there. Schema-constrained generation, covered in topic 5.2, makes literal_error on a closed enum structurally impossible — a decoder that can only emit tokens leading to draft, issued or paid will never produce OUTSTANDING — and it removes missing for required fields for the same reason. The decimal_parsing failure survives, because "1.250,00" is a perfectly well-formed string and no grammar over JSON syntax knows that this particular string is a number written in a different locale.

"Structurally impossible" is a strong enough claim to be worth the step under it. At each position the model scores every token in its vocabulary and the sampler draws from those scores. Constrained decoding inserts one operation between the two: it keeps a grammar derived from the schema, works out which tokens could still lead to a document that grammar accepts, and pushes the score of every other token far enough down that it cannot be drawn. Inside a status field constrained to three literals, the continuations available after the opening quote are the ones spelling draft, issued and paid and nothing else, so OUTSTANDING is not improbable, it is unreachable. missing closes by the same route in reverse: a required key is a place the grammar refuses to let the decoder leave, so the only paths out of the object are paths that emitted the key. decimal_parsing is where that machinery runs out, because the grammar's alphabet is JSON syntax and every candidate rendering of an amount is well-formed as something. Deciding that 1.250,00 is a German spelling of a number rather than a malformed one requires knowing which locale produced the document, which is a fact about the invoice and not about JSON, so there is no tightening of the syntax that makes it go away.

That sorts the traffic arriving at the repair loop into two groups. One group is errors the generation configuration could have prevented and did not, which are a signal to change the configuration rather than to build a better retry. The other is everything a grammar cannot express: value-level constraints, cross-field consistency, anything requiring knowledge of the source document. Turning constrained decoding on in that service dropped first-attempt failures from 4.3% to 1.6%, and the composition of the residual 1.6% shifted almost entirely to locale and formatting problems, which is the population the loop is actually good at.

Those two figures are worth converting into the form that decides where effort goes next. Moving from 4.3% to 1.6% takes out a bit under three of every five first-attempt failures, and it takes them out by never generating them, so they cost nothing in tokens, nothing in latency and nothing in the loop's own budget for going wrong. The residual is both smaller and harder, since what survives a grammar is by definition the part no grammar reaches. That is the right shape for a repair loop's workload and it is not the shape most of them have.

The corollary is that a repair loop sitting downstream of a weak generation configuration will look far more valuable than it is, because it is absorbing failures that a parameter change would have deleted. Repair costs a full extra round trip; in that service the first attempt ran at a p50 of 2.1 seconds — the median, the time half of requests come in under — and the repair added 2.8 seconds on top, so a request that went through the loop landed near 5 seconds and the endpoint's p99, the threshold the slowest one percent of requests sit above, was made almost entirely of repairs. Preventing a failure is worth several times what recovering from it is worth, and the loop exists for the failures prevention cannot reach.

The claim about the p99 is an arithmetic consequence rather than an observation, and it follows in three steps. A p99 is the boundary of the slowest one per cent of requests. Every repaired request takes well over twice as long as a clean one, which puts all of them above every clean request in the ordering. Repairs are several per cent of traffic, which is more than one per cent, so the slowest one per cent is drawn entirely from inside the repair population and the boundary sits somewhere in the middle of it. The consequence is that the tail latency of the endpoint stops being a fact about the model's speed and becomes a fact about the repair rate, which is a different number with a different owner and a different fix.

Where precise feedback stops helping#

Better error feedback raises conformance pressure, and conformance pressure applied to a field the document cannot answer produces invention rather than an error. If the schema requires purchase_order_number and the invoice has no purchase order on it, the first attempt may honestly omit the field and fail validation with Field required. Feed that back precisely and the second attempt returns "PO-000000", or "N/A", or a number lifted from the invoice's reference line. The loop converges, the dashboard shows a successful repair, and the pipeline has manufactured a purchase order.

The fix for that class of failure is in the schema rather than in the repair message — an optional field, or a sentinel the model is explicitly told to use when the document is silent — which means a repair loop is not a substitute for getting the contract right, and a rising repair rate is sometimes evidence about the contract rather than about the model. That thread is picked up in the last chapter of this topic.

A repair request is a bug report, and the same things make it useful here as anywhere else: the location, the rule, the offending value, and enough context to check the fix against reality. Strip the context and you still get a green validator, which is why the pass rate alone cannot tell you whether a repair loop is working.

What the loop has bought so far is a reliable path from a response that does not satisfy the schema to one that does, at a known token cost, with feedback the model can act on. What it has not bought is any opinion about whether the response is right. Every measurement in this chapter that touched correctness came from a human audit or a labelled set sitting outside the loop, because the validator inside the loop was only ever asked a question about shape. That distinction splits the failures arriving at your pipeline into two populations that need entirely different handling, and sending one of them through the machinery built here makes things worse rather than better.