Validation, repair loops, retry-on-invalid

Read these first

• Schema-constrained generation

What you will know

• Validate a response and retry with the validation error fed back

  Return structured, actionable validator feedback and preserve the original request context so the next attempt can correct the specific failure.

• Bound the repair loop so it cannot run forever

  Set an attempt and budget ceiling, then choose an explicit fallback such as rejection, escalation, or partial completion when the ceiling is reached.

• Distinguish a malformed response from a wrong-but-valid one

  Schema validation catches syntax and type violations, but business-rule checks are needed for values that parse successfully yet produce unsafe or incorrect outcomes.

• Log repair rate as a quality signal

  Track attempts against total responses and segment the rate by model, prompt version, schema, and input class so aggregate improvements do not conceal a failing slice.

• Diagnose whether a high repair rate comes from schema constraints, prompt or model behavior, or input distribution using segmented repair metrics

  A high rate is not proof that the schema is wrong; compare validator failure categories and affected cohorts before changing the contract or the model.

Chapters

1. 1

   Feeding the validator's own words back to the model

   A repair attempt that appends `the previous response was invalid, please try again` succeeds 31% of the time because it names nothing. Send the field path, the constraint that failed and the value that was rejected, alongside the original request context. Dropping the source document to save tokens raises the pass rate precisely because conformance is what a model can produce without the source, so the metric being celebrated is the one the design cannot fail.

   Read chapter
2. 2

   Malformed and wrong-but-valid need different machinery

   Response A does not parse; response B parses, validates, and files a misaligned button as P0. The schema proves shape, and only a business-rule check catches a refund that exceeds the per-customer cap. Retrying B is resampling an opinion until you like it, and logging the two together destroys the one signal that would have told you the severity definition is broken.

   Read chapter
3. 3

   A retry cap is only as good as its exhaustion path

   Three attempts means nothing until you decide what happens on the fourth. Rejection, escalation and partial completion are worked through as explicit outcomes, including why a refund exceeding a balance the model was never given must escalate rather than retry. The chapter also takes apart the popular fallback of returning HTTP 200 with partial fields greyed out: styling is not a contract, and no machine consumer can tell that response from a complete one.

   Read chapter
4. 4

   Repair rate is a quality signal, and an average

   Counting repair attempts against total responses turns a hidden cost into a tracked one. The catch is that a rate is a weighted average: a small tenant failing half the time barely moves a number dominated by your largest one, which is how a 4% dashboard sat still for two months while one customer got nothing usable. Segment by model, prompt version, schema and input class before trusting the aggregate.

   Read chapter
5. 5

   Reading a rising repair rate before loosening the schema

   Repair rate climbs from 4% to 12% over six weeks with no deploys — same prompt, same schema, same pinned model — so the thing that changed is the input. Before touching the contract, look at the breakdown of validator errors and which cohort they concentrate in: failures spread across fields you never edited exonerate Monday's change. Making three required fields optional improves nothing; it stops counting the failure and sends nulls to consumers that needed values.

   Read chapter