Verifier-first design

Read these first

• Validation, repair loops, retry-on-invalid

What you will know

• Run a representative batch of outputs through a schema, compiler, test suite, or constraint checker and record false accepts, false rejects, and runtime.

  A checker that catches syntax but accepts semantically wrong results creates false confidence, while over-rejection drives unnecessary repair attempts.

• Instrument generation, verification, and retries separately, then compare their p50/p95 latency and per-success cost.

  The apparent cheapness of verification can disappear when tail latency, repeated generations, or expensive database-backed checks dominate.

• Build a generate-then-verify loop and measure the accuracy gain

  Use the same model, prompts, and evaluation set for the baseline so rejected outputs and repaired outputs are counted consistently.

• Rule a problem out because nothing can verify it

  Record which quality claims remain subjective and why sampling or human review is insufficient for the required reliability target.

• Point at four places in your own system where this same pattern applies

  The useful instances may be distributed across parsing, retrieval, tool execution, code generation, and post-write validation rather than sharing one implementation.

Chapters

1. 1

   The generate-verify asymmetry

   A stochastic generator becomes a dependable component only when something independent can reject its bad work for far less than the work cost to produce. That asymmetry, not the retry, is what does the job. Against the repair loops you already build around schema validation, the new move is choosing the check before committing to the problem, so verifiability becomes a selection criterion rather than a wrapper bolted on afterwards.

   Read chapter
2. 2

   A checker's accept set

   Running EXPLAIN on generated SQL removes syntax and unknown-column errors and leaves every semantically wrong query untouched, now shipping with a verifier's endorsement attached. Measuring a checker means putting a representative batch through it and recording false accepts, false rejects and runtime. Re-retrieving cited chunks catches invented quotations and still accepts a real quote with an unsupported conclusion stacked on top of it.

   Read chapter
3. 3

   Measuring a checker with independent labels

   A checker cannot report its own false accepts: the outputs it wrongly approved are indistinguishable, to it, from the ones it rightly approved. Filling in either error column means hand-labelling a sample of accepted and rejected outputs independently of the checker and scoring the checker against that set. The same labelled set is what turns a loop into a measured one: run the evaluation set with the verifier disabled and again with the full generate-then-verify loop, holding model, prompts and set constant, and attribute the delta to the loop instead of reporting a conformance rate.

   Read chapter
4. 4

   Pricing one accepted output

   A constraint check that applies a candidate write inside a rolled-back Postgres transaction costs 20ms at p50, 900ms at p95, and rejects 20% of first attempts. Instrument generation, verification and retries separately, then price one accepted output in generations, checks and wall-clock at the tail, where the slow checks land on the requests that already retried. Includes the point at which the loop is worse than a human review queue.

   Read chapter
5. 5

   The state a check binds

   That Postgres check is a verifier because it runs against the state that will actually accept the write: the same indexes, foreign keys and in-flight transactions that decide at commit time. Move it to a read replica to dodge the load on production and the property disappears — replication lag and concurrent writes mean a unique index that will reject at commit can accept at check time, turning a cheap deterministic checker into a false-accept generator. Any optimisation that changes what the check runs against is a change to its accept set, not an infrastructure detail.

   Read chapter
6. 6

   Model judges as verifiers

   Weigh EXPLAIN against sending the same query to a larger model for review. The deterministic check is reproducible, independent of the generator and cheap; the model judge covers more ground while being stochastic, correlated with the generator exactly where the generator is weakest, and expensive enough that — priced with the rejection-rate-times-tail model from two chapters back — a rejection loop can cost more than the generation it guards. The cases for taking the judge anyway are named here.

   Read chapter
7. 7

   Conformance rate and accuracy

   A service reporting 99.4% valid output is reporting schema conformance. Hand-label 200 accepted outputs, find 11 carrying the wrong value in a field, and the real accuracy of accepted work is about 94.5%. Raising the retry ceiling from three to five cannot touch those, because every one of them passed on the first attempt and the loop never saw it. This is what a loop with no measurable accuracy gain looks like from the inside.

   Read chapter
8. 8

   Checkers you already own

   Parsing, retrieval, tool execution, code generation and post-write validation each hold a generator paired with something that can reject its output cheaply, and they rarely share an implementation. Learning to spot the instances is the point, and to rank candidate features by how much checking machinery already exists: a generated migration applied to a scratch copy of the production schema is judged by constraints, foreign keys and a test suite that all already exist and cost nothing to reuse.

   Read chapter
9. 9

   Declining unverifiable work

   An agent that rewrites contract clauses while preserving legal meaning at 99.9% reliability has no cheap detector of failure other than the lawyer it was meant to replace, so the answer is no. A rubric scored by a second model checks structure, which was never the hard part, and hands reviewers a reason to read less carefully. Record which quality claims stay subjective, then propose the adjacent problem that is checkable.

   Read chapter