A verifier stops the loop on a fact, a critic on an opinion

An evaluator-optimizer is a loop with two roles in it. A generator produces a candidate output. An evaluator examines that candidate and either accepts it or reports what is wrong with it. The generator produces another candidate with that report attached, round after round, until the evaluator accepts. It exists because a model's first attempt at a structured task is often close to usable without being usable, and a second attempt that has been told exactly which part failed is far likelier to land than a first attempt with a longer prompt would have been. What this topic adds is the objective signal in its title, which means filling the evaluator slot with a check that computes its verdict from the candidate instead of forming an opinion about it — a JSON Schema the output either conforms to or does not, arithmetic that either reconciles or does not, a compiler, a test suite. A check of that kind is a verifier.

The defining property of a verifier is reproducibility: running it twice on the same output returns the same verdict twice. That sounds like a small thing and it decides everything downstream, because the verdict is what a run's record says about why the system stopped. If the verdict is a sample, the record is a sample. This chapter takes a claims-extraction service — structured extraction of fields from insurance claim PDFs, built as one of these loops — and works through what the check inside it does and does not establish. The examples are insurance claims because the requirements are unusually easy to state out loud; nothing about the argument is specific to insurance.

Where the reproducibility comes from#

The property is worth deriving rather than asserting, because the derivation is what tells you when it fails.

Take the two candidate stopping conditions apart into their inputs. A reviewing model receives a prompt and the candidate, and returns text drawn from a probability distribution over token sequences. Drawing from a distribution is a sampling step, and a sampling step consumes randomness — so two draws with the same inputs are two draws, not one draw repeated. Even pinned to a fixed model name and a fixed prompt, the verdict is a realisation of a random variable, and asking it again is asking again rather than looking again.

A check written as code receives the parsed candidate and evaluates a fixed sequence of predicates over it. There is no sampling step in that sequence. Every predicate reads a field of the object, compares it against a literal or against another field of the same object, and returns a boolean. Run it twice on the same object and every predicate takes the same branch, because nothing it reads has changed. That is the whole mechanism: the verdict is a function of the object, so equal objects give equal verdicts.

The derivation names its own precondition, which is that every predicate reads only the object. A predicate that reads anything else inherits that thing's variability, and the reproducibility claim has to be narrowed to say so. The constraint list below has exactly one such line, and it is worth finding before the code is read as an example of the rule rather than as a specimen with a flaw in it.

The check that replaced the reviewing model#

Here is the check the claims team put in the evaluator slot. It is not clever, and its lack of cleverness is the point: every line of it is a statement about the parsed output that a second engineer can read and disagree with.

from decimal import Decimal
from datetime import date
import re

POLICY_RE = re.compile(r"^POL-\d{8}$")

def verify(claim: dict) -> list[str]:
    """Return a list of unmet constraints. Empty list means pass."""
    problems: list[str] = []

    for field in ("policy_number", "loss_date", "deductible",
                  "line_items", "total_claimed"):
        if claim.get(field) in (None, "", []):
            problems.append(f"missing required field: {field}")
    if problems:
        return problems

    if not POLICY_RE.match(claim["policy_number"]):
        problems.append("policy_number does not match POL-########")

    loss = date.fromisoformat(claim["loss_date"])
    if not (date(2015, 1, 1) <= loss <= date.today()):
        problems.append(f"loss_date out of range: {loss.isoformat()}")

    line_sum = sum(Decimal(str(li["amount"])) for li in claim["line_items"])
    if line_sum != Decimal(str(claim["total_claimed"])):
        problems.append(
            f"line items sum to {line_sum}, total_claimed is {claim['total_claimed']}"
        )

    if Decimal(str(claim["deductible"])) > Decimal(str(claim["total_claimed"])):
        problems.append("deductible exceeds total_claimed")

    return problems

Read the predicates against the precondition from the section above. The regex compares a field to a literal pattern. The sum compares one field against a total computed from another field. The deductible comparison compares two fields. All four of those read the claim and nothing else, so all four are reproducible in the strong sense — same object, same verdict, on any machine, in any year.

The loss-date range is the exception, and it is the exception because of date.today(). The lower bound is a literal date and never moves. The upper bound is read from the machine's clock at the moment of the call, so it moves every night at midnight. A claim whose loss_date sits a week in the future fails today and passes in a fortnight, with the same bytes and the same verifier version. That does not make the check worthless — rejecting future loss dates is a sensible rule — and it does mean the reproducibility claim has to carry a condition. What is reproducible is the verdict of the constraints that read only the object. A constraint that reads a clock, a network service, an environment variable or a random seed is reproducible only relative to that thing, and if the record of a run has to mean something to an auditor later, the value that constraint read has to be recorded alongside the verdict. This is the first instance of a pattern the whole topic runs on: the guarantee is exactly as wide as the thing the check actually reads.

The loop around the check is four lines of control flow. Generate a candidate, run verify, and if the list of problems is non-empty, feed those problems back into the next attempt as text and try again. Termination happens when the list is empty, which is a fact about a Python object rather than a report about anybody's satisfaction.

def extract(document: str, model, max_attempts: int = 3) -> dict:
    feedback = ""
    for attempt in range(1, max_attempts + 1):
        candidate = model.extract(document, feedback=feedback)
        problems = verify(candidate)
        if not problems:
            return candidate
        feedback = "The previous attempt failed these checks:\n" + "\n".join(problems)
    raise Exhausted(candidate, problems, attempts=max_attempts)

The trace line an auditor gets for a run that stopped this way names the verifier module and its version, the constraint list it evaluated, the digest of the input document, and the attempt number. She can take the stored output, run the same verifier at the same version herself, and get the same verdict. She does not need the model, the prompt, or a GPU, which matters more than it sounds: the artefact and the check outlive the model that produced them, so the record of why the system stopped remains meaningful after the vendor deprecates the endpoint that produced it.

Version is not decoration in that trace line. A constraint list is a living thing — the claims team amended theirs four times in the first quarter, adding a currency field, widening the loss-date window for latent-damage claims, and twice loosening a rule that turned out to reject legitimate documents. Every amendment changes what a stored pass means, so the version has to travel with the output and the amendments need the same review discipline as a schema migration. Teams that skip this end up with a database column called verified whose meaning varies by row and cannot be recovered, which is worse than not having the column, because a report that counts it is silently mixing four different claims.

The feedback the check produces is worth a moment on its own, because it is the other half of what makes this loop cheap. A critic returns prose that has to be interpreted, varies in wording between attempts, and sometimes contradicts what it said last time. The constraint list returns the same sentence for the same violation every time, names the field, and quotes the two numbers that disagree, which gives the next attempt something narrow to act on. That regularity follows from the same fact as the reproducibility: the message strings are formatted from the object, so the same violation on the same object renders the same characters. It is what lets you count failures by constraint, and the counts become the operational signal that later chapters lean on.

The two designs, side by side#

Both designs are loops with a generator and something that ends them, and it is worth being precise about which columns actually differ.

The useful way to read a table like this is not to score the columns and pick the winner but to look for a row where one column dominates the other, and then for the row that runs the other way. Column B is better on the stop, on auditability, on the shape of the feedback and on cost per attempt, and those four are not independent — every one of them is the reproducibility derivation cashed out in a different currency. Four apparently separate advantages produced by one property is a sign you are looking at one advantage described four times, and it should make you less impressed rather than more.

There is exactly one row where the direction reverses, and it is coverage. A critic can object to something nobody anticipated; a constraint list cannot, because a constraint list is a list. That is the row that makes B dangerous, and the mistake it invites is reading coverage as a level. Coverage in design B is not "high" or "low". It is an enumeration, and you wrote it. Design A's coverage is unbounded and unreliable, which is a genuinely different trade rather than a worse one.

The row on build cost carries a residual worth stating too. A prompt and a rubric is a one-off; domain constraints written as code are a standing maintenance obligation, and the four amendments in one quarter are what that obligation looks like in practice. The cost per attempt row is where B wins by orders of magnitude, and the cost per quarter row — which is not in the table — is where it gives some of that back.

What passes and is wrong#

The shallow reading of the table is that B gives you a guarantee where A gives you a judgement. What B guarantees is conformance to what it encodes, which is a narrower thing than it sounds when you say it quickly. Here is an output the claims extractor produced on the first attempt for a scanned two-page claim, and it passed verify with an empty problem list.

{
  "policy_number": "POL-40118256",
  "loss_date": "2025-03-14",
  "deductible": 500.00,
  "line_items": [
    { "description": "Water damage remediation", "amount": 4200.00 },
    { "description": "Flooring replacement",     "amount": 3150.00 },
    { "description": "Contents",                 "amount": 1130.00 }
  ],
  "total_claimed": 8480.00
}

The policy number on the document was POL-40118526. The model transposed two digits, and the transposition produced a string that satisfies ^POL-\d{8}$ as completely as the correct one does. The loss date on the document was 14 March in the claimant's narrative and 11 March on the adjuster's cover sheet, and the model picked one without noting the conflict. The third line item is not on the document at all; the model has seen enough claims to know that contents cover usually appears alongside water damage, and total_claimed is internally consistent because the model computed it from the line items it had invented. Every constraint the verifier encodes is satisfied. The loop stopped on the first attempt, and the design holds no opinion whatsoever about any of this, because there is nothing in verify that could have one.

The stumble to name here is not a slip in the arithmetic. It is a belief about what a defence covers, and it is tempting for a reason worth stating: a required field feels like a guarantee that the field will be right, when what it is is a guarantee that the field will be present. Those two are the same thing only when the generator has no way to produce a plausible value it did not read. A language model always has a way. Look at what stays fixed across the two cases. Run verify on the correct extraction of this document and it returns an empty list; run it on the fabricated one above and it returns an empty list. The verdict is constant while the property everybody cares about — did this come off the page — varies completely. A check whose output does not move when the dangerous property moves is not measuring that property, whatever it is named after.

Requiring a field does have a real effect, and it is the opposite of the intended one under a retry loop. The requirement is stated in writing to the generator on every attempt after the first, and the cheapest way to satisfy a statement that a field must be present is to fill it. This is the class of failure that verifier-first design calls a false accept, and the thing that changes when a verifier becomes a stopping condition rather than a gate is where the false accepts end up. A gate that runs once and rejects sends the bad output back or drops it. A loop that runs until the gate opens is looking for outputs the gate will accept, and it will find them, which is the subject of chapter 4.

The requirement no constraint list can hold#

Ask what an extraction feature actually promises and you find that most of the requirements are relational: they are claims about the correspondence between the output and the source document, not properties of the output on its own. verify can assert that policy_number has eight digits after a prefix. It cannot assert that those are the eight digits printed in the header of page one, because doing so requires reading page one, which is the task. The same holds for the completeness requirement — every line item on the document appears in line_items, and no others — and for the disambiguation requirement, which is that when the document states the loss date twice and inconsistently, the extractor uses the adjuster's figure and flags the discrepancy.

The clean way to see the boundary is to sort the feature's requirements into two piles. One pile holds properties of the output considered alone — the identifier has the right shape, the dates parse, the arithmetic is consistent, the enum value is one of six permitted strings — and every requirement in that pile can be written as code that takes the output as its only argument. The other pile holds properties that quantify over the input as well: this field came from that box on the page, these items are all of the items, this interpretation is the one a trained adjuster would have chosen.

That sort is the closing identity of this chapter, and it is worth stating with both halves. What it licenses is a fast and complete answer for the first pile: if a requirement can be written as a function of the output alone, a verifier can decide it, and the loop can stop on it with everything the first section derived intact. What it leaves behind is the second pile, and the residual there is not a gap that better engineering closes. Everything in the second pile requires an oracle — something that knows what the document says — and the model under test is the only oracle in the room. The two piles are not "checked" and "not yet checked". They are "decidable from the output" and "decidable only by reading the input again".

You can push the boundary outward by making the constraints less self-referential, and the pushing is instructive because of what survives it. Requiring that every extracted string appear verbatim somewhere in the document text kills the invented contents line and does nothing about the transposed digits, since POL-40118256 will not appear verbatim either, but a wrongly selected real string will. Requiring a bounding-box citation for each field and re-reading the pixels under it is stronger and costs a per-field pass of OCR, the optical character recognition step that turns scanned pixels back into text. What you cannot do is write a constraint that certifies the extraction is faithful without, somewhere in it, doing the extraction again by another route and comparing — which is a real design, and it costs what a second independent extractor costs. The boundary moves; it does not dissolve, because the only thing that decides a question about the input is something that reads the input.

Where a critic still belongs#

This is not the writer–critic loop with better tooling, and the distinction is worth holding onto because a widely-copied piece of reference material blurs it. The evaluator-optimizer example in the Claude Cookbook runs a generator and an evaluator with separate prompts, feeds the evaluator's feedback back into the generator, and loops until the evaluator emits PASS. Structurally that is this chapter's loop. The signal is a model's judgement wearing a keyword, so it inherits every property in column A of the table above while looking like column B — a keyword is a token the model chose, and choosing a token is the sampling step the derivation began with. Google's ADK gets the shape right in a different way, with LoopAgent making the exit condition an explicit, declared part of the contract rather than something buried in a prompt. Chapter 7 comes back to what you can and cannot claim when a threshold on a judge is the only signal available.

It is also not a replacement for the critic in cases where the critic was catching things you never anticipated, which is the coverage row of the table and the second pile of the closing identity, arriving together. The two compose: run the verifier as the loop's stopping condition, because it is cheap, deterministic and precise about what it wants, and run a critic on a sample of accepted outputs as an audit rather than a gate. The arithmetic of that arrangement is what makes it affordable — one critic call per sampled output instead of one per attempt — and the structural point is sharper than the arithmetic. A critic inside the loop is something the generator is optimising against. A critic outside it is something whose opinions get compared against outputs the loop already accepted, which is the only configuration in which those opinions are evidence about anything.

The writer–critic split from 11.6 buys you independent criticism, and it hands you a loop whose exit is an opinion. Swapping the critic for a check that is decidable — a JSON Schema, a set of constraints over the parsed object, a compiler, a test suite — changes what "done" means, from a reviewer was satisfied to this artefact has this property. Where every constraint reads only the artefact, the property is the same on Tuesday as on Friday, on your machine as in the auditor's, under model version 20250219 as under whatever replaces it. Where a constraint reads a clock or a service, the property is as stable as that thing is, and the record has to say so.

The claims team kept verify and added the audit sample, and the auditor's eleven-claim question now has a mechanical answer for every one of them. What the answer covers is the first pile and nothing else. The next question is harder and is the one that actually decides whether the loop helps: the constraint list is an objective, the loop is an optimiser, and what gets optimised is the list rather than the thing the list was standing in for.