Evaluator-optimizer with an objective signal

Read these first

• Writer–critic (generator–evaluator) with a *separate* critic
• Verifier-first design

What you will know

• Replace a subjective critic with an objective verifier

  A deterministic pass/fail signal makes termination auditable, but it can still encode the wrong objective or miss important quality defects.

• Loop until the verifier passes rather than until the critic is content

  Unbounded retries turn a brittle test into a latency and token-cost multiplier, so attempts and exhausted runs need explicit limits.

• Measure pass rate, attempts, latency, and token cost against a fixed baseline on a task set

  A higher verification rate is not an improvement if it comes from excessive retries or leaves end-to-end quality unchanged.

• Build a verifier for a real task and run it against both valid and adversarial outputs

  Boundary cases expose vacuous checks, parser exploits, and false positives that ordinary successful examples conceal.

• Write a decision record that compares evaluator-optimizer with a single pass and a critic loop under quality, latency, cost, and failure-recovery constraints

  The pattern is unjustified when no reliable objective signal exists or when retries cost more than the defects they prevent.

Chapters

1. 1

   A verifier stops the loop on a fact, a critic on an opinion

   Looping until a schema-and-constraint check passes gives you something a critic cannot: a termination condition anyone can audit and reproduce. It also gives you exactly as much as the check encodes, so a JSON output with every required field filled by a plausible fabrication passes and the design holds no opinion about it. This chapter draws that line before anything is built on top of it.

   Read chapter
2. 2

   The loop optimises the signal you gave it, not the goal

   A text-to-SQL feature that retries until the query parses and executes reaches a 99% verification rate while users report wrong numbers, because a query joining the wrong table runs perfectly. The loop drove toward runnable, which is what it was asked for. Learn to read a verifier as the objective it actually encodes, and price the one that would catch wrong answers: golden result sets, a reference query, a checked schema mapping.

   Read chapter
3. 3

   Your fixtures never included an output built to pass wrongly

   Passing every valid fixture and rejecting every malformed one says nothing about the case that matters: an output deliberately constructed to satisfy the check while being wrong. Build the adversarial set — vacuous checks any output satisfies, parser quirks that let malformed data through, false positives that reject good outputs — and see why a non-empty-result rule is a slightly less vacuous check rather than a fix.

   Read chapter
4. 4

   More attempts means more search for outputs that satisfy the check

   The pass-but-wrong outputs you constructed by hand are not hypothetical: the loop is a search, and what it searches for is whatever the verifier accepts. A code-fix agent looping on your test suite can satisfy it by special-casing or editing the failing test; an extractor fills a required field with a plausible fabrication because the field is required; a SQL generator rewrites until rows come back. Every extra attempt you permit buys more of this, which is why a class of inputs whose attempt count is climbing is a signal to read the outputs that passed rather than the ones that failed — and why the independent quality measure you ask for should be taken specifically on the highest-attempt runs.

   Read chapter
5. 5

   Unbounded retries turn the hardest inputs into an unbounded tail

   Median latency unchanged, p99 from six seconds to 71, and a handful of requests that never return: the loop is retrying the hardest inputs indefinitely, which is precisely why only the tail moved. Bound the attempts explicitly and decide what an exhausted run returns — the best failing attempt with a flag, a fallback path, or a handoff — because eventual passing is not a latency budget.

   Read chapter
6. 6

   Pass rate against your own stopping signal rises by construction

   Any loop that retries until a check passes drives that check's pass rate toward 100, so 71% becoming 96% reports how many attempts were permitted rather than whether the outputs got better. Ask instead for the attempts distribution, the added latency and token cost, and an independent quality measure. For a code-fix loop, what settles it is the single-pass rate against the looped rate on the same tasks.

   Read chapter
7. 7

   Without a real signal, a threshold on a judge is still a critic loop

   With no objective check available for summarisation, scoring each draft out of ten and looping until eight is a critic loop wearing a number: the writer converges on whatever phrasing that judge rewards, and switching providers changes the tastes rather than making them objective. There are still conditions under which you ship it: calibrate the judge against human labels on a sample so the threshold means something outside the loop, keep the judge's rubric out of the writer's prompt, bound attempts hard, and watch for drift toward judge-pleasing phrasing over time. Close by writing the decision record — this pattern against a single pass and a critic loop, on quality, latency, cost and recovery.

   Read chapter