Without a real signal, a threshold on a judge is still a critic loop

Some tasks have no property of the output that a check could decide. Summarising a support ticket is one: there is no schema to conform to, no compiler, no test suite, no golden result set, and no arithmetic that either reconciles or does not. The obvious way to get an evaluator-optimizer loop working anyway is to put a strong model in the evaluator slot, ask it for a score out of ten against a rubric, and loop until the score reaches a threshold. The exit condition is now a numeric comparison evaluated in code, which looks exactly like the mechanical stop of chapter 1.

The subject here is what that design inherits from the objective version, what it does not, and what has to be attached to it before it can be shipped honestly — because for a large class of features it is the only design available. The example throughout is a support handoff summariser: when a ticket moves between teams, it summarises the thread so the receiving engineer does not have to read forty messages. The judge scores each draft out of ten, its score and critique go back to the writer, and the loop stops at eight.

What a score threshold is and is not#

Structurally this is the loop of chapter 1 with the verifier slot filled by a model. It generates, evaluates, feeds the evaluation back, and terminates on a condition that is checked mechanically. What is missing is the property that made the objective version worth the trouble, and the derivation from chapter 1 says exactly which one. A verifier's verdict is reproducible because every predicate reads the artefact and nothing else, and no sampling step appears anywhere in the chain. Here the comparison against eight is a pure function of the score, and the score is a token sequence drawn from a distribution. Putting a deterministic comparison on the end of a sampling step does not remove the sampling step; it only moves it one line up.

Read the middle column against the right-hand one before reading it against the left. On four of the six rows the judge behaves like the prose critic rather than like the verifier, and the two rows where it differs — "same verdict on re-run" and "rising pass rate means" — are both improvements in legibility rather than in kind. A sampled score is more repeatable than sampled prose because a number has fewer ways to vary, and a threshold gives you a ceiling you can point at. Neither of those changes what the loop is optimising.

The word "objective" is doing the damage in the original proposal, and it is worth being precise about which property it was supposed to name. A number is not objective because it is a number; it is objective when the thing it measures exists independently of the measurer. Eight out of ten from this judge is a fact about this judge.

The defence usually offered is that the judge is a different model from the writer, so the score is independent. That defence is worth taking apart rather than dismissing, because half of it is right. Independence of weights removes correlated blind spots — the writer's characteristic mistakes are not the judge's characteristic mistakes — which is a real benefit and is why the previous topic argued for a separate critic. What it does not do is convert a preference into a property. The writer is optimising against the judge's tastes either way, and given enough attempts it converges on whatever phrasing that particular judge rewards. A different provider has different tastes, so switching changes which tics appear rather than removing them. The Gradient-Guided Multi-Judge Prompt Optimization work is instructive here because it uses several judges as the signal specifically so that no single set of tastes can be satisfied cheaply, and even there the target is a prompt improved offline rather than a per-request stopping condition.

Termination and the audit trail, taken apart#

Two claims made for the design are half-true and the halves matter.

It terminates reliably, which is correct once there is a cap, and it is correct for a slightly worrying reason: the score is elastic. A summary that scores 6 gets a critique, the writer applies it, and the score moves, because the judge is disposed to reward visible responsiveness to its own feedback. So the loop almost always converges within three attempts even on tickets where no good summary exists.

Set that beside chapter 5's unsatisfiable inputs. An objective verifier on an input with an empty accept set runs forever, which is unpleasant and also informative — the non-termination is the signal that the input was impossible, and a no-progress detector can turn it into a diagnostic. A judge loop never gives you that signal, because there is no input for which no output scores eight. Reliable termination is therefore not evidence that a good output was found; it is evidence that the judge can always be satisfied, which is the same fact wearing better clothes.

The audit trail is real in that a stored score and critique explain why the loop stopped. It is weak in that nobody can reproduce the verdict, and the measurement is easy to make. The team re-scored the same 300 accepted summaries a week later, same judge model name, same prompt, temperature 0: 12% of scores moved by at least a point, and 3% crossed the threshold, meaning that on re-run those summaries would not have been accepted.

Temperature 0 is where people expect this to stop, and the reasons it does not are worth listing because each one is outside your control. Sampling at temperature 0 selects the highest-scoring token, but the scores themselves come from floating-point accumulation whose order depends on how the request was batched with other requests and on which hardware served it, so ties and near-ties resolve differently between calls. And providers update the checkpoint — the particular saved set of weights being served — behind a stable model alias, so the model name in your record does not pin the model. An auditor who wants to know why claim 4471 was accepted can re-run a constraint list on her own machine. She cannot re-run this, and the score she would get if she tried is not the score in the record.

Anchoring the threshold to something outside the loop#

Sometimes there is no objective signal and the feature is still worth having, which is the situation for summarisation, tone, helpfulness and most writing tasks. The design becomes defensible when the judge's threshold is anchored to something outside the loop.

Calibration is the anchor and it is a one-off measurement with a maintenance cost. Take 150 summaries spanning the score range, have two people who receive these handoffs rate each as acceptable or not against a written definition of acceptable, and measure how the judge's threshold behaves as a classifier of that human label.

                          human: acceptable    human: not
judge score >= 8                  97                23      precision 0.81
judge score <  8                  13                17      recall    0.88

That table repays being read slowly, because the two numbers on its right are computed over different populations and mixing them up is the commonest error in reading one. Precision is a fact about the accepted row: of everything the loop shipped, 97 were acceptable and 23 were not, so 97 divided by the row total gives 0.81. Recall is a fact about the acceptable column: of everything a human would have accepted, 97 got through and 13 were rejected, so 97 divided by the column total gives 0.88. One denominator is "what the loop delivered" and the other is "what was good", and no arithmetic converts one into the other without knowing both.

Translate each into a rate a person can act on and keep the population attached. Accepting at 8 ships roughly one unacceptable summary in five — one in five of the summaries it ships. Rejecting below 8 throws away about one acceptable summary in eight — one in eight of the acceptable summaries that exist. Those two are not comparable quantities, and quoting them side by side without their populations is how a threshold gets moved on the strength of an arithmetic that was never done.

Now the threshold means something outside the judge's own frame, the loop can be argued about in terms of the 19% rather than the 8, and a threshold of 9 can be priced against a threshold of 8 in the same units. The inter-rater agreement between the two humans, meaning the share of summaries on which they returned the same verdict, belongs in the record too. At 84% here, it caps how good the judge can look: a classifier cannot agree with a label more reliably than the label agrees with itself, so the ceiling on this judge's measured precision is set by the two raters and not by the model. That is a fact people forget to check before blaming the model.

Three things go alongside calibration, and each is a consequence of something already derived. Pin the judge to a dated model version and treat a version bump as a change requiring recalibration, because the 150 labels are what the threshold means and they were collected against one checkpoint. Keep the judge's rubric out of the writer's prompt, for the reason chapter 4 gave about passing a verifier's source to a generator: a writer that can read the scoring criteria optimises the criteria directly, and the convergence that took 2.4 attempts happens on the first one — the tell is that mean scores rise while the human-labelled acceptable rate does not. Bound the attempts hard, at two rather than the five you would allow an objective check, because the marginal attempt against a preference buys judge-pleasing phrasing at a much higher rate than it buys quality, and unlike an objective check there is no residual population of genuinely-fixable outputs waiting at attempt four.

Write the record while the numbers are in front of you#

The comparison that matters is between four designs, and it needs to be written down at the point where you have measured, because the argument recurs every time someone new joins and every time a model is upgraded.

Compare the third column against the second row by row and one of them dominates. The verifier loop is cheaper per delivered output, has a defined exhausted state, has an auditable stop, and its quality gain is bounded but demonstrable. The critic loop wins nothing except on the first row, and only in the specific case where the critic notices something no constraint could encode — the coverage row of chapter 1's table, which is the one thing an enumeration cannot buy. So the honest reading is that the critic loop is dominated wherever a decidable property exists, and survives only where one does not, which is the same territory the fourth column occupies. The fourth column's advantage over the second is a bounded attempt count and a threshold that can be priced; that is what the extra cost per delivered output is buying.

The record itself is short and its most useful section is the trip-wires, because a decision record without a revisit condition is a decision that will be re-argued from scratch.

decision: summarisation-handoff-loop
date: 2026-04-14
chosen: judge-threshold loop, cap 2, threshold 8, judge pinned to 20260219
rejected:
  - single pass: 44% of summaries rated unacceptable in the 150-item calibration
  - verifier loop: no decidable property of a summary corresponds to useful
  - critic loop: same signal, no threshold, unbounded attempts
accepted_costs:
  - ~19% of accepted summaries are unacceptable to a human reader
  - p99 latency 11 s, cost 2.6x single pass
  - stop condition is not reproducible; not usable as evidence of review
trip_wires:
  - judge mean score and human acceptable rate diverge by >8 points on the frozen set
  - mean summary length exceeds 160 words
  - judge checkpoint changes without recalibration
  - single-pass acceptable rate on the frozen set comes within 5 points of the looped rate
review: 2026-10-01 or on any trip-wire

The accepted_costs block is what separates a record from a rationalisation, because every line in it is a number somebody would rather not write down. The 19% is the precision figure restated as a shipping rate. The note that the stop condition is not usable as evidence of review is the audit-trail section restated as a constraint on what the feature may be claimed to do, and it is the line that matters if anybody later asks whether these summaries were checked.

What drift looks like before anyone notices#

Drift is the failure that arrives slowly for this class of loop, and the slowness is what makes it dangerous: no single week's outputs look wrong, and the aggregate a year later is a feature nobody would have approved.

The monitoring that catches it is the calibration set re-run. Take the frozen 150-item human-labelled set monthly and plot the judge's mean score against the human acceptable rate. A widening gap is the signal, and the gap can open from three unrelated causes: a checkpoint update behind the model alias, a prompt change on either side, or a writer that has found a new tic. All three produce the same chart, which is why the chart is a trigger to investigate rather than a diagnosis.

Between monthly runs, mean output length is a cheap proxy worth alerting on, and it is a proxy for a reason rather than by coincidence. Almost every way of pleasing a rubric-following judge adds material — a section heading, a restatement, a closing recommendation — and almost none removes any, so length is the observable that moves first when the writer is converging on the judge. This team's summaries went from 118 words at launch to 189 words at six weeks with the human rating flat, which is the whole story in one line. It is also, notably, past the trip-wire in the record above, which fires at 160 words: the measurement that revealed the drift had already breached the condition the team wrote to detect it, which is what happens when a trip-wire is written after the fact rather than wired to an alert.

The limit of all of this is the last trip-wire in the record, and it is the one that eventually retires the loop rather than tuning it. A loop of this kind is worth exactly the gap between the single-pass rate and the looped rate on a fixed set. That gap narrows as models improve, silently, because nothing in the system reports it — the mean score and the termination rate both stay flat while the base model gets better at first attempts. The equivalent line in the code-fix agent's record is the same sentence with the frozen task set substituted, and there it is the difference between 62% and 89% suite-green that a future model may close on its own.

Everything in this topic reduces to a short sequence you can run against a loop you already own. Write the accepting predicate as an English sentence ending in and nothing else, and see whether the sentence mentions the thing you promised users. Build twenty outputs designed to satisfy it while being wrong, and count how many the check catches. Log attempts with the first failed constraint, then read the outputs that passed at the top of that distribution rather than the ones that failed. Pick the cap from the marginal yield curve and give the exhausted path a type the caller cannot ignore.

The loop that survives all five is a small amount of code, and the reason it is trustworthy has almost nothing to do with the loop and almost everything to do with the check inside it. Where no such check exists, the loop is not a weaker version of the same thing. It is a different design, with a stopping condition that is a preference, and the work moves from building the check to anchoring the preference against people, pinning it, and writing down what you have agreed to accept.