Evaluating a critique pass

A critique pass is cheap to add and expensive to evaluate honestly. Eleven lines appended to a prompt produce a number within an afternoon, and the number arrives in a pull request description with a latency figure and a token multiplier beside it, which is enough material for a decision-shaped conversation and not enough for a decision. What is missing is not more precision on the same measurement. It is a different measurement.

This chapter uses a SQL-generation feature as its worked example, and the measurements below all come from one frozen 200-case eval — frozen meaning a fixed set of cases that nobody adds to between runs, so two numbers measured against it are measuring the same thing — scored on execution match, the share of generated queries that return the same rows as the reference query.

Two cases is inside the noise, twice over#

Start with the claim under test: a second turn saying review the query above and correct any mistakes before returning it moved execution-match accuracy from 71% to 72%, took p95 latency — the figure that the slowest 5% of requests exceed — from 3.4s to 6.9s, and raised output tokens per request 2.4x, from 310 to 744. One point on 200 cases is two cases. There are two separate sources of variance that swamp two cases, and it helps to keep them apart, because they have different fixes.

The first is sampling error from the size of the eval set. With 200 cases at around 71%, the standard error on a single accuracy estimate — how far the measured figure typically lands from the true one, purely because the 200 cases are a sample — is sqrt(0.71 * 0.29 / 200), which is 3.2 percentage points. A 95% interval on the baseline alone spans roughly 65% to 77%. That interval is the wrong tool for a paired comparison — running both configurations on the same 200 cases is much more sensitive than comparing two independent samples — but it does establish the scale, and the scale is that a one-point difference is a rounding artefact.

The second is generation variance. The pipeline ran at temperature 0.2, and the eval was scored once. Re-running the unchanged baseline five times on the same 200 cases gave 69.5%, 71.0%, 71.0%, 72.5% and 73.0% — mean 71.4, standard deviation 1.3 points, spread 3.5 points end to end. The critique configuration's 72% falls comfortably inside the range the baseline produces by itself. Before any comparison is meaningful, the eval has to be run enough times per configuration to know how much the thing moves when nothing has changed, and three runs is usually enough to see it.

Why a paired design is the more sensitive arrangement is worth stating rather than asserting, since the interval quoted earlier is about to be set aside. When two configurations are measured on two different samples, part of the gap between them is simply which questions each sample happened to ask; a set with more hard joins in it scores lower whatever produced the answers. Running both configurations over the same frozen 200 removes that source of variation completely, because every case is put to both. Nothing is left to explain except the cases that came back different, and those are far fewer than 200. So the interval running from roughly 65% to 77% establishes the scale of the problem and settles nothing: it is sized for a comparison nobody here is making.

The right statistic for a paired eval is built from the cases that changed answers. Lay out the 2x2:

Read the row and column totals before anything else, because both headline percentages are sitting inside this table. The baseline was correct across the top row: a hundred and thirty plus twelve, which is a hundred and forty-two of the 200 cases, and that is the 71% quoted above. The critique pass was correct down the left column: a hundred and thirty plus fourteen, a hundred and forty-four of the same 200, which is the 72%. The one point separating the two headlines is the difference between fourteen and twelve, laid over a denominator of 200. An aggregate accuracy delta is not a summary of what the change did. It is two of these four cells subtracted from each other, with the other two discarded and the split between the two that mattered discarded along with them.

The 130 and the 44 carry no information about the change; both configurations did the same thing. The evidence is entirely in the 26 discordant cases, split 14 fixed against 12 broken. Under the null hypothesis that the critique turn is neither helping nor hurting, each discordant case is a coin flip, and seeing 14 or more heads out of 26 has a two-sided probability of about 0.85. There is no signal here whatsoever.

Throwing away most of an eval set feels wrong, so it is worth being exact about why those two cells go. A case both configurations got right, and a case both configurations got wrong, came out the same way whether or not the critique pass ran. Nothing about such a case distinguishes the two configurations, so it can neither support nor undermine a claim that one is better than the other. The 130 and the 44 come to a hundred and seventy-four of the 200, and that is the portion of the eval which was never testing this change at all. What remains is the 26 that flipped.

The test on those 26 is worth doing on paper once. Start from the assumption that the pass neither helps nor hurts. Under that assumption a case that flipped was equally likely to flip in either direction, so the discordant cases behave like a run of fair coin tosses and the question becomes how surprising fourteen heads out of 26 is. Twenty-six tosses land on an even thirteen-and-thirteen split more often than on any other outcome, and fourteen is one step away from it. The observed result is therefore just about exactly what the no-effect assumption predicts, and the probability of landing at least this far from even, counting both directions, comes out near 0.85. A p-value that close to one is not weak evidence of an effect. It is the measurement reporting that it saw nothing at all.

from math import comb

def mcnemar_p(fixed: int, broken: int) -> float:
    """Two-sided exact test on discordant pairs."""
    n, k = fixed + broken, min(fixed, broken)
    tail = sum(comb(n, i) for i in range(0, k + 1)) / 2 ** n
    return min(1.0, 2 * tail)

print(mcnemar_p(14, 12))   # 0.8450...
print(mcnemar_p(26, 0))    # 2.98e-08  -- what a real 26-case effect looks like

The contrast in that second call is the reason to run the test at all. If the critique turn had fixed 26 cases and broken none, the same headline — plus thirteen points, not one — would have been unambiguous on the same 200-case eval. The size of the eval is rarely the binding constraint; the ratio of fixed to broken is.

The reason the eval size drops out is visible in what the test consumes. It never looks at 200. It looks at the number of discordant cases and the split within them, so adding cases to an eval helps only to the extent that the extra cases flip. A change that alters a quarter of the answers produces twenty-five flips per hundred cases added; a change that alters one answer in fifty produces two. Two evals of identical size can therefore have wildly different ability to detect the same effect, and the quantity that decides it is a property of the change rather than of the set.

That relationship is worth turning around into a planning number before you build the eval. To reach significance at the conventional threshold you need roughly a 3:1 split among the discordant cases at around 20 discordant pairs, or about 2:1 at 40 pairs, or about 1.5:1 at 100. If a change is expected to be a modest improvement — fixing three cases for every two it breaks — then detecting it at all takes on the order of a hundred cases that flip, which on a task where a critique turn changes a quarter of the answers means an eval of four hundred and up. An eval too small to distinguish a modest real improvement from noise will still produce a number, and that number will be argued about in a pull request.

The twelve broken cases are the actual finding#

The 2x2 also destroys the mental model that a headline invites, which is that the change makes some answers slightly better and leaves the rest alone. The critique turn was rewriting a substantial fraction of the answers, and on 12 of them it rewrote a correct query into an incorrect one.

Reading those 12 is ten minutes well spent. Eleven of the 14 fixes were join problems — a missing join condition, an ambiguous column, a cross join where an inner join was meant. Nine of the 12 regressions were the critique turn widening or dropping a filter, and they had a consistent shape:

-- baseline (correct)
SELECT customer_id, SUM(amount)
FROM   orders
WHERE  status = 'settled'
  AND  created_at >= '2026-01-01' AND created_at < '2026-04-01'
GROUP  BY customer_id;

-- after the critique turn: "the date range should be inclusive of the
-- quarter end, and 'settled' may exclude late-settled orders"
SELECT customer_id, SUM(amount)
FROM   orders
WHERE  status IN ('settled', 'pending_settlement')
  AND  created_at >= '2026-01-01' AND created_at <= '2026-04-01'
GROUP  BY customer_id;

The critique invented a requirement, justified it fluently, and produced a query that runs, returns rows, and is wrong. Compare that to the failure mode it fixes: a missing join condition produces either a SQL error or a row count so absurd that the user notices immediately. The two sides of the ledger are not the same kind of event. Eleven fixes to loud failures and nine regressions to silent ones is a worse product at 2.4x the token spend, even in the world where the aggregate had moved by five points instead of one.

That asymmetry is what a headline accuracy number is structurally incapable of showing, because accuracy weights every case at 1.0.

Making severity part of the number#

If the failures have different costs, the eval should say so. This does not require a formal loss function; a three-tier weight assigned once, per error class, gets almost all of the benefit.

How a weight changes a verdict is simple enough to state and easy to skip past. Accuracy counts every case at one, so the ledger reads fourteen against twelve and the fixes win by two. Assign the error classes different weights and the two sides stop being counted in the same units. Nine of the twelve regressions are silently wrong rows, weighted five times a loud failure; eleven of the fourteen fixes are loud failures weighted at one, and the remaining three are unnecessary scans weighted at half of that. Nine regressions at five apiece therefore outweigh all fourteen fixes put together by more than three to one, and the sign of the result turns over. None of the counts moved. What moved is the claim that a case is a case.

SEVERITY = {
    "silently_wrong_rows": 5.0,   # query runs, returns plausible wrong data
    "query_errors":        1.0,   # fails loudly, user retries
    "unnecessary_scan":    0.5,   # correct but expensive
}

def weighted_delta(flips: list[tuple[str, str]]) -> float:
    """flips: (direction, error_class); direction in {'fixed', 'broken'}."""
    return sum(
        SEVERITY[cls] * (1 if d == "fixed" else -1)
        for d, cls in flips
    )

# 14 fixed (11 query_errors, 3 unnecessary_scan)
# 12 broken (9 silently_wrong_rows, 3 query_errors)
#   -> 11*1.0 + 3*0.5 - 9*5.0 - 3*1.0 = -35.5

Minus 35.5 against a plus-one accuracy headline is the same run, scored twice. Publishing the weights alongside the result also forces the argument to happen in the open, since anyone who disagrees with a 5x penalty on silently wrong rows can say so and the number moves.

The cost side deserves the same treatment, expressed per unit of benefit rather than as a multiplier. At 744 output tokens against 310, on 90,000 SQL generations a month, the critique turn adds about 39 million output tokens, which at $15 per million is $585 a month. Against 14 fixed cases in 200, or 7% of requests, that is roughly $0.09 per corrected query — cheap, if the correction were free of side effects. Against the net two cases it is $0.65 per net correction, and against the severity-weighted total it is money spent to make the product worse. Reporting a cost multiplier invites the response that tokens are cheap; reporting dollars per net corrected case invites the response the evidence actually supports.

That paragraph runs four steps, and the last two are where the argument is actually won. The token step takes the difference per request, 744 against 310, and multiplies it by the monthly volume of 90,000 — a difference rather than a total, because the baseline's tokens were already being paid for — which is where about 39 million extra output tokens comes from. The money step prices that at $15 per million. The two rates that follow then divide the same $585 by two different denominators, and the choice of denominator is the entire argument. Dividing by the 14 cases fixed treats the broken ones as though they had not happened, and gives about $0.09. Dividing by the two net cases — fourteen fixed less twelve broken, which is what the user population actually receives — gives $0.65, seven times as much money for exactly the same spend.

The severity-weighted score has no denominator available to it at all, and that is the point rather than a gap in the method. There is no cost per net corrected case when the net is a loss; the money buys a product that is worse along the dimension the weights were written to capture. So the third framing is not a harsher version of the second, it is a different statement, and it is why the weights have to be agreed before the run rather than negotiated once a result is on the table.

Questions that decide a reported gain#

A teammate reporting that a self-critique turn moved their support-triage classifier from 84% to 89% has a more interesting number than one point, and the same questions apply. Their eval is 60 examples drawn from last quarter's tickets, scored once, with the headline recorded and nothing else. Five points on 60 examples is three examples.

The third row's arithmetic looks like a sampling nicety and is not. A class making up a fifth of live traffic should turn up about twelve times in 60 examples if the sample were representative of production. Finding four means the eval carries roughly a third of the exposure it should for that class, so a change that helps everywhere except there will score better on the eval than it deserves, and a change that helps only there will score worse. Neither error is repaired by running the eval more times. Repetition reduces noise, and a sample pointed in the wrong direction is not noisy — it is consistently wrong in the same direction on every run.

The kill condition in the fourth row is the one that decides it even in the good case. If the 89% survives more examples, repeated runs and a fresh sample, and the flips turn out to be six low-severity misroutes fixed against two escalations now landing in the general queue, the change should not ship. The aggregate improved and the product got worse, because a billing question routed to the wrong specialist costs a few hours and an escalation that never escalates costs a customer. "Run it on more examples" answers only the noise question, and noise was never the only question.

Leakage deserves its own emphasis because it is the least visible of these. A critique prompt refined over an afternoon against the same 60 examples is a model of those 60 examples. The honest version holds out a set that was never read during prompt iteration, which for a 60-case eval means the eval needs to grow before it can be split at all.

Keeping the eval set frozen#

The word frozen is doing real work in all of this and is easy to let slide. An eval set that grows whenever someone adds an interesting failure is not comparable across time, so a five-point gain measured in April against a set that gained forty cases in March is measuring the cases as much as the change. The workable arrangement is a versioned set that changes on a schedule, with every reported number carrying the set version, and a separate growing pool of new cases that becomes the next version. When the set does turn over, re-run the current production configuration on the new version first, so that the baseline moves for a known reason rather than showing up inside somebody's change.

None of this makes a frozen eval a good predictor of production quality. It is a sample from a distribution that drifts, it typically over-represents the cases someone found interesting enough to write down, and it scores against a rubric that is itself an approximation. A change that is clearly positive on a well-run paired eval can still disappoint in production, and the eval's job is to catch the changes that are clearly not positive before they cost anything.

A critique pass is therefore not something an accuracy delta can approve or reject on its own. The three numbers in that pull request were all true and none of them was evidence: the delta was two cases against a baseline that moves 3.5 points by itself, the token multiplier said nothing about dollars per corrected case, and the accuracy metric weighted a silent wrong answer the same as a loud one. The 2x2, three runs per configuration, severity weights agreed in advance and a cost expressed per net correction are four cheap artefacts, and together they turn the same afternoon's work into something that can be argued with.

Nor is any of this an argument that critique turns are a bad idea. The finding here is narrower and it points somewhere specific, since the pass was good at joins and destructive on filters. A pass restricted to the half it does well — asked to check join conditions and column qualification, forbidden from touching the WHERE clause — is a different experiment, and a promising one, which is why the pull request went back rather than being closed. What people reach for instead, when one round of critique returns a disappointing plus-one, is more rounds.