Comparing two critics means holding six things constant

The argument in the roadmap review lasted forty minutes and neither side had any evidence. The engineer who had built the separated critic showed the Northwind consolidation catch, which is a genuinely convincing artefact: a recommendation that would have tripled a storage bill, caught by the critic and missed by reflection. The staff engineer across the table remembered a review from March where the separated critic had flagged three claims as unsupported that were all fine, and reflection had quietly fixed a section total that was out by $4,400. Both examples were real. Both were single draws from distributions nobody had characterised, and the meeting resolved by seniority.

What settles this is a comparison narrow enough that the only thing differing between the two arms is the thing under test. That is harder than it sounds for this particular pattern, because the two arms differ naturally in call count, token spend, output length and formatting, and each of those differences is an alternative explanation for whatever the grader sees.

The six knobs, and what each one ruins#

The first four are bookkeeping, and they are still where most comparisons quietly go wrong, because each one is normally violated by accident rather than by choice. The task set changes because someone regenerated the sample to include this week's accounts. The writer prompt changes because the same pull request that added the separated critic also tightened the writer's instructions, and nobody separated the two edits. The revision budget changes because the separated arm was given a retry on parse failures that the other arm did not have. None of these are arguable once noticed; the discipline is to write down the six values before the run and to check them against the code afterwards.

The middle column is the one to read carefully, because each entry names a rival explanation rather than a nuisance. If the task set moved, then "the separated arm scored higher" and "the separated arm got easier accounts" produce identical numbers and the data cannot separate them. If the writer prompt moved, the same is true of "the critic helped" and "the writer got better instructions". A controlled comparison is not an attempt to reduce noise; it is an attempt to leave exactly one story that fits the result. Every knob you fail to hold adds a story, and a result compatible with three stories decides nothing.

The last two are where comparisons of this pattern fail even when everyone is being careful, so they are worth taking one at a time.

Token budget is awkward because the arms are not naturally equal. On Meridian, a self-critique arm costs one writer call carrying the plan and the draft, then one reflection turn appended to it, then one revision turn — three turns on a growing transcript, 6,100 tokens per item all told. The separated arm costs a writer call, a fresh critic call that re-sends the brief and the sources, and a revision call, which is 4,100 tokens because nothing accumulates the writer's reasoning but the sources are sent twice. Those are close enough to compare at one revision pass each, and they diverge sharply at two, where the self-critique transcript grows quadratically and the separated arm grows linearly. Reporting the comparison at a fixed number of passes with the token cost printed beside the score is more honest than trying to equalise both, since forcing equal tokens means handicapping one arm's passes and then arguing about whether the handicap was fair.

Those two growth rates are worth deriving, since the choice of where to report rests on them. In an appended transcript, every call re-sends everything said before it, so the second call carries one turn, the third carries two, and the tokens billed over a run are the running total of a rising sequence — which grows with the square of the number of turns. In a fresh-context design each critic call carries the same fixed bundle: the brief, the sources and the current draft, and nothing from any earlier call. So the tokens billed grow in direct proportion to the number of passes. At one pass the separated arm is already the cheaper of the two here, and the reason is not the growth rate — one pass is one pass — but the size of what accumulates: the self-critique arm's single transcript carries the writer's plan and reasoning, which the separated arm never re-sends. The growth rates decide what happens after that, and they say the gap widens with every pass added.

Blinding fails in a way that is specific to this pattern and easy to miss. The separated critic's findings are categorised and severity-tagged, so the revision step tends to produce structured edits, and Meridian's separated-arm outputs came out an average of 94 words shorter with a "Recommendations" heading present 91% of the time against the self-critique arm's 54%. A spot check where a colleague was shown twenty unlabelled outputs and asked to guess the arm came back 78% correct. A grader that can identify the arm is not grading blind, whether the grader is a person with an opinion about the project or a model with a preference for structured text.

Put a number on how much of that the heading tell alone explains, because the remainder is the part the fix does not reach. Somebody guessing "separated" whenever the heading is present and "self-critique" whenever it is absent would be right on 91% of separated outputs and on the self-critique outputs that lack the heading, which is a little under half of them, so the two together average to a little under seven guesses in ten. The colleague reached closer to eight. So the heading is not the only tell, and the obvious remaining one is length: the separated arm's outputs are 94 words shorter on average, and a reader who has seen a few pairs can use that. This is where the defence in the last column of the table stops short of the job. Stripping heading markers and bullet glyphs holds formatting constant, and the property that decides whether the grader is blind — identifiability — keeps varying through a channel the normalisation never touches.

The harness#

ITEMS = json.load(open("eval/fixed_set_v3.json"))   # 120 frozen item ids

def run_arm(item, arm: str) -> str:
    brief, sources = load(item)
    draft = write(brief, sources)                    # identical both arms
    if arm == "self":
        findings = reflect_in_context(brief, sources, draft)
    else:
        findings = criticise(draft, brief, sources)  # fresh context
    findings = above_threshold(verified(findings, draft, sources))
    return revise(draft, findings, sources) if findings else draft

def normalise(text: str) -> str:
    """Strip formatting tells so the grader cannot identify the arm."""
    text = re.sub(r"^#+\s*", "", text, flags=re.M)     # heading markers
    text = re.sub(r"^[-*•]\s*", "", text, flags=re.M)  # bullet glyphs
    return re.sub(r"\n{2,}", "\n\n", text).strip()

rows = []
for item in ITEMS:
    a, b = normalise(run_arm(item, "self")), normalise(run_arm(item, "sep"))
    left, right = ((a, b), (b, a))[random.random() < 0.5]   # order shuffled
    rows.append({"item": item["id"], "left": left, "right": right,
                 "left_is_self": left is a})

json.dump([{k: r[k] for k in ("item", "left", "right")} for r in rows],
          open("eval/for_grading.json", "w"))          # key withheld

Two details in that harness do most of the work. The arm key is written to a separate file the grader never opens, which is what makes the blinding survive somebody helpfully checking a result mid-run. And the left/right order is shuffled per item, because a grader — human or model — that always sees arm A on the left develops a position preference within a dozen items.

One property of the arms limits what any harness can hold constant here, and it is inherent rather than an oversight. write runs inside run_arm, so each arm generates its own draft, and the two arms are therefore criticising two different drafts of the same brief. That cannot be avoided for the self-critique arm, whose whole design requires the draft and the reflection to sit in one transcript. So the writer prompt is held constant and the writer's output is not, and every per-item difference carries one draw of writer variance on top of the arm effect. Pairing still helps, because both drafts answer the same brief and inherit the same account difficulty, and the residual variance is a reason the intervals below are wide rather than a reason to distrust the design.

Normalising formatting has a cost worth stating: it destroys the part of quality that lives in presentation, so an arm that genuinely produces better-structured reviews will not be credited for it. When structure matters to the product, the alternative is to grade on rubric dimensions that are indifferent to it, and to measure structure separately with a mechanical check rather than through a grader who can use it as an identifier.

The design is paired: both arms run on the same item, and the grade difference is computed within the item before being averaged. That matters more than the sample size here, because account reviews vary enormously in difficulty — a two-workspace enterprise account with a mid-quarter plan change is a harder brief than a single-seat startup — and an unpaired comparison spends most of its statistical power on that variation rather than on the arms.

Who grades, and how you know they can#

The grader is a judge, which means it has its own error rate and needs to be checked before its verdicts are used to decide anything. Meridian grades with a model against a five-point rubric, at roughly $0.02 per pair, which is what makes a 120-item paired run affordable to repeat weekly. Trusting it required calibrating against people: 60 pairs graded by two account managers, then by the model, giving a Cohen's kappa of 0.68 on the three-way better/worse/same judgement. That is decent agreement, and it is also low enough that a 0.05 difference in mean grade is inside the grader's own noise, which is a second reason the headline result above should not be read as a win.

The affordability claim is one multiplication and worth doing, because it is what makes the weekly cadence real rather than aspirational. Two cents a pair across a hundred and twenty pairs is under three dollars a run. That is why the whole comparison can be repeated after every prompt change instead of being treated as a quarterly event, and it is the strongest practical argument for a model grader over a human one — not that it is better, since the kappa says it is not, but that it can be run again on Tuesday.

Position bias in a model grader is large and measurable. Grading the same 120 pairs twice with the order flipped changed the winner on 17 of them, and averaging the two orders removed the effect. Human graders are subject to the same thing plus fatigue, which is why the pairs are shuffled and why nobody should be asked to grade 120 pairs in one sitting.

Set that 17 against what the run is trying to detect and the scale of the problem is clear. Flipping nothing but the presentation order changed the answer on about one item in seven, while the difference between the two arms is a twentieth of a grade point. So the instrument's own sensitivity to an irrelevant variable is large compared with the effect it is being pointed at. The averaging fixes the bias, in the sense that a bias applied equally in both orders cancels. It does not shrink the noise the flips reveal, and that noise is what the interval below is made of.

Where a frozen set stops describing production#

This is an offline comparison on a frozen set, which buys control and gives up representativeness. The 120 items were drawn from archived reviews with known defects, so the defect base rate in the set is far higher than in Thursday's batch, and any absolute number from it — precision, catch rate, mean grade — is a statement about the set rather than about production. Ratios between arms transfer reasonably well; absolute rates do not, and a dashboard that reports the eval set's catch rate as though it were the production one will show a comfortable number while the tail does whatever it likes.

The asymmetry between ratios and absolute rates has a reason worth knowing, because it tells you which numbers to quote. Selecting items for having known defects raises the defect density, and it raises it for both arms at once. A ratio between the arms divides that common factor out, so the ratio survives the selection. An absolute rate does not divide anything out and carries the selection with it into whatever it is quoted at. The rule that falls out is simple: report differences and ratios from a curated set, and get absolute rates from sampled production traffic or from nowhere.

The set also wears out. Prompts get tuned against it, and after a month of tuning the numbers reflect fit to those 120 items rather than to the task, which is the ordinary overfitting story arriving through a side door. Holding 40 items out from the start and touching them only at decision points is the cheap defence, and re-drawing the whole set from recent incidents every quarter is the real one.

Two things the harness cannot establish at all: how either arm behaves on inputs unlike anything in the set, and how it behaves when a source is missing rather than merely unhelpful. Both are production questions and need production instrumentation.

With the harness built, a question that has been deferred twice becomes cheap. Swapping the critic's model is a one-line change to criticise, the arms stay matched on everything else, and the same run that produces an aggregate and a per-class breakdown will also tell you whether two models fail on the same items or on different ones — which is the only thing that decides whether the second model is worth its invoice.