Eleven findings per draft is a critic without a threshold

By April the critic was returning an average of 10.8 findings per draft. Meridian's revision step acted on all of them, which meant the second draft differed from the first in about 62% of its sentences, and a third of reviews went round a second time because the revised draft picked up findings of its own. Token spend per review had gone from 2,300 to 8,900. Wall-clock from brief to sendable draft had gone from 9 seconds to 31. On the blind quality grading, reviews produced with the critic scored 3.42 out of 5 and reviews produced with no critic at all scored 3.38, a gap the confidence interval swallowed whole.

A sample of what the critic was returning explains the flat line better than the aggregate does.

[critic  |  draft 2026-04-11-northwind  |  11 findings]
 1  high     recommendation contradicts stated retention constraint
 2  medium   overage figure conflicts with entitlement matrix row 4
 3  medium   "significant growth" — 6% quarter on quarter
 4  low      passive voice in section 1 opening
 5  low      "utilise" — house style prefers "use"
 6  low      section 3 is longer than section 2; consider balancing
 7  low      the closing sentence repeats the opening framing
 8  low      three sentences begin with "Additionally"
 9  low      consider naming the account manager earlier
10  low      bullet list would read better as prose here
11  low      "leverage" appears twice

Findings 1 through 3 are worth a revision pass. Findings 4 through 11 are a copy editor with no brief, and they cost the same to produce, the same to act on, and considerably more in damage, because the revision step rewriting eight sentences for wording is eight chances to introduce a defect into text that was correct. Two of April's three worst regressions were introduced during revision, in sentences that no finding had objected to but that sat next to one that did.

The critic was never told what counts as a defect#

A prompt that says "find problems with this report" has handed the model an open-ended question, and an open-ended question about a piece of writing has an unlimited number of true answers. Any text can be described as improvable, so a critic asked for problems will keep finding them until it runs out of output tokens or politeness, and it will do so at every quality level. That is why the finding count did not fall as the writer improved: the last four items in the list above would have been produced for a flawless report.

The step under that is worth making explicit, because it explains what the finding count is actually measuring. The model produces findings until something stops it, and under an open prompt the only things that stop it are the output token limit and whatever conversational sense of sufficiency the weights impose. Neither of those is a function of the draft. So the count is set by the prompt and the sampling, and a draft with six real defects and a draft with none produce lists of similar length made of different material. A count that does not depend on the input carries no information about the input, which is why watching the finding count under an open prompt tells you nothing and why the same count under closed categories tells you a great deal.

The open prompt is tempting for a reason that sounds like rigour. Enumerating categories feels like telling the critic what to find, which feels like pre-deciding the answer and giving up the chance to be warned about something you did not anticipate. In practice the open critic does not warn you about anything you did not anticipate either, because the defects it surfaces are the ones most available in the text — surface features of style, which are visible in every sentence, rather than a conflict with a document that requires the source to be consulted. The unanticipated defect arrives through the incident log and the labelled set, and the answer to it is a new category, added deliberately, with its own measurement.

Closing the question means enumerating the defect classes the loop exists to catch, and refusing everything else. Meridian's list came from the incident log rather than from a whiteboard, which is the part worth copying — each category is there because a defect of that shape reached a customer.

CATEGORIES = {
    "contradicts_source":      "Claim conflicts with a supplied document.",
    "unsupported_claim":       "Claim no supplied document establishes.",
    "internal_inconsistency":  "Two parts of the report disagree.",
    "constraint_violation":    "Recommendation conflicts with a constraint "
                               "stated in the report or brief.",
    "arithmetic_error":        "A figure does not follow from the figures "
                               "given.",
    "missing_required_section":"A section the brief requires is absent.",
}

SEVERITY = """
high   — a reader acting on this would take a wrong action, be billed
         differently than stated, or receive a factually false claim.
medium — the claim is unsupported or overstated; a careful reader would
         notice the gap but would not be misled into action.
low     — accurate and supported, but weaker than it could be.

Severity describes the consequence to the reader, not how strongly you
disagree. Style, tone, word choice, structure and length are not defects
and must not be reported in any category.
"""

Two details in that prompt earn their words. Severity is defined by consequence, so the model has an external anchor and cannot slide into rating its own conviction, which is what produces a list where a preference about "leverage" arrives at the same weight as a wrong overage rate. And the exclusion is explicit and enumerated, because a model that has been given six categories will otherwise file a wording preference under unsupported_claim and phrase it so that it fits.

Two seams in that prompt are worth seeing before it is copied, because both leave a decision to the model that the prompt meant to take away. The exclusion says structure is not a defect, and one of the six categories is a missing required section, which is a structural property; a model reading both has to work out that the exclusion means presentational structure while the category means a section the brief demanded, and nothing in the text says so. The other seam is between arithmetic_error and internal_inconsistency: two section totals that disagree are two parts of the report disagreeing and a figure that does not follow from the figures given, so the same defect can be filed under either label depending on which sentence the model read last. Neither seam produces a wrong finding. Both produce findings whose category is unstable, which matters later, because every per-class measurement in this topic matches findings to labelled defects by category.

Measure the precision of what gets through#

Dropping low-severity findings improves the numbers whether or not the surviving findings are any good, so the threshold has to be accompanied by a measurement of what survives it. The measurement that matters first is precision: of the findings the loop acts on, what fraction correspond to a real defect.

Building that is manual and finite. Take 200 above-threshold findings from a week of production traffic, strip the severity and the model's explanation, and have two people who know the domain label each one as a real defect or not, adjudicating the disagreements. Meridian's first pass scored 0.71 precision with a Cohen's kappa of 0.79 between labellers — kappa being agreement corrected for the agreement two people would reach by guessing, where 1.0 is perfect and anything under about 0.6 means the labels are too unstable to tune against — and the errors clustered: 31 of the 58 false findings were unsupported_claim raised against sentences whose support was in a document that had not been retrieved, which is the retrieval failure from chapter 3 showing up as a number instead of an anecdote.

That 58 is not an independent figure; it falls out of the other two. Precision of 0.71 over 200 labelled findings means a little over a hundred and forty were real, and the remainder — the 58 — were not. So the sentence is stating one measurement twice, and checking that it closes is worth two seconds because it confirms the sample and the precision refer to the same set of findings rather than to two different weeks.

The clustering figure is the one that carries a decision, and it is worth converting into a ceiling. Thirty-one of the two hundred labelled findings were false for one identifiable reason, so eliminating that reason entirely — perfect retrieval — would raise precision by a little over fifteen points and put it in the mid eighties. That is a ceiling on what fixing retrieval can buy on this measurement, which is a great deal, and it is also the end of what that fix reaches: the other twenty-seven false findings have other causes, and they are the ones a better critic prompt or a different model would have to address. Knowing which of two projects is capped where is most of what a precision breakdown is for.

The labelling cost two people about three hours each, once, and it is the piece teams skip. Skipping it leaves you tuning the critic prompt against the end-to-end quality score, which moves slowly, has a wide interval, and responds to the writer, the retrieval and the revision step as well as the critic — so a week of prompt edits produces a set of movements you cannot attribute. Arize's LLM as a Judge 102 treatment of meta-evaluation makes the same argument in more general terms: the judge is a model with an error rate, and until you have measured it you are trusting an unmeasured component to tell you whether the measured ones are working.

Precision alone can be gamed by a critic that says almost nothing, so it needs recall beside it, and recall needs a different artefact — a labelled set of known defects like the 120-review set, where you know what is there and can count what was found. The two measurements answer different questions and neither substitutes for the other.

Note why the two artefacts in the first column cannot be swapped, since teams try to build one set and use it for both. Precision has to come from production traffic, because it asks what fraction of the findings the loop actually raised are real, and that fraction depends on the mix of drafts the loop sees. Recall has to come from a set with known defects, because it asks what fraction of the defects present were found, and production traffic does not come with a list of what is wrong with it. One artefact starts from findings and looks for defects behind them; the other starts from defects and looks for findings in front of them. The third row is the one that decides whether the loop stays, and the first two are what tell you where to work when it does not move.

Two proposals that move the number and not the problem#

The proposal that always arrives at this point is to keep the prompt as it is and truncate the critic to its top three findings. It does fix real things: revision churn falls immediately, the token bill falls, latency falls, and the change ships in one line. Taken as a treatment of the symptom it is a good one, and taken as the whole answer it hides three problems that the threshold approach surfaces.

The critic is doing the ranking, and nothing has established that it ranks well. Truncating to the first three items of an eleven-item list produced by an unconstrained prompt gave a labelled precision of 0.42 against the threshold approach's 0.71 on the same traffic, because the ordering an open-ended critic emits reflects the order things occurred to it rather than their consequence. Second, the fabricated and preference-shaped findings are still being generated and are simply no longer visible, so the drop log that told you 15 findings quoted sources that did not exist now tells you nothing about the eight items you threw away. Third, a cap is a constant applied to a variable quantity: a genuinely bad draft with six real defects ships with three of them fixed, and the loop reports the same three-finding shape it reports for a clean one, so the count stops carrying information about draft quality.

Thresholds behave the other way round. A clean draft produces zero findings above medium and skips revision entirely, which on Meridian's traffic is 38% of reviews and is where most of the token saving actually came from. A bad draft produces seven and gets all seven.

The other standing proposal is to paste the grading rubric into the critic prompt so its findings line up with how quality is scored. It is tempting because it is the obvious way to make the two numbers agree, and it does: on Meridian, adding the rubric raised the graded score from 3.71 to 3.94 within a week.

The score moved because the critic learned to enforce the things the grader rewards. The grader gave a point for an explicit next step in each recommendation, so the critic began raising findings whenever a recommendation lacked one, and the writer began appending next steps to recommendations that did not need them. In a side-by-side where account managers picked which of two reviews they would rather send, rubric-aligned reviews won 51 to 49 against the pre-rubric ones, which is a coin flip attached to a 0.23-point rise in the metric. The rubric is a proxy for what a good review is, and pointing the optimiser directly at a proxy is how the proxy stops being one.

Put the two results beside each other and the disagreement is the whole finding. The metric rose by 0.23 and the people who send these reviews were split fifty-one to forty-nine on which they preferred, which is what an unchanged product looks like when a hundred choices are made. Both measurements are competent. They disagree because one of them is the thing anybody cares about and the other is a stand-in for it, and the rubric edit acted on the stand-in directly. The residual is that a hundred pairwise picks cannot detect a small genuine preference either, so the honest statement is that the rubric bought a measurable metric rise and no detectable product change.

There is a second problem with the rubric as a first move: it says nothing about evidence. A critic given the rubric and no quotation requirement produces findings that are well-aligned with the score and still unverifiable, so invented faults survive alongside the aligned ones and the drop log catches nothing. The ordering that works is categories first, then evidence spans, then a severity threshold, then measurement — and once precision is at 0.71 and you can see which categories the misses fall into, you have something to point at that is more specific than a rubric.

Every number in this chapter has been a comparison against a baseline: 3.71 against 3.38, 0.71 against 0.42, 4,100 tokens against 8,900. None of them are trustworthy yet, because they came from runs made at different times, on overlapping task sets, with a grader who could often tell which pipeline had produced which review from its shape. Getting a comparison that survives contact with a sceptical reviewer requires holding rather more constant than that.

The awkward part of that admission is that it applies to the recommendation this chapter makes as well as to the proposals it declines. Closed categories with a threshold look better than an open critic on every column here, and every one of those columns was measured under exactly the conditions the paragraph above says cannot be trusted. The argument for the recommendation is the mechanism rather than the numbers: an open question has unlimited true answers and a closed one does not, and that is true independently of what any run reported.