A separate critic judges the artefact, not its own process

Meridian drafts quarterly account reviews for a data-platform company: it pulls usage from the warehouse, plan and entitlement facts from billing, and writes an eight-hundred-word review that an account manager skims and sends. In the January batch, the review for Northwind Logistics recommended consolidating their two workspaces onto one to cut cost. Four paragraphs above that recommendation, in a table the same run had produced, sat the reason it was wrong: the two workspaces ran different retention policies, ninety days and seven years, because one of them held shipment records that Northwind's regulator requires them to keep. Consolidation would have forced seven-year retention onto everything and roughly tripled the storage line the recommendation claimed to reduce.

The reflection loop from 11.5 had run on that draft. It is worth reading what it said, because the failure is not that the pass was skipped.

[writer/reflect  pass 1  |  1,412 in / 318 out]
The draft is well structured and follows the plan: usage summary,
cost breakdown, then two prioritised recommendations. The
consolidation recommendation is supported by the storage figures in
section 2 and follows from the stated goal of reducing spend. The
second recommendation could be more specific about timelines.

Revision: tightened the timeline language in recommendation 2.
No other changes needed.

That is a competent reading of the draft against the plan the model had just written for itself, and it is exactly the wrong reading. The pass evaluated whether the draft executed the plan, and it did. Nothing in the reflection prompt asked whether the plan was any good, and by the time reflection ran, the plan was no longer a proposal under review — it was a set of premises earlier in the same conversation, which the model had already committed to in writing and now had every textual reason to treat as settled.

What Meridian is missing at this point is a reader who was not in the room when those premises were set, and the arrangement that supplies one has a standard name. In a writer–critic pair — the same structure is called a generator–evaluator pair in the evaluation literature — one model call, the writer, produces the artefact, and a second call, the critic, receives that artefact together with the brief it was supposed to satisfy and returns a list of specific findings rather than a rewrite, which the writer then revises against. What makes the second call a separate critic is that it starts a new conversation instead of continuing the writer's, so the plan, the justifications and the draft's own defence are not in front of it and it has no earlier commitment of its own to protect. The arrangement exists because generated text is often expensive to get wrong and nobody is going to read every output by hand, which is why it turns up in report and email drafting of Meridian's kind, in code-review bots that comment on a diff before a human does, in retrieval-backed answers checked against the passages they cite, and in evaluation harnesses that grade one model's output using another. Through this topic the writer's job stays fixed and the critic is the component under design.

The author is the worst reviewer of their own pull request#

Every backend team knows what happens when the person who wrote a change reviews it. They do not review it. They re-read it, confirm it matches what they intended, and approve. The defects that survive are the ones where the intention was wrong, because checking output against intention cannot find a bad intention. This is why review is assigned to somebody else, and the reason it works has little to do with the second reviewer being smarter. They are simply not the person who decided.

That is an analogy, and where it breaks is the whole subject of the next chapter, so it is worth marking the boundary now. A second human reviewer brings two things the author lacks: a mind with no stake in the change, and a different body of knowledge — different systems worked on, different outages remembered, different corners of the codebase read last month. The split built in this chapter delivers the first of those and none of the second. Everything a separated critic knows, the writer knew. So the part of human review that catches "you have misremembered how this API behaves" has no counterpart here, and every claim in this chapter is about the other part.

A self-critique pass reproduces the author-reviews-own-PR shape almost exactly. The reflection turn appends to a transcript that already contains the model's plan, its intermediate reasoning, its stated justifications for each choice, and the draft itself. Continuing that transcript is continuing a document in which those justifications are already asserted as true, and the most probable continuation of a document that argues for a decision is more argument for the decision. The model is not being stubborn in any psychological sense; it is completing a text whose earlier half is a defence.

Walk that mechanism one link at a time, because it is the only thing this chapter's fix acts on. A model call produces the continuation its weights rank as most likely given everything in the context window. The reflection call's context window contains, in order, the brief, the plan, the reasoning that produced each recommendation and the draft. That sequence is a document of a recognisable kind: a proposal followed by its own argument. Documents of that kind continue in a recognisable way, with more argument, qualification and elaboration rather than with retraction. So the most likely continuation of that particular window is a favourable reading, and a prompt that says "now critique this" changes the surface of the continuation — it will contain the word finding and some criticisms — while leaving the document it is continuing exactly as it was. The prompt is competing with several hundred tokens of prior commitment, and the prior commitment does not have to argue.

The shape of the fix is old enough to have been borrowed rather than invented. Generator–evaluator pairs predate language models by decades: a genetic algorithm has a fitness function that is not the mutation operator, a compiler has a verifier that is not the optimiser, and a proof search has a checker that is not the searcher. In every one of those, the evaluator is deliberately a different piece of machinery with different inputs, and nobody would think of implementing the fitness function by asking the mutation operator whether it was happy with its work. The language-model version of that architecture is the same trade: you give up the evaluator's access to the generator's internal state in exchange for an assessment that is about the output.

The borrowing is partial and it is worth saying which part did not come across. In every one of those three systems the evaluator is a different kind of machinery: a fitness function is arithmetic and a mutation operator is not, a proof checker implements a decision procedure and a proof search does not. That difference in kind is what gives their evaluators an error profile unrelated to their generators'. The language-model version reuses one piece of machinery for both roles and varies only what is put in front of it. So the architecture transfers and the guarantee does not, and a design document that cites the compiler analogy as a reason to trust the critic has taken the shape and left the substance behind.

Change the setup so that the second call is a fresh conversation, with its own system prompt, its own stated role, and nothing in context except the artefact and the specification it was supposed to meet, and the second call has no defence to continue. Here is the same draft, same model version, same temperature, run that way.

[critic  pass 1  |  1,106 in / 402 out]
finding 1  severity: high  category: unsupported_recommendation
  span (draft): "consolidating both workspaces onto workspace-prod
  would remove the duplicate storage charge"
  The report's own section 2 lists retention as 90d for workspace-eu
  and 7y for workspace-prod. Consolidation would apply one retention
  policy to both data sets. The report does not state which, and does
  not estimate the storage impact of the longer policy. The claimed
  saving is not supported by the figures given.

finding 2  severity: low  category: vague_commitment
  span (draft): "in the coming quarter"
  No date given for the migration review.

Same weights, same sampling parameters, a different position in the conversation, and the defect that mattered came out. That is the entire mechanism of this chapter, and it is smaller than it first looks: the critic did not know anything the writer did not know. It knew the same things and had no stake in the conclusion.

Notice which evidence finding 1 rests on. It cites the report's own section 2 — the retention figures were in the draft the whole time, and were in the reflection pass's context too. Nothing was fetched, nothing was looked up, and no fact entered the system between the two runs. The difference is entirely in what the two calls were positioned to do with material both of them could see.

Building the split so that it is actually a split#

Two things have to be true for the second call to be a separate critic rather than a self-critique pass wearing a new label. It runs in its own message list, not appended to the writer's. And its system prompt gives it a role and a job of its own, phrased in terms of the artefact rather than in terms of the writer's work.

WRITER_SYSTEM = (
    "You write quarterly account reviews for platform customers. "
    "Use only the figures supplied. Aim for 800 words."
)

CRITIC_SYSTEM = (
    "You are a reviewer at a data-platform company. You are given a "
    "quarterly account review written by someone else and the brief it "
    "was written against. Your job is to find claims the report's own "
    "figures do not support, and recommendations that contradict the "
    "constraints stated in the report. You did not write it and you are "
    "not responsible for defending it. Report findings only."
)

def write(brief: str) -> str:
    r = client.messages.create(
        model=MODEL, max_tokens=2000, system=WRITER_SYSTEM,
        messages=[{"role": "user", "content": brief}],
    )
    return r.content[0].text

def criticise(draft: str, brief: str) -> str:
    r = client.messages.create(
        model=MODEL, max_tokens=1500, system=CRITIC_SYSTEM,
        messages=[{                       # fresh list; writer turns absent
            "role": "user",
            "content": f"<brief>\n{brief}\n</brief>\n\n<report>\n{draft}\n</report>",
        }],
    )
    return r.content[0].text

The line that carries the design is the one constructing the critic's messages. It is a new list. The writer's turns are not in it, the writer's plan is not in it, and no assistant turn precedes the critic's. If instead you had written messages = writer_messages + [{"role": "user", "content": "Now critique that."}], you would have a reflection pass with a longer prompt, however carefully the wording distanced the model from its work, because the defence would still be sitting in the context window as prior assistant text.

Two near-misses are worth naming because both look like separation from the outside. The first is keeping the draft in an assistant turn — [{"role": "user", "content": brief}, {"role": "assistant", "content": draft}, {"role": "user", "content": "Critique this."}] — which is a transcript in which the model has apparently just produced the report, and produces verdicts measurably closer to the reflection pass than to the critic above. The second is passing the writer's plan to the critic "for context", usually added by someone reasoning that the critic will judge better if it understands the intent. It will judge the execution of that intent better and the intent itself worse, which is the trade you were trying to get out of.

The first near-miss is worth one more sentence because of how little it takes to fall into. Nothing in that message list is a lie — the model did produce the draft, and putting it in an assistant turn is the honest encoding. But a role label is not a record of history to a model; it is a token in a document, and an assistant turn containing a report is the same textual situation as having just written one. The separation is achieved by what the document looks like rather than by what actually happened, which is why the encoding decides the outcome.

Which context each arm carries is the whole of what has changed, and it is worth laying out plainly.

Three of those six rows differ, and the last one is the one that will matter most in the next chapter. Read the table by what it holds fixed rather than by what it changes and the argument gets sharper. The brief is fixed, the draft is fixed, and the weights are fixed — the two arms are given the same specification, the same artefact and the same model at the same temperature. Everything that varies is the conversational surround: three rows describing what the model has apparently already said and what role it apparently already holds. So the experiment is unusually clean, since only one family of thing moved. The same cleanness is why the last row cannot be argued with. Holding the weights fixed is what makes this a measurement of stance, and it is also what guarantees that anything the defect depends on which lives in the weights is present identically on both sides.

Where the split has nothing to offer#

There is a class of task where separating the roles changes nothing measurable, and recognising it early saves a quarter of confused dashboards. When the defect the loop is meant to catch is one the model can detect from the artefact alone with high reliability — a JSON payload that violates a schema, a summary that exceeds a word count, a citation format that does not match a style guide — the reflection pass already catches it, because commitment to a plan does not blind a model to a mechanical property of the output. Meridian's over length class was caught 40 of 41 times by reflection and 41 of 41 by the separated critic, which is not a difference worth a second call, and for that class a two-line assertion in code is cheaper and exact.

The word exact is doing real work in that sentence and is worth unpacking, because it is the strongest claim available anywhere in this topic. Counting the words in a document and comparing the count against a number stated in the brief is a total function: it terminates, it has no failure mode that depends on phrasing, and it returns the same answer every time. So the deterministic version of the over length class is not 41 of 41 as a measured result; it is 41 of 41 by construction, and it would be 41 of 41 on any set. Both model arms are being paid to approximate something a comparison operator already decides. Every class where that is true should leave the model entirely.

The split earns its calls on judgement-shaped defects: claims the evidence does not support, recommendations that conflict with constraints, conclusions that overreach the data. Those are the ones where the writer's own justification is the thing standing in the way, so removing the justification from context is what changes the outcome. The test to apply before spending the call is whether you can name the commitment reflection would be protecting. If the answer is that reflection has nothing at stake on a defect of that shape, the split will not move its numbers.

It is also worth saying what the split cannot repair even on those defects. If the reason a claim is wrong lies outside the draft and outside the model's knowledge, then a critic reading only the draft has no way to reach the truth, and it will read a confident false statement as a supported one. Meridian's reviews quote plan pricing constantly — per-seat rates, included allowances, overage thresholds — and none of that lives in any model's weights, because it is one company's internal price list that changes every few months. On that class of claim the separated critic caught almost nothing, which is the next chapter's subject, and the reason it fails there has nothing to do with stance.

So the whole of what has been bought is a reviewer with no stake in the draft, obtained for one call by changing which message list the second call runs in. On the defect classes where the evidence is inside the artefact and the brief, that is worth a great deal, and Meridian's own numbers on the constraint class are the demonstration. On the classes where the evidence lives somewhere else, the stance is irrelevant, because a reviewer with no stake and no access is still a reviewer with no access.