Unaided self-critique

Self-critique is the practice of asking a model to examine output it has just produced, state what is wrong with it, and revise it, either as a further turn in the same conversation or as a fresh call carrying the answer across; run as a repeating cycle of generate, critique and revise, the same pattern is usually called reflection. It exists because the alternatives that reliably work are expensive. A large language model (LLM) writes its answer in a single pass, with no built-in moment where it stops to check a figure against the source or reconsider a step, and closing that gap properly — a person reading every output, a purpose-built verifier for each field, a second model from another vendor — costs money, latency or engineering weeks that one more paragraph of prompt does not.

You meet the pattern as a "review your answer and correct any mistakes" line appended to a system prompt, as the reflect node in an agent framework's graph, and as the generate–critique–refine loop that a run of 2023 papers made a standard building block. All of it rests on one premise: that a model asked to look again can locate the mistakes it has just made. That premise is cheap to test, it fails in a specific and predictable direction, and the direction is the subject of this chapter.

Why the check comes back clean#

Take a summariser that turns 8-K filings and company press releases into three-sentence digests for a markets newsletter. It reads a release opening with the line Cadence Foods completed its acquisition of Northmoor Dairy on 12 November 2019, and produces a digest saying the merger closed in March 2021 and added roughly $340M in annual revenue. The revenue figure is correct. The date is wrong by sixteen months. A review step sits downstream for exactly this class of mistake — a second turn appended to the same conversation, asking the model to check the summary against the source and flag any factual errors:

messages = [
    {"role": "system", "content": "Summarise the filing in three sentences."},
    {"role": "user", "content": SOURCE},          # 1,180 tokens of press release
    {"role": "assistant", "content": SUMMARY},    # the digest, with "March 2021"
    {"role": "user", "content":
        "Check this summary against the source and flag any factual errors."},
]

The summary is accurate. The parties (Cadence Foods, Northmoor Dairy), the
transaction type (completed acquisition), the closing date and the revenue
contribution are all consistent with the source document. No corrections
needed.

The digest said March 2021 because, conditioned on a press release about a mid-market food acquisition, that is a high-probability date. The 2020–2021 period was dense with exactly this kind of transaction, the phrasing of the release matched a great many 2021 announcements, and the model's parametric prior over "when do acquisitions like this close" put substantial mass there. The date was not copied from the source, it was generated, and it was generated because it scored well.

Two routes can put a claim into a digest, and which route a given claim took is the whole story of this chapter. A claim can be copied: the string is in the source, attention locates it, and the model reproduces it. Or it can be generated: nothing in the source fixes it, and the model emits whatever its training has made likely in that slot. The party names were copied, and the quoted line of the release is where they came from. The date was generated. Both arrive by the same forward pass and both land in the digest inside the same flat declarative sentence, so the finished text carries no mark saying which is which. Anything downstream that wants to tell them apart has to go back to the source and look.

Now run the review turn. The question being asked is, in effect, whether the claim in the summary is plausible given the source. The machinery available to answer that question is the same distribution over text that produced the claim in the first place. March 2021 still scores well under that distribution, for the same reasons it scored well four hundred tokens earlier. The error is invisible to the checker because the checker's notion of "does this look right" is the very function that emitted it.

That result is worth laying out link by link, because each link is separately checkable and a reader who has only the conclusion cannot predict the next case. "Flag any factual errors" asks for a judgement, and a judgement is satisfied by a verdict. A verdict is text, and text is what the model produces from its distribution. So there is no step in this task that fails if the source is never consulted: "the summary is accurate" is a well-formed, plausible, on-topic completion whether or not a single token of the press release was attended to. Nothing forces the lookup. The one thing that could still raise an alarm, then, is the model's own sense that March 2021 sits badly in this document — and that sense is the same scoring function that put March 2021 there. It scored the date well when generating and it scores the date well when reviewing, because it is one function evaluated twice. The highest-probability continuation of a source, a summary and a request for review is therefore a confirmation, and a confirmation is what came back.

The contradicting sentence was in the context the whole time. Cadence Foods completed its acquisition of Northmoor Dairy on 12 November 2019 sat about a thousand tokens above the review request, in plain text, unambiguous. The check did not fail because the evidence was missing. It failed because the task it was given could be completed without reaching for the evidence, and the model's estimate of its own output is not a second opinion on that output. This is the part that is genuinely hard to hold onto: the failure is not a shortage of information and it is not a shortage of capability, so neither more context nor a larger model addresses it.

Two further pressures make it worse. The summary is sitting in the context window as a completed assistant turn, which is strong evidence, in the statistical sense the model actually operates on, that the summary is the sort of thing that follows from the source — the conversation so far asserts it. And the reviewing turn has been given no task with a verifiable outcome; "flag any factual errors" invites a judgement, and in the absence of a discrepancy the model can actually locate, the highest-probability continuation of a document containing a source, a summary and a request for review is a short confirmation.

How much of this is the shared weights and how much is the transcript can be separated with a small change to the prompt. Presenting the same 300 digests to the same model as work written by someone else — "a junior analyst produced this summary; check it against the source" — took the flag count from 5 to 19 of the 41 digests that contained a slip, with false alarms rising from 9 to 34. Neither the model nor the source changed, only the attribution, and the model became roughly four times more willing to find fault and nearly four times more likely to invent one. That framing effect is real and cheap to exploit, and it is bounded by the same ceiling as everything else here, because the 22 slips it still missed are the ones that follow from priors the model holds no matter whose name is on the text.

The Google DeepMind paper Large Language Models Cannot Self-Correct Reasoning Yet is the cleanest statement of the general result. Across reasoning benchmarks, models asked to review and revise their own answers with no external signal did worse after the revision than before it, and the earlier literature that reported gains had, in several cases, been using the ground-truth label to decide when to stop revising — which is not a self-correction loop, it is a loop with an oracle in it.

What a targeted check finds instead#

The interesting part is that the model is not incapable of finding this error. It is incapable of finding it when asked the question that way. An audit of 300 digests found 41 containing at least one factual slip, and the same model was pointed at those 300 outputs under four different check designs.

Read the slips-found column as a fraction of 41 before reading anything else, because the denominator is what makes those counts mean something. The second turn found 5 of the 41 digests that carried a slip, a little over one in ten; the other thirty-six came back clean and were not. The fresh call with identical wording found 8, still under a fifth. The quote-or-NOT-FOUND design found 33, which is four in five. Those are not neighbouring quantities, and the first two rows are not one rewording away from the third.

The false-alarm column is where the design argument is actually settled, and it has to be read alongside the slips found. Flagging more things is not by itself evidence of a better check, because the cheapest way to raise the count of slips found is to lower the bar for complaining, and that raises the false alarms with it. The first two rows are that trade happening. Moving from a second turn to a fresh call found three more slips and took the false alarms on the 259 clean digests from 9 to 21: the true catches rose by about half, the spurious ones more than doubled. Real finds per false alarm went down. The fresh call did not become more discerning; it became less agreeable.

The junior-analyst reframing described above is the same effect at higher amplitude, and running the same division on it is what shows that. It took the flags from 5 to 19 and the false alarms from 9 to 34, and the striking thing is that those two multipliers are almost identical — a little over one true catch for every two false ones before the reframing, and a little over one true catch for every two false ones after it. The proportions did not move. Attribution shifted where the model set its threshold and left its ability to tell a slip from a clean digest exactly where it was. Any intervention that scales both columns by roughly the same factor has bought willingness, not judgement, and the first column on its own cannot tell you which you bought.

The third row is the one worth staring at. Nothing changed about the model or the source; what changed is that the task stopped being an appraisal and became an extraction with a checkable failure mode. Asked to quote the sentence containing "March 2021", the model has to actually look, and there is no sentence to quote, so it writes NOT FOUND. The judgement about whether the summary is good has been replaced by a lookup whose answer is in the context.

Slips found and false alarms read together are what confirm that reading, and they are the reason this row cannot be explained as another threshold move. Against the second turn it finds more than six times as many slips — thirty-three against five — while raising fewer false alarms, six against nine. Every threshold move trades those two counts against each other, by construction: a bar low enough to catch more real slips is low enough to catch more imaginary ones. This row improves both at once, which no setting of a bar can do. Only a change in the question can.

"Has to actually look" is the step, and it is worth being exact about what forces it. The required output is a span of the source. Producing a span means locating one, and there is no way to satisfy that format from the prior alone except by fabricating a quotation — which is a loud failure rather than a quiet one, since a downstream string comparison catches an invented quote for nothing. The appraisal imposed no such requirement. "The summary is accurate" is complete and well-formed whether or not a source token was ever consulted, and that asymmetry, rather than any difference in wording or effort, is the entire distance between the first row and the third.

What the third row buys is grounding, and grounding is not accuracy. It is not a hard check: eight of the 41 slips still got through it, and six clean digests were flagged anyway. A model asked for a quotation can return a span that nearly matches, quote a sentence carrying the figure in some other role, or write NOT FOUND against a fact the source states in different words. The gain is that its remaining failures now require it to do something specific and inspectable rather than to say nothing, and a wrong span is a thing a reviewer can be shown.

The fourth row is a reminder about scope. Twelve lines of Python caught every date error in the set and none of the nineteen errors that were something else — a misattributed quote, a revenue figure assigned to the wrong subsidiary, a product name that did not appear in the release at all. It was also free, deterministic, and available for logging and alerting in a way a model's opinion is not.

import re

DATE = re.compile(r"\b(?:\d{1,2} )?(?:January|February|March|April|May|June|July|"
                  r"August|September|October|November|December) \d{4}\b|\b\d{4}-\d{2}-\d{2}\b")

def unsupported_dates(summary: str, source: str) -> list[str]:
    """Dates asserted in the summary that appear nowhere in the source."""
    return [d for d in DATE.findall(summary) if d not in source]

# unsupported_dates(SUMMARY, SOURCE) -> ['March 2021']

That function does not understand anything. It also does not have a bad day, does not vary by 4% between runs at temperature 0.7 — the sampling setting that decides how much randomness goes into each token choice — and costs nothing per call, which is a combination no prompt can offer.

The same weights with a different context window#

The second row of that table is the one people get wrong, and it is worth being precise about, because it is the crux of a design argument that comes up on every team that builds one of these.

Two engineers are reviewing a contract-clause extractor. One proposes a second turn in the same prompt — the clause, the extracted value, the model's working, then "check your answer and correct it". The other proposes a separate call to the same model, given only the clause text and the extracted value, with no prior reasoning in context. The framing usually offered for the second design is that it is an independent check.

That framing is tempting for a reason worth stating rather than waving away. Everywhere else in engineering, independence is a property of wiring: two power feeds are independent when they do not share an upstream breaker, two replicas when they do not share a rack, two tests when they do not share a fixture. Find the shared connection and remove it, and independence follows. The fresh call does remove a shared connection — the transcript — and it is a short step from there to concluding that the two runs are now independent. The step is wrong because the shared component in this system is not the transcript. It is the weights, and no arrangement of context windows unshares them.

What actually differs between them is the contents of the context window, and that has real consequences. The second-turn design carries the first answer's justification forward, so the check is conditioned on a chain of reasoning that already concluded. If that chain contained a wrong step — the extractor read "the notice period is thirty days" from a sentence about termination for convenience and applied it to the auto-renewal clause — the check inherits the wrong step as established context and reasons onward from it. It also inherits the social shape of the exchange: the model is being asked to disagree with something it just said, in a transcript where it said it. The fresh call carries none of that. It sees a clause and a claimed value, with no story about how the value was arrived at, so it evaluates the claim on its own terms.

There are practical differences too. The second turn is cheaper if the provider's prompt cache is hot — providers charge less for a prompt prefix they have already processed and still hold — since the source document prefix is already there; the fresh call re-sends the clause but skips the reasoning trace, which for a long chain-of-thought can be the larger of the two. The second turn can catch process errors — a step that contradicts an earlier step — because it can see the process. The fresh call cannot see the process at all, which is precisely why it is not dragged along by it.

What does not differ is the part that caused the summariser's failure. It is the same model, with the same training distribution, the same priors about when food-sector acquisitions close and what a thirty-day notice period usually attaches to, and the same systematic blind spots. If the extractor's error came from a prior — the model believes auto-renewal clauses carry thirty-day notice because that is the modal contract — then the fresh call holds that same belief and will confirm the extraction with confidence. De-anchoring from a specific bad reasoning chain is a real gain, and it is a different thing from independence. A checker whose errors are correlated with the generator's is not independent no matter how clean its context window is.

So the two designs split the error population rather than ranking it, and that split is the thing to reason from when choosing between them. Errors produced by a bad step — a value lifted out of the wrong sentence, a figure carried into the wrong row — live in the reasoning chain, and dropping the chain drops them. That is a real population and it is why the fresh call's first column is above the second turn's at all. Errors produced by a prior live in the weights, and both designs carry the weights unchanged. The Northmoor date is in the second population; so is the thirty-day notice period. That is why the second row of the table moves by three slips rather than by twenty-eight, and it is how to predict in advance what a fresh-context critic will do on a system you have not measured yet: ask which population your errors are in.

Criticism of text the model did not write#

None of this says models are bad at criticism in general. They are quite good at reviewing text they did not write, where they have no prior commitment to it and no anchoring on a reasoning chain; that is why "review this colleague's PR description" works better than "review your own answer". They are also good at applying a rule they have been handed — style guides, schema constraints, required sections — because those checks have a determinate answer that the model can look up in the context rather than assess.

That first claim needs its condition attached, and this chapter's own junior analyst supplies it. What makes somebody else's text easier to review is not the absence of a byline; it is that the errors in it were not drawn from the reviewer's priors. A person writing a PR description makes a person's mistakes, and a person's mistakes are uncorrelated with a model's, so the model's sense of what looks wrong is genuinely informative about them. Relabelling the model's own output as a colleague's changes the byline and imports none of that property, which is exactly what the flag counts showed: more complaints at the same proportion of them being right. Read the claim as being about whose mistakes are in the text, not about whose name is on it.

Nor does it say that feedback is useless. When the correct date is supplied from outside, models generally do incorporate it, though the Feedback Friction work is a useful corrective on how completely: even with correct, explicit, external feedback, models often absorb it only partially and plateau well short of the accuracy the feedback would allow, particularly when the correction contradicts a strong prior. Plan for a correction step to land most of the time rather than all of it.

And it does not say the review turn in that summariser was worthless. Five slips in 300 documents is a real catch rate. What made it a net negative was that it created a false impression of coverage across four months, and a control nobody can distinguish from a no-op is worse than an absent one, because the team stops looking.

The failure at the top of this chapter is not that the model is a poor critic. It is that the review was pointed at a question the model answers with the same machinery that produced the answer, so a well-scoring wrong date scores well twice. Rewrite the question into one whose answer sits in the context — quote the sentence, or write NOT FOUND — and the same model, on the same documents, finds 33 of 41 instead of 5. Move the check out of the model altogether, to a regex or a schema or a recomputation, and its errors stop being the generator's errors at all.

What the summariser needed, in one phrase, was a checker whose mistakes are not the same mistakes. That phrase does most of the work in every design that follows, and it turns out to be a scale with several rungs on it rather than a property a system either has or lacks — the fresh-context critic sits somewhere in the middle of it, and knowing exactly where determines what you can expect it to catch.