Answer quality across reflection rounds

Reflection, in the sense used here, is a loop: generate an answer, critique it, revise it against the critique, and repeat some fixed number of times, returning what the last round produced. It generalises the single critique turn into an iterative one, on the intuition that if one pass of review improves an answer then several passes improve it several times.

That intuition holds for a small number of rounds and then reverses, and the reversal is not gradual noise — it is two specific mechanisms, both of which can be measured on your own data in an afternoon. The measurements in this chapter come from a documentation-QA agent answering questions about an internal deployment CLI, scored against a frozen 240-case eval at three runs per case.

A round asked to find fault will find one#

Here is one question tracked through such a loop. The user asked whether deployctl promote validates that referenced secrets exist before promoting.

ROUND 1  (correct)
  No. `promote` does not check secret references. If a referenced secret is
  missing, the promotion succeeds and the pods fail at start-up. Validate
  with `deployctl lint --secrets` beforehand.

ROUND 2  (critique: "the answer does not mention configuration options")
  No, not by default. `promote` does not validate secret references unless
  strict reference checking is enabled. Run `deployctl lint --secrets`
  first, or enable strict mode.

ROUND 3  (critique: "the previous answer is vague about how strict mode is set")
  `promote` performs no secret validation unless `--strict-refs` is passed
  or `strictRefs: true` is set in `deploy.yaml`. With strict references
  enabled, promotion aborts on a missing secret.

ROUND 4  (critique: "the previous version omits default behaviour and migration")
  The previous answer was insufficient because it did not state the default.
  `--strict-refs` defaults to false for backwards compatibility. Projects
  created after the v4 CLI release have `strictRefs: true` in the generated
  `deploy.yaml`. To adopt it on an older project, add the key and run a
  canary promotion first.

There is no --strict-refs flag. There is no strictRefs key. Round 1 was correct and complete. By round 4 the answer has a flag, a default, a version history and a migration procedure, all of it fluent, none of it real, and the hedging that made round 1 easy to disbelieve is gone.

Across the frozen eval the same loop took accuracy from 78% at round 1 to 74% at round 5, so this transcript is not an unlucky case selected to make a point. It is the shape of the aggregate, written out on one question. One feature of round 4 is worth noticing on its own, because it recurs across the eval: the answer opens by explaining the previous version's insufficiency. That is the loop reporting on its own progress inside the artefact it is producing. A round asked to improve something will narrate an improvement whether or not it made one, and to a reader skimming for evidence of care, the narration is the most persuasive part of the output.

The critique step in that loop was a prompt ending in identify the weaknesses in the answer above. Given an answer with no weaknesses, that instruction still has to be completed, and the highest-probability completion of "identify the weaknesses" is a list of weaknesses. Nothing in the sampling process is willing to return an empty list on the grounds that there is nothing to say.

The step underneath that is a presupposition, and it is why the instruction is dangerous rather than merely unhelpful. Identify the weaknesses in the answer above takes for granted that weaknesses are there; the sentence admits no reading on which none exist. Completing it means producing a list, and the model is not adjudicating whether the presupposition holds — it is continuing a document, and documents that open that way go on to contain lists of weaknesses. So the failure is not that the model looked at a correct answer and misjudged it. It is that the request left no well-formed way to report having found nothing, and an instruction with no null output is an instruction that will always produce work.

The measurement is easy to make and worth making on your own data. Take 100 answers that a human has confirmed correct, run one critique-and-revise round on each, and count what happens.

The first row is the loop as most people write it, and it is editing a correct answer two-thirds of the time. Twenty-two of those hundred confirmed-correct answers came back wrong. The third row is the lookup-shaped check from the earlier chapters applied to the loop: give the round a task with a determinate answer that lives in the context, and the manufacture rate falls to something a system can live with. It also gives the round permission to do nothing, which the first instruction never does.

Reading the three count columns across rather than down is what separates the two rows that otherwise look like the same idea at different strengths. Divide the last column by the one before it and what comes out is the share of edits that did damage. The first row changed 68 answers and broke 22 of them: roughly one edit in three turned a confirmed-correct answer wrong. The second row changed 31 and broke 9, which is close to the same proportion, a little under one in three. The third row changed 12 and broke 2, about one in six.

Those three fractions say what each instruction actually bought. Adding "if there are none, output NONE" cut the critiques by more than half and the edits with them, and since the harmful share stayed roughly where it was, the absolute damage came down in step. What that instruction supplied was a licensed way to say nothing — not a better eye. The third instruction cut the volume further and cut the harmful share on top of it, which is a different kind of change: it permits an edit only where a specific claim failed a lookup, so the edits it does make are aimed at something. The second row moves a threshold. The third changes the question, and that is a difference in kind rather than in strength.

The gap between rows one and three is roughly the whole of the difference between a critique loop that helps and one that degrades. It is not a subtlety of model capability; it is what the round was asked for.

Each round conditions on the last round's justifications#

The second mechanism is what turns a single bad edit into the round-4 answer above, and it is a property of the context rather than the instruction.

The dotted edge that is missing is the whole diagram. Every round after the first draws on exactly the same source material as round 1, plus a growing pile of the model's own assertions and the reasons it gave for them. Round 3 is not deciding whether --strict-refs exists; it is conditioned on a transcript in which --strict-refs was already introduced and defended, and the probability of contradicting an assertion that has been made twice and justified once is low. The flag becomes a premise, and subsequent rounds elaborate premises rather than test them.

That also explains the hedging. Round 1 says "No" with a caveat because the model's uncertainty over the answer is genuinely spread. By round 4 the context contains three increasingly detailed statements of the same claim, so the distribution over the next answer has collapsed onto them, and confident text is what you get from a collapsed distribution. The output gets more assertive as a direct consequence of iteration, independent of whether it is getting more correct. Answers that read as more careful are, in this loop, evidence of more rounds rather than of more accuracy.

What that confidence is tracking is worth stating exactly, because it is where a reader's belief most often goes wrong. A hedge appears in an answer when the continuations available to the model disagree with each other about what the answer is; the hedge is that disagreement surfacing as text. After three rounds the transcript holds the same claim stated three times with reasons attached, so the continuations now agree — but they agree with the transcript, not with the documentation. The assurance in a round-4 answer is a reading of how internally consistent the conversation has become. It is at no point a reading of how well supported the claim is, and nothing on the surface of the output separates the two.

Both mechanisms compound: mechanism one keeps injecting new claims, mechanism two makes each injected claim harder to remove.

The disappearing hedges are worth treating as a harm in their own right rather than as a stylistic side effect. Users calibrate on hedges. An answer that says "not under the current policy, though I would check with the platform team" gets checked; an answer that states a default value and a migration procedure does not, because it carries every surface signal of having been looked up. The round-1 answer to the promote question would have sent a careful engineer to deployctl --help within a minute, and the round-4 answer sends them to write a deploy.yaml key that the CLI will silently ignore. Measured across the eval, the loop moved answers from 44% hedged to 6% hedged while accuracy fell four points, which is a system getting less accurate and more believable at the same time, in that order of magnitude.

Truncating the context between rounds does not fix mechanism two, and it is the first thing people try. Passing only the latest answer forward, with the earlier rounds dropped, removes the accumulated justifications and does reduce the elaboration somewhat — on this eval, invented entities at round 5 fell from 38 to 26 — while leaving the claim itself in place, since the latest answer is precisely where --strict-refs now lives. What actually removes an invented entity is a check that can see it is not real, which means an inventory of real flags and commands, which the CLI's own help output provides for free.

The size of that drop is the diagnosis rather than a disappointing result. Cutting the context removed about a third of the invented entities and none of the claim that produced them, which is exactly what you would expect if the accumulated justifications drive the elaboration while the seed rides forward in the latest answer. Anything the loop keeps in order to iterate on is a channel for the second mechanism, and the latest answer has to be kept or there is no loop. So the fix cannot come from pruning inside the loop. It has to come from a fact the loop could not have generated — an inventory of real flags is one — because only an outside fact can retire a claim, where truncation can at best slow its growth.

Why a model judge reports monotone improvement#

Anyone who has run one of these loops has probably been told, or has said, that accuracy improves with each round. That claim is unusual enough to demand its artefacts, and the artefacts to ask for, in order, are the eval set and its size, the grader — whatever decides an answer is correct, whether a string comparison, a rubric applied by a person, or another model call — the number of runs per configuration, and the per-round scores rather than a first-versus-last comparison.

The reason to ask for the grader first is that a model grader has a bias pointing straight at this loop's output. Asking a same-family judge which of two answers was better made it prefer the round-5 answer to the round-1 answer in 74% of cases, on a set where round 1 was in fact correct 78.3% of the time and round 5 only 74.2%. A judge that rewards elaboration will report monotone improvement from a loop whose only reliable output is elaboration. The other likely explanation for such a claim is that it was an impression formed by reading a handful of outputs, and the two are hard to tell apart after the fact, which is why the per-round scores matter more than the conclusion drawn from them.

That preference rate is worth one more look, because it distinguishes a biased judge from a merely unreliable one. A judge with no systematic lean, comparing two answers of genuinely similar quality, lands near an even split. A preference running at roughly three cases in four is not noise; something in the answers is driving it consistently in one direction.

The per-round figures later in this chapter — one row per round, carrying accuracy, answer length, hedge rate and invented-entity count — identify what. Length, hedging and invented entities each move monotonically across the five rounds, in one direction and never back. Accuracy is the only column that turns around. A grader reporting that the loop improves monotonically is therefore reporting one of the three monotone columns faithfully, and it cannot be the column it believes it is reporting.

Where iterative refinement did work, and why#

The loop pattern comes from a family of 2023 results — Self-Refine, Reflexion and their descendants — that did report real gains, and the difference between those setups and this one is not subtle once you look for it. Reflexion's loop ran on tasks with an environment: code that either passed its unit tests or did not, a game that either was won or was not, a search that either found the item or did not. Each round received a fact from outside the model. Self-Refine's stronger results were concentrated on tasks with a checkable or preference-shaped objective, where the feedback had something to attach to.

A documentation-QA loop with fixed retrieval has no such channel. Round 3 knows exactly what round 1 knew. Under those conditions the loop is a sampler over the model's own distribution with an ever-longer prefix of its own assertions, and Large Language Models Cannot Self-Correct Reasoning Yet is the result that says what happens: without external feedback, the second pass is on average worse than the first, and the published gains that suggested otherwise were largely produced by loops that used the true label to decide when to stop.

The experiment that finds the peak#

The measurement is cheap because one run produces every data point: log the answer at the end of every round, then score all five snapshots against the same frozen eval. The loop does not need to be re-run per configuration.

def run_loop(question: str, rounds: int = 5) -> list[str]:
    answers, ctx = [], build_initial_context(question)
    for _ in range(rounds):
        answer = generate(ctx)
        answers.append(answer)
        ctx = append_critique_and_answer(ctx, critique(ctx, answer), answer)
    return answers          # snapshot per round, scored independently

# 240-case eval, 3 runs per case, scored on exact-match + an
# invented-entity check against the CLI's flag inventory:

Round 2 is the peak, and the shape after it is a straight line down accompanied by three columns that all move together: longer, less hedged, more invented entities. The invented-entity count is the column worth instrumenting permanently, because it is computed against a static inventory of real flags and commands, so it costs nothing and it is the leading indicator — it starts climbing at round 3, a round before the accuracy loss shows up as a headline.

The word headline there is carrying the timing, and it repays unpacking. Round 3's accuracy of 79.6% is still above round 1's 78.3%, so anybody comparing the loop's output against its own first round at that point sees a gain and has no reason to look further. The first round on which that comparison turns negative is round 4. The invented-entity column had already gone from 5 to 14 a round earlier. An instrument that moves before the metric everyone is watching moves is worth more than a more accurate instrument that moves at the same time, and this one costs a string comparison against a list you already have.

Why the curve has a peak at all is the part worth being able to reconstruct, because it follows from two quantities moving in opposite directions rather than from anything specific to this CLI. A round's net effect is the real faults it repairs minus the correct answers it damages. At round 1 there is a genuine population of wrong answers — a bit over one in five of the eval — so a round asked to find fault has real faults available to it, and some of what it edits are corrections. That is where round 2's gain of a little over three points comes from. But a round that succeeds shrinks the population it was feeding on. By round 3 most of what remains is already correct, so the same instruction has less to repair and more to spoil, and the repairs term falls every round. The damage term does not fall with it, because the instruction's willingness to edit was never conditional on anything being wrong.

The second mechanism then pushes the damage term the other way as the first term decays. Each round's inventions ride forward as premises, so the later rounds are editing a cleaner set while conditioned on claims the earlier rounds made up. That is the shape the columns record: invented entities roughly triple between round 2 and round 3 and keep climbing, while accuracy gives back about two points, then close to three, then about two and a half. The peak exists because one term decays while the other grows, and nothing inside the loop is arranged to notice when they cross.

Retrieval is not the whole story#

The obvious reading of the invented flags is that retrieval is weak — the docs for promote were thin, so the model filled gaps. Retrieval probably is weak, and improving it is worth doing. It is not a substitute for capping the loop, because the loop is what turns a single gap-filling sentence into four rounds of documented behaviour with defaults and a migration path. Round 1 produced no invented flag on this question at all; the flag arrived at round 2 and was elaborated three more times. With retrieval improved and the loop untouched, the amplifier is still there, waiting for the next question whose docs are thin.

The other thing worth being clear about is that none of this argues against iteration as such. A loop with a real external signal each round — tests executed, a schema validated, a fresh retrieval issued against the claim just made — behaves differently, because something outside the model's distribution enters the context at every step.

What a reflection loop without such a signal produces, past its peak, is elaboration: more words, more structure, more confidence, and more invented detail, all of it moving in the same direction and all of it easy to mistake for quality. The peak is cheap to locate and it is specific to a model version, a prompt and a retrieval corpus, so it is a number to re-measure rather than a constant to remember. Cap the loop below it, ask each round for a lookup rather than an appraisal, and count invented entities against a real inventory, and the loop keeps the region where it helps and loses the region where it does not.

Which raises the question of what an external signal looks like when the task is not code with tests, and what it takes for a revision decision to be defensible after the fact rather than merely well-intentioned.