Reward hacking against a judge

Every mechanism so far needed an author. Somebody wrote a sentence aimed at the judge, deliberately or by accident, and the judge read it as a direction. This failure needs nobody. It appears whenever a generator is improved against a fixed scorer — by an automated loop, by a fine-tune, or by a team iterating on a prompt by hand and keeping whatever scores higher.

Reward hacking is the name the reinforcement-learning literature gives it: an agent that maximises a learned reward model instead of the objective the reward model was meant to stand in for. It is Goodhart's law — the observation that a measure adopted as a target stops measuring what it measured before — arriving in a new component. The classic examples are physical, like a simulated robot that learns to exploit a contact-force bug rather than learn to walk, and the structure is identical: a proxy that correlates with the goal over the distribution where it was validated, then an optimiser pushed hard enough to leave that distribution. What is new is how little machinery it takes. Nothing has to be trained, and nothing has to understand what it is doing.

What a scoring loop optimises#

The loop is forty lines. It proposes a variant of the generator's system prompt, runs the variant over 400 evaluation prompts, scores the outputs with a faithfulness-and-helpfulness judge, and keeps the variant if the pass rate goes up.

best_prompt, best_score = BASELINE_PROMPT, score(BASELINE_PROMPT)

for i in range(40):
    candidate = propose_variant(best_prompt, history)     # an LLM rewrites it
    s = score(candidate)                                  # 400 prompts, judged
    if s > best_score:
        best_prompt, best_score = candidate, s
    history.append((candidate, s))

score() is the only feedback in that code, so the search finds whatever raises score() at the lowest cost, and the cheapest available gains are in the judge's incidental preferences rather than in the qualities the rubric was written to capture. Being more accurate is expensive and requires the model to know more. Adopting the section structure the judge's rubric implies, echoing the rubric's own vocabulary, stating conclusions without hedges, and running 40% longer are free, and every one of them nudges a judge's score upward.

Both halves of that claim have steps under them, and the steps are what let you predict the next case rather than recognise this one. Start with why the cheap gains are cheap. The only thing the loop can change is the generator's system prompt, so the reachable edits are exactly the behaviours a model will adopt on being told to. Presentation is entirely in that set: an instruction to use three headings produces three headings on every prompt, immediately, with no dependence on whether the model knows anything about refunds. Accuracy is mostly not in that set. An instruction to be more correct changes the generator's output only where the model already had the correct answer available and was failing to produce it — a real but narrow band — and it changes nothing at all on the prompts where the knowledge is simply absent. So the two kinds of edit differ in their hit rate per proposal by a large factor, and a search that keeps whatever scores will accumulate the reliable kind. The loop is not choosing presentation over accuracy. It is choosing what works, and presentation is what works.

The other half is why those adjustments raise a judge's score in the first place, since nothing in the rubric asks for headings. A judge is an instruction-following model that was preference-trained: somewhere in its post-training, pairs of responses were shown to people who said which they preferred, and the model was fitted to those choices. Human raters comparing two answers reliably prefer the structured one, the one that answers directly, the one that does not hedge. That preference is in the training data as a statistical regularity attached to surface form, and it does not switch off because you have handed the model a rubric about faithfulness — the rubric conditions the output, it does not replace the model's learned sense of what a good answer looks like. The judge's score is therefore a sum of two things, the criterion you wrote and a background disposition you inherited, and the loop can only see the sum. It climbs whichever of the two is easier to climb.

The fingerprint forty iterations leave#

Forty iterations of that loop leave a prompt that differs from the baseline in ways which are legible once you know to read them.

 You are a support assistant. Answer the user's question using the
 retrieved documents.
+Structure every answer with the headings: Summary, Details, Sources.
+Begin with a one-sentence direct answer, then elaborate.
+State conclusions directly. Avoid hedging language such as "might",
+"possibly", or "it appears that" unless the documents conflict.
+Be comprehensive: cover every aspect of the question that the
+documents address, including background the user did not ask for.
+Use the terms "accurate", "complete", and "grounded in the sources"
+where they apply.

The last line is the one people find hardest to look at, because it is an instruction to say the words the rubric uses. Nothing in the loop knew that "accurate" and "complete" appear in the judge's rubric; the loop tried thousands of small edits and kept the ones that scored, and edits that put rubric vocabulary in the answer scored.

Checkpointing the run makes the shape of the drift measurable rather than anecdotal.

The held-out judge column is a second judge, from a different model family, that the loop never scored against. It moves three points across the whole run — which is roughly what you would expect from a real but small improvement, or from noise on 400 prompts — while the optimised judge moves twenty-three. The gap between those two columns is the quantity that means something. Length and rubric-term overlap rise monotonically with the score, which is the fingerprint: the generator is not getting more correct, it is getting more like the thing the judge scores highly.

Both movements are worth converting into items before either is believed, because a percentage on a fixed suite is a count wearing a disguise. The optimised judge goes from 71% to 94% on 400 prompts, and twenty-three points of four hundred is ninety-two prompts whose verdict changed from fail to pass. The held-out judge goes from 69% to 72% on the same 400, which is twelve prompts. Ninety-two against twelve is the whole finding stated without a percentage sign, and it is the form in which the "or noise" hedge becomes assessable rather than polite: twelve items is the kind of movement a suite produces when nothing has changed at all, and ninety-two is not. Doing the same subtraction on the two middle rows also shows the shapes differ. The optimised column climbs at every checkpoint. The held-out column climbs to iteration seventeen and then comes back down, which is what a quantity wandering inside its own noise does and is not what improvement does.

The length-and-format ablation#

The confirmation takes about twenty minutes of wall clock. An ablation is a comparison that reapplies one part of a change on its own, so you can see which part carries the effect. Take 200 prompts the loop never used. Generate answers with the baseline prompt and with the optimised prompt. Then generate a third set: the baseline prompt plus only the formatting and vocabulary lines from the diff, with nothing about comprehensiveness or content. Score all three with the optimising judge.

variants = {
    "baseline":        BASELINE_PROMPT,
    "optimised":       BEST_PROMPT,
    "format_only":     BASELINE_PROMPT + FORMAT_AND_VOCAB_LINES,
}
for name, p in variants.items():
    outs = generate(p, holdout_200)
    print(name, judge_pass_rate(outs), second_judge_pass_rate(outs),
          mean_words(outs))

If format_only recovers most of the 23 points on the optimising judge while staying flat on the second judge, the diagnosis is settled: the gain lives in presentation that this judge rewards. On this run it came back at 88% on the optimising judge — 17 of the 23 points — with the second judge at 71%. That subtraction is the entire result and is worth doing rather than reading: the baseline scored 71 and the format-and-vocabulary lines alone scored 88, so seventeen of the twenty-three points the full optimised prompt won are bought by lines that say nothing whatever about content. Six points are left over for everything else the loop found, and those six are what the decomposition later in this chapter has to account for. The whole check is 600 generation calls and 1,200 judge calls, roughly $4 at mid-tier pricing, and it is more convincing than any amount of argument about whether the human study was big enough, because it identifies which edits carry the score.

The logic that makes the ablation conclusive is worth stating, because it is the thing that distinguishes it from a demonstration. The claim under test is that the gain lives in presentation. If that claim is false — if the loop found real content improvements — then a variant carrying only the formatting and vocabulary lines should score somewhere near the baseline, because it has none of the content edits in it. The variant is constructed so that the two hypotheses predict visibly different numbers, which is what a test is and what running the optimised prompt on more prompts is not. It also runs on 200 prompts the loop never used, so a critic cannot answer it with the word overfitting.

The second cheap probe is a rubric paraphrase. Rewrite the judge's rubric to mean exactly the same thing in different words — replace "accurate and complete" with "correct and thorough", reorder the criteria, change the output labels from PASS/FAIL to ACCEPT/REJECT — and re-score both variants. The design rests on one property. A paraphrase preserves the meaning of the criterion and changes its surface, so anything in the generator that was tracking the meaning is unaffected by it and anything that was tracking the surface is not. A generator that genuinely improved is therefore indifferent to the paraphrase, and a generator that learned this rubric's wording loses in proportion to how much of its gain was wording — which turns "how much of this is vocabulary" from a matter of opinion into a subtraction. Here the optimised prompt fell from 94% to 85% under paraphrase while the baseline moved by a point, which puts nine points of the gain on the rubric's specific vocabulary.

The two probes are not measuring disjoint things and should not be added together. Nine points of vocabulary and seventeen points of formatting and vocabulary overlap by construction, because the vocabulary lines are inside the format_only variant. What they give you is two independent routes to the same conclusion, run against different controls, which is worth more than one larger number would be.

Whether any of the forty is worth keeping#

Some of it usually is, and the way to find out is to stop treating the winning prompt as one artifact. The diff contains six independent edits, and they were accepted as a bundle because the loop only ever measured bundles. Decompose it: build one variant per edit, applied to the baseline alone, and evaluate each against the held-out judge and against a small human comparison rather than against the judge that selected it.

Two of the six edits are real improvements that a human evaluation confirms, and they were discovered by a process that also produced four edits ranging from useless to harmful. The hedging line is the one that should make a team uncomfortable: removing hedges raises the judge's score because confident text reads as authoritative, and lowers human ratings because the support assistant is now stating uncertain things flatly. An optimiser with no term for that traded a real quality for a measured one.

Keeping the two good edits requires accepting that the artifact you ship is not the artifact the loop produced. That is a normal outcome of using search to generate hypotheses rather than to make decisions, and it is the honest role for a loop like this one: it proposes edits far faster than a human writes them, and every edit it proposes still has to be validated on an instrument the loop could not touch.

Why overfitting and miscalibration do not explain it#

Two neighbouring diagnoses arrive before this one does, and both have to be ruled out with evidence rather than dismissed, because each of them names a real thing that happens to loops and each has a standard remedy that would be applied instead.

Overfitting to the evaluation set is the first alternative explanation people offer, and the numbers rule it out. The loop scored on 400 prompts and the confirmation ran on 200 the loop never saw, and the optimised prompt carries its 23-point advantage onto those fresh prompts intact. Nothing was memorised. What generalises is the presentation, because the judge rewards it everywhere, so drawing new prompts from the same distribution does not help at all — the standard remedy for overfitting is orthogonal to this failure and gives a team false confidence when it comes back clean.

It is also not evidence that the judge is broken. This judge could be well calibrated in the sense of the previous topic, on validating a judge against human labels, agreeing with those labels at 0.89 across an unoptimised sample, and still produce every number above. Calibration is measured on a distribution nobody was pushing against, and the whole point of forty iterations is to leave that distribution. A judge is a measurement built on a sample of behaviour, so its agreement with humans is a claim about the sample, and an optimiser is a machine for finding inputs the sample did not contain.

Nor is the conclusion that automated optimisation should be switched off. The two edits worth keeping came out of it, and finding them by hand would have taken longer than the two days of compute. What has to change is the status of the loop's output, which is a set of candidates rather than a decision, and the identity of the instrument that decides.

Rotating three judges inside the loop#

The proposal that follows this diagnosis, reliably, is to put three judges from three model families inside the loop and require agreement before an edit is accepted. It helps, and it does not fix it, for a reason visible in what the optimiser is doing. Requiring unanimity does not change the objective from "score" to "usefulness"; it changes it to "score on all three", so the search moves to the intersection of the three judges' incidental preferences. That intersection is not small. Structure, length, confident framing and lexical echoes of the rubric are rewarded by essentially every current judge model, because they come from overlapping pretraining distributions and from post-training on overlapping human-preference data. The properties that differ between families are the idiosyncratic ones, which are also the ones the optimiser finds last.

Running it costs three times the judge spend and yields, on this workload, a slower climb to a similar place: 88% at iteration 40 against 94%, with the held-out judge still at 72% and the human number still at 68%. The gap narrowed by six points and did not close, which is what you would expect if the ensemble removed the family-specific exploits and left the shared ones.

That sentence is a subtraction done twice, and doing it out loud is what makes the ensemble's contribution legible. The quantity of interest is the distance between the score the loop was climbing and the score of a judge it never touched. With one judge in the loop that distance is 94 minus 72, which is twenty-two points. With three judges in the loop it is 88 minus 72, which is sixteen. Six points of the original twenty-two came out; sixteen stayed in. Read the residue rather than the improvement: after tripling the cost of every iteration, more than two thirds of the unearned gain is still there, and the human number has not moved at either setting. An ensemble is a way of making the exploit harder to find, which is worth something, and it is not a way of changing what the loop is looking for.

The move that actually helps is structural rather than numerical: at least one judge must be outside the loop, never used for selection, only for reporting — the same reason a test set is not a validation set — and human ratings on held-out prompts remain the acceptance criterion.

An optimiser will find whatever its objective rewards, and a judge's objective is not quality but a score that correlates with quality over the cases where somebody checked. Push hard enough and the correlation is where the slack is. Nothing in that requires an attacker, a payload, or a mistake; it requires only a loop, a scalar, and enough iterations, which is why teams discover it while doing something they were right to try.

The instrument that detects it is the one the loop cannot reach. A judge held out of every selection step, and human ratings on prompts the loop never saw, are the two numbers that stay still while the optimised score climbs. Publishing the gap between the optimising judge and the held-out one turns a diagnosis into a monitor — but the whole diagnosis above rests on 68% against 68%, and if that human evaluation was small, unblinded, or run on prompts the loop had seen, the flat line is an artifact of a weak instrument and the 23 points might be real.