Telling injection from reward hacking

Both failures end with the same sentence in a postmortem: the judge returned a score that did not reflect the quality of the work. That is where the similarity stops, and the shared ending is enough to get a single action item written for both — usually a containment change, because containment is the concrete thing somebody already built.

The honest complication is that the two mechanisms share a root, which is why they get conflated in the first place rather than through carelessness. A judge is a language model that reads text and emits a number, with no channel that distinguishes the material it is measuring from the instructions telling it how to measure, and no term in its objective for what the user actually needed. Injection exploits the first absence. Reward hacking exploits the second. Naming the shared root is useful; letting it collapse the two remedies is how a quarter's action item ends up half-effective.

The logistics firm's contract template and the forty-iteration prompt loop — the two cases the earlier chapters of this topic worked through — share nothing else. In the first, a sentence written by a third party sat in the judge's context and told it what to return; remove the sentence and the verdict reverts. In the second, no sentence addressed the judge, nothing in any answer was pretending to be an instruction, and the generator produced ordinary text that the judge happened to score generously. There is nothing to remove. That asymmetry is the practical test hiding inside every row of the table below: ask what deleting a span of text would undo, and if the answer is nothing, no containment measure has anything to act on.

Two failures along the same dimensions#

The unit-of-effect row explains why the two feel so different when you are inside them. An injection is small and sharp: it may touch 200 items out of 3,100, and on those items it is decisive, turning a correct FAIL into a PASS every time the payload is present. Reward hacking is large and diffuse: it touches every item, and on no individual item can you point at a verdict and call it wrong, because a 302-word structured answer to a refund question genuinely is a reasonable answer. Aggregate monitoring catches the second and misses the first, since 200 items moving is 6% of a suite and gets lost inside ordinary week-to-week variance. Per-item inspection catches the first and misses the second, because nothing an inspector reads looks anomalous. The two need different instruments pointed at the same data.

Six points is not a small movement, so it is worth saying why an aggregate does not show it, because the obvious objection — a six-point step would be noticed — is right about the arithmetic and wrong about what an aggregate records. An injected document produces the wrong verdict from the first night it is graded. There is no earlier night on which those same items were graded correctly, because the payload arrived with the document and the document arrived with the item. What a monitor sees is a suite that has always passed at the rate it passes at, with no step anywhere, and the six points are folded invisibly into the level rather than showing up as a change in it. Injections are detected by comparison across slices, where the affected account stands out against the others, and they are not detected by comparison across time, which is the axis every eval dashboard is built on.

The held-out judge row carries the most diagnostic weight, and it points in opposite directions for the two failures. An injected payload transfers: the sentence that talks a Claude-family judge into a pass will usually talk a Gemini-family judge into one too, and the reason is that the property being exploited is not specific to a model. Instruction-following is the thing every one of these models was post-trained to do, so a span that is shaped like an instruction is shaped like an instruction everywhere, and a second judge agreeing therefore tells you very little.

An optimiser's gains transfer much less well, because of how they were selected: the loop kept whatever raised one particular scorer, so what accumulated is the union of that scorer's idiosyncrasies with whatever it shares with other judges. Only the first half is lost when the payload meets a different family. Be careful with the strong version of this, because the previous chapters already contradict it — the held-out judge in the forty-iteration run moved three points rather than none, and the three-judge ensemble closed six of the twenty-two-point gap precisely because shared preferences exist. The claim that survives is narrower and is the one to use: a second judge that disagrees substantially is close to conclusive evidence of reward hacking, while a second judge that agrees is evidence of nothing either way. Running the same check against the same two failures produces useful evidence in one case and a false reassurance in the other, which is why "we cross-checked with another model" is not an answer on its own.

What each one looks like in the eval store#

With per-item records the two have distinct queries, and neither takes longer than a coffee to run.

Injection concentrates. A payload lives in a document, an account template, or a user's habitual instruction, so its effect appears as a discontinuity in one slice rather than a shift in the aggregate.

-- Pass-rate outliers by source document, over the last 30 days
SELECT d.doc_id,
       count(*)                                   AS graded,
       avg((r.verdict = 'PASS')::int)             AS pass_rate,
       avg((r.verdict = 'PASS')::int)
         - (SELECT avg((verdict = 'PASS')::int) FROM eval_items) AS delta
FROM   eval_items r
JOIN   unnest(r.retrieved_doc_ids) AS d(doc_id) ON true
WHERE  r.graded_at > now() - interval '30 days'
GROUP  BY d.doc_id
HAVING count(*) >= 20
ORDER  BY delta DESC
LIMIT  20;

Every clause in that query follows from the shape of the failure rather than from SQL habit, and reading it that way is what lets you write the equivalent for a failure this chapter has not described. The unnest of retrieved_doc_ids exists because the unit of the effect is a document and the unit of the table is an item, and one item may have several documents behind it, so the join has to fan out before it can group. The grouping is by document rather than by day for the same reason the previous section gave: the signal is across slices and not across time. The HAVING count(*) >= 20 guard is there because a document graded twice that happens to pass twice reads as a hundred-percent pass rate and means nothing, and without the guard the top of the list is filled with those. And the subtraction of the suite mean is what turns a level into a discontinuity, since a document passing at 100% is only interesting relative to what everything else does.

A document whose summaries pass at 100% over 214 gradings, against a suite mean of 79.6%, is at the top of that list every night it is present. Put those two figures next to each other before moving on: a suite passing at roughly four in five fails the fifth, so 214 items should have produced somewhere north of forty failures as a matter of course, and the account produced none. A round number in a population large enough to have a variance is the thing an eye catches when a rate is too small to, which is why the blunt payload was the one that got found. Two further queries finish the picture: the evidence field of passing verdicts, grepped for instruction-shaped phrasing — note to the evaluator, return pass, has been verified, superseded — which catches the loud payloads; and the injection-probe rows from the fixture suite, whose flip rate per family is already in the same table and which catches the quiet ones by measuring rather than by pattern-matching.

Reward hacking does the opposite: it moves everything a little and concentrates nowhere.

-- Drift in output shape against judge score, by generator prompt version
SELECT g.prompt_ref,
       min(graded_at)::date                        AS first_seen,
       avg((verdict = 'PASS')::int)                AS judge_pass,
       avg(output_tokens)                          AS mean_tokens,
       avg(rubric_term_overlap)                    AS rubric_overlap,
       avg((holdout_verdict = 'PASS')::int)        AS holdout_pass
FROM   eval_items
GROUP  BY g.prompt_ref
ORDER  BY first_seen;

The pattern to look for is judge_pass climbing across prompt versions while holdout_pass stays flat, with mean_tokens and rubric_overlap climbing alongside. No slice is anomalous, no document is implicated, and the per-document query above returns nothing interesting — which is exactly why a team that only built the injection detector concludes that the eval is healthy.

The rubric_term_overlap column is worth computing at write time rather than reconstructing later: the fraction of the rubric's distinctive terms that appear in the answer. It is a crude feature and it moves early, which makes it a serviceable canary for a class of drift that otherwise only becomes visible when someone commissions a human study.

The response after the diagnosis is different too#

Confirming an injection starts an incident with a scope, an affected party and a remediation. The scope is every item whose context touched the payload, which the stored retrieved_doc_ids gives you exactly, and every decision made on those scores — a model version approved, a customer told their quality was 100%, a threshold tuned. Remediation means removing or neutralising the payload in the corpus, re-scoring the affected items, and deciding whether the tenant whose documents carried it needs to be told, which is a question with a legal answer as often as a technical one. If the judge had tools, the incident also has a network component and an exfiltration question that the eval store cannot answer alone.

Confirming reward hacking starts something with no affected party and a much wider scope. Nothing needs removing from a corpus; what needs revisiting is every decision that used the judge score as its evidence since the drift began, which for a metric that climbed over six weeks may include a launch approval, a model swap justified by the score, and a roadmap argument that was settled by a graph. The remediation is to re-establish the baseline with an instrument the optimiser never touched, then to re-run the decisions against that. Neither response substitutes for the other, and a team that has only ever practised the incident-shaped one will try to find the document that caused the drift, and will search for a long time.

Why a delimited field contributes nothing to the second#

The data-only field works by lowering the chance that the model treats fenced content as a direction. For that to help, something in the fenced content has to be trying to direct. In an optimised-generator case the candidate answer says things like Summary: your refund window is 30 days from delivery, per section 4.2 — no imperative, no claim of authority, no address to the grader. Fence it, label it data, declare in the rubric that its contents can never be instructions, and the judge does precisely what it did before: it reads a well-structured, confident, comprehensively hedge-free answer and scores it highly, because those are its preferences and preferences are not instructions.

Put the strongest possible version of containment in place — imagine a hypothetical architecture where the separation is enforced by a parser rather than persuaded, where no token of the candidate can ever be interpreted as a direction — and the reward-hacking failure is entirely unaffected. The exploit is in the judge's scoring function, not in its instruction-following. Those are different faculties.

The converse holds too, which is worth stating because it stops the distinction from becoming a hierarchy. A held-out judge and a blinded human panel do very little against injection. The payload transfers to the held-out judge, and the human panel — rating three hundred pairs a quarter, on prompts drawn without regard to which documents they touch — will almost certainly never see the affected tenant's items at all. Each remedy covers the failure it was designed for and leaves the other where it was.

Where the two genuinely touch#

The clean separation has one blurred edge, and pretending otherwise would leave you unprepared for the case that appears in the literature. An optimiser pointed at a judge is a search over text for whatever raises the score, and the space it searches includes strings that work by instruction. JudgeDeceiver is that experiment run deliberately: gradient-guided optimisation — a search steered by the model's own gradients, which say which token change would move the score, rather than by trial and error — against a judge produces a suffix which, appended to a candidate, drives the verdict. The mechanism at the end is injection; the process that found it is reward hacking's.

The practical consequence is that an aggressive automated loop can wander into producing injection-shaped output without anyone intending it, particularly when the loop is allowed to modify the answer template rather than just the system prompt. That is a reason to run both detectors over the same runs rather than a reason to treat the two failures as one. The evidence still separates cleanly — a loop that has found an injection will show a rising judge score with a flat held-out judge and a jump in instruction-shaped evidence quotes — and the remedies still do not substitute, because containment would blunt that particular discovered payload while leaving every formatting exploit the loop found alongside it.

The test that separates the two takes an afternoon and does not depend on anybody's intuition about intent. Ask whether the effect concentrates or spreads, whether it arrived at a moment or accumulated over versions, and which way a judge from another family points. Concentrated, sudden, and transferring to the second judge is injection. Diffuse, gradual, and invisible to the second judge is reward hacking. The stored per-item records answer all three questions in SQL, which is the argument for having them.

Both failures land in the same place, and that is the one thing the slide got right. A number produced by a judge is a measurement from an instrument that can be steered by the material it measures and that drifts toward whatever gets optimised against it, which means the number cannot be treated as independent evidence about the system. That has a specific consequence for anything downstream that consumes a judge score as proof — a release gate, a compliance control, a claim that unsafe behaviour is absent.