The judge as an attack surface

Read these first

• Judge failure modes and calibration

What you will know

• Run a prompt-injection case through a judge that receives model output and record whether quoted instructions alter its score or verdict.

  The evaluated text is untrusted input, even when it looks like a harmless answer, document, or tool result.

• Build a judge harness that keeps candidate content in a data-only field, separates rubric instructions from the sample, and tests the boundary with adversarial fixtures.

  This limits instruction confusion but does not remove vulnerabilities in the judge model, parser, tools, or surrounding application.

• Optimize a generator against a fixed judge, then compare judge scores with blinded human ratings on held-out prompts to expose reward hacking.

  A generator can learn rubric artifacts, formatting tricks, or judge-specific preferences without becoming more useful to users.

• Instrument judge and generator versions, prompts, scores, and sampled outputs so a score jump can be traced to a harness change or an exploit.

  Without an audit trail, metric drift and gaming look like genuine model improvement and cannot be reproduced reliably.

• Set a release gate that requires an independent evaluation and blocks security-sensitive decisions from judge scores alone.

  A judge is a fallible measurement channel, not an authorization boundary or proof that unsafe behavior is absent.

Chapters

1. 1

   Prompt injection against a judge

   One account's summaries pass at 100% while the rest of a suite sits near 80%, because a line in that account's document template addresses the evaluator directly. This chapter defines prompt injection in the grading path, shows the single-channel prompt that makes it work, builds the fixture suite that turns an accident into a measured flip rate per payload and position, and separates the mechanism from a rendering bug, from a calibration problem, and from a jailbreak.

   Read chapter
2. 2

   Data-only fields and instruction confusion

   Move the candidate into its own delimited field, label it data, and state in the rubric that its contents are evidence and never directions. What that buys is a lower rate of instruction confusion and somewhere to measure it from, not a boundary anything enforces — the model still reads rubric and sample through one channel. This chapter designs the containment element by element, separates a mitigation from a control by way of the prepared statement it is compared to, specifies the adversarial fixtures, and sets a pass criterion for the harness rather than for the model.

   Read chapter
3. 3

   Tool access in a judge harness

   A citation check that lets the judge retrieve any URL appearing in a candidate answer hands the choice of destination to model-generated text drawn from customer uploads, and every page that comes back is more untrusted text. This chapter works through what a four-line tool definition actually grants, why an evaluation runner is usually more privileged than the service it grades, the fetch client that moves each decision into code, and the fixtures that prove the refusals fire.

   Read chapter
4. 4

   Reward hacking against a judge

   No attacker, no instructions, nobody intending anything: a loop that keeps whatever scores higher against a fixed judge finds the judge's incidental preferences, because they are cheaper to satisfy than the qualities the rubric was written for. This chapter reads the fingerprint such a loop leaves in length and vocabulary, gives the two cheap probes that localise the gain, decomposes the winning artifact into the edits worth keeping, and explains why an ensemble of three judges inside the loop narrows the gap without closing it.

   Read chapter
5. 5

   Sample size for a human evaluation

   A flat human number is evidence only if the comparison was built to move. This chapter sizes a human evaluation against the smallest difference that would change a decision rather than against what the budget allows, sets out the design conditions — held-out prompts, blinding that survives a format tell, pairing, rater calibration — that decide what the number means before its size does, and separates a null result of 68% against 68% from a study too weak to have found anything.

   Read chapter
6. 6

   The evaluation audit trail

   When a pass rate moves overnight, three explanations fit — a genuine improvement, a harness change, or an output pattern that games the judge — and which one you can establish depends entirely on what was written to disk beforehand. This chapter specifies the per-item record that makes the question answerable: generator and judge versions, raw outputs stored beside their scores, tool calls and injection-probe rows. It then sets out the discriminating re-score, the noise floor it has to clear, and the frozen-set gate that catches a rubric edit before it is reported as progress.

   Read chapter
7. 7

   Telling injection from reward hacking

   Both failures end with a judge returning a score that does not reflect the quality of the work, which is why one action item gets written for both. This chapter separates them along the dimensions that matter — who acts, what the unit of effect is, how onset looks, which way a held-out judge points — gives the two eval-store queries that surface each, and settles what a data-only field contributes against a generator that never issues an instruction, which is nothing.

   Read chapter
8. 8

   Judge scores as control evidence

   Zero violations across 2,000 sampled outputs, or 100% clean across 5,000, is exactly the shape a rare class produces from a detector nobody has measured — and one steerable by the content it reads. This chapter works through what a clean month is worth without a recall figure, why a control the measured population can write to is not a control, and what to offer compliance instead: seeded positives that turn the judge into an instrument with stated properties, architectural controls, and a gate that reports the judge without letting it decide.

   Read chapter