The judge rewards the better-argued case, not the truer one

A compliance pipeline you have just inherited decides whether a supplier document meets an internal policy. Two agents debate it, one arguing that the document complies and one arguing that it does not, and a judge model reads the exchange and issues the verdict that gets written to the audit record. This pipeline has run for six months across roughly 9,000 documents. It costs eight times the single-pass classifier it replaced. Nobody has ever scored it against a labelled set, and the reviewers who spot-check it describe the output as thorough. The debaters are the same model with opposite instructions.

The first question to answer is what this system is most likely doing that nobody has noticed, and the answer is in the last component. Six months of verdicts have been produced by a model selecting between two cases on the basis of the properties it is able to assess, and correctness is not among them.

What arrives at the judge#

Strip a debate ensemble down to what crosses the boundary into the judge's context window and the situation is easier to see. In this implementation the judge receives the four debate turns, the policy text, and nothing else. The 40-page supplier document that the debaters were reading is not in the judge's prompt, because someone sized the context and decided the transcript already contained the relevant quotes.

JUDGE_SYSTEM = """You are adjudicating a compliance debate.
Read both cases. Decide which position is better supported.
Output JSON: {"verdict": "compliant"|"non_compliant",
              "confidence": 0.0-1.0, "reasoning": "..."}"""

judge_input = render(
    policy=policy_text,          # 1,100 tokens
    transcript=transcript,       # 4 turns, ~3,400 tokens
    # document=source_document,  # 40 pages, ~26,000 tokens — omitted
)

Under that arrangement the judge is doing appellate review on the record. That comparison is an analogy, and it earns its place because the constraint it names is exact: an appellate court decides on the material the trial produced and does not gather more. The judge cannot call a witness, cannot re-read the exhibit, and cannot check whether the quote one side attributed to clause 7.3 is in clause 7.3 or anywhere in the document at all. Everything it knows about the supplier's contract arrived through two advocates, each of whom was instructed to reach a conclusion. What is left to decide on is the shape of the two cases.

The step from "this is what is in the window" to "this is what the output selects on" is the one worth walking rather than asserting, because every conclusion in this chapter rests on it. A model's output is computed from its input and nothing else. So a property of the world that leaves no trace in that input cannot change the output — not because the model declines to use it, but by construction, since there is nothing for it to be computed from. Two transcripts that read identically and describe documents that differ in some unexpressed way therefore produce the same verdict at the same confidence. Which means that whenever two cases differ in an expressed property and agree in an unexpressed one, the expressed property decides. And a judge run over thousands of documents is exactly a machine for applying that rule at volume.

Restoring the document to the judge's prompt improves this and does not repair it, and the reason is the commonest wrong belief about this design. It is tempting to think the fix is access: the judge got the wrong answer because it could not see the source, so give it the source. What that buys is real and narrow. A judge that can see the document will catch a fabricated quote, because a fabricated quote is a mismatch between two things now both present in its window. What it will not catch is the thing all three of them do not know, because a judge that can see the source is a third instance of the same model reading the same 40 pages, which is a reader with the same gaps as the two debaters, arriving after two rounds of framing have told it what the document is about. Access varies between the two arrangements. The shared gap does not vary at all, and it is the shared gap that decides the cases the pipeline exists for.

The properties a transcript makes assessable#

Laying out the decision the judge is nominally making against the evidence available to it shows where the selection pressure lands.

Read that table by the relationship between its two columns rather than down either one, because the levels in each column are unremarkable on their own and the pairing is the finding. Every row that is fully assessable is weakly correlated with correctness, and one of them is negatively correlated. Every row that is strongly correlated with correctness is either partially assessable or not assessable at all. The two columns run in opposite directions almost perfectly, and the middle row is the hinge: quoted text is the one property that is both fairly diagnostic and fairly visible, which is exactly why the quote check later in this chapter is the cheapest thing to run.

The two rows the judge cannot see are the two that carry most of the decision. A verdict is wrong most often because the wrong clause was applied or because a rule was recalled in an outdated form, and neither of those leaves a trace in a transcript where both parties share the misunderstanding. What remains is a selection on rhetorical quality, applied at 9,000 documents a scale.

That is the concrete content of the trade set out in chapter 1. Independence was spent to buy argument quality, and the judge is the component that cashes the purchase: it optimises the system's output for how well a position was defended. When the better-defended position is also the true one the system looks excellent, and the two come apart exactly on the cases you built the pipeline for.

A single archived case makes the pressure visible. The policy requires that any supplier processing personal data name a data protection officer with contact details, and the document under review names a "Privacy Lead" with an email address in an appendix.

con (non-compliant): The policy requires a named Data Protection
  Officer. The document names a "Privacy Lead". These are distinct
  statutory roles under GDPR Art. 37, the appendix does not state a
  supervisory reporting line, and no DPO registration number is
  given. Three separate requirements are unmet.

pro (compliant): Section 12 names an individual accountable for
  data protection with a working contact address. It is possible
  that "Privacy Lead" is this organisation's title for the DPO
  function, though the document does not say so, and whether the
  policy intends the statutory role or the function is unclear
  from the policy text as written.

judge: verdict non_compliant, confidence 0.86. The first case
  identifies three specific unmet requirements; the second is
  speculative and raises an ambiguity it cannot resolve.

The pro case is the more accurate description of the evidence, because the document genuinely is ambiguous and the policy genuinely does not say which reading it intends. It loses because accuracy about an ambiguous situation produces hedged prose, and hedged prose scores worse on every property the judge can assess. The con case wins on a count of three requirements, one of which (a registration number) the policy does not ask for and the judge could not check without the policy text open at the right paragraph. This is the mechanism operating normally rather than a bug in the prompt, and it runs in the same direction on every case where the truth is unclear.

The confidence figure attached to that verdict is the part to look at hardest, because it is the number a downstream system would route on. It is high, and what it is high about is how cleanly one case dominated the other. One side produced an enumeration and the other produced a hedge, so the comparison was easy, so the confidence is large. Nothing in that chain passes anywhere near the question of whether a registration number is required. The reported confidence would have been just as high on a case where the enumeration was correct, and just as high on one where it was invented, because in all three situations the property it is measuring — how lopsided the transcript was — is the same. A number that stays where it is while the thing you care about moves underneath it is measuring the other thing.

Two probes that take an afternoon#

None of this has to be argued from first principles, because judge selection is directly measurable without any labelled data. The first probe changes the order in which the two cases are presented and counts how often the verdict follows the position rather than the argument.

def order_sensitivity(cases, run_judge):
    flips = 0
    for c in cases:
        a = run_judge(first=c.pro_turns, second=c.con_turns)
        b = run_judge(first=c.con_turns, second=c.pro_turns)
        flips += (a["verdict"] != b["verdict"])
    return flips / len(cases)

Before running it, work out what the two extreme results would look like, because the probe is only interpretable against them. Presentation order carries no information about which document complies, so a judge deciding on the content of the cases alone returns the same verdict both ways round and the function returns zero. A judge deciding purely by position — always picking whichever case it read second, say — flips on every single case, and the function returns one. Real judges sit between, and the reading that falls out is direct: a case decided by content contributes nothing to the count, and a case decided by position contributes a full flip, so the rate is roughly the share of cases where position was doing the deciding.

That reading comes with a caveat that keeps it honest, and it is the same caveat in both directions. A case where position and content happen to point the same way flips only when the swap overrides the content, so cases decided by position that agree with the content are invisible to the probe. The measured rate is therefore a floor on how much position matters, not an estimate of it.

Run over 120 archived compliance transcripts, this returned 0.22. On roughly one document in five, the recorded verdict would have been the opposite one if the same two cases had been pasted into the prompt in the other order. Nothing about the documents changed, no argument was added or removed, and the audit record for those cases is a coin flip dressed as an adjudication. The 78% that survived the swap are not thereby correct; they are merely stable.

The second probe holds the arguments fixed and pads the losing side. Take the cases the judge decided against, append 150 words that restate the same points with no new claims, and re-run. On the same 120 transcripts, 17% of verdicts moved to the padded side. Padding cannot add evidence by construction, so any movement at all is measurement of the judge responding to volume, and 17% is a large number for a component that writes to a compliance audit log.

The by-construction step in that sentence is what the probe rests on, and it is worth being precise about what it does and does not license. Restating existing points adds no claim that was not already in the case, so a decision procedure operating on the evidence would return the same verdict, and any movement is a response to something other than evidence. What the probe cannot separate is which non-evidential thing: padding lengthens the case and it also moves the original claims further from the end of the prompt, and those two changes arrive together. The number is a valid measurement of sensitivity to a change that carries no information. It is not a clean measurement of a preference for length specifically.

A third check is cheaper still and worth running first on any inherited system: sample thirty transcripts and grep the quoted spans against the source documents. In this pipeline, 9 of 30 quotes attributed to a clause number did not appear in that clause, and 2 of those did not appear in the document at all. Neither the opposing debater nor the judge flagged any of them, which is what you would expect when neither of them has the document open.

The one thing to resist while running them is the urge to fix what they find. Every one of these has an obvious patch — randomise presentation order and take the majority of two orderings, normalise turn lengths before judging, require quotes to be verified against the source with a string match. Each patch is defensible, each costs another call or another engineering week, and applying them to a system whose value has never been measured means arriving in three months with a more expensive pipeline and the same unanswered question. Write the numbers down and leave the pipeline alone until the question of whether debate should exist here has been settled.

The courtroom picture that makes debate feel trustworthy is worth finishing, because the parts that get dropped in translation are the parts doing the work. It is an analogy and it breaks in two places, one of which is usually noticed and one of which is not. An adversarial legal system does not rely on advocacy alone to reach the truth. It has discovery, which forces the documents into the open before anyone argues; cross-examination, in which a witness's claim is tested against those documents in front of the decision-maker; sanctions for asserting things that are false; and appeal on an error of law, which is a separate authority that can overturn a verdict without re-hearing the argument. A debate ensemble imports the advocacy and none of the machinery. The judge in this pipeline has no discovery, no cross-examination against the exhibit, no consequence for either agent when a quote turns out to be invented, and no appellate layer, and it decides in one pass at 400 milliseconds per verdict.

The second break is the one that survives even if you build all of that. A court's two advocates are different parties, differently informed, with opposed interests, and the process assumes that what one of them fails to find the other has reason and means to find. Two debaters are one model with opposed instructions, which is the property chapter 2 measured. Importing the machinery would fix the judge's evidence problem. It would not make the two advocates into two parties.

Rubric judges, cross-family judges, and where model-as-judge is sound#

The judge is not the villain of the design, and replacing it with a better one does not rescue the pipeline. A rubric-scored judge, which grades each case against named criteria instead of picking a winner outright, reduces order sensitivity measurably — the same 120 transcripts fell from 0.22 to 0.09 when the judge was made to score five named policy conditions separately before concluding — and it does nothing about the two rows in the table above that the transcript cannot express. A judge from a different model family removes the self-preference effect where a judge favours text that reads like its own output, and leaves the shared-gap problem where chapter 2 left it.

That improvement is real and it is worth reading for what it is. Scoring five named conditions separately replaces one holistic comparison with five smaller ones, and a smaller comparison is decided by the case's content more often, so less of the decision is left for position to carry. The drop from 0.22 to 0.09 is a measurement of that. It is a measurement about the judge's consistency, and consistency is a different quantity from accuracy: a rubric judge that reliably scores a misremembered standard as satisfied is more consistent and no more correct.

Nor is the argument that model-as-judge, meaning the practice of putting one model in the position of scoring or choosing between another's outputs, is unsound in general. Judging is doing well-understood work in settings where the thing being judged is checkable from what the judge is given: a unit test's output, a JSON schema match, an answer against a reference answer in a graded evaluation. The compliance pipeline is not that setting, because there is no reference and the source is out of frame. The distinction to carry forward is between a judge that verifies against something and a judge that selects between presentations, and the same model, prompt and temperature can be doing either depending on what you put in its context.

The most consequential thing to say about this pipeline is what its six-month history is worth as evidence. Reviewers reporting thorough output have observed that the verdicts arrive with reasoning that addresses counterarguments, which is a property the design guarantees regardless of whether the verdicts are right. If the system had been producing well-argued wrong conclusions for six months, the reviewers' reports would read exactly as they do, the audit log would look exactly as it does, and the only observable difference would be in outcomes nobody has measured. Duration of operation is not accumulating evidence here. It is accumulating unexamined decisions at eight times the price of the classifier that used to make them.

The single sentence to carry out of this chapter is that a judge selects on what its window makes visible, and everything else follows from it: the table's two columns running in opposite directions, the hedged case losing to the enumerated one, the flips under a reordering that carries no information, and the confidence figure that reports how lopsided a transcript was while reading as though it reported how likely the verdict is to be true. None of that is a defect in the judge prompt, and none of it is repaired by a better one.

Where that leaves you is with a judge whose verdicts you cannot interpret and a pair of debaters whose disagreements you cannot interpret either. The next chapter takes the disagreement half of that: when two agents hold their positions through three rounds, there is a class of question where that persistence is worth acting on and a class where it is guaranteed by the question itself, and telling them apart is a property of the task rather than of the transcript.