Debate and adversarial ensembles

Read these first

• Parallelization — sectioning and voting

What you will know

• Set two agents to argue opposite positions and surface an error

  Use separate prompts or roles and preserve their intermediate claims so a third party can verify that the disagreement exposed a real defect rather than manufactured conflict.

• Measure whether debate beat a single pass

  Run both approaches on the same labeled cases and compare error rate, calibration, latency, and cost; wins on a few anecdotes are not evidence.

• Calculate the cost multiplier

  Include participant calls, debate rounds, judge or synthesis calls, retries, and longer context windows because nominal parallelism often understates spend and tail latency.

• Classify disagreement as signal or noise using test cases

  Objective tasks can turn conflict into a correctness check, while subjective or underspecified tasks may produce incompatible but valid answers or amplify correlated mistakes.

• Decide against debate for a task where it added nothing

  A simpler single-pass or verifier design is preferable when the measured quality gain is negligible, unstable, or smaller than its operational and financial overhead.

Chapters

1. 1

   Debate trades independence for argument quality

   Sampling one prompt three times and taking the majority exploits statistical independence; two agents arguing assigned positions in front of a judge exploits information sharing. Reading each other's reasoning is not free extra evidence, it is anchoring, and it removes the independence that made the majority meaningful. Here you set up a two-agent disagreement, preserve the intermediate claims, and let a third party check that a real defect surfaced.

   Read chapter
2. 2

   Two instances of one model share the objection neither raises

   Prompted to argue opposite sides, the same model often converges confidently on a conclusion that turns out to be wrong. Opposing instructions do not produce opposing blind spots, so the objection both are missing gets raised by neither, and two rounds of argument then rest on it. Reviewers describing such output as thorough is entirely consistent with a system producing well-argued wrong answers.

   Read chapter
3. 3

   The judge rewards the better-argued case, not the truer one

   The judge is the component that turns an exchange into an output, and what it receives is the transcript: two coherent cases, each answering the other. It is normally the same model family as the debaters, so it shares their blind spots, and where it has no way to check a claim against a source it selects on the properties it can assess — coherence, coverage of the opponent's points, confident framing. That is the concrete meaning of buying argument quality with independence: the system's output is optimised for how well a position was defended, which is exactly why six months of thorough-looking compliance verdicts is not evidence that any of them were right.

   Read chapter
4. 4

   Persistent disagreement only means something on a checkable question

   Two agents argue opposite readings of a contract clause and still disagree after three rounds, each with a coherent case. Whether that is worth acting on depends on the question: where a determinate answer exists, stubborn disagreement usually locates a real defect, while on an underspecified one both readings can be defensible and no number of rounds will settle it. You will build test cases that separate the two.

   Read chapter
5. 5

   Rounds grow input tokens quadratically while calls grow linearly

   Two debaters over three rounds plus a judge is seven calls, not three, and the call count is the smaller error. Each round feeds the whole exchange so far back in, so input tokens rise with the square of the rounds and the judge's call is the largest in the run. Running the debaters concurrently cuts latency and not spend. Measure the real multiplier over a week of production traffic.

   Read chapter
6. 6

   Six tenths of a point on 400 cases is about two tickets

   Single pass scores 91.2%, a two-agent debate with a judge scores 91.8%, at six times the cost and eight seconds more latency, and product wants to ship it. That gap sits inside sampling noise, so nothing has been demonstrated, and routing only low-confidence cases into debate makes an unmeasured effect affordable rather than exploiting a measured one. Compare error rate, calibration, latency and cost on the same labelled set, then inspect which cases each side gets wrong.

   Read chapter
7. 7

   Ground truth for a judgement task is a stratified afternoon's work

   The comparison you want needs labelled cases, and the systems most in need of it — a compliance checker running unevaluated for six months — are exactly the ones that have none. You do not need a benchmark: pull a stratified sample of past decisions covering the policy areas and the outcomes you care about, have one or two domain experts label fifty to a hundred of them blind to which system produced what, then replay both the incumbent and the single-pass arm over that sample. This is the smallest experiment that can tell you whether a debate ensemble should exist, and building the sample is the part people skip because it is the only part that is not a model call.

   Read chapter
8. 8

   A verification pass beats an argument when values are unsupported

   Six fields extracted from supplier invoices go wrong mostly by carrying values that appear nowhere in the document, and you have budget for roughly double the current inference. One call that re-reads the source and checks each field catches that failure far more cheaply than two agents arguing. This chapter writes the case against debate for a task, including how to engage with the claim that it is standard practice for high-stakes decisions.

   Read chapter