A verification pass beats an argument when values are unsupported

Accounts payable extracts six fields from every supplier invoice — invoice number, invoice date, supplier tax ID, currency, subtotal, and total due — across about 12,000 invoices a month, which is 72,000 field extractions. The field-level error rate is 3.1%, so roughly 2,200 fields a month come out wrong, and each one that reaches the payment run is either a delayed payment or a wrong one. There is budget for approximately twice the current inference spend, which is $191 a month, and debate is on the list of things to try because someone read that adversarial ensembles reduce hallucination.

Before choosing anything, somebody sat down with 100 of the wrong fields and sorted them.

Two thirds of the errors are values that do not exist in the source. That single fact decides the design, because a failure where the model asserts something the document does not contain is checkable against the document without any judgement at all, and checkable failures do not need an argument.

The cheapest verifier makes no model calls#

The first pass at the 1,518 unsupported values costs nothing per invoice, because you already have the OCR text in memory — the characters an optical character recognition pass has read off the scanned invoice, which is what the extractor was given in the first place — and you can ask whether the extracted string occurs in it. The naive version fails immediately on real data — the model normalises 1.234,56 to 1234.56, writes dates as ISO, and strips spaces out of tax IDs — so the check has to normalise both sides before comparing.

import re, unicodedata
from datetime import date

def norm_text(s: str) -> str:
    s = unicodedata.normalize("NFKD", s).casefold()
    return re.sub(r"[\s \-–—/.,]", "", s)

def norm_amount(s: str) -> set[str]:
    """Candidate surface forms an amount could take in the source."""
    v = s.replace(",", "").replace(" ", "")
    try:
        cents = round(float(v) * 100)
    except ValueError:
        return set()
    whole, frac = divmod(cents, 100)
    return {f"{whole}{sep}{frac:02d}" for sep in (".", ",")} | {str(whole)}

def is_grounded(field: str, value: str, ocr: str) -> bool:
    hay = norm_text(ocr)
    if field in {"subtotal", "total_due"}:
        return any(norm_text(c) in hay for c in norm_amount(value))
    if field == "invoice_date":
        d = date.fromisoformat(value)
        forms = [d.strftime(f) for f in ("%d%m%Y", "%m%d%Y", "%d%b%Y", "%Y%m%d")]
        return any(norm_text(f) in hay for f in forms)
    return norm_text(value) in hay

Run over the archive, that function flags 1,270 of the 1,518 unsupported values — 84% of the largest failure class, at a cost of about 20 milliseconds per invoice and no tokens. The 248 it misses are mostly amounts reconstructed by the model from line items that are individually present, plus a handful where OCR mangled the character the model happened to get right.

A flagged field is not yet a fixed field. The cheap policy is a single targeted re-extraction that names the failing field and its constraint, and on this corpus 61% of flagged fields came back grounded on the second attempt — grounded meaning the value can be found in the source text rather than only in the model's answer — with the remainder going to the human review queue that already exists for low-confidence invoices. That queue grew by about 500 invoices a month, which the AP team judged acceptable against the payments it prevented.

One call that re-reads the source and returns spans#

The string check cannot touch the second failure class, where the value is real and comes from the wrong line, and it cannot help with format or omission. Those need something that understands the layout, which is one model call that sees the OCR text and the six extracted values and is asked to locate each one, returning the span that supports it — the exact stretch of source text the value was supposedly read from.

VERIFY = """You are checking six extracted fields against an invoice.
For EACH field, return the exact substring of the invoice text that
supports it, plus the label immediately preceding that substring.
If no substring in the document supports the value, return
"span": null. Do not correct anything; only report what you find."""

# response for one invoice
{
  "invoice_number": {"span": "INV-2025-04417", "label": "Invoice No."},
  "invoice_date":   {"span": "03.04.2025",     "label": "Rechnungsdatum"},
  "supplier_tax_id":{"span": null,             "label": null},
  "currency":       {"span": "EUR",            "label": "Currency"},
  "subtotal":       {"span": "1.240,00",       "label": "Gesamtbetrag"},
  "total_due":      {"span": "1.475,60",       "label": "Zahlbetrag"}
}

Two design choices do most of the work here. The verifier is forbidden from correcting anything, so it produces evidence rather than a second opinion, which keeps its output mechanically checkable — every returned span goes back through is_grounded against the OCR text, so a fabricated span is caught by string comparison rather than by trusting the verifier. And it returns the preceding label, which is what catches the wrong-occurrence class: subtotal supported by a span whose label reads Gesamtbetrag is a real number pulled from the total line, and a rule comparing the returned label against an expected-label list for each field flags it without any further inference.

The verifier call carries the invoice text again, around 4,300 input tokens against the extractor's 4,400, and returns about 260 output tokens. That is 1.06 cents per invoice against the extractor's 1.59, so the pipeline lands at 2.1x — inside the budget, with the string check running free in front of it.

A verifier has its own error rate, and leaving it unmeasured repeats the mistake this whole topic is about. Two numbers matter and both come from the same labelled sample. The false-flag rate is how often a correctly extracted field is reported as unsupported, which on this corpus ran at 0.4% of correct fields, or about 280 a month arriving in the review queue for nothing. The corruption rate is how often a flagged-and-re-extracted field comes back worse than it went in, which was around a quarter of the false flags, adding roughly 70 new errors a month. Those 70 are subtracted in the table below rather than quietly ignored, and they are the reason the combined pipeline lands at 0.7% rather than the 0.6% the failure taxonomy alone would predict.

What each option buys for what it costs#

The debate alternative for this task is two agents arguing whether an extraction is correct over two rounds with a judge, which is five calls and 6.4x. It was run on the same 600-invoice labelled sample as everything else.

The debate's poor showing on unsupported values is the interesting cell. Both debaters are the same extraction-capable model, and an agent arguing that the tax ID is wrong has to notice that a specific string is absent from a 3,800-token document, which is exactly the check the model is bad at doing by reading, and exactly the check in does perfectly. The argument spends five calls producing reasons while the failure was a two-line predicate. Where the debate did help was the wrong-occurrence class, because there the two candidate values are both present and an agent can genuinely argue about which line the subtotal comes from, and even there it recovered 63 fields against the verifier's 199 at three times the price.

The general shape of the decision is which failure mode dominates, and the honest version of it starts by labelling failures rather than by choosing a pattern.

Only the bottom branch reaches debate, and it reaches it as a candidate rather than as a conclusion. The branch above it is the one from chapter 2's slip-versus-gap distinction, where independent sampling is the cheaper exploitation of the same variance, and the two branches above that are not ensemble problems at all.

Writing the decision against debate#

A decision not to use something needs a record, because otherwise it gets re-proposed every quarter by whoever read the paper most recently. Four lines in the design document are enough, and they are all numbers already in hand: the failure taxonomy from the 100 labelled errors, the measured field error rate of each option on the same 600 invoices, the cost multipliers, and the budget constraint. What makes that record durable is that it is specific to a failure profile rather than to a technology — if the error mix shifts and wrong-occurrence errors become the dominant class, the conclusion is genuinely open again, and the document says so.

The claim that debate is standard practice for high-stakes decisions comes up here too, and payments are high-stakes in a way compliance verdicts are not, since a wrong total due moves money. High stakes is a reason to demand more evidence for a design, and it applies symmetrically: an unevaluated 6.4x ensemble on the payment path is a large uncontrolled change to a system that moves money, and it leaves 1,090 unsupported values a month in place while doing it. The version of the argument that lands is the taxonomy table, because it is not a claim about debate at all — two thirds of the failures are a value that is absent from a document, and no one defends spending five calls to argue about that once they have seen the row.

Where debate does earn its place#

Nothing in this topic says the pattern is worthless, and the conditions under which it works are specific enough to check against a task before proposing it. The claims in dispute have to be resolvable against material that is in front of the participants, so an argument can terminate on a fact rather than on a preference. The participants have to fail differently, which a same-model pair does not do well and a cross-family pair does partially. The judge has to be able to check the claims it is adjudicating, which means the source in its context and a mechanical check on quoted spans. And the gain has to be measured on labelled cases against the single-pass arm, at a sample size that can see an effect worth the multiplier.

Where those hold, the pattern has a real record: rounds of mutual critique measurably cut arithmetic and factual errors in the original Improving Factuality and Reasoning in Language Models through Multiagent Debate, and the composable-models/llm_multiagent_debate repository is a runnable loop with the exact prompts, which is the fastest way to try the protocol on your own task with your own model rather than reasoning about it. Reading that code first also makes the token arithmetic from chapter 5 concrete, since the transcript concatenation is right there in the loop.

The thing to build first on any task where debate looks attractive is the smallest version that produces evidence: a labelled sample of fifty to a hundred cases, a single-pass arm, a two-round debate arm, and the four columns of error rate, calibration, latency and cost. That takes about the same three days as the compliance experiment and it answers the question for your task rather than for the benchmark in a paper. If the debate arm wins by a margin that clears its multiplier, you have a result nobody can argue with and a transcript format that makes each verdict inspectable. If it does not, the budget goes to a verification pass, a better taxonomy, or more labelled data, all of which compound in a way that another round of argument does not.