Persistent disagreement only means something on a checkable question

A contract review tool runs two agents over a master services agreement and asks whether the customer may terminate for convenience during the initial term. After three rounds they have not moved.

r3 · agent_yes
  Clause 9.2: "Either party may terminate this Agreement for
  convenience on sixty (60) days' written notice." No carve-out
  for the initial term appears in 9.2 or anywhere in section 9.

r3 · agent_no
  Clause 3.1 fixes an Initial Term of twenty-four (24) months.
  A fixed term that can be exited at will is not a fixed term;
  9.2 must be read as operating after the Initial Term expires,
  or 3.1 has no effect.

Both cases are coherent, both cite real clauses, and neither has conceded anything across three exchanges. The engineer watching this has to decide what the persistence means and what to do with it, and the honest answer depends on something outside the transcript entirely.

Two kinds of question that produce identical transcripts#

The termination question above is determinate in one version of the contract and underspecified in another, and the transcripts look the same either way. If the agreement contains a subordination phrase — "subject to clause 3" at the head of section 9, or "following the Initial Term" inside 9.2 — then there is a fact of the matter, one agent has failed to read a phrase that is present, and three rounds of stubbornness is pointing directly at the paragraph where the failure lives. If the agreement genuinely contains both an unqualified termination right and an unqualified fixed term with no ordering between them, then the document is internally inconsistent, both readings are ones a court might take, and the debate can run for twenty rounds without resolving because there is nothing in the input that decides it.

Persistent disagreement is therefore not a signal on its own. It is a signal conditional on the question having a checkable answer, and the same protocol produces it in both regimes with the same confidence and the same quality of argument. A system that escalates on disagreement without knowing which regime it is in will escalate a genuine drafting inconsistency and a missed subordination clause identically, and its reviewers will spend the same amount of time on each.

The distinction is not the same as easy versus hard, which is the substitution most people make. "How many days' notice does clause 9.2 require?" is determinate and trivial. "Does this agreement, read as a whole, permit termination during the initial term?" is determinate and hard, because it requires holding several clauses together. "Is the termination regime in this agreement commercially reasonable for a supplier of our size?" is not determinate at any level of effort, because the criterion that would settle it was never supplied. Only the third one is guaranteed to produce irresolvable disagreement, and it is the sort of question that gets added to a review prompt without anyone noticing that it is different in kind from the two above it.

Building test cases that separate the two#

The useful move is to stop asking what a given disagreement means and start measuring what disagreement means on your task, which requires three classes of test case rather than one.

The first class is seeded defects: take real documents where the answer is known, plant a specific flaw, and check that the pair disagrees. The second is clean cases: the same documents with the answer known and no flaw planted, which measures how often the protocol manufactures disagreement out of nothing. The third is the one people skip, and it is known-ambiguous cases: documents where two domain experts have independently endorsed different readings and signed off that the text does not decide between them. Without that third class you cannot distinguish a system that finds defects from one that finds hard questions, because both look like disagreement.

Sourcing the ambiguous class is the part that takes real time, and it cannot be delegated to the model, since asking the same weights that are debating the question to certify that the question is undecidable reproduces the problem one level up. The procedure that works is to hand two reviewers the same twenty clause-and-question pairs independently, with no access to each other's answers, and keep the pairs where they landed on different readings and both were willing to defend theirs after seeing the other. On the contract tool that took one afternoon with two contract managers and yielded fourteen usable ambiguous cases out of twenty-eight candidates, with the rest collapsing once one reviewer read a clause the other had cited. The fourteen were topped up to twenty from an older batch of escalations that legal had previously closed as drafting defects, which is a good hunting ground because a closed escalation with a note saying the parties were asked to clarify is a documented instance of a text that did not decide its own question. Six per class is enough to see the pattern; twenty per class is enough to put a number on it that survives a rerun.

CASES = [
    Case(id="term-01", doc="msa_2019.pdf", kind="seeded",
         question="Does 9.2 permit termination during the Initial Term?",
         seeded_defect="removed 'subject to clause 3' from 9.1 header",
         expected="no", expect_disagreement=True),
    Case(id="term-02", doc="msa_2019.pdf", kind="clean",
         question="Does 9.2 permit termination during the Initial Term?",
         expected="no", expect_disagreement=False),
    Case(id="term-03", doc="msa_2021.pdf", kind="ambiguous",
         question="Does 9.2 permit termination during the Initial Term?",
         expected=None,          # two experts, two readings, both defensible
         expect_disagreement=True, expect_verdict="underspecified"),
]

Sixty cases split evenly across the three classes, run through a three-round debate, produced this on the contract tool:

Treated as a defect detector across the whole set, persistent disagreement fires 40 times and is pointing at a genuine document defect in 15 of them, which is a precision of 0.375 — precision being the share of the occasions a signal fires on which it turns out to be pointing at something real. On the subset where a determinate answer exists — the seeded and clean classes together — the same signal fires 21 times and is right 15 times, giving 0.71, and the three seeded cases where the pair agreed on the wrong answer are the correlated failures from chapter 2 showing up on schedule. The difference between 0.375 and 0.71 is entirely accounted for by the ambiguous class, where the signal is firing correctly and means something the escalation queue is not equipped to act on.

Making "underspecified" a verdict the system can return#

The design consequence is that a debate over a question with no determinate answer should not be forced to produce a winner, and forcing one is the default in every implementation that gives the judge two choices. The judge in the contract tool had permits and does_not_permit in its schema, so on all 19 ambiguous cases it picked one, and 11 of those picks came back does_not_permit with confidence above 0.8. A reviewer reading the audit trail sees a decided question.

Adding the third option changes what the pipeline can express:

{
  "verdict": "underspecified",
  "confidence": 0.91,
  "unresolved_because": "Clause 3.1 fixes a 24-month Initial Term and
    clause 9.2 grants termination for convenience with no ordering
    between them. Neither clause is subordinated to the other.",
  "would_resolve_if": "the parties confirm whether 9.2 was intended
    to operate during or only after the Initial Term",
  "positions": ["permits", "does_not_permit"]
}

That output is more useful than either verdict, because the action it implies is real: send this clause pair to a human with the specific question that would settle it. It also changes the economics of the whole design, since the cases where debate is expensive and irresolvable are now identified rather than resolved, and the reviewer's time goes to the 19 documents that need a decision instead of being spread across 40 escalations.

The cost of adding the option is a rise in abstentions — runs where the system declines to answer and hands the case on rather than committing to a verdict — on cases that were determinate all along, because a model handed an easy out will take it. On the same 60 cases, the three-verdict judge returned underspecified on 19 ambiguous cases and also on 5 of the 40 determinate ones. Whether that trade is acceptable depends on what an abstention costs you relative to a wrong confident verdict, and it is a threshold to tune with the abstention rate on the clean class as the guardrail.

What more rounds do and do not fix#

The instinct on seeing three rounds of deadlock is to allow a fourth and a fifth. On the contract set, extending to six rounds resolved 3 of the 19 ambiguous cases, and in all three the resolution came from one agent producing a fabricated qualifying phrase that the other accepted. Convergence happened, the transcript got longer and more confident, and the resolution was an artefact of one side running out of genuine material and generating something to fill the turn. Deadlock on an underspecified question is the correct behaviour, and rounds spent trying to break it are buying fabrication.

On determinate questions the picture is better and still has a ceiling: rounds four through six flipped 2 of the 3 seeded cases from wrong agreement to correct disagreement, which is a real gain on a small base, at double the token bill.

Disagreement is not uncertainty, and the queue treats them the same#

Two agents holding opposite positions is a fact about a protocol with assigned roles, and low model confidence is a fact about a distribution over answers. Teams route on them interchangeably because both arrive at the escalation queue as "the system was not sure", and they come apart in both directions. On the twenty clean cases the pair disagreed four times while the flat prompt returned the correct answer at a mean self-reported confidence of 0.88, so the protocol produced conflict on cases the model was not conflicted about. In the other direction, on the three seeded cases where both agents landed on the same wrong reading, there was no disagreement to route on at all and the flat prompt's confidence was 0.79, which is above the threshold most queues use. A disagreement flag and an uncertainty flag select different subsets of your traffic, and only one of them is available without running the debate.

The practical consequence is that disagreement is not a substitute for a calibration measurement. If what you want is a signal that says "a human should look at this", the cheaper candidate is the single pass's own confidence, measured against outcomes on a labelled set to see whether a reported 0.8 corresponds to being right eight times in ten. That measurement costs one run over cases you have already labelled, and if it turns out well calibrated on your task, a debate that fires on a different and largely disjoint subset has to justify itself against it rather than alongside it.

Where this leaves the escalation queue#

Nothing here says to route documents by determinacy using the same model, though that is the obvious implementation and it half-works. Asking the model in advance whether a question has a determinate answer given a document is itself a judgement it can get wrong, and on this set a pre-classification pass agreed with the experts on 51 of 60. The nine errors were mostly in one direction, marking genuinely ambiguous questions as determinate, which is the direction that puts irresolvable cases back into the defect queue.

The more durable fix is at the level of the question rather than the case. Review prompts accumulate questions over time, and the ones that produce permanent deadlock are usually identifiable by inspection: they contain an evaluative term with no supplied criterion, or they ask what a document means rather than what it says. Auditing the question list once, and splitting the evaluative ones out into a separate path that never claims a verdict, removes more irresolvable debates than any amount of per-case routing.

You now have a way to tell whether a debate's disagreements are worth acting on, which makes the design defensible on the quality axis for a specific class of question. It says nothing about whether you can afford it. Those six-round experiments cost more than twice the three-round ones despite adding only twice the calls, and the reason is the shape of what gets re-sent on every turn, which is the subject of the next chapter.