Declining unverifiable work

A reliability target is an arithmetic commitment, not a statement of ambition. Measuring a rare failure costs roughly the inverse of how rare it is, so a target of 99.9% on work that only a specialist can judge implies a labelling budget denominated in that specialist's hours — and if the specialist is the person the feature was meant to free up, the budget cannot be paid out of the savings.

That is the shape of every declined proposal in this chapter, and it is worth separating from two things it is not. It is not doubt about the model, which may well produce excellent output. And it is not a permanent no: the same material usually contains a smaller artefact with a real checker attached, and finding it is the useful outcome of the conversation.

Where the arithmetic settles it#

Legal spends about 400 hours a month rewriting contract clauses into plainer language for a sales team's standard agreements. The proposal is an agent that produces the plain-language version while preserving the legal meaning, at 99.9% reliability, because anything less would not clear the general counsel. The demo was good: three clauses in, three rewrites that read better than the originals, and a lawyer in the room who agreed all three were fine.

The refusal rests on the target. The company processes roughly 2,000 clauses a month, so 99.9% means two failures a month, and detecting a failure rate that low requires labelling on the order of 3,000 clause pairs before the estimate is tight enough to distinguish 99.9% from 99.4%. Each of those labels is a lawyer reading an original and a rewrite side by side, which takes about twelve minutes. Six hundred hours of legal time to establish the number once, against 400 hours a month saved — and the measurement expires whenever the model version, the prompt or the clause mix changes. The only instrument that can decide meaning preservation is the person the agent was meant to free up, so the agent's quality claim can never be established more cheaply than doing the work by hand.

Each step of that is one arithmetic operation and the conclusion only lands if they are done rather than gestured at. A tenth of a per cent of two thousand clauses is two clauses, so the target is a commitment that no more than two rewrites a month change the meaning. To measure a rate that low you need a sample in which failures actually occur: at one in a thousand, a sample of three thousand expects three of them, and three is about the smallest number of events from which any estimate can be formed at all. Three thousand labels at twelve minutes each is thirty-six thousand minutes, which is six hundred hours.

Now put the two numbers in the same units, because that is where the proposal dies. The feature saves 400 hours a month. Establishing that it works costs 600 hours of the same people's time. If that were a one-off, it would pay back inside two months and be an easy yes. It is not a one-off: the measurement is a property of one model version, one prompt and one clause mix, and any of those changing invalidates it. A team that adjusts the prompt monthly is spending six hundred hours a month of legal time to save four hundred. The target is not ambitious. It is arithmetically unpayable, and the reason it is unpayable is that the measuring instrument and the resource being saved are the same people.

Notice that this argument never mentions how good the model is. Improve the model until it genuinely fails once a month instead of twice and every figure above gets worse, because a rarer failure needs a larger sample to establish. That is the counter-intuitive result worth carrying out of this chapter: raising a reliability target raises the cost of proving it faster than it lowers the cost of the failures, and past some point the proof is the thing you cannot afford.

The substitutes that get proposed, and what each one covers#

The first counter-proposal is always a second model scoring the rewrite for meaning preservation on a five-point scale. Everything from the comparison of deterministic checks against model judges applies unchanged, and one thing gets worse: a judge asked whether two texts mean the same thing is being asked a question with no reference answer, so its verdicts cannot be calibrated against anything except the same lawyer-hours the exercise was avoiding.

The second counter-proposal is better and deserves to be taken seriously, because a surprising amount of legal text is mechanically checkable. Numbers and durations in the rewrite must match the original exactly. Defined terms — capitalised phrases bound in the definitions section — must appear with identical spelling and must not be introduced or dropped. Cross-references must resolve to the same clause. Party names must be unchanged. Modal verbs carry obligation, so a shall becoming a may, or a must not becoming a need not, is a diff worth rejecting on. Negations must survive. This is perhaps eighty lines of code, runs in single-digit milliseconds, and would reject a genuine and embarrassing class of rewrite.

It also accepts every failure that matters. Adding "reasonable" before "efforts" changes an obligation without touching a number, a defined term or a modal. Splitting one sentence into two can silently rescope which noun a trailing qualifier applies to, which is one of the most common ways a rewrite changes meaning and is invisible to every check in the list. The mechanical checks shrink the residual and do not remove it, and the residual is where the 99.9% lives.

The reason those checks miss what they miss is worth stating as a property rather than as a list of examples, because the property predicts the next example. Every item on that list is a token-level identity: this number equals that number, this capitalised phrase appears in both, this modal verb is unchanged. Identity between tokens is decidable by comparison, which is exactly why the checks are cheap. Meaning, though, is carried as much by structure as by tokens — which noun a qualifier attaches to, what scope a clause has, what an added adjective does to an obligation. A rewrite that changes structure while preserving every token the checker compares is not an exotic adversarial case; it is what "rewriting into plainer language" is. So the mechanical checks are not merely incomplete. They are systematically blind to the operation the feature performs, and complete on the things the feature was not supposed to touch.

What survives the conversation is a different product with no reliability target on it. An agent that proposes a plainer rewrite alongside the original, runs the mechanical diff and shows its findings, and requires a lawyer to accept or reject each one, is a drafting aid whose failures are caught by design rather than by hope. Legal's 400 hours might drop to 250. Nobody has to certify 99.9% of anything, because no output reaches a contract without a decision by someone whose job it is to make that decision.

Set the two versions side by side in hours and the comparison stops being a matter of ambition. The autonomous version saves four hundred hours a month and costs six hundred to certify, repeatedly. The drafting aid saves a hundred and fifty hours a month and costs nothing to certify, because there is no claim being made that anybody has to establish. A hundred and fifty hours a month against a saving that is negative once its measurement is paid for is not a compromise; it is the larger number. The reason the first version looks better in a meeting is that its savings are stated and its verification cost is not, which is the same omission the whole of this topic has been about, arriving at planning time instead of in a cost model.

Writing the refusal down so it survives the meeting#

A verbal "we can't verify that" gets relitigated every quarter by someone who was not in the room. The durable version is a short record kept next to the design docs, naming the claim, the checkers considered, what each covers, and why sampling is insufficient at the required reliability.

proposal: clause-rewriter-autonomous
claim: "rewrite preserves legal meaning"
required_reliability: 0.999
checkers_considered:
  - name: mechanical-diff
    covers: [numbers, durations, defined_terms, cross_refs, party_names, modals, negation]
    cost: 4 ms
    residual: "scope changes via added qualifiers or sentence splitting"
  - name: llm-judge-meaning
    covers: "claims full coverage"
    cost: $0.06
    residual: "uncalibratable; no reference answer exists without lawyer review"
  - name: lawyer-review
    covers: everything
    cost: 12 min / clause
    residual: none
sampling_verdict: >
  Establishing a 0.1% failure rate to within half a point needs ~3,000 labelled
  pairs = 600 lawyer-hours, against 400 hours/month saved. Re-required on every
  model, prompt or clause-mix change.
decision: declined as autonomous; approved as suggestion-with-diff, no reliability target
review_date: 2026-02-01

The residual and sampling_verdict fields are what make this more than a no. They say what would have to change for the answer to change, which is the only thing that makes a refusal revisitable on evidence rather than on enthusiasm.

Sorting the claims a document makes#

The second proposal in the same quarter looks different and is not. Leadership wanted an agent that drafts incident post-mortems from logs, alerts and the chat transcript, on the reasonable grounds that engineers hate writing them and they arrive late or not at all. The proposal that comes with it is a second model scoring the draft against a post-mortem rubric — completeness, timeline accuracy, action items — and regenerating anything below threshold.

Sorting the claims a post-mortem makes by what could decide them is more productive than arguing about the rubric.

Six of seven rows have a real checker and the seventh is the reason anybody reads the document. A rubric scored by a model produces a number over all seven, and its score is dominated by structure, which is the part the deterministic checks already cover for free. The causal narrative is unverifiable by construction, because the counterfactual it asserts — that the incident would not have happened without this condition — was never run.

That phrase is the load-bearing one and it is worth unpacking, because "root cause" sounds like the sort of thing a sufficiently careful reader could check. A causal claim is a claim about a world that did not happen: had the client's retries been bounded, the outage would not have occurred. Every record the company holds is a record of the world that did happen. There is no query over PagerDuty, the deploy log or the metrics store that returns a fact about the other one, because the other one produced no records. That is not a gap in the tooling and no amount of instrumentation closes it — a counterfactual is only settled by running the experiment, and running it means causing the outage again with the retries bounded.

Compare that against the row above it. "Detection at 14:22 UTC" is a claim about a record that exists, so checking it is a lookup. Every one of the six checkable rows has that shape: the document asserts something, a system somewhere holds the same thing, and the check is a comparison. The seventh row asserts something no system holds and none could. Sorting a document's claims by that single test — does a record of this exist anywhere — is what turns "can we verify a post-mortem" from an argument into a table, and the answer it gives is not a percentage. It is a list of which sentences are decidable and which one is the reason the document is written.

The adjacent problem is worth more than the one declined#

class TimelineEntry(BaseModel):
    at: datetime
    source: Literal["pagerduty", "ci", "chat", "metrics", "alertmanager"]
    source_id: str
    text: str                 # must be reproducible from the source record

def verify_timeline(entries: list[TimelineEntry], sources: SourceRegistry) -> Verdict:
    for e in entries:
        record = sources.fetch(e.source, e.source_id)      # re-fetched, not cached
        if record is None:
            return Verdict(False, f"{e.source}:{e.source_id} does not exist")
        if abs((record.timestamp - e.at).total_seconds()) > 1:
            return Verdict(False, f"{e.source_id} occurred at {record.timestamp}, entry says {e.at}")
        if normalise(e.text) not in normalise(record.body):
            return Verdict(False, f"text for {e.source_id} not found in the record: {e.text[:80]!r}")
    if entries != sorted(entries, key=lambda x: x.at):
        return Verdict(False, "entries are not in chronological order")
    return Verdict(True, None)

Each of the four checks in that function corresponds to a way a generated timeline goes wrong, and the order they appear in is the order of increasing subtlety. Fetching the record by source and ID rejects a phantom event outright: if there is no PagerDuty incident with that ID, nothing further needs deciding. Comparing the record's timestamp against the entry's, within a second, rejects a real event placed at a plausible but wrong time, which is the failure that quietly reorders a narrative. Requiring the entry's text to appear inside the record's body rejects a real event with words put into its mouth. And the final comparison — the entries against the same entries sorted by time — needs no source at all, because chronological order is a property of the list rather than of any record in it. The first three checks are grounded in external systems; the fourth is an internal consistency check, and it is free.

The sources.fetch call is doing something the comment flags and it is easy to skim past. The record is re-fetched rather than read from whatever the model was given, which is the same requirement the citation checker had and for the same reason: a checker that reads the generator's copy of the evidence is checking the generator against itself. Re-fetching is what makes the source registry an independent party rather than a cache of the model's input.

The whole verifier runs in about 400 ms against systems the company already pays for, its rejections carry reason strings precise enough to regenerate against, and its accept set excludes invented events, misattributed quotes, wrong timestamps and phantom deploys. What it accepts is a true, complete-looking chronology that omits the one event that explains everything, which is a real residual and a much smaller one than the original proposal carried.

That residual is not an oversight in the implementation and no amount of care would remove it, which is worth seeing because the same limitation applies to every checker in this topic. Look at what the function iterates over: entries, the list the model produced. Every check inside the loop is a question about an entry that exists. An event the model never mentioned is not an entry, so it is never fetched, never compared and never rejected — it is outside the domain the loop runs over. A checker built this way can decide whether everything present is true and can say nothing whatever about whether anything is missing, because the generator chose what the checker would look at. Detecting omission requires the opposite construction: iterate over the sources, and ask which records have no entry. That is a different and more expensive check, and knowing it is the shape of the fix is more useful than knowing that a residual exists.

The argument that this is worth more than the declined feature has three parts. It removes 2.4 of the 3.5 hours rather than the one hour of writing — which is about two-thirds of the effort against under a third, so the declined feature was attacking the smaller half of a job it could not verify while the accepted one attacks the larger half of a job it can. It leaves the causal reasoning with the people who have the context to do it, and it improves that reasoning, because a complete chronology assembled in seconds is a better input to thinking about cause than a partial one reconstructed from memory two days later. And the timeline is useful during the incident, not only after it, which the post-mortem drafter never could be.

The pattern that runs through the declined clause rewriter and the accepted timeline builder is that the checker was chosen before the problem was, and the problem was then adjusted until a cheap checker fit it. That is the practical form of everything in this topic, and it is what turns a refusal into a counter-proposal rather than an obstruction.

When the next proposal arrives, the first three things to establish are what artefact the model will produce, what will reject a bad one and at what price against the price of producing it, and what remains in the accept set once that check passes. If the second answer is a person whose time the feature exists to save, the feature as framed is not available, and the useful move in the room is to find the adjacent artefact in the same material that something cheap can decide. Then instrument it, label a sample, and let the two arms tell you whether the loop bought anything — which is where this stops being a design question and becomes an ordinary engineering one.