A needs-review state beats an uncalibrated 87% confident

The claims tool showed a small grey badge under every extracted figure: 87% confident. Adjusters had been using it for four months and had developed a working rule of thumb, which they described in user interviews as "anything in the eighties or nineties I just accept, below eighty I open the document". When the team finally graded a sample of 900 outputs against the source policies, the results grouped by displayed confidence looked like this: items shown 90–100% were correct 84% of the time, items shown 80–90% were correct 61% of the time, and items shown 70–80% were correct 58% of the time. The badge that read 87 was attached to a bucket where two in five figures were wrong, and the adjusters had been taught by the interface to accept exactly that bucket without opening the document.

Nothing in the system had claimed the number was a probability of correctness. It was a token-level score transformed through a couple of steps in a helper function nobody had looked at since it was written, and it was displayed because it was available. The users supplied the interpretation, and the interpretation they supplied is the only one that badge can plausibly carry.

Where the reading comes from#

Users have one strong prior for a percentage attached to a prediction, and it comes from weather. "70% chance of rain" is a probability that behaves properly: national forecast offices have published verification statistics for decades, and across large samples it rains on close to 70% of the days forecast at 70%. That is calibration earned over an enormous number of resolved predictions with an unambiguous outcome, and it took the field a long time to get there — Brier's scoring rule dates from 1950, and operational probability-of-precipitation forecasts followed in the 1960s. When a person sees a percentage next to a machine's prediction, that is the frame they bring, and no amount of tooltip text displaces it.

The two-digit precision makes it worse rather than better. A label reading "high confidence" invites a rough interpretation; a label reading 87% signals that something was measured to the nearest point, which is a claim about instrumentation quality that the underlying number does not support. Self-reported confidence from a language model — asking it to rate its own certainty — has the additional property of clustering on round human-sounding values, so a system displaying self-reported scores shows 85, 90 and 95 for most of its output and almost nothing in between, and users read the spread as meaningful variation.

Whether a score is calibrated at all is a measurable property, and topic 9.13 covered how to establish it: reliability curves, the difference between separation and calibration, and what a cutoff actually buys you. The question here is different and comes after: given a score whose calibration you have measured, what do you put in front of a user, and what do you do when the measurement says the score is not calibrated and will not be by the release date.

States the system can defend#

The alternative is to display something whose definition is a check that ran, so that the label is a statement about the system's own evidence rather than an estimate of truth. That constrains you to a small number of coarse states, which is the point — a state you can define in one sentence is a state a user can learn, and a state backed by a check is a state you can defend in an incident review.

Two things change when the display moves to this vocabulary. The label becomes falsifiable — if something marked "verified against source" turns out wrong, there is a bug in the matcher and you can find it, whereas an 87 that turns out wrong is not wrong about anything in particular. And the label stops competing with the actual remedy, which is showing the source: an adjuster who can click a figure and see the highlighted line in the policy does not need a confidence estimate, because they can settle the question in three seconds. Provenance beats any confidence expression where provenance is available, and a large fraction of the cases where teams reach for a score are cases where they could have shown the evidence instead.

def presentation_state(field, extraction, ledger, second_pass):
    """One function, four outcomes; each branch is a check that ran or failed."""
    if field.name in ledger:
        return State("from_your_records", source=ledger.ref(field.name))
    if extraction.span is None:
        return State("unverified")
    if not field.type.validates(extraction.value):          # date in future, negative premium
        return State("needs_review", reason="failed_format_or_range")
    if second_pass and second_pass.value != extraction.value:
        return State("needs_review", reason="extractors_disagree",
                     alternative=second_pass.value)
    return State("verified", source=extraction.span)

{
  "field": "annual_premium",
  "value": 1840.00,
  "state": "needs_review",
  "reason": "extractors_disagree",
  "alternative": 18400.00,
  "source": {"page": 3, "bbox": [112, 604, 198, 620]},
  "score": 0.87
}

The raw score stays in the payload and out of the interface. It is genuinely useful for routing, for the thresholds behind the review queue in the previous chapter, and for the analysis that tells you whether the states themselves are behaving — and none of those consumers are the end user.

The objection, which always arrives#

Proposing this in a design review produces a predictable set of responses, and arriving with only the design is how the proposal loses.

Product's objection is that the badge depresses conversion, and it is usually correct. On this product, adding the four states dropped the one-click accept rate on labelled items by 4.1%, which across the funnel was worth about £34,000 a quarter in adjuster throughput. The counter-argument is that the drop was concentrated almost entirely in the needs_review and unverified buckets, which is the mechanism working rather than a side effect of it, and that the metric under discussion is the wrong one. Accept rate measures whether users click; what the business pays for is corrected claims, reopened cases and regulatory findings, and the same period showed downstream corrections falling from 2.9% to 1.6% of processed claims, worth more than the throughput lost. Bringing both numbers changes the conversation from taste to a trade.

The competitive objection — the rival product shows nothing and looks more confident — is real and does not have a satisfying answer. What it has is an asymmetry worth naming: the cost of looking slightly less slick is bounded and known, and the cost of an adjuster processing a run of wrong figures with no signal is not. It also has a design answer, which is asymmetric disclosure. Say nothing in the good state; a verified figure gets a subtle underline that reveals the source on hover, and only needs_review gets an actual visible marker. The interface then reads as clean and confident, and the 4% of fields that carry a warning carry it against a quiet background where it is noticeable.

The objection that the model is usually right and users will learn to ignore the label is the one to take most seriously, because it is true whenever the label fires too often. A warning attached to 30% of fields becomes part of the page and stops being read within a fortnight, which is the same fatigue that emptied the review queue in the previous chapter, arriving through a different door. That sets a design constraint: the flagged state has to be rare and it has to be predictive. Keeping needs_review at 4% and measuring that flagged items are wrong at roughly six times the rate of unflagged ones gives the label its meaning; if that ratio degrades to 1.5, the label is noise and should be removed rather than defended.

Support's objection is that any visible hedge generates tickets, and it should be priced rather than argued with. Measured over the first month, labelled results produced about 0.8 extra contacts per 1,000, mostly asking what "unverified" meant, and most of them disappeared after the wording changed from "unverified" to "not found in your document". A support cost of that size is a line item; it becomes an argument-ending objection only when nobody has measured it.

When there are no fields to label#

A structured extraction makes this easy, because there are discrete values to attach states to. A conversational surface has one blob of prose containing a dozen claims of varying support, and a single badge on the whole answer is either wrong about most of it or so weakly worded that it says nothing.

The workable approach is per-claim provenance rather than per-answer confidence: split the answer into its assertions, check each against retrieved sources, and mark the individual sentences that no source supports. That produces an answer where three sentences carry a citation the user can open and one carries a small unsupported marker, which is a genuinely different reading experience from a paragraph with a disclaimer under it. The check is the same entailment machinery topic 9.13 covered, run at sentence granularity and used for display rather than for a routing decision.

What does not work is letting the model write its own hedges. Asking a model to express uncertainty in its prose produces hedging that tracks the register of the question rather than the evidence, so a well-supported answer to a vaguely-worded question comes back full of "it appears that" while a fabricated citation arrives in flat declarative English. The distribution of hedge phrases in generated text is close to uncorrelated with whether the content is right, which makes it the worst of both options — it looks like a signal and carries none. The same argument applies to letting the model choose its own badge: whatever the label says, it should be the output of a check the system ran, not a claim the generator made about itself, because a label produced by the same process that produced the error inherits the error.

Wording turns out to matter more than the taxonomy. The four states above went through three rounds of copy before they stopped generating support contacts, and the pattern in what worked was that every state described what the system did rather than how it felt: "not found in your document" instead of "unverified", "read from your ledger" instead of "high confidence", "two readings disagreed — check page 3" instead of "needs review". Each of those tells the user what to do next, which is the actual job of the label; a state that leaves the user knowing only that something is uncertain has moved a problem from the system into a person without giving them anything to act with.

Placement follows the same logic. A marker attached to the specific number is read; a banner at the top of the page is read once, on the first visit, and then stops existing as far as the user's attention is concerned.

Where disclosure is the wrong instrument#

Telling a user that the system is unsure is only useful if the user can do something with that information. An adjuster with the policy document in front of them can check; a consumer told that their eligibility assessment is "unverified" has no means of verification and receives nothing but anxiety and a support contact. Where the user cannot resolve the uncertainty, the honest options are to resolve it internally before showing anything, to route it to someone who can, or to withhold the output — and abstention is the mechanism topic 9.13 dealt with.

Disclosure is also not a guardrail for a consequential action. A label on a screen does not create a window, a cancel path or a restorable prior state, so for anything irreversible it belongs alongside the mechanisms from chapter four rather than in place of them. Its scope is genuinely narrow: it governs what a person believes about an output they are about to act on.

That completes the kit — budgets by error class, slicing, bounded radius, holds, idempotent effects, sized review, and calibrated disclosure. The remaining problem is that no feature ships with all of them, and the decision about which one goes is usually made three days before a launch by whoever is most tired.