A confidence threshold routes an action three ways

Every decision about human oversight so far has been made per class of action: refunds are gated, ticket searches are not, subscription cancellations get three business days and a named owner. A class is a coarse unit, and a rule written at the level of the tool gives an enormous refund and a trivial one the same treatment for as long as the rule stands. Routing is the alternative: deciding, for each proposed action individually, whether it executes without a human, goes to a gate, or is refused. The support agent this topic has followed — a ticket-handling agent at an analytics vendor that searches tickets, drafts and sends replies, applies account credits, issues refunds and cancels subscriptions — is where the figures below come from.

Routing needs a number per action, and that number is a confidence signal: anything computed at proposal time that estimates how likely this particular action is to be wrong. Two independent design questions follow. The first is which signal to compute. The second is where to put the boundaries on it, which the signal itself cannot answer, because a score of 0.7 means nothing until somebody has said what a wrong action costs.

Four signals, and what each of them knows#

The signal is a design choice with real differences in cost and in what it can detect.

That table is a gap rather than a ranking. A verifier and a retrieval check answer a question with a determinate answer — is this amount inside this balance, does this clause appear in a document the agent opened — and anybody can recompute it later and either agree or contradict. A judge score and a self-report are impressions: not recomputable to the same value, and contradictable only by another impression.

The ordering is not accidental. It follows from that gap. Deterministic verifiers are first because they are cheap, exact and auditable, and because most of what teams want a confidence score for turns out to be expressible as a check: this refund exceeds the invoice balance, this recipient is not on the account, this invoice already has a credit against it. A verifier that returns already_credited would have caught the double refund from the third chapter without any notion of confidence at all.

Judges are worth their 900 milliseconds where the question is genuinely a judgement — does this reply answer the question the customer asked, is this refund consistent with the policy text — and they cost enough attention to design that treating one as a drop-in scorer produces a number that correlates with reply length. Self-report belongs on the list because it is free and it is what everyone tries first, and its failure mode is the most instructive one in this chapter.

The tempting belief is that a model reporting low confidence is a defence against the model being wrong, and it is tempting because the number is real and it does move. What it moves with is the wrong thing. A self-reported score is the model's sense of how clear the request was, and clarity of the request and consequence of the action are unrelated properties of a ticket. The score stays near the top of its range while the quantity that makes an action dangerous varies underneath it, so a large refund and a trivial credit on equally plain tickets carry the same number. Anything routing on that score is sorting its queue by how well customers write.

Three routes, and the one that gets left out#

Routing has three arms, and the difference between the second and third is what most designs miss. Executing means the effect happens without a human. Gating means a person is shown a specific proposed action and asked to authorise it. Refusing means the agent stops and the case goes to a human as a case — the ticket returns to the queue with the agent's findings attached and no proposed action at all.

Three arms rather than two is a design decision, and the argument for it runs as a chain. The reason refusal is a distinct arm is that a gate is only useful when the proposed action is probably right. A gate offers two buttons, which can express only agreement or disagreement with what is already there, so the card assumes the action on it is probably right and the operator's job is to confirm it. When the system's own estimate says otherwise, that assumption is false. Handing an operator a card for an action the system has low confidence in asks them to do the agent's job through a two-button interface, using a card that was designed to confirm a plausible action rather than to construct a correct one. They cannot, so they either approve something they have not evaluated or reject and start over — and rejecting and starting over is the refuse arm, reached by the expensive route. Below some level of confidence the right move is to stop proposing.

The floor comes first, which is the whole point of drawing it that way. A confidence score can move an action from gate to refuse and it can never move an action below the floor established when the gate set was chosen: cancel_subscription is gated at a score of 0.99 because a wrong one takes out a tenant, and no signal is trustworthy enough to change that. Scores tune the middle of the distribution; the floor bounds the worst case.

That is an ordering rather than a level, and the distinction is what makes it usable. The floor is not a very high threshold that scores must clear — if it were, a sufficiently confident score would eventually clear it, and the diagram would have the score feeding into the floor test rather than sitting downstream of it. The two answer different questions: the floor asks how bad the worst outcome of this action class is, which does not change from call to call, and the score asks how likely this call is to be wrong, which says nothing about magnitude.

The edges leaving the operator in that diagram carry the gate's own outcomes back into the same three arms. Edit returns to the verifiers rather than to execute, because an edited action is an action nobody has checked. Reject and expire both land on refuse, which is why refuse has to be a real destination: both leave a case that still needs handling.

Deriving the first thresholds from what each error costs#

A threshold on a score has no meaning until the score is expressed as a probability of the action being wrong, and the boundaries then fall out of arithmetic over three costs. Take the support agent's refunds, with an average value of $80.

• Wrong unattended execution costs H: the $80, about 20 minutes of someone's time unwinding it at roughly $14, and a share of complaint and dispute risk. Call it $140.
• A gate costs c_g: about 45 seconds of operator attention, near $0.75, plus a median 12 minutes of added latency for the customer, valued at $0.50 for this queue. Call it $1.25.
• A refusal costs c_r: the case is handled entirely by a person, about 12 minutes at $9, and the customer waits longer. The outcome is correct, so there is no error term.

A gate does not eliminate the error; it catches some fraction k of wrong actions, and the probe measurement from the previous chapter is where k comes from. With k = 0.40:

E[execute] = p·H
E[gate]    = c_g + p·(1-k)·H
E[refuse]  = c_r

execute beats gate while   p·k·H < c_g
                           p < 1.25 / (0.40 × 140) = 2.2%
gate beats refuse while    c_g + p·(1-k)·H < c_r
                           p < (9 − 1.25) / (0.60 × 140) = 9.2%

  p < 2.2%    execute unattended
  2.2% – 9.2% gate
  p > 9.2%    refuse and hand the case over

Write p for the probability that this particular action is wrong and H for what a wrong unattended action costs, the $140 above. Then E[execute] = p·H says that executing costs H when the action turns out wrong, which happens with probability p, and costs nothing the rest of the time. Multiplying a cost by its probability is what an expected cost is: the average bill per action over a great many of them.

The middle line adds one term and one structural feature that carries most of the teaching. Writing c_g for what a gate costs, the $1.25 above, the c_g in E[gate] = c_g + p·(1-k)·H sits outside the p term, because the operator is interrupted on every gated action whether or not that action was going to be wrong. There is no version of gating in which you pay only for the cards that mattered, since knowing which those are is the thing the gate was for. Then, of the actions that were wrong, a gate catches the fraction k, so 1-k is the fraction that gets through, and those still cost H each.

Writing c_r for what a refusal costs, the $9 above, E[refuse] = c_r has no p in it at all: a refused case is handled from scratch by a person, the outcome is correct, and there is no error term to add. That asymmetry is why the routes rank differently at different values of p. Two of the expressions grow with p and the third does not, so refusing eventually wins however expensive the person is.

Execute beats gate exactly when p·H is less than c_g + p·(1-k)·H. Subtract p·(1-k)·H from both sides, which is allowed because subtracting the same quantity from both sides leaves an inequality true. On the right that term cancels and leaves c_g. On the left, p·H minus p·(1-k)·H has p·H in common, so factor it out, and what remains in the bracket is one minus (1-k), which is k. The left side is p·k·H, giving the condition p·k·H < c_g that the block states without showing the step.

Divide both sides by k·H, permitted with the inequality pointing the same way because k is a fraction of wrong actions caught and H is a cost, so both are positive — dividing by a negative number would reverse it. The result is p < c_g/(k·H). The numerator is what asking a person costs; the denominator is the harm the gate removes per unit of p, smaller than H because a gate catches only the fraction k. So the boundary says: gate only once the harm you would remove has grown to exceed the price of asking. Substituting the gate cost of $1.25 over the product of the assumed catch rate 0.40 and the $140 of harm gives the 2.2% in the block.

The gate-versus-refuse boundary comes out the same way with one fewer cancellation. Gating beats refusing while c_g + p·(1-k)·H < c_r. Subtract c_g, leaving p·(1-k)·H < c_r - c_g, the extra cost of sending the case to a person rather than putting a card in front of one. Divide by (1-k)·H, again positive and again preserving the direction, and the boundary is p < (c_r - c_g)/((1-k)·H): keep gating while the harm still leaking through the gate is cheaper than the difference between a refusal and a gate. Substituting $9 minus $1.25 over the product of 0.60 and $140 gives 9.2%.

Now substitute the catch rate the probes actually measured, k = 0.15. The execute boundary moves to 1.25 / (0.15 × 140) = 6.0%, and the gate-versus-refuse boundary to (9 − 1.25) / (0.85 × 140) = 6.5%. The band in which gating is the best of the three options is half a percentage point wide, which is the arithmetic saying that a gate catching 15% of bad actions is barely worth interrupting anyone for. That is not an argument for deleting the gate; it is an argument for the card and volume work that raises k, and it puts a number on what that work is worth.

Both boundaries moved, and the directions are readable off the algebra without recomputing anything. In p < c_g/(k·H) the catch rate k sits alone in the denominator, so a smaller k makes the fraction larger and pushes the execute boundary up. In p < (c_r - c_g)/((1-k)·H) the denominator carries 1-k, so a smaller k makes 1-k larger, makes the fraction smaller, and pushes the gate-versus-refuse boundary down. The two close in from opposite sides at once, which is why a band that was several percentage points wide collapses to half of one. The strength of a gate is the width of the region in which gating is the right answer, and a weak gate is worth reaching for over a sliver of p and no wider.

Given p, a catch rate k measured from probes, and the three costs H, c_g and c_r, the comparison ranks all three routes and hands back both boundaries, so a threshold stops being a number somebody was comfortable with and becomes something a colleague can dispute by disputing an input. It also prices improvements to the gate, since the width of the gating band is a function of k.

The step this arithmetic depends on is turning a score into p. Scores from judges and verifiers are not probabilities, and treating a 0.7 as a 30% chance of being wrong is how these systems end up mis-tuned in both directions at once. Calibration is a lookup table built from outcomes: bucket historical actions by score, count how many turned out wrong, and read the empirical rate. It needs roughly 500 resolved outcomes per class before the buckets mean anything, and outcomes take 7 to 30 days to mature, so the first month of any deployment runs on uncalibrated scores. The workable interim rule is to choose cutoffs by volume rather than by probability — set them so the gate band contains the number of cards your rota can read attentively — and to replace them with the cost arithmetic once outcomes exist.

That is the second tempting belief, and it feels like a technicality. A score is an ordering: this action is more suspicious than that one. A probability is a rate: out of a hundred actions carrying this score, about this many turn out wrong. Every expression above multiplies p by a cost, and multiplying an ordering by a cost produces a number with no meaning. A score is a quantity that stays fixed while the rate of being wrong behind it moves — which is what happens when the model version changes or the ticket mix shifts — and a team that has substituted a score for p will not notice, because the arithmetic still runs and still returns boundaries.

def route(action, ctx) -> Route:
    if policy_floor(action):                     # from the gate-set design
        return Route.GATE
    checks = run_verifiers(action, ctx)          # deterministic, 3 ms
    if checks.hard_failures:
        return Route.REFUSE                      # not a confidence question
    p = calibration[action.klass].p_wrong(
        verifier_score=checks.score,
        judge_score=judge(action, ctx) if action.klass in JUDGED else None,
        support=retrieval_support(action, ctx),
    )
    band = THRESHOLDS[action.klass]              # per class, versioned, dated
    if p < band.execute:
        return Route.EXECUTE
    if p < band.gate:
        return Route.GATE
    return Route.REFUSE

Hard verifier failures return REFUSE before any score is computed, because amount exceeds invoice balance is not a matter of degree. Mixing deterministic failures into a score is how a policy violation becomes a 0.62 that lands in the gate band and gets approved by someone who assumed the system would not propose something out of policy.

Each link in that argument is checkable. A verifier failure states that a rule was broken, and a rule is either broken or not. Blending it into a score turns a definite statement into a contribution, and a contribution can be outweighed — a strong retrieval match and a confident judge pull the blended number back into the gate band. The card that reaches an operator then looks like every other card, and the one signal that would have made them stop is the one the blending removed.

Moving the thresholds on evidence#

Thresholds set once are wrong within a quarter, because the volume, the agent and the operators all change. The rates from the previous chapter are what move them, per action class, on a slow cadence with a minimum sample size and a written record of each change.

The gap in that table is between its first three rows and its last two. The first three are observations about the routing — where actions of this class are landing and whether that placement produces good outcomes — so the threshold is the right lever. The last two are observations about something thresholds do not control: an expiry rate is a fact about whether the named owners were available, and a falling probe catch rate is a fact about k, the gate's own strength.

Reaching for the threshold in either of those cases fails in the same direction. A rising expiry rate looks like a class bothering people unnecessarily, so the reflex is to loosen — which sends more of an already-unwatched class through unattended at the moment the evidence says nobody is watching. A falling catch rate looks like a gate finding nothing, so the reflex is again to loosen. But a falling k shrinks the band from both ends, as the algebra showed, so the arithmetic is asking for better cards rather than a wider execute region.

Two disciplines keep this from oscillating. Changes need hysteresis, so the condition that loosens a class is stricter than the condition that tightens it, and a class that has just moved is frozen for a full window. And one class moves at a time, with the date and the reason recorded next to the threshold, so that when the reversal rate rises in March somebody can see that the refund band was widened in February. A monthly cadence with a 30-day outcome window is a reasonable default; anything faster is reacting to noise, since 500 outcomes per class per week is more volume than most of these systems have.

Both are design decisions about the measurement rather than about the agent. Nothing forces one class at a time; the reason to is that outcomes take weeks to mature, so two simultaneous changes produce one delayed signal attributable to neither. Without hysteresis a class near a boundary crosses back and forth on sampling noise, and every crossing resets the clock on the window.

The row about keeping 5% gated is the part that gets dropped for being wasteful. Without it, a class that leaves the queue produces no further human judgement, and the only remaining signal about it is post-action outcomes, which detect the errors that someone complains about and nothing else. A residual sample costs a handful of cards a week and keeps a measurement alive.

What a threshold cannot do#

A confidence signal reduces how many wrong actions reach production and says nothing about how bad the worst one is. The distribution of harm is not the distribution of error, and an agent that is wrong 0.5% of the time with a bounded downside is a different proposition from one that is wrong 0.5% of the time where one of those errors cancels an enterprise account. That is the job of the floor and of the scoping work from the architectural mitigations topic, and no amount of threshold tuning substitutes for either.

That limit is a residual of the arithmetic rather than an unrelated caution, and it is visible in the expressions. Each carries a single H, one number standing for what a wrong action costs. Collapsing a distribution of possible harms into one figure is what taking an expectation does, and it is right for a policy that runs over many actions, because the average is what you will pay. It is wrong when one member of the distribution can end the business relationship, since no term in the comparison carries the size of the worst case. Two agents with identical p, identical k and identical average harm get identical boundaries even when one has a tail the other does not. The floor covers what the expectation dropped, which is why it sits before the score.

The second residual is the calibrated p the comparison rests on. Outside the conditions where p is a real rate — roughly 500 resolved outcomes per class, matured over 7 to 30 days — the boundaries are arithmetic performed on a number that is not what the arithmetic assumes, and they will be confidently wrong rather than obviously wrong. The first month of a deployment sits outside that domain by construction.

Calibration also drifts under exactly the conditions that matter. A new product, a new refund policy, a change of model version, a seasonal shift in ticket mix — each moves the relationship between score and outcome, and the lookup table built in January describes January. Recomputing it on a rolling window and alerting when the observed rate in a bucket diverges from the predicted one is the maintenance this approach carries, and it is the thing most likely to be quietly abandoned after the engineer who built it moves teams.

Building any of this in the order it was presented would be a mistake, since routing needs telemetry that does not exist on day one. What works is to start with the narrow gate set defended on reversibility and blast radius, put the approval in front of the effect and bind it to the payload so retries and parallel branches cannot walk around it, make the pending state a durable row with an owner and an expiry, and render a card somebody can actually decide from. Run that for a quarter, with probes and post-action outcomes recorded from the first week. The routing thresholds are what you write once those numbers exist, and they are worth writing because the alternative — every action of a class getting the same treatment forever — is what put 5,312 cards a week in front of four people in the first place.