2,000 wrong outputs a month is the number, not 99% accuracy

Two weeks before launch, the eval dashboard for the ticket-routing step read 99.1% on a held-out set of 2,400 tickets, and the room took that as the end of the accuracy conversation. The feature reads an inbound support email, assigns it to one of eleven queues, and sets a priority; once it is on for every customer it runs about 200,000 times a month. Nobody in the review converted the percentage into a count. At 99.1% over 200,000 runs the feature produces roughly 1,800 wrong routings a month, about sixty a day, and whether sixty a day is a rounding error or the reason support misses its response-time commitments in week three depends entirely on what one misroute does after it happens.

Everything in this topic starts from one property of that routing step, and it is worth naming before the arithmetic gets going. The model inside it is a dependency that is wrong sometimes, and no amount of prompt work, fine-tuning or provider upgrade takes that rate to zero, because the component is statistical by construction: it produces the most plausible output given its inputs, and plausible and correct are different things. That is what this topic means by an imperfect component — a part of the system whose failures are a normal operating condition rather than a defect awaiting a fix, in the way that a network link drops packets and a disk occasionally returns a read error. Designing around an imperfect component is the work of arranging everything else so that the mistakes it is guaranteed to make stay cheap, contained and visible: the volume you run it at, where its output lands, what commits and when, who checks the result, and what the user is told when the system is unsure. The accuracy of the model is more or less fixed by the time the product is being built, whereas the consequences of its errors are almost entirely yours to shape, and that asymmetry is why the design work is worth more than another week of prompt tuning.

You meet this work at a specific moment, which is the one in the room above: a feature has been evaluated, a number exists, and somebody has to decide whether to turn it on for everybody. It comes back in design review, when someone asks what happens if the model gets this wrong; in incident review, when the answer turns out to have been nothing in particular; and in the interview loop for lead roles, where a question of the form "it measures 97% accurate, would you ship it" is standard and has no good answer that does not go through the material in this topic. The problem all of it solves is narrow and expensive. A model wrong at an ordinary, respectable rate still produces mistakes that look entirely reasonable on the way past, and a plausible mistake landing in a system with nothing arranged around it is how a defensible error rate turns into a product incident with a name, a postmortem and a customer on the call. Treating the model as a fallible dependency — the same way you already treat a third-party API that returns 500s — is what keeps those two things apart.

The first piece of that arrangement is the one the launch review skipped. An error budget is a written statement of how many wrong outputs a feature is allowed to produce, justified by what one of those outputs costs and who absorbs it, together with what happens when the number is exceeded. It is the artefact that turns "99.1% on the eval set" into a claim a business can agree to or refuse, and writing one begins with a conversion nobody in that room performed.

That conversion is the whole move. A percentage is a property of a model on a dataset; a count of wrong outputs is a property of a product in a month, and only the second one can be priced. Round 99.1% to 99% and the arithmetic gets easier to hold in your head: one in a hundred, 200,000 runs, 2,000 wrong outputs a month. The same rounding works in the other direction and is the more common shape in interviews — 97% accurate, 100,000 runs a month, so 3,000 wrong outputs a month. Neither number is good or bad on its own. The question that makes them answerable is what a single one of those wrong outputs costs, who absorbs that cost, and whether anybody notices it happened.

The same 1% across five features#

Detection is the variable that changes the answer most, and it is the one teams tend to leave out. An error a user sees immediately and fixes in two seconds costs almost nothing beyond irritation; the same error committed silently into a downstream system costs whatever it costs when it surfaces three months later, plus the cost of the investigation that finds it. Here are five features, all of them running 200,000 times a month, all of them wrong 1% of the time, so all of them producing 2,000 wrong outputs a month.

The middle three rows are the interesting ones. Ticket routing at 2,000 errors a month is an operational cost you can put in a budget line and defend: £8,000 a month against, say, £140,000 a month of agent time saved is a trade the business will take. The refund agent at the same error rate burns £90,000 a month in unrecoverable credits, which may still be worth it if the feature deflects enough contacts, but it is now a conversation with finance rather than a conversation with engineering. The VAT row is worse than its average suggests, because the cost is not uniform across the 2,000 — most are caught in reconciliation and cost half an hour, and the 8% that reach a filed return carry a different kind of exposure entirely.

Getting from "1% error" to a number you can defend means writing the cost model down, because the model is where the assumptions become visible and arguable.

from dataclasses import dataclass

@dataclass
class ErrorCost:
    monthly_runs: int
    error_rate: float          # measured, on traffic that looks like production
    p_detected_before_commit: float
    cost_if_caught: float      # rework, re-review, a retry
    cost_if_not_caught: float  # what it costs once it has landed

    def wrong_outputs(self) -> float:
        return self.monthly_runs * self.error_rate

    def monthly_cost(self) -> float:
        n = self.wrong_outputs()
        caught = n * self.p_detected_before_commit
        return caught * self.cost_if_caught + (n - caught) * self.cost_if_not_caught


routing = ErrorCost(200_000, 0.01, 0.95, cost_if_caught=4, cost_if_not_caught=180)
refunds = ErrorCost(200_000, 0.01, 0.02, cost_if_caught=1, cost_if_not_caught=45)

print(routing.wrong_outputs(), round(routing.monthly_cost()))   # 2000.0 25600
print(refunds.wrong_outputs(), round(refunds.monthly_cost()))   # 2000.0 88220

Routing comes out at about £25,600 a month rather than £8,000 once you account for the 5% of misroutes that are never caught and become a breached SLA at £180 apiece. That £180 and that 5% are estimates, and they should be labelled as estimates with a source — a sample of thirty historical misroutes, a support lead's judgement, a contractual credit schedule. An error budget built on a guess that is written down and dated can be corrected when the guess turns out to be wrong. An error budget built on a percentage that sounded respectable has nothing in it to correct.

Two kinds of error that do not multiply#

The arithmetic above assumes every error is one unit of the same thing, so that halving the rate halves the cost. Two categories break that assumption, and both of them tend to be the ones that end careers.

A regulated error costs what it costs regardless of how many there are, because the unit of harm is the event and not the volume. One auto-dismissed KYC alert on an account that later turns out to be laundering money is a reportable failure, an internal investigation and a conversation with a supervisor; the fact that it was one in 200,000 is a mitigating detail in that conversation rather than a defence. The same shape appears in a wrongly suppressed adverse-event report in pharma, an incorrectly processed erasure request under GDPR where the clock is measured in days rather than in error rates, and any decision that produces a document a regulator can subpoena. For these, the budget is not a rate. It is a statement that the model does not make this decision unaided, that a human record exists for every instance, and that the failure mode you are budgeting for is the review being skipped rather than the model being wrong.

An abusive case breaks the arithmetic from the other side, because its rate is chosen by an attacker instead of measured from your traffic. Your 1% is an estimate of how often the model is wrong on the inputs your users happen to send. It says nothing about how often the model is wrong on inputs crafted specifically to make it wrong. A refund agent that is 99% accurate on organic support emails can be close to 100% wrong on an email written by someone who has spent an afternoon probing it, and the moment that email works it gets posted somewhere and arrives ten thousand times. Measured accuracy has no predictive power over adversarial accuracy, which is why the architectural mitigations from topic 7.4 sit in front of these paths rather than a threshold: the control is a hard cap on what the action can do, not a confidence number attached to the decision to do it.

A budget with more than one line in it#

That gives you four classes of error rather than one, distinguished by what a single instance does. A harmless error is one the user or the system notices and retries, at a cost dominated by irritation. A costly-but-reversible error has real money attached and a path to undo it. An irreversible error commits value or state you cannot get back. A regulated error creates an obligation the moment it occurs. The budget is per class, and the classes are denominated in different units, because "0.5%" and "zero unreviewed instances" are not the same kind of statement.

feature: support_ticket_routing
monthly_runs_estimate: 200000
budget:
  harmless:
    definition: wrong queue, corrected by the receiving agent within one pickup
    threshold: <= 2.0% of runs
    rationale: 4,000/month at ~£4 rework = £16k, against £140k of agent time saved
    on_breach: page nobody; review at the next weekly
  costly_reversible:
    definition: wrong priority causing a first-response SLA breach
    threshold: <= 60 incidents/month
    rationale: £180 credit per breach; £10,800 is inside the support credit pool
    on_breach: alert the on-call product owner; consider raising the routing threshold
  irreversible:
    definition: ticket auto-closed as spam and never seen by a human
    threshold: 0 — auto-close is disabled; the model may only deprioritise
    rationale: no detection path exists, so no rate is defensible
  regulated:
    definition: a ticket containing a GDPR erasure request routed away from the DPO queue
    threshold: 0 unreviewed; keyword rule overrides the model and logs every match
    rationale: one instance is a reportable event with a 30-day statutory clock

The interesting part of that document is the last two entries, where the threshold is zero and the mechanism is not a model change. Auto-close is off because no detection path exists, and a rate you cannot measure is a rate you cannot budget. The erasure-request line is enforced by a deterministic keyword rule that runs before the model and cannot be overruled by it, which is a worse classifier and the right control, because its failures are the kind you can enumerate.

Before you can say whether 97% is good#

Given a feature at 97% accuracy running 100,000 times a month, the arithmetic gives you 3,000 wrong outputs and nothing else, and there are five things you need before the number becomes a judgement.

The first is what one wrong output does, expressed as an action rather than a category. "It gets the category wrong" is not an answer; "it routes a refund request to the billing queue, where it waits an average of eleven hours instead of two, and about 5% of those become a chargeback" is. The second is who absorbs it — the user, an internal team, a customer's customer, or a regulator — because a cost absorbed by a team that never agreed to absorb it is a cost that will surface as an escalation rather than as a line in a budget. Support taking on 4,000 extra touches a month is a headcount conversation, and having it before launch is the difference between a feature and a grievance.

The third is whether anyone notices, which is the detection rate in the model above and is usually the weakest-known number in the set. Teams estimate it optimistically because they imagine an attentive user, and the honest way to get it is to sample: take fifty known-wrong outputs from a shadow run, follow what happened to each, and count how many were caught. The fourth is how the errors are distributed across the wrong outputs — whether the 3,000 are 3,000 small ones or 2,900 small ones and 100 catastrophic ones, since the mean cost is a poor summary of a distribution with a tail. The fifth is what happens with no AI at all, because the comparison that matters is not against perfection. If the manual process it replaces runs at 94% with a two-day backlog, then 97% with no backlog is an improvement on both axes and the conversation is about whether the remaining 3,000 are worse-shaped than the 6,000 that existed before. If the rule-based system it replaces runs at 99.7% and is merely tedious to maintain, the feature is a regression that has been described as an innovation.

Two of those five are usually obtainable in an afternoon from logs, and the other three take a week of work that nobody schedules. A launch review where the answer to all five is "we would need to look into that" is a review where the accuracy number cannot be interpreted, and the useful outcome is to say so and put a week in the plan.

What this is not#

An error budget is not an accuracy SLO. An SLO on model accuracy commits you to a number produced by an evaluation harness, and the number moves when the eval set is refreshed, when a provider silently updates a model, or when traffic drifts. A budget commits you to a consequence — pounds, breached SLAs, unreviewed regulated events — and stays meaningful across all three of those changes. It also is not a promise that errors under the line are free; they are paid for, by someone, and the budget's job is to name who. Support absorbing 4,000 misroutes a month is a real cost that shows up as agent headcount, and it should be agreed with the person who owns that headcount rather than assumed.

It is also not a substitute for reducing the error rate, and the trap there is that a comfortable budget removes the pressure to improve the model at exactly the point when improvement is cheapest. The budget's function is to make the trade explicit, so that shipping at 97% is a decision someone made with the cost in front of them rather than a number that went unchallenged in a review.

There is one assumption underneath all of the arithmetic so far, and it is load-bearing: that the 2,000 wrong outputs are spread more or less evenly across the 200,000 runs. Everything in the cost model — the per-error price, the detection rate, the claim that support can absorb four thousand misroutes — depends on those errors arriving as background noise. They do not, and the next chapter is about what happens to the budget when they arrive in a clump.