Errors cluster, so the aggregate can hide one customer's outage

The extraction feature had been live for five weeks at 97% and nobody had escalated anything, which is why the call on the Thursday was confusing. Northbeam Logistics, on a £180,000 annual contract, had been re-keying almost every invoice by hand since go-live and had finally given up and written to their account manager. The dashboard had not moved. The dashboard could not move: Northbeam sent about 1,900 documents a month against a total of 100,000, so even at zero accuracy they could shift the headline number by less than two points.

The postmortem found that Northbeam's warehouse staff photograph paper invoices with a phone app rather than uploading PDFs, and on phone photographs the extraction ran at 63%. That is not a variant of the 97% story. It is a different story with the same arithmetic, and the error budget written in the previous chapter cannot distinguish them, because 3,000 errors in 100,000 runs is 3,000 errors in 100,000 runs whether they are three in every hundred for everybody or every third request from one account.

The p99 habit, applied to correctness#

Backend engineers already refuse to accept a single mean for latency. Nobody reports "mean response time 140 ms" and considers the question closed, because the mean is compatible with 99% of requests at 90 ms and 1% at five seconds, and the 1% is where the pager lives. The instinct to ask for p99, and then to ask for p99 per tenant and per endpoint, is well established for latency and almost entirely absent for model accuracy, where teams still hand around a single scalar produced by one eval run over one sampled set.

Accuracy has no percentiles — a single output is right or wrong, so there is no distribution over a request to take a quantile of. What replaces the percentile is the slice: accuracy computed separately over subsets of traffic chosen so that the subsets differ in ways the model is likely to care about. The headline is then the traffic-weighted average of the slices, which means it is dominated by whichever slice is largest and tells you nothing about the smallest.

The last two rows are the account-level incident. Mobile photo capture is 5% of runs and a third of all errors; one tenant inside it is 1.9% of runs and 23% of all errors. Both facts were present in the logs from day one and neither was visible in the number that was reported weekly.

The dimensions worth slicing by are the ones along which your inputs actually vary, and they are usually a short list: tenant or account, input channel or format, language, document or record type, and the specific field or entity being produced. That last one matters more than teams expect, because an extraction that is 97% accurate over all fields can be 99.4% on invoice number and 71% on tax total, and the tax total is the field that reaches the accounts payable ledger. Slicing by output field turns "the model is 97% accurate" into "the model is unusable for the one thing finance needs it for", which is a materially different sentence to bring to a review.

Small slices lie in the other direction#

The obvious way to run this — group by every dimension, sort by accuracy, look at the worst — produces a wall of terrible-looking segments that are mostly noise. A tenant with eleven requests and two errors reads as 82% accurate, and it might be 82% or it might be 97% having a bad week. Two protections make the output usable: a minimum sample size before a slice is allowed to raise anything, and a confidence bound rather than a point estimate, so that ranking is done on the pessimistic end of the interval rather than on the sample mean. The Wilson interval is the usual choice for a proportion of this kind, because it stays sensible at the small sample sizes where the textbook normal approximation starts returning bounds below zero.

from math import sqrt
from collections import defaultdict

Z = 1.96  # 95%

def wilson_lower(successes: int, n: int) -> float:
    """Lower bound of the Wilson interval — pessimistic accuracy for this slice."""
    if n == 0:
        return 0.0
    p = successes / n
    d = 1 + Z**2 / n
    centre = p + Z**2 / (2 * n)
    margin = Z * sqrt(p * (1 - p) / n + Z**2 / (4 * n**2))
    return (centre - margin) / d

def flag_slices(events, dimension, min_n=200, budget=0.95):
    """events: iterable of (attributes dict, correct: bool)."""
    tally = defaultdict(lambda: [0, 0])          # key -> [correct, total]
    for attrs, correct in events:
        cell = tally[attrs[dimension]]
        cell[0] += int(correct)
        cell[1] += 1
    out = []
    for key, (ok, n) in tally.items():
        if n < min_n:
            continue                             # not enough evidence to act on
        lo = wilson_lower(ok, n)
        if lo < budget:
            out.append((key, n, ok / n, lo))
    return sorted(out, key=lambda r: r[3])

for key, n, acc, lo in flag_slices(month_events, "tenant_id"):
    print(f"{key:<14} n={n:<6} acc={acc:.1%}  lower bound {lo:.1%}")

Run against the month above, the tenant dimension reports northbeam with n=1,900, accuracy 63.0% and a lower bound of 60.8%, which is far enough below the 95% budget that no sampling story explains it. A tenant with n=210 at 91% comes back with a lower bound of 86.6%, which is a genuine flag but a weaker one, and a tenant with n=40 does not appear at all.

The honest cost of this is multiple comparisons, and it is worth stating plainly rather than discovering later. If you slice by five dimensions with a few hundred distinct values between them and flag anything whose 95% lower bound falls below budget, you will flag some slices that are performing exactly at budget, and the number is not small — at 200 slices and a nominal 5% error rate you should expect roughly ten spurious flags per run. The fix is not to abandon the method but to treat a flag as an instruction to look, not as a finding: pull twenty actual failures from the slice and read them. If they share a cause, it is real; if they look like twenty unrelated one-offs, it was noise. Reading failures is also the only reliable way to find the segment you did not think to slice by, which is the case this whole apparatus is weakest against. Northbeam was findable because someone had thought to log capture_source. A cluster caused by a document layout nobody has a field for is invisible to every group-by you can write, and surfaces only when someone clusters the failures themselves — by embedding the inputs, or by reading a sample of a hundred — and notices that thirty of them have a logo in the top right.

Where the labels come from#

None of this works without knowing which outputs were wrong, and on 100,000 monthly runs you are not grading 100,000 outputs. Two things make the measurement affordable, and both have to be decided before the feature ships rather than after the escalation.

The first is logging the slice dimensions at request time. An accuracy number you can slice is an accuracy number where every graded event carries the attributes you will later want to group by, and attributes that were not recorded cannot be reconstructed — a month later you can see that a document failed and you cannot see that it arrived through the phone app, because that field was in the upload service's memory and never made it into the inference log. The list is short and cheap: tenant, input channel, detected language, document or record type, output field, model identifier and prompt version. The last two matter because a slice table computed across a provider version change is comparing two systems, and the resulting "drift" is a deployment, not a data problem.

The second is stratified sampling. A uniform random sample of 1,000 outputs a month gives a headline accuracy with a margin of about ±1 point and gives you nineteen samples from the mobile photo channel, which is not enough to say anything about it. Sampling by stratum instead — 300 from native PDFs, 300 from scans, 300 from photos, and a floor of 100 from any tenant above a contract-value threshold — costs the same grading effort and produces per-slice intervals that are actually usable. The headline is then recovered by re-weighting each stratum by its true traffic share, which is one line of arithmetic and is the step people forget, producing a "97%" that is really the unweighted mean of the strata and is wrong by several points.

Grading is the expensive part, at somewhere between 30 seconds and three minutes per item depending on how much of the source document a grader has to read, so 1,200 items a month is between ten and sixty hours of skilled work. Where the output is checkable against a system of record — an extracted invoice total against the amount eventually paid, a routed ticket against the queue it ended up in after human handling — a large share of that labelling is free and continuous, and building that comparison is usually the highest-value week of instrumentation on the whole feature. It also produces labels on production traffic in the proportions production actually has, which no curated eval set does.

Once the slices exist, the alerting should follow them rather than the headline. A per-tenant rule that fires when any account above a minimum volume drops below its budgeted accuracy for two consecutive weeks would have caught Northbeam in week two, at the point where the fix was a preprocessing change and an apology rather than a renewal conversation. The threshold in that rule is the per-class budget from the previous chapter applied at the level errors actually arrive at.

Whether the 3% is stable#

Two systems can produce identical slice tables in a given month and behave completely differently over time, and the difference determines what you do about it. A stable cluster means the same slice fails at the same rate every month: Northbeam's photographs were 63% in March, 64% in April and 62% in May. That is a defect with an owner and a fix — better OCR preprocessing, a capture-quality check in the app, or a decision that the photo channel is unsupported and should say so. It is predictable, which also means it is plannable.

An unstable cluster is the harder case. The failures move: this month it is one tenant's German invoices, next month it is a layout that a different tenant started sending after they changed accounting software. The rate averages 3% and never sits still, which usually indicates that the model is sensitive to something you are not measuring, and that any per-slice fix will be obsolete before it ships. The diagnostic is cheap — compute the slice table for three consecutive months and check whether the same keys appear in the bottom ten — and the answer changes the remediation from "fix that slice" to "reduce the model's sensitivity", which is a much larger piece of work and should be scoped as one.

Burstiness is the same question at a shorter timescale, and it is the one that damages downstream capacity planning. Northbeam's 703 monthly errors do not arrive at 23 per day. They arrive when the warehouse does its Monday batch, so about 160 of them land inside two hours. Anything sized on the average — a review queue, an alerting threshold, a rate limit on corrective actions — is sized wrong for that Monday. A queue provisioned for 100 items a day is not a control when 160 arrive before lunch and the reviewers spend the afternoon clearing them at speed.

What slicing does not do#

Slicing is a measurement discipline and it changes nothing on its own; a beautifully sliced dashboard on a feature with no per-tenant kill switch has told you precisely which customer is being harmed while you cannot stop harming them. It is also not the same activity as a fairness or bias audit, which shares the mechanics and differs in what it is for: a fairness audit slices by protected characteristics because disparity across those groups is itself the harm, whereas this slicing is about finding concentrations of ordinary defect. The methods overlap; the obligations, the documentation and the people who need to sign off do not, and treating one as a substitute for the other will satisfy neither.

Slicing also does not help with the case where every slice is fine. A model can be 97% accurate uniformly, with errors spread evenly across every tenant, language and format, and still take down a customer — because the harm from a wrong output is not only a function of how many there are and who receives them, but of where the output lands. One wrong output written into one place can reach ten thousand people, and the next chapter is about bounding that.