Review capacity, not review policy, is the actual control

The design document said that low-confidence outputs would be routed to human review, and the compliance reviewer signed it off on that sentence. The system classifies 50,000 items a day and routes about 3% of them, which is 1,500 reviews landing in a queue every day. The operations team that inherited the queue has four people on it, working roughly seven productive hours each, so the queue arrives at 1,500 items against 28 person-hours, which is 67 seconds per item if reviewers do nothing else and take no breaks. Whether the control described in that document exists depends on a number nobody in the room computed: how long it actually takes to review one of these correctly.

For the item in question — a classified support interaction with a 250-word transcript, a suggested category and a suggested refund amount — the answer measured on a trained reviewer working carefully was about 90 seconds. At 90 seconds, 1,500 items a day needs 37.5 hours of focused work against 28 available. The queue cannot be cleared by anybody being diligent, which means it will be cleared by somebody being fast, and that is not a policy decision anyone will make explicitly. It happens by the third week.

What four seconds buys#

Six months on, the dashboard for this queue reads: 99.4% approval rate, median review time four seconds, zero backlog. It is a natural reading that the model has improved to the point where the reviewers rarely find anything, and it is the reading most often offered, because the alternative implicates the team you are looking at. The four-second median rules it out on its own. A 250-word transcript takes about 50 seconds to read at a normal reading speed of 300 words per minute, and the fastest defensible review — glance at the category, confirm it matches an obvious keyword, approve — does not go below 15 to 20 seconds. Four seconds is the time it takes to move a mouse to a button that is already under the cursor and click it. Whatever is happening in that queue, reading is not part of it.

That is an inference, and the interesting work is turning it into evidence, because "I think your team is rubber-stamping" is a claim that needs more than an argument about reading speed before you take it to the person who runs the team.

Seeding is the strongest of these and the least often implemented. You inject items with known-wrong model output into the live queue at a low rate, indistinguishable from real work, and measure how many are rejected. On this queue, 200 seeded items over two weeks — deliberately mis-categorised interactions, several with refund amounts an order of magnitude too high — came back with 11 rejections. A 5.5% catch rate on errors that were obvious to anyone reading the item is not a marginal result that needs careful interpretation, and it converts the conversation from an accusation about effort into a measurement of a system.

The offline audit sizes the damage. Pulling 500 approved items and having two senior people grade them properly found 6.2% wrong, which against 1,500 approvals a day is about 93 bad decisions reaching customers daily, all of them carrying an audit trail that says a human approved them. That last detail is what makes a rubber-stamped queue worse than no queue at all: it manufactures documented human sign-off for machine errors, and every downstream process, every incident review and every regulator treats that sign-off as meaningful.

This is alert fatigue with a different label#

The mechanism is familiar to anyone who has run an on-call rotation or sat near a security operations team. A monitoring system that pages twelve times a night, where eleven are noise, does not produce twelve careful investigations; it produces a reflex acknowledgement and, eventually, a missed real page. The literature on this in aviation and clinical decision support calls it automation bias — the tendency to accept a system's recommendation because it is there, more strongly when the operator is under time pressure and when the system is usually right. Studies of clinical decision support have found double-digit rates of clinicians reversing a correct decision after a wrong automated prompt, and those are professionals reviewing a handful of cases, not a queue of 1,500.

Three properties of a review queue make the effect worse, and all three are things you choose when you build it. The reviewer sees the model's answer before forming their own, which anchors them. The interface makes approving cheaper than rejecting, since approve is one click and reject requires selecting a reason and often typing. And the base rate is high — if 97% of what arrives is correct, then approving everything scores 97%, and a reviewer who is measured on throughput is being paid to do exactly that.

Changing those is more effective than exhortation. Requiring the reviewer to enter the category themselves before the model's suggestion is revealed converts confirmation into an independent judgement, and it costs about 20 extra seconds per item, which is a real cost that must come out of volume somewhere. Making reject as cheap as approve — a single click with the reason inferred, corrected later if needed — removes the asymmetry. Measuring reviewers on agreement with a gold set — a fixed pool of items whose correct answer has been established in advance by someone senior — rather than on items per hour changes what the job optimises for.

Sizing the queue from the capacity you have#

The routing threshold and the staffing level are the same decision, and treating them as independent is what produces the failure above. Working from capacity backwards gives a defensible routing rate.

def max_review_rate(daily_volume, reviewers, hours_per_shift, seconds_per_item,
                    focus_factor=0.75):
    """Largest share of traffic that can actually be reviewed, not merely routed."""
    seconds_available = reviewers * hours_per_shift * 3600 * focus_factor
    reviewable_items = seconds_available / seconds_per_item
    return reviewable_items / daily_volume

print(f"{max_review_rate(50_000, 4, 7.5, 90):.2%}")   # 0.90%
print(f"{max_review_rate(50_000, 4, 7.5, 20):.2%}")   # 4.05%

Four reviewers at 90 seconds an item can genuinely review 0.9% of 50,000 daily items, not 3%. That leaves three options, and the choice among them is a product decision rather than an engineering one: staff to 3% by hiring nine more reviewers at whatever a reviewer costs; route 0.9% by raising the confidence threshold, which means accepting that the items between the old and new threshold go through unreviewed and pricing that against chapter one's budget; or reduce the per-item cost, by presenting less, pre-computing the checks a reviewer performs manually, or splitting the item into a 10-second triage and a 90-second deep review for the fraction that triage flags.

The focus_factor is the number people argue about and it should stay conservative. Reviewers attend meetings, handle escalations, and cannot sustain continuous judgement for seven hours; 0.75 is generous rather than pessimistic, and a plan that only works at 0.95 is a plan that fails in its first week.

The routing rule itself also deserves more thought than "send the low-confidence ones", which is the default and is right only for some features. For a high-volume, low-risk, reversible action, a person on every uncertain item is expensive and mostly wasted, and a sampled audit of 200 items a day — chosen randomly, weighted toward the segments chapter two identified as clustered — buys a much better estimate of the true error rate per hour of reviewer time. For an irreversible or regulated action, sampling is the wrong instrument entirely, because the whole point is that a specific instance must not proceed unchecked, and that review has to sit in front of the commit hold rather than after it. Most systems need both, applied to different action classes, and a single global review rate is a sign that the classification from chapter one was never carried through into the design.

What you do on the Monday#

The measurement is only useful if it leads somewhere, and the sequence matters because the first instinct — tell the reviewers to slow down — changes nothing, since the queue arrives at the same rate the following morning.

The first move is to stop claiming the control in places where the claim has consequences. If a compliance document, a customer contract or a data protection assessment says that these decisions receive human review, then a queue catching 5.5% of errors is a statement to the regulator that is not true, and correcting the document is more urgent than fixing the queue. It is also the step most likely to unlock the headcount, because a control described accurately as "sampled quality assurance" attracts different scrutiny than one described as "human approval".

The second is to shrink what is routed until it fits the capacity that exists, using impact rather than volume. On this queue, the items where the suggested refund exceeded £200 or where the category carried a regulatory obligation came to about 240 a day, which four reviewers can genuinely handle at 90 seconds with room left over. Everything else moves to a sampled audit — 150 randomly chosen items a day, graded properly, producing an error estimate rather than a decision. Total coverage falls from 1,500 to 390 items, and the number of errors actually caught goes up by roughly a factor of eight, which is the counterintuitive part to bring to the meeting where you propose reviewing less.

Third, change what the interface and the incentives ask for. Hide the model's suggestion until the reviewer has entered their own answer, make rejection a single click, and replace items-per-hour on the team dashboard with catch rate against seeded items. Seeding then stays on permanently at about 1% of queue volume, which is fifteen items a day, cheap enough to run forever and the only continuous evidence that the control still works. A queue whose seeded catch rate is published weekly does not silently decay, because the decay is visible while it is happening rather than six months later.

The last piece is an escalation path, since a reviewer who is genuinely unsure has nowhere to go in most queue designs and will approve rather than block the line. A third option beside approve and reject — send to a senior reviewer, with no penalty and a target of a few percent of items — costs little and converts a class of silent wrong approvals into a small, visible workload. Watching that escalation rate is also a decent proxy for whether the routing threshold is set sensibly, since a rate near zero usually means the queue is receiving cases that were never ambiguous.

What review is not#

A human in the loop is not a correctness guarantee, and quoting one as a mitigation without a number attached is the same error as quoting a model's accuracy without a volume. Careful reviewers on tasks like this run somewhere between 85% and 96% themselves, so a reviewed pipeline does not go to zero errors; it goes to the product of two error rates, which is a large improvement and not an elimination. Review also adds latency — an item held for a queue that clears twice a day is an item delivered a day late — and that latency is part of the cost of the control.

The uncomfortable property is that review capacity does not scale gracefully with caution. Doubling the routing rate with fixed staff halves the time per item, which lowers the catch rate on every item including the ones that were being caught before, so a nervous team that widens the net can reduce the number of errors it stops. The control has a maximum, and past that point the only remaining place for uncertainty to go is to the person using the product.