What a 96% filter costs at eighty thousand uploads a day

A team shipped an image-injection filter and reported it caught 96% of their red-team set. The number felt like a grade — 96 out of 100, a solid B-plus — and it went into the launch review as evidence the risk was handled. What that number actually measured was the filter's performance against forty payloads the same team had written. The denominator was their own imagination. Somewhere out past it were the payloads they had not thought of, and the filter's score against those is not 96%; it is unknown, because there is no test set for the attacks you cannot conceive. The 96% is a real measurement of a fake population, and mistaking it for a coverage estimate is the single most common error in evaluating one of these systems.

Meanwhile the same launch review had a second number that was fully real and got a fraction of the attention: in shadow mode — the filter scoring live traffic and recording its verdict without rejecting anything — it flagged 2.1% of legitimate uploads. The product processes 80,000 uploads a day. That is 1,700 real users a day whose genuine screenshots the filter would reject, every day, measured against actual traffic with a solid denominator. One number is a comforting fiction with no denominator; the other is a hard fact about 1,700 blocked people. The launch review weighted them the other way around.

The number with no denominator, and the one that is all denominator#

The asymmetry is worth stating carefully because it decides how you should read every filter metric you will ever produce. Detection rate against a red-team set answers "of the attacks we wrote, how many did we catch?" That is a fine question for regression testing — it tells you whether a change broke coverage you previously had — and a useless one for estimating real-world protection, because the attacks that matter are the ones not in the set. False-positive rate against live traffic answers "of real uploads, how many do we wrongly reject?" and that question has a denominator you did not invent: every legitimate upload your users actually send. Moving the filter's threshold trades the second number, which you can measure, against the first, which you cannot. Loosen the threshold and false positives fall while detection of unknown attacks falls by an unmeasurable amount; tighten it and false positives climb by a number you can watch in a dashboard. You are always trading a measurable cost against an unmeasurable benefit, and it is very easy to talk yourself into paying the measurable cost because the benefit sounds large.

This is the honest reading behind the receipt-scanning question: your upload filter blocks all 40 payloads in your red-team set and flags 3% of legitimate uploads, on a product handling 50,000 uploads a day — what is the honest reading? Blocking 40 of 40 is not "perfect coverage"; it is "perfect against the 40 we wrote." The 3% is 1,500 legitimate uploads rejected daily, on a product whose entire job is to accept uploaded receipts. That is a heavy, measurable price paid for attack coverage that cannot be measured at all, and calling the filter a success on the strength of the 40-of-40 is exactly the mistake the chapter opened with.

Move the threshold, or move the filter#

The interview question puts the same trade in a form that demands a decision. You built an injection filter for uploads. It stops 96% of your red-team set. In shadow mode it flags 2.1% of real uploads, and you process 80,000 a day, which product correctly points out is 1,700 blocked users a day. What do you do? The shallow answer is to move the threshold until the false positives are tolerable and accept the reduced detection rate as the cost of usability. That answer treats the filter as the boundary and the threshold as the only knob, so the whole problem collapses into a single slider between "block real users" and "let attacks through." Slid either way, it is a bad trade, because it is trading against the unmeasurable benefit.

The better answer refuses the framing. The filter is not the boundary; the capability boundary and the provenance taint from the previous two chapters are the boundary, and they do not depend on the filter catching anything. Once the architecture holds — an image turn cannot issue a refund, tainted content cannot trigger an irreversible action without a human — a payload that gets past the filter reaches a model that cannot act on it in a way you would regret. That changes what the filter is for. It is no longer the thing standing between an attacker and your money; it is a rate limiter sitting behind the boundary, thinning the volume of injection attempts that reach the model so the model spends fewer tokens processing obvious garbage and your logs are quieter. A rate limiter should be tuned to almost never reject a legitimate user, because its job is throughput reduction, not protection, and the protection lives elsewhere. At 1,700 blocked users a day you have a rate limiter tuned as if it were a wall, and the fix is to loosen it dramatically — accept that more injection attempts reach the model — precisely because the wall behind it is what actually stops the damage.

Measure the boundary on its own terms#

The trap in all of this is measuring the whole defence with the filter's yardstick. Attack success against an architectural control is not a detection rate, and reporting it as one hides what the control actually does. The capability boundary does not "catch" payloads; it makes them inert. The right measurement is not "what fraction did we detect" but "on the turns where a payload got through, could it have done anything irreversible?" — and for a correctly enforced boundary the answer is no, independent of whether the filter flagged the image. That is a different quantity, measured differently, and it is the quantity that matters.

Measuring the boundary also means measuring what it costs, because the taint that never clears and the confirmations that gate irreversible actions are not free. The honest instrumentation records, per conversation, which turns ran degraded, which tools they lost, how many tasks failed to complete because a needed tool was withheld, and how many extra human confirmations the boundary forced. Those are the boundary's costs, and they are as real as the filter's false positives — a support agent that loses the refund tool on every turn after a screenshot arrives will escalate more, complete fewer refunds unattended, and push more work to humans. You cannot decide whether the boundary is tuned correctly without those numbers, and no detection rate contains them.

The checklist item is to patch the injection architecture from Stage 7 and measure attack success, false positives, latency, and cost before and after. Run it as a genuine before-and-after, with each control changing one thing, so you can attribute movement to a cause.

The rows that improve and the rows that regress are both the point. Attack success drops to roughly zero and stops depending on the filter; false positives fall by an order of magnitude because the filter no longer has to be a wall; and the two new costs — extra confirmations and a few percent fewer unattended completions on image turns — appear on the ledger where they belong, as the measurable price of an architecture that holds. A defence with no visible cost is usually a defence that isn't doing anything, and the goal of measuring before and after is to see the costs clearly enough to tune them, not to make them disappear.

Getting a denominator you did not write

The red-team score has no denominator, and there is a partial remedy that costs a few hours a month: harvest attacks from live traffic instead of inventing all of them. Keep the classifier scoring every upload even when the threshold is loose enough to block almost nothing, so that flagging becomes a logging decision rather than a rejection, and pull a fixed sample from the high-scoring tail for human review each week. On this product that meant 20 uploads a week out of the roughly 160 a day the loosened filter flagged, or 480 reviewed over six months.

Eleven of those 480 were genuine injection attempts. Eight belonged to families already in the red-team set — white-on-white instruction text, a fake system banner across the top of a screenshot, a block of text styled to look like a tool result. Three did not, and one of them never touched the pixels at all: it rode in the EXIF block, the metadata a camera or an editor writes inside an image file alongside the picture data.

Each family was added to the regression set the week it appeared, and within a release the filter's score against the whole set was back around 96%. A score that returns to 96% after every discovery is describing a set that keeps being repaired to match what the filter already handles, which is the behaviour of a regression test working correctly and says nothing about the families nobody has met yet. The numbers worth putting on the dashboard are the ones the harvest produces directly: how many novel families were found this quarter, how long each had been arriving in traffic before anybody noticed, and how many uploads were reviewed to find them. A quarter with zero novel families is ambiguous until you can show the sampling rate that failed to turn any up.

Operating the whole thing#

The seven chapters have moved from the channel your text filters cannot see, through the three paths a single file takes, the scaling attack your own resize develops, the trust boundary that never sat at the upload form, the capability limit that holds where model compliance does not, the provenance taint that carries that limit across a conversation, and finally the measurements that tell you whether the assembly is tuned right. What you have now is not a checklist you run once but a system you operate. Treat every image — uploaded, fetched, attached, retrieved — as untrusted; keep the model able to read it and unable to act irreversibly on it; carry that constraint with the derived content; and watch four numbers, knowing that one of them is honest and one of them flatters you. The filter's red-team score will keep looking like a grade. Read it as a regression test, put the wall behind it, and spend your attention on the false-positive rate and the completion rate, because those are the numbers your users and your finance team will feel.