Reducing agent autonomy

The containment in the previous chapter works by re-deriving every authority-bearing fact from trusted state, and it runs out when there is no trusted state to re-derive from. An agent that reads inbound vendor emails and updates a purchasing system — changes a delivery address, adjusts a quantity, updates payment terms — is one of those cases. "Change the delivery address to the one in this email" has no independent source to verify against, because the address is the thing the vendor is legitimately allowed to change. Injection testing puts the payload success rate at 4% for messages that cause an unauthorised update, and the business case for the agent is twenty minutes of saved work a day for one purchasing clerk.

What is left once containment has run out is a decision about how much authority the agent gets, and the honest answer is that the strongest available mitigation is to turn the autonomy down. Engineers reach for it last because it feels like a defeat: every prior chapter tried to make the agent safe while keeping it autonomous. An agent that drafts an update for a person to apply has a completely different risk profile from one that applies the update itself, and the difference is not a percentage on the same axis. It is the presence or absence of an attacker-reachable write path.

The trade is not injection against convenience#

The way to make the decision legible is to lay the two designs against the dimensions that move a launch call, rather than arguing about the success rate in isolation. For the supplier-invoice version of the same problem — Design A posts an invoice to the ledger autonomously, Design B extracts the fields and presents them for a person to approve at about 15 seconds per invoice across 400 invoices a day — the comparison looks like this.

The row that most changes how people vote is the injection risk stated as a capability rather than a percentage, because "4% success rate" is abstract and "anyone who can email you an invoice can change where your payments are sent" is not. For Design A, the risk is exactly that: the population who can reach the agent is the population who can send an invoice, which is unbounded and unauthenticated, and a successful payload posts directly to the ledger. For Design B the same population can, at most, cause a proposed entry to appear in a human's review queue, and to turn that into a ledger change they must also defeat the reviewer. Phrasing the risk as the set of people who can trigger it, crossed with what a trigger can reach, is what makes the two designs feel as different as they are; the percentage flattens a capability difference into a scalar and hides the thing the reviewer needs to weigh.

Read down the last three rows and the decision is not close for a write into a financial system. Design A converts "anyone who can send you an invoice" into "anyone who can write to your ledger", throttled only by the 4% success rate, and 4% of a channel that receives hundreds of adversary-authored documents a day is a working exploit that runs unattended. Design B costs about 100 minutes of clerk time a day and removes the write path entirely, because every ledger entry now requires a human action that an injected invoice cannot itself perform. The 100 minutes is a real cost and it is the price of not selling an attacker a write path into purchasing for the value of twenty minutes of saved work.

Two different clocks are running in that comparison and they belong to two variants of the same problem, so it is worth keeping them apart before either number is quoted. The twenty minutes is the daily saving claimed for the purchasing agent, the one reading vendor emails for one clerk. The hundred minutes is the daily review cost of the draft-and-approve design for the supplier-invoice variant, and it is where it is because four hundred invoices at about fifteen seconds of review each is six thousand seconds of clerk time, which is a hundred minutes. Each design has to be weighed against its own saving rather than against the other's, and the reason both numbers appear in one paragraph is that they point the same way: in the variant where the review cost is largest, the write path being closed is a ledger, and in the variant where the saving is smallest, the write path being closed is the payment address. The argument does not depend on borrowing one variant's figure for the other, and it would be a weaker argument if it did.

Rubber-stamping is a real decay and not a refutation#

The strongest objection to Design B is that the approval step is theatre. Finance argues that within a month the clerk will click approve without reading, so you have paid 100 minutes a day for a control that does nothing, and you may as well ship A and save the time. Half of this is correct. An undifferentiated approval prompt — the same dialog four hundred times a day, approve mostly identical entries — does decay into reflexive clicking, and a control that depends on sustained human attention to routine events will lose that attention. Pretending otherwise is how approval steps get designed badly.

The conclusion does not follow from the correct half, because it discards what survives the decay. Even a rubber-stamped approval changes the attack from "send an invoice, the ledger updates" to "send an invoice, and also get a human to click a button", and that second clause is not nothing. It caps unattended exploitation: an attacker cannot run this at scale in the middle of the night, because every success now waits on a human action, so the volume an adversary can push through is bounded by human throughput rather than by network throughput. It also produces an audit trail with a named approver, which turns an anonymous automated write into an action attributable to a person, and attribution changes both the forensics after an incident and the behaviour before one. A write path that requires a human step, even a degraded one, is categorically smaller than a write path that requires nothing.

The right response to rubber-stamping is to fix the gate rather than delete it. Make it rarer and more informative, so the human attention it consumes is spent where it matters.

def needs_human_approval(invoice, proposal):
    # Auto-apply the boring majority; gate only the entries where an
    # injected change would actually move something.
    if proposal.matches_open_purchase_order() and proposal.amount <= po_total(proposal):
        return False                      # reconciles against trusted state: apply
    if proposal.changes_bank_details or proposal.changes_remit_to_address:
        return True                       # exactly the field an attacker wants
    if proposal.amount > APPROVAL_THRESHOLD:
        return True
    return False

A gate that fires on every invoice trains the clerk to ignore it. A gate that fires only when the proposed change touches bank details, redirects a payment address, or exceeds a threshold fires rarely enough that each firing carries signal, and it fires on exactly the changes an injected invoice would make. If ten of four hundred daily invoices trip the gate rather than all four hundred, the clerk is reviewing forty times fewer prompts, and each one is a genuine anomaly rather than a routine confirmation, so the attention that decayed under an undifferentiated prompt has somewhere worth going. The 100 minutes of review time collapses toward a few minutes spent on the entries that could actually cause harm, which weakens the finance objection on its own terms: the cost that made rubber-stamping look inevitable was itself a symptom of a gate that fired too often. Put the two arithmetics side by side and the collapse is stark: four hundred entries at about fifteen seconds each is six thousand seconds, and ten entries at the same fifteen seconds is a hundred and fifty, which is two and a half minutes against a hundred. The point that is easy to miss is that this survives the per-item review getting slower. Even if each of the ten took half again as long as a routine confirmation, because it is now a genuine anomaly being read properly, the total is still a small number of minutes. Two factors are moving in opposite directions here — how many reviews and how long each one takes — and the count is falling by a factor of forty while the duration rises by a fraction, so the product is dominated by the count. That is the general shape of every good gating decision: buy attention back by firing less often, then spend the surplus on making each firing worth reading. The boring majority — invoices that reconcile against an open purchase order, the buyer's own record of what was ordered and at what price — apply automatically, which is the containment from the previous chapter doing the work it can, and the residue that cannot be reduced to a trusted-state check is where the human is spent. Approval and containment are not alternatives; containment handles what reduces to trusted state, and approval handles the remainder.

Putting numbers on the cap makes it easier to defend in a launch review than the argument from principle. The purchasing mailbox receives roughly 300 vendor emails on a working day, and every one of them is authored by someone outside the company. At a 4% payload success rate, an attacker who sends fifty crafted messages during business hours lands two unauthorised writes; sending five thousand overnight lands about two hundred, and in the autonomous design all two hundred are applied before anyone opens a laptop. Under the gated design the same five thousand messages produce at most five thousand queued proposals, none of which becomes a ledger entry until a clerk acts, and a queue that has grown by five thousand overnight is itself the alert. The ceiling on exploitation moves from network throughput to human throughput, and the attacker loses the one property that made the channel worth attacking, which was that it ran while nobody was watching.

The step in that comparison worth making explicit is what each design multiplies the attacker's message count by. In the autonomous design the chain is messages sent, times the success rate, equals ledger writes, and every term is under the attacker's control or fixed by the model, so the output scales with how fast they can send. In the gated design the same product gives queued proposals rather than writes, and the conversion from a proposal to a write is a separate factor supplied by a person acting during working hours. Multiplying by something an attacker cannot increase is what turns an overnight campaign into a morning's worth of clicking, and it is why the reduction is a change in kind rather than a smaller percentage. Autonomy is the term in that product that the team owns; the send rate and the payload success rate are not.

What the approver is shown decides what the gate is worth#

A gate that fires ten times a day is worth whatever those ten screens contain. The first version of the purchasing gate rendered the model's own summary of the email — "Vendor requests delivery address update for PO-4471 to their new distribution centre" — above an approve button and a reject button. That screen hands the approver a description of the change written by a pipeline the attacker was trying to steer, so a message carrying both the injected instruction and a plausible cover story for it produces a summary that agrees with the injection. The clerk reads a coherent, ordinary-sounding request and approves it, and the audit trail records that a named person reviewed and accepted the change, which is worse than no gate at all because the record now claims a review happened.

Say plainly what that screen was asking the clerk to do. It asked her to decide whether a proposal was legitimate, and it gave her, as her evidence, a description of the proposal written by the pipeline that produced it. If the proposal is honest the description is honest. If the proposal was steered, the same steering produced the description, because both came out of the same model reading the same email. So the summary agrees with the proposal in every case, including every case the gate exists to catch, and a piece of evidence that agrees with the claim regardless of whether the claim is true carries no information about it. This is the shape to watch for anywhere a check is designed: the hint the reviewer is given has been written by the party that would be lying. It is tempting because the summary is genuinely the most readable thing on the screen and because the model is usually right, and the cases where it is usually right are exactly the cases nobody needed a gate for.

What the screen has to carry is the material the model did not author. A field-level diff against the record of trust — the remit-to address, meaning the destination a payment against this vendor actually goes to, Unit 4, Brookvale Estate becoming PO Box 8812, Willemstad — shows the approver the two strings without an intermediary. The raw source, quoted rather than summarised, with the sender address and whether the domain matches the vendor on file, lets them see the request in the words it arrived in. A one-line statement of the effect of approving, naming the system and the record that will change, closes the gap between reviewing a description and authorising a write.

The general form of the rule is that the approval surface must not consist solely of model output, for the same reason the previous chapter's authoriser must not take its inputs from the model. A human gate is a check like any other, and a check whose evidence is supplied by the component you are checking has been talked out of its job.

Three answers, and what changes each one#

The full decision has three outcomes and it is worth being explicit that "require approval" is the middle one, not the default. Ship autonomously when the action is contained by a check against trusted state, so there is no attacker-reachable write path to gate — a refund verified against order ownership and total does not need a human, because the payload cannot move it. Require approval when the action cannot be reduced to a trusted-state check and its blast radius is more than you will accept unattended, which is the invoice bank-detail change and the purchasing address update. Decline the agent when even the gated design does not clear the bar — when the approval load is so high it will certainly be rubber-stamped and the blast radius of a slipped approval is catastrophic, or when the saved work is so small that neither shipping nor gating is worth the exposure at all.

What would flip the purchasing decision from B to A is a change in one of the inputs, and naming them keeps the decision honest rather than dogmatic. If the writes could be constrained to a set the attacker cannot profit from — address changes only to addresses already on the vendor's verified record, amounts only within an existing purchase order — the action would reduce to a trusted-state check and ship autonomously, because containment would have caught up with it. If the success rate were driven to zero by a control you could actually operate and measure, the write path would close.

Neither of those is true today, which is why the answer today is to draft for a human and gate the dangerous residue. The autonomy was the expensive variable, it was the one actually under the team's control, and turning it down was the move that bounded the damage the model's compromise could reach — which is the property this entire topic has been chasing, from the first triage meeting where the visible incident was the wrong one to worry about.