Comparing mitigations by residual exposure

The choice between mitigations usually arrives as a comparison that has already been settled by whoever proposes the cheapest one: it keeps the feature and it blocks the demo, so it strictly dominates. That reasoning uses the one property both options have in common. What separates them is the residue — what each one costs to run, and what each one leaves exposed after the demo is forgotten.

Take an engineering assistant that reads private source repositories, runs shell commands in a container, and fetches URLs to read documentation. A proof of concept exfiltrates a repository secret — an API key committed to a .env, the plain-text file of environment variables that sits in the checkout, which the shell tool can cat — through the URL fetcher, by planting instructions in a documentation page the assistant retrieves. Two mitigations are available and both turn that proof of concept red: default-deny egress from the container with an allowlist of documentation hosts, or removing the fetcher entirely and pre-caching a documentation corpus the assistant reads locally.

The four dimensions that actually separate them#

Two mitigations that block the same attack still differ on task success, latency, operational complexity, and residual exposure, and those four are where the decision lives. Task success is whether the ordinary workflow still works, and how well. Latency is what the mitigation adds to a normal request. Operational complexity is the standing work the mitigation creates for as long as it exists — the tickets, the reviews, the on-call surface. Residual exposure is the exfiltration that still works after the patch, the door the mitigation did not close. The proof-of-concept result is a fifth column, and it is the same for every row, which is exactly why it cannot be the basis for the choice.

That last sentence is doing real work and is worth deriving rather than asserting, because "it blocks the demo" is the argument that actually gets made in rooms. A comparison ranks options by their differences: whatever contribution a column makes to one row's score, it makes to every other row's score too when every row holds the same value, so adding it, weighting it heavily, or dropping it entirely leaves the ordering exactly where it was. A column with no variation across the candidates cannot separate them, in the same way that adding the same constant to every score changes no positions. The proof-of-concept column is that column here by construction — it was used to filter the shortlist, so every candidate that reached the table passes it. Which means an argument of the form "this one blocks the demo" is not an argument for this one at all; it is a restatement of why it is on the list.

Read down the residual-exposure column, because it is the one that gets skipped and it is the one that decides. The allowlist keeps the fetcher, which means it keeps an outbound capability and merely narrows where it can point, and a narrowed outbound leg is defined by what it still permits: documentation hosts are not inert. Many serve user-generated content, so the attacker can host the payload on an allowlisted host. Many redirect, so an allowed host answers with a 302 that sends the request onward to a host you never allowed. A query string appended to an allowed URL carries the secret to that host's access logs, and a DNS lookup for an attacker-controlled subdomain carries it out even when no HTTP connection to the outside is permitted. The allowlist has not removed the leg; it has drawn a boundary through it, and the residue is every point on that boundary the attacker can still reach.

The four rarely agree, and when they disagree the residual-exposure column is the one to weight, because it is the only one of the four that changes shape after you ship. Task success, latency and operational complexity are point measurements you can take on the day and re-take whenever you like; residual exposure is a claim about what an attacker can still do, and it moves as hosts are added, as approvers tire and as the system grows consumers nobody has looked at. Choosing on the three you can measure today is how a design ends up optimised for the week of the review.

Removing the fetcher removes the capability, so its residue on this leg is nothing — there is no fetcher to argue the model into using, which is what a real removal means. The cost is in the task-success column: the assistant now reads only what the pre-cache captured, so documentation published after the last crawl is invisible, and a question whose answer lives on a page nobody thought to cache gets a worse answer. The residue that remains is on other legs the shell tool still provides — the container can still write files, still run git push, still open a socket the egress rules permit — so removing the fetcher narrows the exfiltration surface without claiming the agent is now safe, which is the honest scope of any single-leg removal.

Picking between the three on the day of the review reads the first three columns, and there is a relation between two of the four that is worth having in hand before the pick is made. Read the operational-complexity column and two rows say the work is high and permanent: the allowlist, which needs a ticket answered every time a host is missing, and the approval gate, which needs a person present at every outbound action. Now read the residual-exposure column for those same two rows. The allowlist's residue widens as hosts accumulate, and the gate's residue widens as the approver stops reading. The third row, whose ongoing cost is a scheduled crawl, has a residue that stays the size it was on the day it shipped. The two columns agree row for row, and they agree for a reason rather than by coincidence: permanent operational complexity means the control's strength is a function of a continuing stream of human decisions, and a stream of decisions taken by whoever is on hand gets cheaper for the person taking them the longer it runs. So the operational-complexity column is doing a second job nobody asks it for. It estimates what the mitigation costs to run, and it also forecasts which way that row's residue will move once the review is over — worth reading off before the pick, since the two rows that cost the most to keep are the two whose protection is quietly leaving.

The allowlist becomes the surface it was meant to shrink#

The reason the allowlist looks dominant is that on day one it is. The fetcher still works, the demo is red, latency is unchanged, and the only cost is a short config file. The cost is not in the artifact; it is in the artifact's trajectory. Every allowlist accumulates hosts, because every legitimate documentation source someone needs becomes a ticket that says "the assistant can't reach our vendor's docs, please add docs.vendor.example," and that ticket does not get routed to the person who wrote the threat model. It gets answered by whoever is on call, who sees a reasonable request to unblock a documentation host and approves it, because from where they sit it is a documentation host and the assistant is supposed to read documentation. Over a year the allowlist grows from three hosts to forty, and several of the forty serve user-generated content or redirect or log query strings, and the boundary that was tight on day one is porous by the time anyone audits it — loosened one reasonable ticket at a time by people optimising for the assistant working, which is their actual job.

The growth is not bad luck and it is not a failure of discipline, and seeing why makes it predictable rather than surprising. Look at what generates changes to the list. A missing host produces a person who cannot do their work, who files a ticket, which produces an addition. A present host produces nobody: no one is blocked by a documentation domain being reachable, no one notices the cost, and no ticket is ever filed asking for a removal. So the process has a source of additions and no source of subtractions, which makes the set monotone — it can only ever grow. A monotone set under a steady stream of locally reasonable decisions has no equilibrium to settle at; it does not stabilise at the right size, it stabilises at whatever size the year produced. Anything that is going to shrink it has to come from outside that loop, which means a scheduled audit with an owner and a mandate to delete, and an audit is a standing cost that shows up in the same column the allowlist looked cheap in.

So the allowlist's operational complexity is not a one-time setup cost; it is a standing liability that transfers the security decision to the least-invested reviewer on a recurring basis. That is the residue that should worry you more than the removed fetcher's staleness, because staleness is a visible, complained-about, quantifiable degradation — someone notices the answer is out of date and files a bug — while allowlist creep is invisible until an incident, and each individual expansion is locally reasonable. A control that decays through reasonable local decisions is harder to hold than a control whose cost is felt and measured.

Where the approval gate fits, and where it does not#

The third row changes character with the workload, and the clearest way to see that is a different system: an agent that reads private customer records, ingests supplier emails, and can send email. Proposal A is default-deny egress plus an allowlist of internal hosts. Proposal B is a human approval step before any outbound email. Both block the proof of concept, and again the choice is the residue. A removes the capability at the network, so it holds unattended — no human in the loop, no decay from fatigue — and its residue is the gap in coverage: mail that leaves through an internal relay the egress rule permits was never in scope of a network egress control, because the mail left through allowed infrastructure. B keeps the capability and puts a person in front of it, so its residue is the person: an approver who sees the intent of each outbound message, which is genuinely more than a network rule can see, and who over hundreds of routine approvals stops reading and starts clicking, so the control degrades toward a rubber stamp exactly when volume makes it matter.

The two residues are different in kind, which is why the choice is not obvious. A holds without attention but is blind to intent and misses the relay path. B sees intent but requires attention it will not durably get. If outbound volume is low and each message genuinely warrants a look, the approver stays engaged and B's intent-visibility is worth its latency. If volume is high, B decays into clicking and you have paid seconds of latency per message for a control that has stopped controlling, and A's unattended hold is the better residue despite its blind spot. The decision is a reading of the operational reality, not a ranking of the mitigations in the abstract.

Residues that differ in kind cannot be compared on a single scale, so the tie-break is a question about observability rather than about size. A's residue is enumerable: the set of internal relays that can carry mail outward is a list somebody can write down this afternoon, and once written it can be closed, accepted or monitored, because it is a fact about the configuration. B's residue is not in the configuration at all. Reading the approval policy tells you the gate exists and tells you nothing about whether anyone reads the prompts, since the mitigable part — the gate being present, wired and mandatory — is exactly the part that stays true while the control stops working. That gives a usable rule for choosing between residues of different kinds: prefer the one you can enumerate, and if you take the one you cannot, commit in the same decision to the measurement that would reveal it, which for an approval gate is the distribution of time-to-approve rather than the approval rate.

Watching what the pick left exposed#

Having picked, you monitor the thing the pick left exposed, not the thing it fixed. The removed fetcher does not need watching; it is gone. What needs watching is the staleness the removal introduced and the legs it did not touch. Track the age of the cached corpus and the rate of questions the assistant answers "I don't have documentation on that," because those are the signal that the task-success cost is climbing. Watch the shell tool's remaining outbound surface — file writes, git operations, any socket the egress policy still permits — because those are the residual legs and the next proof of concept will aim there. And keep the canary in place, on a schedule, so that if a future change quietly restores an outbound path the endpoint log tells you before an attacker does.

That inversion is the habit worth taking from the comparison. The instinct after shipping a mitigation is to build a dashboard for the attack you blocked, which will report zero forever and teach you nothing, since the route it watches no longer exists.

The general test for whether a metric is worth an alert falls out of that: a reading is informative only to the degree that some plausible state of the world would have produced a different reading. A counter on a removed fetcher has one possible value, because the component that could increment it is gone, so observing it rules nothing out and its information content is zero no matter how prominently it is charted. A counter on the corpus age has many possible values and their differences mean different things about the product. A canary that still runs against a live path can, in principle, fire. The instinct to watch the fixed thing is really an instinct to watch the thing you understand, and the two are not the same — you understand the removed route precisely because it no longer has any behaviour left to surprise you with. The residue is the part of the system that can still surprise you, so it is the part that earns a metric and an alert.

Comparing on the four dimensions tells you which mitigation to ship when a leg is available to break or constrain. It assumes a leg is available — that you can remove the fetcher, deny the egress, or gate the send without destroying the product — and that assumption holds for most systems and fails for an important class of them.

A support agent that must read the customer's account, read the customer's inbound message, and reply to the customer by email has all three legs by design, and none of them is optional: remove the account read and it cannot help, drop the inbound message and there is nothing to answer, block the reply and it cannot respond. There is no leg to remove and no clean pick to make. The next chapter is about that case — what constraining a leg means when removal is off the table because every leg is the product, and what you would have to see in production to believe the constraint is holding.