Writing a separation requirement

Working groups reach for a one-sentence requirement because a one-sentence requirement is what six companies can sign. The model must not execute instructions that originate in untrusted data is the shape they converge on, and it is unusable as written.

Three things are missing and each of them is the kind of gap that only shows up when somebody tries to test compliance. It names no component that does the refusing, so there is nothing to point at when an operation goes through. It has no verifier: given a system and a payment that happened, no procedure decides whether the requirement was met, because "executed an instruction" is not a property of an observable event. And it does not say what happens when provenance is unknown, which is the common case, so an implementation that treats unlabelled input as trusted is compliant with the text. A requirement that fails open when it does not know is not a security requirement, it is a preference.

Write it so it can be failed#

The version below is longer, which is the price of being testable. It is written for a payments context where an agent can initiate transfers, and the clauses are ordered so that each one closes a hole in the one before it.

R1  Every value admitted to the agent's context is assigned a provenance
    label by the component that ingests it — retriever, email fetcher, web
    tool, user session. The label is metadata in the orchestrator's own
    memory. It is never a string inside the prompt, and the model never
    assigns or modifies it.

R2  Labels propagate through orchestration. Any tool argument that is
    copied from, derived from, or selected on the basis of a labelled
    value inherits that label. A value produced by the model after reading
    a labelled value is labelled.

R3  Before executing any operation in the MONEY_MOVING set, the tool
    router evaluates the labels on the call's arguments and on the
    decision path that produced it. If any is UNTRUSTED, the router
    refuses to execute and returns an error to the agent. The refusal is
    not overridable by the model's output.

R4  A refusal may be lifted only by a confirmation record created in a
    separate authenticated request by a principal with authority over the
    source account, referencing this operation's exact arguments.

R5  Unknown provenance is UNTRUSTED. If the label store, the policy
    service, or the confirmation service is unavailable, the router
    refuses. The property fails closed.

Enforcing component: the tool router, a non-model process owned by us,
with the policy service and the confirmation store it depends on.

R1 is where most drafts go wrong, because the obvious implementation is to write the label into the prompt as <document trust="untrusted">, and chapter 1 established what happens to markers that live inside the string. The label has to be a field on an object in the orchestrator — your own process, the code that assembles each request, calls the model and dispatches the calls it asks for — keyed to the span, so that the router can read it after the model has finished and without asking the model anything.

R2 is the clause that makes the requirement hard to implement and is also the one without which it is worthless, since an attacker who cannot get an untrusted value into an argument directly can usually get the model to restate it. If a document says the correct payee for this invoice is IBAN GB29… — an IBAN being the international bank account number a transfer is addressed to — and the model writes that IBAN into a transfer call, the value has passed through the model but its origin is unchanged. Whether your orchestrator can track that is the practical test of whether you can implement R3 at all.

Laid end to end the five clauses are not a list of good ideas, and it is worth seeing that each one exists because the clause before it has a hole. R1 alone gives you labels that live in the orchestrator, and an attacker defeats it by getting the model to restate an untrusted value in its own words, since the restatement is a new value with no label. R2 closes that by making anything derived from a labelled value inherit the label, and now you have accurate labels that nothing consults. R3 closes that by putting a component in front of the money-moving operations that reads the labels and refuses when the call originated in untrusted data. That would refuse every legitimate invoice payment, so R4 adds the one path back — a confirmation created out of band by a principal with authority — and now the property holds whenever the label store and the confirmation service are answering. R5 closes the last hole by fixing the answer when they are not. Five clauses, four holes, and no failure mode left over that the previous four permit: that is what makes it a requirement rather than a statement of intent, and it is also the reason it cannot be shortened without knowing which hole you are reopening.

R5 is the clause that vendors and internal teams both try to negotiate away, on the grounds that refusing when the policy service is down converts a security control into an availability risk. That is exactly what it does, and the trade is deliberate: an agent that stops moving money during a policy-service outage has an incident, and an agent that keeps moving money during one has no boundary, because "make the policy service unreachable" becomes a step in the attack.

The 99.7 percent answer#

The vendor's response to a requirement like this is now standard. Their model accepts a dedicated data field in the request, the model is trained to treat content in that field as inert, and they publish a compliance figure — 99.7 percent on their internal evaluation. The question is whether that satisfies R1 through R5, and the answer is that it does not satisfy any of them, for a reason that is structural rather than a matter of the number being too low.

A trained tendency is a prior over inputs, and R3 requires a decision procedure with a defined output on every input including ones nobody has seen. No amount of training converts the first into the second, which is the point chapter 1 made about roles and which applies unchanged to a dedicated field: a field the model is trained to ignore is a role marker with better marketing. There is no component in the vendor's architecture that refuses anything; there is a model that usually declines, and "usually declines" has no failure-closed behaviour because there is no failure to detect.

The arithmetic is worth doing in front of the working group, because it converts a structural objection into one the finance representative will carry for you. A mid-sized payments agent handles on the order of 12,000 money-moving operations a month. At 99.7 percent compliance, 0.3 percent is 36 operations a month reaching a payment rail against the field's semantics, and at a mean transfer of €4,200 that is about €151,000 a month of unauthorised movement, before considering that an attacker who finds one working payload does not distribute their attempts uniformly across your ordinary traffic. Compliance percentages are the shape of a quality metric, and quality metrics are averages over a distribution that an adversary is choosing.

Do that arithmetic slowly in the room, because it is three multiplications and each one changes what the number is about. Start with the complement: a figure quoted as compliance leaves a residue, and the residue here is three tenths of a percent. Multiply the residue by the monthly volume — three thousandths of twelve thousand — and you have thirty-six operations, which converts a proportion into a count of events. Multiply that count by the mean transfer and you have a figure in euros, which converts a count of events into an amount of money, and money is the unit the finance representative is authorised to have an opinion about. Nothing in those three steps is disputable, which is what makes the sequence useful: the objection to the vendor's field is structural and hard to argue in a working group, and the same objection expressed in euros a month needs no shared theory of language models at all.

The figure's provenance matters as much as its value. It was measured on the vendor's payload set, which by construction predates the attack anyone will write against your specific product, and the previous chapter covered what a static corpus measures compared with an adversary holding your configuration. Ask the vendor what the number is under an adaptive attacker with the field schema and eight hours; the answer, when anyone has run it, is not 99.7 percent.

What would satisfy the requirement from a vendor is a mechanism rather than a rate: a signed provenance envelope the router can verify independently of the model's output, or an API contract in which arguments to a tool call are returned with the input spans they were derived from, so R2 becomes checkable without the orchestrator reimplementing dataflow analysis. Those are plausible products and none of them exists as a shipped guarantee today. Until one does, a vendor claim can lower the frequency term and the router is still the component that has to refuse.

What you can build today#

The patterns that achieve the property without any vendor cooperation are catalogued in Design Patterns for Securing LLM Agents against Prompt Injections, and the working system in Securing AI Agents with Information-Flow Control shows the label-propagation machinery of R2 built end to end. What follows is the shape they share and the price each charges.

Read down the cost column and the five patterns turn out to be one idea priced five ways. The property every row buys is the same: untrusted text does not determine a privileged argument. What differs is which freedom is surrendered to buy it. The action-selector gives up data-derived arguments, the plan-then-execute gives up adaptive control flow, the dual model gives up having one call do everything, the code-then-execute gives up not owning an interpreter, and confirmation gives up unattended operation. There is no row that costs nothing, and the absence of such a row is the finding rather than an omission in the table. If a pattern claiming the property at no cost is proposed to you, the question to ask is which of those five freedoms it is quietly still using, because the property is exactly the statement that one of them has been given up.

In practice the dispatcher that enforces R3 is small, and it is small in the same way the confirmation check two chapters ago was small.

MONEY_MOVING = {"transfer", "pay_invoice", "update_payee"}

def dispatch(call: ToolCall, labels: LabelStore, ctx: RequestContext) -> ToolResult:
    if call.name in MONEY_MOVING:
        taint = labels.taint_of(call.arguments) | labels.taint_of(call.decision_path)
        if Label.UNTRUSTED in taint or labels.unavailable:      # R3, R5
            record = ctx.confirmations.matching(call.name, call.arguments)
            if record is None or record.actor_authority < Authority.ACCOUNT_OWNER:
                return ToolResult.refused(
                    "untrusted_provenance",
                    origins=labels.explain(call.arguments),      # for the audit log
                )
    return TOOLS[call.name](**call.arguments)

The expensive part is labels.taint_of, which is where R2 lives; taint is the long-standing name in security engineering for a mark attached to an untrusted value that every value derived from it inherits. It is worth being concrete about how it is implemented, because the honest version is coarser than the requirement makes it sound. Attributing a specific token in a tool argument to a specific input span is not something a hosted chat API exposes, so nobody does true span-level tracking against a black-box model. The implementable approximation is turn-level: if any untrusted span was in the context window when the model produced this call, every argument of the call is untrusted.

That approximation is a deliberate choice of which way to be wrong, and the reasoning behind it is short enough to hold. The true relation you want is whether this particular argument came from that particular span, and it is not observable through a hosted API. Faced with an unobservable fact, an implementation can guess in one of two directions. Guess that arguments are clean unless proven dirty, and the check passes whenever your evidence is missing, which is most of the time, so the property fails open and R5 is violated in spirit on every call. Guess that arguments are dirty whenever anything dirty was present, and the check refuses in cases it did not have to, which costs product and never costs safety. Only the second is compatible with a requirement that has to hold under uncertainty. The cost of that choice is real and it shows up as refusals a user experiences as bugs, which is why the fix is to remove the uncertainty by segmenting the call rather than to loosen the guess.

That over-approximates badly — a transfer whose payee came from the user's own typed instruction is refused because an unrelated document happened to be in the same context — and the fix is architectural rather than analytical. You segment the work so that the model call which produces a money-moving argument has no untrusted spans in its context at all, which is the dual-model pattern in the table, with the quarantined call returning {"payee_iban": str, "amount_cents": int} and its free text discarded. The typed values are still labelled untrusted and still require confirmation; what the segmentation buys is that the decision to call transfer was made in a context the attacker never entered.

The flows the policy refuses#

The honest thing to tell a working group is that the cost of this design is not the round trip or the code. It is that the flows customers want most are precisely the flows the policy refuses. "Read this invoice and pay it" is a money-moving call whose payee and amount are derived from an untrusted PDF, so R3 refuses it every time, and the product becomes "read this invoice and show me a payment I confirm". The confirmation screen has to render the derived values prominently, since a user who confirms without reading the IBAN has restored the vulnerability with extra steps, and measuring whether they read it is a design problem nobody has fully solved.

Two further things are worth saying before somebody assumes otherwise. The requirement does not make the agent's output trustworthy: a labelled value the user confirms is still whatever the attacker wrote, so the confirmation screen is carrying real security weight and cannot be a spinner with a Continue button. And it is not a sandbox. Restricting what a process can touch and refusing an operation because of where its arguments came from are different controls, and a container that limits blast radius does nothing about a transfer the agent was entitled to make with a payee the attacker chose.

Set against that, the property you get is the one no rate provides: for any payload, written by anybody, against any model version you might swap in next quarter, an unconfirmed transfer with a payee taken from a document does not execute. The guarantee survives a model upgrade, which is worth restating to a working group that has spent the meeting discussing model behaviour, because every behavioural mitigation on the table has to be re-measured after each upgrade and this one does not.

The clean case here is a transfer, where ownership of the source account and the identity of the confirming principal are both facts a service can check without consulting a model. Not every guarantee people ask for has that shape. When legal asks for an assurance about what the user meant, there is no service that can be queried, and what you can honestly promise when the property in question has no attacker-independent test is the next chapter.