Enforceable guarantees and intent

Some security properties can be moved out of the model and some cannot, and telling them apart before you promise one is the difference between a commitment you can keep and a sentence that gets quoted back at you. The criterion is narrower than it first looks, and it is worth stating before any example: a guarantee is only as independent as the worst input to its decision procedure. If any input to the check is text an attacker can influence, the check inherits the attacker's reach no matter how it is built.

That criterion is worth deriving rather than accepting, because in this form it sounds like a slogan and it is a consequence of two ordinary facts. A guarantee is a claim that some outcome cannot occur. Whatever component makes that claim true does so by running a decision procedure whose output determines whether the outcome happens. A decision procedure is a function of its inputs, which means that for any input, an adversary who can set that input can explore what the function does across the values it accepts and pick one that yields the answer they want — unless the function's answer happens not to vary with that input, in which case it was not really an input. So the reachability of the outcome is bounded below by the reachability of the most attacker-controllable input. Adding more inputs the attacker cannot touch does not repair this, because the attacker is not required to defeat all of them, only to move the one that is theirs. Guarantees do not average over their inputs. They are set by the worst one, and a check with four clean inputs and one dirty one is exactly as independent as the dirty one.

The product this chapter works through is an inbox assistant. It reads a user's mailbox and can archive, label and reply, it has been in production for four months, and the requirement on the table is that an email from a stranger must not be able to cause the assistant to send a reply the user did not intend. That request is not unreasonable — it is close to what a customer would assume is already true — and there is no component in the system that would enforce it.

Why one guarantee moves outside the model and the other does not#

The assistant already carries a guarantee of the enforceable kind. It cannot read or send from a mailbox other than the one belonging to the authenticated user, and the reason that holds has nothing to do with the model's cooperation. The mail service resolves a session to a user ID, looks up mailbox ownership in a table, and returns 403 on a mismatch, so a payload instructing the assistant to forward the CFO's inbox produces a tool call that fails at the API. Ownership is a fact stored somewhere the attacker cannot write, it is decidable in constant time by a process that never sees the email, and its truth does not depend on anything in the request body.

"The user did not intend this" has none of those properties. There is no table of intentions. The only evidence of what the user wanted is the text of their request, and everything the assistant did afterwards was influenced by a document the attacker partly wrote, so any test for intent has to consult material that is inside the attacker's reach.

The second row is worth pausing on, because it is the one that looks enforceable and is only mostly so. The participant list has to come from the mail store's own record of who was on the thread, since the Reply-To, Cc and From headers of the inbound message are fields the sender wrote, and an implementation that builds the allowlist from headers has handed the attacker the allowlist. Even done correctly the gap remains that a stranger who emails the user is a participant, so the guarantee permits a reply to the attacker themselves — which is exactly the channel an exfiltration payload wants, since a reply containing a summary of the thread goes to the person who asked for it by being on it. The row still earns its place because it prevents the reply reaching a fourth party, and describing it to legal requires naming that residual case rather than rounding it off. A guarantee with a stated exception is usable; one whose exception surfaces later is worse than none.

The table has a property worth noticing, because it means you never have to judge the last column directly. Read the third column and the fourth and they agree everywhere: where the attacker cannot influence the deciding input, the guarantee is enforceable, and where they can, it is not. The one row that is neither a clean yes nor a clean no in the third column is neither in the fourth, and its qualifier — "only by being on the thread" — is precisely the qualifier that reappears as "with a narrow gap". So the fourth column is not an independent judgement about each property. It is the third column restated. That is what turns the criterion from a slogan into a procedure: to decide whether something can be guaranteed, you do not reason about the property at all, you enumerate the inputs to the check that would decide it and ask which of them the attacker can write. The properties sort themselves.

The last two rows are the ones legal is asking about, and the fourth row is where the tempting answer lives.

The classifier reproduces the problem one layer down#

The natural proposal is a checker: before sending, a second model receives the user's original request and the drafted reply, and answers whether the draft is consistent with the request. Block the sends that fail. It is a day of work, it produces a metric, and on a first evaluation it looks like it works — on a set of 200 drafts, 160 legitimate and 40 injected, one team measured 94 percent precision and 88 percent recall, which is to say that of the drafts the checker blocked, 94 percent really were injected, and of the injected drafts in the set, it blocked 88 percent.

Those two figures are not the same kind of thing, and the difference decides how much of the good news survives leaving the evaluation set. Recall is computed inside the injected group alone — how many of the hostile drafts were caught — so it does not depend on how many legitimate drafts were in the set beside them. Precision is computed across the blocked group, which mixes both, so it depends directly on the ratio of injected drafts to legitimate ones in whatever population it was measured on. That population was a fifth injected. A real mailbox is nothing like a fifth injected; hostile drafts are rare and legitimate ones are the overwhelming majority, so the same classifier with the same behaviour will block far more legitimate drafts for every injected one it catches once it is deployed. The precision figure is therefore an artefact of the evaluation mix, and the recall figure is a statement about the classifier against those particular forty payloads. Neither is a statement about the checker's behaviour in the product, and the second one is the only one that even purports to be.

The diagram shows what the evaluation does not. The attacker's text reaches the checker as well as the assistant, because the checker cannot judge consistency without seeing the thread, and a checker that sees only the draft cannot tell an appropriate reply from an inappropriate one. Both boxes shaded red take input the attacker controls, so the same payload gets two attempts, and the second attempt is easier because the checker's task is narrower and its decision is a single token. A paragraph in the original email establishing that this account's replies are routed through an authorised-forwarding process, and that including the forwarding address in the reply body is the expected format, reads as consistent to a consistency checker, since the assistant's draft now matches the context the checker is reading.

When that team re-ran the evaluation with 40 payloads written after the checker existed and aimed at it, recall fell from 88 percent to 31 percent. The classifier was not badly built and a better classifier does not fix this, because the failure is the one from chapter 1 appearing in a component whose only defence is the same prior. A check whose input the attacker controls is not an independent authoriser, whatever its measured accuracy.

Hold the two recall figures against each other rather than reading them as a before and an after. Against payloads written before the checker existed, it caught close to nine in ten. Against payloads written afterwards and aimed at it, it caught under a third, which is barely more than a third of its former rate. Only one thing differs between the two sets, and it is not difficulty, cleverness or length. It is whether the payload's author had seen the checker. That is the quantity the first evaluation could not measure and the one every deployed system is exposed to, and it is why a first evaluation of any detector is, structurally, a measurement taken before the thing it measures exists.

The false positives are the second half of the cost and they land on ordinary users. Six percent of legitimate replies were blocked, and the blocked ones skewed towards the long, unusual, multi-topic threads where the assistant is most useful, because those are the drafts a consistency judgement finds hardest. A control with 6 percent false positives on the flows people care about most gets a threshold adjustment within a fortnight, then a second one, and by the time it is comfortable enough to leave on it is catching almost nothing.

What you can actually promise, and how to phrase it#

The honest response separates two kinds of statement and never lets one wear the other's clothes. A guarantee names a component and an operation that component refuses; a mitigation names a measurement, a method and a date. Writing both, clearly labelled, is a stronger position than one blurred sentence, and it is the version that survives the sentence being quoted back to you in a deposition.

The guarantees available on this product today, with no new modelling work, are these. The assistant cannot access or send from any mailbox other than the authenticated user's, enforced by the mail service's ownership check. It cannot send to a recipient who is not a participant on the thread being replied to, enforced by the send guard comparing recipients against the participant list the mail store returns rather than any address appearing in the message body. It cannot include content retrieved from a different thread, enforced by per-thread context scoping in the retriever. Every send is written to an immutable audit record containing the draft, the source thread and the message that triggered it, and every send is delayed 30 seconds and revocable by the user within that window.

What cannot be guaranteed is stated as plainly. Nobody in the industry can currently guarantee that the content of a reply is free of influence from a message the user did not write, and the phrasing legal needs is that the assistant may be induced by a malicious inbound message to draft an unintended reply the user would not have written, that this is a known limitation of language models rather than a defect in this implementation, and that the controls above bound where such a reply can go and make it reviewable and revocable rather than preventing it. Adding a measured figure is fine as long as it is labelled as a measurement — injection success on the current 60-payload suite, measured on a date, against that build.

The product change that makes a real guarantee possible#

Intent cannot be externalised, and an act can. The change that converts the fourth row of the table into the third is to stop trying to test what the user meant and instead create an observable artefact of what the user approved: a send requires an authenticated click on a rendered draft, and the confirmation is bound to that exact draft rather than to the conversation.

def confirm_send(draft_id: str, ctx: RequestContext) -> ConfirmToken:
    d = drafts.get(draft_id)
    return tokens.issue(
        user=ctx.user_id,
        payload_hash=sha256(canonical(d.to, d.cc, d.subject, d.body)),
        ttl=timedelta(minutes=10),
    )

def send(draft: Draft, token: ConfirmToken | None, ctx: RequestContext) -> Result:
    if token is None or token.expired or token.user != ctx.user_id:
        return Result.refused("no_confirmation")
    if token.payload_hash != sha256(canonical(draft.to, draft.cc,
                                              draft.subject, draft.body)):
        return Result.refused("draft_modified_after_confirmation")
    if set(draft.to) - thread_participants(draft.thread_id):
        return Result.refused("recipient_off_thread")
    return mail.send(draft)

The hash comparison is the load-bearing line. Without it the model could get a confirmation for a benign draft and then send a different one, which is the same substitution attack in a new place, and with it the guarantee becomes checkable: what the user saw is byte-identical to what left the building. Nothing in that function reads the email, judges consistency, or consults a model, so there is no input for a payload to reach.

Run the criterion over that function and it is clear why this one works where the classifier did not, and the difference is not effort or cleverness. The inputs to the send decision are a token issued in an earlier authenticated request, a hash computed over fields the user was shown, a clock, and a participant list the mail store returns. An attacker who writes the original email influences none of the four. They can influence what the draft says, and a draft that says something hostile still has to be rendered, shown and clicked before the hash matches — so the attacker's reach now stops at what a person looked at and accepted. That is the whole trick, and it is worth stating in the general form. Intent is a fact about a mind and there is nowhere to look it up. An approval is an event, events leave records, and a record is a row a service can check without asking anybody what they meant. Where a property you have been asked to guarantee has no attacker-independent test, look for the act that would have accompanied it and guarantee the act instead.

The cost is a product regression and it should be stated as one. The assistant stops being able to clear a mailbox unattended, which was the demonstration that sold it, and a measured version of this change put median time-to-send at 3 minutes against 8 seconds before, with roughly 12 percent of drafts never confirmed. There is a defensible middle setting where replies to participants already in the user's contacts send automatically and replies to anyone else require the click, which recovers most of the unattended value and leaves the guarantee intact for exactly the case legal asked about, since a stranger is by definition not in the contact list. That version is worth building, and the boundary it draws — known correspondent, unknown correspondent — is a fact in a table rather than a judgement about text, which is why it can be enforced at all.

Every requirement worth arguing about reduces to the same audit, and it is short enough to run on your own system this week. List what the agent can do; for each, write the guarantee somebody would want; name the component that refuses when the guarantee is violated. The rows with a component are your guarantees and you can say so in writing.

The rows whose only candidate enforcer is a model reading attacker-influenced text are not guarantees today, and they are the roadmap: either a capability to remove, an operation to move behind a confirmation bound to its payload, or a fact you can put in a table so a service can check it without asking a model anything. That is the whole of what this topic has been arguing, arrived at from the other end — not that a prompt cannot be made stronger, but that strength in a prompt is a rate, and a rate is not a row you can put a component's name against.