Containing a persuaded model

Take the system section of a billing assistant, print it, and go through it with a highlighter asking one question per line: if a customer read this aloud in a webinar, would anything bad happen? On an eleven-line prompt, seven lines survived without a mark. They said things like answer only from the retrieved article, use the customer's local currency, and if the question is about a chargeback, say that fraud will be in touch within one business day. Publishing those would be mildly embarrassing and operationally irrelevant, since knowing an assistant's tone rules does not let anyone do anything.

Four lines got marked, and they were the whole exposure. The refund thresholds at $500 and $2,000, the three account flags that route to fraud, the Q3 sunset that Growth customers must not be told about, and a sentence describing how the assistant should recognise a customer who is probably about to churn. Those four lines are the reason a pentester printing the prompt counts as a security finding, and none of them needed to be in a prompt.

The secret belongs behind a call, and the sunset belongs nowhere#

Two of the four are the same fix. A threshold is a policy decision, and a policy decision is a function call:

# before
Refunds above $500 require manager approval; above $2,000 require finance.
Accounts flagged chargeback_risk, dispute_open or manual_review route to fraud.

# after
To find out whether a refund is permitted, call refund_policy with the amount.
To find out where a ticket should go, call routing_decision.

Extraction of the second version yields the names of two tools and the fact that a policy exists. The thresholds now live in the billing service, which already had them, and the model learns a decision for one specific amount and one specific customer, at the moment it needs it, through a call your code makes and your code authorises. An attacker who extracts the whole prompt has learned that the company has refund rules. This is not a secret; the terms of service say so.

The third line is a different mistake and the more instructive one. Never tell a Growth-plan customer that Growth is being sunset in Q3 requires telling the model about the sunset in order to forbid it from mentioning the sunset, which means the sensitive fact is now in the context of every Growth conversation, protected by an instruction that the previous chapter already priced as a probability rather than a boundary. The correct handling is not a stronger instruction. It is to leave the fact out of the prompt entirely and to scope retrieval so that the roadmap document is not in the index the assistant queries. A model that has never been told about the sunset cannot be persuaded to reveal it by any phrasing, in any language, at any temperature. Withholding beats forbidding every time, and the reason teams reach for forbidding is that the fact usually got into the prompt by accident — someone pasted the internal FAQ — and deleting it feels like losing a capability rather than removing a liability.

The asymmetry underneath that is worth stating as a mechanism rather than as advice. An instruction not to mention something is evaluated by the model while it generates, against everything else in its context, and it wins or loses on a weighting — which is what a rate is. A fact that is not in the context is weighted against nothing, because the model's output is conditioned on the tokens it was given and no token sequence recovers a fact absent from all of them. That is an invariant, and it is enforced by the retrieval scope and the prompt assembly rather than by the model's disposition. The condition it rests on is worth making somebody demonstrate before it is claimed: the fact has to be absent from every channel into the context, so a roadmap document still in the index, an earlier turn in which the assistant mentioned it, or a summary generated from either one puts it back, and the withholding is only as complete as the slot audit of the previous chapter was.

The fourth line, the churn-detection heuristic, comes out and goes into a feature flag the model never sees.

Two cheap checks on the way out#

Once the prompt has been drained of anything worth extracting, the remaining question is whether it stays drained, because prompts grow. A canary handles that. Put an opaque string in the system section that has no meaning, appears nowhere else in the codebase, and is never mentioned to the model, then reject any response containing it:

CANARY = "cnr_8f2a4e1b90d7"
CUSTOMER_ID = re.compile(r"\bcus_[a-z0-9]{10}\b")

def check_response(text: str, session: Session) -> Verdict:
    if CANARY in text:
        return Verdict.block("system_prompt_leak")
    foreign = set(CUSTOMER_ID.findall(text)) - {session.customer_id}
    if foreign:
        return Verdict.block("cross_customer_identifier")
    return Verdict.allow()

Against a pentester's hundred extraction variants, the canary check blocked 68 and alerted on all 68, at a cost of eight tokens per request and roughly 40 microseconds of substring search. The 32 it missed were paraphrases, where the model restated the instructions in its own words and never emitted the literal string. A canary detects copying, not disclosure, and reporting it as an extraction defence overstates it by a factor of three.

Being exact about the mechanism is what decides which claim the check supports. The canary is a substring match against a string the model was never told about and that appears nowhere else in the system, so a hit is proof that part of the system section was reproduced character for character. Everything else follows from that. A model that restates the escalation rules in its own words has disclosed everything the rules contain and emitted none of the characters being matched, and about a third of the variant set did precisely that. Nothing in that third is exotic — asking for a summary rather than a printout is enough — which is why the honest name for the control is verbatim-leak detection rather than extraction defence.

The second check in that function is the one that matters more and gets built less often. It is deterministic, it is specific to your identifier formats, and it catches the failure that is genuinely a breach rather than an embarrassment: a response mentioning a customer identifier that is not the identifier of the session that asked. The same shape extends to account numbers, invoice ids, and email addresses, and it costs a compiled regex per pattern. A model-based policy check on top of this — a small model asked whether the response discloses another customer's data — adds around 180 ms at p50, the median request, and $0.0004 per request, and is worth adding after the deterministic checks rather than instead of them, because it catches paraphrase and it is itself a model reading attacker-influenced text.

The check runs on text the customer is already reading#

Both checks in that function take a finished string, and most assistants stream. With the first token reaching the browser at 340 ms and the full answer landing around 2.1 seconds, a system section quoted from the middle of a long answer puts the canary on the wire at roughly 1.4 seconds, by which point about 900 tokens have already been painted on the customer's screen. A check that runs after the last token detects that leak and cannot prevent it, which makes it an alerting control wearing the clothes of a blocking one, and the distinction shows up in a report as the difference between we were notified and it did not go out.

The cheap version of the fix is a delay line in the gateway: hold a rolling window of the last 200 tokens and release each token only once the window containing it has cleared both checks. The customer sees the first token at about 600 ms rather than 340 ms and the stream then runs at its normal rate, and a match inside the window is cut before it leaves the process and replaced by the same refusal a blocked non-streaming response would get. Buffering the entire response instead would push perceived first-token latency to the full 2.1 seconds, which one product team measured as a 9% drop in completed sessions on the support surface and declined. The window still leaves 260 ms of exposure at the tail of the stream, which is a real gap and a considerably smaller one than the whole answer.

Those figures are one quantity seen twice, and deriving it is what makes the trade legible rather than arbitrary. The stream paints its first token at 340 ms and finishes around 2.1 seconds, and roughly nine hundred tokens are on screen by the time the canary reaches the wire at about 1.4 seconds — so those tokens arrived over about a second, which is a rate close to one token per millisecond. Two hundred tokens at that rate is about a quarter of a second of text. That is where the extra wait before the first token comes from, and it is the same quarter-second that remains unprotected at the end, because the buffer's depth measured in time is the delay at the front and the gap at the back. Widening the window buys more of both. Narrowing it risks splitting the string the check matches on across two windows, which is a way of having no check at all.

Neither check survives contact with production unless somebody budgets for tuning it. The cross-customer identifier rule fired four times a day in its first week and three of those four were legitimate, because the agent-facing summary surface quotes the merchant identifier of the counterparty on a disputed charge, which is the entire purpose of that surface and looks exactly like a leak to a regex comparing against session.customer_id. Scoping the rule to customer-facing surfaces and adding the disputed counterparty to the allowed set for the duration of a ticket took most of a day and brought it down to about one alert a week, all of them worth reading. A blocking check with a 75% false-positive rate is switched off within a fortnight by whoever is on call at the time, so the tuning is part of shipping the control rather than a follow-up ticket that competes with everything else.

A persuaded model should be able to request, not to decide#

The refund incident from chapter 3, where a poisoned community article published to Arcadia Pay's help centre reached an unrelated customer's session, moved $4,180 because issue_refund did what it was told. The model produced a tool call with a customer id and an amount, and the handler executed both. Any design where the model supplies the identity of the subject of an action has made the model an authorisation component, which is the classic confused deputy: the application's privileges, pointed by someone else's instructions.

def issue_refund(session: Session, amount_cents: int) -> RefundResult:
    policy = billing.refund_policy(
        customer_id=session.customer_id,        # never a model argument
        amount_cents=amount_cents,
    )
    if not policy.permitted:
        return RefundResult(status="denied", reason=policy.reason)
    if policy.requires_approval:
        ticket = approvals.create(session.customer_id, amount_cents)
        return RefundResult(status="queued", ticket=ticket)
    return billing.refund(session.customer_id, amount_cents,
                          idempotency_key=session.request_id)

The customer id comes from the authenticated session and is not in the tool's schema, so the model has no way to name a different customer even if it wants to. The amount is checked against the same policy service the human agent console calls. Above the approval threshold, the call becomes a queued request that a person sees, which turns a successful injection from a payout into a row in an approval queue with an odd-looking justification attached. The blast radius of a fully persuaded model is now bounded by what a fully persuaded model is permitted to do, and that boundary is enforced in Python by code that never reads the prompt.

Naming the pattern locates the defect precisely, which is why it is worth naming. A confused deputy is a component holding a privilege that somebody else gets to aim: the application is the one entitled to call the billing service, the model decides which customer and how much, and so the application's authority is exercised on a subject chosen by whoever wrote the text the model happened to read. The correction is not to make the model choose better. It is to delete the parameter, and deleting it is total in a way no instruction is — customer_id is absent from the tool's schema, so no argument the model can emit names another customer, and a model entirely convinced it should refund somebody else has no field in which to say so. The persuasion still succeeds. It arrives at a function signature that does not accept it.

Read-only tools deserve the same treatment with less ceremony. A search tool scoped to the session's own tickets cannot be talked into returning someone else's, because the scope is a WHERE clause and not a request.

Two weeks, sequenced by what each day changes#

Laid out as a fortnight for one engineer, with the column that matters being not what each item is but what it changes about the finding:

Twelve and a half days, with the cheapest and least load-bearing item last, which is the reverse of the order a ticket-closing instinct suggests.

The third column is the sort key, and it is worth saying why rather than leaving it as a preference. Sorting by cost puts the half-day item first and produces a week at the end of which the finding is exactly where it started. Sorting by what each item changes about the finding puts first the two rows that make a successful extraction return nothing worth having, then the row that turns a successful persuasion into a queued request rather than a payment, and only then the row that makes the attempt marginally less likely to work. Each of the first three changes the finding's severity on the day it lands, which is a statement a report can carry with no expiry on it. The last changes a probability, which is a statement that needs a measurement date and goes stale when somebody else ships a model. The days total the same either way; what differs is what the report says after the first three of them.

A downgrade, not a resolution#

The model can still be persuaded, on every request, at whatever rate the current phrasing achieves. Nothing above changes that, and a report claiming injection is resolved would be false. What changes is the outcome of a successful persuasion: instead of policy disclosure and a refund, you get an assistant saying something unhelpful and, if it is a serious attempt, an alert. That is a downgrade from an incident to a nuisance, and downgrades are what security programmes actually deliver.

There is a configuration where this is not enough, and it is worth recognising before committing to it. If a product genuinely requires untrusted content in the context, high-privilege actions available to the model, and free-form output going somewhere consequential, no arrangement of these controls is sound, because the first component defeats the reasoning about the other two. The honest move is to remove one of the three — narrow the tools, put a human on the consequential path, or keep untrusted documents out of the privileged call and summarise them in a separate one whose output is data.

Equally, this is more than most internal tools need. An assistant that reads a public documentation set, holds nothing sensitive, and can call nothing, has a prompt whose extraction costs nobody anything, and building an approval queue for it is theatre.

What the three controls share is that none of them asks the model to behave. The prompt holds no secret, so extraction returns nothing worth having; the response is matched against strings and identifier patterns that do not depend on intent; and the tool arguments that decide who an action applies to come from the authenticated session rather than from generated text. Each is enforced by code that never reads the prompt, which is what lets a report state an invariant rather than a rate.

Every control described here assumes you can answer a question that the refund incident could not answer for nine days: what was actually sent to the model on the request that went wrong. The logs held the template name, the model id, and the response, and reconstructing the prompt took an engineer two days of guessing.