Prompt extraction and authorization boundaries

Two things get written into system prompts on the assumption that the model will hold them. The first is material that must not come out: internal pricing, policy the competition should not read, occasionally a credential. The second is a constraint on what the assistant may do, phrased as an instruction and trusted as though it were an access-control rule. Both assumptions are testable, both fail, and they fail differently enough that the repairs are different.

This chapter measures the first with an extraction suite and fixes the second by moving it out of the prompt entirely, into the handler behind the tool. The distinction it turns on is between a control that holds because a text predictor usually declines and a control that holds because your database was consulted; only the second is a boundary, and knowing which of the two you have is the difference between a threat model and a hope. What remains in the prompt afterwards is not nothing, and the last part of the chapter is about that residue, because reading this as "take refunds out of the system prompt" gets the conclusion backwards.

What an extraction test establishes#

Run a few hundred attempts at getting a prompt's contents out — direct requests, reformulation as translation or summary, role-play framings, requests in other languages, and content injected on each untrusted path the system has — and some proportion succeeds. On one nine-hundred-word prompt the number was eleven of two hundred, about five and a half percent, and the number is the least interesting part of the result. What matters is that the successful attempts do not share a pattern you could write a rule against, and that closing each one individually produces a prompt that fails differently on the next round.

The rate is still worth converting, because it decides how much luck an attacker needs. Eleven divided by two hundred is a little over five in every hundred attempts, which is the same figure written another way: roughly one attempt in twenty. Somebody sitting down for an afternoon makes many more than twenty attempts, and each new framing is a fresh draw rather than a retry of a blocked one, because nothing in the model's own operation records that the previous framing failed. Something outside it can count attempts and cut them off, and that is a separate control with its own evasions. Absent one, a rate of this size is not a wall; it is a delay, priced in minutes.

That gives the precise version of the claim: content in the context window is data the model is conditioned on, and the model's decision not to reproduce it is a behaviour, not a permission check. A behaviour has a distribution. It holds on most inputs and fails on some, the failing set is not enumerable in advance, and an attacker gets to sample from the space of inputs at their own leisure while you get to write a fixed number of sentences beforehand. Treating the last line of a prompt as a control means the security property of your system is "a text predictor usually declines", which is a true statement about the model and an unusable one for anything you would put in a threat model.

Why the content is reachable at all is the step underneath that, and it is worth stating as a mechanism, because "the model might reveal it" sounds like a lapse when it is the ordinary operation of the thing. The request reaches the provider as one sequence: the operator's block, the tool definitions, every turn so far, in order. Producing a reply is a loop that scores every candidate for the next fragment of text against that whole sequence, picks one, appends it, and runs again. The secret is inside the sequence being conditioned on, so it is present at every one of those scoring steps, and the refusal that usually follows a request for it is one high-scoring continuation among others rather than a gate that runs before them. No step in the loop asks whether a fragment may be disclosed, in the same way that no step in it asks whether a fragment is true.

The belief that makes this surprising is that the prompt is hidden. It is hidden in the product: the user is looking at a chat box, the operator's block is not rendered anywhere, and the response body does not echo it back. Hidden in the interface is a fact about what the browser draws rather than about what the process reads. The model reads the whole sequence and then writes the text the user is shown, which puts the secret and the channel out of the system on the same side of the boundary, and a secret is only secret from a component that never sees it.

It is worth building the test as an artifact rather than an afternoon, because an afternoon's result expires with the next model version or prompt edit. The shape that survives is an eval suite — a fixed set of inputs replayed against the running system on a schedule and scored automatically, so that behaviour is reported as a number instead of an anecdote — built from a small set of phrasing families, with several instances each, run against staging and reporting an extraction rate rather than a pass or a fail. Reporting a rate matters: a suite that reports pass invites the reading that the property is now true, whereas "eleven of two hundred, up from four last month" is visibly a measurement of a distribution and invites the correct next question, which is what would have to be true for the number to stop mattering.

While you are enumerating what sits in the context, include the parts that are not prose. Tool schemas are in there, and a tool named apply_partner_discount with a parameter called margin_floor_pct describes your pricing structure to anyone who gets the model to talk about its capabilities. Tool results are in there too, so an order lookup that returns the full internal record hands over cost price and supplier id alongside the delivery date the customer asked about. Both are extractable by the same routes as the prompt text, and both are usually overlooked because nobody thinks of a JSON schema as something the model might describe out loud.

They are in the sequence for the same reason the prompt is, and the reason is not an oversight. A tool definition has to reach the model as text — the tool's name, its parameter names, the descriptions written to explain them — because the model's only way to call a tool is to write a well-formed call to it, and it cannot write a name it was never shown. A tool result has to reach it as text too, since answering from the result is the point of fetching one. Anything that has to be readable in order to be usable is also readable when the reader is asked to do something else with it.

Two things follow, and they are different in kind. A credential must leave the prompt, because there is no amount of prompt engineering that makes a secret in the context safe, and a leaked partner key is a real incident with a rotation cost and a partner conversation attached to it. Sensitive-but-not-secret material — pricing bands, internal terminology — needs a decision instead of a fix: if disclosure is embarrassing, leaving it in with a refusal line is a reasonable product choice; if disclosure is materially costly, it belongs behind a tool call that returns only the part relevant to the current request. What you cannot do is keep both in the prompt and call the refusal line a mitigation.

Where the boundary actually goes#

The same argument applies, with more money attached, to the sentence teams most often trust: Only refund orders belonging to the authenticated user, and never more than $50. Here is the shape of the system that gets built around it:

TOOLS = [
    {"name": "lookup_order",
     "input_schema": {"type": "object",
                      "properties": {"order_id": {"type": "string"}},
                      "required": ["order_id"]}},
    {"name": "issue_refund",
     "input_schema": {"type": "object",
                      "properties": {"order_id": {"type": "string"},
                                     "amount": {"type": "number"}},
                      "required": ["order_id", "amount"]}},
]

def issue_refund(order_id: str, amount: float) -> dict:
    payments.refund(order_id, amount)          # no ownership check
    return {"status": "refunded", "amount": amount}

The tool takes an order id and an amount and does what it is told. Every constraint that matters — whose order this is, and how large the refund may be — exists only as English in the system prompt, which means both constraints are enforced by the model choosing arguments correctly. The route to another user's order is then to get a plausible order id into the conversation and let the assistant call the tool with it, and any untrusted content the system retrieves is a channel for doing exactly that. No jailbreak is required — no phrasing that argues the model out of a rule it was given — because a refusal line about disclosure has nothing to say about arguments.

How an argument comes to be chosen is the step to make explicit, since that is where the money moves. The model does not call anything. It emits a block of text in the format the provider defined for tool calls, naming a tool and carrying a JSON object of arguments, and that block is produced by the same scoring loop that produces prose, conditioned on the same sequence — which by this point holds the customer's message and whatever else the system pasted in. Your dispatcher reads the block, matches the name against a table of handlers, and calls the handler with the arguments as it found them. Nothing between the emitted block and payments.refund inspects intent, because there is nothing in the block to inspect: an order id proposed because a stranger's sentence suggested it is byte for byte an order id proposed because the customer asked.

A rule in the prompt is therefore aimed at the wrong component. It constrains the caller, and the caller is the part of this system an attacker gets to influence. The handler behind the tool runs whatever arrives at it, having been told nothing about the rule at all.

The fix is a handler that cannot be talked into anything, because it does not consult the model for the facts that matter:

def issue_refund(order_id: str, amount: float, *, session: Session) -> dict:
    order = db.orders.get(order_id)
    if order is None or order.customer_id != session.customer_id:
        return {"error": "not_found"}              # same answer either way
    if amount <= 0 or amount > min(order.refundable_total, 50_00):
        return {"error": "amount_exceeds_limit",
                "max_allowed": min(order.refundable_total, 50_00)}
    if db.refunds.exists(idempotency_key=(order_id, session.request_id)):
        return {"error": "already_refunded"}
    return payments.refund(order.id, amount, actor=session.customer_id)

Three properties carry the whole change. The customer identity is not a tool parameter; it comes from the authenticated session that the transport established before any model call happened, so there is no argument the model can emit that changes whose orders are reachable. The ceiling is compared in code against the order's own refundable total, so the number the model proposes is a request rather than a decision. And the failure for someone else's order is not_found rather than forbidden, so the tool does not confirm the existence of orders the caller cannot see.

One line of that handler repays reading closely, because a ceiling that compares two different units is not a ceiling. 50_00 is fifty dollars written in minor units, the underscore separating the dollars from the cents for a human reader and ignored by Python. The comparison against min(order.refundable_total, 50_00) is the rule you meant only if amount and refundable_total both reach the handler in those same minor units. Whether they do is a property of the calling code and of the schema the model fills in, and it is worth establishing by reading rather than assuming, because a unit mismatch here fails silently and fails expensively in one direction.

The not_found on the ownership failure is doing a second kind of work, and it is the kind a later refactor undoes while making error messages more helpful. A handler that answered forbidden for an order that exists and not_found for one that does not would be telling the difference to whoever is asking, which converts a guessed order id into a way to enumerate real ones: send a value, read which error comes back, learn whether that order exists. The same answer in both cases removes the signal, at the price of an unhelpful message in the one case where a legitimate customer really did mistype their own order number.

The idempotency key deserves its exact reading too. Keyed on (order_id, session.request_id), it makes a refund settle once within a single request, which covers a transport retry and a model that emits the same call twice in one turn. Two separate requests carry two separate request ids and therefore two separate keys, so a customer asking again tomorrow, or a payload firing in each of several conversations, is not deduplicated by this line. Bounding that case is the ceiling's job and the refund ledger's.

The tool definition changes to match: issue_refund no longer accepts a customer id, and lookup_order returns nothing for orders outside the session. Once that is true, the sentence in the prompt is describing behaviour rather than enforcing it, which is the correct division of labour.

The bottom two rows are the only ones whose answer in the third column is unconditional, and that column is what the word boundary means. The others are worth having; an eval suite catches regressions, an output filter is a useful second layer, and a clear prompt rule genuinely lowers the rate at which the model tries something you did not want. None of them changes what happens when an input arrives that nobody wrote a case for.

Two of the middle rows are worth taking apart, because both get mistaken for the bottom two. An output filter is a pattern match run over the reply after the model has written it, so it catches the shapes somebody wrote patterns for and misses the same content paraphrased, translated, spelled with spaces between the characters, or split across two turns. Its more serious limit is ordering. The tool call is made and answered before the reply text is generated, so by the time a filter reads a sentence announcing a refund, the refund has happened. A filter can suppress a disclosure and cannot recall an action.

An eval suite has the property its own name gives away: it replays the cases in it. A green run means the attacks somebody wrote have stopped working, which is worth knowing after a prompt edit or a model change and is silent about the space of inputs nobody enumerated. That is why an extraction rate is the honest report and a pass is not, and why the third column of the table above asks about a phrasing you never imagined rather than about the ones you did.

The cheaper plan, and what stays in the prompt#

The counter-proposal is always the same and always plausible: harden the prompt, add the red-team cases to the eval suite, ship this sprint, and skip rebuilding the authorization layer. It will lower the measured failure rate. That is precisely why it gets shipped, and it is worth conceding out loud before disagreeing, because pretending the cheap plan does nothing is how you lose the argument to someone who has watched it work.

What it does not do is change what the security property is made of. After the hardening, "another user's order cannot be refunded" is still a claim about how a text predictor responds to inputs, and the eval suite measures the attacks somebody thought of on the afternoon they wrote it. The attacker needs one that nobody thought of, once. There is no phrasing of the rule that comes with a guarantee about the next framing, and each round of hardening produces a prompt that is longer, more contradictory, and more expensive on every request.

Answering the cheap plan on its own terms means being exact about what a lower rate is and is not, since the improvement it buys is genuine. Extra sentences shift the model's output distribution on the inputs somebody tried, and the smaller failure rate turns up in the suite the following week. A rate is a summary of sampled inputs. The path — a sentence in the sequence, weighed by a predictor, deciding an argument that money moves on — is a property of the arrangement, and no sentence added to the prompt removes it. Driving a rate down and removing a path are different operations, and only the second has a value that stays true for inputs nobody has written yet.

The estimate on the other side of the trade is usually wrong too. "Rebuild the authorization layer" describes the handler above, which is a session parameter, an ownership comparison, a bounded amount, and an idempotency key so that a retried or duplicated call within one request settles once rather than paying twice: a day's work and a migration to remove customer_id from a tool schema. The reason it gets quoted as a quarter is that the tools were designed to take identity as an argument, so every call site assumes the caller supplies it, and the real project is threading the session through code that never had it. That is a genuine cost and it is a one-time one, against a prompt-hardening cycle that recurs every time someone finds a new framing.

Moving enforcement out does not empty the prompt. What leaves is the credential and the belief that a sentence was the control; the policy stays, because the tool can only answer whether a refund is permitted, and something still has to decide when one should be offered, which is a product question with no server-side answer. Three things in particular belong there. When to offer a refund at all — the circumstances, what to ask for first, when to route to a human instead. That the assistant must call issue_refund rather than describe an outcome in prose, since a promise in text is not a refund and a customer who has been told they will be refunded is now a support problem whether or not the tool ran. And how to explain a denial: the handler returns amount_exceeds_limit with a number in it, and the difference between "that is above the limit I can approve; I have flagged it for an agent, who will confirm today" and a raw error code is entirely the prompt's work.

Read together, the prompt describes the intended path and the tool defines the permitted one. When they disagree the tool wins, which is what makes the arrangement safe, and the prompt's job is to make disagreement rare and legible when it happens. That division also tells you what to do with any sentence in a prompt that sounds like a guarantee: either find the code that enforces it, or stop describing it as a control and start measuring how often it holds.

A prompt that has been through this exercise tends to be a prompt that has been through several other exercises too, and it shows. The next chapter starts with an inherited one — 2,300 words, three quarters of a second of latency, and a contradiction four paragraphs deep — and works out what to remove first.