Re-classifying untrusted values downstream

A quarantine is a statement about one pair of contexts: this call sees untrusted content and holds no tools, that call holds tools and does not see the content. It is not a statement about the values themselves. A notes string that arrives in a purchase-order record is the same attacker-authored string it was inside the supplier's PDF, and nothing about having transited a validated schema changed a byte of it or reduced what it can do to a reader that treats it as instructions.

That matters because the record outlives the extraction that wrote it. The value is stored, and later read by people and systems that were not part of the design the quarantine was drawn around — a buyer's screen, a nightly digest, a search index, an export to a partner. Each of those is a sink with its own rules, and the string arrives at all of them carrying no mark of where it came from. The quarantine holds; the assumption that it made the value safe is what fails, and it fails somewhere nobody was looking.

The boundary is a place, not a treatment#

The equivalent that most backend engineers already carry is output encoding for cross-site scripting. Input sanitisation — cleaning a string once, on the way in, and treating it as safe thereafter — is the approach the web industry spent a decade abandoning, because there is no single "clean" form: the escaping that makes a string safe inside HTML text is wrong inside an HTML attribute, wrong inside a URL parameter, wrong inside a JavaScript string literal and wrong inside SQL. What replaced it is contextual encoding at each sink, with the value carrying no assumption of safety between them. Attacker-authored prose in an agent system needs the same discipline, and the sinks are just different: a human's screen, another model's context window, a search index, an export to a third party.

The diagram makes the shape of the mistake legible: the quarantine sits on one edge out of five, and four other edges leave the same store carrying the same string. Whoever drew the original design reviewed the first edge and never looked at the rest, which is the normal outcome, because the other four were built later by other people who saw a database column.

The count of edges is the argument in miniature. Five edges touch that store: one writes into it and four read out of it. The review looked at the writing edge, and looking at a writing edge can establish exactly one thing — what the value is and where it came from. The claim being made, that the value is safe, is not a property of the value at all; it is a property of what a reader does with it, and readers are on the other four edges. So a review of the write edge cannot establish safety even in principle, however carefully it is conducted, because the evidence it would need is on edges outside its scope. That is why the failure is normal rather than negligent. Everybody reviewed the thing the design was about, and the design was about the wrong edge.

Drawing the other four is a mechanical exercise and worth doing at the moment the first one is designed, since that is the only moment anybody is thinking about the boundary at all. Take the column the free text lands in and enumerate its consumers rather than its writers: every query that selects it, every export or replication job that carries it to a warehouse, every index that tokenises it, every screen that renders it, and every service whose API returns it. Most of those are findable with a search for the column name across your repositories and your warehouse job definitions, and the ones that are not findable that way — a downstream team reading a replica, a partner integration pulling a report — are exactly the ones that will surprise you, which is an argument for the label being in the data rather than in a document.

The comparison with output encoding is worth keeping and worth bounding, because it covers one half of the problem exactly and the other half not at all. What transfers is the discipline: no value is safe in the abstract, safety is decided at the sink, and the treatment is chosen by where the value is going rather than by where it came from. What does not transfer is the availability of a treatment. Escaping works for HTML because HTML has a grammar, so there is a mechanical transformation that provably turns any string into one the parser reads as text rather than as markup. A model context has no such grammar. There is no character sequence that marks the following prose as data, because the thing consuming it does not parse — it weighs meaning, which is the same conclusion topic 7.2 reaches from the prompt side. So the row for a human's screen has a real encoding available and the row for another model's context does not, and that asymmetry is why the wrapper type below can offer a for_display and can only refuse for a tool-holding context. Where the analogy runs out is precisely where the answer stops being a function and starts being a rule about which sinks the value may reach at all.

Re-classification means the label travels with the value#

Since there is no treatment that makes the string safe everywhere, the only durable mechanism is to record that the value is untrusted alongside the value, so that every later reader is confronted with the fact rather than having to know it. In a database that is a column; in code it is a type that cannot be used where a plain string is expected.

class Untrusted:
    """Attacker-authored text. Cannot be interpolated; must be unwrapped
    by a sink-specific function that states how it will be consumed."""
    __slots__ = ("_value", "_origin")

    def __init__(self, value: str, origin: str) -> None:
        self._value, self._origin = value, origin

    def __str__(self) -> str:                      # blocks f-strings and print
        raise TypeError(f"untrusted text from {self._origin} used as a string")

    def for_display(self) -> str:
        return escape_markup(self._value)          # plain text, no links or images

    def for_model_context(self) -> str:
        raise TypeError("untrusted text may not enter a tool-holding context")

    def for_index(self) -> tuple[str, dict]:
        return self._value, {"trust": "untrusted", "origin": self._origin}

The for_model_context method exists and always raises, which is deliberate — the failure arrives as a TypeError in CI on the branch that adds a new consumer, naming the file and the line, rather than as a Slack post at six in the morning. A team that needs untrusted text in a model context for a genuine reason then has to add an explicit escape hatch with a name like for_model_context_without_tools(), and that name is a thing a reviewer can search for across the repository.

A wrapper type only holds inside one process in one language. The moment the value is serialised to JSON for another service, exported to a warehouse, or read by an analytics job in a different runtime, the type is gone and only the string survives. That is why the label has to exist in storage as well, and why the two mechanisms are complements rather than alternatives: the type catches the mistakes made by the team that owns the code, and the column catches the mistakes made by everyone downstream of them.

ALTER TABLE purchase_orders
  ADD COLUMN notes_trust text NOT NULL DEFAULT 'untrusted'
    CHECK (notes_trust IN ('untrusted', 'staff_authored'));

-- what an agent-facing service is allowed to select from
CREATE VIEW purchase_orders_agent_safe AS
  SELECT id, supplier_id, status, ordered_quantity, delivered_quantity,
         amount_eur, delivery_date
  FROM purchase_orders;

The view is the blunt version of the same idea and is often the one that ships first: agents read a projection that contains only the constrained fields, and reaching the free-text column requires deliberately querying the base table. It is coarse — some agent will legitimately need the notes eventually — and it converts an invisible default into a visible request.

The last row is worth dwelling on because mixed-provenance fields are where these schemes quietly fail. If buyers append their own comments to the same column, the column's trust label degrades to the weakest contributor, and any rule that says "staff notes are fine to show as markdown" becomes an injection vector the moment one supplier-authored sentence sits above one staff-authored one. Keeping the two apart in storage is cheaper than any rule about reading them.

The reason mixing degrades to the weakest contributor is a property of where the rule is attached rather than of the values. A rule that says "this column may be rendered as markdown" is evaluated per column, so it applies identically to every value the column holds, and a rule applied uniformly to a mixed set is only as safe as the least safe member. The column's effective trust is therefore the minimum over its contributors, and a minimum is unmoved by any number of good values — one supplier-authored sentence appended under a hundred staff-authored ones sets the level for the whole column. Splitting the storage changes what the rule is attached to, so each rule is now evaluated over a set whose members share a provenance, and the minimum over a set of one kind is that kind. This is also why per-value labels have to be per value: a trust column with one row per record works, and a trust column describing a field that several parties write into is recording the same minimum in a more expensive way.

The attack that survives a correct quarantine#

Suppose the notes hole is closed, every crossing field is content-constrained, and every downstream reader treats the free text as untrusted. There is still an attack, and it is the one worth designing against, because it does not involve smuggling any text at all.

The quarantined model is choosing values from a set you defined, under the influence of a document the attacker wrote. The attacker therefore gets to choose within that set. Each individual value is valid; the combination is a lie. A delivery note engineered so the extraction reports status = "received" and delivered_items = 200 against a real, open purchase order for 200 units — where nothing was delivered — passes every validator in the schema, resolves against a genuine PO, and moves the record into the state that three-way matching — the finance check that a purchase order, a goods receipt and a supplier invoice all agree before money leaves — uses to authorise payment. Nothing in the pipeline can tell it from a true report, because the pipeline's only evidence about the physical world is the document.

The other shape is identifier selection. A po_reference that resolves against your own records is much stronger than a regex, and it still leaves the attacker free to name a real PO that belongs to another buyer, if the resolver checks only that the PO exists and is open. The fix is to check the relationship, not the existence — the PO must belong to the supplier whose document this is, and that supplier must be the one whose authenticated channel delivered it — which turns identifier selection from "pick any open PO" into "pick among the POs I was already a party to". The remaining choice set is small, and small is the goal, since it cannot be made empty.

Quantifying the residual is useful in a review. With status reduced to four values, quantity bounded by the ordered amount, amount bounded by the outstanding balance and the PO bound to the sending supplier, an attacker's entire influence over the record is a handful of bits: which of their own open POs to touch, whether to report full or partial receipt, and a number between zero and the ordered quantity. That is a bounded fraud channel against their own account, which is the same channel a dishonest supplier has always had with paper delivery notes, and it is handled the way that has always been handled — three-way matching against a goods-in scan, an independent receipt event from the warehouse scanner, and reconciliation that flags a receipt with no corresponding scan within 24 hours.

Calling that a handful of bits is a claim you can size, and sizing it is what makes the phrase mean something in a review. The attacker's remaining freedom is a choice from each of three sets, and the number of distinct records they can produce is the product of the three: how many of their own purchase orders are open, multiplied by how many of the status values are reachable for a delivery they control, multiplied by the number of quantities between zero and the amount ordered. Every one of those factors is small and every one of them is a quantity somebody at your company already knows, which is the difference from the unconstrained version — there the third factor was every string a language model can emit, and a set that large is not a set anybody reasons about. What the product cannot be is zero. Each factor is at least one, because the workflow requires a real purchase order, a status and a quantity to exist, so the residue is bounded and non-empty by construction, and bounded-and-non-empty is the signal to hand the remainder to a control that does not live in this system at all.

That is the honest end state. Architectural controls compress the attack surface from "arbitrary instructions to a system with tools" down to "a supplier can lie about their own delivery", and the last step is not an AI control at all but the business control that existed before any of this was automated. Provenance-tracking designs such as CaMeL push the boundary further by keeping untrusted values opaque to the planning model entirely, and even there the residual persists in data-dependent control flow, because a system that acts differently depending on an attacker-chosen value has given the attacker a channel however carefully the value is wrapped.

Everything so far has assumed you get to design the system. The more common position is that you are handed a design that already exists, with a ship date, and time to change one thing.