One request sends more than the prompt and the completion

The question on the security questionnaire was one line: list all third parties that receive customer content in the course of processing a request. The platform team answered it in a fifteen-minute meeting — the model provider receives the prompt and returns the completion, and that is the extent of it. Two people signed off.

The infrastructure team then ran a two-hour capture at the egress gateway — the single hop every outbound connection from the cluster has to pass through, which makes it the one place where egress, the traffic leaving your network, can be observed in full — filtered to the traffic generated by a single synthetic assistant request carrying a marked payload, and produced this:

One request, five vendors, roughly 165 KB of customer content crossing the boundary against the 9.4 KB the team had in mind. The synthetic request was deliberately unlucky — it hit a timeout and a 500 — but nothing in it was exotic, and every one of those calls was in the codebase, written on purpose, by someone on that team.

The paths that are easy to forget are the ones that are conditional#

What the fifteen-minute answer describes is the happy path of the main call. Everything missing from it is either a repetition of that call, a satellite of it, or a consequence of it failing, and each of those categories hides for a different reason.

Retries repeat the payload and nobody counts them because the SDK does them. The Anthropic and OpenAI clients retry on connection errors, 408, 409, 429 and 5xx by default, twice, with backoff. From the application's point of view one function was called; from the network's point of view the customer's data left the building three times, potentially to three different edge locations. During an incident where the provider is degraded, the multiplier is not theoretical: a bad twenty minutes at 40,000 requests an hour produces tens of thousands of duplicate transmissions.

There is also a layered version of this that produces multiplication rather than addition. If the SDK retries twice, a gateway in front of it retries twice, and a job runner retries the whole task three times on failure, a single logical request can become eighteen transmissions during a bad minute. The fix is to own retries at one layer and disable them at the others — set max_retries=0 on the client and handle the policy in code where you can count it, or leave it on the client and turn it off in the gateway. Either is defensible; having all three on is a decision nobody made.

Tool calls cross the boundary in both directions, and the return direction is the one that carries weight. The arguments the model produces go out with the next request, which is expected. The tool result also goes out — that is the entire point of a tool loop — and a result is frequently a database row. A tool called lookup_account returns the customer's plan, their billing address and their contact email, and those fields are now in the messages array of the following request, having never appeared in anything the user typed. Teams that carefully redact user input and never look at their tool schemas end up sending more personal data through the tool channel than through the prompt.

{
  "role": "user",
  "content": [{
    "type": "tool_result",
    "tool_use_id": "toolu_01Hb",
    "content": "{\"account_id\":\"AC-88213\",\"holder\":\"Marcus Bell\",\"email\":\"m.bell@northgate.co.uk\",\"plan\":\"enterprise\",\"billing_address\":\"14 Chandler Row, Salford M5 4QT\",\"open_disputes\":2}"
  }]
}

The system prompt is the third omission, and it leaks in the other direction. It usually carries internal identifiers, the names of internal services, escalation policies and thresholds — the sentence "if the account balance exceeds £50,000, route to the retention team and offer up to 15% off" is a commercial disclosure sitting in a payload that goes to several vendors. The sharper problem is few-shot examples, the two or three sample exchanges pasted into a system prompt to demonstrate the answer shape you want. Prompt engineering by copying three good production transcripts into the system prompt is the single most common way one customer's content ends up inside another customer's request, and it is invisible to every control discussed so far, because the content is in the template rather than in the input. It also means that content survives every deletion request, since nobody thinks of a prompt template as a data store. Sending the system prompt is unavoidable; knowing what is in it, and where each sentence came from, is the part that is optional and usually skipped.

The observability exporter is the one that surprises people who have read the previous chapters, because it is the same span they carefully redacted. Redaction at ingest only helps if the exporter runs after it. The default OpenTelemetry GenAI instrumentation captures prompt and completion content as span attributes, so a repository that adds opentelemetry-instrumentation-anthropic to its requirements file and configures an OTLP endpoint — the ingest URL for the OpenTelemetry Protocol, which is the wire format collectors and vendors agree on — has, in two lines, created a second full copy of every prompt in a vendor's cluster, on that vendor's retention schedule, in that vendor's region.

Crash reporting is the same story with worse defaults. Sentry's server-side integrations attach the HTTP request to the event, and request.data is the parsed body — which for an assistant endpoint is the user's message. The default scrubber removes fields named like password, secret, api_key and card numbers; it has no opinion about a field called message. An exception thrown anywhere inside the handler ships that body to a third party under a 90-day retention policy that nobody chose.

Half the rows should not exist#

An inventory read carefully is a list of things to remove, and four of the eight rows in the opening table did not survive review.

The alerting rule was deleted the same day. Quoting the first 400 characters of a matching prompt into an incident channel is a convenience that puts customer content into a chat vendor's retention, in whatever region that vendor's workspace lives, readable by everyone in the channel and searchable forever. The replacement quotes the trace id and a link.

The error reporter was reconfigured rather than removed, because crash reports are genuinely useful. Sentry and its equivalents let you strip the request body before send, and the before_send hook is where that belongs — dropping request.data entirely and attaching the trace id and the span's structural attributes instead gives you the stack trace, the correlation key and none of the text. The team measured the loss in debugging capability at approximately zero, since the exception messages were never about the content.

The telemetry exporter was fixed by moving redaction upstream of it, which was the first chapter's rule applied one layer out: the exporter can only export what the span carries, so a span built under a field-class policy exports safely by construction. What had gone wrong was that the auto-instrumentation package added gen_ai.prompt after the application's own redaction ran.

The embeddings vendor was the interesting one, because the call is necessary and the content is necessary — a retrieval query has to be embedded to be useful. What changed was the vendor, from a general-purpose API to a model served inside the same VPC, on the reasoning that a 1.1 KB payload leaving the network 40,000 times a day for a task that a small local model does adequately is a poor trade. That is a build-versus-buy decision that the inventory reframed by making the recurring privacy cost visible next to the hosting cost.

What is left after that pass is two destinations that receive content because they cannot function without it. Those two are the ones the contract paragraph has to name, and a two-vendor list is a defensible thing to put in front of a customer in a way that an eight-vendor list is not.

An inventory nobody maintains is worse than none#

The obvious response is a spreadsheet listing destinations, owners and data classes, reviewed quarterly. It goes stale between the first and second review, because the thing it describes changes whenever anyone adds a dependency, and the spreadsheet has no way of finding out. Six months on it is a document that says "we have thought about this", which is the worst possible artefact to hand an auditor, since it is a specific claim that is now specifically wrong.

The version that stays true is derived rather than maintained. Three mechanisms do most of the work, and they are cheap in a way that quarterly review is not.

The first is a default-deny egress policy, which turns the network itself into the record. If outbound traffic from the inference namespace is blocked except to an allowlisted set of hostnames, then the allowlist is the inventory, it cannot drift from reality, and adding a destination requires a change to a file that a human reviews. The failure mode inverts usefully: a new vendor integration fails in staging with a connection error rather than succeeding silently in production.

The second is a canary assertion in the test suite. Run a representative request through a recording proxy — a proxy the test points every outbound client at, which forwards each request and keeps a copy of its host, path and body for the assertions to read — with a marked payload, then assert on both the set of destinations and the presence of the marker in each body. This catches the case the allowlist cannot — an already-allowed destination that starts receiving more than it used to, which is what happens when someone enables prompt capture on the telemetry SDK.

CANARY = "EGRESS-CANARY-4f19bd"

ALLOWED_WITH_CONTENT = {"api.anthropic.com", "guardrails.vendor.io"}
ALLOWED_NO_CONTENT = {"otlp.observability.io", "ingest.errorreporting.io"}


def test_only_expected_destinations_receive_content(recording_proxy):
    run_assistant_turn(user_message=f"help me with {CANARY}", force_error=True)

    seen = recording_proxy.requests_by_host()
    unexpected = set(seen) - ALLOWED_WITH_CONTENT - ALLOWED_NO_CONTENT
    assert not unexpected, f"new egress destination: {unexpected}"

    for host in ALLOWED_NO_CONTENT:
        for req in seen.get(host, []):
            assert CANARY not in req.body, (
                f"{host} received customer content in {req.path}")

force_error=True is doing real work in that test. The error-reporting path only exists when something throws, so a test that exercises the happy path will never see it, and the happy path is exactly the version of the system that the fifteen-minute meeting described. The same argument applies to retries, which is why a second test induces a timeout and asserts on the duplicate-transmission count rather than treating retries as invisible.

The third mechanism is a build-time check that fails when a new outbound HTTP client appears — a lint rule over the dependency graph, or simply a test that the set of hostnames in configuration matches the allowlist. It is the crudest of the three and it catches the integration that someone added on a Friday with the intention of documenting it on Monday.

What an inventory is and is not for#

This is not a threat model, and treating it as one produces the wrong conversations. Nothing on the list is an attack; every entry is a deliberate integration doing its job. The inventory exists so that statements about your data handling are checkable — so that when the contract paragraph from the previous chapter says which systems process customer content, the list of systems is derived from the network rather than from memory.

It also stops one step short of the two questions people immediately want to ask of it. It tells you that spans go to otlp.observability.io; it does not tell you how long that vendor keeps them, which is a contractual question, or which region their cluster runs in, which is the residency question. Those are the next two layers, and an inventory is what makes them askable at all — you cannot check the residency of a destination you did not know you had.

The honest limitation is coverage. A capture at the egress gateway sees what happened during the capture window, and conditional paths only appear when their condition fires. The 500 in the opening table appeared because the synthetic request was constructed to cause one; the failover path that routes to a different region when the primary provider errors did not appear at all, because the primary did not error that afternoon. Degraded paths are where the interesting behaviour lives, and they are systematically under-represented in any observation of a healthy system — which is the reason a region setting that looks correct in the console can still be wrong.