Ambient authority

The two questions a multi-tenant server — one deployment serving several unrelated customer organisations, each of them a tenant — answers on every request are which tenant the call is for and with whose authority the work is done. Per-request identity settles the first. This chapter is about the second, and about the specific way it goes wrong on a server where the first has already been fixed: a code path that never asks whose data to return, because it was built to use a credential the process already holds.

Ambient authority is the name for that credential, and it is borrowed from capability-based security, where the distinction is between authority a subject wields because of who it is and authority it wields because it was handed a specific, designated capability. It arrives through the environment rather than through a parameter — a module-level singleton, an environment variable read inside the function, a machine role attached to the pod (a cloud identity issued to the process itself, which any line of code in it can spend), a database connection already holding a privileged role, a filesystem the process can read, or simply a network position from which an internal service answers anyone who can reach it. Hardy's confused deputy is the same defect stated as a story: a program with authority of its own, given a name by someone else, uses its authority on that name. Every finding in this chapter is a confused deputy, and the name is always an identifier the caller chose.

The property that makes it invisible#

A specimen is more useful than the definition. This handler refuses to write across a tenant boundary, correctly, and leaks anyway:

async def add_to_collection(identity: Identity, collection_id: str, document_id: str):
    title = await resolve_title(document_id)          # ← the finding
    ok = await repo.add_to_collection(
        identity=identity, collection_id=collection_id,
        document_id=document_id, label=title,
    )
    if not ok:
        raise ToolError(f"Cannot add '{title}' to that collection.")

resolve_title is forty lines away in a utility module, has no tenant parameter, and never needed one:

SERVICE_DOCS = DocsClient(api_key=os.environ["DOCS_SERVICE_KEY"])  # module import time

async def resolve_title(document_id: str) -> str:
    row = await SERVICE_DOCS.fetch(document_id, fields=["title"])
    return row["title"]

Nobody who wrote either function believed they were writing an access control decision. One of them was fetching a display label; the other was rendering an error. Between them they build a working cross-tenant read oracle with a rate limit of one document per HTTP request, and the tenant filtering in the repository layer is entirely correct throughout — the insert is refused, the row never appears, and the title is in the refusal.

The chain that turns a display label into a read oracle is worth walking link by link, because each link is checkable and each is easy to skip. The caller chooses document_id, so the identifier crossing into resolve_title is a value the caller controls. resolve_title fetches with a credential that spans every tenant, so the fetch succeeds for any identifier that exists anywhere in the product. The fetch is issued before the repository is consulted, so the refusal that the repository is about to produce has not happened yet and cannot prevent it. And the returned title is interpolated into the refusal, so the value the privileged fetch produced is handed back to the caller as the price of a good error message. Every one of those four is a property somebody chose deliberately, for a reason that was correct locally; the leak is the composition, which is why a reviewer looking at any single link sees nothing to stop on.

Read the expression await resolve_title(document_id) on its own and there is no credential in it, no identity, no permission, nothing a reviewer scanning a diff would stop on. That is the property, and it is why a per-tool code review passes over these paths: they do not look like data access, because the thing that makes them data access is stored somewhere the call site does not mention.

Passing the credential converts the problem into one a reviewer and a compiler can both see:

async def resolve_title(docs: DocsClient, document_id: str) -> str:
    row = await docs.fetch(document_id, fields=["title"])
    return row["title"]

Now resolve_title(document_id) does not type-check, every call site names the client it is using, and a grep for the parameter enumerates the whole surface. This does not by itself stop anything — a caller can still pass SERVICE_DOCS — and that is the correct order of operations, because you cannot fix a set you cannot list.

The belief that has to be dislodged before any of this lands is that the repository's tenant filter is the access control layer. It is tempting for a good reason: it is true of every path that goes through the repository, those are most paths, and they are the paths a reviewer thinks to check. What makes it false is a property of the object rather than of any particular call. SERVICE_DOCS is one client, constructed once at import, byte-identical on a Northwind request and on a Contoso request, and it therefore carries no information at all about which tenant is asking. A control that has to tell two tenants apart must vary when the tenant varies, and a module-level singleton is by construction the same thing on both calls. So the problem is not that resolve_title forgot to check. It is that resolve_title was never in a position to check anything, and no amount of care inside it changes that while the credential it uses is fixed.

The confused-deputy story is a picture rather than a description, and it is worth saying where the picture stops. In Hardy's telling there is a deputy, a name supplied by someone else, and an authority the deputy already holds, which maps cleanly onto the handler, the document id and the startup client. What the story does not carry is that the deputy here is usually two functions written months apart by two people, neither of whom saw the other's half. Nobody in the codebase is confused, because nobody ever held both facts at the same time. The story explains why the defect is possible; it understates how comfortably it survives review.

Listing the outbound calls that are not queries#

The token path and the query path are the two things every review looks at, which means that in any server that has already been reviewed, the surviving leaks are on paths that are neither. That is the whole exercise, and it is mechanical: take one request, follow it to every point where the process talks to something outside itself, and strike out the ones that go through the tenant-filtered repository. What remains is the list.

Doing that from memory produces a shorter list than doing it against a running process, because the calls that leak are the ones nobody remembers writing — a title resolver in a utility module, a metrics decorator that tags a span with a document name, a health check that touches a shared bucket. The version that works is to run a single tool invocation against a staging instance with every outbound socket recorded, then reconcile the recording against the repository's own query log. In CPython an audit hook — a callback the interpreter invokes whenever the program performs a sensitive operation such as opening a socket or a file — gets you the recording in a few lines:

import sys

WATCHED = {"socket.connect", "urllib.Request", "open"}

def audit(event: str, args: tuple[object, ...]) -> None:
    if event in WATCHED:
        sys.stderr.write(f"EGRESS {event} {args!r} req={current_request_id()}\n")

sys.addaudithook(audit)

Reading a request id out of a context variable is acceptable here in a way it was not in the handler, because this record is diagnostic and a wrong attribution costs a confusing log line rather than a tenant's data. Run one add_to_collection call through that hook on a server of moderate size and the output is longer than expected: in one codebase of roughly 30,000 lines the reconciliation produced 23 distinct outbound destinations for a single tool call, of which nine were repository queries, six were telemetry and tracing, and eight were the list — the eight nobody could have named beforehand.

Those three counts close, which is what makes the exercise a reconciliation rather than an impression. Nine plus six plus eight is twenty-three, the same twenty-three the hook recorded, and there is no residual category: every destination the process opened has been placed in exactly one bucket. That closure is the whole reason to do this against a recording instead of from memory. A list assembled from memory has no total to check itself against, so a forgotten destination leaves no trace of having been forgotten; a list that must add up to a recorded number cannot quietly lose one, because the sum comes out short and somebody has to go and find the missing row. Taking the nine repository queries out of the twenty-three leaves fourteen non-query destinations, and of those fourteen the six telemetry and tracing endpoints were ones the team could have named in advance. The eight that remain are the finding.

Nine rows is more than most teams expect and fewer than most servers have. The rows are not exotic; every one of them is a normal engineering decision that was correct in a single-tenant service and became an authority boundary when the second customer arrived.

Two of the numbers on this page count different kinds of thing, and keeping them apart matters more than it looks. The table's nine rows are categories — kinds of outbound call — while the eight are destinations observed on one recording, of one tool invocation, in one codebase. A category with no instance in your server contributes a row and no destination; a single category can contribute several destinations. Neither number is therefore a check on the other, and a team that finds eight rows in its own reconciliation has not thereby covered the table. The six telemetry endpoints show the same distinction from the other side: they are instances of the log-and-trace row, so they sit inside the table, and they were still counted apart from the eight, because the eight is a count of destinations nobody could have named rather than a count of destinations that leak. Being nameable in advance is a different property from being harmless, and the table's inclusion of that row is what says so.

Two rows are worth dwelling on because they fail differently from the rest. A presigned URL is authority that has been detached from the request entirely — once minted, it is a bearer credential that anyone holding the string can redeem for the next fifteen minutes or seven days, with no further reference to the identity that caused it to exist. The check happened at mint time; the access happens whenever. A presigned URL signed with SigV4, the AWS scheme that puts a request's signature into its query string, using a role's temporary credentials can carry an expiry of up to seven days, and the common default in application code is an hour, so a URL that reaches a log aggregator, a chat transcript or a model provider's request record remains redeemable long after the request that produced it has been forgotten. And a cache is authority that has been detached from time: whatever the first caller's credential fetched is what the next caller reads, so a cache populated by a privileged path serves privileged results to unprivileged callers for the rest of the TTL.

Those two rows share a shape worth naming, because it recurs everywhere once you have a name for it. An authority check and the access it authorises have been separated, and something in between is holding the result. A presigned URL separates them in space: the check runs inside your process and the redemption runs at the storage service, with nothing of yours left in the path to object. A cache separates them in time: the check ran on the request that filled the entry, and the access runs on every request that reads it. Wherever a check and its access are separated by a stored intermediate, the question that decides whether the separation is safe is what identifies the stored thing — and when the answer does not include the caller, the store is a mechanism for lending one caller's authority to the next.

Two days before a second customer#

Suppose you inherit this server and there is a date on the calendar. The list above is the first day's work, done against the real call graph rather than from memory, and the second day is spent on whichever rows exist.

Given a server where the token path and the query layer are both already correct, the row to open first is name resolution — anything that turns an id the caller supplied into a human-readable string without going through the repository. It is first because it is the one nobody classifies as data access, it is usually implemented once and called from four or five places, and the thing it returns is exactly what a document title is: the sensitive part. What you should expect to find is a single helper of a dozen lines, a module-level client above it, and no tenant parameter anywhere in the file. If instead the helper takes an identity and the client is built per request, you have learned something real about the codebase in twenty minutes and can move to object storage.

What has to be true before the second customer is enabled is narrow enough to finish. Every module-level or import-time client is either deleted or moved behind a factory that will not produce one without an Identity, which is a change you can make mechanically and verify with a search for the module names. The list of non-query outbound calls is written down, in the repository, with the tenant enforcement point named for each row. And the paths you found are exercised by tests that run as one tenant and reach for another, since a fix with no test is a fix with a half-life of one refactor.

What can honestly wait is the structural work: splitting the search index per tenant instead of filtering it, moving enforcement into a gateway, replacing shared caches with per-tenant namespaces where the current keys are merely long rather than wrong. Those are weeks, they are the right destination, and none of them is a reason to hold a launch when the mechanical fixes close the paths that a tester actually walked.

A credential nobody had to pass in is a credential nobody has to think about, and that is the whole of the defect. The nine rows in the table are not clever attacks; they are the outbound calls a normal server makes on a normal request, and each of them was written by someone solving a display problem, a latency problem or an observability problem in a codebase that had one customer. Making the authority visible at the call site does not remove it, and it converts a search of everybody's memory into a search of the source.

Two things it explicitly does not buy. It changes nothing about blast radius, meaning how far a single compromised request or process can reach once it is loose: if add_to_collection now receives SERVICE_DOCS as an argument, the tester's loop still works, and the only difference is that the diff which introduced it would have been easier to catch. And it is not the same distinction as scope — a token minted with too many scopes is over-privileged, and it is still passed in, still logged, still attributable to a caller, so authority can be perfectly scoped and remain ambient, because the defect is in how it is obtained rather than in how much of it there is.

What is left is the question of what the credential in the argument should be. A single integration account that every request borrows is one answer, and the reason it is the wrong one has to do with what a single compromised request can then do — and with the parts of the system where there is no request at all.