Tenant leaks outside the handler

Every correction so far has been about what a tool handler does: which identity it reads, which credential it is handed, what a test proves about it. This chapter is about the code on either side of it. A cache lookup that happens before the handler runs, a retry that fires after the response has been sent, an error mapper that turns an exception into a message, and a worker that picks up what the handler enqueued are all part of the request's path and none of them is part of the handler's body.

The defect they produce is characteristically reported as a cosmetic problem: a string appears somewhere it should not, and the obvious repair is to stop printing it. That repair is almost always wrong, because a string naming another tenant's data is evidence that something resolved that data inside a request authenticated as somebody else.

The boundary is crossed before the message is written#

Take a tool that accepts either an export id or a human name, and fails on the name branch with this:

ToolError: AccessDenied: s3://acme-exports/t_bravo/2026/07/payroll-q2-final.xlsx

Reading that as a message that leaks internal bucket paths misses what it says. t_bravo is the storage prefix of another tenant, Bravo, and payroll-q2-final.xlsx is Bravo's file, and both of those strings were produced by the server, inside a request authenticated as somebody else, before any error existed. The AccessDenied came from S3, at the very end, and it is the only reason anyone found out. Sanitising the message removes the report and leaves the resolution.

Read the sequence in the order it happened rather than in the order it was noticed. The server took a name the caller supplied. It resolved that name against an index spanning every tenant. The resolution returned Bravo's storage key, which the request now held. Only then was the key presented to a client that had no authority over anything of Bravo's, and only then did S3 refuse. The boundary was crossed when the name was resolved and enforced when the key was presented, in that order, and the error string comes last of all. A repair applied to the last event changes what the caller is told about the earlier ones and changes nothing about whether they happened. That is why "stop printing internal identifiers" is not a partial fix; it is a complete fix to a different problem that happens to share a symptom.

The path that produced it is two functions:

@alru_cache(maxsize=8192, ttl=300)
async def resolve_export_key(name: str) -> str:
    row = await exports_index.lookup(name=name)   # index spans all tenants
    return row["s3_key"]

async def download_export(identity: Identity, name_or_id: str) -> bytes:
    key = await resolve_export_key(normalise(name_or_id))
    return await storage_for(identity).get(key)   # tenant-scoped client, correct

Every individual decision in that pair is defensible. The storage client is per-tenant, exactly as the chapter on credentials as arguments requires, which is why the download failed rather than succeeding. The resolver was written when the product had one customer and a name was globally unique. The cache was added when a dashboard started calling the tool in a loop. A mistyped name matches another tenant's export because normalise lowercases and strips punctuation, and two tenants had both produced a file whose normalised name was payroll q2 final.

The consequences of a lookup like that go past the error text. The cache key is name, with no tenant in it, so for the five minutes of its TTL after one tenant's request populated the entry, every tenant asking for that normalised name gets that key without the index being consulted at all. At a 92 percent hit rate — which is what the cache was added to achieve — 92 percent of calls on this path never execute the lookup, so an enforcement check added inside exports_index.lookup would run on fewer than one call in ten and would look, from the outside, like it was working.

Both halves of that sentence are one subtraction. If ninety-two calls in a hundred are answered from the stored entry, the remaining eight are the ones that reach the function the entry was memoising, and eight in a hundred is fewer than one in ten. A check written inside exports_index.lookup therefore runs on those eight and on nothing else. A test suite that exercises this path a handful of times, against a cold cache, hits the miss branch almost every time, watches the check fire, and records it as working — while production takes the hit branch on more than nine calls in ten and never consults the check at all. The general shape is worth carrying past this example: putting an enforcement check behind a cache sets that check's coverage equal to the miss rate, and a cache is deployed precisely in order to make the miss rate small. The two goals point in opposite directions, and the cache was added second, by somebody optimising latency who had no reason to open the function they were wrapping.

The fix belongs on the path that produced the string. The resolver takes an identity and the index query filters on it; the cache key becomes (tenant_id, name); and the AccessDenied stops being reachable because there is no longer a way for one tenant's request to hold another's key. Redacting the message is worth doing as well, and it is worth being clear about what it buys: with the resolver fixed, redaction protects against the next defect of this shape rather than this one. Doing redaction instead removes the signal that told you the resolver was broken, and the next occurrence of the same class arrives with no ticket attached.

A request leaves the process by more exits than the response#

The reason a complete per-tool matrix misses this is structural. Such a matrix covers the surface — every tool, every argument — and the leaks that survive it are not on the surface. They are on the paths a request touches on its way out, most of which run after the handler has returned or before it is entered.

Four of those exits deserve individual attention, because each fails in its own way.

Retries are the exit that reorders time. A retry wrapper that captures the call at the first attempt and re-issues it later must re-resolve the identity rather than replay the captured one, since the wrapper is frequently shared infrastructure that outlives the request — an after_response hook, a background reconciliation loop, a client-side retry inside an HTTP library configured once. The failure that shows up in production is subtler than the wrong identity: a call that failed for B lands in a retry queue, the queue is drained by a worker that sets up its context from whatever request is currently active, and B's work executes with A's credentials or A's work with B's. Both directions are cross-tenant, and only one of them is visible to the user who reports a bug. Retries also multiply idempotency: a POST retried three times against a shared idempotency store with an untenanted key can return the stored result of another tenant's identical request, which is the same defect as the cache with a different name.

Caches are the exit that reorders callers. The rule that catches most of it is that a cache key must contain every input to the decision the cached value depends on, and when the value depends on who asked, the key contains who asked. Memoisation decorators — wrappers that store a function's return value under its arguments and hand it back on the next matching call — are the common source, because the decorator is applied to a function whose parameters are the key, and the identity was deliberately not a parameter. Negative caching, storing the fact that something was not found so the lookup is not repeated, is the case teams forget: caching "no such export" for A and then serving that miss to B is a leak in the direction nobody tests, and it produces a support ticket about missing data rather than about exposed data. Shared Redis instances make it worse only in blast radius; a per-process lru_cache leaks between the tenants that happen to hit the same pod.

The rule about keys follows from what a cache is, which means it does not have to be remembered as a separate fact. A cache is a promise that a stored value may stand in for recomputing the function, and the promise holds exactly when every input the function actually used is reflected in the key — because two calls sharing a key are, by the cache's own definition, the same call. When the value depends on who asked and the key does not mention who asked, two different calls have been collapsed into one, and the cache then behaves perfectly correctly while returning a wrong answer. Memoisation decorators produce so many of these for a mechanical reason rather than a careless one: the decorator builds its key out of the function's parameter list, so a function that takes an identity gets a correct key for nothing, and a function that reaches for ambient authority gets a key that has no way to mention the caller. The two chapters before this one were about making that parameter exist. This is a second, separate cost of its absence.

Logs and traces are the exit that crosses an organisational boundary rather than a technical one. A span attribute carrying document.title, an exception captured with local variables attached, a structured log line with the full tool arguments — each of those sends tenant data to a destination governed by a different set of rules than your database. An error aggregator groups by stack signature and shows the most recent occurrences, so an engineer opening a grouped issue routinely sees payloads from several tenants side by side; a support console that renders "last error" for a customer will show whatever string the server produced, including a storage key belonging to somebody else. If the customer contract says their data stays in one region, the trace exporter is a data path and needs the same answer as the database.

Background jobs are the exit whose enqueue side the chapter on credentials as arguments covered. What matters here is the far side: a job that fails and logs, a dead-letter payload that an operator inspects, a job whose retry runs after the acting user's access has been revoked. Work leaving the request means data leaving the request's enforcement, and whatever the worker writes to logs or to a shared index is outside every test that calls a tool and inspects a response.

Testing a path rather than an endpoint#

These exits are testable, and the tests look different from the ones a tool matrix produces because the unit under test is a path. For caches, the case is two requests in sequence with no reset between them — B's request, then A's identical request, asserting A's result differs — plus a structural check that every cache key constructor takes a tenant, which is enforceable as a lint rule against the memoisation decorators in use rather than as a test. For retries, the case is a call that fails deterministically for B, followed by A's traffic, with the retry executed after B's connection is closed; the assertion is on which credential the retried call presented, which usually means asserting on a recorded outbound request rather than on a response. For logs and traces, the case is to run a request as A and grep the emitted spans and log lines for B's canary strings, which is the assertion helper from the previous chapter pointed at a different output stream. And for background jobs, the case is to force a job to the dead-letter queue and inspect the payload, since the payload is a durable artefact that a human will read.

The common property is that none of these assertions can be made on a tool response, which is why a suite that only inspects responses cannot cover them regardless of how many tools it enumerates.

What the error should say#

Once the resolution path is fixed, the response for an out-of-scope reference is a design choice with a real trade-off, and it is not obvious in one direction.

The third row is the one to build, and the correlation id — an opaque reference that identifies a request without describing it, safe to show a caller because it means nothing until someone looks it up — only works if the internal record behind it is complete enough to be useful: the id itself, tenant, tool, the id as the caller supplied it, and the decision that was made, stored where a support engineer can query by correlation id and see only records belonging to the tenant they are helping. Timing is worth a thought too, since a server that takes 4 ms to reject an id that does not exist and 21 ms to reject one that exists in another tenant has published the same oracle it just removed from the message. That difference is measurable over a few hundred requests, and the practical mitigation is to make both branches do the same amount of work rather than to chase constant time.

Why a few hundred requests suffice is not obvious and is worth one step, because the step also explains why the obvious countermeasure fails. Any single timing is noisy: scheduling, garbage collection and network jitter move it around by more than the gap being looked for, so one measurement of each branch decides nothing. What distinguishes the two branches is that their difference is systematic — present in every sample, always in the same direction — while the noise is not. Averaging many samples therefore leaves the difference where it is and shrinks the scatter around it, and the attacker does not need either true latency, only two averages far enough apart to put in order. That is also why equalising the work is the mitigation and adding a random delay is not: a random delay is more noise, and noise is precisely what averaging removes.

None of this is an argument that error messages should be vague. A message that says "no export named 'payroll q2 final' in this workspace" is specific, useful, and names nothing outside the caller's scope, because the scope was applied before the name was resolved. Vagueness is what teams reach for when they cannot be sure the string is safe, and being sure is a property of the path rather than of the formatter.

Everything in this chapter and the three before it is a defect in application code, which means each has a fix in application code — a parameter added, a key extended, an attribute dropped from a span, a resolver scoped. That is the good case, and it is where most incidents live. It is also a bounded case: retries, caches, logs and error text are paths where the process did something on behalf of the wrong tenant by accident.

The question the next chapter takes up is what the process could do on purpose, and whether any amount of care on these paths changes the answer.