Authorization has to be in the key, not around it

At 09:14 an administrator at Globex asked the assistant "when does our contract renew?" The request was authenticated, the session carried tenant_id=globex, a tool call read the contract record for that tenant, and the model produced "Your current agreement renews on 14 March 2027." The response was stored in the cache under the embedding of the question text. At 11:42 a finance manager at Acme typed the same nine words. The nearest neighbour scored 0.99, comfortably over the 0.93 threshold, and the assistant returned Globex's renewal date in 40 milliseconds without calling a tool, without reading a contract, and without any component in the request path having an opinion about it.

Nothing in that trace is a bug in the sense of a line of code that behaves differently from how it reads. Authentication worked. The authorisation middleware ran and correctly confirmed that the Acme user was entitled to use the assistant. The tenant-scoped data access layer, which has tests proving it cannot read another tenant's contracts, was never invoked, because the cache returned before the code that would have called it. The control was on the path the cache exists to skip.

The check ran, and it ran on the wrong side of the lookup#

Web infrastructure worked this out a long time ago and gave it a name. A shared HTTP cache that stores an authenticated response and serves it to the next requester is the oldest cache-poisoning-adjacent defect there is, and the fix has been in the specification since HTTP/1.1: the response declares Vary on whatever request dimension changed it, or it declares itself private and is not stored in a shared cache at all. Every CDN has a cache key configuration where you list the headers and cookies that participate. A semantic cache is a shared cache holding authenticated responses, and it usually ships with a cache key consisting of one thing that was never part of the authorisation decision.

The reason this lands so often is that a cache is naturally written as a wrapper around the expensive part of the handler, and it is placed as early as possible precisely because everything after it costs money. In a service that grew organically, the work sitting behind that point includes tenant resolution, entitlement lookups, row-level filters and the audit log write, none of which announce themselves as security controls in the diff that adds the cache. Provider prompt caching never raises the question, because its key is the bytes you sent and it returns computation rather than content, so a team that has shipped one and reasons by analogy has no prior for asking where the authorisation boundary sits relative to the lookup.

The fuzzy key makes the failure worse than its HTTP ancestor in one specific way. An HTTP cache at least requires the URL to match, so the collision is between requests for the same resource. Here the key is approximate, so an Acme user does not need to type Globex's question — anything within 0.93 of it will do, and the previous chapters established how wide that neighbourhood is. "when's our renewal date" scores 0.95 against the stored Globex entry.

Whatever could change the answer belongs inside the key#

The fix is to make the cache entry unreachable by a request that is not equally entitled, which means the identity dimensions become part of the lookup rather than a gate around it.

def scope_key(ctx: RequestContext) -> str:
    """Every dimension that can change what a correct answer says."""
    parts = [
        ctx.tenant_id,
        ctx.plan,                       # enterprise vs starter: different limits
        ",".join(sorted(ctx.roles)),    # admin sees billing detail; member does not
        ctx.data_region,                # residency changes retention answers
        ctx.locale,
        EMBED_MODEL_VERSION,
    ]
    return hashlib.sha256("|".join(parts).encode()).hexdigest()[:16]

def lookup(question: str, ctx: RequestContext):
    scope = scope_key(ctx)
    return index.search(
        namespace=scope,                # partition, not a post-filter
        vector=embed(question),
        k=1,
    )

Two details in that snippet carry most of the weight. The scope has to be recomputed from the authorisation source of truth on every request, because a scope hash cached on the session survives a role being revoked and reintroduces the same defect one layer down. And the partition has to be applied inside the approximate nearest-neighbour search rather than to its results. Approximate nearest-neighbour search, usually shortened to ANN, is what a vector store does instead of comparing a query against every stored vector: it walks a graph or a set of buckets toward the closest entries and accepts that it may miss a few, which is what buys the millisecond lookup. Filtering after a top-k retrieval, where the index hands back its k best guesses and the caller throws away the ones that do not belong to it, is the tempting shortcut, since it works in a test with a hundred entries, and in production it produces silent recall collapse: the top 10 neighbours are all other tenants' entries, they are all discarded, and the tenant's own perfectly good cached answer three positions further down is never considered. Every serious vector store has a way to push the filter into the traversal, and the difference between the two shows up as a hit rate that mysteriously falls as the index grows.

The last row is where the design gets honest about its cost. Partitioning by scope turns one cache with 40,000 entries into 4,000 caches with ten entries each, and a cache with ten entries has no hit rate worth having. Tenant-scoped semantic caching is close to worthless on a long tail of small accounts, and the arithmetic does not improve with a better index. What survives partitioning is the class of question whose answer does not depend on the scope at all, which is where the allowlist comes from.

The write gate is the real control#

The allowlist for a support assistant is shorter than most teams expect and holds only requests whose correct answer is fully determined by published content: how a feature works, what a term means, what the documented steps are for an operation, what the published pricing tiers contain. Those can share a single global namespace across every tenant, because there is nothing tenant-specific in the answer, and they are also the questions with the highest repeat rate, so the allowlist captures most of the available saving anyway.

The denylist is where the judgement sits, and every entry on it corresponds to a way an answer can be right for one requester and wrong for the next.

The rule that does most of the work is the first one, and it is worth stating as a mechanical check rather than a taxonomy: if the generation invoked any tool that reads customer state, the response is not eligible to be written. That does not require a classifier or an intent taxonomy, because the tool-call trace is already in the response object, and it is exactly the signal that the answer depends on something outside the question text.

UNSAFE_TOOLS = {"get_contract", "list_orders", "get_invoice", "get_usage",
                "get_entitlements", "search_tickets"}

def may_store(question: str, result: Generation, ctx: RequestContext) -> bool:
    if UNSAFE_TOOLS & set(result.tools_called):
        return False                       # answer depends on tenant state
    if result.finish_reason != "stop" or result.refused:
        return False
    if POSSESSIVE_RE.search(question):     # my / our / this account
        return False
    if not intent_allowlist.contains(result.intent):
        return False                       # default deny on unclassified
    return True

Multi-turn conversation is the case that slips past all of the above, because the question text that gets embedded is usually the latest user turn and the meaning of that turn lives in the ones before it. "What about for express shipping?" is a complete question inside its conversation and an ambiguous fragment outside it, and it will embed close to every other question containing those words regardless of what preceded them. Two conversations that reach the same follow-up from different starting points get the same key and deserve different answers. The workable rules are to cache only turn one of a conversation, or to embed a resolved question — the follow-up rewritten as a standalone query, which a RAG pipeline may already produce for retrieval — and never to embed a bare follow-up. Keying on the whole conversation history instead is not a fix, because it drives the hit rate to approximately zero: no two conversations share a prefix.

Enforcing on the write side rather than the read side matters because the two are not symmetric in their failure. A read-side rule that is missing one case serves a dangerous entry that is sitting in the cache waiting; a write-side rule that is missing one case fails to store something cacheable and costs you a little hit rate. Entries that were never written cannot be served, so every subsequent bug in the retrieval path is bounded by what the write gate allowed in. The classifier deciding intent allowlist membership has its own error rate, and the default on an unclassified or low-confidence intent is not to store.

How many customers saw someone else's data#

The cross-tenant defect has one redeeming property, which is that it is exactly countable, provided the logging schema from the previous chapter was in place. Every hit recorded the id of the matched entry, and every entry recorded the tenant whose request created it.

SELECT r.request_id, r.tenant_id AS served_to, e.tenant_id AS answer_from,
       r.user_id, r.ts, r.similarity
FROM   responses r
JOIN   cache_entries e ON e.entry_id = r.matched_entry_id
WHERE  r.cache_hit
  AND  e.tenant_id IS NOT NULL
  AND  e.tenant_id <> r.tenant_id
ORDER  BY r.ts;

On this system the cache had been live for nine days across 1.24 million responses, of which 380,000 were hits. The query returned 412 rows: 156 distinct users across 47 tenants, each of whom received an answer generated inside a different customer's account. Each row names the request, the recipient, the source tenant and the timestamp, so the notification list is a SELECT DISTINCT away and the exact text each user saw is recoverable from the entry. That is a bad morning and a set of disclosure emails with real content in them, and it is finite.

Two limits on that count are worth stating rather than discovering during the write-up. The query only sees as far back as the response log retains, so a thirty-day retention on a cache that has been live for nine days is fine and the same defect found after four months is not. And it counts exposures rather than consequences: the log establishes that 156 people were shown a sentence containing another customer's contract date, and it says nothing about how many read it, believed it, forwarded it, or made a decision on it. Regulators and customers generally want the first number, which is why having it within an hour of the page changes the shape of the conversation considerably.

The same post-incident review surfaced a second problem. The refund policy changed a fortnight ago, and the cache is very likely still answering with the old one — same mechanism, no tenant boundary crossed, no query that will find it, because there is no column recording which version of the policy any entry was built from. The tenant fix does nothing for it, and the next chapter is about why the reflex response of shortening the TTL is a number without an argument behind it.