Tenant isolation enforced in the repository layer is not enforced

Every other failure in this topic removes something from a result list, which is what made each of them detectable. A tenancy failure adds something. One customer's documents appear in another customer's results, the list is full, the latency is normal, the fill ratio is one, and the recipient has no way to know that the contract they are reading belongs to someone else. Nothing measured so far moves.

Two words do the work here and they are worth separating before anything else. A convention is a rule that holds for the code paths that currently exist, because the people who wrote them followed it. A boundary is a rule enforced by a component that every path must go through whether or not its author knew the rule existed. The difference is not one of rigour. It is a difference in what happens to code nobody has written yet.

Why an application predicate is default-allow#

Follow what a filter does to a query, one step at a time, because the conclusion is uncomfortable and each step is checkable on its own.

A metadata filter is a conjunction of conditions a document must satisfy. Adding a condition to a conjunction can only shrink the set of documents that satisfy it. So removing a condition can only grow that set. The tenant predicate is one such condition, added by application code. Therefore a code path that omits it does not fail, error or return nothing — it returns strictly more than it should, which is every tenant's documents. That is what default-allow means: the absence of the rule is indistinguishable from a successful query with a broader scope.

Now run the same steps on a store that resolves the tenant from the credential the caller presented. The credential names a namespace or partition. The store looks up data by that name. A query arriving without a name has no location to read from, so the store has nothing to return and says so. Omission produces an error rather than a wider answer, which is default-deny. The two designs differ in exactly one respect — whether the scope is an argument the caller supplies or a property of the caller — and every other difference between them follows from that one.

The consequence is that the safety of a default-allow design is a claim about the completeness of a set of code paths, and that set grows. A convention holds for the paths that exist and offers nothing about the ones merged next quarter.

The failure that happens with the predicate applied correctly#

The leak most worth studying has nothing to do with a missing predicate, which is what makes it the useful example. To reduce cost, a team adds a result cache in front of retrieval: hash the query text, look up the cached document list, return it on a hit. The cache sits in the API layer, above the repository, and the key is sha256(query_text) + ":" + str(k).

The leak is a property of the key, and it can be derived from the key alone without knowing anything about the retrieval system underneath. A cache is a function from key to value: the same key returns the same value. This key is computed from the query text and k. The tenant is not among its inputs. Therefore two requests that differ only in tenant compute the same key. Therefore they must receive the same value. There is no configuration of the cache, no size, no eviction policy and no expiry that avoids this, because it is not a bug in the cache — it is what the cache was asked to do.

Every filter in that diagram was applied correctly. The vector store was never asked a question it should have refused. The repository added tenant_id = A exactly as designed, and tenant B received ten chunks of tenant A's contracts because a layer above the repository decided that two identical strings deserved the same answer.

That combination names the stumble in this chapter, and it is a belief about a defence rather than a slip. The integration tests asserting that a query from tenant A returns nothing belonging to tenant B continued to pass throughout — before the cache was added, after it was added, and on the day it leaked. Their verdict did not move when the dangerous property moved, because they exercise the repository and the repository never misbehaved. A test that returns the same answer whether or not the system is leaking cannot be used to decide whether the system is leaking. It is measuring a component, and the property being claimed belongs to the composition.

Cache keys are a place where the tenant boundary can be erased by an engineer who is not thinking about tenants at all, and the cache is the highest-value optimisation in the whole pipeline, so it gets added early and by someone optimising latency.

The same hazard exists in a quieter form for embedding caches. Caching text → vector is usually safe on its own, since a query vector is not tenant data, but the cache becomes an oracle: a hit reveals that someone previously embedded that exact string, and on a set of customer-specific identifiers or deal code names that is a real disclosure. Note what the attacker measures here — not the value, which is harmless, but the time it took to return. Any cache whose value or timing is derived from another tenant's activity needs the tenant in the key, and the reliable way to get that is to build cache keys from the same scope object that carries the tenant, so that constructing a key without one is a type error rather than an oversight.

Where the boundary can actually live#

The distinction that matters across the options is the default-deny versus default-allow split derived above, applied to each layer that could hold the rule.

The second row is the whole table in one line, and every other row is a consequence of it. A layout that makes another tenant's data unaddressable cannot leak it through a forgotten predicate, so the weak points listed in the last row move upward, to how a namespace name is chosen and how a credential is minted. Those are single, reviewable code paths rather than an open-ended set.

The row about small-tenant latency deserves more attention than it usually gets, because it connects this chapter to the measurements in the rest of the topic. Take a system holding 84 million chunks across 400 enterprise tenants. Divided evenly that is a little over two hundred thousand chunks each, and as a fraction of the whole it is one four-hundredth — a selectivity of about a quarter of a percent. Put that on chapter 2's sweep and it lands at the top of the recall cliff, where the mean result count is beginning to slip and recall is already well below its unfiltered value. The tenant predicate is not merely the predicate with the highest stakes. On a corpus shared between hundreds of tenants it is also, structurally, a predicate selective enough to trigger the failure that topic has spent five chapters on, and any tenant smaller than the average sits further down the curve still.

For 400 tenants and 84 million chunks, the two rows in the middle of the table are both workable and they solve slightly different problems. Per-tenant namespaces or collections give each tenant an index of their own inside one cluster, which removes the selectivity problem entirely — a tenant with 41,000 chunks is searching a 41,000-vector index at 100% selectivity, so there is no cliff and no routing decision. The cost is per-namespace overhead, which is why engines document a ceiling in the low thousands of collections and why this design stops working for a product with a million self-serve accounts.

Partitioning within a single collection keeps one index and one set of build parameters while colocating each tenant's vectors physically, so the tenant predicate becomes a partition selection rather than a filter, and the exact-scan plan from chapter 3 gets a sequential read instead of a scattered gather. That last point is worth connecting to the arithmetic there: the gather, not the arithmetic, was what made a mid-sized scan slow, and colocation is precisely the change that removes it. Qdrant's multitenancy guidance and its filterable-HNSW work describe this shape in detail — a payload index on the tenant key plus tenant-aware storage layout — and the operational effect is that the tenant filter stops being the predicate that causes recall problems and becomes the one that makes everything else cheaper.

Row-level security is worth a separate look when the vectors already live in Postgres under pgvector, the extension that adds a vector column type and vector indexes to an ordinary Postgres table. Row-level security is a Postgres feature that attaches a predicate to a table itself, so the database evaluates it on every statement no matter who wrote the query, which makes it the one option here where the enforcement is a database object rather than a service configuration. A policy such as USING (tenant_id = current_setting('app.tenant_id')::uuid) applies to every statement from every connection, including the ad-hoc psql session someone opens during an incident and the ORM query written by a developer who has never heard of the tenant model.

Its weak point is the session variable, and the failure has the same shape as the cache. The policy is a correct predicate evaluated against whatever scope app.tenant_id currently holds, and a connection pool that hands out a session without resetting it carries the previous request's tenant into the next one. In both cases the rule is right and the scope it was evaluated against is wrong, which is why both are defended the same way: bind the scope at the same moment you acquire the resource — set the variable in the code that takes the connection, build the cache key from the object that carries the tenant — so the two cannot come apart.

Migrating from an application predicate to any of these is the awkward part, because it usually means re-writing the whole index. The path that avoids a flag day is to build the new layout alongside the old one, dual-write during ingest, run the retrieval path against both for a period while comparing result sets query by query, and cut over per tenant starting with the smallest. Comparing result sets rather than trusting the migration matters here, since a namespace whose vectors were partially copied returns short lists rather than errors, and by now that symptom should look familiar — it is the last branch of the previous chapter's triage tree, where an exhaustive scan of the eligible set also comes back short.

What turns either of these from a layout into a boundary is the credential. If the token the retrieval service uses to reach the store is scoped to one tenant's namespace or partition, a query that omits the tenant does not return the whole index; it fails. That inverts the failure mode of every forgotten code path, because a nightly re-embedding job that forgets the predicate now throws a permission error on its first batch instead of quietly reading everything, and a copied snippet in a related-documents panel breaks in staging.

A proof is a probe that fails#

The artefact to hand a security reviewer is an executable attempt at the attack, run against the lowest layer available, asserting that the store itself refuses. Tests that go through the repository prove the repository works, which was never in doubt.

@pytest.mark.parametrize("victim,attacker", CROSS_TENANT_PAIRS)
def test_store_refuses_cross_tenant_read(victim, attacker):
    doc = seed_document(tenant=victim, text="Project Harbour valuation memo")
    vec = embed("Project Harbour valuation memo")

    client = raw_store_client(credential_for(attacker))   # bypass the repo

    # 1. no predicate at all: must not surface the victim's document
    hits = client.query(vector=vec, top_k=50)
    assert doc.id not in {h.id for h in hits}

    # 2. explicit predicate naming the victim: must be denied, not filtered
    with pytest.raises(PermissionDenied):
        client.query(vector=vec, top_k=50,
                     filter={"tenant_id": {"$eq": victim}})

    # 3. direct fetch by id: must not resolve
    assert client.fetch([doc.id]) == []

    # 4. the cache must not answer across the boundary
    warm_cache(tenant=victim, text="Project Harbour valuation memo")
    as_tenant(attacker)
    assert doc.id not in {h.id for h in search("Project Harbour valuation memo")}

Case two is the one that separates a boundary from a filter, and the distinction is in what it asserts rather than in what it sends. Both a boundary and a correctly-written filter cause the victim's document to be absent from the response. Only a boundary raises. If the attacker's credential can express a predicate naming another tenant and receive an empty result rather than an error, the isolation is still being computed by whoever assembled the predicate, and the test would pass identically on a system with no boundary at all. Asserting the exception is what makes the test capable of failing for the right reason. Case four exists because the cache incident above passes cases one through three on the day it leaks.

The same probe belongs in production as a synthetic canary — a scripted transaction the system runs against itself on a schedule, so that a broken property announces itself rather than waiting to be discovered — running every few minutes between two reserved tenants that exist only for this purpose. The schedule is the point. The properties it asserts are broken by configuration changes and cache deployments rather than by code that a CI run would see, so a check that only runs on merge is checking the wrong events. A canary that has been green for 400 days is the closest thing to an answer for a reviewer's question, and it is an ongoing claim rather than a snapshot.

Identity above the boundary, and friction below it#

Index-level isolation does not decide who the caller is. Authentication and authorisation establish which tenant a request belongs to, and if that resolution is wrong — a session token accepted for the wrong organisation, a tenant id read from a client-supplied header — then perfect index isolation faithfully serves the wrong tenant's data to a correctly-scoped query. Every claim in this chapter is conditional on the identity above it being sound, and that condition is not a technicality: the probe above seeds a document, mints a credential and asserts a refusal, and all three steps take the mapping from credential to tenant as given. A reviewer should be made to check that mapping separately.

It also makes some legitimate work harder, which is the honest cost. Global search across tenants for an internal support tool now requires an elevated credential and an audit trail. Corpus-wide analytics need a separate read path. Building a shared eval set that spans tenants — the stratified sample of the fourth chapter, which deliberately draws queries from many tenants — stops being a query and becomes a data-handling decision with a policy attached. Those frictions are the boundary working, and a team that keeps quietly minting a bypass credential to get around them has replaced the boundary with a convention again, in a place nobody reviews.

Everything in this topic converges on the same operational habit: the correctness of a filtered retrieval path is a measurement rather than a property of the code. The filter's order decides which question the index answers, its selectivity decides whether the index can answer it, the plan decides what the answer costs, and none of those three announce themselves in a status code.

Before the next multi-tenant retrieval feature ships, the numbers worth having in hand are the selectivity distribution of real traffic, the plan each band routes to, the filtered recall for each band and each significant tenant, the fill-ratio histogram from live queries, and a green cross-tenant probe. Building that instrumentation first makes the rest of the work — choosing plans, tuning budgets, deciding where the tenant boundary sits — a matter of reading numbers instead of arguing about defaults.