A green isolation test that never held anything worth leaking

An isolation test asserts that one tenant's query never returns another tenant's documents, where a tenant is one customer organisation among many served from a single deployment. It is the artefact a team points at during an access review, and it is usually the only automated evidence that the authorization filter in the retrieval path does anything at all. That makes it worth asking what an assertion of that shape can and cannot establish.

An assertion that something is absent is satisfied in more ways than one. It is satisfied when the defence removed the forbidden item. It is satisfied identically when the forbidden item was never a candidate, when the assertion looked somewhere the item would not have appeared anyway, and when nothing during the run could have moved it. Only the first of those four has anything to do with the code the test is named after, and a test written without attention to the other three will sit inside a pipeline for years, run on every commit, and be incapable of the one outcome it exists to produce.

What the original test asserted#

def test_tenant_isolation(client):
    seed_tenant("acme", ["acme_doc_1.txt", "acme_doc_2.txt"])
    seed_tenant("northwind", ["northwind_doc_1.txt"])

    resp = client.query(as_tenant="acme", text="what are our shipping rates?")

    assert "northwind" not in resp.answer.lower()

Six lines, and three separate decisions are buried in them. The first is what the fixture holds: two tenants, three files, whose contents are not visible from the test and were chosen by whoever wrote it. The second is what the assertion reads: resp.answer, the generated prose, lowercased. The third is invisible because it is an omission — nothing between the seeding and the query changes, so the test observes the system in exactly one permission state.

Each of those decisions is a place the test can be satisfied without the authorization filter having been consulted. That is the property to check first in any test of this shape, and the way to check it is to ask what defect would turn it red. If the answer is "none that we can name", the test is not weak evidence of isolation; it is no evidence, in the way a thermometer with a painted-on reading is no evidence of temperature.

The belief that makes this survive is that a green assertion is evidence because it is a mechanical check rather than an opinion. It is tempting for a real reason: the test does run, it does exercise the query path, and it does compare a string against a response the system genuinely produced. What makes it false is that its outcome is a quantity that stays green while the dangerous property varies completely: whether the authorization filter exists in the codebase at all can change without the test changing colour. An indicator that cannot move when the hazard moves is measuring something other than the hazard.

Three ways this stays green while the system leaks#

The fixture is the first and least visible failure. northwind_doc_1.txt contained three paragraphs of placeholder text about a fictional widget catalogue, written by whoever set the suite up so that the file would not be empty. Acme's query was about shipping rates, and the cosine similarity between that query and a widget catalogue — the usual measure of how close two embeddings sit, running from 0 for unrelated text to 1 for identical text — was 0.11, well below anything that would enter a top-8.

Follow that number through to the assertion, because the chain is short and each link is checkable. Placeholder text was chosen for convenience, so its subject has nothing to do with shipping rates. Topical distance is what an embedding measures, so the document sits far from the query in the space the search traverses. A retrieval call that returns the eight nearest chunks therefore never reaches it, since a score of about a tenth is nowhere near the eighth-best score in a corpus that contains documents actually about the query. A document that is not in the candidate set is never presented to the authorization predicate, so the predicate is never evaluated. And an assertion that no forbidden document appears in the answer is satisfied by a run in which no forbidden document was ever a candidate. The test passed because the retrieval was not interested in Northwind's document, and it would have passed identically with the authorization filter deleted from the codebase. What the test measured was the embedding model's sense of topical relevance, presented as a security property.

The observation point is the second. The assertion reads resp.answer, which is the generated text, and a leak reaches that string only if the model quotes the tenant name verbatim. A model handed Northwind's rate card among its passages will often summarise the numbers without naming the source, so the answer contains the leaked content and none of the leaked identifier. Chapter 1's leak surfaces are entirely invisible from here as well, since a candidate set that passed through the trace, the log and the reranker before being filtered leaves resp.answer untouched.

The fixture's immobility is the third. Both tenants are seeded in a fixture that runs once, and no grant changes during the test, no document moves between folders, no user is removed from a group, and no second request runs concurrently with a write. Every failure mode chapter 5 and chapter 6 described lives in the transitions, so a suite that only ever observes a static permission state has excluded the entire category by construction. The Northwind bug was in that category: a document had been re-shared into a folder whose resource_id mapping was updated asynchronously, and for a window the chunk pointed at a resource whose grants included Acme.

A fourth mechanism is worth naming because it defeats otherwise well-built suites. The test index held three documents; production held 4.1 million. Several vector engines choose their filtering strategy by estimated selectivity, taking an in-traversal path when the predicate is selective and an over-fetch-then-filter path when it is not, so a three-document index exercises a code path that production never takes. The step to see is that selectivity is a ratio, not a count. An engine estimating what fraction of the index survives the predicate finds that most of a three-document fixture does, since one tenant holds two of the three files, and that almost none of a corpus of millions does, since one tenant there is a sliver. Those two estimates fall on opposite sides of any threshold the engine uses to choose a strategy, so the fixture does not merely under-exercise the production path — it reliably selects the other one. Isolation tests that run against toy indexes can be correct about a strategy the system does not use.

Those four defeats are worth reading as a partition rather than as a list, because they sit at four different layers and are independent of one another. The first is about what was stored, the second about what was read, the third about what changed during the run, and the fourth about what the engine did with the predicate. Independence is the uncomfortable part: fixing the assertion leaves the fixture inert, fixing the fixture leaves the suite blind to transitions, and fixing all three still leaves a toy index selecting a strategy production never uses. There is no single repair, which is why what replaces the original test is a set of properties rather than a better assertion.

Rebuilding it so that it can fail#

The property a security test needs, and that ordinary tests get for free, is that there exists a plausible defect which turns it red. That is what the original lacked, and it is the thing to design for first rather than to check afterwards.

Start with canaries: documents seeded into each tenant carrying a unique token that appears nowhere else in the corpus, the codebase or the internet, and that is greppable, meaning a plain literal string a text search finds without needing to understand the format it is embedded in — CANARY-NORTHWIND-7f3a91c4 is fine, and the format matters only in that a regex must be able to find it in a JSON blob, a log line, a span attribute and a stream of model output. Give the token its own field and also embed it in the prose, so it survives both structured extraction and paraphrase.

Uniqueness is what makes the canary an instrument rather than a label. A search for the string has no false positives anywhere in the system, so a single hit is proof of presence and needs no interpretation, and that property is what lets the assertion be applied to surfaces whose format the test does not model — a vendor's outbound request body, a span attribute, a cache entry. Embedding it in two places buys the two ways it can travel: the field survives anything that copies structure, and the prose survives anything that reads and rewrites text.

Then make the fixture adversarial rather than incidental. The reason the original could not fail is that Northwind's document had no reason to rank for Acme's query, so the fixture has to be built the other way around: for each query in the test set, the other tenant must hold a document that is a near neighbour of that query in embedding space. The reliable way to construct that is to copy the querying tenant's own highest-ranking document into the other tenant, with the canary token added, so that the two are near-duplicates and the ranking actively prefers the one the caller may not read. If the authorization filter is removed, that document is the top hit; if the filter works, it is absent. That is the property that makes the assertion meaningful.

That construction is the earlier causal chain run backwards, link for link. The forbidden document is a near-duplicate of a document that legitimately wins, so it sits close to the query in the same space. Sitting close to the query means it necessarily enters the candidate window rather than falling outside it. Entering the candidate window means the authorization predicate is evaluated against it. And once the predicate is the only thing standing between the document and the response, the assertion is a statement about the predicate.

Worth separating in that paragraph is the requirement from the decision. The requirement is that a plausible defect must turn the test red, and it would be met by any fixture whose forbidden document ranks. The decision is to build the fixture from the query set — deriving each planted document from the query it must attract — rather than authoring documents independently and hoping they rank. That decision is what makes the property hold by construction instead of by inspection, and it is what stops the fixture decaying when the embedding model is upgraded and yesterday's near neighbour is no longer near.

Assert on identifiers and telemetry, never on prose. The retrieved chunk ids, the citation list, the captured spans, the outbound reranker payload, the cache entries written during the request and the raw model input are all places the canary can appear, and each of them is a place a real leak has appeared in some production system.

CANARY = "CANARY-NORTHWIND-7f3a91c4"

@pytest.mark.parametrize("q", NEAR_DUPLICATE_QUERIES)   # 50 paraphrases
def test_acme_cannot_reach_northwind(q, capture):
    with capture.telemetry() as tel:
        resp = client.query(as_tenant="acme", text=q)

    assert not any(c.doc_id.startswith("northwind/") for c in resp.retrieved)
    assert not any(c.doc_id.startswith("northwind/") for c in resp.citations)
    assert CANARY not in tel.spans_json
    assert CANARY not in tel.log_lines
    assert CANARY not in tel.outbound_http          # reranker, embedding, model
    assert CANARY not in tel.cache_writes
    assert CANARY not in resp.answer


def test_the_isolation_test_can_fail(monkeypatch):
    """Negative control: with the filter disabled, the suite must go red."""
    monkeypatch.setattr(retrieval, "grant_predicate", lambda p: None)
    with pytest.raises(AssertionError):
        test_acme_cannot_reach_northwind(NEAR_DUPLICATE_QUERIES[0], Capture())

The two assertions at the top do the security work, and the one at the bottom is where most suites start. Checking resp.retrieved and resp.citations catches a leak at the layer where it happens, before a reranker, a filter or a generator has had any chance to hide it; checking resp.answer catches only a leak that survived every one of those stages and then got quoted verbatim. The four assertions between them cover chapter 1's surfaces, and every one of the seven is written as a plain substring test because the canary's uniqueness makes a substring test sufficient.

The negative control is the part most suites omit and the part that would have caught the original. The term comes from laboratory work, where a control arm is run with the mechanism under test deliberately removed, and it earns its place because the run that is supposed to produce a result must actually produce one before the run that is supposed to produce nothing means anything. A test asserting the absence of something is only as good as your evidence that the thing could have been present, and disabling the enforcement and requiring the suite to go red is that evidence, produced automatically on every run. If someone later swaps the fixture for placeholder text, the negative control fails and the build breaks, which is the correct outcome.

The borrowed term is worth holding at arm's length, because the word negative names what was removed rather than what is expected, and a reader who takes it to mean the run that should stay quiet has the direction backwards. What is removed here is the grant predicate, which is the thing being protected rather than the thing being measured, so the expected outcome is that the detector fires and the build goes red. The name is the one in common engineering use and is worth keeping. The expectation a reader may import with it is not, and a red build from this particular test is the suite working.

Around that core, the cases that cover the transitions are ordinary parametrized tests once the machinery exists: revoke a grant mid-flight and query 100 ms later, re-share a document between tenants and assert the old tenant loses it within the SLO from chapter 6, run a query concurrently with an ACL update in both orders, delete a grant whose subject has an active session token, forge the tenant header from chapter 2, and issue a query as a subject whose grant set resolves to nothing and assert on the outcome type from chapter 4 rather than on an empty list.

Read the middle column as a set of gaps rather than as a ranking, because no two rows observe the same thing and dropping any one row leaves a blind spot no other row covers. The canary row is about reach, the fixture row about candidacy, the two assertion rows about layer, the paraphrase row about repeatability, the mutation row about time, the negative control about the instrument itself, and the scale row about the engine. That is the same independence the four failure modes had, which is why the table functions as a checklist and why a suite scoring seven out of eight is blind in exactly one nameable way rather than seven-eighths safe.

One thing the table does not supply is a strength for any of its rows. There is no measured leak rate here, no runtime for the suite, and no figure for how often the near-duplicate fixture actually surfaces the canary across the fifty paraphrases. The paraphrases exist because approximate-nearest-neighbour traversal is non-deterministic and one phrasing may miss what another catches; what fraction of them would catch a given defect is not stated, so the parameter is a hedge of unquantified strength rather than a coverage guarantee, and it should be described that way to anyone who asks what the suite proves.

Whether per-tenant indexes are the permanent answer#

Giving every tenant its own index is a real option as a permanent architecture rather than only as an emergency one, it carries real costs, and the decision turns on the shape of the tenant base rather than on a principle.

The cost row closes exactly, which is worth checking rather than taking on trust: the tenant count in the row label, multiplied by the per-index floor in the cell beside it, reproduces the stated total for empty indexes. What that closure licenses is the claim the section is built on — the fixed cost is knowable before any migration starts and rises in direct proportion to the tenant count, so it is a number a team can put in front of a decision rather than a risk they have to argue about. The small-tenant row is that same fact seen from the other side: the floor is charged per index whatever the index holds, which is why a 40-chunk tenant pays it in full.

The last row is the one that decides whether it counts as a fix for this class of bug. Choosing which index to query is itself an authorization decision made from the caller's identity, so a service that routed on a forged X-Tenant-Id header queries the wrong index just as happily as it built the wrong predicate. Per-tenant indexes move the enforcement point; they do not remove it, and chapter 2's failure survives the migration untouched. The licence that row grants is narrow: it says the architecture changes where a mistake is made and not whether one can be.

The intra-tenant row is the one that decides whether it is sufficient. In most document products the interesting permission model is inside the tenant — this folder is HR-only, that contract is visible to the deal team, this person left the project last month — and the tenant boundary is the easy half. A team that ships per-tenant indexes and treats permission-aware retrieval as solved has secured the boundary that customers assume and left the one they actually complain about.

Signing off on it as defense in depth is straightforward when the tenant count is in the hundreds and tenants are large, because the per-index floor is then a rounding error and the isolation is physical, provable and easy to explain to an auditor. Signing off on it as a replacement for retrieval-layer enforcement is not defensible, and signing off on it as a reason to leave the blind test in place is how the same incident recurs with a different mechanism.

What to do with all of this on Monday is narrower than the topic makes it sound. Find the enforcement point in your retrieval path and confirm the predicate is built from a verified claim; check whether anything logs, traces or caches the candidate set before the filter; open the one isolation test you have and try to make it fail by deleting the filter. Those three take an afternoon between them, and the answers determine which of the earlier chapters you need — the revocation arithmetic, the fail-closed pricing, the outcome types — rather than all of them at once.

The third of those is the one that generalises past this topic. A test that asserts an absence is only worth the evidence that the thing could have been present, and the cheapest way to obtain that evidence is to break the defence on purpose and watch what happens.