In a shared index, filtering after the search costs you the top-k

The complaint from the field was that the agent had become forgetful, and only for small customers. A dental practice with two users said it kept re-asking things it had been told the week before; the large logistics accounts reported nothing. The memory store was correct — the practice's records were all present, all carrying the right namespace — and the retrieval endpoint returned results without error. What it returned, for a query where ten memories should have surfaced, was two.

The measurement that explained it took an hour. Retrieval was asked for limit=10 against a shared index of 12.1 million vectors covering 340 organisations, and the query ran as a similarity search with a tenant filter applied to the result set. A search of that kind returns the top-k, meaning the k records whose vectors sit closest to the query's, where k is however many the caller asked for and was ten here. Recall — the fraction of the records that should have been returned for a query that actually were — was measured against a hand-labelled set of 200 queries and came out at 0.94 for the four largest tenants and 0.31 for the median tenant, which held about 4,200 vectors — roughly 0.035 percent of the index. The filter was doing its job perfectly. It was doing it after the index had already chosen which ten vectors to return.

Two orderings that look identical from the outside#

The distinction is whether the tenant predicate constrains the search or trims its output, and the same client library can express both with almost the same code.

# Post-filter: the index picks 10 nearest neighbours from all 12.1M vectors,
# then the application discards the ones belonging to other tenants.
hits = index.query(vector=q, top_k=10)
mine = [h for h in hits if h.metadata["org_id"] == p.org_id]   # often 0-2 results

# Pre-filter: the predicate is part of the search, so the index only ever
# considers vectors that satisfy it and returns 10 of them.
hits = index.query(vector=q, top_k=10, filter={"org_id": {"$eq": p.org_id}})

The second form is one keyword different and produces a fundamentally different query. Post-filtering also hides inside patterns that do not look like filtering at all: a re-ranker that rescores the candidate list with a slower and more accurate model, and a deduplication pass that collapses near-identical text both operate on whatever the search returned, so if the tenant check lives downstream of either, the search itself was unconstrained.

The failure is quiet in exactly the way that makes it dangerous to a test suite. Nothing errors, no record crosses a boundary, and a fixture containing two tenants with fifty records each will pass, since at that scale a tenant holds half the index and post-filtering returns five of ten. The behaviour only diverges as the ratio between one tenant and the whole index grows, which means it appears in production and not before.

What the arithmetic predicts, and what it does not#

There is a small piece of arithmetic under this, and it is worth doing carefully, because doing it carelessly produces a number that contradicts the measurement and makes the whole diagnosis look wrong.

Write f for the tenant's share of the index — its record count divided by the total. Take a query and imagine, for the moment, that the tenant's own records are no more likely to be near it than anybody else's. Under that assumption each of the k nearest neighbours belongs to this tenant with probability f, independently, so the expected number of the tenant's own records among the top k is k · f. For the dental practice this is ten times thirty-five thousandths of a percent, which rounds to nothing: the expected number of its own records among any ten nearest neighbours is close to zero. Turned around, the same relation says how far you would have to over-fetch to recover k of your own: fetch N candidates and you expect N · f of them to be yours, so N has to be k divided by f. For the dental practice at 0.035 percent of the index, recovering ten of its own records in expectation requires fetching roughly 29,000 candidates, which no index will serve at interactive latency.

Now put that beside the measurement, because they disagree. The arithmetic says the median tenant should recover essentially none of its records, and the measurement says it recovered about a third of them. Both are correct, and the gap between them is the assumption in the middle. The k · f calculation compares the tenant's records against the wrong reference: it treats the query as if it were unrelated to the tenant, when in fact the query came out of that tenant's own conversation and its own records are about that tenant's own subject matter. Relevance and tenancy are strongly correlated in a memory store, so the tenant's vectors sit far closer to its own queries than a random draw would put them, and recall lands well above the independent-draw figure.

So the arithmetic is a floor rather than a prediction, and knowing which it is changes what you may conclude from it. What it licenses is the direction and the scaling: shortfall under post-filtering gets worse as f falls, and the over-fetch factor needed to compensate grows as one over f, which is why over-fetching cannot be tuned into a fix. What it does not license is a predicted recall for a given tenant, and a team that computes k · f, gets a number near zero, and then observes a third of the records coming back will conclude the diagnosis was wrong when it was the reference class that was wrong. The relation is simple enough to reason about in your head once the number is on a dashboard: a tenant holding a fraction f of the index will find, in expectation, k · f of its own records among the top k under post-filtering. That rule holds as stated for a query unrelated to that tenant, and understates the real figure for the queries a memory system actually issues. Any tenant below roughly 10 percent of the index is already losing a substantial part of what it asked for, and how much depends on how tightly its own content clusters.

That is also the answer to the most common accidental version, which is over-fetching — asking for top_k=100 and filtering down — which improves the symptom without fixing anything, because the required over-fetch factor scales with how small the tenant is. The smallest tenant on the platform sets the requirement. Any fixed over-fetch is a threshold that the next customer to sign up may fall below.

The vulnerability that post-filtering does create#

Losing recall is a quality problem. There are two ways it becomes a disclosure problem, and both are properties of how post-filtering works rather than bugs on top of it.

The first is that the foreign records are fetched. In post-filtering, the other tenants' vectors and their payloads are returned by the index into your process, and the payload is usually the memory text itself. If your retrieval span records the candidate list — and observability tooling for RAG records candidates by default, because that is the thing engineers want to see when relevance is wrong — you have written another tenant's memories into your trace store, where they are searchable by anyone with access to the traces and subject to whatever retention that store has. Nothing leaked to a user, and a copy of tenant B's data now exists in a system that has never been reasoned about as tenant B's data.

This one is worth separating from the recall problem, because it does not depend on the recall problem at all. A large tenant with excellent recall is still receiving other tenants' payloads on every query — its filter simply discards more of them silently. Fixing the shortfall by over-fetching makes this half strictly worse, since the process now handles a hundred foreign payloads where it previously handled ten.

The second is the empty-result path. When post-filtering returns nothing, someone eventually adds a fallback: retry without the filter, or widen the search, or fall back to a keyword query over the same table. Each of those is written under the assumption that the filter was the problem, and at least one of them will be written by someone who does not know why the filter was there. A retrieval path whose degenerate case is "return fewer results than asked for" attracts exactly this kind of repair.

Four layouts, and what each actually costs#

Those figures come from the same 12.1-million-vector workload measured on one machine, with the p95 column reporting the latency that 95 percent of queries came in under, and the shape matters more than the digits: pre-filtering costs a few milliseconds over post-filtering and buys back two-thirds of the recall, while physical separation is faster still and pays for it in memory and operational surface. Read the table by the gaps between adjacent rows rather than by the level of any one row, because the levels are properties of this machine and the gaps are properties of the designs. The first-to-second gap is the largest movement anywhere in the table, obtained for the smallest price. The second-to-third gap buys a little more recall and takes latency down rather than up, because routing to one segment searches a smaller graph. The third-to-fourth gap buys the last point of recall and costs about 5 GB and 340 separate graphs to keep warm, which is why the last increment of isolation is the expensive one.

The middle row carries a caveat worth understanding, because "just pre-filter" is not free at the algorithm level. HNSW, which stands for hierarchical navigable small world and is the structure most vector indexes are built on, is a proximity graph: each vector is a node, each node holds links to a few of its nearest neighbours, and a search walks from node to node choosing whichever link leads closer to the query. A filtered search walks that same graph while discarding nodes that fail the predicate. When the predicate accepts one node in 2,900, whole regions of the graph become unreachable — the traversal arrives at a neighbourhood, finds no eligible neighbours to continue through, and terminates early. The search then returns short, which is the same symptom as post-filtering reached by a different route.

Engines handle this in different ways, and knowing which one you are running determines whether pre-filtering actually delivers the recall in that table. Qdrant builds additional links for filterable payload fields and estimates predicate cardinality to switch to a payload-index scan when the filter is selective enough. pgvector added iterative index scans in 0.8, which continue pulling from the index until enough rows satisfy the filter instead of returning short. Milvus and Pinecone offer a partition key and namespaces respectively, which is the third row: the tenant becomes part of the physical layout rather than a predicate evaluated during traversal. An engine with none of these will silently return fewer results under a selective pre-filter, which reproduces the original symptom through a different mechanism.

-- pgvector: the predicate is in the same statement as the ORDER BY, and
-- iterative scans keep the index from returning short under a selective filter.
SET LOCAL hnsw.ef_search = 100;
SET LOCAL hnsw.iterative_scan = 'relaxed_order';
SET LOCAL hnsw.max_scan_tuples = 40000;

SELECT id, text, embedding <=> :q AS distance
FROM memories
WHERE org_id = current_setting('app.current_org')   -- also enforced by RLS, chapter 3
  AND scope IN ('user', 'org')
ORDER BY embedding <=> :q
LIMIT 10;

Writing the predicate into the same statement as the ordering is what makes it a pre-filter; the row-level policy from chapter 3 sits underneath as the control that survives someone deleting the WHERE clause during a refactor. The two are not redundant, because they fail in different ways: the policy protects correctness of access, and the explicit predicate plus the scan settings protect recall.

Finding out which one you are running#

The question "does our filter run before or after the search" is answerable in production without reading the client library's source, and two signals settle it.

The first is a shortfall metric. Record, for every retrieval, the requested limit and the number of results actually returned, and chart the fraction where the second is smaller than the first, broken down by tenant size. Post-filtering produces a shortfall rate that rises monotonically as tenant share of the index falls, which is the scaling law from the arithmetic above showing up as a curve on a dashboard, and it is usually the earliest visible symptom of the whole problem. A correctly pre-filtered search over a small tenant returns a short list only when that tenant genuinely has fewer than k records, which is a flat rate uncorrelated with index size. The distinguishing feature is therefore the slope rather than the level, which is why the metric has to be broken down by tenant size to say anything at all.

The second is a paired probe. Take a query, run it with the tenant predicate and again without it, and compare the two candidate lists. Under pre-filtering the unfiltered run returns a mostly different set of documents while the filtered run still returns k of the tenant's own; under post-filtering the filtered run's results are a strict subset of the unfiltered run's, which is the definitive signature. The subset relation is the part that carries the proof: post-filtering can only remove items from a list the index already chose, so every survivor must appear in the unfiltered run, and a single filtered result absent from the unfiltered run rules post-filtering out entirely. This costs one extra query and can be run as a scheduled canary — a small probe that runs continuously in production so that a regression announces itself rather than waiting for a customer to notice — against a synthetic tenant seeded with a known number of records, which is the same fixture the next chapter builds on.

Both signals need the tenant's share of the index alongside them to be interpretable, so it is worth exporting a per-tenant record count as a gauge.

Choosing, and what the choice is not#

Per-tenant collections are the right default when tenants are few and large — a B2B product with 200 enterprise customers, where 14 GB of index is unremarkable and the isolation argument to a security reviewer takes one sentence. A shared index with a partition key is the right default when tenants number in the thousands or arrive continuously, because 8,000 HNSW graphs is an operational problem regardless of how safe it is, and most of them will be cold. Hybrid layouts are common and sensible: the largest twenty tenants get their own collections, everyone else shares an index partitioned by tenant, with a routing layer that knows which is which.

What none of these layouts is, is a substitute for the identity work in chapter 4. A per-tenant collection whose name is chosen from a tool argument is the confused deputy again with better latency. Physical separation removes the failure mode where a missing predicate returns everything; it does not remove the failure mode where the right predicate is applied to the wrong tenant.

Nor does index layout say anything about the write path. A pre-filtered search over an index into which the re-embedding job wrote records with a defaulted or missing tenant will faithfully return those records to whoever the default names, and the search was correct — the data was wrong before the query ran.

Both of those residuals have the same shape as the two the arithmetic left behind. Choosing a layout fixes which records a correctly identified, correctly written query can reach, and it decides nothing about whether the identity was right or whether the record was written with the right tenant on it. Isolation is a property of the whole system, and the only way to know you have it is to seed two tenants, ask every path for the other one's data, and assert that nothing comes back.