An empty result and an unknown permission set are different answers

The return type of a retrieval function is usually a list of passages, and an empty list is the value it returns when it has nothing to give. That single value is doing several jobs at once. It is returned when the search ran cleanly and found nothing similar, when the caller's grants excluded everything the search would have found, and when the grant resolution itself failed and the service never had a permission set to filter with. Those three situations call for different behaviour from every layer above, and an empty list tells none of them apart.

The consequences run in two directions. Downstream, every component treats an empty list as a fact about the world and acts accordingly — caches it, declines to retry it, scores it as a legitimate answer — which quietly converts a failure into a result. Outward, the response the caller receives has to be identical whether a document existed and was withheld or never existed at all, because any difference between those two responses is a channel through which someone can enumerate documents they cannot read.

Three situations, one empty list#

The retrieval function at the centre of this usually has the signature def retrieve(query, principal) -> list[Chunk], and that return type is where the information is lost. At least three distinct outcomes collapse into an empty list, and they call for different behaviour at every layer above.

Read the table down its columns rather than across. The middle two rows agree on every column except the last, which is why they may share a response and must not share a log line. The fourth row disagrees with all three others on every column after the first, which is why it cannot be a value of the same type at all.

The fourth row is the one chapter 3 arrived at, and the reason it needs its own outcome rather than an empty list is that everything downstream treats an empty list as a fact about the world. A caching layer will cache it. A retry policy will not retry it. An evaluation harness will score it as a legitimate "no answer" and the retrieval-recall metric will quietly improve, because a query that returns nothing cannot return anything wrong.

That last consequence deserves a moment, because it is the one that makes the defect self-concealing. A metric that counts wrong answers improves when answers stop being produced. So a degradation that suppresses results moves the dashboard in the direction that means "better", and the worse the degradation, the better the number looks. None of that happens if the unknown case leaves the retrieval layer as an error rather than as data: a 503, meaning the service is temporarily unable to answer, is retried, is not cached, and is counted as a failure by everything that watches it. Encoding it as a distinct value costs one type and removes an entire class of silent degradation:

@dataclass(frozen=True)
class Retrieved:
    chunks: list[Chunk]          # may be empty: a real, permitted no-match
    grant_scope: int             # how many resources the caller may read
    resolved_at: datetime

class PermissionsUnavailable(Exception):
    """Grants could not be resolved. Never a 200."""

def retrieve(query: str, principal: Principal) -> Retrieved:
    grants = resolve_or_fail(principal)      # raises PermissionsUnavailable
    hits = index.query(vector=embed(query), top_k=8, filter=grants.predicate())
    return Retrieved(chunks=hits, grant_scope=grants.size, resolved_at=now())

The type is doing the work an assertion could not. Retrieved can only be constructed after resolve_or_fail returns, so a value of that type is itself evidence that grants were resolved; there is no path that produces a Retrieved with an unknown permission state. The failure leaves by a different exit entirely, which means a caller who forgets to handle it gets an exception rather than a plausible empty answer.

The two branches at the bottom of that diagram converge on the same response and diverge in the log, which is the shape the disclosure rule and the diagnostic requirement jointly force. The user is told one thing in both cases; the operator is told two different things.

The grant_scope field is there for the diagnostic that is otherwise impossible. A caller whose grants resolved to zero resources and a caller with 40,000 resources and an unlucky query both get an empty chunks list, and only the first is a symptom — of a misconfigured account, a broken group sync, or a probe. Recording the scope alongside the result makes "how many callers today had a scope of zero?" a query rather than an investigation.

Saying the same thing about documents that exist and documents that do not#

The disclosure rule is that the response must not vary with the existence of resources the caller cannot see. Web APIs have carried a version of this for years in the 404-versus-403 convention, where 404 means the thing was not found and 403 means it was found and you may not have it: returning 403 on a private repository tells an anonymous visitor that the repository exists, so services that care return 404 for both, and only reveal the difference to someone who already had a reason to know. The same rule applies to a retrieval endpoint, with one adjustment for the case where the caller has already been given a reference.

If a user clicks a citation the assistant produced, or follows a link a colleague shared, they know the document exists, and answering "not found" is a worse experience for no security benefit — a 403 with "you don't have access; request it from the owner" is both truthful and actionable, because the existence has already been disclosed by the reference itself. The distinction that decides it is whether the resource identifier came from the caller's own guess or from a channel the system already trusted them with. Search and question-answering are the guess case, so they get uniform "nothing found". Direct fetch by identifier is the reference case, and can be more specific once the identifier is verified as one the caller legitimately received.

That leaves the channels that carry the same information without any wording at all. These are side channels: properties of the response other than its content — how long it took, how big it was, how much it cost — that vary with data the caller was never shown and therefore report on it. The timing channel is only the most obvious. A response that reports how many documents were searched, a UI that shows "12 sources considered, 2 cited", a token-usage field in the response, or a rate limiter that charges per candidate retrieved all reveal something about the shape of the corpus the caller cannot see. Streaming makes latency easier to measure rather than harder, because time-to-first-token is a cleaner signal than total duration.

Here is the stumble, and it is a belief that a defence works. Making both paths return the same sentence feels like it closes the channel, and it is tempting because the sentence is the only part of the response anyone designed. What it achieves is holding one quantity constant — the response text — while the dangerous property, whether documents matching the query existed and were removed, continues to vary. Every other observable property of the response is free to move with it, and the work the engine did to fill a top-k it could not fill is exactly the sort of thing that moves.

The column that decides how much work each row deserves is the last one. Three rows can be closed completely, because the signal is a field the response chose to include and removing the field removes the channel. Two rows are timing, which can only be raised in cost. Reading the table as a set of gaps between "mitigation" and "residual" is what tells you where effort is worth spending.

Padding latency is the mitigation that engineers resist, and it deserves an honest framing rather than a recommendation. You cannot close a timing channel completely in a system whose work depends on data, and a fixed floor of 400 ms on empty responses costs every legitimate no-match user 230 ms. That figure is the floor minus the time a genuine no-match already takes, and it is worth noticing which of the two empty-response timings it is measured against: the cost is borne by the fast path, the one where nothing existed, because the slow path was already closer to the floor. Padding is a tax on the honest case by construction.

What the floor achieves is raising the number of samples an attacker needs from a couple of dozen to something large enough to be visible in the logs, which is why the second half of the mitigation is detection: a principal generating a high rate of empty results, especially with rare proper nouns, is a probe signature. One company ended up alerting on more than 30 zero-result queries from one principal in ten minutes, which fired four times in the first month and caught two red-team runs and two genuinely confused users. Those two categories account for all four firings with nothing left over, which is what a workable threshold looks like: it is not that the alert never fires on legitimate behaviour, it is that when it does the operator learns something either way.

An empty context does not stop the model#

The last property of zero passages is that it does not, by itself, prevent an answer. A generator handed an empty context block and a question about parental leave will answer from what it learned in pretraining — the general corpus it was trained on long before it saw a line of your customer's documents — and the answer will be fluent, plausible and specific — twelve weeks fully paid, an additional four weeks unpaid, notification eight weeks in advance — because that is what a parental leave policy usually says. None of it came from the customer's documents, and the user has no way to tell.

In a permission-aware system this failure is worse than ordinary fabrication, because it is indistinguishable from the thing you were trying to prevent. A user who was correctly denied access to the real policy receives a confident description of a policy, reports that the assistant leaked a document they should not have seen, and the ensuing investigation finds no unauthorized retrieval anywhere in the trace — which is true, and takes a day to establish.

The instruction-level defense is necessary and insufficient. "If the context is empty, say you don't know" works most of the time and fails under pressure from a user who insists, from a long conversation where earlier turns supplied plausible-looking context, or from a system prompt that also says to be helpful. Those three failures share a cause worth stating: an instruction is one influence among several on a distribution over outputs, so it competes with the others rather than overriding them, and the cases where it loses are the cases where something else in the context pushes harder. The structural defense is to not call the model at all:

result = retrieve(query, principal)          # raises -> 503

if not result.chunks:
    return Answer(
        text="I couldn't find anything on that.",
        citations=[],
        generated=False,                      # no model call was made
    )

return generate(query, result.chunks, principal)

Three lines, and they remove the entire class. They work because they change the kind of guarantee on offer: an instruction asks a model to behave a certain way given an input, while an early return means the input never reaches the model. The first can be argued with and the second cannot.

The general-knowledge route, and the channel that stays open#

Separating the outcomes fixes what the system knows about itself; it does not decide what the product should do with a question the corpus cannot answer. The early return above costs the assistant the genuinely useful ability to answer a general question when nothing is retrievable — "what does a term sheet usually contain?" — which some products want. If you want that, the way to have it without reopening the hole is to route it explicitly: a separate mode, a different system prompt, an answer visibly labelled as general knowledge rather than sourced, and no citations attached. What breaks systems is the implicit fallback, where the same code path silently changes what it is doing based on whether retrieval happened to return anything.

The timing channel is the limit that cannot be closed, only priced. A system whose work depends on data will always take longer when there is more data to work on, and the floor raises the sample count an attacker needs rather than removing the signal. The residual column in the table above says so plainly for the two timing rows, and the honest way to record this is as a control with a stated strength — so many samples before the difference is separable, with an alert on the rate of attempts — rather than as a channel that has been closed.

This is also the behaviour that makes cross-tenant testing subtle, and it is worth flagging before the tests arrive later in this topic. A canary test that asserts tenant B's secret string does not appear in tenant A's answer will pass against a system that leaks, if the model paraphrases; and a test that asserts the assistant cannot answer B's question from A's account will fail spuriously if the model answers correctly from pretraining. Both are reasons the assertions belong on retrieved identifiers and traces rather than on answer prose.

Keeping these outcomes apart assumes the grant resolution is cheap and current, which held in every example so far because grants were resolved at query time from an authoritative store. A large class of production systems does the opposite, copying the answer into the index at ingest time so that no resolution is needed at all. That choice removes the resolution step this chapter has taken for granted, and it moves the entire cost onto the moment a grant changes — which is what makes the next chapter's revocation problem unfixable by tuning.