Ninety structured records are a lookup, not a search problem

Two systems can meet the same requirement — answer a person's question out of an organisation's own documents — and fail in completely unrelated ways. One of them holds each document whole and picks the right one by name. The other cuts every document into passages, turns each passage into a list of a few hundred numbers called an embedding, and at question time returns the passages whose numbers sit closest to the question's numbers. That second arrangement is retrieval-augmented generation, and the search inside it is an approximate nearest-neighbour search — approximate because it gives up the guarantee of finding the genuinely closest passages in exchange for being able to do it over millions of them. Closeness is reported as cosine similarity, the cosine of the angle between two of those lists, which reads one when they point in exactly the same direction and zero when they share no measured signal.

Which of the two to build is a design decision, and a requirement almost never names it. "An employee must be able to ask how long the waiting period is on parental leave" is satisfied by either. The habit is to reach for retrieval, because a chunker and a vector index are what an assistant looks like from the outside, and habit is a poor way to pick between two designs whose failure surfaces have nothing in common. There are two edges where the habit is wrong — one below, where the set is small and structured enough that a key settles the answer, and one above, where everything fits in the prompt — and the arithmetic that locates both of them takes an afternoon.

A key that already determines the answer#

An internal benefits tool answers questions about 90 employee benefit policies. Each policy is a short structured record — eligibility, waiting period, coverage limits, exclusions — edited by HR through a form, stored in a table, averaging 400 words. The team's plan was to chunk the 90 records, embed them, and stand up a vector index, because that is what building an assistant looks like.

Four properties of that data decide the design, and each one is checkable by looking at a single record. The records share a schema, so every policy has the same fields in the same places. A question names a field and a value: waiting period, on parental leave, for an employee. A query that names a field and a value has an exact answer set — the records that match, and no others — which is a property of the data model rather than an opinion about it. An exact answer set over 90 records is computable by reading all 90 and keeping the matches, which at this size costs nothing. Those four links compose into a conclusion the team's plan contradicts: nothing in the chain needs the records to be ordered by how similar they are to anything, so ranking is answering a question nobody asked, and chunking, embedding and approximate nearest-neighbour indexing are the machinery for producing a ranking.

The first thing that arrangement does is take a question whose answer is determined by a key and route it through an approximate nearest-neighbour search. "Parental Leave (Employee)" and "Parental Leave (Contractor)" are two records with nearly identical vocabulary, different waiting periods and different coverage; their embeddings sit at a cosine similarity around 0.94, which is well inside the band where retrieval order is decided by phrasing accidents. When the search returns the contractor policy for an employee's question, nothing in the system registers an anomaly. The score is high, the passage is topically perfect, and the model produces a fluent answer stating a waiting period that is wrong by eight weeks, on a question where the user needed an exact number.

The tempting belief to name here is that the similarity score is a safety net — that a wrong record would show up as a lower number somebody could threshold on. It does not, and the reason is the strongest form this kind of mistake takes: the score barely moves across the distinction it is being asked to make. The employee policy and the contractor policy sit at 0.94 to each other, so a question phrased for one of them scores against both at values that are close together and both high. A quantity that reads the same when the right record is returned and when the wrong one is returned carries no information about which happened. It is a defence that is constant while the thing it must discriminate varies, and a threshold placed on it cannot separate the two cases at any setting, because there is no setting at which one passes and the other fails.

The same conclusion falls out of treating end-to-end correctness as a product rather than a list. A right answer requires three things at once: the record exists in the system, the query selects that record rather than a neighbour, and the answer is derived from the record's contents. Those three are multiplied rather than added, since each is a chance of not being the step that ruins the answer, and a zero anywhere makes the whole product zero however good the other two are. A design that reads the record out of a table by its identifier has nothing in the middle position at all: selection is settled by the data model, and its factor is one. A design that ranks by cosine over near-duplicate records inserts a factor strictly below one where the other design has none, and gains nothing elsewhere to pay for it. That is the entire argument against the vector index here, and it does not depend on how good the embedding model is.

Ninety records is not a search problem because the ambiguity in "how long is the waiting period for parental leave" is about which of two named policies applies, and that is a question with 90 possible answers that can be enumerated. Resolving it is classification against a known list, and a classifier can be wrong in ways you can inspect, log, count and confirm with the user. An ANN search over a corpus this small is doing an approximate version of a decision the data model already answers exactly.

Ninety names the model can choose between#

The design that follows from the chain above hands the model the list and asks it to choose, rather than handing it a ranking and asking it to trust one.

POLICIES = load_policies()          # {policy_id: {"name": ..., "fields": {...}}}
NAMES = {p["policy_id"]: p["name"] for p in POLICIES.values()}   # 90 entries


def answer_benefit_question(question: str, employee: Employee) -> Answer:
    resolved = client.chat.completions.create(
        model=GEN_MODEL,
        temperature=0,
        messages=[{"role": "user", "content": RESOLVE_PROMPT.format(
            question=question,
            employment_type=employee.employment_type,   # narrows 90 to ~40
            region=employee.region,
            policies="\n".join(f"{pid}: {name}" for pid, name in NAMES.items()),
        )}],
        response_format={"type": "json_object"},
    )
    pick = json.loads(resolved.choices[0].message.content)
    # {"policy_id": "PARLEAVE_EMP_EU", "confidence": "high", "alternatives": [...]}

    if pick["confidence"] != "high":
        return Answer.clarify(options=[NAMES[p] for p in pick["alternatives"]])

    record = POLICIES[pick["policy_id"]]
    return generate_from_record(question, record, employee)   # whole record, no chunking

The resolved record goes into the prompt entire, which removes the chunking stage and every failure that lives in it: no waiting period separated from the policy name it belongs to, no exclusion list truncated at a token boundary, no ranked mixture of two policies. The whole 400-word record is roughly 550 tokens, so even three candidate policies fit comfortably. The employee's own attributes narrow the candidate list before the model sees it, which is the sort of filtering that a vector search can only approximate through metadata predicates that someone has to remember to add.

The comment on the employment_type line is where that narrowing happens, and the word approximate in the sentence above is the load-bearing one. Employment type is a field on the record and on the employee, so filtering on it removes candidates by equality: 90 becomes about 40, a little under half the list gone, before the model reads a single policy name. Nothing in that step is approximate and nothing in it can rank a non-matching record first. A predicate bolted onto a vector query reaches the same effect only for as long as somebody remembers to attach it, and a query that is missing it fails silently — the search still returns results, still scores them highly, and is now ranking over exactly the candidates the filter existed to remove.

That 550 is worth putting next to the figure the retrieval design produces, because it also explains the running cost. Three whole policies together are still under the 2,400-token prompt a five-passage retrieval design assembles, and one policy alone is a little under a quarter of it. The lookup design sends the shorter prompt of the two, which is the direction the per-query prices in the table below run in, and what it sends is one complete document rather than five fragments of several.

Two properties of this design matter more than its simplicity. Ambiguity becomes explicit — the resolver can return low confidence with alternatives, and the tool asks which policy the user means, rather than silently ranking one above the other. And the answer is auditable in a way that a similarity score is not: the log records which policy identifier was selected, HR can be shown the mapping from question to policy, and a wrong answer is traceable to one resolution decision instead of to a ranking.

Those two properties are the honest statement of what the middle factor became, rather than a claim that it disappeared. A model asked to pick from a list can still pick wrong. What changed is the shape of the error: it is a choice among 90 named options, so it can be logged as an identifier, counted per policy, replayed against a corrected expectation, and — this is the part a score cannot do — declined, because the resolver has a value it can return that means I am not sure, and here are the candidates. A cosine of 0.94 has no such value. The resolver's factor is below one and measurable; the ranker's factor is below one and constant across the cases it must separate.

The same call from the other direction#

The lower edge is a body of documents small and structured enough that a key resolves the answer. There is an upper edge as well, and it moves every time a model provider ships a longer window.

An 800-page product manual — roughly 400,000 tokens, revised twice a year, serving 30 internal support staff — is the case where it bites. The 400,000-token figure is the interesting one, because a corpus that fits in one context window is a corpus where retrieval is a choice rather than a requirement. That was not a real option two years ago, when 8k and 32k windows meant the manual had to be cut apart to be used at all, and the entire ingest-to-index half of the pipeline existed to make that cutting survivable.

What a whole manual costs to read every time#

What it costs is the part that gets waved through, and the arithmetic is short enough to do in full. At a typical mid-tier price of $3 per million input tokens, 400,000 tokens is $1.20 per query before the model writes a word — four tenths of a million tokens at three dollars a million, and the two figures multiply cleanly with nothing left over. At 30 staff asking eight questions a day, that is 240 queries and $288 a day, around $6,300 a month, for a tool serving thirty people. Two hundred and forty queries at a dollar twenty is $288 exactly, with no third term in it, which is what makes the daily figure the one that cannot be argued with once its two inputs are agreed.

Prompt caching, where the provider keeps the processed form of a prompt prefix so that the next request carrying the identical prefix skips re-reading it, changes the arithmetic and introduces its own: cache reads commonly price around a tenth of base input, so a cached read of the manual is about $0.12, while writing the cache costs a premium over base, around $1.50 for this prefix. Whether the bill looks like $28 a day or $288 depends entirely on how often the cache is warm, which depends on the TTL — how long a cached prefix survives without being used — and on traffic shape. Those two daily figures stand in the same ratio as the two token prices, ten to one, and the reason is that the same 240 queries sit on both sides of the comparison and cancel out of it. Nothing about traffic volume separates the warm case from the cold one. The only quantity that differs is the price of a single read, which puts the whole spread between a manageable bill and an indefensible one inside a ratio the provider publishes.

Support questions arrive in bursts around shift starts and go quiet for an hour at a time, so with a five-minute TTL a plausible day has twenty or so cache writes and 240 reads, landing near $59 a day — about $1,300 a month, against roughly $43 a month for the retrieval pipeline at $0.006 a query. That sum closes: twenty writes at a dollar fifty is thirty dollars, two hundred and forty reads at twelve cents is a shade under twenty-nine, and the two components add to the stated total with no third term hiding in it. The one thing worth naming is that the reads column is the whole day's traffic rather than the day's traffic minus its twenty warmings, so the twenty queries that write the cache are being charged at both prices and the estimate errs high. The gap it errs by is small against the gap between $59 and $288, which is the comparison the figure exists to make.

Every monthly figure above is a daily one converted at some assumed number of trafficked days, and the assumption is not a measurement. Every one of those numbers is a guess until someone measures the actual inter-arrival distribution, and that measurement takes an afternoon of log analysis that nobody has done. The per-query column is the part that is safe to reason with, because each of its entries is a price multiplied by a token count and neither input depends on how the month is shaped.

Latency behaves the same way. A 400k-token prompt has a time to first token measured in tens of seconds when the cache is cold, and single-digit seconds when it is warm, against a few hundred milliseconds of retrieval plus a 2,400-token prompt in the pipeline design. For a support agent with a customer on the line, a cold-cache query is a different product.

The table is more useful read across each row for the gap between columns than down each column for a level. The cost row is a clean ladder: retrieval is three times the lookup, a warm full-context query is twenty times retrieval, and a cold one is ten times warm — each step an exact whole-number multiple, which is the signature of three different mechanisms rather than three points on one curve. The update row splits the outer two columns from the middle one: reading a source of truth live cannot be stale and re-sending the manual cannot be stale, while only the middle column runs a background reconciliation that can fall behind, which makes it the only entry in that row where staleness is possible at all. The failure row splits differently. Only the left column fails in a way that announces itself, since a resolver can ask back or write an identifier into a log, where an unretrieved passage and a buried answer both look exactly like a confident answer.

Two of those rows are near-negations of each other and the place they fail to cancel is informative. Retrieval fits when the corpus exceeds the window; full context fits when the corpus fits. Those two conditions would between them cover every corpus and leave the choice decided by size alone, except that the third column asks for headroom and the size row prices that headroom at under about half the window. The residual is real and it has a name: a corpus that fits in the window with no room left for the question, the instructions, the conversation so far and the answer. In that band neither stated condition holds cleanly, and the answer is retrieval, because the column that claims to handle a fitting corpus has already declined the ones that fit tightly.

The last row is a constraint that decides the question outright in some organisations. A single prompt containing the whole manual cannot be filtered per user, so the moment any part of the corpus is restricted, the design either sends restricted content to every user or maintains one cached prefix per permission configuration, which multiplies the cache-write cost by the number of distinct configurations. That is the zero factor from earlier arriving in a different column: per-user filtering is not expensive in the third design, it is absent, and no improvement to the other rows recovers it.

Where the argument turns round#

For the benefits tool, the condition that flips this back toward retrieval is a change in the shape of the questions rather than in the size of the corpus. If people start asking "which policies cover physiotherapy", the answer is spread across records and no single key resolves it, and if HR replaces the structured form with free-text policy documents of 30 pages each, the record no longer fits in a prompt alongside its neighbours. Both are worth watching for, and neither is a reason to build a vector index today against 90 records that a form still produces. The general test is whether a person who knew the data perfectly would answer the question by looking something up or by reading around; the first is a lookup and the second is a search.

Both flips are worth carrying away as thresholds to watch rather than as a rule to memorise, because the figures in them belong to this benefits tool and not to anyone else's data. The quantity to sample is the fraction of real questions that name one record against the fraction that require reading several. The quantity to watch on the records is whether the fields are still fields, since the moment HR is pasting prose into a text area the schema has stopped constraining the answer even though the table has not changed shape.

There are concrete conditions under which the manual team should keep the pipeline. If the labelled set shows the retrieval design beating the full-context design on pinpoint questions — the ones where the answer is a torque value on page 611 and the surrounding 399,000 tokens are distractors — then the earlier chapter's recall-versus-accuracy result, where feeding more retrieved passages raised the chance the gold passage was present and lowered answer accuracy anyway, is operating at extreme scale, and the pipeline wins on quality rather than on cost. If the corpus is going to grow, and one 800-page manual becomes twelve product manuals next year, the window that fits today does not fit then, and the pipeline is the design that survives the growth. If parts of the manual are restricted to certain support tiers, the per-user filter is only available in the retrieval design. And if p95 time to first token above four seconds is unacceptable, the cache has to be warm essentially always, which at 240 queries a day it will not be.

The pinpoint condition is worth reading as a ratio rather than as a pair of page references. Distractors and answer close on the whole manual with nothing unaccounted for — 399,000 tokens against a corpus of 400,000 — so the answer-bearing region is the thousand tokens left over, about one part in four hundred of what is sent. What the labelled set settles is whether a model's accuracy on that needle holds up at that dilution. That is a property of the model and of the question type, it changes with every model release, and it is not something a team can reason its way to from the outside.

The general shape here is that retrieval is a compression strategy for a context window that cannot hold everything, and the pressure on it comes from both directions. Below, a corpus small and structured enough that a key resolves the answer exactly. Above, a corpus small enough to fit in a window whose price is falling and whose length is growing. Retrieval is the right architecture in the wide middle where the corpus is too large for the window, too unstructured for a key, or too permission-segmented to send wholesale — which is where most real corpora live, and will keep living, because corpora grow faster than context windows have.

What remains is to build the thing over your own documents. The order that keeps every later decision measurable is the order this topic ran in: get the eight contracts right so versions, deletions and access labels survive every arrow; stand up the plain path over a small representative slice and keep it as the control; put spans on each stage before you need them; label two hundred real questions with their gold paragraphs — the bottom of the two-to-three-hundred range the labelled-set chapter argues for, and enough to resolve the differences anyone is arguing about; then change one thing at a time and read the paired difference. The first number that comes out of that sequence — the plain path's pass rate on a set of questions your users actually asked — is worth more than any architecture diagram you could draw beforehand, including the one at the start of this topic.