A baseline you can trace beats four optimisations you cannot attribute

A retrieval-augmented assistant answers a question from documents rather than from what a model absorbed in training. A search over a prepared store of chunks — passages of a few hundred words cut out of each document — picks the ones most likely to contain the answer, an application pastes them into a prompt, and a model writes the answer from them. Scoring is by cosine similarity, the measure of how closely two lists of numbers point in the same direction, and taking the highest-scoring top k means keeping the k best of those scores, with k a small number. That is the whole of the plain path, and it is small enough to write in one function.

Four additions are standard enough that most teams plan them before they have measured anything. Hybrid search combines dense retrieval over embeddings with BM25, the word-matching score behind conventional search engines. A cross-encoder reranker re-scores a shortlist by feeding the question and each candidate passage through a model together, which is more accurate than comparing two vectors and far more expensive. Query rewriting sends the user's question to a model first and searches with what comes back. Semantic chunking splits documents on topic boundaries instead of on token counts. Each is a real technique that has earned its place somewhere. The problem is what happens to a team's ability to know anything when all four arrive at once, and that problem is arithmetic rather than engineering.

One measured difference, four unknown contributions#

Nothing in a retrieval system reports the value of a component directly. There is no reading anywhere that says what a reranker is worth. What can be obtained is a score for a whole system on a fixed set of questions, and the only way to turn that into a statement about one component is to score two systems and subtract. The plain path answers some fraction of the question set correctly; a variant with one change answers some other fraction; the number that carries information is the difference between them, and neither of the two scores means much on its own.

That subtraction works for a reason worth stating rather than assuming, because everything else here depends on it. Both runs are scored on the same questions, against the same documents, by the same grader, with the same embedding model and the same generation model. Each of those contributes something to the score — an easy question set inflates both, a lenient grader inflates both, a corpus missing a topic depresses both — and each contribution enters the two scores identically. In the subtraction those identical contributions appear once with a plus and once with a minus, and they cancel exactly. The cancellation is permitted because the terms are literally the same quantity in the two runs rather than merely similar ones. A quantity minus itself is zero, with no residue.

What survives the cancellation is one-directional: the contribution of the single thing that differed between the runs, and nothing else. That is the whole reason a difference is interpretable while a level is not. It also says precisely when the method breaks. Swap in a fresh question set between the two runs and the shared term is no longer shared, so it does not cancel, and the residual now contains a question-set difference of unknown sign added to the effect being measured. Re-ingest the documents, upgrade the grader, or let a model version roll forward underneath, and the same thing happens. A sampling temperature above zero is the small version of the same fault: the shared term becomes random rather than fixed, so it differs between runs and leaves noise of unknown size in the residual. That is what temperature=0 in the function below is protecting. Reproducibility is not tidiness; it is the precondition that makes a subtraction a measurement rather than a sample.

Now change four things at once. The measurement is still one difference, because there is still only one variant to score. Written out, that difference equals the reranker's contribution, plus hybrid search's contribution, plus the rewriter's contribution, plus the chunker's contribution, plus whatever the four do to each other when combined. That is one equation with four unknowns in it before the interaction terms are counted, and a single equation in four unknowns has no unique solution — many different assignments of value to the four components produce exactly the observed total. Two of those assignments can disagree completely about whether a given component helped or hurt, and the measurement cannot tell them apart, because both satisfy it. The same holds for latency: one end-to-end p95 figure is consistent with a wide range of per-component costs, and it constrains only their sum.

The tempting belief this defeats is that a good end-to-end number vindicates whatever shipped with it. It does not, and the sharpest way to see why is that the end-to-end number is constant while the thing it is being used to decide varies. Hold the release fixed and vary the reranker's true contribution across the range the equation permits — the other three components absorb the difference, and the reported score does not move. An instrument whose reading stays put when the quantity you are asking about changes is not measuring that quantity. It is measuring the sum, which nobody asked about.

Counting the systems that were never built makes the cost concrete. Each of the four changes is on or off, so the number of distinct systems is the product of the per-component option counts: two settings for each of four independent components, multiplied together, which is sixteen. Reading it as a product is what makes the next observation obvious. Pin any one component at a fixed setting and that factor drops to one, which collapses the space along that axis and leaves the product over the components still in play. That is exactly what an evaluation harness does when it can force a component to its default, and it is why a harness that can only run one configuration is worth so much less than one that can pin components.

Four binary changes make sixteen configurations. Landing them at once samples one of the sixteen and provides no way to reach the other fifteen except by removing things from a system that now has users, which is a harder political act than not adding them. One sampled and fifteen unreachable account for the whole space with nothing left over, and knowing that before the sprint starts is the point: the number of systems the plan forecloses is calculable at planning time, from the plan alone, without measuring anything.

Every question a team actually wants answered — is the reranker worth its latency, does hybrid search fix the acronym queries — is a comparison between two configurations differing in one switch. The plain path and the all-four release differ in every switch at once, which is the single comparison in that space of sixteen containing no such pair.

As an analogy, and it is kinder to the situation than the situation deserves: changing four things and measuring once is a chemist tipping four reagents into one flask and recording a single colour. It breaks where the analogy is too generous. The chemist can run the four reagents separately afterwards, at the same cost as the first experiment, and recover what each contributed. Here two of the four changes are made at ingest time, so recovering their separate contributions means rebuilding an index; and the software version has users on it, which the flask does not.

The plain path is genuinely small#

The whole of a naive retrieval path is four operations, and writing it out is the fastest way to see how little there is between a question and an answer.

from openai import OpenAI

client = OpenAI()
EMBED_MODEL = "text-embedding-3-small"   # the same model that embedded the chunks

PROMPT = """Answer the question using only the passages below.
If the passages do not contain the answer, say you don't know.
Cite the passage number you used, like [2].

{context}

Question: {question}
Answer:"""


def answer(question: str, principal: Principal, k: int = 5) -> Answer:
    qvec = client.embeddings.create(model=EMBED_MODEL, input=question).data[0].embedding
    hits = retrieve(qvec, principal, k=k)          # filtered cosine search, ch. 3

    context = "\n\n".join(
        f"[{i + 1}] ({h.payload['source_url']})\n{h.payload['text']}"
        for i, h in enumerate(hits)
    )
    prompt = PROMPT.format(context=context, question=question)

    resp = client.chat.completions.create(
        model="claude-sonnet-4-6",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
    )
    return Answer(
        text=resp.choices[0].message.content,
        rendered_prompt=prompt,
        chunk_ids=[h.payload["chunk_id"] for h in hits],
        ranks={h.payload["chunk_id"]: i + 1 for i, h in enumerate(hits)},
        scores={h.payload["chunk_id"]: h.score for h in hits},
        usage=resp.usage,
    )

Embedding the question, taking the top k — the k highest-scoring chunks, with k a small number such as five — rendering them into a prompt and making one model call is the whole of it. The single most important constraint in that function is that EMBED_MODEL is the same constant the ingest job used. Query and document vectors must live in the same space or the cosine between them is arithmetic without meaning, and the failure is silent — you get ranked results, plausible-looking scores, and rankings uncorrelated with relevance. It is worth asserting on it: store model_id in the collection's metadata at build time and have the query path compare against it on startup rather than discovering the mismatch through a quality investigation six weeks later.

Using the same embedding model on both sides is a requirement, and it comes from the arithmetic rather than from anybody's judgement. A cosine compares two lists position by position, and the meaning of the number sitting in a given position is whatever the model that produced it made it mean. Two different models assign different meanings to the same position, so lining their outputs up and multiplying pairs produces a number with no referent. Nothing raises an error, because multiplying and summing floats is always legal, which is exactly why the check has to be an explicit assertion on a stored model_id rather than something the code path discovers on its own.

Everything else in the function is a design decision, and each could reasonably have gone the other way: how many passages to take, whether to make one model call or let the model search again, temperature=0, fixed-size chunks. The value of naming them as decisions is that a decision is a thing you can later vary and measure, whereas a requirement is not, and a team that cannot tell which of its settings are which will waste a sprint A/B-testing something the arithmetic already forbids.

The second constraint is that Answer returns four things beyond the text. The rendered prompt, the chunk identifiers, their ranks and their scores are what make an answer traceable, and they cost nothing to keep. Without them the only artefact of a failed question is a wrong string, and every subsequent conversation about it is speculation. With them, any wrong answer can be interrogated: was the right paragraph in the hit list, at what rank, and did it survive into the prompt.

Standing this up over 4,000 support articles is a day of work and costs very little. At an average of 1,100 words per article, fixed 512-token chunks with 64 tokens of overlap produce roughly 18,000 chunks and 7.4 million embedding tokens, which at text-embedding-3-small pricing is about $0.15 for the whole corpus and around eleven minutes wall-clock at 200 chunks per batch. The index is 18,000 × 1,536 floats, about 110 MB before quantisation, which fits in memory on any machine you already own. The recurring cost is one embedding call and one generation call per question; on 5-passage prompts averaging 2,400 input tokens, that is fractions of a cent per query.

Every figure in that paragraph is a product of things known before any code runs, which is why the sizing question can be settled on paper. The number of chunks is the corpus length in tokens divided by the stride, where the stride is the chunk window less the overlap — the overlap is why a corpus produces more chunk tokens than it contains, since each overlapping region is embedded twice. The embedding bill is the resulting token count multiplied by the model's published per-token price. And the index figure is the cleanest of the three: chunk count times dimensions times the bytes in a single-precision float lands exactly on the stated megabytes, with no residual and nothing hidden in it. What that closure licenses is a decision made in a planning meeting. Whether an index fits in the memory of a machine the team already owns is a multiplication of three numbers, and a team that does it before writing the ingest job never discovers halfway through that it needs a different machine.

Representative is a stronger requirement than small#

Four thousand articles is small enough to index whole, and many corpora are not. When the real corpus is five million documents, the temptation is to take a uniform 1% sample and build against that, which produces a slice with a specific defect: the answer to a question frequently lives in a document whose neighbours were not sampled, so the baseline manufactures corpus gaps that do not exist in production and every early failure looks like retrieval doing badly.

The mechanism there is worth following, because "sample less" sounds like a quantity decision and it is really a structural one. A question is answered by a specific document, not by the average document. Uniform sampling keeps each document independently with the sampling fraction, so the chance that the one document answering a given question survives is that same small fraction. For a question needing two documents together, both have to survive, and the chance of that is the fraction multiplied by itself — smaller again. Every question whose answering document was dropped now fails, and it fails looking exactly like a retrieval failure, because the observable symptom is identical: nothing relevant came back. The measurement is not merely noisy. It is biased in one direction, and the direction is towards blaming the retriever.

The slice that works is chosen by completeness rather than by randomness. Take every document in three or four whole categories — one product area, one region, one document type — so that within the slice, any question a user could ask about those categories has its answer present. The slice should also cover the format range, because a corpus of clean HTML articles plus scanned PDFs plus exported spreadsheets has three different parse failure modes, and a slice containing only the first tells you nothing about the other two. A hundred thousand chunks spanning three complete categories and all three formats is a better instrument than a random million.

Completeness works by driving that survival probability to one inside the chosen categories, which removes the manufactured-gap term from the measurement entirely rather than shrinking it. The requirement is that a failure observed in the slice be a failure that would also occur in production; which categories to take and which formats to include are decisions, and the requirement is what makes some of those decisions wrong.

The other requirement on the slice is that its questions are answerable by someone on the team, since the labelled set the following chapters depend on has to be produced by a human who can tell a right answer from a plausible one. Choosing a slice about a product area nobody in the room understands means the evaluation work stalls waiting on a domain expert's calendar, and the baseline sits unmeasured for three weeks, which is the same outcome as not having built it.

The plain path as the permanent control#

The instinct that the naive path is a throwaway prototype is the thing to resist. It is the control, and it stays in the repository, behind a flag, for as long as the system is being changed. Every subsequent claim — the reranker is worth its latency, hybrid search fixes the acronym queries, the semantic chunker improves answers about tables — is a claim about a difference, and a difference needs two systems evaluated on the same questions. The naive path is the second system, permanently.

It also does something the optimised version cannot: it tells you whether retrieval is the problem at all. A plain path over 4,000 articles that answers 71% of realistic questions correctly says the corpus contains the answers and the basic mechanism works, so the remaining 29% is a bounded engineering problem. A plain path that answers 22% correctly usually means something structural — the corpus does not contain the answers, or the chunks are fragments of tables with no headings, or the questions people actually ask are about combinations of documents rather than lookups in one — and no reranker addresses any of those. Finding that out in week one costs a day. Finding it out after four optimisations have been built on top costs the sprint and leaves the team arguing about which of their four components is broken.

Those two figures close on each other, and the closure is what licenses the word "bounded". The answered and unanswered shares are complements of one whole, accounting for every question in the set with no third category and no residual, so naming one names the other. That is why 71% supports a plan and 22% does not: in the first case the failures are a minority of a set whose majority already works, and that working majority is itself evidence that the mechanism and the documents are both sound. In the second the complement is more than three-quarters of the set, far too large to be a matter of ranking within a candidate list that was retrieved correctly. Both readings carry one condition worth demanding out loud — that the question set is representative of what users ask — and a set written by the team that built the retriever tends not to be.

The table below sets each proposed change against the evidence that has to exist before the change is worth building. The gold chunk named in its right-hand column is the passage a human has identified as the one that answers a given question.

Read the right-hand column downwards and the same instrument appears in every row: the rank of the gold chunk under plain dense retrieval. Four different decisions, one measurement, and it is a measurement only the plain path can take, which is the practical reason the control has to exist before any of the four is built.

The rows are not, however, a clean partition of where the gold chunk can be, and the place they fail to partition is the informative part. The hybrid row asks for failures ranked below 10; the rerank row asks for failures inside the top 50 and outside the top 5. Ranks six through nine satisfy both. A failure landing there qualifies as evidence for two different changes at once, which means the two are competing explanations of the same observation rather than additive ones. A team that counts qualifying failures per row and adds the counts will overestimate what building both is worth, and will be puzzled when the combined gain comes in under the sum.

The last row breaks the pattern in a different way, and this break has a consequence for how it can be tested. Its evidence is not about rank at all — the gold paragraph is retrieved, and the problem is what the chunk boundary did to it. That is because semantic chunking is an ingest-side change while the other three are query-side changes. Query-side changes can be evaluated by re-running the question set against an index that already exists. Changing the chunker changes every chunk identifier and every vector, so evaluating it means rebuilding the index first. It is the most expensive of the four to test, and its qualifying evidence has to be collected from the existing index before the rebuild, because after the rebuild the old ranks are gone.

Keeping the control runnable costs something and the cost is worth budgeting for explicitly. The mechanism that works is a single configuration object threaded through the answer function — retriever type, k, rerank on or off, rewrite on or off, chunk strategy — with the naive settings as the defaults, so that any combination can be run from the evaluation harness without a branch or a redeploy. What that costs is discipline at the seams: every new component has to be written as an optional stage with a pass-through, and the temptation to inline the reranker into the retrieval function has to be refused each time. The price of refusing it is perhaps an hour per component. The price of accepting it is that six months later, comparing against the plain path means reverting code.

That configuration object is what makes the pinning above physically possible. A component that can be switched off from the harness is a factor that can be set to one; a component inlined into the retrieval function is a factor permanently stuck at its shipped value, and every configuration that would have varied it is unreachable without a code change.

Each row of that table names a measurement the plain path can produce and the optimised system cannot, because once all four are in place there is no run in which the gold chunk's rank under plain dense retrieval is known. The right-hand column is also a filter on effort: on a corpus of well-formed prose articles with no product codes, the hybrid-search row usually has close to zero qualifying failures, and building it anyway is a week spent on a class of error the corpus does not contain.

The order in which these arrived historically is a mild argument for the same discipline. The 2020 paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks introduced exactly this shape — a dense retriever over a Wikipedia index feeding a generator, with no reranker, no rewriter and no hybrid stage — and it was the plainness that made the result legible, because the comparison was against a generator with no retrieval at all. Every component in the four-item list above was added later by someone measuring against a version without it.

Where the plain path stops#

It is a baseline, not a destination, and being clear about what it cannot do prevents the opposite mistake of defending it past its usefulness. Single-vector dense retrieval over fixed chunks is weak on queries containing rare exact strings, because an embedding of ERR_4013 carries little of that token's specificity. It has no mechanism for questions that require combining two documents, since the top-k are retrieved independently and nothing reconciles them. It cannot answer aggregate questions — how many policies mention X — because retrieval returns a sample rather than a count. And its recall degrades on questions phrased differently from the corpus's vocabulary, which is the gap query rewriting exists to close.

Those four limits are also the honest scope of the claim that the plain path is the right first build. It is the right first build for a system whose questions are answered by one passage each, phrased in something like the vocabulary of the documents. Where a team already knows its users ask aggregate or multi-document questions, the plain path will score badly and the low score will be correct rather than diagnostic, so the thing to measure is not accuracy but how many of the labelled questions are of a shape the architecture can answer at all.

Fixed-size chunking has a specific pathology worth naming while the baseline is being built, because it will show up in the failure analysis later: a 512-token window with no structural awareness splits tables from their headers and steps from their preconditions, so a chunk reading "3. Confirm the refund in the admin console" retrieves for refund questions and does not say which of four refund flows it belongs to. The overlap parameter is a blunt hedge against it. Knowing that the baseline has this behaviour is different from fixing it before you know how often it fires.

The one thing the plain path must not skip is the tracing, which is why Answer carries what it carries. Even so, what it carries is per-answer and after the fact. It says which chunks were retrieved for one question; it does not say that the embedding call took 40 ms and the generation took 2.8 s, or that p95 moved after Tuesday's release while p50 stayed flat. Turning "the assistant feels slow" into a named stage needs timing emitted per stage on a shared trace, which is the next thing to build.

The plain path is worth building first because of what a single measurement can and cannot be divided among. One score is one number, and one number cannot be apportioned across four components. Every difference the plain path makes measurable is a difference where everything except one component cancels exactly, leaving a residual that belongs to that component alone. Keeping it in the repository behind a flag, with a configuration object that can force any later addition back to its default, is what preserves that cancellation for as long as anyone still needs to know which change earned its latency.