The framework's three-point win is usually a changed default

Retrieval-augmented generation is a pipeline: a chain that turns documents into chunks, chunks into vectors, a question into a ranked set of retrieved chunks, and those chunks into a rendered prompt for a language model. It can be written by hand in a few hundred lines against an embedding API and a vector store, or assembled from a framework — a maintained package that supplies the same stages as configurable components, with a default already chosen for every setting the author did not think to set. Both arrangements run the same stages in the same order. What differs is who chose the settings, and whether anybody wrote the choice down.

That difference is what makes a head-to-head comparison harder than it looks. When one build scores higher than another over a corpus of the same documents and the same evaluation questions, the useful question is not which is better but which of the several dozen settings that differ between them produced the gap. A framework's win arrives as a bundle — a splitter, a unit of size, a distance convention, a prompt template and a retrieval rule, all changed at once, none of them named in the branch description. Adopting the bundle without naming its parts means owning an improvement that cannot be tuned, cannot be defended in review, and will disappear silently the first time a dependency moves.

The case worked here is a reimplementation of a hand-written pipeline on a framework, over the same 412-document corpus, the same text-embedding-3-small, the same generation model at temperature 0, and top-k of 5, run against the same twenty-question eval. The hand-written version scores 16. Three questions it gets wrong, the framework version gets right; one question it gets right, the framework version gets wrong.

The comparison was run against the 16/20 build deliberately, before the table-aware chunker from the previous chapter merged, so that exactly one thing differed between the two runs: the implementation. Every other input was pinned to the same value, which is what makes the four disagreements worth investigating at all.

Calling that a comparison of implementations is where the reasoning usually goes wrong, and the step that gets skipped is short enough to write out. An implementation is not a thing that possesses a quality; it is a set of decisions, one for every setting the stages expose. Two implementations that differ are two different sets of decisions. A measured score gap is a single number, and a single number cannot be divided among a set of differences unless that set has one element in it. So the gap attributes to "the framework" only if the framework changed one thing — and a framework's entire value is that it changes many things so that nobody has to choose them. The gap is uninterpretable as it stands, and the work is to shrink the set of differences until each surviving one can be moved on its own and measured.

What has to be identical before a diff means anything#

Holding the inputs constant is more work than it sounds, because several of them are not parameters you pass but behaviours you inherit.

That table is a product rather than a checklist, and reading it as one changes how much slack it has in it. A measured difference is attributable to the implementations only if the corpus was the same, and the embedding model was the same, and the generation model was the same, and the retrieval depth was the same, and the query text was the same, and the scorer read the same field from both. Those six conditions multiply. Each is one when it holds and zero when it does not, and what the comparison is worth is their product. A single zero anywhere in the column takes the whole thing to zero, and the other five being immaculate buys back none of it, because there is no partial credit in a product containing a zero. That is why the pinning work is all front-loaded: a comparison is normally invalidated before it produces a number rather than after.

The fetch_k row is the one that most often invalidates a comparison before it starts, because the framework is not doing top-5 similarity at all: it is doing top-20 similarity followed by maximal marginal relevance down to 5, a selection rule that chooses each next chunk for a weighted mix of similarity to the query and dissimilarity to the chunks already chosen, so the five that come back are deliberately less alike than the five nearest ones would be. That is a different retrieval algorithm wearing the same parameter name. Confirming it takes one line in the framework's constructor signature.

Why that row is so easy to miss is worth naming, because it is this mistake in its strongest form. The defence against an unfair comparison is the sentence "both sides run top-k of five", and that sentence is true on both sides. It is equally true when one side returns the five nearest chunks and the other returns twenty and then selects five for a blend of closeness and mutual difference. A defence that reads identically whether the two retrievers are running the same algorithm or two different ones cannot tell you which case you are in, and a shared parameter name is exactly that kind of defence — it is shared vocabulary, and vocabulary is not behaviour. The thing that does move with the fault is the constructor signature, which is why reading it beats asserting on the number.

The judging row deserves the same suspicion, and it is the one that produced a false disagreement on this comparison before anything real was found. The hand-built pipeline returns a string and the eval checks whether the expected substring is in it; the framework returns a dictionary whose result key holds the answer and whose source_documents key holds the chunks, and the first version of the comparison harness stringified the whole dictionary. That made the check pass whenever the expected substring appeared anywhere in the retrieved text, which is a much weaker condition, and it credited the framework with a query it answers wrongly. Comparing two implementations means the scorer reads the same field from both, and the cheapest guard is to assert on the answer's length: a 4,000-character answer is a serialised dictionary.

The scorer failure has the same shape one layer out. A substring check exists to separate a correct answer from an incorrect one, and stringifying the returned dictionary makes it pass whenever the expected text appears anywhere in the retrieved chunks. Retrieval succeeding is not what was being measured; it was what was being assumed. So the check returned the same verdict on an answer that was right and an answer that was wrong, as long as retrieval had worked at all — a verdict that does not vary with the thing it exists to discriminate. The length assertion is crude and it works for the one reason that matters: an answer and a serialised dictionary differ in a quantity that does vary.

Once the inputs are pinned, the diff runs over four intermediates, and each has a form you can compare mechanically.

Getting the framework's intermediates out is the only part that takes ingenuity, and it is fifteen lines. Every framework has a way to enumerate what it stored and a hook that sees the final prompt, and if the hook is missing, wrapping the model client is always available.

# 1. dump what the framework actually indexed
records = store.get(include=["documents", "metadatas"])
with open("build/fw_chunks.jsonl", "w") as f:
    for doc, meta in zip(records["documents"], records["metadatas"]):
        f.write(json.dumps({"text": doc, "source": meta.get("source"),
                            "n_tokens": len(ENC.encode(doc))}) + "\n")

# 2. capture the exact string it sends, by wrapping the client
class Capturing:
    def __init__(self, inner): self.inner = inner
    def invoke(self, prompt, **kw):
        Path("build/fw_prompts").mkdir(exist_ok=True)
        Path(f"build/fw_prompts/{uuid4().hex}.txt").write_text(str(prompt))
        return self.inner.invoke(prompt, **kw)

chain.llm = Capturing(chain.llm)

Six defaults, and which of them moved the score#

Comparing the dumps took under an hour and produced a table with no mysteries left in it.

Read that table across rather than down. The absolute values in the middle two columns are settings, and settings are not interesting on their own; the gap between them is the finding, and each gap is a different kind. Two rows describe a difference in behaviour that can move a ranking, two describe a difference in how a quantity is written down, and two describe a difference in what text the model is given. Those three kinds want different evidence, and mixing them is how a comparison ends up arguing about chunk size when the disagreement is in the prompt.

Two of the rows can be checked against each other, and the check is more useful when it fails than when it passes. The unit row says the framework's thousand is a thousand characters, roughly two hundred and fifty tokens against the hand-built four hundred. If both splitters cut fixed windows, the chunk counts would stand in the inverse ratio of those sizes, so the framework would hold about half again as many chunks as the hand-built pipeline — more than that, since its overlap repeats text and adds chunks. It holds about a third more. The prediction overshoots, and the reason is sitting in the row above: a recursive splitter cuts at paragraph breaks before it cuts at offsets, so a short paragraph is emitted whole and a long one is windowed only when nothing else fits. Its chunks are paragraph-shaped rather than window-shaped and their lengths vary. The counts confirm from the outside what the splitter's configuration says from the inside, and two instruments agreeing about a mechanism is worth more than either of them stating it.

The distance row is one column written as the negation of the other. The framework reports one minus the cosine, and subtracting a quantity from a constant reverses order without disturbing it, so sorting ascending on the framework's number selects exactly the chunk that sorting descending on the hand-built number selects. Rankings are identical and only the logged figures are mirrored, which is what the consequence column says. That licenses a genuine reassurance — no eval question can move on this difference — and it should raise the thing that does move, which is any threshold carried across unchanged. A floor that used to admit everything at or above some similarity, applied without inversion to a distance, now admits everything at or below it, which is its own complement; a floor that rejects nothing is the failure the sixth chapter describes, arriving here by a different road.

Two of these were worth betting on before any of the numbers came in, and both paid.

The splitter is the first bet, because splitting on paragraph boundaries rather than at a fixed token offset is a genuinely different rule and it is the rule most likely to matter on a policy corpus where paragraphs correspond to complete statements. Confirming it took twelve minutes: port the recursive splitter into the hand-built pipeline, rebuild the index in 4 minutes 40 seconds, and rerun the eval. Two of the three wins moved over. The mechanism is the one from the previous chapter running in the favourable direction, since a paragraph-aligned boundary never leaves a sentence's subject in one chunk and its number in the next, and both of the recovered queries were exactly that failure.

The prompt template is the second bet, because it is the one intermediate whose contents can differ in ways that no parameter documents. Confirming it took ten minutes: take one captured prompt out of build/fw_prompts/, transplant its instruction text into the hand-built render, rerun. That accounted for the third win and, in the same run, for the single loss. The framework's just say that you don't know is a confidence instruction rather than a text-presence instruction, which is precisely the over-refusal that the fourth prompt variant in the fifth chapter was rejected for, and the query it loses is a runbook question the hand-built version answers from a chunk that plainly states the answer.

The hand-built pipeline with the recursive splitter ported and the framework's ordering adopted, keeping its own abstention wording, scores 19 of 20. The framework scores 18. Neither implementation is better than the other; the gap was two defaults, one of which was an improvement and one of which was a regression, and until they were named the branch looked like a uniform three-to-one win.

The subtraction under that sentence is worth doing slowly, because the cancellation is what licenses it, and because one of those two defaults turns out to carry an improvement and a regression at the same time. Twenty questions went to both builds. Sixteen of them the two builds answer the same way, so those sixteen contribute an identical amount to each score and drop out of the difference exactly — that exact cancellation is what a paired comparison buys, and it is only permitted because both sides saw the same questions against inputs pinned to the same values. What survives the subtraction is the four disagreements: three the framework wins, one it loses, a net of two, which is the whole distance from sixteen to eighteen.

Assigning those four leaves nothing over. Porting the splitter moved two of the three wins, so the splitter is worth two points. Transplanting the instruction text moved the third win and produced the single loss in the same run, so the template's ordering is worth one point and its abstention wording costs one. Two, plus one, minus one, is two; sixteen plus two is eighteen. Every point of a three-to-one win now has a named default behind it, and the residual is zero rather than small.

The best available combination falls out of the same accounting. Take the splitter and take the ordering and keep the hand-built abstention wording, and the score is sixteen plus two plus one, which is the nineteen the hand-built pipeline reaches once both improvements are ported. The framework's own eighteen sits one point below that, and the missing point is exactly what its abstention wording costs. So the claim that neither implementation is better than the other is not a diplomatic hedge. It is arithmetic with no remainder in it, and it says something stronger than a tie: either implementation can hold either score, because the score is a property of the defaults rather than of the code they sit in.

The metadata row deserves a note even though it did not decide anything here. Prepending source: policies/refunds-2024.pdf to the chunk text before embedding means the file path participates in similarity, which helps when paths are descriptive and hurts when they are docs/export_final_v3.pdf. On this corpus it moved the mean gold-chunk score by 0.011, which is inside the noise of the score distribution and not worth acting on, but it is the kind of default that quietly explains a gap on a corpus with better-named files.

Adoption without losing the instruments#

Deleting the implementation is a reasonable outcome. A maintained package brings loaders for file formats you have not written parsers for, batching and retry logic you would otherwise get wrong, and integrations with rerankers and hybrid search that are real work to build. What cannot be deleted is the apparatus this topic built, because none of it is a framework feature: the per-document ingest manifest, the per-query trace record with the rendered prompt in it, the eval set with its empty-context control, and the rule that comparisons run against a pinned index. Every one of those has to be reconstructed on top of the framework, using the same dumping hooks the diff used, and a migration that ships without them has traded a pipeline you can debug for one you cannot.

The second condition is that every default the diff surfaced becomes explicit configuration with a test behind it. A framework's defaults move between minor versions, and the failure mode is precise: a bump restores chunk_size to a different value or rewrites the built-in template, three eval queries regress, and nobody connects the two events because the change was in a lockfile.

def test_retrieval_defaults_have_not_moved():
    splitter = build_splitter()
    assert splitter._chunk_size == 1000 and splitter._chunk_overlap == 200
    assert splitter._separators == ["\n\n", "\n", " ", ""]
    assert retriever.search_kwargs == {"k": 5}
    assert retriever.search_type == "similarity"          # not "mmr"
    assert sha256(PROMPT.template.encode()).hexdigest()[:12] == "0f3b1d9ac224"

Asserting a hash of the prompt template is blunt and it is the assertion that would have caught the one real regression in this comparison. Pin the framework version too, so the test failing is a deliberate event during an upgrade rather than a surprise on a Tuesday.

That test is stronger than it looks in one respect and weaker in another, and the difference is worth holding. It is strong because it asserts on the value a setting holds rather than on the name it is exposed under, which is the failure the retrieval row demonstrated — search_type == "similarity" is a claim about behaviour where k == 5 was only a claim about vocabulary. It is weak because every line of it names an attribute somebody thought to check. A default that moves in a component with no attribute exposing it produces no test failure at all, and the test will keep passing while the behaviour changes underneath it.

The third thing worth keeping is the hand-built pipeline itself, not as production code but as a differential harness. It is about 140 lines, has no dependency beyond numpy and a client library, and running both implementations over the same twenty questions on a schedule turns any future divergence into a signal: when the framework's answers stop matching the reference on queries where they used to agree, something in the dependency moved. That is a stronger check than the defaults test, because it catches changes to behaviour that no attribute exposes, and it costs about a minute of CI a week.

Weighing the dependency itself belongs in the same decision and is usually left out of it. The framework and its transitive packages come to roughly 240 MB installed against 60 MB for numpy plus a client, import time goes from 0.3 to 1.9 seconds, which matters if any of this runs in a function with cold starts, and the upgrade cadence means a pinned version will be several minors behind within a quarter. None of that argues against adopting it — the loaders alone are worth more than the disk — and all of it belongs in the same document as the eval numbers, because the team is choosing an operational commitment rather than a score.

The reasoning to reject is the one that skips all of this: it wins three to one on the same corpus with the same model, so their retriever is better tuned, adopt it and move on. That reads a defaults difference as a quality difference, and it leaves the team owning an improvement it cannot explain, cannot tune, and cannot defend. The three queries are not held by anything — no configuration line records why the splitter is what it is, no test fails when it changes — so the improvement survives exactly until the next dependency bump.

Where the method stops working#

The stage-by-stage diff depends on determinism, and it holds only while both sides are pinned to the same model versions at temperature 0. Comparing across generation models, or across a provider's silent version rollout, breaks the assumption that a single run per query is evidence, and the honest version there needs repeated runs and a difference large enough to survive the variance — on a twenty-question set, a one-query difference between two models is not measurable at all, which is a separate reason twenty questions is a floor rather than an instrument.

The exact cancellation earlier in this chapter is the first thing to go when that assumption fails. Sixteen questions dropped out of the subtraction because both builds answered them identically, and they answered them identically because every input was pinned and nothing in either run was sampled. Once a provider is free to serve a different model version to the two runs, those sixteen stop being a shared term and start being sixteen more places the two scores can differ for no reason anybody can name. The residual then stops being zero, and a residual of one or two questions on a set of twenty is indistinguishable from the effect being measured.

It also says nothing about the things that decide framework adoption in practice: how the dependency behaves in your build, how large it is, how often it breaks on upgrade, whether its abstractions survive the day you need a reranker between retrieval and rendering, and whether its evaluation hooks give you the trace record or a summarised version of it. Those are worth recording alongside the eval numbers, because they are what the team will actually live with.

A framework is a set of defaults with a package around it, and a comparison against one is a comparison of two settings tables. The three-to-one win in this chapter survived being taken apart, and what survived was not a better retriever: it was a splitter worth two points, an ordering worth one, and an abstention rule that cost one, each of which either implementation can hold. That is the whole of the finding, and it is only available to somebody who dumped four intermediates and read them.

What the six inspectable boundaries have produced is a set of habits that transfer to any implementation: a corpus that reports its own gaps, one durable record per query with the exact prompt in it, an eval whose score responds to retrieval rather than to the model's weights, and a rule that no change ships without the whole set rerun against a pinned index. Take those into whatever you build the pipeline on next. The framework will hide the stages again, and you will know which four artifacts to pull back out.