Removing two anti-patterns at once needs the pair of numbers

The pull request description reads: "Adds cross-encoder reranking and reduces k from 20 to 5. Answer quality is much improved — see the three examples below. Context is now 640 tokens instead of 2,510, so this is also a cost win." Underneath are three side-by-side answers, and all three are better after the change.

Three examples chosen by the person who made the change are not evidence about the change. They are evidence that at least three queries improved, which is what you would expect from almost any perturbation of a retrieval system. The reviewer's problem is that two things moved at once, and the two are different kinds of thing. Reranking puts a second and more expensive model over the candidates the retriever already returned and reorders them; a cross-encoder is the usual choice because it reads the query and one candidate together and scores that pair, rather than comparing two vectors that were computed separately. k is how many candidates the retriever is asked for in the first place, and cutting the kept set to five leaves fifteen of the original twenty discarded.

Reordering changes which chunks sit at the top of the list. Truncating changes which chunks exist at all. Both changes sit inside a pipeline whose stages run in a fixed order — retrieve, rerank, truncate, assemble, generate — and they act in opposite directions on the two numbers that matter. If the pair moved in the intended direction the change is good. If precision rose while recall fell, the same three examples would still look better and the system would be answering fewer questions correctly, and a report carrying one number reads exactly the same in both cases.

One measurement, two unknowns#

Give each change a name for the amount it moves whichever metric is being read. Call r the effect of adding the reranker on its own, and t the effect of truncating on its own, each measured as the difference from the system as it stands today. A report that ships both changes and measures once produces a single number, the difference between the system before and the system after; call that m.

If the two effects simply added, the report would be asserting m = r + t. That is one equation containing two unknowns, and one equation in two unknowns has no unique solution. Every pair of values that sums to m fits it exactly as well as every other: the reranker doing all the work and truncation none of it, truncation doing all of it and the reranker none, or one of them doing more than m while the other pulls the total back down. The measurement cannot distinguish those, because the measurement is their sum and a sum has discarded its parts by the time it is written down.

The additive form is itself an assumption, and it is a design decision about the analysis rather than a property of the system. Two changes that touch the same stage can help each other or work against each other, so the honest form carries a third term: m = r + t + i, where i is the interaction — whatever the two changes do in combination that neither does alone. Written that way the single before-and-after measurement is one equation in three unknowns, which is worse than it looked.

Two further measurements close it. Run an arm with the reranker alone and r is read off directly. Run an arm with truncation alone and t is read off directly. Substituting both into m = r + t + i leaves one unknown, so i = m − r − t. The subtraction is permitted because all three measurements are differences against the same baseline, taken on the same frozen query set, which puts them on a common origin and makes them comparable term by term; against different baselines or different query sets the subtraction would be meaningless. That is the entire reason the grid below has four arms and not two.

When i comes out at zero the identity closes: the measured whole equals the sum of the parts with no residual left over. That licenses one specific claim — on this query set, the two changes are separable, so a decision about one can be taken without reference to the other. It licenses nothing outside that set. A zero interaction here does not exclude a large one on traffic nobody labelled, and the smaller the query set, the wider the range of true interactions that remain consistent with a measured zero.

Two recall numbers, because k changed#

The first thing that goes wrong in reports like this is the metric name. Recall@20 before and recall@5 after are not comparable, and putting them in the same column labelled "recall" is how a regression gets shipped. Separate the two things the pipeline does.

Candidate recall is measured at the retrieval depth: did the retriever offer the gold chunks at all, at whatever k it was asked for. Context recall is measured over what actually entered the prompt after ranking, truncation and assembly. The first is a property of the index and the query encoder, and it goes up when you widen the candidate set. The second is a property of the whole pipeline, and it is the one users experience, because a chunk that was retrieved and then discarded is as absent as a chunk that was never found.

Two consequences follow from where in the order each one is measured, and both are identities rather than findings. The first is that when nothing is discarded between retrieval and the prompt, the two numbers must be equal — the set being scored is the same set — so any system that keeps everything it retrieves will report candidate recall and context recall as one figure twice. The second is that truncation happens after retrieval has finished, which means it cannot move candidate recall at all. That term cancels exactly, and its cancelling is what makes candidate recall usable as a control: whatever moves it, truncation is not among the candidates.

For the change in question, candidate recall went from 0.94 at k=20 to 0.96 at k=50, since the candidate set widened to feed the reranker. Context recall went from 0.94 to 0.91. Reporting the first pair alone says the change improved retrieval; reporting the second pair alone says it degraded; both are true statements about different stages, and only having both explains what happened.

Run the grid, because on a fixture it is affordable#

Two binary changes make four arms, and there is no reason to test them one at a time when the fixture runs forty queries in three minutes. The full grid separates the effect of ordering from the effect of truncation, which the two-arm before-and-after cannot.

Arm C is the one the two-arm comparison never runs, and it carries the attribution. Reordering alone leaves context precision unchanged at 0.14, because precision over the whole context does not care what order the chunks are in; what it buys is that the truncation in arm D throws away the right chunks instead of arbitrary ones — arm B keeps five of twenty in the retriever's order and discards the other fifteen, while arm D keeps five of fifty in the reranker's order and discards the other forty-five. The precision gain belongs to the truncation, and the recall protection belongs to the reranker, and neither change is worth making without the other. Arm B is what the team would have shipped if they had taken the cheap half of the proposal.

The precision column reading 0.14 twice is worth separating into its two parts, because only one of them is forced. Context precision is the share of the chunks in the context that are relevant, and a share does not depend on the order in which the things being counted are listed. Reordering a fixed set therefore cannot move it, and to that extent the identity is guaranteed before anyone runs anything. Arm C does more than reorder, though: it widens the candidate pool to fifty and keeps the reranker's top twenty, so its context is not arm A's context and the sets being scored are different. The two arms landing on the same figure says something the order argument does not cover, which is that the wider pool delivered the same number of relevant chunks into a twenty-chunk context as the narrower one did.

Multiplying the share by the number of chunks kept turns each precision cell into a count of relevant chunks per query, and read that way the column tells the mechanism plainly. Arms A and C both hold a little under three relevant chunks. Arm D holds a little under three as well, in a context a quarter of the size, which is why its precision comes out at slightly more than four times arm A's — the numerator held and the denominator fell. Arm B holds about half as many, a little under one and a half, which is the same fact its context recall reports from the other side. The residual worth naming is that each of these is a mean over forty queries, and a mean can sit still while individual queries trade relevant chunks in and out.

Now put the four arms through i = m − r − t. On context precision, truncation alone is worth fifteen points, reordering alone is worth nothing at all, and the two together are worth forty-four. The sum of the parts is fifteen against a measured whole of forty-four, so the interaction is twenty-nine points — nearly two-thirds of the effect, and larger than either change contributes on its own. On context recall the same subtraction runs the other way. Truncation alone costs thirty-three points, reordering alone gains two, and the two together cost three. The sum of the parts is minus thirty-one against a measured minus three, so the interaction is plus twenty-eight points.

The reranker's real contribution is that twenty-eight points, and none of it appears in arm C. Reordering a set that will be kept whole is worth two points of recall; reordering a set that is about to be cut to five is worth twenty-eight more. In neither column does m = r + t close, and neither residual is small enough to be rounding — the interval computed below for the aggregate precision difference spans about fifteen points end to end, and the interaction is roughly twice that width. "Neither change is worth making without the other" is the plain-language form of an interaction term that dominates its own sum of parts.

The reason a single before-and-after number survives review is that it stays constant across situations that call for opposite decisions. Arm D's precision gain of 0.44 is the headline whether the reranker did the work, or truncation did, or the two fought each other to that net figure. A reviewer who treats the number as a defence against a bad change is believing something tempting for good reasons: it is a real measurement, taken on the real system, and it moved in the intended direction. What it cannot do is vary with the thing that decides the shipping question. Arm B is the demonstration — take the cheap half, and context precision still rises by fifteen points, groundedness barely moves, p95 improves, context tokens fall by nearly three quarters, and context recall collapses by thirty-three points. Every column a short report would print looks like an improvement. The column it would not print is the one that changes the verdict.

The grid costs four arms times forty queries, so 160 retrievals and 160 generations, which is roughly twelve minutes and about $2 in model calls at commodity prices. Both of those figures are the single-arm cost multiplied by four and nothing else: four arms of forty queries is the retrieval count, and four runs of a three-minute fixture is the clock. There is no shared work between arms to discount, because each arm is a full pass over the same queries with a different configuration. That affordability is specific to fixtures. The same factorial design in a production A/B test needs four traffic buckets running for two weeks each to reach significance, which nobody will authorise, and it is the reason the fixture exists.

Numbers without intervals invite the wrong conclusion#

Forty queries is a small sample, and differences on the table above range from obviously real to obviously noise. A paired bootstrap over the query set tells you which is which. A bootstrap draws a new forty-query sample from your forty queries with replacement, thousands of times over, and reads off how much the average wanders across those samples, which is an estimate of how much it would wander across other query sets you might have labelled. It is paired because both arms answer the same forty queries, so the per-query difference is what gets resampled and the correlation between the arms removes most of the variance.

The step that looks like a trick is drawing from your own forty queries to learn how a different forty would have behaved, and it is worth saying why that is allowed. The forty are a sample from some population of questions, and the sample is the only description of that population available. Drawing from the sample with replacement reproduces the shape of drawing from the population it came from, so the spread of the resampled averages estimates the spread of averages across other samples that population could have produced. The step that is not allowed is the one it looks like it might license: if the forty were not drawn from the traffic you care about, resampling them harder does not repair that, and the interval will be confidently narrow around the wrong centre.

def paired_ci(a: dict[str, float], b: dict[str, float],
              n: int = 10_000, alpha: float = 0.05) -> tuple[float, float]:
    """95% CI on the mean per-query difference b - a."""
    qs = sorted(a)
    diffs = [b[q] - a[q] for q in qs]
    means = []
    for _ in range(n):
        sample = [diffs[randrange(len(diffs))] for _ in diffs]
        means.append(sum(sample) / len(sample))
    means.sort()
    return means[int(n * alpha / 2)], means[int(n * (1 - alpha / 2))]

Applied to arms A and D, context precision improves by 0.44 with an interval of [0.36, 0.51], which is not something forty queries produce by chance. Context recall drops by 0.03 with an interval of [-0.09, 0.02], so the honest statement is that no recall loss was detected and one as large as nine points is not excluded. Groundedness moves by 0.01 with an interval of [-0.05, 0.07] and should be reported as unchanged rather than as a small improvement, since a reader who sees 0.91 and 0.90 in adjacent columns will otherwise invent a story about it.

There is a second source of variance that the bootstrap does not capture, which is the judge. Groundedness and answer quality are scored by a model, and that model returns different verdicts on repeated calls even at temperature zero once batching and provider-side changes are involved. Score each arm three times and look at the spread before believing any difference smaller than it: on this fixture, repeated grading of the same forty answers moves aggregate groundedness by up to 0.02, which is twice the difference between arms A and D. Pin the judge to a specific model version, keep the rubric in the repository next to the query set, and record both in the report, because a judge upgraded midway through an experiment invalidates every comparison across it.

The belief worth giving up here is that a confidence interval is a defence against noise in general. The bootstrap resamples which queries are in the set and treats each query's score as fixed once it has been assigned, so the interval it reports does not widen when the grader becomes less reliable: it stays the width it was while the thing it appears to protect against gets worse. The repeat-grading spread therefore has to be measured separately and printed beside the interval rather than folded into it. Arm B's three-point groundedness drop clears that spread by a single reporting step, which is not enough to lean on either.

The query set has to be frozen for the same reason the judge has to be pinned, and it is the rule most often broken with good intentions. Someone finds a failing production query during the work, adds it to the set, and now the before numbers were computed on a different population from the after numbers. Keep the frozen set for the comparison, collect new queries in a separate pending file, and fold them in as a new version afterwards with all arms rerun. Version the set explicitly, since a comparison against a number from three months ago is meaningless if the set changed underneath it.

Adding the failing query is this chapter's own failure arriving at a different level. The before number now describes one population of questions and the after number describes another, so their difference carries the system change and the population change added together, with one measurement for the two of them. Adding a query known to fail moves the population in a known direction as well, which makes the contamination systematic rather than random, and nothing in the report will show it.

Per-class numbers deserve the same interval treatment and usually tell you more than the aggregate. On this change, the eight identifier queries show no movement at all, which is expected because a reranker cannot fix what dense retrieval never returned, and the nine procedural queries show the largest context-precision gain and the entire recall loss, because they are the queries with many gold chunks and truncation to five cannot hold them.

The recall half of that breakdown reconciles with the aggregate, which is what makes it worth trusting. Nine queries out of forty losing nine points each comes to a shade over two points spread across the whole set, against a measured aggregate loss of three points, and the gap between them is one reporting step on figures printed to two decimal places. The claim that the procedural class carries the entire recall loss is therefore licensed: no other class shows a loss, and the procedural share on its own reproduces the aggregate to within the precision the table is printed at. The precision half does not close the same way and it is worth saying so rather than implying it does — the two classes named account for seventeen of the forty queries, the rest are not broken out, and no arithmetic available here forces the procedural gain to be the largest one.

Latency and tokens belong in the same table#

A quality report that omits the serving numbers is asking the reviewer to approve a trade with one side hidden. Report p50 and p95 rather than the mean, because a reranker's cost lands in the tail, and report tokens per query as a distribution too, since the mean hides the queries where assembly overflows the budget.

For arm D, the numbers are 655 context tokens against 2,510, which at $3 per million input tokens is a saving of $5.60 per thousand queries, against a reranker running at $1.40 per thousand on a hosted endpoint. That is a net saving of about $4.20 per thousand queries and 200 ms added at p95. The saving is real and it is also not the reason to make the change; if the reranker had cost $8 per thousand the change would still be right, because 0.14 context precision was producing wrong answers. Putting cost in the report keeps it a factor rather than the argument.

Each of those figures comes out of the table by one multiplication, and the column closes with nothing left over. The difference between the two token counts is a little over eighteen hundred tokens a query, so a thousand queries is a little under two million tokens, and two million tokens at three dollars a million is the $5.60. Subtracting the reranker's $1.40 from that leaves the $4.20 exactly, with no rounding to absorb. The added latency is the same subtraction on the p95 column, 1.9 s against 2.1 s, which is the 200 ms.

Reading the same two columns as gaps between arms settles which change is paying for what. Adding the reranker without truncating moves context tokens from 2,510 to 2,530, and adding it on top of truncation moves them from 640 to 655; twenty tokens in one case and fifteen in the other, which is nothing against a saving measured in thousands. The entire token saving is therefore attributable to truncation, and the reranker's cost is paid in latency rather than in tokens — arm A to arm C is two hundred milliseconds at p95, and arm B to arm D is three hundred. Those two are the same change measured against different baselines and they differ by one reporting step, which is the p95 column's own resolution, so the honest reading is that the reranker costs somewhere around two to three hundred milliseconds at the tail and the grid cannot say more precisely than that.

The report itself is worth standardising so that a reviewer does not have to reconstruct what was measured.

{
  "change": "add cross-encoder rerank; keep top 5 of 50",
  "query_set": "fixture-v3 (40 queries, frozen 2025-02-11)",
  "arms": ["A baseline", "B truncate only", "C reorder only", "D both"],
  "primary": {"context_precision": {"A": 0.14, "D": 0.58, "ci": [0.36, 0.51]}},
  "guard":   {"context_recall":    {"A": 0.94, "D": 0.91, "ci": [-0.09, 0.02]}},
  "serving": {"p95_ms": {"A": 1900, "D": 2100},
              "context_tokens_p50": {"A": 2510, "D": 655}},
  "per_class": {"procedural": "precision +0.51, recall -0.09",
                "identifier": "no change (expected)"},
  "not_measured": ["queries outside the labelled set",
                   "reranker timeout path", "cold-cache latency"]
}

The guard key is the discipline. Naming one metric as the thing you are trying to move and another as the thing you are not allowed to break, before running the arms, removes the freedom to pick whichever number came out well afterwards. The not_measured key does similar work at the other end, since a reader who knows what was left out can weigh the result instead of assuming it covers everything.

The limits of a forty-query fixture#

These numbers describe forty labelled queries on a constructed corpus, so they are evidence about a mechanism and not an estimate of the effect on real traffic. Production traffic contains query shapes nobody labelled, and the recall loss on procedural queries could be three times larger there if procedural questions are more common than the fixture's 22%. Roll out behind a flag, and watch the behavioural signals that do not need labels: thumbs-down rate, the share of queries followed by a reformulation within two minutes, and escalations to human support.

The deeper limit is that every arm in the table is a setting somebody knew about. The grid varies rerank and k because those were in the design document; it does not vary the similarity threshold that the vector client applies by default, or the maximum context length the assembly step silently enforces, or the query rewriter that strips punctuation before retrieval. Those are not in the table because nobody wrote them down, and they are usually where the largest single effect is hiding.

Two changes and one number is one equation in two unknowns, and no amount of care in taking the measurement repairs an equation that has no unique solution. The repair is more equations: an arm for each change on its own, against the same baseline, on the same frozen query set, so that the leftover can be computed rather than assumed away. On this change the leftover was the larger part of the effect in both columns — twenty-nine points of the precision gain and twenty-eight points of the recall protection — which is the arithmetic form of a proposal that could not have been split in half.

The pair of numbers in the title is the other half of the discipline. Report recall and precision together, name one of them as the thing being moved and the other as the thing that is not allowed to break before any arm is run, and put latency and tokens in the same table so that a reviewer sees both sides of the trade at once. A report that does all of that is still evidence about forty labelled queries and nothing wider, and writing that limit into the report is what stops the next reader from treating it as more.