A cosine floor and an 18-of-20 can both be measuring nothing

Two numbers get quoted in most arguments about whether a document-search system is working, and both routinely do less than they appear to. The first is a cosine similarity: an embedding model turns each passage and each question into a list of numbers, and the cosine of the angle between two such lists is the score a vector search ranks on. The second is a pass rate on an evaluation set, which is a fixed list of questions with expected answers, rerun after every change so that two builds can be compared. Neither number is wrong. Both are read as measuring something they do not measure, and in both cases the repair is the same question: does this number change when the thing it is supposed to be about changes?

A rejection floor is the obvious use of the first number — pick a value, discard any retrieved chunk scoring below it, and refuse to answer when nothing survives — and a proposal to set that floor at 0.75 is the case worked below. The difficulty is that a cosine similarity is a number about geometry and carries no information about correctness on its own. Where one vector sits relative to another, under one embedding model over one set of documents, is the whole of what it reports, and nothing in that geometry knows whether the passage answers the question.

What the number is actually a function of#

An instrument tells two situations apart only when its reading differs between them, and how far apart those readings are sets how finely it can be read. A thermometer returning the same value in a warm room and a cold one is not an imprecise thermometer; it is not a thermometer. So the question to ask of a rejection floor is not how precise the score is. It is how much of the variation in that score is caused by the thing the floor exists to detect, and how much by everything else. On this quantity, everything else wins comfortably.

The score depends on the embedding model's geometry before it depends on anything about your documents. Embedding spaces are anisotropic to different degrees, meaning vectors occupy a narrow cone rather than spreading over the sphere, and the width of that cone sets where "unrelated" sits. Measuring it takes two minutes: sample 5,000 random query-chunk pairs from your own corpus, compute the scores, and look at the distribution.

A floor of 0.75 rejects essentially every chunk in the first row, including the correct ones, and accepts essentially every chunk in the second, including pure noise, and the only thing that changed between those two rows is a string constant naming the model.

Read the rest of that table by the distances inside each row rather than by the levels down each column. Levels are not comparable between rows, which is the finding itself. Distances are: how far the correct pairs sit above the unrelated ones is the room a floor has to fit into, and it survives the change of scale that destroys everything else. The first row and the last differ in every figure they contain, and agree exactly on one quantity — the distance from the ninety-ninth percentile of random pairs up to the floor that keeps 95% of gold pairs is the same in both. Two different models, every absolute number moved, the usable margin unchanged.

The second row is where that pattern breaks, and the break is the informative row. There the floor needed to keep 95% of gold pairs sits below the score that one random pair in a hundred already reaches. That is the arithmetic statement that no setting of a floor on this model both keeps the correct chunks and excludes the noise: the two requirements have crossed over, and there is no operating point between them. The third row is the opposite case and closes exactly, since the floor for 95% gold recall and the ninety-ninth percentile of random pairs are the same number there — the two criteria meet at a single value with nothing left to trade.

The third row is also the same model as the first on a narrower corpus, where every document is about the same product and the mean of the random-pair distribution shifts up by 0.13, so a floor calibrated on the full corpus is wrong the moment you deploy the same code for one product line. The gold-pair mean in that row barely moves. The shift is therefore not a translation of the whole picture but the noise floor climbing into the signal, and what closes behind it is the room a floor needed.

Query length moves it too, in a direction that catches people out. A three-word query has a short vector built from little context and tends to score lower against everything, so short queries produce lower top-1 scores than long ones on the same corpus, and a fixed floor becomes a filter that rejects terse users. On this corpus, one-to-four-word queries had a mean top-1 score of 0.54 against 0.67 for queries of twelve words or more, with the same rate of correct answers in both groups. That last clause is the control that makes the observation mean something: correctness was held constant across the two groups while the score moved, which is the defining shape of a variable the score responds to and a floor cannot distinguish from the one it is meant to catch.

None of this makes the score useless. It means the score is an ordinal quantity, reliable for ranking chunks against each other within one query on one index with one model, and an unreliable cardinal quantity for comparing across queries, across corpora or across models. Ranking is exactly what retrieval needs it for; thresholding is exactly what it is bad at.

The reason ranking survives what thresholding does not is a cancellation. Every candidate inside one query is scored against the same query vector, by the same model, over the same index, so whatever offset the model's geometry and the corpus's house style contribute is common to all of them and drops out of any comparison between two candidates. Sorting reads only those differences. A threshold reads the raw value, where the offset is still present and is the largest term in it, which is why the same number is trustworthy in a > between two chunks and untrustworthy in a > against a constant.

Calibrating a floor you can defend#

If you want a rejection rule anyway — and there are good reasons to want one, since abstaining beats answering from junk — the floor has to come from labelled pairs on your own corpus with your own model. The material for that already exists: the eval set carries a gold chunk id per question, which gives positives, and any chunk retrieved for a question whose gold chunk is elsewhere gives a supply of negatives.

def score_distribution(cases, index_version, k=20):
    pos, neg = [], []
    for c in cases:
        hits = retrieve(c["q"], k=k)
        for h in hits:
            (pos if h["chunk_id"] == c["gold"] else neg).append(h["score"])
    return sorted(pos), sorted(neg)

def sweep(pos, neg, floors):
    for f in floors:
        kept_gold = sum(s >= f for s in pos) / len(pos)
        kept_junk = sum(s >= f for s in neg) / len(neg)
        print(f"{f:.2f}  gold kept {kept_gold:.0%}  junk kept {kept_junk:.0%}")

Twenty questions gives twenty positives, which is too few to trust a floor to two decimal places, so the real version of this exercise labels a few hundred pairs by hand. Even at twenty the shape of the answer is visible on this corpus:

The quantity to compute across each row is the difference between the two middle columns, because that difference is the entire purchase a floor makes: gold kept above it, junk kept below. At 0.45 the two sit close together near the top, so the rule is discarding almost nothing of either kind. At 0.75 they sit close together again near the bottom, discarding almost everything of both. The widest daylight is at 0.62, where four fifths of the gold survives and under a third of the junk does, and that is the best this instrument does on this corpus — bought at the price of two correct answers turning into abstentions.

The first row's note looks inconsistent with its own numbers and is not, which is worth working out rather than skipping. At 0.45 the filter plainly removes chunks, since the retrieved non-gold column is not at 100%, and the note still says the rule never fires. Both are true, because what fires is the refusal, and the refusal is triggered only when nothing at all survives the floor. Gold chunks kept stands at 100% in that row, so every one of the twenty questions retains at least one chunk, so the surviving set is never empty and the refusal is unreachable. The row below it drops one of the twenty positives and still reports no change, for the same reason: losing a gold chunk empties nobody's context, and one case moving or failing to move is exactly what a set of twenty cannot resolve.

The distributions overlap heavily, which is the honest finding, and it holds on every corpus where documents share a house style. A single scalar threshold on top-1 score is a weak instrument, and the versions that work better in practice use the shape of the ranking rather than its absolute level — the gap between rank 1 and rank 5, or the ratio of top-1 to the corpus-wide mean for that query. Those are still calibrated quantities that need labelled pairs; they are just less sensitive to the model swap that breaks a fixed floor. Each of them is also a difference or a ratio taken inside one query, which is why they inherit the cancellation that a raw threshold throws away.

There is a cheap habit that makes the scores in your trace records readable across model changes, which is to compute the random-pair distribution as part of every build and store its mean and standard deviation in the index file. Reporting a score as its distance from that baseline in standard deviations — 0.71 on a corpus whose random pairs sit at 0.28 with a spread of 0.06 is seven standard deviations out, while 0.83 against an ada-002 baseline of 0.74 with a spread of 0.03 is three — costs one line at query time and makes two builds comparable. It does not turn the number into a relevance probability, and anyone treating it as one has recreated the original mistake in nicer units, but it does mean a score in a record from March is legible after the embedding model changes in June.

The version that works best on this pipeline is not a score rule at all. It is the abstention instruction from the previous chapter, which asks a model looking at the actual text whether the text states the answer, and that question is closer to what you want than any function of a vector distance can be. The score is a property of the retrieval geometry; whether the answer is present is a property of the words.

The same suspicion, turned on the eval#

Removing one input, holding every other input fixed and reading the change in the score is an ablation, and it is the ordinary way of finding out how much of a result was resting on the part you took away. A support team maintains its own acceptance set of twenty questions and reported 18 of 20 correct on the current build, which is a better number than the engineering eval's 16. Running that set again with the context block emptied — everything else identical, same model, same temperature, same instructions — it still gets 15 right.

def empty_context_render(hits, chunks_by_id, question, template):
    return template.format(context="", question=question), 0, []

before = run_eval(support_cases, INDEX, render_fn=render)
after  = run_eval(support_cases, INDEX, render_fn=empty_context_render)

The ablation works by cancellation, and the cancellation is what makes its residual mean anything at all. Two runs go over the same twenty questions, with the same model, the same temperature, the same instruction block and the same expected answers. Every one of those terms enters both runs identically and drops out of the difference between them. One input differs, so the movement in the score is attributable to that input and to nothing else. What is left standing when the context is gone is the part of the score that was never resting on retrieval, and naming that residual is the point of the exercise, because it is the part that no work on the first five stages can move.

Fifteen of those twenty questions are answerable from the model's weights, so the 18 was never evidence that the retriever works. The retrieval-attributable part of the score is three questions, which is the only part of the number that moves when you change anything in the first five stages, and a change that broke retrieval entirely would show up on that set as 18 dropping to 15 — a difference the launch thread would have read as noise.

Reading which questions survive makes the pattern obvious. What is a webhook? passes with an empty context because it is a general question about a general concept. How do I rotate an API key? passes because the model has seen a thousand documentation pages describing the same flow, and the answer it produces happens to match the company's. Who approves a refund above €5,000? fails without context, because the answer is a role name that exists only inside this company, and what is the SLA credit for a four-hour outage on Business? fails because the number is arbitrary and corpus-specific.

The engineering eval was built differently — each question was written against a gold chunk id, which forces the question to be about something a specific chunk states — and it behaves accordingly.

The engineering set drops from 16 to 5, so eleven of its twenty questions are genuinely measuring retrieval, and its ceiling is 19 when the gold chunk is handed over directly — the twentieth question fails even with perfect retrieval, which makes it a generation or prompt problem and an interesting case in its own right. The gap between 16 and 19 is the headroom available to the retrieval stages, and it is a far more useful number to carry into a planning discussion than either raw score.

The support set's first and last columns close against each other, and that identity is the more brutal of the two readings. Full context and gold chunk only are the same number there, eighteen either way, which says a perfect retriever handing that set exactly the right passage every time scores what the current one already scores. Its two failures are therefore not retrieval failures, and no work on chunking, embedding, ranking or budgets will ever move them. A set with three questions of retrieval-attributable score and zero headroom above its current score is not a weak instrument for the decision in front of this team. It is unconnected to it.

The random-context column is the check that catches a subtler failure. A model that answers correctly with five irrelevant chunks in front of it is ignoring the context, which is the same pathology as answering from weights, and it shows the confinement instruction is not binding. The support set losing only one question between empty and random says its passes do not depend on the context at all. The engineering set loses one question across the same step, and that agreement is worth recording as a separate quantity from either raw score: it is roughly what putting irrelevant text in front of the model costs, independent of how much a set depends on retrieval. One question on each set is far too little to put a rate on, and the direction being identical on two sets that agree on almost nothing else is the part to keep.

Questions that belong in a different set#

The ablation does not say the fifteen questions are bad questions. What is a webhook? is a question users ask, and an assistant that answers it well is doing its job; the question simply belongs in a different set, one that guards against the pipeline breaking in ways that make the model refuse or ramble. Keeping it as a regression check is reasonable, and counting it towards a retrieval score is not.

The distinction is between two different jobs a question can do, and a set can only do one of them well. A question that passes with an empty context is sensitive to the model, the temperature and the instruction block, and insensitive to everything before them, so it detects a broken template and cannot detect a broken index. A question written against a gold chunk is the reverse. Mixing the two produces a single number that moves for both reasons and identifies neither, which is how 18 of 20 came to be quoted in an argument about retrieval.

Whether the model read the context at all#

It does not measure groundedness either. A question can fail the empty-context run and still be answered from the model's weights when context is present, because the presence of plausible-looking context does not force the model to read it. Detecting that needs the citation check from the previous chapter, where each cited chunk is verified against the claim, or a counterfactual run with a deliberately altered chunk.

The counterfactual version is the sharper instrument and takes about twenty minutes to build, because the eval already names a gold chunk per question. Copy the index, edit the gold chunk's text to change one factual token — €5,000 becomes €8,000, thirty days becomes forty-five, eu-west-1 becomes eu-central-1 — re-embed only that chunk, and rerun. An answer that follows the edited document is reading the context; an answer that keeps the original figure is reading the model's weights while the correct-looking context sits in front of it. On the engineering set, 17 of 19 answers followed the edit, one kept the original figure, and one hedged by giving both. That single stubborn case is a real finding: the question was about a widely documented HTTP status code, and the model's prior was strong enough to override the document, which is exactly the situation where a RAG pipeline over internal exceptions to a public standard will fail and where the confinement instruction has to be strengthened.

Those three counts close, and they close against a denominator that is not twenty. Seventeen followed the edit, one kept the original figure and one gave both, which comes to nineteen with nothing unaccounted for, and nineteen is the number in the last column of the table above — what this set scores when handed the gold chunk directly. The twentieth question fails even under perfect retrieval, so it has no correct answer for an edited document to move and belongs outside the measurement. The denominator here is the count of cases the instrument can actually read, not the size of the set, and a reader checking the arithmetic against twenty would conclude that a case had gone missing.

Twenty gold-chunk questions as a floor#

And a set that discriminates is not automatically a set that represents traffic. The engineering eval is built from gold chunks, which biases it towards questions with a single clean answer in a single chunk, and real questions frequently span two documents or presuppose something false. The twenty questions are a floor rather than a sample, and treating them as an estimate of production accuracy is a separate mistake from the one this chapter is about.

The two mistakes are worth keeping apart, because the repairs have nothing in common. A set that does not discriminate is repaired by writing questions against gold chunks and rerunning the ablation until the empty-context score comes down. A set that does not represent traffic is repaired by sampling real questions from a log, which will pull in exactly the multi-document and false-premise cases the gold-chunk construction excludes — and those cases will lower the score, which is the point of them.

What the ablation has bought is a number worth defending. Sixteen out of twenty on a set where empty context scores five means the score responds to the retrieval stages, which makes it usable as a before-and-after instrument for the change nobody has been willing to make yet: the two remaining confident wrongs both involve a table split across a chunk boundary, and fixing that means changing the chunker. Neither of the two numbers this chapter started with would have survived that use — one because a threshold on it varies with the model rather than with correctness, the other because most of it was never touching retrieval at all.

Unlike every change in the previous chapter, changing the chunker invalidates every vector in the index, and the next chapter is about what it costs to find out.