A recall floor on hard queries gates the change, and the record makes it reversible

An acceptance criterion is the rule that decides, before a change reaches users, whether it is allowed to. For a compression change — fewer dimensions, int8 instead of float32, bits instead of bytes — that rule has to be about retrieval quality, since retrieval quality is what compression spends. A recall floor is the plainest form of one: a level a measurement must stay above, where recall at a cutoff of k is the fraction of a query's known relevant documents that appear in the top k results an index returns. A slice is a named subset of an evaluation set whose members share one property; here the property is a kind of difficulty, so that near-duplicate wordings sit in one slice and rare entity names in another.

Both terms exist because of what a single averaged number does. An evaluation set assembled the way most are assembled is dominated by easy queries, and an average over it stays almost still while the hard part of the set falls apart underneath it. A change that costs a fifth of recall on the queries an expert user actually struggles with can pass an averaged check, ship, and take weeks to be noticed — and if the index it produced carries no record of how it was built, the weeks after that are spent guessing which of a dozen build settings moved. A floor per slice fixes the first half of that. A manifest travelling with the index fixes the second.

An average will approve almost anything#

One 240-query evaluation set at Halvorsen, a specialty insurer whose adjusters search claims documents, had accreted over two years and it was mostly easy, because the queries people write when asked for examples are the ones with obvious answers. Splitting it by difficulty made every decision in this topic look different.

The n column sums exactly to 240, and that is worth noticing rather than skipping. The four slices are a partition of the set and not a selection from it: every query in the set belongs to exactly one slice, so a rule that puts a floor on each slice has covered every query, and no query can fail quietly by sitting outside the slices being watched. A set of difficulty slices that covered only part of the set would give a gate with holes in it, and the holes would be wherever nobody thought to look.

The before and after columns are not a hypothetical either. They are one real change measured — a minor-version upgrade of Halvorsen's vector cluster in November, which triggered a segment rebuild and recalibrated the quantisation along the way. Twenty-one points on the queries that mattered is the near-duplicate row's −0.21 read as points, and the four points on the average that crossed no alarm is the weighted-mean row's −0.041, rounded. The table is the incident, before anyone knew there was one, and that is what licenses everything that follows: the criterion in this chapter is not proposed against an imagined regression, it is the check that would have stopped a change which actually shipped.

Four points on the mean, twenty-one on the slice where adjusters spend their attention. The mean behaved this way because 62% of the set was in the slice that was immune, and a delta of −0.01 over that much of the set pulls the aggregate towards standing still no matter what the rest of the table does. That proportion was an accident of how the set was assembled rather than a reflection of traffic. Weighting by real traffic would help and would not fix it, since the hard queries are the ones users retry, abandon and escalate, and their business value per query is many times higher than a lookup that any keyword search would have satisfied.

The belief that has to be given up here is that a standard evaluation set is a control. It is a tempting belief, and it is tempting for a good reason: the set is fixed, it is versioned, the same queries run before and after, and running the same measurement on both sides of a change is exactly the shape of a controlled comparison. What breaks it is the composition. When most of a set sits in a slice that a given failure mode cannot touch, the number the set returns is very nearly the same number whether the near-duplicate slice held or lost a fifth of its recall — and a defence that returns the same value in both of the cases it exists to tell apart is not a control at all, whatever its statistical form. Its stability reads as reassurance while being a fact about the set's makeup.

The criterion that survives this is a floor per slice rather than a mean over the union. Every slice must clear an absolute recall level and must retain a stated fraction of its own baseline, and a single failing slice rejects the change regardless of what the aggregate does.

Building the set that hurts#

Four families cover most of what compression damages, and three of them can be mined rather than written.

Near-duplicate candidates are the largest family in any corpus with versioned documents. Find them by clustering the existing float32 embeddings at high similarity and keeping groups where the members differ in a way a human cares about — an effective date, a limit amount, a jurisdiction. Each group yields a query that must retrieve one specific member, and the other members are the negatives.

Hard negatives are mined from the index you already have. Run the current production query set, take the top 20 results, remove the ones marked relevant, and keep what is left; those are documents the embedding already thinks are close, and they are the ones whose ordering against the true answer a compressed representation will scramble.

def mine_hard_negatives(queries, index, judged, per_query: int = 8):
    out = []
    for q in queries:
        hits = index.search(q.vector, k=20)
        negs = [h.id for h in hits if h.id not in judged[q.id]][:per_query]
        out.append({"query_id": q.id, "positives": sorted(judged[q.id]), "negatives": negs})
    return out

Rare entities come from the corpus rather than from the query log, since by definition nobody has searched for most of them. Take the vocabulary of the corpus, keep terms appearing in fewer than fifty chunks, and generate a query per term from the chunk that contains it. Halvorsen produced 180 such queries in an afternoon, of which 24 survived a review for being things an adjuster might plausibly ask.

Short keyword queries have to come from the log, filtered to those under six tokens, and they need human judgments because the intent is often ambiguous. This is the family that costs real labelling time, and it is worth it because it is the family that behaves least like the benchmark data the model was trained on.

Halvorsen ended with 90 hard queries and expanded to 400 over two months. Labelling took about three hours of adjuster time per hundred queries with a tool that showed the top 50 candidates from the exact float baseline.

Slice size is the constraint that decides how sharp the gate can be. A recall estimate of 0.89 over 24 queries has a standard error of about 0.073, so a floor set at 0.88 will reject configurations that are fine and accept configurations that are not, roughly a third of the time each. That standard error is the measured spread of per-query recall across those 24 queries, divided by the square root of the slice size, and it is worth saying plainly that it is not the binomial standard error for a proportion of 0.89 over 24 trials. Recall for a single query is a fraction — how many of that query's several relevant documents came back — rather than a hit or a miss, so per-query values scatter more widely than a coin flip does. Assuming a Bernoulli variable underneath would produce a smaller and much more flattering number, and a gate built on it would look sharper than it is.

Reaching a standard error near 0.03 needs about 120 queries per slice, and the step from one of those figures to the other is the part a reader can rebuild and then reuse on their own slices. A standard error is a spread divided by the square root of the sample size. The spread is a property of the queries themselves and does not move when you collect more of them, so the sample size is the only term available. Dividing an error by some factor therefore means multiplying the sample by that factor squared, since only the square root of the sample reaches the error: quadruple the sample and the square root doubles, which halves the error. Going from 0.073 down to about 0.03 is a division by roughly two and a half, so the sample has to grow by two and a half squared, which is about six. Six times twenty-four is a hundred and forty-four, and about 120 is that same statement with the false precision taken off it.

That arithmetic is why the expansion to 400 queries mattered more than any refinement of the threshold. Moving a floor a hundredth in either direction changes nothing a standard error of 0.073 cannot swamp, while multiplying the slice sizes is the only move that makes any threshold mean something. Until a slice has that many, the honest position is that it produces a warning requiring a human to look rather than an automatic rejection, and saying so in the gate's output prevents the number being treated as more decisive than it is.

The gate#

SLICES = ("distinctive", "near_duplicate", "rare_entity", "keyword")
ABSOLUTE_FLOOR = 0.88
RELATIVE_FLOOR = 0.97
K = 200                      # the first-stage cutoff the reranker consumes

def gate(candidate: dict[str, float], baseline: dict[str, float]) -> list[str]:
    failures = []
    for s in SLICES:
        if candidate[s] < ABSOLUTE_FLOOR:
            failures.append(f"{s}: recall@{K}={candidate[s]:.3f} < absolute floor {ABSOLUTE_FLOOR}")
        if candidate[s] < RELATIVE_FLOOR * baseline[s]:
            failures.append(
                f"{s}: recall@{K}={candidate[s]:.3f} < {RELATIVE_FLOOR} x baseline {baseline[s]:.3f}"
            )
    return failures

Three choices in those constants carry the meaning, and each of them is an earlier chapter turned into a number. The baseline is exact float32 search over the full-width vectors rather than the current production index, for the reason chapter 4 gave when it argued about controls — an approximate baseline lets a compressed index score well by making correlated mistakes, because both the baseline and the candidate skip the same regions of the vector space and agree about the answer they both missed. The cutoff is 200 because that is what the reranker consumes, which is chapter 3's point — that truncation damages deep ranks long before it touches the top ones — turned into a constant: measuring at the depth the second stage actually reads is the only measurement that predicts what the user sees.

Both floors are present because either alone is gameable, and the two ways of gaming them point in opposite directions. A relative floor asks only how far a slice fell from where it was and never where it now is, so a slice that was already weak is licensed to get weaker with every build. The rare-entity slice enters at 0.89, a single point above the absolute floor of 0.88, and 0.97 of 0.89 lands below 0.88 — a relative-only gate would approve the weakest slice in the set sinking under exactly the level the absolute floor exists to hold.

An absolute floor has the mirror blindness. It asks only where a slice now is, so it approves any change that leaves a strong slice above the floor, however much of that slice's own baseline the change destroyed. The distinctive slice enters at 0.97, and a fall to 0.89 would clear 0.88 with room to spare while handing back most of the margin that slice had; the relative test for it sits a little above 0.94 and rejects the same result immediately. The pair is not redundant because a slice's absolute level and its retained fraction are independent facts about it, and a change can be comfortable on one while failing badly on the other.

The shadow stage and the rollback trigger both run on overlap@10, meaning how many of the ten documents the compressed index served for a query also appear in the ten exact search would have returned, which is a proxy for recall that needs no labels and is therefore available on live traffic. The chain that makes it available is short and worth walking. Recall is defined against a list of the documents that were relevant, so measuring recall requires that list; production has no such list, because nobody has told the system which documents were the right ones for the query a user typed a minute ago. Exact search over a sample of the corpus needs no list at all — it just scores every vector and sorts. It also has no tuning parameters: no ef_search to set, no quantised distance kernel, no graph to have been built well or badly. Its ranking is therefore a fact about the vectors rather than a setting somebody chose, and that is what makes it usable as a reference. Agreement between what the compressed index served and what exact search would have returned is then measurable on any query at all, labelled or not.

What that buys is drift detection from any source, since anything that moves the compressed index away from the exact ranking shows up in the same number: a new document distribution that the calibration no longer covers, a partial reindex that left half the segments on an older scale, a store upgrade that changed a default. What it cannot tell you is whether the assistant answered correctly. Agreement with exact search is agreement about which documents come back, and exact search is not an oracle about relevance — it is the same embedding model's opinion, computed without approximation. A change that leaves overlap@10 untouched can still be a change that made answers worse, and a labelled offline measurement is the only thing that says so.

The record that travels with the index#

An index rebuilt without provenance is an artefact nobody can interrogate: the collection is healthy, it answers queries, and there is no way to read out of it whether the vectors are 1024 dimensions or 512, whether quantisation is on, or what quantile the calibration used. The fix is a manifest written at build time into the collection's own metadata, so that reading the index tells you how it was made.

{
  "manifest_version": 2,
  "collection": "claims_chunks_2026_03",
  "embedding": {
    "model": "text-embedding-3-large",
    "revision": "2024-01-25",
    "native_dims": 3072,
    "truncate_to": 1024,
    "transform_order": ["truncate", "l2_normalise", "centre", "quantise"]
  },
  "quantisation": {
    "type": "int8_symmetric_per_dimension",
    "clip_quantile": [0.001, 0.999],
    "calibration_sample": "s3://halvorsen-eval/calib-2026-03-04.npy",
    "scale_sha256": "9f2c41ab6d0e7c5382b1f0a9d4e77c31",
    "centre_sha256": "0b71de44c9a2f6108e35bb27ac9d5e60",
    "keep_originals": false
  },
  "index": { "kind": "hnsw", "m": 32, "ef_construct": 256, "ef_search": 200, "metric": "dot" },
  "evaluation": {
    "harness_commit": "a41d9c7",
    "baseline": "exact_float32_3072",
    "query_set": "eval/hard-v4.jsonl",
    "recall_at_200": {
      "distinctive": 0.962, "near_duplicate": 0.912,
      "rare_entity": 0.897, "keyword": 0.928
    }
  },
  "acceptance": { "absolute_floor": 0.88, "relative_floor": 0.97, "measured_at_k": 200 },
  "rollback": {
    "metric": "overlap@10_vs_exact",
    "threshold": 0.85,
    "window_days": 3,
    "previous_alias_target": "claims_chunks_2025_11"
  }
}

Three of those fields are earlier chapters written down as data rather than carried in someone's head. The clip_quantile pair of 0.001 and 0.999 is the same pair chapter 4 states as percentiles when it fixes where int8 clipping should cut, written here as fractions of one instead of as percentages; it is the same cut, not a different one, and reading it as a different number is the easiest misreading in the file. The transform_order field encodes chapter 4's ordering — truncate, then re-normalise, then centre, then quantise — as data, so that the query path and the indexing path can assert they agree rather than each following whatever its own code happens to do. The keep_originals field is there because chapter 6, which totals the real bill of a compressed index, showed it as the setting that decides whether quantisation saves anything at all, and it is exactly the kind of thing that gets flipped by a migration.

Where this lives matters as much as what it contains. A manifest in a wiki is documentation and drifts; a manifest in the repository is closer and still describes intent rather than state. Writing it into the collection's own metadata — Qdrant's collection payload, an Elasticsearch index mapping _meta block, a row in a retrieval_index_manifest table keyed by collection name — makes it an attribute of the artefact, so a rebuild that does not write one produces an index the query path can refuse to serve. That last consequence is the point of the location: an attribute of the artefact can be required, and a document elsewhere cannot. Halvorsen's search service reads the manifest at startup, compares the truncate_to and transform_order against its own configuration, and fails to start on a mismatch, which converts the entire class of asymmetry bugs from chapter 2 — where a query vector and a stored vector go through different transforms and the distance between them stops meaning anything — into a deployment failure.

The two hashes are the mechanism rather than a label, and the chain they close runs in one direction. The build job computes the scale and centre arrays from the calibration sample. It hashes them. It compares those hashes against the ones in the manifest, and refuses to write the collection if either differs without an accompanying evaluation run. A library upgrade that silently changes a default therefore fails the build instead of degrading production. The property doing the work in that chain is stated in one line: the default is no longer where the value comes from. The manifest is, and the code reads it, so a change of default becomes a disagreement between two recorded values rather than an invisible change of behaviour.

The rollback block is the same idea applied to the way out. A threshold without a named previous target is a wish, and the manifest names the collection to swap the alias back to, which makes the rollback a one-line operation someone can perform at three in the morning without understanding quantisation. Watching the collection after it has shipped runs on the same comparison the shadow stage used. Halvorsen samples 500 live queries a day, runs them offline against a brute-force float32 index over a corpus sample, and compares the top 10 the compressed index served against the top 10 exact search returns. That index ran at overlap@10 of 0.94 in steady state, and the alarm at 0.85 fires on drift whatever caused it. Neither of those figures is comparable with the per-slice floors above them: overlap@10 is a label-free live proxy for agreement, while 0.88 and 0.97 govern a labelled offline measurement of recall, and a system that treats the two as the same quantity will draw the wrong conclusion from one of them.

Answer quality moves separately, and mined negatives go stale#

A recall floor is a statement about the candidate pool and says nothing about whether the assistant answers correctly. Halvorsen kept both, and the relationship between them is loose: the eleven-point answer-quality drop from chapter 3, where truncation cost deep ranks, corresponded to a twelve-point recall@100 loss, and other changes have moved one without the other. The looseness is structural rather than a measurement artefact. A change can lose recall on documents the generator was never going to use, and it can leave recall untouched while reordering the pool so that a weaker passage arrives first. Retrieval gates are fast and cheap and run on every candidate configuration; answer evaluation is slow and expensive and runs on the ones that pass.

Hard-negative sets also go stale, and they go stale in a direction that makes them too easy. Negatives mined from the index of March are the documents that index found confusing, and once a change fixes some of that confusion the set no longer probes the boundary. A set that no longer probes the boundary still returns a number, and the number goes up, which is the worst possible failure mode for a gate: it reports improvement caused by its own decay.

Re-mining quarterly against the current production index is the maintenance, and it has a consequence for how the two floors are read. Because each re-mining produces a set of different intrinsic hardness, the absolute floor is not comparable across re-minings — 0.88 against one generation of negatives and 0.88 against the next are not the same demand. That is an argument for keeping the relative floor as the primary criterion, since a slice compared against its own freshly measured exact baseline is self-consistent whatever the set's difficulty, and for treating the absolute one as a backstop against a slice collapsing far enough that the relative comparison stops being informative.

The manifest, the hard set and the gate are what let the rest of this topic be used rather than merely known. The dial from chapter 3, where truncation trades dimensions against deep ranks, has real positions; the formats from chapters 4 and 5, int8 and binary with rescoring, have measured costs; and the bill from chapter 6 has line items.

What makes any of it safe to turn is that a change now has a criterion before it ships, a record while it runs, and a named way back. The criterion is a floor on the slice that would break first rather than an average that cannot see it break. The record is an attribute of the index itself rather than a description of what someone once intended, which is why a silently changed default now stops a build. And the way back is a collection name written down in advance, so that reversing a bad decision costs one command rather than another three weeks.