Constraint classes and trap passages

A query names a topic and applies conditions to it. The conditions are where retrieval fails, and a constraint class is a name for one kind of condition that similarity cannot represent: whether the passage states a fact or its negation, which of two similarly named entities it is about, which direction a relationship runs in, and whether the reader is allowed to see it. Chapter 1, on writing a rubric for a named task, established recency as one of these: a superseded 2021 refund policy page, retrieved at a cosine similarity of 0.91 for a question about the window currently in force, is the nearest neighbour in a policy corpus and the wrong document. There are four more classes that come up constantly.

The example everyone reaches for is not one of them. What is the capital of France? retrieving a passage about restaurants in Paris describes a passage that is unhelpful, and a model handed it will usually say that it does not contain the answer, or will answer from parametric memory — the facts absorbed into its weights during training, which for a landmark like Paris are reliable and for your corpus are absent — and be right. It exercises nothing, because the failure it produces is a shrug. The distinction worth holding on to is between evidence that is insufficient and evidence that is misleading. Insufficient evidence degrades visibly: the reader gets nothing and can see that they got nothing. What is worth building is a trap passage: the top-ranked result for a query, stating something that a plausible reader with that passage in hand would take as the answer, where the thing they would take is definitively wrong.

Four constraints similarity cannot carry#

Polarity is the one that survives the most engineering effort. Ask a bi-encoder — the standard embedding retriever, which encodes the query and each passage separately and compares the two finished vectors — to score "covered" against "not covered" and you get a similarity in the high 0.8s, because the two sentences share their subject, their object, their register and every content word except one. This is not a defect of a particular model so much as a consequence of how the representation is built: a passage embedding is a fixed-length pooled summary of its tokens, trained by a contrastive objective on pairs judged relevant by topical association, and nothing in that objective ever rewards separating a statement from its negation. Reranking with a cross-encoder — a model that takes the query and one passage together as a single input and scores that pair, instead of embedding each side alone — helps here because the model sees query and passage jointly and can attend to the negation, but it helps unevenly, and it only reranks what the first stage retrieved.

The phrase one token of signal against four hundred words in the table is carrying a mechanism, and the mechanism is short enough to walk. A passage embedding is produced by running every token of the passage through the encoder and then pooling the per-token results into one fixed-length vector. Pooling is an averaging operation. Each token therefore contributes a share of the finished vector that falls as the number of tokens rises, and the word not is one token among the four hundred. Its influence on where the passage lands is roughly its share of the passage, and it gets no larger share for being the word that reverses the meaning, because nothing in the pooling step knows which token carries the meaning. That is why the effect gets worse as passages get longer, and why it does not respond to a better encoder.

The reason a cross-encoder helps then follows with no extra assumption. A cross-encoder never pools the passage into a vector of its own. It reads the query and the passage as a single sequence and produces one score for that pair, so the word not in the query is compared against the passage's tokens directly rather than against a summary in which it has already been averaged away. The help arrives with a condition attached, and the condition matters more than it looks: a reranker only ever reorders the candidates the first stage handed it, so the depth of that hand-off decides which documents are even eligible to be rescued. A target sitting eleventh is out of reach of a reranker fed ten candidates and within reach of one fed fifty, and that is a decision about the retrieval stage rather than about the reranker.

Entity failures are quieter and more frequent. In a corpus of customer contracts the name Northwind Ltd and the name Northwind Solutions Inc differ by two tokens out of six, and once both appear inside four-hundred-word passages about mutual confidentiality obligations the resulting vectors sit at around 0.94 on a scale where unrelated documents sit at 0.2. Measured on one contracts corpus, queries naming a company with at least one near-name sibling retrieved a passage about the wrong legal entity in the top position 31% of the time, against 4% for companies with distinctive names. The remedy is not a better encoder; it is metadata filtering, which means the entity has to be resolved to an identifier before the vector search runs and applied as a hard predicate rather than left to the similarity score.

Those two percentages are a controlled comparison, and that is what makes them worth more than either would be alone. The queries have the same shape, the corpus is the same corpus, the passages are the same length and the same encoder scored all of them. One thing differs between the two groups: whether a similarly named company exists in the index at all. The top-position error rate is close to eight times higher in the group where one does. That isolates the cause and rules out the two explanations a team reaches for first. The encoder does not handle contracts badly, and it does not handle entity questions badly, because it handles those same questions at four percent when there is nothing in the corpus to confuse them with. What it cannot do is order two passages whose distinguishing information is two tokens wide and whose remaining four hundred words are interchangeable, and no amount of retraining widens those two tokens.

Permission is the odd one out, because the constraint is not in the text at all. Whether an outsourced support agent may read an internal-only escalation runbook is a fact about the requester and the document's access metadata, and an encoder sees neither: it reads the passage and scores it against the query, and an internal runbook is usually the best topical match precisely because it is the most specific document on the subject. Nothing in the ranking layer can fix that, and no rubric dimension makes it visible after the fact, so the filter has to run before the search rather than over its results — the requester's scope narrows the candidate set, and the ranker only ever sees documents that reader is allowed to see.

Why the order of those two operations decides the outcome is arithmetic rather than architecture, and it is worth stepping through, because filtering after the search is the cheaper thing to build and the thing most systems do first. Rank first and the internal runbook still wins the top slot. Dropping it afterwards leaves the reader with one fewer result and promotes nothing, because the best document that reader is allowed to see was ranked on its merits against the whole corpus and may have finished below the cut, in which case it was never returned to be promoted. Filter first and the runbook is not a candidate at all, so every one of the five slots is spent on documents the requester may read. Same query, same encoder, same scores. A different candidate set is the only lever there is here, because a similarity score has no way to represent who is asking.

Direction is the class that produces the most confidently wrong output, because both passages are true statements about the same two parties, so nothing in the retrieved text signals to the model that it has the relation backwards. A passage reading "Acme Corp sublicenses our SDK to its resellers under Schedule 3" and a query asking whether we sublicense Acme's SDK are near-identical in vector space and opposite in meaning, and a model given the first will answer the second with a citation.

Building the trap on purpose#

The construction is mechanical once the class is chosen. Take a fact your corpus states, write a query for the complement of that fact, and confirm two things: that the corpus does contain the true answer somewhere, and that the passage stating the true answer ranks below the one stating the complement.

from openai import OpenAI
import numpy as np

client = OpenAI()

CORPUS = {
    "sla/coverage.md#regional": (
        "Regional coverage. The 99.9% monthly availability commitment applies "
        "to us-east-1, us-west-2, eu-west-1, eu-central-1, ap-southeast-1, "
        "ap-northeast-1, and five further regions listed in Schedule A."
    ),
    "sla/coverage.md#exclusions": (
        "Exclusions. The availability commitment does not extend to "
        "sa-east-1, af-south-1, me-south-1 or any region marked Preview in "
        "the console, and no service credits accrue for incidents there."
    ),
}

QUERY = "which regions are not covered by the SLA?"


def embed(texts: list[str]) -> np.ndarray:
    resp = client.embeddings.create(model="text-embedding-3-small", input=texts)
    vecs = np.array([d.embedding for d in resp.data])
    return vecs / np.linalg.norm(vecs, axis=1, keepdims=True)


doc_ids = list(CORPUS)
doc_vecs = embed([CORPUS[k] for k in doc_ids])
q_vec = embed([QUERY])[0]

for doc_id, score in sorted(
    zip(doc_ids, doc_vecs @ q_vec), key=lambda p: -p[1]
):
    print(f"{score:.3f}  {doc_id}")

The fixture repays reading before the result does. sla/coverage.md#regional names six regions and then says five further ones are listed in Schedule A, which is the eleven regions the coverage page claims in total. sla/coverage.md#exclusions names three regions and adds a catch-all for anything marked Preview. Both passages are about SLA regions, both are written in the same vocabulary by the same author, and the whole of the difference between them is the word not and the fact that one list is the complement of the other. A query asking which regions are not covered is asking for the second passage, and every content word in that query points at the first.

On the two-document corpus the exclusions passage usually wins, because it contains the word "not" and there is nothing else to compete with. Against a real twelve-thousand-document index the coverage passage wins, and the reason is instructive: the coverage page is the canonical, heavily-linked, keyword-dense document about SLA regions, and in a corpus that size the exclusions appendix competes with six other passages that discuss regional availability at greater length. A trap that reproduces on a toy corpus and disappears at scale is not the trap you have in production, so build it against the real index or you are testing an artefact of the fixture.

0.891  sla/coverage.md#regional
0.874  pricing/regional-availability.md#intro
0.860  status/incident-2025-11-04.md#scope
0.843  sla/definitions.md#availability
0.712  sla/coverage.md#exclusions

The gap between the top-ranked passage and the exclusions passage is 0.179, which is wide enough that no threshold tuning saves you: setting a cutoff that admits the exclusions passage admits several hundred other things per query, and setting one that keeps precision drops the only correct document. Threshold tuning is where teams spend the week after discovering this, and it is the wrong lever, because the ordering itself is what is wrong.

Reading the listing downwards shows why a cutoff has nowhere to stand. Four passages occupy the span between the top result and the target, and none of them is a stray — a pricing page on regional availability, an incident scope note and an SLA definitions page are all genuinely about the subject. A cutoff placed just under 0.712 to catch the exclusions appendix therefore catches those four as well, plus everything else in a twelve-thousand-document index that happens to land anywhere in that band, and in a corpus that size a band nearly two tenths of similarity wide is not a fringe. It is where most of the topically related material lives. A cutoff placed above 0.712 excludes the one document that answers the question. There is no third position available, and the shape of that dead end is worth carrying to other problems: a cutoff can only separate documents the score has already separated, and here the score has put the truth below its own complement.

From an anecdote to something that keeps firing#

An engineer who finds this failure typically demonstrates it in a notebook, opens a ticket, watches someone add the exclusions text to the coverage page's first paragraph, confirms the query now works, and closes it. Six months later the docs team splits the SLA page in two for readability and the failure returns, silently, with nobody in the building remembering that it was ever a known issue.

What survives is a case in the evaluation set, named by its constraint class, with the trap document pinned by identifier.

- id: sla-polarity-exclusions
  constraint_class: polarity
  query: "which regions are not covered by the SLA?"
  must_retrieve:
    - doc: sla/coverage.md#exclusions
      within_k: 5
  must_not_outrank:
    trap: sla/coverage.md#regional
    target: sla/coverage.md#exclusions
  guard_docs:            # if these vanish, the test is meaningless, not passing
    - sla/coverage.md#exclusions
    - sla/coverage.md#regional
  answer_must_contain: ["sa-east-1"]
  answer_must_not_contain: ["eu-west-1"]
  added: 2026-02-11
  note: >
    Nearest neighbour states the complement of the fact requested. Fixed by
    reranking or by a polarity-aware query rewrite; not by threshold tuning.

The guard_docs block is the field that makes the difference between a test and a decoration. A negative assertion — this trap must not outrank that target — passes trivially once the trap document is deleted or renamed, and corpora churn constantly. Without the guard, the day someone merges the exclusions text into a new page is the day the test turns green for the wrong reason and stops protecting anything. With it, the run fails loudly with a message that says the fixture is stale, which sends a human to look at it.

import pytest, yaml

CASES = yaml.safe_load(open("eval/constraint_cases.yaml"))


@pytest.mark.parametrize("case", CASES, ids=lambda c: c["id"])
def test_constraint_case(case, index):
    corpus_ids = index.doc_ids()
    missing = [d for d in case["guard_docs"] if d not in corpus_ids]
    assert not missing, f"fixture stale for {case['id']}: {missing} not in corpus"

    ranked = [hit.doc_id for hit in index.search(case["query"], k=10)]

    for req in case.get("must_retrieve", []):
        assert req["doc"] in ranked[: req["within_k"]], (
            f"{req['doc']} not in top {req['within_k']}: got {ranked[:req['within_k']]}"
        )

    rule = case.get("must_not_outrank")
    if rule and rule["trap"] in ranked and rule["target"] in ranked:
        assert ranked.index(rule["target"]) < ranked.index(rule["trap"])

The test function makes that argument mechanically, and the order of its statements is the load-bearing part. The guard assertion runs first, before any search happens, and fails if a guard document is missing from the corpus. Consider what the function would do without it. The must_not_outrank rule is wrapped in a condition requiring both the trap and the target to appear in the ranked list, so on the day the trap document is deleted that condition is false, the assertion never executes, and the test passes. Nothing is printed, nothing is reported as skipped, and the suite's green count goes up by one. That is the failure every negative assertion has by construction: it is satisfied by the absence of the thing it forbids exactly as readily as by the thing being handled correctly. The guard converts absence from a silent pass into a loud and differently worded failure, which is the only outcome that puts a person in front of the fixture.

Where the suite runs matters as much as what it asserts. Thirty cases against a live index cost thirty embedding calls and thirty searches, which is roughly four seconds and under a cent per run, so it belongs in the pull-request pipeline for anything touching the retriever, the chunker, the embedding model identifier or the index build. The cases that also assert on generated text — answer_must_contain and its negative twin — cost a model call each and are slower and noisier, so they run nightly against the staging index rather than per commit, and a failure there opens a ticket instead of blocking a merge. Splitting the suite by cost this way is what keeps it from being disabled the first week it makes someone's build take four minutes.

The ratio between the two halves is what decides whether the suite survives its first year, and it is worth computing rather than feeling. Thirty embedding calls and thirty searches in roughly four seconds is a little over a tenth of a second per case, end to end, which vanishes inside a pipeline that is already installing dependencies. Four minutes is sixty times four seconds, and sixty times is the distance between a check nobody notices and a check somebody switches off. Splitting the suite on that ratio, rather than on which cases feel most important, is what keeps the cheap half running on every commit — and the cheap half is the half that catches a regression before it merges.

Sourcing the cases is the part that scales badly if you rely on imagination. Sitting down to invent traps produces polarity examples and stops, because polarity is the one everybody thinks of. The productive source is the complaint log: take the queries where an answer was marked wrong, retrieve for each of them, and read the top passage. When the top passage supports the wrong answer the user received, you have a trap already built by production, and all that remains is to name its constraint class. Twenty complaint queries typically yield five or six cases, spread across classes you would not have guessed.

Diagnostic instruments and coverage maps#

Constraint cases are diagnostic instruments, not a quality measure. A suite of thirty of them, all passing, says that thirty specific known traps are handled and says nothing about the other 11,970 documents in a twelve-thousand-document corpus. They are deliberately adversarial and deliberately unrepresentative, so their pass rate must never be reported alongside recall as though the two were the same kind of number — a system can pass every constraint case and be worse on real traffic than one that fails four of them.

Putting the two counts side by side shows why the comparison is not merely unwise but ill-defined. Thirty cases against twelve thousand documents is a quarter of one percent of the corpus, and that quarter of a percent was chosen adversarially — every one of those documents is in the suite because it is hard. A recall figure is computed over a sample drawn to look like real traffic, which is the opposite selection rule. So the pass rate answers are these thirty known traps still handled, and it cannot be stretched into an answer to how often does retrieval work, because its denominator was picked for difficulty rather than for representativeness. Printed next to recall the two read as commensurable, and the first one then moves for reasons the second is structurally unable to see.

Storing constraint_class as a first-class field is what keeps the set readable as a coverage map rather than as a folder of query strings: four polarity cases, nine entity cases, one direction case and no permission cases is a legible statement about where your testing is thin. It is also the field that tells you when the map has stopped matching your domain, since a class your corpus generates constantly and your suite has never sampled is exactly the gap that produces a surprise.

They also do not indict embeddings as a technique. A lexical ranker fails the polarity query too, and fails it for an unrelated reason: the coverage page and the exclusions appendix share nearly every content word, "not" is a stopword — one of the very common words a search engine's text pipeline deletes before indexing — in most analyzers, and the coverage page is longer and better linked. The two families fail the same query with different mechanisms.

That coincidence carries more information than either failure does alone. Two rankers built on unrelated principles — one comparing dense vectors, one counting term matches — return the same wrong document for the same query, and they arrive there by routes that share nothing: the dense side averages the negation away while pooling, the lexical side deletes it before the index is even written. When two independent instruments land on the same wrong answer, the cause sits upstream of both of them, and here it sits in the query. Not is the only part of that question distinguishing it from its opposite, and neither family has anywhere to put a single function word. That is what makes polarity a constraint class rather than a defect in one model, and it is why the fix has to live in a different stage — a rewrite that turns the negation into a positive request for the exclusions list, or a scorer that reads query and passage together — rather than in a different retriever.

The habit worth taking from this chapter is small: when a wrong answer arrives, look at the top-ranked passage before looking at anything else, and ask whether a careful human reading only that passage would have made the same mistake. If they would, you are holding a trap rather than a miss, it belongs to a class, and the class is what tells you which other queries to go and check.

The temptation after a chapter like this is to conclude that the answer is to switch families, or to run both and take the union. Whether that buys anything is an empirical question, and answering it requires being precise about what each family actually scores and about how to summarise per-query labels into one number that can be compared.