Rare identifiers and error codes

Identifiers are the query class where lexical retrieval wins outright: error codes, internal service names, configuration keys, ARNs, feature flags — strings that exist in one place because somebody invented them and documented them once. Rarity is the property doing the work, and it is measured by document frequency, the count of documents containing a term at all. A df of 1 against 4.4 million chunks produces an inverse document frequency near the top of what the formula can express, and that weight arrives from corpus statistics rather than from anybody telling the engine that error codes matter.

The comparison is against dense retrieval, in which every chunk is turned into a fixed-length numeric vector by an embedding model, the query is turned into a vector by the same model, and the chunks whose vectors sit closest to the query's come back. What matters here is that closeness in such a space means resemblance, and an error code is a string whose whole purpose is to be distinct from the things it resembles.

Why the arithmetic is not close#

Take ERR_CONN_REFUSED, a code a platform team invented and documented in one place, so that it appears in exactly one chunk of a 4.4-million-chunk engineering corpus. Its BM25 inverse document frequency is ln((4,400,000 − 1 + 0.5)/(1 + 0.5) + 1) = 14.89. Every competing document has to build its score out of ordinary vocabulary, and each of those terms has a ceiling — the saturation limit from the previous chapter, which multiplies the term's IDF by k1 + 1:

The right-hand column is one multiplication per row and it is worth doing, because the whole argument of this chapter is that column. The saturation ratio from the previous chapter climbs towards k1 + 1 and stops there, so the most a term can ever contribute is its inverse document frequency times that ceiling — a little over two at the conventional setting. Multiply 3.83 by it and refused tops out at 8.4. Multiply 3.02 and connection tops out at 6.6. Multiply 1.85 and error tops out at 4.1. Add those three and the total is 19.1, which is the ceiling quoted in the sentence below, arrived at by adding a column rather than by assertion.

Read the right-hand column as a budget. A document with no occurrence of the identifier can extract at most 8.4 from refused however many times it repeats it, at most 6.6 from connection and at most 4.1 from error, for a hard ceiling of 19.1 — and reaching anything near that means repeating all three dozens of times, which no real page does. The definition page scores 14.89 from a single occurrence of the identifier, which is already nearly twice what any competitor can extract from refused at any term frequency, and every further query word it matches adds on top. There is no ranking accident available here: the one document containing the rare token wins.

The dense retriever's behaviour is equally explicable and considerably less useful. Its embedding model has no entry for ERR_CONN_REFUSED in its vocabulary either, so the tokeniser breaks the string into subword pieces — something like ERR, _, CONN, _, REF, USED — and the pooled output vector is a blend of those fragments' contextual representations. What that vector encodes is roughly "an uppercase snake-case token about connections and refusal", which is a genuinely good summary of the topic and carries no information distinguishing this code from any other connection error. Measured cosine similarities against the query on that corpus:

query: "ERR_CONN_REFUSED"

  0.784  Debugging network connectivity in staging
  0.771  TLS handshake failures and how to read them
  0.769  Egress proxy runbook
  0.762  Connectivity troubleshooting FAQ
  0.741  Sidecar error codes reference   <- contains the definition
  0.738  Connection pool exhaustion

The correct document is not missing from the neighbourhood. It sits fifth, four hundredths of a cosine point below a page that never mentions the code, and it is below the cut in any pipeline that retrieves the top four. This is what semantic similarity does to an exact token: it maps it to the region of things that resemble it, and resemblance is precisely the property the user was not asking about.

The spread in that list is worth noticing separately from the ordering. Six results within 0.046 cosine of each other is not a ranking with a clear winner; it is a set of near-ties, and near-ties are what you get whenever the query carries no signal the model can discriminate on. A downstream reranker will sometimes rescue this and sometimes not, and either way it costs a cross-encoder pass over every candidate — 40 to 200 ms for a top-50 rerank on a hosted model — to reorder documents that a term lookup would have separated by a factor of five in score. The lexical index is not competing with the reranker here; it is removing the need for one on this query class.

The reason those six results are near-ties is the same reason the ordering is wrong, which is worth seeing as one fact rather than two. The query is a token the model never learned as a unit, so what its vector encodes is the blend of the fragments it was broken into, and that blend is a topic rather than a name. Every page about connection failures matches a topic about connection failures roughly equally well. So the spread of the scores measures how much the query distinguishes among the documents, and a spread of 0.046 across six results is the model reporting that it cannot distinguish among them at all. The ordering inside that band is then decided by whatever tiny differences happen to exist, which is why the correct page can sit fifth without anything having gone wrong in the model. A tight spread is not a mild version of a good ranking. It is the absence of one.

Whether the lexical side has this advantage available is a single lookup, and it is worth doing before you form an opinion about any specific query:

# how rare is this term, actually?
curl -s 'localhost:9200/docs/_count' -H 'Content-Type: application/json' \
  -d '{"query": {"term": {"body.exact": "err_conn_refused"}}}' | jq .count

1

A df of 1 to a few hundred against millions of documents means BM25 will win the query outright. A df in the tens of thousands means it will not, and the reason is entirely visible in the IDF column of the table above.

Where the boundary between those two sentences sits can be worked out rather than felt, and the budget table supplies everything needed. A single occurrence of the identifier wins outright when its inverse document frequency alone exceeds the best a competitor can extract from any one ordinary term, and the best figure in that column is the 8.4 belonging to refused. So the question is what document frequency gives an inverse document frequency of 8.4. Since the weight is the logarithm of roughly the collection size over the document frequency, an answer of 8.4 needs that ratio to be e to the power of 8.4, which is a few thousand to one — so df has to be a few thousand times smaller than the collection, and on 4.4 million chunks that is about a thousand documents. Below a thousand the identifier wins from one occurrence. Above it, the identifier is competing on the same terms as ordinary vocabulary and the win becomes a matter of degree.

Recording the win as a rank#

An anecdote about one error code convinces nobody, including yourself in three months when someone proposes removing the lexical index. The thing worth building is a small harness that takes a list of query-and-gold-chunk pairs, runs both retrievers, and reports where the gold document landed:

from dataclasses import dataclass

@dataclass
class Case:
    query: str
    gold_id: str
    kind: str          # "error_code" | "identifier" | "service_name"

def rank_of(results: list[str], gold_id: str) -> int | None:
    """1-based rank of the gold chunk, or None if outside the result set."""
    try:
        return results.index(gold_id) + 1
    except ValueError:
        return None

def head_to_head(cases: list[Case], k: int = 20) -> None:
    rows = []
    for c in cases:
        lex = rank_of(bm25_search(c.query, size=k), c.gold_id)
        vec = rank_of(vector_search(c.query, size=k), c.gold_id)
        rows.append((c.query, c.kind, lex, vec))
    for q, kind, lex, vec in rows:
        print(f"{q:<28} {kind:<13} bm25={str(lex):>4}  vec={str(vec):>4}")

    def mrr(idx: int) -> float:
        return sum(1 / r[idx] if r[idx] else 0.0 for r in rows) / len(rows)

    print(f"\nMRR  bm25={mrr(2):.3f}  vector={mrr(3):.3f}")

Run over a dozen cases drawn from real agent queries, the pattern is not subtle:

MRR  bm25=0.944  vector=0.216

Four of the twelve gold chunks never appear in the dense retriever's top 20 at all, which is the result that matters operationally: a reranker cannot fix it, because a reranker only reorders what retrieval handed it. The dashes are the argument.

The two headline figures can be bounded from the rank column without recomputing them, which is the check worth doing on any summary statistic sitting under a table. Mean reciprocal rank averages one over the rank of the first correct result, and a miss contributes nothing. Ten of the twelve lexical rows are at rank 1, and ten ones over twelve rows is already above eight tenths before the remaining two rows contribute anything at all, so a lexical figure in the nineties is the only place the average can land. Four of the twelve dense rows are misses contributing zero, so even if the other eight were all at rank 1 the dense figure could not exceed eight twelfths — two thirds. It comes in at 0.216, well under a third of that already-generous bound, because the eight it did find sit at ranks like 14, 12 and 11 where a reciprocal is worth almost nothing. The dashes cap the metric and the deep ranks do the rest.

Two details of the harness matter more than they look. The cases have to come from real traffic rather than from imagination, because a list of identifiers you invented will over-represent the ones you happen to remember and will miss the shapes your users actually type; four weeks of query logs filtered to strings containing an underscore, a digit-letter run, or two internal hyphens will produce a candidate list in an afternoon. And the gold label has to be a chunk id rather than a document id, since a document can be split across six chunks of which only one contains the definition, and scoring the document as a hit when the retriever returned the chunk describing something else is a way to give yourself a benchmark that never fails.

Reporting MRR alongside the ranks keeps a single dramatic row from carrying the argument. Mean reciprocal rank rewards getting the answer at position 1 and decays quickly — rank 2 contributes 0.5, rank 10 contributes 0.1, a miss contributes 0 — which matches how a retrieval-augmented pipeline actually consumes results when it passes the top three chunks to a model and discards the rest.

The score gap is also inspectable, which the cosine similarities are not. Every Lucene-based engine will explain how it arrived at a number:

curl -s 'localhost:9200/docs/_explain/chunk:sidecar-error-codes%233' \
  -H 'Content-Type: application/json' \
  -d '{"query": {"match": {"body": "ERR_CONN_REFUSED"}}}' \
  | jq '.explanation | {value, description}'

{
  "value": 32.41,
  "description": "weight(body:err_conn_refused in 118) [PerFieldSimilarity]"
}

That value can be checked against the budget table before anything else is read, and the check passes: 32.41 is just under the 32.8 the table gives as the most this term can ever contribute. A score above its own ceiling would mean the explain output and the table were computed against different corpus statistics, which is the failure the check exists to catch. Descending into details gives the IDF, the term frequency, the field length and the value of avgdl used, so a ranking decision can be reconstructed from four numbers. Nothing equivalent exists for a cosine similarity of 0.784 — it is a single scalar with no decomposition, which is why debugging a dense retrieval failure so often ends in re-embedding rather than in an explanation.

Where the identifier advantage runs out#

The score gap has a second use beyond explanation. When a query contains a term with df in the single digits and one document scores five times the runner-up, the retriever has effectively identified a unique answer rather than produced a ranking, and that condition is cheap to detect at query time — compare the top score against the second, or simply check whether any query term has df below a threshold. A system that notices this can shorten the context it assembles, skip the reranker, and hand the model one chunk instead of five, which cuts both latency and the chance that four topical distractors pull the answer off course. Dense retrieval offers no equivalent signal, because a cosine of 0.784 means nothing in isolation and the gap between first and second is routinely four hundredths whether the top result is right or wrong.

Three qualifications keep this from being a general claim about lexical retrieval, and each of them can be checked before the argument is made rather than after it is contradicted.

The win depends entirely on the token surviving indexing. If the analyser splits ERR_CONN_REFUSED into err, conn and refused — which the standard tokeniser will do at the underscores — then there is no rare term, df jumps from 1 to a few hundred thousand, and BM25 returns the same topical soup the embedding did. The advantage in the table above is available only to an index configured as described earlier in this topic, and the term dump is how you confirm you have it rather than assume it.

The advantage also shrinks as df rises. SIGKILL 137 scored a vector rank of 3 rather than a dash precisely because it is a widely documented code that appears in 4,100 chunks; its IDF is 6.98 instead of 14.89, and the lexical index returns 4,100 candidates that must then be ordered by something other than rarity. For a truly common code — an HTTP 404, appearing in 380,000 chunks — lexical retrieval offers almost nothing: it identifies a vast candidate set and ranks within it by term frequency and length, which is close to arbitrary for this purpose. That row is the threshold above being confirmed by the results table rather than by an argument. A thousand documents was the point at which one occurrence of an identifier stops outscoring the best a common word can reach, and SIGKILL 137 sits at four thousand — comfortably past it, with an inverse document frequency of 6.98 against the 8.4 ceiling belonging to refused. So the prediction was that its lexical advantage would narrow, and what the table shows is a dense rank of 3 rather than a dash. The identifier win is a property of rare identifiers, and rarity is measurable with one df lookup before you form an opinion.

The third qualification is the one users generate. BM25 matches the string that was typed, so ERR_CONNECTION_REFUSED — a plausible misremembering of the actual code — has df = 0 and returns nothing, while the dense retriever, being indifferent to the exact characters, would have surfaced the same four topical pages it always does and at least one of them might have helped. Fuzzy matching, edit-distance query expansion and dense retrieval all address this and lexical exactness does not. An identifier the user gets slightly wrong is the case where the property that makes BM25 win becomes the property that makes it fail.

Rare exact strings are the clearest case in this topic, and one number carries it: an IDF of 14.89 against a competing budget of 19.1 that no real document collects. The harness, the MRR and the _explain output exist to turn that arithmetic into evidence somebody else can check, because "vector search is bad at error codes" is an opinion and "four of twelve gold chunks outside the dense top 20, MRR 0.944 against 0.216" is a measurement with a date on it.

Two further query classes behave the same way for related but distinct reasons, and building one standing case for each is what turns twelve rows into a regression test rather than a demonstration.