Dense retrieval has nothing to match an error code against

Dense retrieval is the arrangement where a query and every stored passage are turned into vectors by one embedding model, and a query is answered by returning the passages whose vectors sit closest to it. The whole arrangement rests on one property of the model: text a person would call similar in meaning comes back as vectors pointing in similar directions. An identifier is a string chosen to be unique rather than to mean anything — an error code, a part number, a ticket reference, a symbol name from a codebase. It has no neighbourhood of similar-meaning text for the model to have learned, because its meaning is a fact about one document rather than a fact about language, and that is the property this chapter is about.

The failure this produces is worth caring about out of proportion to how much traffic it touches. An identifier is what somebody types when they already know exactly which document they want and are using the search box as a lookup, so the queries that break are the ones arriving with the highest expectation and the least tolerance for a near miss. They also break quietly: a dense retriever handed an identifier returns a full page of plausible, topically adjacent material rather than an empty result, which reads to everyone downstream as a successful search.

The failure has no bug in it#

At 02:41 an on-call engineer copies a string out of a log line and pastes it into the assistant: E_2231_TLSHANDSHAKE. That system's corpus contains a runbook whose title is that exact string and whose first paragraph explains that the error is emitted when the sidecar proxy is restarted while a connection pool is draining. What comes back is three pages of generic TLS troubleshooting, a certificate rotation guide, and a page about handshake timeouts in the load balancer.

query: "E_2231_TLSHANDSHAKE"

rank  cos    document
   1  0.793  ops/tls-troubleshooting.md#0    "Common TLS failures and how to diagnose…"
   2  0.788  ops/cert-rotation.md#2          "If the handshake fails after rotating…"
   3  0.784  net/lb-handshake-timeouts.md#1  "Handshake timeouts observed at the edge…"
 ...
 156  0.711  ops/e2231-tls-handshake.md#0    "E_2231_TLSHANDSHAKE. Emitted when the…"

The runbook is at rank 156 with a cosine of 0.711, below a hundred and fifty-odd documents that contain no instance of the string at all. Retrieval did not fail to find it in the sense of an index bug; the index is intact and the document is in it. The vector for the query and the vector for the document simply are not close, and increasing k to 200 to reach rank 156 would put two hundred chunks in the prompt to catch one.

The listing is more useful read as a gap than as a set of levels. From rank 1 to rank 156 the cosine falls by eighty-two thousandths, and inside that narrow band sit a hundred and fifty-odd documents ordered by nothing the user cares about. Meanwhile the property that decides whether a document can answer the question at all — does it contain the literal string — flips from absent to present exactly once across the whole band, at the bottom. The score is very nearly constant while the thing that matters varies completely, and a quantity that stays flat while the dangerous property swings is a quantity you cannot steer by.

The tokenizer is where the meaning goes#

Run the identifier through a subword tokenizer — the step that chops text into the word fragments an encoder actually reads, since no model has a vocabulary entry for every string anyone might type — and the reason becomes visible.

>>> tok.tokenize("E_2231_TLSHANDSHAKE")
['E', '_', '22', '31', '_', 'TL', 'SH', 'AND', 'SH', 'AKE']

>>> nearest(embed("E_2231_TLSHANDSHAKE"), among=IDENTIFIER_VOCAB, n=4)
[('E_2119_TLSTIMEOUT',   0.938),
 ('E_2240_TLSVERIFY',    0.931),
 ('E_2231_TLSHANDSHAKE', 0.926),
 ('E_3018_TLSRENEGO',    0.919)]

The chain from that first line to the ranking in the section above has four links, and each one is checkable on its own. A tokenizer's vocabulary is fixed at training time and chosen by how often each fragment appeared in the training text, so a string it has never seen is cut at whatever boundaries that vocabulary happens to offer — which is why the cut falls after TL and again after SH, places no author of the identifier would have chosen. Each fragment then carries a vector learned from every context that fragment ever appeared in, so AND here is not a piece of the word handshake at all; its vector was learned from the conjunction and from every other word those three letters sit inside. The encoder composes those fragment vectors into a single vector for the string, and what survives composition is whatever the fragments have in common across their training contexts. For TL, SH and AKE that common ground is real: they co-occur with TLS and handshake text. For 22 and 31 there is none, because digits appear everywhere and mean nothing consistent.

The result is a vector that is well determined in the direction TLS error code and near-arbitrary in the direction which TLS error code, and nearest-neighbour search on it can only return what the well-determined part points at. The encoder is not confused about what kind of thing this is; it places the string firmly in a neighbourhood of TLS-related error codes. What it cannot do is separate members of that neighbourhood, because the difference between 2231 and 2240 is carried by two subword pieces out of ten and those pieces have no meaning that training could have attached to them. The identifier's own embedding does not even rank itself first among its siblings, which is the clearest statement of the problem: an encoder trained to map semantically similar text to nearby vectors is behaving correctly, and the property the user needs is character identity rather than semantic similarity.

Reading the second listing as a band rather than as four scores sharpens the same point. Its four siblings span nineteen thousandths from top to bottom, roughly a quarter of the width of the retrieval band above, and that quarter is being asked to separate four strings where the wider band separates a hundred and fifty-six documents. There is no resolution left at that scale. The ordering is what the prose above reads, and only the ordering: that listing scores a query-side embedding against a stored vocabulary of identifier embeddings, and whether those two sides were produced by the same pass is not stated here, so no argument below rests on the exact self-score.

As an analogy, the model is recognising the identifier the way a reader recognises an unfamiliar chemical name as some kind of chemical without knowing which compound — the shape of the word carries the category and nothing finer. It breaks at the step that matters: a reader who cannot place the compound can still copy the letters down and look them up, whereas the encoder has discarded the letters by the time it produces a vector, and no later stage can get them back.

None of the usual dials reach this. There is no similarity threshold that admits the runbook and excludes the hundred and fifty documents above it, because a threshold needs a cliff in the score-versus-rank curve to cut at and this curve has none; rank 156 sits at 0.711 and rank 1 sits at 0.793 with everything else spread smoothly between. Raising k until the runbook is included means a candidate set of two hundred, and two hundred chunks at 128 tokens each is a little over 25,000 tokens of context to catch one relevant chunk. That also multiplies the precision problem from chapter 2 by ten: a candidate set of twenty holding one relevant chunk is nineteen irrelevant chunks to one, and a candidate set of two hundred holding the same one chunk is a hundred and ninety-nine to one. A better embedding model does not help either, because a model that placed E_2231_TLSHANDSHAKE far from E_2240_TLSVERIFY would be a worse model on every task embeddings are trained and evaluated for. Placing similar meanings close together is a requirement the model is built to satisfy; preserving character identity is a different requirement, and asking one instrument to satisfy both is where the design error lives rather than in any setting.

Everything with this shape breaks the same way. Part numbers, ticket ids, SKUs, function names, rare surnames, chemical designations, table names in a schema, version strings. They are exactly the tokens a user types when they know precisely what they want, which is why the failure clusters in the queries where expectations are highest.

Measure how much of your traffic is this#

The reason to measure rather than assume is that the fix costs real operational money, and the size of the problem varies enormously between corpora. A regex sweep over query logs takes an hour and produces the number that justifies or kills the proposal.

IDENT = re.compile(
    r"""(?x)
      \b[A-Z]{2,}[_-]\d{2,}[A-Z_]*\b     # E_2231_TLSHANDSHAKE, HX-2140R
    | \b[A-Z]{2,}-\d{2,}\b               # JIRA-4417
    | \b[0-9a-f]{7,40}\b                 # commit sha
    | `[^`]{2,}`                         # anything the user backticked
    | \b\w+\.\w+\(\)                     # method references
    """)

hits = [q for q in queries if IDENT.search(q)]
print(len(hits) / len(queries))

Over thirty days and 41,300 queries on the system in question, 7.8% matched. Those queries also carried a thumbs-down rate of 19% against 8% for the rest, and 31% of them were followed within two minutes by a reformulated query from the same user, against 11% overall. Three thousand two hundred queries a month, failing at more than twice the base rate, is a case for spending on infrastructure. Had the number come back at 0.4% with no difference in satisfaction, the correct decision would be to leave dense-only alone and spend the effort elsewhere.

Those figures close on each other, and it is worth checking that they do before spending anything on the strength of them. Applying 7.8% to 41,300 queries gives a little over three thousand two hundred, which is the monthly figure the paragraph above states, so the count and the share are the same measurement written twice rather than two claims that have to be believed separately. The one residual is that the window is thirty days and a month is not exactly thirty days, which moves the figure by less than the sweep's own rounding does.

The two comparisons in that paragraph do not share a denominator, and reading them as though they did is the easiest mistake available here. The thumbs-down figure is set against the rest of the traffic, while the reformulation figure is set against the whole of it, identifier queries included. Converting one to the other is arithmetic the chapter's own numbers support: a slice that is a small fraction of traffic and fails at 19% pulls an all-traffic rate only a little above the 8% the rest achieves, so the identifier slice still fails at better than twice the all-traffic rate, and the claim survives whichever denominator you pick. Doing that conversion is also the honest way to check the reformulation pair, since an all-traffic rate of 11% with a small slice at 31% implies a rate for the rest a couple of points below 11%, which is what a consistent set of measurements should imply.

Worth splitting that 7.8% by shape before choosing a fix, because two populations behave differently. Bare-identifier queries, where the entire query is the string, were 61% of the matches. Mixed queries like "why do I get E_2231 when the proxy restarts" were the remaining 39%, and they need the semantic half as much as they need the exact match, so any fix that routes them away from the dense retriever entirely will make them worse.

Those two shares sum to the whole with nothing left over, which says the two populations are exhaustive and disjoint: a fix covering one of them leaves exactly the other uncovered, and there is no third bucket to hope for. The real residual is where the boundary between them falls. A query counts as bare when the regex matches it end to end rather than merely finding a match inside it, so a trailing question mark or a stray word moves a query from one population to the other without changing what the user wanted. That boundary is a design decision, and it is the same decision the routing code below has to make.

Four fixes, in ascending order of what they cost you#

The last row needs a word of explanation, since it is the one nobody meets in a quickstart: learned sparse retrieval trains a model to rewrite each document and each query as a set of weighted terms, including terms the text never contained, and stores those in an ordinary keyword index, which buys exact matching plus a limited kind of synonym handling at the price of a GPU pass over the corpus at index time. The mechanics of fusing two rankings were covered in topic 8.13, so the question here is only which row to buy. The decision turns on whether the identifier vocabulary is closed. Error codes usually are: they are generated from an enum in the source tree, there are four hundred of them, and a build step can emit the complete list. That makes the first row exact rather than approximate, and exact beats fused on the queries it covers, at a fraction of the cost.

Reading the table down the last column rather than across the rows is what makes it a decision procedure. Each row fails on a different property of the traffic — mixedness, extraction coverage, and word form — so the rows are not four grades of the same product but four instruments with different blind spots. That is why the ordering is by cost rather than by quality: none of them dominates, and buying a later row does not retire an earlier one.

def route(query: str, idx: Indexes) -> list[Chunk]:
    m = IDENT.fullmatch(query.strip())          # note: fullmatch, not search
    if m and (doc := idx.identifier_map.get(m.group(0).upper())):
        return idx.chunks_for(doc) + idx.dense(query, k=5)
    return idx.dense(query, k=50)               # mixed and natural-language queries

The fullmatch is doing the work. A query that consists of nothing but an identifier is unambiguous and can be routed; a query that merely contains one is a sentence with an identifier in it, and hard-routing it discards the part that says "when the proxy restarts". Even in the routed case the function keeps five dense hits, because the on-call engineer will want the surrounding troubleshooting material and the runbook alone may not be enough.

The same word ties the router to the sweep. The sweep counted a query as an identifier query when IDENT.search found a match anywhere inside it, and the router treats a query as routable only when IDENT.fullmatch consumes all of it, so the bare population and the mixed population are precisely the queries the two calls disagree about. The 61% and the 39% are therefore not an estimate of what the router will do; they are a measurement of it, taken before it was written.

The lookup table earns its place only where the vocabulary can be enumerated. As soon as users type customer names, internal ticket references, symbol names from a codebase or arbitrary quoted phrases, the set is open and no build step can produce it, and the lexical index becomes the cheapest thing that covers the whole class. That is the honest form of the general advice: add a lexical retriever and fuse its results with the dense ones so an exact match on the identifier ranks the runbook whatever its vector does. It is the right default answer when nothing more specific is known about the corpus, and it is not the cheapest answer when the corpus is known well.

Whichever row you buy, the fixture is where the claim gets checked before the infrastructure exists. The eight identifier queries in the query set are the smallest useful arm: dense-only puts the gold chunk in the context for three of them, and a BM25 arm fused into the same candidate budget puts it there for all eight, with the runbook at rank 1 or 2 in seven cases. Eight queries is a small sample and the effect is the maximum possible size, which is the situation where a small sample is fine, since the difference between three and eight is not something noise produces.

That last sentence is worth making explicit rather than asserting, because it is the step a reviewer should be able to redo. The queries the two arms agree on carry no information about which arm is better; only the ones where they disagree do, and here every query the arms disagree about went the same way, five of them, with none going the other. Under a null hypothesis that the arms are equally good, each disagreement is a coin toss, and five tosses all landing the way these did happens about one time in thirty. The residual is real and worth stating: eight queries drawn from one query set are not eight independent draws from the traffic, and the effect measured here is at its ceiling by construction, so this licenses building the thing rather than forecasting how much it will move a production metric.

What the second index actually costs to run#

The build is the small part. A BM25 index over 1.1 million tokens is trivial, and over a real corpus of a few hundred million tokens it is a few gigabytes and an hour. What follows is the part that shows up in incidents.

Two indexes have to be written together, and they will drift. A document deleted from the vector store and left in the lexical one produces a citation to a page that returns 404, which reads to users as the assistant inventing sources. The mitigation is a single ingest transaction with a reconciliation job that compares document counts and identifier sets nightly, and an alert when they differ by more than a threshold you pick deliberately rather than by zero, since transient drift during a reindex is normal.

The query path fans out, so latency becomes the maximum of the two arms rather than the average, and a lexical index that is usually 4 ms will occasionally be 300 ms on a query with a very common term. Each arm needs its own timeout and a defined behaviour when it expires, which is the same discipline the reranker needed in chapter 2 and is forgotten for the same reason. Budget 15–40 ms added at p95 for the fan-out and merge on a healthy system, and put the arm's status in the trace.

One structure explains both of those numbers. A lexical index stores, for each term, a posting list naming every document the term occurs in, and building it is a single pass that counts terms and appends to lists, with no model in the loop and nothing to learn — which is why the build is cheap. Answering a query means walking the posting list of each of its terms, and a list holds one entry per document containing the term, so a very common term has a very long list and the query pays for its commonest term rather than its rarest. The same structure that makes the build trivial is what puts a heavy tail on the reads.

The 15–40 ms budget has a condition attached to it that is easy to lose. A fanned-out request finishes when its slowest arm finishes, so the request is inside a time limit only when both arms are, and the chance of that is the chance for one arm multiplied by the chance for the other. Two arms that each meet a time limit nineteen times in twenty therefore meet it together only about eighteen times in twenty, which means the limit that was each arm's ninety-fifth percentile is closer to the pair's ninetieth. The multiplication is only valid when the two arms are independent, and they are not independent when they share a host, a thread pool or a garbage collector — in which case the tails arrive together and the pair is worse still. Measure the fused percentile directly rather than composing it from the arms.

Paraphrase, and stale documents that match exactly#

A lexical arm does nothing for paraphrase, which is the failure dense retrieval exists to fix, and topic 8.13 made the case that the pair is only worth it where the two halves genuinely disagree. If your query log sweep finds almost no identifier traffic and your users write full sentences, the second index is cost without benefit and the correct design is dense-only.

The arms also swap roles at the edges of the identifier class, which is the residual the table's last column names as a word-form mismatch. A code the user has mistyped, or remembered as E_2231 when the document writes E_2231_TLSHANDSHAKE, is a string the lexical index cannot match at all, while the dense arm still lands in the right neighbourhood and may well be the better of the two. The same goes for codes whose text is ordinary language a tokenizer does have entries for.

It also does not interact with the previous chapter's problem in the direction people hope. A lexical match on E_2231_TLSHANDSHAKE will happily return the 2023 copy of the runbook alongside the current one, at a higher score if the old copy repeats the string more often. Exact matching finds strings, and status filtering decides which documents are allowed to be found; neither substitutes for the other, and a system that adds lexical retrieval to an unfiltered index makes stale documents easier to surface than they were before.

The old copy outranking the current one is the scorer working as designed. An inverted index raises a document's score when the query term occurs in it more often, and a superseded page carries no marker that would tell the scorer to hold back, because currency is not a property such an index represents. That is the failure at the top of this chapter arriving from the other side: the instrument measures precisely what it was built to measure, and the property the reader needs is a different one.

There is one more belief worth naming, because it is the reason this failure survives review. A team looking at an aggregate retrieval metric sees it barely move when the identifier slice collapses, since that slice is a small share of traffic and the aggregate averages it away. The metric holding steady is read as evidence that nothing is wrong, when what it actually shows is that the metric cannot see the slice. The only reading that works is the sliced one, which is why the sweep comes before the fix rather than after it.

The two halves of a hybrid retriever are complementary for a single reason worth carrying out of this chapter. A string that occurs almost nowhere gives an embedding model nothing to have learned, and gives an inverted index its highest possible score, because a term that appears in one document out of many is the most discriminating evidence such an index can hold. The property that ruins one representation is the property that makes the other work, and that is why the pair is two instruments with different domains rather than a hedge against uncertainty — you can say in advance which one will answer a given query by looking at how rare its rarest token is.

Five chapters in, the list reads as a set of prohibitions: do not chunk at 128 tokens, do not stuff k=20, do not skip the reranker, do not embed everything, do not run dense-only. Each of those is the right design for some corpus, and the next chapter says which.