Phrase queries and code-like tokens

Semantic search ranks by proximity between embeddings rather than by words a query and a document share, and its tolerance for surface variation is the property being bought. Two strings from the same deployment runbook, embedded with the same 1024-dimension model, show what that tolerance costs:

A: "deploy to staging before production"
B: "deploy to production before staging"

cosine(A, B) = 0.971

One of those describes a release process and the other describes an outage. A retriever ranking by cosine treats them as very nearly the same request, so a user who typed one has a good chance of being handed a chunk containing the other. This is not a defect in that particular model; it is what the training objective asks for. Embedding models are trained contrastively on pairs a human would call equivalent, and word-order permutations of the same content words are, for almost every pair in almost every training set, equivalent. The model learned to ignore the thing that distinguishes A from B because ignoring it was rewarded ten million times. Past the rare identifiers of the previous chapter, two further classes lose to the same tolerance, and the three together cover most of what a lexical index buys a system that already has semantic search. All three arrive at the same symptom from different places, which is why dense retrievers are hard to debug from their output alone: a plausible neighbour first and the correct chunk somewhere below the cut.

How order-insensitive your particular model is happens to be measurable in about twenty lines, and running the test on your own documents is more persuasive than any general claim:

import itertools, random
from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("BAAI/bge-large-en-v1.5")

def permutation_gap(sentence: str, trials: int = 8) -> float:
    words = sentence.split()
    base = model.encode(sentence, normalize_embeddings=True)
    sims = []
    for _ in range(trials):
        shuffled = words[:]
        random.shuffle(shuffled)
        sims.append(float(base @ model.encode(" ".join(shuffled),
                                              normalize_embeddings=True)))
    return sum(sims) / len(sims)

print(permutation_gap("deploy to staging before production"))   # 0.958
print(permutation_gap("retry on 429 but not on 500"))           # 0.949
print(permutation_gap("the reconciler reads from the ledger"))  # 0.941

Randomly shuffling the words of a sentence — destroying the syntax entirely — moves the embedding by four to six hundredths of a cosine point. Whatever the model is representing, word order contributes a few percent of it, and a ranking decided by cosine will resolve almost entirely on which content words are present. Transformer encoders do have positional information available to them, so this is a property of what the training objective rewarded rather than an architectural limit, and it is stable enough across current general-purpose embedding models to plan around.

Word order that carries the meaning#

The class is queries where the content words are the same in the right answer and the wrong one, and the relationship between them is what the user is asking about. Retry policies, ordering constraints, precedence rules, and any documentation of the form "X takes priority over Y" all sit here.

A concrete case from a payments corpus: a client library retries on 429 and does not retry on 500, and exactly one chunk says so. Querying for it produced the following:

query: "retry on 429 but not on 500"

vector search, top 4
  0.912  "Error handling: the client retries on 500 but not on 429"   <- inverted
  0.897  "Retry and backoff configuration"
  0.884  "HTTP status codes returned by the payments API"
  0.881  "Idempotency keys and safe retries"
  ...
  0.842  "Retry semantics: 429 is retried, 5xx is surfaced"           <- correct, rank 11

The top result states the opposite of the truth with a cosine of 0.912, and an agent given that chunk will write code that retries the wrong status class. The correct chunk is at rank 11.

Put that gap next to the permutation measurement and it becomes clear that no adjustment to the ranking recovers this. The inverted chunk sits at 0.912 and the correct one at 0.842, a distance of seven hundredths. The permutation test says the entire contribution of word order to this model's embedding is four to six hundredths — that is how far a sentence moves when its syntax is destroyed completely. So the whole of the signal that could distinguish the correct chunk from the inverted one is smaller than the distance the correct chunk would have to travel. Even a version of this model that weighted word order as heavily as it possibly could, within the range this test measures, would not reorder those two. The failure is not that the model weighs order too lightly. It is that the quantity being weighed is smaller than the gap it must close.

Lexical retrieval does not automatically win this either — BM25 scores terms independently, as established when we looked at what the formula does not model, so retry 429 500 scores the inverted chunk and the correct chunk almost identically. What wins it is a phrase query, a different query construct built on the positions stored alongside the postings:

{
  "query": {
    "bool": {
      "should": [
        { "match_phrase": { "body": { "query": "429 is retried", "slop": 2 } } },
        { "match_phrase": { "body": { "query": "retries on 429",  "slop": 2 } } },
        { "match": { "body": "retry 429 500 backoff" } }
      ]
    }
  }
}

slop is the number of position moves allowed between the query terms, so slop: 2 matches "429 is always retried" as well as the literal phrase. With that clause the correct chunk moves to rank 1 and the inverted chunk drops out of the top 5, because the inverted chunk contains 429 and retried but not in that arrangement. Positions cost index space — enabling them roughly doubles a postings list, and index_options: docs versus positions is a real storage decision — and this is what you buy with them.

What positions add is the one thing a bag of terms cannot hold, and the reason the phrase clause works follows directly from it. A postings list without positions records that a document contains 429 and that it contains retried, which is true of both chunks. With positions it records where, so the engine can ask whether the two terms sit within a stated number of moves of each other. The inverted chunk contains both terms and arranges them the other way round, so the distance between them under the query's ordering is large and the clause does not fire. Nothing about the scoring changed; the index simply started storing a fact it had previously been discarding, and that fact is exactly the one the query turns on.

Phrase matching is also more expensive at query time than the term matching described earlier in this topic, because after the postings intersection identifies candidate documents the engine must walk the position lists inside each candidate and check adjacency. On a two-term phrase over a collection where both terms are common, that second pass dominates: a phrase query on connection refused runs roughly 4 to 8 times slower than the equivalent boolean query on the same index. The reason the multiplier lands there rather than somewhere smaller is that the second pass is per candidate rather than per query — two common terms produce a large candidate set, and every member of it has its position lists read and compared. On a phrase whose terms are rare the candidate set is tiny and the second pass costs almost nothing, which is why the same construct can be free on one query and expensive on the next. Elasticsearch's index_phrases mapping option trades space for that time by indexing two-term shingles as ordinary terms, so 429 is becomes a single dictionary entry and the phrase check becomes a term lookup, at a cost of roughly 20 percent more index size on the field. The limit of that trade is in the word two: a shingle covers exactly the adjacent pair it was built from, so a three-term phrase still has to fall back on positions, and turning shingles on does not let you turn positions off.

The bigger practical problem is that a phrase clause has to be constructed, and users do not type quotation marks. Generating the candidate phrases from the query — as in the should block above, where two orderings of the same relation are both tried — is a small piece of query-side logic, and the cheap version is to shingle the user's query into overlapping bigrams, the adjacent word pairs of what was typed, and add each as an optional low-boost phrase clause. That raises recall on order-bearing queries without harming anything else, since a phrase clause in a should position only ever adds score.

Code-like tokens, where near-misses are worse than misses#

The second class is function names, configuration keys, flags, environment variables and SQL fragments. What makes it distinct from the rare-identifier case is that the wrong answers are not merely unhelpful; they are plausible enough to be used.

query: "max_connections"

vector search, top 5
  0.893  max_pool_size            (different subsystem, different semantics)
  0.881  maxConnections           (a deprecated camelCase alias)
  0.877  connection_limit         (per-tenant quota, not a server setting)
  0.864  max_connections          <- correct, rank 4
  0.859  max_client_conn          (pgbouncer, not postgres)

Every one of those is a real configuration key that exists somewhere in that corpus, and all five are within 0.034 cosine of each other. An agent that retrieves the top three and writes a config change gets max_pool_size, which is syntactically valid, silently ignored by the server it was written for, and produces a bug that surfaces under load a week later. A retrieval miss that returns nothing is recoverable; a retrieval miss that returns a neighbouring key that looks correct is not, because nothing downstream flags it.

The spread is the reading that matters, and it is worth computing rather than eyeballing. Top to bottom, those five results span a little over three hundredths of a cosine point, which is less than a random shuffle of a sentence costs on this model. In other words, the whole difference between the correct configuration key and four keys belonging to other subsystems is smaller than the difference between a sentence and the same sentence with its words in a bag. That is the geometry doing exactly what it was built to do: names that describe similar things sit close together, and these five names describe similar things. The model is not confused. It is answering a question about resemblance correctly, and resemblance is the one property the user was not asking about.

The lexical answer here is the keyword field from earlier in this topic, queried exactly:

res = es.search(index="docs", size=5, query={
    "bool": {
        "should": [
            {"term":  {"config_key.exact": {"value": "max_connections", "boost": 12}}},
            {"match": {"body": "max_connections"}},
        ]
    }
})

max_connections has df = 61 on that corpus against 4.4 million chunks, so its IDF is 11.19 and the exact-term clause puts the right chunk first with a score gap of 4.3x over the second result. The keys that confused the embedding do not contain the string at all and score zero on the term clause.

That document frequency lands the query on the right side of the threshold from the previous chapter, which is the check to make before promising anybody an outright win. The boundary was around a thousand documents, the point at which an identifier's inverse document frequency drops to what a saturated common word can reach. Sixty-one is two orders of magnitude inside it, and the resulting 11.19 is well above the 8.4 that was the best any ordinary term in that budget table could manage. So the outright win here is predicted by one lookup rather than discovered by running the query, and a sibling key that happened to appear in fifty thousand chunks would have been predicted not to win.

The three failure mechanisms are different even though the symptom is the same. The first is a vocabulary problem in the embedding model's tokeniser, the second is a property of the contrastive training objective, and the third is the geometry of a space where semantically related names sit close together by design. No single change to the dense side fixes all three, which is the reason a larger embedding model is a poor first response to any of them.

The argument against the larger model is worth stating in its strongest form, because it is the proposal that will be made in the room. Size changes the capacity of a model; it does not change what the model was rewarded for. Every one of these three mechanisms traces to the objective rather than to the capacity. A tokeniser without an entry for an invented string still has no entry for it at a larger size. A training set in which permutations of the same content words are labelled equivalent still labels them equivalent at a larger size. A space in which sibling configuration keys are neighbours is a space that was asked to put similar names near each other, and doing that better is doing it more thoroughly. So a bigger model is a defence whose result does not move when the thing it must discriminate changes, and that is the property to test for before spending on one: run the permutation measurement on the candidate model, and if the number is the same, the money buys nothing on these three classes.

Keeping them as standing cases#

A one-off comparison decays. The version that keeps paying is a file of cases and a test that runs them, so that a change to the analyser, the chunker, the embedding model or the b parameter has to explain itself:

# retrieval_cases.yaml
- id: rare-string-err-conn-refused
  query: "ERR_CONN_REFUSED"
  gold: "chunk:sidecar-error-codes#3"
  class: rare_string
  require: { lexical_rank_at_most: 1, vector_rank_at_most: null }

- id: order-bearing-retry-429
  query: "retry on 429 but not on 500"
  gold: "chunk:payments-retry-semantics#1"
  class: order_phrase
  require: { lexical_rank_at_most: 2, vector_rank_at_most: null }

- id: code-token-max-connections
  query: "max_connections"
  gold: "chunk:pg-server-config#7"
  class: code_token
  require: { lexical_rank_at_most: 1, vector_rank_at_most: null }

import pytest, yaml

CASES = yaml.safe_load(open("retrieval_cases.yaml"))

@pytest.mark.parametrize("case", CASES, ids=lambda c: c["id"])
def test_lexical_holds_its_cases(case):
    ranks = {
        "lexical": rank_of(bm25_search(case["query"], size=20), case["gold"]),
        "vector":  rank_of(vector_search(case["query"], size=20), case["gold"]),
    }
    limit = case["require"]["lexical_rank_at_most"]
    assert ranks["lexical"] is not None and ranks["lexical"] <= limit, (
        f"{case['id']}: lexical rank {ranks['lexical']} exceeds {limit}; "
        f"vector rank was {ranks['vector']}"
    )
    record_metric(case["id"], ranks)      # keep the history, not just the assert

Asserting on the lexical rank and merely recording the vector rank is deliberate. The lexical bound is a contract — if max_connections stops ranking first, something in the analysis chain changed and you want the build to say so. The vector rank is an observation, and turning it into an assertion produces a flaky test that fails whenever anyone re-embeds, without telling you anything actionable.

The asymmetry follows from what each side is made of. A lexical rank is a function of the terms in the index and the constants in the similarity, and both of those change only when somebody edits a mapping or a config file — so a lexical rank that moves points at a specific, recent, human action. A dense rank is a function of every vector in the collection, and re-embedding regenerates all of them at once, so a change of model moves every rank in the file simultaneously and in no particular direction. An assertion on that is not measuring the change under test; it is measuring that a change happened, which the deploy already told you.

Three cases is a floor rather than a target. Finding candidates for new ones does not require reading logs by hand, because two of the three classes are recognisable by shape:

import re

CODE_TOKEN = re.compile(r"""
    \b(?: [a-z]+(?:_[a-z0-9]+)+          # snake_case config keys
        | [a-z]+(?:[A-Z][a-z0-9]+)+      # camelCase function names
        | [A-Z]{2,}(?:_[A-Z0-9]+)+       # SCREAMING_SNAKE error codes
        | [a-z]+(?:-[a-z]+){1,3}         # hyphenated service names
    )\b""", re.X)

ORDER_WORDS = re.compile(r"\b(before|after|but not|instead of|rather than|"
                         r"unless|precedence|overrides?|takes priority)\b", re.I)

for q in query_log:
    if CODE_TOKEN.search(q):
        bucket("code_token", q)
    elif ORDER_WORDS.search(q):
        bucket("order_phrase", q)

The queries that go the other way#

Presenting these three classes without their counterweight would be dishonest, since they are a minority of traffic on almost every corpus. The same harness over the same index, on paraphrase-shaped queries, produces the mirror image:

The mechanism is the exact inverse of the three classes above. Where an embedding's tolerance for surface variation destroys the distinction between max_connections and max_pool_size, that same tolerance is what lets it connect "stop working after a while" to "expire after 30 minutes of inactivity" across a vocabulary gap with no shared content word at all. Blur is not a defect that a better model will eliminate; it is the property being purchased, and the three classes in this chapter are its price.

The two shares measured in the scenario add up in a way that is worth checking against this paragraph, because it is what keeps the chapter proportionate. Code tokens were 18 percent of four weeks of traffic and order-bearing queries 4 percent, which is twenty-two percent between them, and the rare-identifier class from the previous chapter sits on top of that. Seventy percent of traffic phrased as a question in the user's own words leaves roughly the right amount of room for those classes and for everything belonging to none of them. Two independent counts, taken for different reasons, describing one traffic mix without contradicting each other is the evidence that neither was chosen to make an argument.

Zero term overlap means zero BM25 score, and no tuning of k1, b or the analyser reaches these. On a query log where 70 percent of traffic is phrased as a question in the user's own words, an index that only does lexical matching loses badly on volume even while winning decisively on the three classes above.

Word order and code-like tokens fail for different reasons than rare identifiers do, and all three fail the same way from the outside: a plausible neighbour at rank 1 and the correct chunk somewhere below the cut. What separates them in practice is the construct that recovers each one — positions and a phrase clause for order, an unanalysed field and an exact term for code tokens — and the fact that both constructs are configuration rather than research.

Measuring the share of your traffic each class represents is what keeps this proportionate, and the honest result on most corpora is that these are a minority worth a day's work rather than a redesign. Running the comparison the other way, on the volume that dense retrieval owns, requires knowing what the dense side is doing well enough to configure and cost it, rather than treating it as an opaque service that returns cosine numbers.