Shared topic drowns negation, which is why 0.94 is correct

An operational corpus is full of pairs of sentences that differ by one word and mean opposite things. A runbook — the written procedure an on-call engineer follows during an incident — says to restart a service, and three paragraphs later says not to restart it under some condition. Both sentences name the same service, the same command and the same subsystem. To a retrieval system built on embeddings they are as close as two sentences ever get, and this chapter is about why that number is right, why the two things people reach for to fix it cannot work, and what does.

The number in question is a cosine, the similarity score a vector search returns, running from −1 to 1. Two sentences that are exact negations of each other land around 0.94, and reading that as alarming is the first thing to correct, because a system designed around the belief that 0.94 is a bug will spend its effort on the model and none of it on the parts of the pipeline that can actually help. The claim to hold onto is narrower and more useful: the score is measuring subject matter accurately, and polarity was never among the things being measured.

The number, and how far it generalises#

pairs = [
    ("The API returns 200 on success.",
     "The API does not return 200 on success."),
    ("Restart the primary with pg_ctl restart after the checkpoint completes.",
     "Do not restart the primary with pg_ctl restart after the checkpoint completes."),
]
for a, b in pairs:
    va, vb = embed([a, b])            # text-embedding-3-small, unit norm
    print(round(float(va @ vb), 3))
# 0.941
# 0.968

The vector arithmetic topic already showed that a negation scores high, so the number itself is not the news. What matters for a system built on chunks is how the number moves with length, because the single distinguishing token's share of the pooled vector shrinks as everything around it grows.

The third row is the operationally important one. At chunk sizes anyone actually uses, a passage and the same passage with one instruction reversed are 0.994 apart, which is well inside the range where floating-point noise and a different tokenisation of a stray character can reorder them. The fourth row says the space is doing something sensible in the meantime: two genuinely different chunks from the same runbook sit at 0.812, comfortably below either negation pair, so the ranking is dominated by subject matter exactly as the training objective specified.

Reading 0.94 as alarming comes from treating the score as a percentage of agreement, as though the model were reporting that the two sentences are 94% the same claim. The score reports the cosine of an angle in a space whose axes were arranged to put related subject matter together, and the range it actually occupies is narrow: unrelated English sentences from this model land around 0.04 to 0.2, and anything about the same system in the same register starts at 0.7. On that scale 0.94 says the pair is unusually similar in subject matter, which is true of a sentence and its negation by construction.

The mechanism is pooling, from chapter 1, applied to a difference of one token. Mean pooling gives each token position a weight of 1/n in the final vector, so the perturbation from inserting not is bounded by roughly one token's worth of magnitude out of nine, or out of a hundred and eighty. That bound is worth testing against the table rather than accepting, because the table can check it. Read the first and third rows as distances from perfect agreement rather than as levels: the shortfall below 1 is about six hundredths at nine tokens and about six thousandths at a hundred and eighty-four, so it shrank by a factor of ten. The token count over the same two rows grew by a factor of twenty. If the negation's influence were purely its own 1/n share of the mean, those two factors would match; the shortfall falls only half as fast as pure averaging predicts, and the difference is attention.

Attention propagates the negation into the neighbouring token representations rather than confining it to its own position, so the changed word contaminates several positions instead of one, which is why the number is 0.941 instead of 0.99 at nine tokens. That propagation is a fixed, small multiple rather than a growing one, so it slows the collapse without preventing it. The size of the propagation is set by what the contrastive fine-tune rewarded, and chapter 2's account of the pair sources says what that was: topical relevance, at enormous scale, with negatives drawn from unrelated documents. No gradient step ever asked this model to move a sentence away from its own negation, so the geometry contains no direction that does it.

Both of the obvious escapes are closed#

Switching the distance metric cannot change a ranking, and the reason is four lines of expansion. Squared Euclidean distance between two vectors is the dot product of their difference with itself: ‖a − b‖² = (a − b)·(a − b). Multiplying that out term by term gives a·a − 2(a·b) + b·b, where the cross terms combine because a·b and b·a are the same number. For unit vectors a·a and b·b are both 1 by definition, since the dot product of a vector with itself is its squared length. Substituting leaves ‖a − b‖² = 2 − 2(a·b), which the vector arithmetic topic established in general.

Read what that expression permits. The only place the pair appears is inside a·b, multiplied by −2, so squared distance is a strictly decreasing function of cosine: raise the cosine and the distance falls, with no exceptions and no region where the relation reverses. Taking a square root cannot reorder anything either, since the square root is increasing over non-negative numbers. So sorting a candidate list by ascending distance and sorting it by descending cosine produce the same sequence, item for item. Applied here it means the negation sits second in both orderings and always will:

import numpy as np
V = embed([query, claim, negation, other_chunk])   # all unit norm
q, rest = V[0], V[1:]

cos = rest @ q
l2 = np.linalg.norm(rest - q, axis=1)
print(np.argsort(-cos), np.argsort(l2))     # [0 1 2] [0 1 2]
print((2 - 2 * cos - l2 ** 2).max())        # 2.4e-07

The last printed number is the identity checked against the machine: the largest discrepancy between the algebra and the measured distances is 2.4e-07, which is float32 rounding rather than disagreement. The two argsort outputs are the same permutation, which is the consequence stated as data.

The reason the metric swap keeps getting proposed is that the names sound like they describe different notions of meaning, when they describe different formulas over the same coordinates. Nothing in the choice between them consults the text; the vectors were fixed before the metric was applied, and a function of a number cannot recover information the number never carried. This is the strongest form a bad defence takes: it is not merely weak against negation, it is exactly constant in it. Whatever the polarity of the two texts, the metric swap returns the identical ordering, so no experiment on any corpus could ever show it helping.

Inner product on non-normalised vectors is the one metric change that genuinely reorders results, and it does so by reintroducing magnitude, which correlates with chunk length rather than with polarity. Whichever of the three you pick, the polarity of a sentence is not among the things being compared.

Adding dimensions fails for a different reason. Capacity and relation are separate questions, and the missing thing is a relation. Running the same short pair through four models of different sizes, listed with their average retrieval score on MTEB — the public Massive Text Embedding Benchmark that most model cards quote — shows the number barely moving:

The right way to read that table is to look for the cell that is missing. If capacity bought separation there would be a row where a larger dimension count produces a lower negation cosine, and if benchmark quality bought separation there would be a row where a higher MTEB average does. Neither exists. The dimension column spans a factor of eight and the last column does not descend with it; the benchmark column spans over thirteen points and the last column does not descend with that either. The model with seven times the dimensions of the smallest and the best retrieval benchmark average in the group scores the pair at 0.933, which is not a meaningful improvement on 0.918 and is nowhere near a separation. The highest negation cosine in the table belongs to a mid-sized model, which is what a column looks like when it is unrelated to the two beside it rather than weakly related to them.

More coordinates give the encoder more room to place topics apart from one another; a direction that encodes "this sentence asserts the opposite of that one" would have to be created by training pairs that demand it, and the extra room does not conjure them. This is also why "use a better model" stalls: the leaderboard the model climbed measures topical retrieval, so improving on it optimises the axis that was already working.

What a runbook corpus actually needs#

Everything that fixes this sits outside the vector, in a stage that can look at two texts together rather than at one point each. That phrase is the whole of the design constraint, and it follows directly from the last chapter's floor: a stored vector was computed from one text with no access to the other, so a relation between two texts cannot be read out of it by any later arithmetic. Two of the options below are named after that stage: a reranker is a second model that rescores the handful of candidates vector search returned and reorders them, and an entailment check asks an NLI model whether one text follows from, contradicts or is neutral towards another. The realistic options differ in what they catch and in what they cost:

Putting numbers on the top two rows is worth the afternoon it takes. Labelling 240 queries drawn from the on-call channel with the polarity the correct answer needs, and measuring how often the top-ranked passage carries that polarity, gave 61% for dense retrieval alone and 89% after reranking the top 50 with bge-reranker-base. Adding an NLI pass over the surviving top 5 took it to 93% and cost 18 percentage points of the pipeline's p95 latency, the figure 95% of requests come in under, from 310 ms to 366 ms. Those are the numbers to bring to the decision, since the argument otherwise runs on whether a reranker "feels" necessary. Read the two improvements as steps rather than as a total: the reranker buys the large majority of what is available, and the NLI pass buys a much smaller further slice at a cost that is a visible fraction of the latency budget. Whether that second step is worth taking is a question about what the wrong polarity costs in this particular system, and the pair of figures is what makes it answerable.

A reranker is the usual first move because it improves ordinary retrieval quality as well, and because it changes one component rather than the corpus. It is a cross-encoder, meaning it takes the query and the candidate passage concatenated as a single input and outputs a relevance score, so the two texts are in the same forward pass and their relation is available to the model. That is the property a bi-encoder embedding cannot have, since each side was encoded without knowledge of the other.

The branch on the right is the part that gets designed wrong. The instinct after an incident of this kind is to filter out passages containing negation terms, and that discards the answer, because for a query asking whether an action is safe the prohibition is the correct passage. What the pipeline needs is for both the instruction and its prohibition to reach the generation step with their polarity labelled, so the decision is made where both texts are visible. Retrieval's job is to not lose either one.

Chunking deserves more credit than it usually gets in this list, since it addresses the 0.994 row instead of compensating for it. Splitting runbook sections so that each imperative and each prohibition is its own chunk, with the section heading prepended for context, moved the claim-versus-negation cosine in this corpus from 0.994 back to around 0.95 and made the reranker's job tractable, because a reranker over 50 candidates cannot separate two passages that are 99.4% identical either. Notice which quantity that change moves. It does not teach the model anything about polarity; it shortens the text so the distinguishing token gets a larger share of the pooled mean, which is the same 1/n arithmetic from earlier run in the useful direction.

What is still broken after all of it#

None of these remedies teach the embedding anything. They add a second stage that can see relations, and every one of them is bounded by what the first stage returned: if the prohibition ranks 84th and you rerank the top 50, the reranker never sees it, and the recall problem is upstream of everything in that table. Retrieval depth, hybrid search and chunking are the levers there, and each costs latency.

That bound is worth stating as a rule, because it decides where to spend effort. A reranker is a precision instrument and cannot raise recall; it can only reorder what arrived. So the two figures to watch are different in kind: how often the right passage is anywhere in the candidate set, which only the first stage controls, and how often it ends up on top, which the second stage controls. Reporting a single accuracy number over the pipeline hides which of the two moved, and a team that improves reranking while the candidate set quietly degrades will see the number hold steady and the failures get worse.

Negation is also one member of a family. The same argument — a single vector has no channel for a relation between two texts — covers numeric magnitude, so "retry after 30 seconds" and "retry after 300 seconds" sit at 0.97; it covers temporal validity, so a deprecated procedure and its replacement are near-identical; it covers argument roles, dates and units. A system that solves negation with a negation-term list has solved one instance and will meet the next one as a fresh incident, because the list is a patch over one surface form of a structural gap.

So 0.94 is the model reporting, correctly, that two sentences are about the same thing, and the two instincts it provokes are both dead ends for reasons that can be checked before spending anything. The metric swap is an identity on unit vectors, provable in four lines and confirmed to seven decimal places by the code above. More dimensions and a better benchmark average move the negation cosine around within a few hundredths, in no consistent direction, because the benchmark scores the axis that already works. What is left is everything outside the vector: a chunking policy that stops packing an instruction and its prohibition into one average, and a second stage that reads two texts at once.

Having changed the chunking and added a reranker, the natural next step is to check whether the space looks healthier than it did, and the tool everyone reaches for is a scatter plot of the vectors in two dimensions. That plot answers a different question from the one it appears to answer.