Semantic similarity scoring

Semantic similarity, in the form almost every suite uses it, is an embedding cosine: each text is turned into a vector of several hundred numbers by an embedding model, and the cosine of the angle between the two vectors is reported on a scale where 1.0 means they point the same way. It is the natural repair for the overlap metrics of the previous chapter, since it credits a paraphrase that shares no wording, and it is cheap enough to run on every example of every build.

Two steps hide inside "the cosine of the angle between the two vectors", and both matter to everything below, so they are worth doing rather than accepting. The raw comparison available between two lists of numbers is the dot product: multiply the two lists together position by position and add the results. That number is the length of the first vector, times the length of the second, times the cosine of the angle between them — three separate facts multiplied into one figure. Two of the three are properties of how the vectors were produced rather than of what the texts say, since a longer or more emphatic passage can come back as a longer vector without pointing anywhere new. Anything that puts a threshold on a raw dot product is thresholding that blend, and a text can clear the bar by being long rather than by being close.

Dividing the dot product by the two lengths removes exactly the two facts nobody wanted. The division is permitted because a length is never negative and, for any vector that is not all zeros, never zero, so the operation is always defined and cannot flip the sign of what it acts on. What survives is the cosine by itself, which is bounded between −1 and 1 whatever the two texts were. That boundedness is the reason a fixed threshold like 0.85 is a conceivable thing to write down at all: a number with a floor and a ceiling can carry a line, and a number that grows with the length of its inputs cannot. The intuition to carry forward is that every trace of size has been scrubbed out, leaving a statement about direction and nothing else.

What it answers is whether two texts are about the same thing. That is a real question, and it is not the question a correctness gate is asking. The failure of this model class is not an argument against using a model to judge text at all, because a cross-encoder does the job a bi-encoder cannot.

Semantic similarity answers "same topic", not "same facts"#

Here are four pairs from a claims assistant's 500-example suite, scored against a gate that passes an answer when its cosine against the reference answer is at least 0.85. The first column is what the reference says, the second is what the model said, the third is the gate's verdict.

reference                              prediction                             cos    gate
Your claim was approved on 14 March.   Your claim was not approved on         0.94   PASS
                                       14 March.
The deductible remaining is $1,250.00. The deductible remaining is            0.98   PASS
                                       $1,520.00.
Policy P-88214 covers this repair.     Policy P-88241 covers this repair.     0.99   PASS
Coverage applies after the excess is   Excess first, then coverage. The       0.79   FAIL
paid. Submit receipts within 30 days.  receipt window is 30 days.

The three passes are a reversed decision, a $270 error, and a policy number that belongs to a different customer. The failure is a correct answer written as two short clauses instead of two long ones. This is not a threshold set slightly too low; it is a measurement that assigns nearly the same value to a sentence and its negation, and a lower value to the same content rearranged.

The belief that table is aimed at is that a similarity score is a soft correctness score — that 1.0 would mean right, that 0 would mean wrong, and that the numbers in between grade how nearly right an answer is. It is tempting for three good reasons. It is continuous where exact match is binary, so it looks like strictly more information. It needs no shared wording, which is the precise defect of the overlap metrics and makes similarity feel like their repair rather than a different instrument. And it agrees with human judgement on every pair anyone tries first, because the first pairs anyone tries are a good answer against an answer about something else entirely. The belief passes its own smoke test, which is why it takes a distribution rather than an anecdote to dislodge it.

An embedding model is trained to place texts that are about the same thing near each other in a vector space, using objectives built from co-occurrence, paraphrase pairs and retrieval-style contrast. Under that training, the direction of a vector is dominated by the topic, the register and the entities present — the things that determine whether a document is a relevant hit for a query. A negation adds one short, extremely common token and leaves every other feature of the sentence untouched, so the vector barely moves. A digit substitution changes a numeric token that carries little of the sentence's semantic mass and leaves the rest identical. An entity substitution that preserves the entity's type — one policy number for another policy number — is nearly invisible for the same reason.

Laid out as steps, the chain from that training objective to the 0.94 in the table is short, and every link in it can be checked. The objective rewards putting together texts that co-occur, that paraphrase one another, or that answer one another. What varies between texts related in any of those ways is subject matter, entities and register. What does not vary is truth value, because a passage and a passage disputing it are related in exactly those ways too. The model therefore has no pressure to spend any of its dimensions on the difference between a claim and its denial, and considerable pressure to spend them on the properties that do separate its training pairs. A sentence and its negation differ by one short function word and share every entity, every content word and the entire register, so almost no dimension moves. A tiny movement is a tiny angle, and a tiny angle has a cosine close to 1.0.

The same mechanism produces something that surprises most people the first time they measure it, and it is the sharper form of the point: general-purpose embedding models place antonyms very close together. Two opposite words occur in nearly identical contexts, since the sentences either one appears in are otherwise the same sentences, so a training signal built on context has nothing available to separate them with. Opposition is not distance in these spaces. Two texts that contradict each other are, by every measurement the model was trained to make, about the same thing.

There is a reason underneath that one, and it is worth having before any of the repairs get tried, because it rules most of them out in advance. Correctness is not a property of a text. It is a relation — between a text and the world, or between a text and a source document — and the input to an embedding model is one text on its own. There is no position in the resulting vector where "contradicts the policy" could be recorded, because the second thing the relation is about was never present when the vector was computed.

The compression of corrupted and sound answers into the same narrow band of scores is easier to see as a distribution than as anecdotes. Take the 500 reference answers, embed them, and measure the cosine of each against its own prediction, then against a randomly chosen different reference from the same suite:

import numpy as np

pred = np.array([embed(p) for p in predictions])   # hosted embedding endpoint
ref  = np.array([embed(r) for r in references])
pred /= np.linalg.norm(pred, axis=1, keepdims=True)
ref  /= np.linalg.norm(ref,  axis=1, keepdims=True)

matched  = (pred * ref).sum(axis=1)
shuffled = (pred * ref[np.random.permutation(len(ref))]).sum(axis=1)

print(matched.mean(),  matched.std())    # 0.931  0.041
print(shuffled.mean(), shuffled.std())   # 0.706  0.078
print((matched >= 0.85).mean())          # 0.962

The two lines that divide by np.linalg.norm are the division worked through above, in the form that is cheapest at scale rather than the form it was described in. Each row is divided by its own length first, which leaves every vector pointing exactly where it did with a length of one. Multiplying two unit-length vectors position by position and summing then gives their dot product, and dividing a dot product by two lengths that are both one changes nothing — so the sum on the next line is the cosine, with no further step needed. Doing it in this order matters when there are 500 of them, because each vector is normalised once and then reused against every partner it is compared with.

Three numbers come out of that script, and the relationship between them is the whole argument. Matched pairs average 0.931, with a standard deviation of 0.041 — most matched pairs therefore sit within about four hundredths of that mean. Shuffled pairs, each prediction scored against somebody else's reference, average 0.706 with a standard deviation of 0.078. The gate at 0.85 sits a shade under two matched standard deviations below the matched mean, and a shade under two shuffled standard deviations above the shuffled mean. Being equidistant in those units is not an accident of the number chosen; it is what a line drawn to separate two populations looks like.

The last printed figure says the same thing from the other side. A threshold that far out into the lower tail of the matched cloud leaves 0.962 of matched pairs above it, so the ones falling below are the small remainder — of 500 examples, around twenty of them.

The threshold of 0.85 sits between those two clouds, which is exactly where a retrieval threshold belongs: it separates an answer about this customer's claim from an answer about someone else's mortgage. That separation is real and the metric is good at it. It is also a distinction the suite has already made, because each example was paired with its own reference by construction. The gate is spending its entire discriminating power on a question whose answer was fixed before the model ran.

The distinction it is being asked to make lives inside the matched cloud. A negated answer, a wrong amount and a swapped identifier all sit at 0.94 or above, comfortably inside the top half of that distribution, while a correct answer with an unusual structure sits at 0.79 and falls out. There is no threshold that separates the corrupted answers from the sound ones, because the two populations are not separated along this axis at all.

Every candidate line can be checked against the figures already on the page, and they all fail in one of two ways. Raising the bar to 0.95 puts it above the matched mean of 0.931, so it cuts through the middle of the population it exists to protect and fails hundreds of correct answers; it still passes the wrong amount at 0.98 and the swapped policy number at 0.99. It would catch the negation at 0.94, and that is the most that can be said for it — the catch comes from this one corruption happening to land below this one line, not from any property that sorts corruptions from sound answers. Lowering the bar passes everything, including the three defects. Moving the line changes which corruption survives and does not reduce how many do, because the corrupted answers are not sitting on one side of the sound ones to begin with.

Three numbers on one field are three questions#

The same misunderstanding shows up in a milder form on an extraction dashboard, where a single field carries three scores that look like a progression:

patient_dob    exact match 0.71    token-level F1 0.86    embedding cosine 0.97

Read left to right this looks like a strictness ladder — a harsh measure, a lenient one, and something in between — and the natural conclusion is to report the middle number as a balanced view. The three are not measuring the same property at different tolerances. They are answering different questions, and on this particular field two of the three answers are close to meaningless.

Exact match at 0.71 is the honest bad news: 29% of the extracted dates are not the string the consumer expects, and most of that is format variation of the kind type-aware equality handles. Token F1 is an F-score computed by splitting each side on non-alphanumerics and comparing bags of tokens, which for a date means the multiset {1985, 04, 12}. That comparison cannot see order, so 1985-04-12 and 1985-12-04 score a perfect 1.0 against each other — a transposed month and day, the single most common date-extraction error and the one most likely to survive to production, is invisible to it. When the date is partly wrong it awards partial credit: 1985-04-11 against 1985-04-12 scores 0.67, and two thirds of a date of birth is not two thirds of an answer. It is a wrong patient.

Both of those figures are worth computing rather than accepting, because the working is what shows the measure could not have behaved otherwise. Splitting 1985-04-12 on non-alphanumerics gives the three tokens 1985, 04 and 12, and splitting 1985-12-04 gives 1985, 12 and 04 — the same three tokens in a different order. A bag has no order, so every token on each side finds a partner. Precision is three matched out of the three the prediction offered, recall is three matched out of the three the reference held, and the harmonic mean of two numbers that are both one is one. The perfect score is not a rounding artefact or a threshold effect; it is the only answer the definition permits, because the two multisets are equal.

The partial case runs the same way. 1985-04-11 and 1985-04-12 agree on 1985 and 04 and disagree on the final token, so two of the three tokens on each side find a partner. Precision is two thirds, recall is two thirds, and the harmonic mean of two equal numbers is that number, which is the 0.67. What the arithmetic never reaches is which token disagreed. A wrong year, a wrong month and a wrong day each cost exactly the same two thirds, because the measure counts partners and holds no representation of what a position means.

Cosine at 0.97 is close to a constant. Every ISO date is a short string of digits and separators with no topical content to distinguish it from any other, so two unrelated dates of birth score around 0.96. The metric's entire dynamic range on this field is about three hundredths, and the observed 0.97 is one hundredth above the floor. It does not report accuracy; it reports that the model emitted something date-shaped.

The row that licenses that reading is the one giving each measure's value on two unrelated dates, and the idea in it generalises well past this field. Every measure has a floor, which is whatever it scores on a pair chosen to be wrong, and a reported value carries information only in the distance between it and that floor. Exact match scores 0.00 on two unrelated dates, so its whole range is available for saying something. Token F1 scores between 0.00 and 0.33, since two unrelated dates may happen to share a component, so most of its range is available. Cosine scores around 0.96 against a ceiling of 1.0, which leaves the few hundredths in between as the entire space it has to express any difference at all.

Reading a value without its floor beside it is what makes the three look like a ladder. Put the floors back and the ordering stops being a progression from strict to lenient and becomes a ranking by how much room each measure has left to say anything, on which cosine comes last by a wide margin. That question — what does this instrument score when it is deliberately given a wrong answer — costs one line to ask and is the fastest way to find out whether a number on a dashboard is reporting anything.

Which of the three belongs in a release gate is then straightforward, and it is none of them as written. The gate should be the parsed comparison — both sides through the consumer's date parser, equal dates or not — with exact match kept alongside as its strict lower bound, useful because a widening gap between the two tells you formatting drift is increasing. Token F1 comes off the dashboard for this field; it is defensible on long free-text fields such as an address, where recovering most of the components genuinely is partial success, and indefensible on a date. Cosine comes off outright, since a number that varies by 0.03 across every possible input cannot inform any decision.

The model class is wrong, not the idea of using a model#

Nothing above argues that a learned model cannot judge two texts. It argues that a bi-encoder cosine cannot, and the distinction is worth holding onto, because the failure has a specific cause with a specific set of alternatives.

A bi-encoder embeds each text independently into a fixed vector and compares the vectors, which is what makes it fast enough to index millions of documents: the corpus is embedded once, offline, and a query is one vector multiplication away from every hit. That independence is also the source of the blindness. The model never sees the two texts together, so it cannot represent a relation between them; it can only place each one in a space and let the geometry stand in for the relation. Negation, numeric disagreement and entity substitution are relations, not topics, and there is nowhere in a single-text vector for them to live.

It is worth being exact about which half of that is the design win and which is the cost, because they are the same property seen twice. Since each text is embedded without reference to any other, a corpus can be embedded once and the vectors kept on disk. A query then costs one embedding call plus arithmetic against stored numbers, and that arithmetic is a multiplication a machine performs billions of at a time. None of it survives if the score depends on the pair, because a pair-dependent score has to be computed per pair and cannot be precomputed for pairs that do not exist yet. A bi-encoder cosine therefore cannot arbitrate between two answers about the same subject, and the reason is not that it was trained badly or tuned wrongly. The independence that makes the index affordable is the same independence that leaves the relation unrepresented, which is why no quantity of training data moves it.

A cross-encoder reads both texts in one pass and outputs a score for the pair, which is why rerankers are cross-encoders and why they cost orders of magnitude more per comparison. A natural language inference model is the same architecture pointed at the right labels: given a premise and a hypothesis it returns entailment, contradiction, or neutral, and contradiction is exactly the verdict the negated claims decision needs. Running an off-the-shelf NLI model over the four pairs above labels the negation a contradiction and the reorganised answer an entailment — the two judgements the cosine gate got backwards.

That does not make NLI a drop-in gate. It compares a hypothesis against a premise you supply, so it answers "does this text follow from that one" rather than "is this true", and pointing it at a human reference answer only tells you whether the model's answer follows from one person's phrasing. Aimed at the source material instead, and applied to one assertion at a time, it becomes something much closer to a correctness check.

Fine-tuning the embedding model is the repair people try first and it does not reach far enough. Training on in-domain pairs tightens the matched cloud and improves retrieval, and it leaves the structural problem intact, because the objective still asks the model to place single texts near each other. A negation-sensitive embedding is a research problem, not a Tuesday.

Being honest about why that is hard is worth more than the verdict on its own, because "harder than it looks" is what people hear and it is not what is being claimed. The difficulty is not effort, or data volume, or picking better pairs. The objective that makes an embedding useful — pulling together texts about the same thing — is the same objective that pulls a sentence and its negation together, so the two are not independent settings that can be turned separately. Buying sensitivity to a negation means giving direction over to a feature the retrieval behaviour is actively trying to ignore, and the space a model has for expressing distinctions is finite, so a distinction added is not free. That is a trade for somebody training a model to make deliberately, and there is no version of it that arrives as a configuration change.

What similarity is genuinely good at#

None of this makes embeddings a bad tool. They are the right instrument for a family of jobs that all reduce to find me things in the same neighbourhood, and in those jobs the topical sensitivity that ruins a correctness gate is the whole point. Retrieving candidate passages for a retrieval-augmented generation pipeline, the RAG arrangement in which a model answers from documents fetched at request time, deduplicating near-identical support tickets, clustering a month of user queries to find the themes, flagging an answer that has wandered off the topic of the question entirely, routing a message to the right queue — cosine does all of these well, cheaply, and at a scale nothing else touches.

The dashboard version of that is worth keeping deliberately rather than by inertia. A similarity check earns a place next to an answer-quality number for exactly one purpose, which is detecting that the model has started answering a different question, and it earns it only if the number beside it is something that can fail on a wrong fact.

There is one number that makes the difference legible to anybody reading the dashboard, and it costs one extra line: report the metric on shuffled pairs alongside the real one. When a PM puts 0.97 on a customer slide as extraction accuracy, the thing to say is that two randomly paired dates from the same suite score 0.96 on it, so the number certifies that the field contains a date rather than the right one. The number that survives scrutiny is the parsed-date agreement — 93% on the current build, with the 7% broken out by cause — and it is a better slide anyway, because it is a claim the team can stand behind when a customer's auditor asks how it was measured.

Correctness is a property of the relationship between a claim and the world, and cosine measures the angle between two strings; there is no threshold on the second that implies the first. The gate at 0.85 was a topic-and-fluency check wearing a correctness check's badge, and it held for two months because it produced a plausible number in a plausible range every week.

That completes the survey. Exact match cannot see the formats a consumer treats as identical, and normalizing it into usefulness quietly invents equalities the system does not have. Type-aware comparison handles anything with a parser and has nothing to say about open text. Overlap counts shared wording, which is neither necessary nor sufficient. Similarity counts topical proximity, which is blind to the edits that flip truth. Each has been examined for what it cannot see, which raises the obvious question of what the check would look like that sees those things — the one all four have been implicitly compared against.