Before and after score 0.94, and so do paraphrases

Before anything ships, you spend an afternoon on the cheapest experiment available: take a week of real questions from the order-support surface, find what each one would have matched against, and label whether the match was safe. Twelve thousand questions came through in that week, deduplicated to 8,300 distinct strings. For each question you embed it, search the questions that arrived before it, keep the nearest neighbour, and write down two things — the cosine score, and whether a human who reads both questions says the first one's answer would have been correct for the second.

Nine hundred pairs came back with a score above 0.90, which is the region any deployment would be operating in. Of those, 604 were genuine paraphrases where the same answer serves both, and their scores run from 0.92 to 0.97. The other 296 differed by a single qualifier — before against after, my against our, this year against last year, standard shipping against express — and would have needed different answers. Their scores run from 0.93 to 0.96. The dangerous band sits entirely inside the safe one.

The measurement takes an afternoon and settles the question#

The harness is short, and worth writing before the cache rather than after the incident, because it is the same code you will want for the threshold sweep later.

import numpy as np

def mine_pairs(questions: list[str], vecs: np.ndarray) -> list[dict]:
    """For each question, find its nearest earlier neighbour. Chronological order."""
    pairs = []
    for i in range(1, len(questions)):
        sims = vecs[:i] @ vecs[i]              # unit vectors -> cosine
        j = int(np.argmax(sims))
        if sims[j] >= 0.90:                    # only pairs a cache could admit
            pairs.append({"new": questions[i], "cached": questions[j],
                          "score": float(sims[j]), "safe": None})
    return pairs

# Label `safe` by hand, or with a strong model plus a human audit of 100.
# Then bucket:
for lo in [0.90, 0.92, 0.94, 0.96]:
    band = [p for p in pairs if lo <= p["score"] < lo + 0.02]
    unsafe = sum(1 for p in band if p["safe"] is False)
    print(f"{lo:.2f}-{lo+0.02:.2f}  n={len(band):4d}  unsafe={unsafe/len(band):.0%}")

Bucketed, the week of traffic looks like this.

Labelling 914 pairs sounds heavier than it is. A reviewer who knows the product handles roughly 350 an hour, because most judgements are obvious on sight and only the ambiguous ones need thought, so the whole sample is under three hours of one person's time. Labelling them with a strong model and hand-auditing a hundred is a reasonable substitute when the sample is larger; on this corpus the model agreed with the human on 94 of 100, and all six disagreements were pairs the human also called borderline. What you cannot substitute is the sampling: the pairs have to come from real traffic in arrival order, because a set of paraphrases someone writes by hand will contain the paraphrases they thought of and none of the qualifier collisions that make the distribution overlap.

A few of the pairs sitting at 0.95 are worth reading out loud, because the aggregate makes them abstract. "Do I get charged for the return label" against "do I get charged for the replacement" scored 0.95 and needs two different answers from two different policy sections. "When will my refund arrive" against "when will my replacement arrive" scored 0.96. "Can I change the delivery address" against "can I change the delivery date" scored 0.95. In each case a human reading the two sentences side by side would not hesitate for a second, and the vectors are as close as two phrasings of the same question.

The shape to notice is that the unsafe share stops falling somewhere around 0.93 and then flattens. Between 0.92 and 0.98 the proportion of dangerous matches barely moves, from 33% down to 27%, while the number of pairs available to admit collapses from 341 to 84. Every point of threshold you spend above 0.94 buys you almost no additional safety and costs you most of your volume. That flat region is what "the bands overlap" means when you stop describing it as two ranges and look at the density.

Why the two words that decide the answer barely move the vector#

The pairs that survive to high scores are the ones where the topic is identical and the difference is grammatical. Both sentences in the cancellation pair are about cancelling an order, and the seven tokens they share dominate a pooled representation into which "before" and "after" each contribute a single position. Those two tokens are also unusually poor at carrying distance, because they are high-frequency function words that appear in the same syntactic slot, and a model trained to put a query near the document that answers it has never been shown a training pair whose negative differs only by a temporal preposition. Retrieval corpora do not contain that distinction, so the objective never asks for it.

There is a second effect that makes the numbers read as more comfortable than they are, and it is worth calibrating on. Sentence embeddings from these models are anisotropic — they occupy a narrow cone of the space rather than spreading over the sphere — so cosine similarity has a compressed dynamic range. On this corpus, two questions picked at random from the support queue with genuinely unrelated intents ("where is my package" against "how do I change my billing address") score 0.78. A question about orders against a question about a completely different product line scores 0.71. The floor is not zero, it is somewhere near 0.75, and the entire interesting region is 0.75 to 1.00. Read against that scale, the gap between a paraphrase at 0.94 and a policy-inverting qualifier at 0.94 is not a small difference the model nearly captured; there is no difference at all.

The instinct at this point is that the embedding model is underpowered, and the fix is a larger one. It measures out at 0.93 for the cancellation pair with text-embedding-3-large against 0.94 for -small, and 0.95 with a well-regarded open bi-encoder, meaning a model that embeds each of the two sentences on its own and compares the resulting vectors, rather than reading both together and scoring the pair. The ranking of the pairs does not change, and the paraphrase band moves with the qualifier band because both are made of the same kind of sentence. Larger models are trained on the same objective against the same kind of corpus, so they are better at the thing they were asked to do and no better at the thing nobody asked for.

No threshold separates them, and 0.97 is the worst place to stand#

The conservative-sounding move is to raise the bar until the dangerous pairs fall outside it. Set the threshold at 0.97 and look at what is on either side. Of the 604 genuine paraphrases, 43 score at or above 0.97, so 93% of the legitimate hits you were building this for are now misses and the cache is serving something like 2% of eligible traffic. Meanwhile the qualifier-differing pairs top out at 0.96 as a population, but their density is highest at the top of their range, precisely because the pairs that share the most words are the pairs that differ by the least — a single swapped qualifier is the most lexically similar way for two questions to mean different things. The matches that squeeze past a high bar are disproportionately drawn from the class you were trying to exclude.

That is the arithmetic behind the claim that raising the threshold is the worst of both. You give up the savings that justified the project and you concentrate the remaining traffic on the highest-risk matches, which leaves you defending a cache that saves 2% of spend while its surviving hits carry a worse error profile than the ones you rejected.

Lowering the threshold is at least honest about what it is doing. At 0.92 you serve 34% of eligible traffic and roughly a third of those hits are answers to a materially different question, which on 12,000 weekly questions is somewhere around 1,300 wrong answers a week reaching customers who asked about cancellation windows and refund eligibility. Nobody proposes that once it is written in those units, and the units are the subject of the last chapter of this topic.

What is actually available on a workload like this#

Three things remain on the table once the threshold is off it.

The slot-normalised key is the one most teams skip and the one that fits this surface best. If the questions in the queue reduce to a modest set of intents with a small number of qualifiers each, then parsing "can I cancel my order after it ships" into ("cancel_order", timing="post_shipment") and using that tuple as an exact key gives you the paraphrase-collapsing behaviour you wanted from embeddings, with the difference that its mistakes are classification errors you can count, hold a confusion matrix for — the grid that records how often each intent was read as each other intent — and fix by adding training data. It is no longer a cache in the sense that the word usually means, and calling it one keeps the conversation on the wrong axis.

Fine-tuning the encoder on hard negatives is the version of this that actually moves the embedding numbers, because you are supplying the training signal the original objective never had. Mining the negatives is straightforward once you have the labelled pairs from the harness above, since every pair marked unsafe is by definition a hard negative. What makes it a real commitment rather than a quick win is the operational tail: your cache keys are now tied to a model version you control, and every retrain invalidates the entire index and requires a full re-embed before the threshold you measured means anything again.

The workload decides this, not the technique#

None of the above is a verdict on semantic caching. Run the same harness against a public documentation assistant for the same product and the picture separates cleanly: paraphrases of "how do I rotate an API key" cluster from 0.94 to 0.99, while questions that need a different page — key rotation against key revocation, the Node SDK against the Python SDK — sit between 0.80 and 0.87, with almost nothing in between. On that corpus a threshold at 0.91 admits nearly every paraphrase and essentially no cross-topic match, because the distinguishing content is a noun that the embedding model was trained to represent and not a preposition that it was not.

The difference between the two surfaces is what the answers depend on. Documentation answers turn on topic, and topic is what these vectors encode. Order-policy answers turn on a modifier attached to a stable topic, and that is the axis along which the representation is thinnest. Before setting a number, the useful question is which of those two shapes your traffic has, and the harness answers it in an afternoon on data you already have.

On a surface where the bands do separate there is still a number to choose, and choosing it by looking at two ranges and picking something between them is not a method. The next chapter takes the labelled sample the harness produced and turns it into the two curves the decision is actually made on.