One cache costs money, the other returns wrong answers

Your support assistant sends a 12,400-token system prefix on every request — tool schemas, formatting rules, an escalation matrix and a compressed policy digest — and the provider's prompt cache is hitting on it 78% of the time. The billing export bears that out: cached input reads are charged at a tenth of the base rate, so the prefix that would have cost roughly $0.037 per call at $3 per million input tokens costs about $0.011 on average once the hit rate is applied, and the line item for input tokens fell by two thirds in the month you shipped it. A colleague has now posted in the channel that the obvious next step is a semantic cache, "the same idea, one level up — you've cached the prefix, now cache the answer."

The proposal is concrete enough to price. Embed each incoming question, search a vector index of questions you have already answered, and if the nearest neighbour is close enough, return the answer you produced for that neighbour without calling the model at all. On the head of your traffic distribution that is a large win: the top 200 phrasings account for about 41% of eligible volume on this surface (eligible meaning the requests you would be willing to let a cache answer at all), a typical answer is 900 output tokens at $15 per million, and skipping generation entirely saves the $0.0135 of output plus the whole 1.8-second time to last token, replacing it with a 40-millisecond vector lookup.

The two mechanisms are not the same idea at different granularities, and the difference is not one of degree. A prompt cache is keyed on an exact byte prefix and returns nothing except computation you would otherwise have redone; the tokens the model generates are the same tokens it would have generated on a miss. A semantic cache is keyed on approximate meaning and returns a previously generated response in place of one the model never produced for this request. It exists because the same questions arrive over and over in slightly different words, and paying a model to write the same answer for the four hundredth time is money and latency spent on nothing; you meet one sitting in front of a support assistant, a documentation bot or a retrieval pipeline, arriving either as a library wrapped around the model client or as a feature of the vector store a team already runs. The first can waste money. The second can answer a question nobody asked, in 40 milliseconds, with the same confident tone as a correct answer.

What each one is actually keyed on#

Prompt caching, as the previous topic laid out, matches on a prefix of the token sequence at declared breakpoints, byte for byte. Change one character in the escalation matrix and every request after it misses until the new prefix is written again. The object stored is the attention key/value state for those tokens, which is a function of exactly those bytes and the model weights, so a lookup either finds state computed from an identical prefix or it finds nothing. There is no near miss. The failure modes are all economic: a prefix that varies per user never accumulates hits, a cache write costs a premium over the base input rate and is wasted if the entry expires unused, and a five-minute TTL — the time to live an entry is granted before it is discarded — against traffic that arrives every seven minutes bills you the write every time and never bills you a read.

A semantic cache stores pairs of (embedding of the request, response text) and retrieves by vector similarity. The lookup is short enough to write out in full.

THRESHOLD = 0.93

def answer(question: str) -> tuple[str, str]:
    q_vec = embed(question)                      # 1536-dim, L2-normalised
    hits = index.search(q_vec, k=1)              # cosine over the stored keys
    if hits and hits[0].score >= THRESHOLD:
        return hits[0].payload["response"], "cache"

    response = model.generate(SYSTEM_PREFIX, question)
    index.upsert(vector=q_vec, payload={"question": question,
                                        "response": response})
    return response, "model"

Nine lines of logic, one tunable constant, and a saving that shows up in the first day of metrics. The reason this shape spreads so fast is that it is easy to build and its win is immediately visible, while its cost is invisible by construction — a bad hit returns a well-formed answer with no error, no exception, no elevated latency and no signal in any dashboard you already have.

The pattern has a short and well-documented history. It arrived as a general technique with GPTCache in 2023, which packaged the embed-and-search loop as a drop-in wrapper around the OpenAI client with pluggable similarity evaluators, and the published measurements since then have been consistent about both halves of the trade: GPT Semantic Cache reports hit rates in the 68–70% range on a customer-query workload, along with the accuracy degradation that appears once the similarity threshold is loosened enough to reach those rates. Both numbers come from the same experiment, and teams reliably quote the first without the second.

Only one of the two caches can return a wrong answer#

Prompt caching is correctness-preserving by construction. The cached artifact is an intermediate computation over bytes the client sent, retrieved only by those same bytes, and feeding it to the model produces the same forward pass as recomputing it, up to floating-point non-determinism that is present with or without the cache. A stale entry does not exist as a category: if the prefix changed, the key changed, so the old entry is simply unreachable. Eviction produces a recomputation. Corruption produces an error. Nothing in the mechanism has a path that ends in a different, plausible answer.

A semantic cache stores an output that was generated for a different input, and admits it on the strength of a distance being under a number you chose. Whether that admission was correct is not a property of the mechanism; it is an empirical property of your query distribution, your embedding model, and the specific pair that got matched. The cache cannot check its own work, because if it could evaluate whether the stored answer fits the new question it would be doing the reasoning it exists to skip.

The middle column matters because it is the neighbour this gets confused with. Hashing the normalised request text and returning the stored response for byte-identical repeats is a genuinely different system: it inherits the freshness problem that any response cache has, but it never matches two different questions, so it has no wrong-question failure mode at all. On a surface where users arrive through canned buttons or a fixed set of suggested prompts, exact-match already captures most of the achievable hit rate, and reaching for embeddings adds risk in exchange for a few percentage points.

Where similarity quietly collapses two questions into one#

The earlier topic on similarity metrics established that cosine similarity measures angle in the embedding space and that the metric has to match how the model was trained. What it does not tell you, and what decides whether this cache is safe, is which linguistic differences the model was ever trained to represent as distance. Retrieval embedding models are trained with contrastive objectives, meaning the model is shown pairs it must pull together in the space alongside pairs it must push apart, and here those pairs are queries and the documents that answer them, where the useful signal is topical: a query about refunds should be near a refund policy document and far from a shipping document. Nothing in that objective rewards separating two refund questions from each other.

The consequence shows up on exactly the words that carry the decision. Three pairs, measured with text-embedding-3-small on real phrasings from an order-support log:

The genuine paraphrase scores lower than all three of the dangerous pairs. Short prompts make this worse, because with eight tokens of input a single changed function word is a small perturbation of a mean-pooled vector, which is what you get when the encoder's per-token vectors are averaged into one; named entities make it worse, because INV-2291 and INV-2843 tokenise into near-identical subword sequences and neither appeared in pretraining as a meaningful unit; negation makes it worse, because "can" and "can't" occupy the same syntactic slot in the same topic. These are not exotic edge cases dredged up to make a point. They are the ordinary shape of customer questions, where the topic is stable across a whole support queue and the distinguishing content is one qualifier.

Where you run it, and where you refuse#

The workload a semantic cache is built for is a public documentation assistant: a pre-sales or developer-docs question surface where the answer depends on published content and not on who is asking, traffic is head-heavy so the top phrasings repeat hundreds of times a day, and the content has a publish pipeline you can hang invalidation off. On a surface like that the cache saves real money, the worst case of a bad hit is a user being shown a nearby documentation answer they can see is nearby, and the correction path is that they rephrase.

The workload to refuse is the one in the opening scene. An account-scoped support assistant answers "when does my next invoice arrive" differently for every tenant, differently for an admin and a read-only member, and differently on the fifth of the month than on the twentieth. Every one of those distinctions lives in context that the question text does not contain, which means the embedding key cannot represent it. Two customers can send byte-identical questions that require different answers, and a cache keyed on the question text will happily give the second one the first one's answer.

Between those poles there is a design that gets less attention than it deserves, which is to cache something other than the final answer. Caching the retrieval step — the set of document chunks a question pulled — keeps the model in the loop, so a slightly-wrong chunk set still passes through generation, which usually notices that the retrieved text does not address the question asked. The saving is smaller, because you still pay for output tokens, and on a RAG surface — retrieval-augmented generation, where the answer is written from documents fetched at request time rather than from the model's own weights — retrieval is a third of latency, so it is not nothing.

The pitch to answer in the channel, then, is that the two caches share a word and not a risk profile. Prompt caching is a cost optimisation whose failure mode is a cost regression, so it can be enabled anywhere and tuned in production against a billing dashboard. A semantic cache is a correctness surface, and the question of whether to run it is settled before the question of what to set the threshold to.

That threshold is where everyone goes next, on the reasonable-sounding theory that the pairs above are a tuning problem — 0.94 admits the cancellation pair, so set the bar higher. The next chapter measures what happens when you try that on a real traffic sample, and finds that on some workloads there is no number that works, which is a different and more useful finding than a badly chosen one.