Senior · 6 chapters
Semantic caching
Embedding-keyed response cache, similarity thresholds, staleness and correctness risk
A cache hit that saves tokens but serves a plausible answer to the wrong user or question is a correctness incident, not an optimization.
Read these first
What you will know
Cache responses keyed by embedding similarity
Similarity can collapse distinct intents, especially when short prompts, negation, or named entities are poorly represented.
Choose a threshold and measure the wrong-answer rate it admits
A threshold that raises hit rate can also raise false positives, so measure cache precision and savings together.
Show semantic caching returning a subtly wrong cached answer
Changed dates, tenant context, permissions, or omitted constraints can make a cached response plausible but invalid.
Write an allowlist and denylist for cacheable requests, including handling for tenant, user, authorization, and freshness context
Personalized or mutable data can turn a previously valid response into a confidentiality or correctness incident.
Evaluate the cache as a quality risk, not just a cost win
Lower model spend is not a win if bad hits increase retries, support load, or user distrust.
Chapters
- 1
One cache costs money, the other returns wrong answers
Provider prompt caching keys on an exact byte prefix and gives back nothing but saved computation, so the worst it can do is spend money you expected to save. A semantic cache keys on approximate meaning and returns a previous answer, so its worst outcome is a fast, confident reply to a question nobody asked. Describing it as the same idea one level up buries the only distinction that decides where you are allowed to run it.
- 2
Before and after score 0.94, and so do paraphrases
"Can I cancel my order after it ships?" against the same sentence with "before" measures 0.94, while genuine paraphrases in your traffic sit between 0.92 and 0.97. The words carrying the meaning barely move the vector, so the dangerous pairs live inside the band you wanted to admit. On a workload like this the threshold is not the control, and the honest question becomes whether to run the cache on this surface at all.
- 3
A threshold is chosen from two curves, not one
The number comes from a labelled sample of admitted hits at several candidate thresholds: for each one, how often the cached answer was actually right for the new question, read alongside the hit rate that threshold buys. Raising to 0.97 because the bands overlap is the worst of both, since genuine paraphrases drop out while qualifier-differing pairs at 0.96 remain the likeliest survivors. A cheap second-stage verifier changes the arithmetic and deserves pricing rather than reflex.
- 4
Authorization has to be in the key, not around it
A user at Acme asks when our contract renews and receives Globex's renewal date in 40ms, because the key was the embedding of the question text and the entitlement check sat outside the lookup. Tenant, plus any role or scope that could change the answer, belongs inside the key so an entry is reachable only by an equally entitled request. This is the defect that is an incident, and the blast radius is countable from the logs.
- 5
An answer that was correct on Monday decays silently
The refund policy changed on Tuesday and customers were still hearing Monday's answer on Thursday, with no error and nothing the cache could detect. Cutting the TTL from 24 hours to one is a guess dressed as a control: the window has no relationship to when the policy moved, and it still cannot tell you who was told what. Record which documents and policy versions produced each answer, evict on version change, and keep TTL as a backstop.
- 6
Three percent of hits is a count of customers per day
A 34% hit rate, 28% off model spend and 3% of admitted hits wrong cannot be decided in those units. Convert the error rate into wrong answers reaching customers each day and look at where they sit, because bad hits concentrate near your highest-traffic questions. The argument that the model is already wrong 5% of the time mistakes an added error for an absorbed one, and a threshold tuned on one query distribution and one embedding model transfers nowhere.
External resources
- GPT Semantic Cache: Reducing LLM Costs and Latency via Semantic Embedding Cachingarxiv.org
Measured hit rates and the accuracy cost of loose similarity thresholds.
- What is semantic caching? | Decagondecagon.ai
Despite being a vendor glossary page, it substantively explains semantic-cache mechanics, threshold tuning, cache invalidation, personalization risks, multi-turn context, and integration with RAG rather than merely promoting Decagon.
- zilliztech/GPTCachegithub.com
Drop-in embedding-keyed cache with pluggable similarity evaluators.
- Semantic Caching for LLMs: Beyond Prefix Cachingtruefoundry.com
Where semantic cache beats prefix cache and where it silently lies.
- What is a semantic cache?Redis
Five-minute model of embedding keys, thresholds and invalidation.
- Cut Your LLM Costs and Latency up to 86% with Semantic Caching | Databases for AIAWS Events
Production numbers plus threshold tuning and staleness handling.