Cache TTL and hit rate

Every cache entry has a time to live (TTL), the interval after which the provider drops it: commonly five minutes on the default tier of a hosted API, with some providers selling a longer life at a higher write premium. The previous chapter valued a cached prefix at nine tenths of its size times the number of times it is read, and left the second term unexamined. That term is not a property of your traffic on its own. Six questions per document is a fact about people; five reads per write is a claim about time, and it is only true if all six questions arrive inside one entry's lifetime.

The gap between those two statements has a direction, and the direction is always the same one. Counting how many questions a reader asks is easy and gets into the plan; counting how far apart they arrive is not, and does not. So a plan that skips the conversion overstates its saving rather than understating it, and on the endpoint measured below it overstated it by about 30%. Everything that follows prices a write at 1.25 times the base input rate and a read at 0.1 times it, which is the pricing shape this topic has used throughout.

Sliding expiry and absolute expiry#

Before the two rules, the reason there is a rule at all. A cache entry is not a record of something that happened; it is reserved capacity. The previous chapter followed a write to the point where the computed state has been copied out of the accelerator's working memory, indexed under a hash of the prefix, and left sitting somewhere it can outlive the request that made it. That somewhere is finite, it is shared with every other customer on the same hardware, and nothing attached to your entry tells the provider whether you will ever come back for it. An entry that is never reclaimed is capacity permanently removed from serving live traffic, so every entry has to be reclaimable.

Which entry to reclaim would be an easy question if the provider knew which ones were about to be read, and it does not — that is the same ignorance the previous chapter gave as the reason the writer is charged rather than the readers. A rule that cannot look forward has to run off the past, and the cheapest rule of that kind is a clock: start a timer on some event, drop the entry when the timer runs out. A time to live is that timer. It is not a judgement about your entry's usefulness and it is not sold to you as one; it is the provider bounding how long any single entry can hold capacity without having to know anything about it.

That leaves exactly one design choice, which is which event starts the timer, and both available answers are in use. Under sliding expiry a read refreshes the entry, so it survives as long as no gap between consecutive matching requests exceeds the TTL, and a conversation that continues for two hours with a question every four minutes lives on a single write. The provider is treating a read as evidence of further reads, which is the best available guess and costs it a bound on how long an entry can persist. Under absolute expiry the clock starts at the write and nothing extends it, so an entry lives exactly five minutes regardless of how much traffic it serves — the provider gives up the guess and keeps the bound. Take one document with questions arriving at 0.0, 1.4, 3.1, 5.2, 7.6 and 9.8 minutes and the two semantics diverge immediately.

One write and five reads under sliding expiry; two writes and four reads under absolute. Read the diagram row by row to see where the second write comes from: the entry written at 0.0 dies at 5.0 under the absolute rule because nothing that happened at 1.4 or 3.1 touched its clock, and the question at 5.2 arrives two tenths of a minute too late to find it.

Price both sequences in the unit the previous chapter used — one send of the prefix at the base input rate is one prefix-equivalent — and each side is two multiplications. Sliding expiry buys one entry at 1.25 and five reads at 0.1 apiece, which is 1.25 + 0.5 = 1.75. Absolute expiry buys two entries, which is 2.5, and four reads, which is 0.4, for 2.9 altogether. Subtracting one from the other leaves a bit over one prefix, and a bit over one prefix out of 2.9 is a little under 40%: sliding expiry costs about 40% less than absolute for exactly the same six questions, produced entirely by a semantic detail in the provider's documentation.

Working in prefix-equivalents is allowed here for the reason it was allowed in the previous chapter, and the reason is worth restating because the comparison has changed shape. Both sides are the same six requests carrying the same six questions and producing the same six answers, and the provider bills the questions and the answers identically whichever expiry rule it runs. Those terms are the same quantity on both sides, so they cancel. Dividing what remains by the prefix's token count and by the base input rate removes the last two quantities the two sides share, and nothing survives except the multipliers — which is why the 40% does not depend on how large the prefix is or on what a token costs.

Both designs exist, and the sliding version is the more common one on hosted APIs, which means the assumption most people make happens to be right often enough that they never check it and are badly wrong when they move. The belief worth naming is not that expiry exists but that a matching request is a hit. Under absolute expiry the question at 5.2 minutes matches byte for byte, the prompt is unchanged, the code is unchanged, and the request is still a write: it pays the premium rather than the read rate, an order of magnitude more for the same tokens. Nothing in the response distinguishes "your prefix was different" from "your prefix was fine and the entry had gone", so a hit rate below expectation is not by itself evidence about the prompt.

If the documentation is ambiguous, twenty minutes and about a dollar settle it. Write an entry with a qualifying prefix, then send the same prefix every four minutes for twenty minutes and record the read counter each time. Sliding expiry gives five consecutive hits; absolute expiry gives a hit at four minutes and a write at eight, and the pattern repeats on a five-minute cycle however faithfully you keep sending. The probe works because the two rules differ in when a write happens rather than in what a write costs, and the response reports cache-read and cache-creation tokens as separate counters — so the question "was this a write" is answered by the payload rather than inferred from the bill. Run the same probe again at a six-minute interval to confirm the boundary sits where you think it does, since a provider that describes its TTL as "at least five minutes" is telling you the observed number will be larger and variable.

Expected writes from a gap distribution#

With sliding expiry, an entry dies exactly when a gap exceeds the TTL, so the number of writes for a document with n questions is one plus the number of gaps longer than the TTL. That sentence is the whole of the model and it is worth taking apart on the six questions above rather than accepting.

Six questions have five gaps between them, one fewer than the number of questions, because a gap needs a question on each side of it. The first question is always a write, because there is nothing to read yet. Every later question is a write if and only if the entry has died since the previous question, and under sliding expiry the previous question is the only thing that touched the clock — so the entry is alive exactly when the gap since that question was shorter than the TTL. Each of the five gaps is therefore a yes-or-no about one write, and the total is one plus however many of the five came out yes.

Turning that into an expected number needs one more step, and it is the step worth being careful about. Give each gap a score of one if it outlives the entry and zero if it does not; then the number of extra writes is the sum of those five scores, and the average value of a score is precisely the probability that a gap is long. The average of a sum is the sum of the averages, which holds whether or not the gaps have anything to do with each other — and they do have something to do with each other, since a reader who pauses once tends to pause again. That is why this particular step is safe and why a step assuming independence would not have been. So the expected number of extra writes is five times the probability that one gap is long, and in general n − 1 times it.

What remains is the probability itself, and it is the only place a distribution enters. Requests nobody is coordinating produce many short gaps and a thin tail of long ones, and the shape with that profile is the exponential — the one distribution for which having already waited tells you nothing about how much longer the wait will be, which is the right assumption when the next question does not know when the last one was asked. Under it, the chance a gap exceeds some length falls off at a fixed rate for each further multiple of the mean gap: if inter-arrival gaps are roughly exponential with mean m, the probability that any one gap exceeds T is e^(−T/m). Only the ratio of the two appears, so a five-minute TTL against a two-minute mean gap and a ten-minute TTL against a four-minute mean gap give the same answer.

That is a claim about the population of gaps rather than about any one reader, and the difference matters because the previous step already conceded that a reader who pauses once tends to pause again. Memorylessness says nothing about whether one reader's gaps resemble each other; it says that within a single gap, the time already spent waiting does not change the remaining wait. Putting the two halves together gives one write, plus n − 1 gaps, each outliving the entry with probability e^(−T/m), which is the function below.

import math

def expected_writes(n_requests: int, mean_gap_min: float, ttl_min: float = 5.0) -> float:
    """Sliding expiry: a new write for every gap that outlives the entry."""
    return 1 + (n_requests - 1) * math.exp(-ttl_min / mean_gap_min)

One row done slowly makes the rest readable. Take the mean gap of 2 min against the five-minute TTL. The ratio of the two is two and a half, so the probability a gap outlives the entry is e raised to minus two and a half, which the table gives as 0.082 — roughly one gap in twelve. Five gaps at that chance is about four tenths of a long gap expected per document, so the expected write count is one plus that, or 1.41. The remaining questions are reads, and six questions minus 1.41 writes is 4.59 of them. The hit rate is the share of all six requests that read rather than wrote, which is 4.59 out of 6, or the 77% in the last column. Every other row is that same sequence with a different exponent.

A write count of 1.41 is worth one sentence of warning, because no document ever costs 1.41 writes. It is an average over many documents, most of which cost one write and some of which cost two, and it is the right quantity to multiply by a day's document count for exactly that reason. Reading it as a prediction about a single document is where the number stops meaning anything; reading it as a rate is where it earns its place.

The curve bends steeply, and that convexity is why an average is a poor input here: the long gaps in a mixed population dominate the answer while the short ones contribute almost nothing. Averaging the gaps and then applying the exponential is not the same operation as applying the exponential to each gap and then averaging, and for a curve that bends this way the second is always the larger of the two. A population where half the documents get six questions in ninety seconds and half get six questions over forty minutes has the same ten-minute mean as a population read steadily throughout, and produces a completely different write count — the fast half contributes 1.0 writes each and the slow half close to 6.0, averaging 3.5 rather than the 2.84 the mean alone predicts. Where request logs exist, compute the gap distribution directly instead of feeding a mean into the formula.

The hit rate at which caching stops paying#

The break-even from the previous chapter was expressed in requests, and it assumed one write. Expiry breaks that assumption, so the write count becomes a variable and the threshold has to be expressed as a proportion instead. Take n matching requests over some period, of which w were writes and the remaining n − w were reads. Sending them all uncached costs n prefix-equivalents. Caching them costs 1.25 for each write and 0.1 for each read, so caching is the cheaper of the two when

1.25w + 0.1(n − w) < n

Multiplying out the bracket splits the read term in two: 0.1n, which is what the reads would have cost if every request had been a read, and −0.1w, which takes back the ones that were writes instead. The left-hand side is now 1.25w + 0.1n − 0.1w, and the two w terms can be collected because they are the same quantity in the same unit — prefix-equivalents charged per write. What their difference means is the interesting part: a write costs 1.25 and displaces a read that would have cost 0.1, so the excess a write pays over a read is the gap between those two rates, charged once per entry.

Subtract 0.1n from both sides, which leaves the direction of the inequality alone because the same quantity has come off each side. The left is now that excess times w. The right is n less 0.1n, which is what a full-price send still costs once the read rate has been taken off it — the saving a read makes against sending the prefix plainly. Then divide both sides by n, positive, so the direction is again unchanged, and w becomes the write share w/n. The threshold is therefore one ratio: the saving a read makes over an uncached send, divided by the excess a write pays over a read. On these rates that comes to w/n < 0.783.

The temptation at this point is to read 0.783 as the hit rate the endpoint needs, which would say caching only pays above about four fifths — close to the opposite of the truth. It is a ceiling on the write share. Every matching request is either a write or a read and nothing else, so the hit rate is whatever is left when the write share is taken away from the whole, which puts it at 21.7%. A hit rate above roughly 22% pays; below it, caching costs more than not caching.

The second row runs the same three lines with the write rate changed and the read rate left where it was. The excess a write pays over a read is now the gap between 2.0 and 0.1 rather than between 1.25 and 0.1, so it is larger. The saving a read makes over an uncached send has not moved at all, because the read rate did not change. A bigger number underneath the same number on top is a smaller ratio, so the write share the endpoint can afford falls to 47.4% — and the hit rate it needs, being the remainder, rises to a little over half. The two cells in that column are complements of one another rather than the same quantity measured twice, which is why each says which way it is counted.

That is why a longer TTL is a real decision rather than a free upgrade. Buying an hour of life at twice the base input rate demands that more than half your requests be hits before it beats sending the prefix plainly, and it beats the five-minute tier only when the extra survival converts enough writes into reads to cover the doubled premium. For an endpoint whose matching requests arrive every few seconds, the five-minute entry never expires under load and the longer tier merely doubles the premium on the handful of writes that remain.

Recomputing an endpoint comparison honestly#

Put measured write counts back into the comparison from the previous chapter. A document question-answering service with a 40,000-token prefix, six questions per document and 400 documents a day sits at the 3.3 min row of the gap table, so a document costs 2.10 writes and 3.90 reads rather than one and five. Price that document the way the two expiry sequences were priced above: 2.10 writes at 1.25 apiece, and 3.90 reads at 0.1 apiece, which come to 3.02 prefix-equivalents between them. Sending the same six questions uncached would have cost 6.0, one per question, so the saving is the difference of the two — allowed because both are already carried in prefix-equivalents, which is what the unit was for. The endpoint therefore saves 40,000 × (6.0 − 3.02) = 119,200 effective tokens per document rather than 170,000, which across 400 documents is 47.7 million tokens, about $143 a day at $3 per million. A ticket classifier at two million requests a day has arrivals so dense that expiry is irrelevant: even assuming a write every five minutes for the whole day, that is 288 writes against two million reads, a hit rate of 99.99%, and a saving still close to 2.7 billion tokens.

The gap between the two endpoints widened rather than narrowed, because expiry took 30% off the endpoint whose reuse depends on human pace and nothing off the endpoint whose reuse depends on request volume. That direction generalises: expiry punishes workloads whose reuse count was already small and is invisible on workloads whose reuse count was large, so applying the correction rarely reverses a ranking and frequently makes one obvious.

Scheduling as a lever on the hit rate#

Because expiry acts on gaps rather than on volume, work whose timing you control can be rearranged to fit inside the window, and this is the one place where a TTL becomes an input to system design instead of a constraint you measure. A nightly job that processes 4,000 documents across 40 tenants, iterating documents in upload order, touches each tenant's playbook every forty seconds or so and keeps every tenant entry alive — but it also holds 40 live entries at once and pays a write for each. Sorting the queue by tenant instead writes each playbook once, reads it a hundred times, and never revisits it, which is both cheaper and simpler to reason about.

The same trick applies to interactive traffic through routing rather than ordering. Requests sharing a prefix that are spread across shards or regions each maintain their own entry, so hash-based affinity on the prefix — routing a request to the shard chosen by hashing its leading bytes rather than to whichever shard is least busy — concentrates them and multiplies the effective arrival rate against a single entry. Neither of these changes the prompt at all; they change how close together in time the matching requests arrive, which is the only variable expiry actually reads.

Two conditions blunt all of it, and the first blunts the longer tier as well. Traffic that stops overnight restarts cold whatever the TTL says, so a longer tier's advantage applies to the working day and not to the first request after a quiet period. And a TTL is an upper bound rather than a promise: entries can be evicted early under memory pressure on the provider's side, which shows up as a hit rate slightly below what the arithmetic predicts and is not something you can do anything about beyond leaving margin in the estimate.

A TTL is not a freshness control#

Expiry has nothing to do with content going stale, and the reason is in how the entry is found rather than in how long it lives. A lookup is a match on the prefix bytes themselves. If an underlying document changes, the bytes that go into the prompt change with it, so the request is looking for something that was never stored — it misses, writes a fresh entry, and the old one sits unmatched until its own clock runs out. There is no path by which a request carrying new content is served an answer computed from old content, because old content is the only thing the old entry can be found by. Teams occasionally set a short TTL "so the cache stays fresh", which buys nothing and costs writes; the belief underneath it is that a hit could hand back a cheaper version of a different answer, and a hit cannot, since a hit is by construction a request whose prefix is byte-identical to the one that was computed.

Nor is it a lever on traffic that has no repetition in it. Extending an entry's life from five minutes to an hour helps only if matching requests arrive in that extra window; a batch job that processes one document an hour with a single question each gets nothing from either tier and pays the premium every time. A TTL is a knob on how much of an existing arrival pattern you capture, and it cannot manufacture arrivals.

The lifetime, then, is what converts a traffic shape into the reuse count the previous chapter multiplied by. Two things decide the conversion — whether a read refreshes the entry, and how long the gaps between matching requests actually are — and both are measurable in an afternoon, one with a probe and one from logs. A plan that skips the conversion is not wrong by a rounding error; it was 30% out on a real endpoint, in the direction of optimism, because counting questions is easy and counting the gaps between them is not.

All of that assumed a single entry per document read by one population of requests. Once a prompt is split into several cacheable segments, each segment has its own population — everyone, one tenant, one conversation — and each of those populations has a different arrival rate against the same TTL. Where the boundaries fall therefore decides who is able to read each entry, and that turns out to be a sharper lever than the lifetime.