A cache entry ages: what invalidation and TTL do to the cost model

An internal policy assistant answers questions over a corpus of HR, security and procurement policy, and the design document for it contained a cost model in three lines. The corpus is 400,000 tokens. It is edited two or three times a week. The feature serves about 2,000 queries a day. Therefore: three cache writes a week, 2,000 cache reads a day, and a bill of roughly $240 a day, which the finance partner signed off without comment because it was smaller than the number in the previous version of the document.

The word carrying the weight there is therefore. Edits invalidate a cached prefix and are easy to count, so a model built out of edit counts feels complete. An entry also has a time to live, abbreviated TTL: the period the provider promises to keep it before discarding it, commonly five minutes on the default tier. It dies of old age whenever the gap between two matching requests exceeds that period, and gaps are a fact about when requests arrive rather than about how many arrive in a day. The difference between counting the one and counting the other is the difference between half a write a day and twenty.

What the three-line model prices#

Take the pricing from chapter 2 and put the corpus size in it. One uncached read of 400,000 tokens at the $3.00 base rate sets the unit, and the two multipliers scale it: a cache write at 1.25× that unit is $1.50, and a cache read at 0.1× is $0.12. Everything in this chapter is those two prices multiplied by a count, so the whole argument reduces to getting the two counts right.

The design document's model is then three writes a week — 0.43 a day — plus 2,000 reads: $0.64 + $240.00 = $240.64. That closes exactly, which is worth noticing, because a model can be arithmetically perfect and still be built on the wrong quantity. Nothing in the addition is wrong. The count of writes is what is wrong, and the addition has no way to say so.

Why an entry dies of old age#

The provider is holding the state it computed while reading 400,000 tokens, and that state occupies memory on a machine somewhere. Holding it costs the provider something whether or not anybody comes back for it. So the entry is discarded after a fixed idle period rather than kept indefinitely, and the consequence for you is that survival depends on the gap between one matching request and the next. An entry is not a stored object with a lifetime of its own; it is a bet that another request will arrive soon enough.

Two thousand queries a day averages one every 43 seconds, which sounds like an entry that never expires, and that average is taken over a day in which the traffic is not remotely uniform. The average is the quantity that holds still while the property that matters moves underneath it. Traffic can shift from evenly spread to heavily clustered without the daily total changing by a single query, and the write count will move by an order of magnitude while the number everybody quotes stays at 2,000.

Counting expiries from the arrival profile#

Expiries follow the gaps between requests, so you count them from the arrival profile rather than from the daily total. For a workload whose requests arrive independently of one another at an average rate of λ per hour — a Poisson arrival process, which is the standard first approximation for user traffic and is close enough here — the probability that any given gap exceeds a five-minute TTL is e^(−λ/12), and the expected number of expiries in that hour is the query count multiplied by that probability.

Two things in that sentence are worth unpacking before the table uses them. The twelve in the exponent is the TTL expressed in the same units as λ: five minutes is a twelfth of an hour, so at λ arrivals per hour the expected number of arrivals inside one TTL window is λ divided by twelve, and the probability that none of them shows up — which is what a gap longer than the TTL means — is e raised to minus that quantity. And the exponent is what makes the profile matter. The probability does not fall in proportion to the rate; it falls exponentially in it, so doubling the rate squares the survival probability rather than halving it. That is why no single average rate can stand in for a day that has both quiet hours and busy ones.

Both totals close on the rows above them with nothing left over: the hour bands add to a full day, the queries add to 2,002, and the expiries add to 19.4. That matters more than it looks, because it means the day has been partitioned rather than sampled. Every query in the day sits in exactly one band and is counted once, so the expiry column can be read as a decomposition of the write count rather than as an estimate of it.

Read the table as gaps between rows and one pair does all the work. The first band and the third have the same corpus, the same TTL and the same prices, and differ only in how closely their requests are spaced. Twelve queries produce 10.2 expiries; five hundred and forty produce none worth printing. Forty-five times the traffic, and the busier band writes nothing at all. The relationship between volume and cost has changed sign inside one table.

Twenty writes a day rather than 0.43, a factor of 46 on the quantity the design document reasoned about. Almost all of it comes from the twelve overnight queries, which are 0.6% of the traffic and produce 53% of the writes, because a lone query at 03:14 warms a $1.50 entry that nobody reads before it expires. Those two percentages come from the same partition, so they can be compared directly: the share of traffic and the share of writes are computed over the same denominators, and the fact that one is a small fraction of a percent while the other is more than half is the whole of the finding. Weekends are worse in the same direction: a Saturday with 300 queries spread over sixteen hours generates around forty expiries against a seventh of the volume.

What twenty writes do to the bill#

Put the corrected count back in and the day costs 20 × $1.50 + 1,982 × $0.12 = $30.00 + $237.84 = $267.84. The design document was wrong by 46× on writes and 11% on money, and the reason is that reads dominate so heavily that the write term barely participates.

The regrouping that shows why is short enough to do here. Write the daily cost as the unit price of one uncached read of the corpus, multiplied by the read multiplier times every query, plus the difference between the two multipliers times the writes. That is the same sum rearranged: a write is being charged as though it were a read, plus the amount by which a write exceeds a read. The rearrangement is legitimate because every request is either a read or a write and there is no third category, so adding the read rate to all of them and then topping up the writes counts each request exactly once.

What that form makes visible is the sensitivity. An error in the write count is not multiplied by the price of a write; it is multiplied by the premium, the gap between $1.50 and $0.12, because a request that was going to be billed anyway merely changes category. The design document undercounted writes by about nineteen and a half a day, and nineteen and a half of those premiums comes to about twenty-seven dollars — which is what the two totals differ by, twenty-seven dollars and twenty cents, the small remainder being the rounding of three writes a week into 0.43 and of 2,002 queries into 2,000. A forty-six-fold error in the count and a one-part-in-nine error in the money are the same error, seen through a multiplier that is small.

That licenses one thing and forbids another. It licenses being sloppy about the write count, in the specific sense that getting it badly wrong still lands the bill within about a tenth of the truth. It forbids reading that tenth as small. Both terms scale with the size of the prefix, so the proportion is a property of the traffic rather than of the corpus and it survives the corpus growing, while the money the proportion is taken from does not stay put at all. The end of this chapter is what happens when that number moves.

The ratio caching cannot change#

A cached long-context design is not cheap; it is exactly one tenth as expensive as the uncached version, and one tenth of a 400,000-token read is still a 40,000-token read. Against a retrieval design sending 6,000 tokens per query, the payload ratio is 66× uncached and 6.6× with a perfectly warm cache, and 6.6× of $36 a day is $238 a day.

Two instruments have now landed on nearly the same number by routes that share no step. One counted 1,982 reads at $0.12 and got $237.84. The other took retrieval's daily cost and multiplied it by a payload ratio, never touching a query count or a read price at all, and got $238. Agreement between a per-item count and a ratio is the check worth running on any cost model, because the two fail in different ways: a count goes wrong when the traffic profile is wrong, and a ratio goes wrong when the payloads are wrong.

The residual between those two routes and the $268 in the table is not rounding. It is exactly the $30.00 of writes, which sits outside the payload ratio altogether — the ratio prices what a query sends and the write term prices what a gap in traffic costs. That is the one part of the bill cache tuning can move, and the table above shows how little of the total it is.

Caching bought a 10× discount on a design that was 66× more expensive, which leaves it 6.6× more expensive, and no amount of further cache tuning changes that ratio because the cache is already at 99% hit rate. The two factors multiply and neither can reach the other: the discount is set by the provider's read multiplier and the payload ratio is set by how many tokens each design sends, and a cache has no opinion about the second. There is a version of this feature where that is the right trade — a corpus where retrieval genuinely fails to find the relevant policy, or a team with no capacity to maintain an index — but it has to be argued on quality or on engineering cost, and not on the claim that the cache made it cheap.

Buying a longer entry lifetime#

One thing does move the write term, for what it is worth. Buying a longer entry lifetime — an hour rather than five minutes, at a write premium of 2.0× rather than 1.25× — removes essentially every timeout except the overnight gap.

The row that surprises people is the read cost, which goes up. Nothing about reading got more expensive; the requests that stop having to write an entry become reads instead, so the read count rises by as many as the write count falls. The two columns' daily query totals differ by two, which is the rounding of 2,002 into 2,000, and once that is set aside the movement closes: reads gain a little under two dollars, writes shed a little over twenty-five, and the difference between the two totals is twenty-three dollars and twenty-eight cents. The lever is not the number of reads or writes on its own. It is that a write costs more than a read, so converting one into the other is a saving whichever direction the individual counts move.

Nine percent, or about $8,500 a year, for a configuration change. Worth doing, and small enough that it does not touch the comparison above.

import math

DAY_PROFILE = [(2, 6), (40, 3), (180, 3), (120, 2), (190, 4), (70, 3), (40, 3)]

def daily_cost(prefix_tokens: int, profile=DAY_PROFILE, ttl_min: float = 5.0,
               base_per_mtok: float = 3.00, write_mult: float = 1.25,
               read_mult: float = 0.10, edits_per_day: float = 0.43) -> dict:
    queries = expiries = 0.0
    for rate_per_hour, hours in profile:
        n = rate_per_hour * hours
        queries += n
        expiries += n * math.exp(-(ttl_min / 60.0) * rate_per_hour)
    writes = expiries + edits_per_day + 1
    reads = max(queries - writes, 0.0)
    unit = prefix_tokens * base_per_mtok / 1_000_000
    return {
        "writes": round(writes, 1),
        "hit_rate": round(reads / queries, 4),
        "cost": round(writes * unit * write_mult + reads * unit * read_mult, 2),
    }

The writes line is the model in one expression: the timeouts computed from the profile, plus the edits, plus one for the first warm-up of the day. The + 1 is the entry that has to exist before any read can find it, and it is the term the design document dropped along with the timeouts. Run it with your own arrival profile before you put a number in a design document. The function makes the dependency explicit: cost is a function of the shape of your traffic, not its volume, and two features with identical daily query counts can differ by a factor of two in write count.

Appending edits to keep the entry warm#

The natural response to "edits destroy the entry" is to stop editing in place. Append a corrections block to the end of the prefix — "Section 4.2 of the travel policy was updated on 14 March; the new per-diem is €95" — and the first 400,000 tokens stay byte-identical, so the entry survives.

It survives, and it is the single most dangerous idea in this chapter. Three things go wrong, in increasing order of how long they take to notice.

The append does not actually preserve the entry, because the breakpoint sits at the end of the corpus and the appended block is inside the cached region. Moving the breakpoint above the corrections block fixes that, at which point the corrections are re-read at full price on every request, which is fine while they are a paragraph and stops being fine at the eighteen-month mark.

The model now has two contradictory statements about the per-diem and no rule for preferring one. In practice it usually prefers the later one, and "usually" is not a property you want on a policy answer; retrieval of a specific clause will sometimes surface the original text in a context where the correction is not obviously about the same clause, particularly once corrections accumulate and start correcting each other.

The third failure is the one that puts a wrong answer in front of a user with nothing to signal it. The corpus in the prompt is whatever was current at the moment the entry was written, and nothing in the caching mechanism relates that moment to the current state of the source documents. Once the team is deliberately engineering for cache survival — appending rather than editing, or rebuilding the prefix on a nightly schedule to keep the daytime hit rate high, which is the more common version — the prefix is a snapshot with an age, and the age is invisible. A policy edited on Tuesday morning is quoted in its old form on Thursday afternoon, confidently, with a section reference that makes it look authoritative. The user has no way to detect it and neither does the model.

Every defence a reader is likely to reach for here is a defence against the wrong failure. A high hit rate says the prefix is being matched, which is a statement about byte-identity and therefore a guarantee that the text has not changed — the exact property that is the problem. An edit-invalidation policy protects the case where somebody edits the source, and the appending trick exists specifically to stop that policy from firing. Neither instrument can see staleness, because staleness is agreement between the cache and a copy of the corpus that is no longer current, and agreement is what both of them are built to confirm.

If you keep the long-context design, the freshness requirement has to be written down as a number — "answers reflect the corpus as of at most 30 minutes ago" — and enforced by a version stamp: hash the assembled corpus, put the hash and its build time in the metric stream, and alert when the served hash lags the source hash by more than the stated bound. That converts an invisible correctness failure into a visible operational one. It does not make the answer fresher; it makes staleness something you can see, which is the most any cache design can offer.

The same model at 1.2M tokens#

The corpus grew. Procurement templates, two acquired companies' handbooks, and a security standard took it to 1.2 million tokens, and every term in the model moves at once.

Reads go to 2,000 × 1.2M × $0.30/M = $720 a day, $263,000 a year, against a retrieval design still sending 6,000 tokens per query at $36 a day. The ratio went from 6.6× to 20×, because retrieval's payload is set by top-k and does not grow with the corpus while the stuffed payload is the corpus by definition. Writes go to $7.20 each, so the twenty timeouts a day now cost $144 rather than $30, and the overnight query that warms an entry nobody reads costs more than a hundred queries during the working day.

One thing does not move, and that is what retires the licence granted earlier in this chapter. Both terms scale with the prefix, so the design document's proportional error is the same here as it was at 400,000 tokens; what grows is the number the proportion is taken from. A tenth of $263,000 a year is not a rounding difference, and the arrival profile stops being worth estimating and becomes worth measuring — at exactly the point where nothing about the model itself has changed.

Two things stop being about money. The corpus may simply exceed the model's context window, and even where it does not, providers cap the cacheable prefix and the number of breakpoints, so a 1.2M-token prefix can be uncacheable in whole or in part on the model you wanted to use. And the re-warm is now a user-visible event: reading 1.2 million tokens from cold takes tens of seconds of prefill, which is the pass in which the model reads the entire prompt before it can produce a single word of the answer, so whoever asks the first question after a quiet period waits while everyone else gets a warm read. Chapter 4 takes the latency and quality axes properly, because at this size they stop being secondary considerations and start being the reason the design is rejected.

A cached prefix is a bet that the next matching request arrives before the entry ages out, and the cost model is the price of that bet placed once per gap in the traffic. Counting the gaps takes an arrival profile, which most teams have in a dashboard and almost none put in a design document, and the count that comes out can be an order of magnitude away from the edit rate that seemed like the obvious driver.

The discount is real and it is bounded. It scales the tokens a design sends; it does not change how many it sends, so it moves a comparison between two designs by exactly one factor and no further. A design that was many times more expensive before caching is fewer times more expensive after it, and whether that is now acceptable is a question about quality and engineering cost rather than about the cache.