A memory store costs more than tokens and milliseconds

The costs weighed so far are the ones an instrument produces: milliseconds, tokens, dollars per task. They share a property that makes them easy to argue about, which is that something automatically counts them. The two costs in this chapter share the opposite property. Nobody counts them, they arrive irregularly, and they land on engineers rather than on a bill, so a feature review that reaches for the numbers to hand will discuss a fifth of a second and 1,500 tokens as though those were the whole trade.

The first is that a store of facts about people is retained personal data, which carries duties: to delete it on request, to retire it on a schedule, to keep it in the jurisdiction it was collected in, and to show a person what you hold about them. The second is that a stateful component has a standing maintenance bill — migrations, backfills, on-call and the upkeep of the eval suite the earlier chapters built — which recurs whether or not anybody is developing the feature. Both are absent from the dashboards of chapters 1 through 5 by construction, because those dashboards measure requests and this cost is not made of requests.

The eight places 47 records live#

An email arrives from a customer's data protection officer: a departing employee has exercised their right to erasure, and everything the product holds about them must be deleted within thirty days. The ticket is assigned to whoever owns the memory service, on the assumption that this is one DELETE statement. The store holds 47 records for that user — preferences about report formatting, the fact that they own the EMEA pipeline, a note that they asked for figures in euros. Deleting the rows takes a second. Finding every other place those 47 facts now live takes the rest of the week.

The reason a diagram like that exists at all is worth stating as a rule rather than as a surprise. A record is written once and then read by everything downstream, and every downstream reader that keeps a copy has created a second location holding the same personal data. Deleting the source removes the source and nothing else, because a copy is by definition not reachable from the thing it was copied from. So the number of places to delete from is the number of components that retain rather than merely pass through, and that number is a property of the architecture — which means it is enumerable in advance, and enumerating it is the deliverable.

Four of those eight destinations were built by the earlier chapters of this topic, which is the part teams do not anticipate. The sampled production traces exist because attribution needed them and they contain memory content verbatim. The eval fixtures exist because the held-out set was built from real customer histories, and although names and identifiers were substituted, a preference phrased in someone's own words is still their data. Backups and the read replica — a continuously synchronised copy of the store that serves queries so the primary is not overloaded — exist because the store is a database and databases get backed up and read from copies. The derived summaries exist because someone added a compaction step and forgot that a summary of personal data is personal data.

Erasure has a fan-out and it needs to be written down#

The obligation is not satisfied by deleting the primary row, which means the erasure path has to be a first-class piece of code with a test, not a runbook step someone follows by hand.

async def erase_user(user_id: str) -> ErasureReceipt:
    record_ids = await primary.select_ids(user_id=user_id)
    await primary.delete(user_id=user_id)
    await vector_index.delete(ids=record_ids)          # tombstone, then compact
    await summaries.delete(user_id=user_id)            # derived from the above
    await traces.redact(user_id=user_id, fields=["writes", "retrieval.candidates"])
    fixtures = eval_fixtures.find(user_id=user_id)     # manual queue: rebuilding
    await backups.schedule_expiry(user_id=user_id)     # cannot rewrite; expires D+35
    return ErasureReceipt(record_ids, fixtures_pending=len(fixtures),
                          backup_clear_at=now() + timedelta(days=35))

The vector index deletes by tombstone, marking the record dead so that searches skip it and reclaiming the space in a later compaction pass. The reason it works that way is the structure of the index: an approximate-nearest-neighbour graph stores each record as a node with links to its neighbours, so removing a node outright means finding and repairing every link that pointed at it, which is expensive and has to happen while queries are running. Marking it dead and filtering it out at query time costs one check per visited node and can be done immediately. The consequence to state plainly is that a record is unfindable within seconds and physically gone somewhat later, and if "deleted" in your policy means "no longer on disk", the compaction schedule is part of your compliance story rather than an implementation detail.

Two lines in that function are honest about limits, and both limits are structural rather than lazy. Backups usually cannot be surgically rewritten, because a backup is an image of the database at an instant and editing it destroys the property that makes it a valid restore point. The standard practice is therefore to record the erasure, suppress the user on any restore, and let the backup age out — which is defensible and needs writing into the retention policy rather than discovered during an audit.

The two numbers in that line do not fit together, and noticing that they do not is the point of writing the receipt. The obligation is thirty days; the backups hold thirty-five days of history, so the last copy of a record erased today clears five days after the deadline has passed. No amount of scheduling closes that, because the backup taken yesterday already contains the record and cannot be edited. Suppress-on-restore is what bridges the gap: the record is still on tape and can never come back into a running system, which is the strongest statement available and is the one the policy has to make rather than the one everyone assumes it makes.

The eighth destination in the diagram is the one you do not operate. Injected memories travel to the model provider inside the request body, so the provider's request logs hold them for whatever period its agreement specifies — thirty days here — and no code you write shortens that. It belongs on the diagram anyway, because an erasure receipt that omits it is describing a smaller system than the one that exists.

Eval fixtures cannot be automatically deleted either, and the reason connects back to the second chapter. The eval reports a proportion over a fixed set of tasks, so removing tasks changes both the denominator and the mix of slices inside it, and two quarters' numbers then describe two different sets. A set that shrinks each quarter as erasure requests arrive also loses power, since the smallest detectable difference grows as the set gets smaller. This team's resolution was to rebuild affected fixtures from synthetic equivalents on a monthly cadence, which costs about a day a month and keeps the set size stable.

Retention has a similar shape: a policy that memory records expire eighteen months after last access is easy to state, and implementing it means a TTL sweep — a scheduled job that deletes records whose time to live has run out — an index compaction, and a decision about whether an expired record that a user still relies on should quietly disappear. Data residency multiplies the whole apparatus, because EU workspaces' memory has to stay in an EU region, which is a second store, a second index, a second backup schedule and a second erasure path.

Access requests arrive alongside erasure requests and are handled less often, because they need a rendering rather than a deletion. Someone asking what the product holds about them is entitled to an answer they can read, which means every record has to carry enough provenance — when it was written, from which session, by which extractor version — to be explained. A store designed as an opaque bag of embedded strings cannot produce that after the fact, because the information was never captured and cannot be recovered from an embedding. Retrofitting provenance onto 4.1 million existing records is a backfill — a one-off pass that rewrites every row already in the store into the new shape — with no source of truth for the older ones.

An incident in a stateful component#

The previous topic covered how memory becomes an attack surface — poisoned writes, cross-session influence, the way retrieved content reaches the model as instructions. This is the same store viewed as an operational liability rather than as an adversarial one, and the difference matters because these costs are incurred with no attacker present at all.

A stateless assistant that leaks something leaks whatever was in that request. A store leaks an accumulated corpus: 4.1 million records, spanning three years, describing what 40,000 people work on and how they want it presented. The size of the exposure is a function of how long the feature has been running rather than of how bad the bug is, which is the property that makes a stateful component different from a stateless one.

The namespace filter bug from the attribution chapter is the illustration close to hand, and it is worth walking in both directions. In the direction it actually failed, records written under an old workspace identifier fell outside the current filter and became invisible, which cost 18 eval failures and some confused users. The same predicate with the comparison inverted returns another workspace's records, and that is a cross-tenant disclosure of durable personal data rather than a wrong answer.

What makes that pair dangerous is what a test of the filter usually asserts. The natural test seeds a workspace's records, issues a query, and checks that the expected records come back. A filter that returns too many records still returns all the expected ones, so the assertion passes; the count of expected records present is constant while the count of other tenants' records leaked alongside them varies freely. The test is a defence against the direction that costs eval failures and no defence at all against the direction that costs a disclosure. A filter predicate that decides correctness in one direction and confidentiality in the other deserves a different class of test than the one it had — one that asserts absence rather than presence, seeding a second workspace and requiring that none of its records ever appear.

The blast radius shows up in incident response time as well as in exposure. A memory-related incident cannot be resolved by rolling back a deployment, because the bad state is already written. Rolling back a stateless service returns it to its previous behaviour and there is nothing else to do; rolling back a writer stops it writing and leaves everything it wrote.

The standing maintenance bill#

The obligations are episodic. The maintenance is continuous, and it is the line most often missing from a memory feature's cost model because it lands in engineering time rather than on an invoice.

Read that table across its columns rather than down them. The money column is nearly empty and the effort column is full, and on the one row that has a money figure the two are wildly mismatched. Re-embedding 4.1 million records at roughly 60 tokens each is 246 million tokens, which at $0.13 per million costs $310 — an amount nobody would discuss. The six engineer-days and the three-week window during which both the old and new indexes must be queried and reconciled are the actual cost, and they recur every time the embedding model is upgraded, which for a team that wants to stay current is more often than eighteen months.

Totalled from the team's own tracker, memory consumed 26 engineer-days per quarter, or roughly 0.45 of a full-time engineer. The conversion between those two figures is the one people skip and then dispute: a quarter holds somewhere around sixty working days once holidays and public days are taken out, so 26 days against it is a little under half a person, permanently. It is not half a person in one block, which is why it is invisible — it arrives as an afternoon here and a week there, spread across whoever is on call. At a fully loaded cost of about $260,000 a year — salary plus the taxes, benefits, tooling and floor space that come with employing someone — that is $117,000 annually. The inference and infrastructure it adds — $3,720 a month in extra tokens across 1.2 million tasks, plus about $1,400 a month for the vector index replica and storage — comes to roughly $61,000 a year. Maintenance is close to twice the running cost, and it is the half that never appears in the cost review because nobody bills engineering time to a feature.

The eval suite row is the one this topic created, and it is worth being clear that it is a permanent cost rather than a project. A held-out set decays in three separate ways: the distractor corpus stops resembling production as stores grow, cases become guessable as the base model improves and starts answering from priors what it used to need a stored fact for, and erasure removes fixtures. All three of those changes move the number in the same direction, because the cases that rot are disproportionately the hard ones — a case that becomes guessable was a hard case that the model got better at, and a case removed by erasure was one of the vivid real histories. A suite left alone for a year therefore reports on a system that no longer exists, and the number it reports will usually be flattering. Five engineer-days a quarter is what keeping it honest cost here, and skipping two quarters of that work is functionally the same as switching the measurement off while continuing to quote it.

Managed memory services move some of these lines rather than removing them. Storage, index maintenance and migrations become someone else's problem; the erasure fan-out, the residency requirement, the eval suite and the retention policy remain yours, because they attach to the fact that you decided to retain personal data rather than to where the bytes sit. A vendor's data processing agreement, the contract under which they handle personal data on your instructions, changes who operates the store and not who is accountable for what is in it.

Two lines that scale on different quantities#

None of this is an argument that memory is too expensive to have. A feature that produced a genuine ten-point gain in task success would justify all of it comfortably, and plenty of products are in exactly that position, particularly long-running assistants where sessions are short and the relationship is long. The purpose of putting numbers on these lines is that they are otherwise absent from the comparison, so the debate happens as though the only cost is 400 milliseconds and 1,500 tokens.

The figures are also specific to this system's scale and workload, and the ratios shift substantially elsewhere. The two cost lines scale on different quantities, which is why the ratio between them is not portable. Maintenance scales with the number of components and migrations, which barely depends on how many users there are; inference scales with traffic, which depends on nothing else.

Follow that through and both extremes come out. A team with 400 users has almost no erasure volume and the same six engineer-days for an embedding migration, so maintenance dominates even harder. A team serving 50 million tasks a month has an inference line that dwarfs the engineering time, and the calculation inverts. What transfers is the set of lines to count, not the values in them.

The ledger is now complete: a measured difference of +2.0 points with a ±2.5-point band, a 13-point gain concentrated in one slice, 51 regressions against 71 gains, +410 ms, +12% per task, and about $178,000 a year in running and maintenance cost carrying obligations that do not go away on their own. Weighing those against each other, in an order that does not let the cheapest argument go first, is the last thing this topic has to do.