Deletion, replay, and residency

A messages table was never really a storage decision. Storage was the cheap part — a few kilobytes of text per conversation, on a database you were running anyway. What the table actually bought was an artefact: a row per turn, with a timestamp, an immutable body, and a foreign key to the run that consumed it. Because it existed, exact replay was available whether or not anyone had asked for it. You could take a run from six weeks ago, read the rows that fed it, pin the model version recorded alongside, and send the identical bytes again. Nobody writes "byte-exact replay" on a requirements list, in the same way that nobody writes down that the floor holds.

That availability cost nothing to arrange, and it is worth seeing why, because the reason is a property of the write path rather than a feature anyone built. A row was written once and never updated, so the text of turn four was the same text on the day of the dispute as it had been on the day it was sent. The outbound request was then assembled from those rows by code running in your own process, which makes the request body a function of the stored rows and of nothing else. A function whose inputs cannot change returns the same output every time it is called, so running the assembly step again over the same rows produced the same bytes. Nobody arranged that; it fell out of writing turns down and building the array from them.

This chapter is about the properties that were consequences of owning the bytes and become terms of an agreement once a provider holds them. Two of them arrive on a clock — an erasure request has a statutory deadline, and a dispute arrives whenever it arrives — and the rest are discovered late, which is the worst time, because the mitigation for every one of them is a local copy that undoes part of the saving that motivated the move.

The capability nobody listed as a requirement#

Exact replay needs five things, and it is worth being precise about them because the difference between owning state and renting it lands unevenly across the list. You need the message sequence as it stood at that turn; the retrieved or tool-produced text that was in context; the system prompt version in force; the tool definitions in force; and the model identifier actually served. Application-owned state gives you the first two by construction and the other three by ordinary discipline — you version your prompts, you version your tool schemas, you log the model string from the response.

The distinction between by construction and by discipline is the one to hold, because only the first survives a team's attention moving elsewhere. A message row is correct by construction: the code path that sends a turn is the code path that writes it, so an input that reached the model and a row that failed to be written cannot both exist. The last three are correct only for as long as somebody keeps them so. Pinning a prompt in version control feels like the same kind of guarantee and is not, because a repository records that a version existed rather than which version was in force when a particular turn was sent. Writing the prompt identifier and the tool-schema identifier into the run record at call time is what closes that gap, and it closes it by converting a fact about your repository into a fact about that request.

Hosted state gives you none of the first two by construction, and the reason is not that providers are careless. It is that a conversation object is a live working set, not an append-only log. Two properties follow from that, and both are correct engineering decisions with the same inconvenient consequence.

Retention windows mean the object has an expiry. A conversation, an uploaded file, and a vector store each carry their own window, typically measured in days to weeks unless you have negotiated otherwise, and they are not necessarily the same window. The transcript can outlive the file it referenced, which leaves exactly the state that looks reconstructible and is not.

That mismatch has a mechanism worth naming, because it decides which pairs of objects can come apart. Each window starts when its own object is created, and the three objects belonging to one case are created at three different moments: the conversation on the first turn, the file when the customer uploads it, the store when an index over that file is built. Three clocks started at three moments and running for three lengths expire on three dates. A window is also a maximum rather than a minimum — it states the longest the provider undertakes to hold the object, which is a different statement from a commitment that the object will still be there tomorrow.

Server-side compaction means the object mutates. When a hosted conversation approaches the context window the provider summarises or drops older material so that the thread can continue, and this is a feature — it is the thing that lets a hundred-turn conversation work at all. It is also destructive to the historical record, because the fold happens at a moment determined by the provider's threshold, using the provider's summariser, with no event delivered to you and no version of the pre-fold state retained on your side. Nineteen turns become fourteen items, and nothing in your database records that it happened.

The destructive part is not the summarising, it is the moment the summary is written. A summariser that ran at assembly time — folding only the copy being sent to the model, and leaving the stored items alone — would cost the same tokens on that turn and lose nothing, because the originals would still be in the object afterwards. Compaction folds the object itself, which is precisely what makes it cheap on every later turn, since the work is done once rather than on each call. That same choice is what removes the pre-fold text, and no request you can send afterwards undoes it. The provider is taking the side of the trade that serves a live conversation, which is the wrong side for a record and the right side for the product.

A third loss sits underneath both retention and compaction, and it holds even for a conversation that has not expired and has not been folded. Replay needs the bytes the model read, and under hosted state the step that turns stored items into those bytes runs on the provider's side. What an items endpoint returns is their record of the conversation in the shape they chose to expose, and the assembled input sits downstream of that: the ordering, the placement of tool results, the formatting of role markers and whatever the provider prepends are all decided by code you cannot run. Reading the items back therefore recovers the material the input was built from rather than the input. That is why the strongest promise available is a best-effort read-back, and why the difference stays invisible until somebody has to defend a reconstruction.

Put the three together and the honest statement of what you lost is narrow and important: not access to the conversation, which you have, but the guarantee that what you can read now equals what the model read then. You can read back what the provider still holds. You cannot demonstrate that it is the input a past run used, and for a dispute, an audit, or a regression bisect, demonstration is the whole job.

Erasure becomes a fan-out with receipts#

The other direction has the same shape and shows up sooner, because erasure requests arrive on a statutory clock. A customer exercises their right to erasure, the statutory right under GDPR and its equivalents elsewhere to have personal data about them deleted on request, within a deadline, with the burden on you to show it happened. Their conversations live in the provider's hosted state and three documents they uploaded live in the provider's hosted file store.

Under the old arrangement this was one transaction. DELETE FROM cases WHERE id = ? with cascades to messages and attachments, committed, done, and the evidence that it happened is the transaction log plus the row's absence — for as long as that log is retained, and provided no replica or backup restore puts the row back. Under the new arrangement it is a distributed operation across a system you do not control, with partial failure as a normal outcome, and the deliverable is not just the deletion but a record that each part of it succeeded.

What changed there is not only the number of calls. It is where the evidence comes from. A local delete and the evidence for it are one event observed twice: the statement commits, and afterwards the same query that would have found the row finds nothing. Action and proof are available to the same party through the same interface, which is why nobody ever had to think about the second one. A delete issued across an API separates them. You send an instruction, you receive an acknowledgement of the instruction, and at no point do you observe the store. The strongest thing you can truthfully say afterwards is that you asked and were told it had been done, which is why the artefact the job has to produce is a receipt rather than a deletion.

def erase(case) -> dict:
    receipts = []

    def call(kind, ident, fn):
        try:
            fn()
            status = "deleted"
        except NotFoundError:
            status = "already_absent"      # idempotent: counts as success
        except Exception as exc:
            status = f"failed: {exc.__class__.__name__}"
        receipts.append({
            "kind": kind, "id": ident, "status": status,
            "at": datetime.now(timezone.utc).isoformat(),
            "request_id": getattr(client, "last_request_id", None),
        })
        return status

    for file_id in case.provider_file_ids:            # three documents
        call("file", file_id, lambda: client.files.delete(file_id))
    call("vector_store", case.provider_store_id,
         lambda: client.vector_stores.delete(case.provider_store_id))
    call("conversation", case.provider_conversation_id,
         lambda: client.conversations.delete(case.provider_conversation_id))

    ok = all(r["status"] in ("deleted", "already_absent") for r in receipts)
    ErasureRecord.objects.create(subject=case.subject_id, receipts=receipts,
                                 complete=ok)
    if not ok:
        raise ErasureIncomplete(receipts)             # retry, then escalate
    return receipts

Five delete calls where there used to be one commit, and the interesting lines are the ones that are not delete calls. already_absent has to count as success or a retry of a partially-completed erasure will report failure forever. Every call needs its own receipt, because "we ran the erasure job" is not evidence about the third document. The receipts have to be stored durably on your side, since the only proof that you deleted something is a record kept somewhere it was not deleted from. And the whole thing has to be retryable, because a 503 on the second file leaves you with a subject whose conversation is gone and whose photographs are not.

The first of those four lines rests on an argument worth making explicit, because on its face treating a missing object as a success looks like a job marking its own homework. A not-found response says that the identifier does not resolve, and that is equally consistent with an earlier attempt of yours having succeeded and with the object never having been created at all. Both are acceptable end states for an erasure, because the obligation is that the data is not there rather than that you were the one to remove it, and that is what licenses collapsing the two into one status. Counting not-found as a failure would make a retry fail on precisely the objects that had already been dealt with, so a partially completed erasure could never be driven to completion by running it again.

Three failure modes are worth naming because they are all silent. Objects created and never recorded — a vector store made during an experiment, a file uploaded by a code path that predates the tracking table — are undeleteable in practice, because the erasure job iterates your records rather than the provider's account. Objects that outlive their reference: delete the case row before the fan-out and the handles go with it. And deletion that is not purge, where the API returns success and the provider's documentation describes removal from backups on a longer schedule, which means the honest answer to "is it gone" has two dates in it, not one.

The silence is not a coincidence in any of the three, and the cause is the same one each time: nothing on the provider's side is organised around your subject. Their objects are addressed by identifier, and an identifier is very nearly all they hold about the object — there is no query that asks which files belong to a person, because a person is a concept in your schema rather than in theirs. So enumeration can only run over your own records, which makes an object created by a code path that never wrote a tracking row unreachable in principle rather than merely overlooked, and which is why the fan-out has to happen before the local delete that would destroy the identifiers it needs. The same asymmetry accounts for the third mode: what you can observe is whether an identifier still resolves, and an identifier stops resolving well before the last copy of the bytes stops existing.

The gap between deletion and purge hides a reporting question that gets settled badly under time pressure. A statutory erasure response typically has to be sent inside a month, and the deletion you can evidence on day two is removal from retrieval, not purge from every replica. The workable answer is to say what is true — the records were deleted on this date, and the provider's documented purge completes within their stated window — and to make sure the receipt row carries both the call timestamp and the provider's request identifier, because in a year's time the person answering a follow-up will have nothing else. What does not work is discovering at that point that the erasure job logged a single line reading "erased case 88213".

The same shape, four more times#

Once you see the pattern — a property that used to be a consequence of owning the bytes becomes a property of an agreement — the rest of the list follows without much argument.

Residency deserves a sentence of mechanism rather than just a row, because it is the one place where providers have shipped a real control. Storage region and inference region are separate facts: the conversation object rests somewhere, and each turn that reads it is served by a model running somewhere, and those are configured independently. Providers now expose a pin for the second — a field on the request or on the agent configuration that constrains which geography serves inference, validated against an allowlist on your workspace every time a request is served rather than once at setup. That is a genuine guarantee and worth using. What it does not cover is the object at rest, the tool execution that happens inside the turn, or the network a hosted search reached from, each of which is governed by a different clause and may have a different answer.

Objects derived from your data are the gap a review tends to miss, because they are neither the conversation at rest nor the request being served. An index built over an uploaded file, a cache of an assembled input and a copy taken for abuse monitoring are each produced from material you supplied, and each is governed by whichever clause names it — in a good many agreements no clause names it at all. The pin earns the word guarantee for a specific and narrow reason worth holding onto: the check runs when each request is served rather than once at setup, so a misconfiguration cannot persist silently behind a settings page that still reads correctly. What a reviewer should make somebody demonstrate is therefore a served request whose response reports the region that served it, alongside the clause covering the object at rest. A screenshot of a settings screen evidences the configuration and not the behaviour, and the two come apart exactly when a rollout is partial.

Three rows in that table are counterintuitive enough to draw out. Legal hold is the mirror image of erasure and the harder one: an obligation to preserve, colliding with a system whose default behaviour is to expire. If a matter requires you to retain everything relating to a customer for the duration of a dispute, a thirty-day provider retention window does not help you retain anything — it actively deletes the material while you are under an obligation not to. The only reliable answer is a copy you control, which means that for any workload with a plausible hold obligation you are back to writing transcripts down, and the saving from removing the messages table evaporates.

Portability is the row people assume is fine because an export endpoint exists. Exports return what the provider stores, which is the compacted working set rather than the original turns, in a shape built for their API rather than for anyone else's. Moving to a second provider is not a matter of replaying an export into a new conversation object — their conversation model differs, their tool-result representations differ, and the summarised items in the middle of your threads have no equivalent anywhere. Portability degrades quietly, in proportion to how long you have been running. Eval fixtures degrade the same way and get their own treatment in chapter 7, because they fail in a form that looks like a model regression rather than an infrastructure problem.

Portability's proportionality to age has a cause, and the cause is compaction again. Each fold replaces original turns with text the provider's summariser wrote, and a fold is never undone, so the share of a long-lived thread that exists only as the provider's prose can only grow. A thread migrated in its first week moves almost intact. The same thread a year later moves as a compacted digest of itself, and the material that has gone is exactly the material a second provider would need in order to reproduce anything.

The fixture row is the one that reads as harmless, and the reason it is tempting is that the handle really is stable: the same string resolves today and next month, which is what a fixture is supposed to look like from the outside. A fixture is meant to be a constant, and a handle is a pointer to something the provider is permitted to rewrite. Substituting the pointer for the value means the input to the suite can change without anyone changing the suite, so a failure against yesterday's pass has two candidate explanations and nothing in the run to separate them. Storing the assembled bodies alongside the handle is what turns the pointer back into a value.

None of this is an argument against hosted state. It is a bill of items that used to be free and now have to be paid for, either in provider guarantees you negotiate or in local copies that undo part of the saving. The practical form of that bill is short: decide in advance whether a past input has to be producible and, if it does, capture request and response bodies yourself rather than relying on read-back; enumerate every provider-side object type your code creates and give each one a row, because the erasure job can only delete what your records name; and write the receipts down, because deletion you cannot evidence is indistinguishable, a year later, from deletion that did not happen.

Which raises the obvious question about the other side of the ledger. The reason anyone accepts this trade is that hosted state removes a database read from every turn and, so the reasoning goes, makes the whole thing faster and cheaper. Chapter 2's trace already contains a hint that half of that expectation is wrong — the request payload got 100× smaller while the token count did not move — and the next chapter runs both versions of the same workflow under a stated method to find out what actually changed.