Snapshot and rollback

A snapshot is a frozen record of a subtree that the filesystem can take in milliseconds and later swap back into place; rollback is that swap, performed after a run finishes so the next run starts from a known state. Both are cheap, both are worth having, and both are routinely credited with a property they do not have.

The property they do have is local. Containment — deciding what a run could reach in the first place — is the work of the three preceding chapters: default-deny egress governs whether an outbound POST is possible at all, per-execution credentials govern how much data the run could reach before it decided to send any, and filesystem scoping governs what it could read locally. Rollback runs after all of those have had their say, which is why the question it answers is what the platform can restore rather than what the run was able to do.

Why a snapshot cannot reach past the disk#

Here is an incident write-up in four lines, of which the second and third are the problem.

14:02:11  run 8f21ac started, snapshot workspace@pre taken (btrfs, 4 ms)
14:02:19  tool call notify_webhook(url=..., body=<412 rows, customer contacts>) -> 200
14:02:26  tool call open_pull_request(repo=infra, branch=agent/cleanup-8f21ac) -> #4471
14:02:31  run finished, workspace rolled back to workspace@pre (11 ms), instance reused

What the rollback restored is exactly what a filesystem snapshot restores. The workspace overlay went back to its pre-run content, so the CSV the run wrote to /workspace/export.csv no longer exists, the four packages it pip-installed are gone, the query results cached in /tmp are gone, and the short-lived warehouse token that had been written to tmpfs is gone with the tmpfs. Every one of those was local state on a disk the platform owns.

The webhook POST and the pull request are on the other side of a line the snapshot cannot see. The 412 rows arrived at an endpoint somebody else operates, were written to their storage, and may already have been forwarded, indexed or read. Pull request #4471 exists, notified the twelve people watching the repository, triggered a CI pipeline that ran for ninety seconds and published a preview deployment at a URL that is now in the build logs. Rolling the filesystem back at 14:02:31 changed none of that, and no mechanism available to the sandbox could have.

The mental model that produces the mistake is the database transaction, and the comparison is worth making precisely because it explains the limit. A relational database can roll back because it owns every effect a transaction produces: writes go to pages under its control, durability is deferred until commit, and the undo log is complete because nothing escaped during the transaction. ROLLBACK works because there was nowhere for an effect to go.

A sandbox owns its local disk and nothing else. The moment a tool call crosses to an external system, the effect is durable in a store the sandbox has no authority over, and the transaction analogy stops applying. Distributed systems have a name for the workaround, which is the compensating transaction — the saga pattern's answer to the fact that you cannot two-phase-commit, meaning get every participant to agree to commit or abort together, across services that do not want to be coordinated. Compensation is available for some effects and it is approximate in every case, because a compensating action is a new event rather than the erasure of an old one.

The diagram splits the run's effects by which side of the snapshot boundary they landed on, and the split is not about severity. Deleting a file is a bigger operation than sending 412 rows, and the file comes back.

It is not about recency either, which is the other intuition worth dismantling, because "roll back" sounds like winding a tape. Read the four timestamps as intervals. Eight seconds of purely local work, then a seven-second stretch in which the first external effect lands, then five more seconds in which the second lands, then the restore. A file the run wrote a second before the restore is recovered; rows delivered twelve seconds before it are not. The line the mechanism draws runs through the machine, not through the clock, and the question it answers for any given effect is who holds the storage it landed in. That also explains why the mechanism feels more powerful than it is. Taking the snapshot took four milliseconds and the restore took eleven, which is three orders of magnitude faster than the events it is being credited with undoing — and speed on the local side is exactly what you would expect from an operation that never leaves the disk.

What compensation can and cannot recover#

Listing the effect classes with their compensating actions makes the boundary usable in a design review, because the question stops being "does rollback help" and becomes "which column is this effect in".

Three things in that table are decided by where the cells are rather than by what they say. The first is that the yes column and the platform owns the storage property are the same three rows, with no exceptions and no partial cases — a file in the workspace, a package installed, a credential cached on disk. There is no row that is half-recoverable, which is what makes the rule usable in a review: you do not have to reason about the effect at all, only about which disk it landed on. The second is that the last column is filled in for every single no row. Compensation always leaves something, so no amount of engineering converts a no into a yes; it converts it into a no with a smaller remainder. The third is the one empty cell. Paid API quota has no compensating action at all, which sits oddly next to rows about customer contacts until you see what it is telling you: it is the only effect whose entire content is the consumption of something, with no artefact left to delete and no party to ask. Effects that consume rather than create are invisible to every mechanism in this chapter.

The rows that matter most are the ones with no compensating action at all. Data that has left is gone in the sense that matters, since the ability to make someone delete their copy is not the ability to make them not have read it, and the ability to close a pull request is not the ability to un-notify. That is why rollback is a recovery mechanism and not a containment one: it restores your ability to keep operating after a bad run, and it does not reduce the harm the run caused outside.

Following one compensating action all the way through is what stops "we can undo it" being said loosely. Closing pull request #4471 does not remove it. It appends a closed state to a record that still exists, still carries its diff, and still sits in the notification history of the twelve people who were watching; the CI run already happened and its artefacts are already published. Deleting a row from your own database does not reach the replica that copied it, the backup taken in between, or the job that read it while it was there. Each of those is the same fact in different clothes, and it is the definition given above doing its work: a compensating action is a new event, and new events pile on top of old ones rather than removing them. An audit trail is a good thing that has this as its cost, which is worth saying plainly, because a system that could genuinely erase its own history would be a worse system in every other respect.

Naming the limit is not an argument against the mechanism, and it is worth being specific about what snapshotting genuinely buys, because a control that only ever gets described by its weaknesses tends to get removed. A run that fills the disk with a runaway log, corrupts a checkout mid-rebase, installs a package version that breaks every later run on that instance, or leaves a half-written Parquet file that the next job reads as valid — all of those are recovered for the cost of a subvolume swap, and without snapshots each of them is an operator paging through a dirty machine. Snapshots also make incident work possible: keeping the post-run snapshot for twenty-four hours before destroying it gives you the exact filesystem the run produced, which is how you answer questions about what it wrote and in what order after the instance itself is gone. The four-line write-up above was only possible because the pre and post snapshots were both still on disk.

Verifying that nothing local persisted#

Even for the effects rollback does cover, "we rolled back" is a claim about a command's exit code, and the interesting failures are the writes that were never inside the snapshot's scope. A snapshot of /workspace says nothing about a file written to /opt/cache, a systemd user unit dropped in ~/.config, which is a service definition the init system will start on that user's behalf, an at job — a one-off command queued with at to fire at a chosen time — scheduled for eleven minutes later, or a process the run left running with a socket bound. Verification means comparing the instance's whole observable state against the state it started in.

#!/usr/bin/env bash
# runs on the host after rollback, before the instance is returned to the pool
set -euo pipefail
RUN=$1; PRE=/var/lib/attest/$RUN.pre; POST=/var/lib/attest/$RUN.post

capture() {   # everything that must be identical, not just the workspace
  find / -xdev -newermt "@$RUN_START" -not -path '/proc/*' -not -path '/sys/*' \
       -printf '%p %s %T@\n' | sort
  ip -o addr; ss -ltnp; ls /var/spool/cron/crontabs /var/spool/at 2>/dev/null
  ps -eo pid,ppid,user,cmd --no-headers | sort -k4
  systemctl --user list-units --no-legend 2>/dev/null || true
}

capture > "$POST"
if ! diff -q "$PRE" "$POST" >/dev/null; then
  diff "$PRE" "$POST" | head -50
  echo "residue detected for $RUN; quarantining instance"
  destroy_instance "$RUN"      # never return a dirty instance to the pool
  exit 1
fi

Three properties of that script carry the weight. It looks at the whole filesystem rather than the snapshotted subtree, which is what catches the write that went somewhere unsnapshotted. It looks at processes, sockets and scheduled jobs, because a run's most durable artefact can be a running program rather than a file. And its failure action is destruction rather than a warning, since an instance that failed verification is precisely the instance you least want to hand to the next tenant. Teams that log the diff and reuse the instance anyway have built a detector with no actuator attached.

The cost is small enough not to be an argument. A subvolume snapshot on btrfs or ZFS, filesystems that can freeze and restore a named subtree as a unit, takes 4–8 ms to create and the rollback is a subvolume swap at 10–15 ms; discarding an overlayfs upper directory is closer to 15 ms plus the asynchronous delete. Verification as written above takes 300–900 ms on a small root filesystem, which is too slow to run inline on a 300 ms budget and is entirely reasonable to run during the instance's return to the pool, off the request path.

There is a blunter alternative that deserves consideration whenever the verification logic starts growing conditionals, which is to skip rollback and destroy the instance. Recreating from a golden image, the known-good base every instance is built from, proves the post-condition by construction — there is no residue because there is no continuity — and its cost is the cold start from chapter 1, on isolation boundaries, so 30–80 ms for a container and 125–200 ms for a microVM. On a workload with seconds of real work per execution, destroy-and-recreate is the correct default and rollback is an optimisation nobody needs. Snapshot-and-rollback earns its complexity specifically where the instance is expensive to rebuild: a warm interpreter with 400 MB of libraries imported, a browser already launched, a just-in-time compiler that has warmed up, a model loaded into memory. The trade is explicit, which is that you are keeping continuity for speed and taking on the obligation to prove what that continuity carries forward.

Adding the figures up shows how lopsided that trade is, and the arithmetic is not the intuitive one. A rollback you actually believe costs the restore plus the verification, which is a handful of milliseconds plus somewhere between a third of a second and most of a second, because a restore without the verification is a claim about an exit code. Set that total against the cost of having no continuity at all. A fresh container is tens of milliseconds and a fresh microVM is a couple of hundred, and both of those are smaller than the verification on its own. On cost grounds, then, a plain container has no case for rollback at any point: proving the instance is clean takes longer than building one that is clean because it has no history. The comparison only turns where the instance carries state a fresh one would not — the warm interpreter, the launched browser, the warmed compiler — and what you are buying there is not the milliseconds of the restore but the seconds of rebuilding that state. Sizing the obligation against that saving is the calculation, and it is the one that sets expectations honestly: continuity is worth its proof only when the continuity itself is worth something.

Staged effects, and the limit of mediation#

The uncomfortable part of the effect table is that the strongest column is empty for exactly the effects that hurt. Research on tool-using agents has started attacking that directly rather than accepting it: Cordon: Semantic Transactions for Tool-Using LLM Agents proposes staging an agent's external effects instead of applying them as they are produced, mediating the authority to commit them, and tracking lineage so that a commit decision can be made over the whole set of effects once the run has finished. In that shape, the webhook POST at 14:02:19 would have been recorded as a pending effect, the run's later behaviour would have been evidence against committing it, and the rollback at 14:02:31 would have discarded it along with the files.

The limit on that approach is the same one that limits everything else in this chapter, which is mediation. An effect can be staged only if it passes through machinery you control, so a tool your harness proxies can be deferred and a call the model makes through a library that opens its own socket cannot. Staging is a strong argument for routing every external effect through a small number of harness-mediated tools rather than letting sandboxed code speak to the internet directly, even where the egress policy would permit the destination.

So the sentence to write in a design document is that rollback restores the instance, not the world. It is worth its few milliseconds, it is what makes a dirty machine an operational non-event, and it keeps the evidence an incident needs — and every effect that crossed to a system you do not own is standing, whether or not the restore succeeded.

For a pooled sandbox, rollback and reset are the same operation described from two directions: rollback is what you owe the run that just finished, and reset is what you owe the run about to start. The verification script above is doing the second job, and it is doing it for one instance at a time under assumptions somebody wrote down. The next chapter is about writing those assumptions down deliberately — what "reset" has to mean before a warm pool serving multiple tenants is something you are willing to sign off, and what proofs you demand before you do.