Signing off a warm pool

Signing off a sandbox design means putting your name to a claim about a boundary, and the claim is only as good as the words in it that have definitions. "Pooled warm Firecracker microVMs, reset between executions" is the right shape for a workload that cannot pay a boot per request, and the whole of the risk is sitting inside the word "reset", which is a cleanup step in most implementations and has to be a security control in this one.

The workload that forces it is worth stating concretely, because the numbers do the deciding. Customers write their own transformation code and run it against their own data in your product, on demand, all day, and the whole call has a 300 ms p95 budget — 95 calls in 100 must finish inside it — because it sits behind a UI control that people click repeatedly.

Three sentences, then the technology falls out#

A threat model that runs to eleven pages gets read once. Three sentences get argued about, which is the useful outcome, and the three are adversary, asset, and the escape you refuse to accept.

The adversary is any customer who can submit code, together with anyone who can influence the data that code processes, since a transformation over an uploaded file is steerable by whoever wrote the file. The assets are other customers' data and credentials, plus the host and the control plane that schedules work onto it. A compromise of one execution must not yield another tenant's data, another tenant's credentials, or code execution on the host, and we accept that a customer can do anything they like inside their own execution with their own data.

The technology follows without much argument. The adversary supplies the code, so the syscall surface is exercised deliberately rather than incidentally, which puts a shared host kernel on the wrong side of the line from chapter 1, on containers, gVisor and microVMs. Multiple tenants share machines, so the boundary has to hold between guests and not only between guest and host. That is a hardware-virtualised boundary, and Firecracker is the version of it that boots in a time worth discussing.

Each step in that derivation is doing work, and it is worth seeing which of them are forced. The first sentence of the threat model says the adversary writes the code, and that is what rules out the shared kernel. The reason is specific rather than a general claim about containers: the untrusted process is choosing which syscalls to make and what arguments to pass, and a shared kernel means those arguments reach the same code that serves everything else on the box. The second sentence says other tenants' data is an asset, and that is what rules out a boundary that only holds between guest and host: a design where every guest is perfectly isolated from the host and reachable from its neighbours satisfies the first requirement and fails the second. The third sentence, the one accepting that a customer may do what it likes inside its own execution, is the one that keeps the scope finite — without it, the design would have to police what the code does rather than what it can reach, which is a different discipline and belongs to the tool-level controls rather than to the sandbox. Only the technology choice is a matter of engineering preference here. The shape of the boundary follows from the three sentences without a decision being made.

Where the 300 milliseconds go#

Budgets get argued in the abstract until someone writes down the line items, at which point the argument usually ends.

That last row is the arithmetic of the table above it, and it is worth doing rather than assuming: with the mint running in parallel with acquisition, as the credential design in chapter 5 assumes, the acquisition stage has 80 ms to work in and has to cover the 55 ms mint inside it. Boot-per-execution needs 125–200 ms and does not fit in 80 ms, and no amount of tuning closes a gap of that size.

The table has two readings and both of them close, which is what makes the row trustworthy. Read serially, the four stages with numbers against them account for two hundred and seventy-five of the three hundred milliseconds, leaving twenty-five for acquiring an instance — and twenty-five milliseconds buys nothing at all. Read with the mint running alongside acquisition, the mint's fifty-five drops out of the sum, the same subtraction leaves eighty, and the acquisition stage has to finish inside that with the mint completing inside it too. So overlapping the mint is worth fifty-five milliseconds, more than doubling what is available for acquisition, and it is still not remotely enough for a boot. Even the optimistic end of boot-per-execution overruns the whole stage by more than half again, which is the sort of gap that does not respond to engineering.

Pooling is what fits, and stating what you have accepted by pooling is the substance of the sign-off. You have accepted that instances have continuity across executions, that continuity is shared between tenants unless placement forbids it, that "reset" is now a security control with a definition and a test rather than a cleanup step, and that some residual risk from co-residency remains no matter how good the reset is.

The reset ladder#

Reset is not one thing, and the levels differ in what they restore and what they cost. Writing them as a ladder makes the choice explicit and makes the gap between "what we do" and "what we claim" visible.

Level 2 is the answer that fits the budget, and it is not the intuitive one, because most teams reading "reset" imagine level 0 or level 1. Firecracker can take a full snapshot of a paused VM's memory and disk state and resume from it; with the memory file backed by userfaultfd, a kernel facility that lets a userspace process supply pages as they are first touched, so pages fault in on demand rather than being copied up front, resume is a handful of milliseconds and the working set materialises as the execution touches it. That restores things no userspace cleanup can reach: a sysctl the previous run changed, meaning one of the kernel's runtime tunables, an iptables rule it added, a kernel module it loaded, an established TCP connection to the warehouse it left open, the guest page cache holding the previous tenant's file contents, and — this is the one that matters most — a compromise sitting in guest kernel memory after a successful local privilege escalation inside the guest. Level 1 cannot touch any of those, because level 1 is a process running under the kernel it is trying to clean.

Read the cost column down the ladder and it does not do what a ladder is supposed to do. Level 0 to level 1 costs twenty extra milliseconds and buys process state and /tmp. Level 1 to level 2 buys everything in that list of kernel-resident state — and costs less than level 1 did, because resuming a paused VM from a snapshot is cheaper than killing processes and re-executing a supervisor. Then level 2 to level 3 leaps by an order of magnitude, to a full boot. So the cost is not monotone in strength: it rises, falls, and then jumps, and the cheapest of the four levels is the second strongest. That single fact is why "reset" gets implemented as level 1 in systems that could afford level 2 — the ladder was assumed to be a price list, and the assumption was never checked against the numbers.

The Leaves column settles the top of the ladder just as cleanly. Levels 2 and 3 leave the same thing: host-level and hardware-level sharing, which is co-residency and not a property of the guest at all. Destroying the VM and booting a new one therefore buys exactly one thing over restoring a clean snapshot, which is that it has no snapshot to reuse, and it charges an order of magnitude for it. Put those two readings together and the choice makes itself. The gap between level 1 and level 2 is enormous and costs nothing; the gap between level 2 and level 3 is narrow and costs everything. A design review that treats the ladder as "how much security can we afford" will land on level 1 and describe it as a compromise, when the compromise it thinks it is making is not available on this table.

Diagram

The two edges into Quarantined are the ones that get dropped during implementation, and they are what make the rest of the machine meaningful. An instance that fails its residue check must not go back to Idle, and an instance that produced an anomaly signal during a run must leave the pool immediately rather than at the end of its shift. Retirement on age and on execution count is the backstop for everything nobody thought of, and it is cheap: 2,000 executions per instance at 40 executions a second across a 60-instance pool means an instance retires roughly every fifty seconds, which is one extra cold boot per fifty seconds of operation.

That figure comes out of two divisions and is worth following, because retirement policy is the part of a state machine people set by feel. Forty executions a second spread over sixty instances gives each instance two thirds of an execution a second, so reaching two thousand executions takes three thousand seconds — fifty minutes of service before an instance is due to retire. Sixty instances each retiring after fifty minutes, staggered, means one leaving the pool roughly every fifty seconds, and one boot every fifty seconds against forty executions a second is a rounding error in both capacity and cost. Now put that fifty minutes next to the other retirement edge, which fires at an hour of age. At this arrival rate the two conditions come due within ten minutes of each other, and the execution count reaches its limit first — so the age edge is doing nothing here, and it is still worth keeping, because it is the one that still fires when traffic collapses and instances sit idle for a shift. Two limits that nearly coincide at today's load are a pair, not a duplication; they diverge exactly when the load does.

Level 2 carries a caveat that has to go in the sign-off explicitly. Restoring the same memory snapshot repeatedly means every restored instance starts with identical kernel entropy state, identical RNG pools, and identical values in any memory the snapshot captured. A guest that generates a TLS session key or a UUID immediately after resume can generate the same one on every instance, which is a correctness and a security problem at the same time. The fix is to inject fresh entropy into the guest after resume, over vsock — the socket type that carries messages between a host and its guests without a network — or through virtio-rng, the paravirtualised random-number device, before any application code runs, and to treat that injection as part of the reset rather than as an optimisation somebody can disable.

The chain behind that caveat is short and every link is checkable. A memory snapshot captures the guest's RAM; the kernel's random-number state lives in the guest's RAM; restoring the snapshot therefore restores that state byte for byte on every instance resumed from it. Anything drawing randomness before something new has been mixed in draws from an identical pool, so two instances resumed from one snapshot produce the same first value — the same UUID, the same session key, the same nonce. What makes this worth dwelling on is that it is the strength of level 2 turning against you. The reason level 2 reaches state no userspace cleanup can touch is exactly that it restores kernel memory wholesale, and the entropy pool is kernel memory. The mitigation has to run after resume and before application code, which puts it inside the reset by construction, and it is the sharpest illustration in this chapter of why the word needs a written definition rather than a name.

What you insist on proving#

A sign-off that says "the reset is adequate" is a sentence you will be reading back to yourself in an incident review. The signable version names the artefacts that produce evidence and who owns keeping them green.

The first is the egress canary suite from chapter 3, on default-deny egress, run against the real pooled image with the real policy, on every change to the image, the policy, the CNI and the runtime, with the verdict coming from the canary rather than from inside the sandbox. The second is a residue test that runs continuously in production, not only in CI: a synthetic tenant's execution plants a marker of each kind — a file in the workspace, a file in /tmp, an environment variable, a sysctl change, a listening socket, an at job, a .pth file in site-packages, a large file read to warm the page cache — and a second synthetic execution scheduled onto the same instance looks for all eight. Any hit pages someone. The third is the mount-table assertion and the collector traversal test from chapter 4, on filesystem scoping, since convenience mounts arrive one merge at a time and no reset restores a boundary that was never there.

The eight markers in that second proof are not a list of things that seemed worth checking; they are chosen against the ladder. A file in the workspace and a file in /tmp fall to level 1, and so does an environment variable on the supervisor. A .pth file in site-packages and an at job survive anything that does not restore the disk they live on. And three of them — a changed sysctl, a socket left listening, and a page cache warmed by reading a large file — are precisely the ones a level 1 reset cannot reach, because they are kernel state and level 1 runs under that kernel. So the marker set spans the boundary between the two levels, which is what lets a single test tell you which one you actually implemented rather than which one the document claims. A residue test made only of files would pass on a level 1 system and prove nothing, and it is the version most teams write first.

The fourth is the credential evidence from chapter 5, on per-execution credentials: no long-lived credential visible from inside the sandbox, tokens minted per execution from context the model cannot influence, and a replay test proving a completed run's token fails sixty seconds later. The fifth is the reset ladder itself, written down, with the level you actually implement and a line per dimension it does not restore. The sixth is placement: a statement of which tenants may co-reside on a host, whether SMT is disabled, and whether any downstream system authorises by source IP, since tap devices and their addresses are recycled with the instance.

The seventh is the least technical and the most predictive of whether any of this is true in six months. Each of the above needs a named owner, a dashboard showing when it last ran and whether it passed, and an alert when it has not run. A test suite that proves a boundary and has not executed since March is a claim about March.

Co-residency, and a guest kernel inside the snapshot#

Level 2 reset plus per-execution credentials plus default-deny egress plus a scoped filesystem leaves a specific and statable residue, and saying so is part of signing rather than an admission that the work failed.

Co-resident guests, meaning guests placed on the same physical machine, share hardware, and microarchitectural side channels between them — the timing differences in shared caches and buffers that let one guest infer what another is doing with them — are not closed by any reset, because they are not state the VM owns. Disabling SMT removes the sharpest of them at a cost of roughly 20–30% of throughput per host. Tenant-aware placement — never scheduling two tenants of materially different sensitivity onto one host — narrows the rest, and it does not eliminate them, and no vendor will tell you it does.

A guest kernel bug that the snapshot restore does not clean, because it was present in the snapshot, persists across every instance restored from that snapshot. That is an argument for rebuilding and re-snapshotting on the same cadence as your patching, and for the age-based retirement edge in the state machine.

Everything in this topic bounds what one execution can take from another. None of it bounds what an execution does with the authority you gave it on purpose, which remains the domain of the tool-level controls from topic 7.5, on tool execution safety, and none of it undoes an effect that has already crossed to an external system, as chapter 6, on snapshot and rollback, established.

The sign-off, then, is not a yes or a no, and it is not a paragraph in a design document either. It is a memo in which every word that carries risk has been given a definition, a level and an owner — "reset" above all, because it is the word a pooled design leans on hardest and the one most likely to mean rm -rf /workspace/* in the implementation. What you are signing is that the boundary is defined, tested continuously, and owned, which is the only form of that claim anybody can actually keep.