Containers, gVisor and microVMs

A sandbox is an environment a program is made to run inside, arranged so that the environment rather than the program decides what the program can reach. Browsers have run strangers' JavaScript this way for two decades, and CI runners execute unreviewed pull requests the same way, so the mechanism is old. What an agent system changes is authorship, since a language model writes the code in response to text that may include a web page or a support ticket an attacker controls, which means the program about to run has an author nobody authenticated. Sandbox design is the set of decisions that fix what such a program can do: the boundary between it and your host, the traffic it may send out, the files it can name, the credentials it holds and for how long, the resources it may burn, and what it leaves behind. You meet it as whatever sits on the far side of a tool called run_code or execute_bash: the container an agent framework starts when a tool call arrives, the throwaway virtual machine behind a hosted code interpreter, the environment a coding agent runs an unattended build in.

This chapter is about the first of those decisions, the boundary itself. The sentence "we run model-generated Python in a container with a seccomp profile and a read-only root filesystem" is a sandbox specification, and it is an answer to some threat models and not to others — a threat model being a written statement of who the adversary is, what they are after, and which outcomes you refuse to accept. Which ones it answers depends entirely on where the boundary physically sits, which is a different question from how strong it sounds. Anthropic's Choose a sandbox environment gives the coding-agent case as filesystem scope, network egress and whether the boundary is per-run, and most engineers meet the whole subject as a block of configuration somebody wrote once and nobody has attacked since.

Three boundaries, three different things on the far side#

In March 2022 a bug in the Linux kernel's pipe buffer implementation, catalogued as CVE-2022-0847 in the public Common Vulnerabilities and Exposures index and nicknamed Dirty Pipe, let any unprivileged local process overwrite bytes in files it only had read permission on. The proof of concept was about two hundred lines of C. It worked from inside a Docker container against files on the host image, because the container and the host were using the same kernel code to splice pages into a pipe, and the container boundary has nothing to say about what that kernel code does once it is executing. Anyone running untrusted code under runc, the low-level runtime Docker uses to start containers, on an unpatched 5.10 kernel that week had a host compromise available for the price of compiling a file.

That is the concrete thing at stake in the choice. A container is a process on the host, with namespaces changing what it can see and control groups, always called cgroups, limiting what it can consume. When code inside it makes a system call — openat, io_uring_setup or splice are three of them, and a syscall is how any program asks the kernel to do anything at all — that call is serviced by the same kernel that services your kubelet, the Container Network Interface plugin that wires up pod networking, and every other tenant on the box. Seccomp, which is short for secure computing mode, narrows which of the roughly 350 syscalls on x86-64 the process may issue at all; the Docker default profile blocks around 44 of them, and a tuned profile for a Python sandbox can plausibly get the allowed set down to 60 or 70. The remaining 60-odd are still a direct line into several hundred thousand lines of kernel C, and kernel bugs are not evenly distributed across syscalls — splice, io_uring and the networking stack have historically produced more of them than getpid has.

Those three seccomp figures deserve to be put next to each other, because the word "seccomp" covers two things that are nothing like each other. Around three hundred and fifty syscalls exist; the Docker default profile blocks forty-four of them. Subtract and roughly three hundred remain available — the default profile removes about one syscall in eight and leaves the overwhelming majority of the kernel's interface open to the process. A profile tuned for a Python sandbox leaves sixty or seventy, which removes something like eight in nine. Those are different controls with one name, and a design document that says "with a seccomp profile" without saying which one has said almost nothing. The gap between them is also where the work is: getting from three hundred to seventy means finding out what your workload actually calls, breaking it a few times, and owning the profile as it drifts.

gVisor moves the line by putting a second kernel in the way. The Sentry is a userspace process, written in Go, that implements the Linux system-call interface itself — something on the order of 260 syscalls, enough for most application workloads. Guest code never talks to the host kernel directly. The Sentry does, through its own seccomp filter that is far tighter than anything you would apply to a general-purpose container, on the order of 55 host syscalls. A guest that finds a bug in splice finds a bug in Go code running as an unprivileged user, and to reach the host it then has to find a second bug in the narrow set the Sentry itself is allowed to use. The cost is that every syscall crosses a userspace boundary, so syscall-heavy workloads slow down measurably: a build that spends its time in stat and read can run 2x to 10x slower, while a numeric workload that spends its time in userspace CPU is barely affected.

Set the Sentry's host-syscall count against the tuned container profile and the numbers nearly coincide, which is the most instructive thing in this section. A well-tuned container lets the untrusted process issue sixty or seventy syscalls to the host kernel; gVisor lets the Sentry issue on the order of fifty-five. Two similar counts, and no similarity at all in what they mean, because the count is not the property that matters. In the container, those calls are issued by the attacker's code, with arguments the attacker chose, aimed wherever the attacker aims them. Under gVisor, the attacker's code issues its calls to the Sentry, and the calls that reach the host are issued by the Sentry with arguments the Sentry constructed to service a request it has already interpreted. So the question a syscall filter answers is "which entry points are reachable", and the question a userspace kernel answers is "who is choosing the arguments" — and only the second one takes a crafted splice invocation off the table. Any comparison that ranks these two by counting syscalls is comparing them on the axis where they happen to agree.

A microVM, meaning a virtual machine stripped down to the handful of devices a workload actually needs, moves the line to hardware. Firecracker boots a guest kernel under KVM, the hypervisor built into Linux itself, with a deliberately tiny device model — virtio-net, virtio-block, a serial console, a one-button reboot device, and almost nothing else. There is no BIOS, no PCI enumeration, no USB, no emulated graphics. The untrusted code runs against its own kernel, and to reach the host it has to get through that kernel and then out of the VMM — the virtual machine monitor, the host-side process that emulates those devices — which is roughly 50,000 lines of Rust that is itself jailed in a container with its own seccomp filter and dropped capabilities. Two independent bugs are required rather than one, and the second one has to be in a very small piece of code.

The diagram is worth reading as a count of hops rather than as a ranking. Each added hop is a place where an exploit has to work again, against different code, written in a different language, with a different bug history.

What the choice actually costs#

Strength is only one column, and on its own it decides nothing. Here is the same three options against the dimensions a design review has to weigh, with figures that are typical rather than guaranteed — measure your own, because they move with kernel version, storage backend and image size.

Before the rows are read individually, it is worth noticing which direction each of them runs, because that is where the "three points on a scale" reading dies. The first two rows are monotone: the boundary moves further from the host kernel as you go right, and what a splice bug buys shrinks as you go right. Not one of the cost rows follows. Cold start is the sharpest case — gVisor's is slower than the microVM's, which means the option most people rank second on strength ranks third on the cost that decides whether the design fits a latency budget. Memory reverses again: gVisor's per-instance floor is tens of megabytes, and the microVM's floor is the VMM plus a guest kernel plus whatever RAM you carve out for the guest, which is an order larger. Steady-state overhead reverses a third time, since gVisor's penalty falls on syscall-heavy work and the microVM's is a flat CPU tax. The finding is not in any single cell. It is that no column dominates the others once you look past the first two rows, so "which is strongest" and "which should we run" are different questions with different answers, and a review that only asks the first has skipped the part where the decision is made.

The device row deserves more attention than it usually gets. Teams reach for a container, then mount /var/run/docker.sock so the sandboxed process can start helper containers, or add --device /dev/nvidia0 because the workload needs a GPU, or grant CAP_NET_ADMIN so a test can set up a tun interface. Each of those re-opens a piece of the host that the isolation choice was supposed to have closed, and the seccomp profile in the design document says nothing about any of them. The GPU case is the sharpest, since the NVIDIA kernel driver is a large piece of closed-source code that both containers and gVisor pass through comparatively thinly, and putting a microVM around it requires device passthrough, which is a different and much wider interface than virtio-block.

The belief that gets people here is that the seccomp profile is the boundary, and it is worth saying why that belief is reasonable and still wrong. A syscall filter constrains one channel: the calls the process issues. A bind-mounted socket is not a call the filter has an opinion about, because talking to a socket is openat, write and read on a file descriptor, and those have to be permitted for the process to do anything at all. So the profile's verdict is identical on a sandbox with a Docker socket mounted and one without, and identical again on a sandbox with a GPU device node and one without. It is a defence that does not change when the thing it is credited with bounding changes, which means it cannot be what is bounding it. What actually bounds the device surface is the list of mounts and devices, and that list lives in a different file, reviewed by different people, on a different schedule.

Noisy-neighbor behaviour separates the options in a way that shows up as latency rather than as a breach. Under cgroups v2 you can set memory.max, cpu.max and pids.max, and those hold for anonymous memory and scheduler time. Page cache, kernel slab allocations for open sockets, and dentry cache pressure are much less cleanly attributed, so a tenant that reads a 4 GB file repeatedly can evict another tenant's working set without ever exceeding its own memory limit. A microVM's RAM is carved out at boot and the guest kernel manages its own page cache inside that allocation, which is why VM-per-tenant remains the honest answer when one tenant's behaviour must not be visible in another tenant's p99, the latency that 99 requests in 100 come in under.

The arithmetic that decides it#

Take a workload of 40 executions a second, each with under 200 ms of real work, running model-generated code. The concurrency you need for the work itself is small — by Little's law, which multiplies the arrival rate by the time each item spends in the system, 40/s × 0.2 s is 8 executions in flight. That is a comfortable number.

Now add a fresh microVM per execution. Boot to a Python interpreter ready to accept work is 125 ms at the optimistic end, and that lands on the front of every request, so the end-to-end p50, the median call, becomes roughly 325 ms and the concurrency requirement becomes 40/s × 0.325 s, or 13 VMs live at any moment. That part is survivable. The part that is not survivable is the churn: you are creating and destroying 40 VMs a second, each of which means a rootfs, meaning the root filesystem image, to attach as a device, a tap device — a virtual network interface on the host that the guest sees as its network card — to create and plumb into the host bridge, an IP to allocate, a jailer chroot to set up and tear down, that being the stripped directory tree the VMM process is itself confined to, and 40 teardowns a second racing the creations. Tap device creation and network plumbing alone commonly costs 10–30 ms of host CPU per VM, and at 40/s that is between roughly half a core and a core and a half burned on nothing but network setup.

Two of those figures are worth deriving rather than reading, because they are what turns an objection into a decision. The concurrency numbers come from Little's law and nothing else: multiply how often work arrives by how long each piece stays in the system. Forty a second times two tenths of a second is eight, and adding the boot to the front of each request lengthens the stay to a third of a second, which is why the same arrival rate now needs thirteen machines rather than eight. That is more than half as many machines again for no additional work done, and it is still the survivable part. The host CPU figure is the other one. Forty network setups a second, each costing ten milliseconds of host CPU, is four hundred milliseconds of CPU spent every second — near enough half a core doing nothing but plumbing — and at the top of the range, with forty teardowns a second racing the creations for the same locks and the same interfaces, it reaches the upper figure. Notice what kind of cost that is. It does not appear in any request's latency, it does not show up as a failing call, and it scales with the arrival rate rather than with the work, so it gets worse exactly when you are busiest.

Teams hit this and reach for the obvious fix, which is a pool of pre-booted VMs that get handed out and reused.

{
  "machine-config": { "vcpu_count": 1, "mem_size_mib": 256, "smt": false },
  "drives": [
    { "drive_id": "rootfs", "path_on_host": "/pool/rootfs.ext4",
      "is_root_device": true, "is_read_only": true }
  ],
  "network-interfaces": [
    { "iface_id": "eth0", "host_dev_name": "fc-tap-7", "guest_mac": "..." }
  ],
  "balloon": { "amount_mib": 0, "deflate_on_oom": false }
}

That configuration is the honest one for a pooled design: a read-only rootfs so the base image cannot be mutated, a fixed 256 MiB with no ballooning so a guest cannot grow into a neighbour's memory, and simultaneous multithreading, SMT, switched off so a co-resident guest cannot share a physical core with another tenant's thread. Writing it down is the easy part. The pool is where the design gets interesting, and the reason is that a pool is a decision to share.

Longer work, GPUs and single tenancy#

Three things would change the answer. Making the work longer — 2 seconds instead of 200 ms — makes a 150 ms boot a 7% tax and a microVM per execution becomes affordable at that rate. Making the workload need a GPU pushes you back toward containers or gVisor with device access, because passthrough widens the VM interface enough to erode the reason you chose it. Making the tenancy single — one customer's code against that customer's data, with no cross-tenant question — lowers what an escape is worth and can justify the cheaper boundary.

The first of those is worth checking rather than accepting, because the percentage is not the boot divided by the work. The boot is added to the work, so the call takes both, and the tax is the boot as a share of the whole call rather than of the part that does something useful. Run the same reading on the original workload and the boot is a bit over a third of a two-hundred-millisecond call, which is why one figure reads as a rounding error and the other as an objection. The lever is the ratio between a fixed setup cost and a variable amount of work, and it moves without anybody touching the sandbox — the same isolation choice that is unaffordable on short calls is nearly free on long ones, which is why "how long is a typical execution" belongs in the isolation discussion and is almost never in it.

Answering the question with "the strongest one" is the move to avoid, and by this point the reason is assembled rather than asserted. Strength orders the first two rows of the comparison table and nothing else in it; the cost rows run in three different directions; the concurrency arithmetic turns a boot time into a fleet size and half a core of host CPU; and the option that wins on strength at this arrival rate is only purchasable as a pool, which reintroduces the sharing it was chosen to remove. Every one of those steps is checkable against a number already on the page, which is what makes the conclusion something a reviewer can argue with productively rather than a preference.

Picking an isolation technology also decides nothing about what happens when code uses the capabilities you deliberately gave it. A microVM with a network interface and a credential in its environment will exfiltrate a database happily, at the full speed of virtio-net, without a single kernel bug being involved. The escape-proof boundary and the leaky design are entirely compatible, which is why the rest of this topic is about egress, filesystem scope, credentials and durable state rather than about hypervisors. The same split runs through topic 6.14, on trusting third-party servers, where a confined server process is shown to bound reach without bounding behaviour.

There is one further confusion worth heading off, because it costs teams a boundary they think they have. A scope a program is told about is not a boundary at all: a list of directories handed to a cooperating process constrains code that is trying to behave and nothing else, which is the subject of topic 6.9, on roots and OS-level sandboxes. Everything in this chapter is about the other kind, where the kernel or the hypervisor holds the description and consults it whether or not the program agrees.

So the answer to "which isolation" is a sentence with three clauses in it: this adversary, these resource caps, this latency budget — and the boundary that follows from them. Strength on its own picks the microVM every time and then loses it to a pool nobody specified. The choice recorded in the design document is in any case a statement about one instance in isolation, and production runs many instances, reuses them, mounts things into them for convenience and shares caches between them to save time. Each of those is a decision to share something, and sharing is the variable that actually determines what one tenant can read from another — which is where the next chapter starts, with a design that keeps cold start under 100 ms and leaves every session's credentials readable by the code beside it.