Filesystem scoping

"Each execution gets a scoped workspace mounted at /workspace" is the sentence a design document usually carries about the filesystem, and it describes where a directory was put. What the process experiences is a mount namespace — the kernel's per-process view of which filesystems are visible at which paths — and in that namespace the only question that decides anything is which paths resolve to something. Scoping holds when nothing outside the intended set can be named from inside.

The distinction matters because placement is stable and naming is not. A directory stays where you put it; the set of things nameable from inside a sandbox grows every time somebody mounts a socket for a test fixture, shares a cache to save install time, or exposes a credentials directory to unblock a colleague. An escape attempt that fails is the only form the evidence takes.

Scope is a property measured from the inside#

Here is what eighteen months of that growth looks like. Somebody ran findmnt, which lists every filesystem currently mounted where the process can see it, from inside a live sandbox during an unrelated investigation and printed this.

TARGET                  SOURCE                       FSTYPE   OPTIONS
/                       overlay                      overlay  rw,relatime
├─/proc                 proc                         proc     rw,nosuid,nodev
├─/sys/fs/cgroup        cgroup2                      cgroup2  ro
├─/workspace            /srv/runs/8f21ac/work        ext4     rw,relatime
├─/opt/cache            /srv/shared/pipcache         ext4     rw,relatime
├─/etc/pki/ca-trust     /etc/pki/ca-trust            ext4     ro
├─/var/run/docker.sock  /run/docker.sock             none     rw
├─/home/agent/.aws      /home/svc-agent/.aws         ext4     ro
└─/tmp                  tmpfs                        tmpfs    rw

Four of those nine lines are bind mounts added after the design review, a bind mount being a host directory or file made to appear at some path inside the sandbox while remaining the same object on the host. The Docker socket line alone ends the discussion, because a process that can write to it can start a container with / bind-mounted and root privileges, which is a host filesystem read and write with no exploit involved. The .aws mount hands over long-lived credentials. The shared pip cache is a channel by which one run writes a .pth file that a later run's Python imports on startup.

This has been understood for a long time and keeps being relearned. chroot, which changes what a process sees as / to some subdirectory, arrived in Version 7 Unix in 1979 and was in use as a build sandbox by the early 1980s, and it was never a security boundary: a process retaining a file descriptor to a directory outside the new root can fchdir to it and walk out, which is why FreeBSD added jails in 2000 to close the gap between "cannot name" and "actually cannot reach". Linux mount namespaces plus pivot_root, which swaps the namespace's root for a new one and then detaches the old one entirely, give the modern version of the same property, and they give it only if nothing punches a hole through afterwards. The four extra mounts in that findmnt output are holes punched afterwards, and they were punched by people who never thought of themselves as changing the security design.

Six vectors account for nearly every escape from a scope that was correct on paper.

The third and sixth are the ones teams miss, because both involve a process that is not the sandboxed one. A symlink inside /workspace pointing at /etc/shadow does nothing while it is being resolved inside the sandbox, where /etc/shadow is the image's own file. It becomes an escape when the harness — the code outside the sandbox that prepares a run, feeds it work and collects its results, and which necessarily runs with more privilege than the run does — tars up /workspace to collect results and follows the link on the host, or when a UI renders a "download artefact" path without normalising it. The sandbox held; the thing reaching into it did not.

# inside the sandbox: resolves to the image's own file, harmless
$ ln -s /etc/shadow /workspace/results/summary.csv
$ cat /workspace/results/summary.csv
root:*:19000:0:99999:7:::           # the image's shadow, no accounts

# on the host, the collector runs as the harness user and dereferences
$ tar -czhf /srv/artifacts/8f21ac.tgz -C /srv/runs/8f21ac/work results/
$ tar -tzvf /srv/artifacts/8f21ac.tgz
-rw-r----- root/shadow  1284 results/summary.csv    # the HOST's shadow file

The -h flag is what does it, and it is in that command because somebody added it two years ago after a bug report about broken symlinks in downloaded artefacts. The fix is on the collector: refuse any entry that is a symlink, refuse any path component equal to .., and resolve the final path with realpath before checking that it is still a prefix of the run directory. Doing the check by string manipulation is where this goes wrong a second time, because os.path.join(base, user_path) returns /etc/passwd outright when user_path is absolute, and os.path.normpath will happily normalise a traversal into a valid host path without ever touching the filesystem to find out what the components really are.

The step underneath that whole example is one sentence, and having it makes the rest predictable. A path is resolved by whoever opens it, in that process's own mount namespace, and not by the file it points at. So /etc/shadow inside the sandbox and /etc/shadow on the host are the same eleven characters naming two different files, and a symlink is a stored string rather than a stored reference. The sandbox writes the string; the collector resolves it; the collector is in the host's namespace. Nothing was smuggled out and no boundary was crossed by the sandbox — a more privileged process reached in, picked up a name the untrusted side chose, and resolved it against its own view. That is why the fix belongs on the collector and why it has to be realpath rather than string arithmetic: normpath reasons about the characters, which is exactly the layer at which the two files look identical, while realpath asks the filesystem and therefore asks in the namespace where the answer matters.

The second and fourth vectors are quieter and share a mechanism, which is that Python's import machinery reads files it was never asked to read. Any file ending in .pth inside a site-packages directory is executed at interpreter startup if its lines begin with import, and sitecustomize.py anywhere on the path runs unconditionally. A writable directory shared between runs — a pip cache, a wheel cache, a PYTHONPATH entry mounted from the image for convenience — is therefore a way for one execution to run code inside every later execution, without needing a network, a kernel bug or a single suspicious-looking file. The same shape exists for Node with NODE_OPTIONS and a shared node_modules, and for shells with a shared ~/.bashrc.

Walking that one step by step is worth the space, because it is the vector engineers most often decline to believe. A Python interpreter, at startup and before a line of your program runs, scans its site-packages directories. Any file there whose name ends in .pth is read. Any line in such a file beginning with import is executed. So write access to a directory on that scan path is not a data problem at all; it is arbitrary code execution inside every interpreter that starts afterwards, with no exploit and nothing that looks out of place to somebody listing the directory. Whether a shared cache reaches that path is the condition to check rather than assume, and there are two ways it does. Either the shared directory is itself on the import path, mounted as a PYTHONPATH entry for convenience, and the .pth file runs directly. Or it is a package cache, in which case one run poisons an artefact and the next run installs that artefact into a site-packages directory, where the same startup scan finds it. The mechanism is the same in both, and it explains why the fix later in this chapter is to make the cache read-only rather than to inspect what goes into it.

What makes all six survive review is that each one appears in a diff as a single line in a compose file or a Helm values file, in a change whose title is about something else. Nobody reviews - /run/docker.sock:/run/docker.sock as a security change, because the pull request is about test fixtures. That is the argument for asserting the mount table in CI rather than relying on review: the assertion is stated once, in a file whose whole purpose is the boundary, and a change that widens the scope has to edit that file explicitly and defend it.

Building the scope, then attacking it#

The configuration below — an OCI runtime specification, which is the JSON that runc and its equivalents read when they start a container — is the shape that survives review. Everything the process can write is per-run and disposable, everything shared is read-only, and the paths that leak host information are masked.

{
  "root": { "path": "rootfs", "readonly": true },
  "linux": {
    "namespaces": ["pid", "ipc", "uts", "mount", "network", "user"],
    "uidMappings": [{ "containerID": 1000, "hostID": 493216, "size": 1 }],
    "maskedPaths": ["/proc/kcore", "/proc/keys", "/sys/firmware", "/proc/1/root"],
    "readonlyPaths": ["/proc/sys", "/proc/sysrq-trigger"]
  },
  "mounts": [
    { "destination": "/workspace", "type": "overlay", "options":
      ["lowerdir=/srv/base/repo", "upperdir=/srv/runs/8f21ac/up",
       "workdir=/srv/runs/8f21ac/wk", "nosuid", "nodev"] },
    { "destination": "/tmp", "type": "tmpfs", "options":
      ["size=512m", "nosuid", "nodev", "noexec", "mode=1777"] },
    { "destination": "/opt/cache", "type": "bind", "source": "/srv/shared/pipcache",
      "options": ["ro", "nosuid", "nodev", "noexec"] }
  ]
}

The user-namespace mapping is the piece that does the most work per line. Container uid 1000 maps to host uid 493216, a subuid — one address in a subordinate uid range set aside for exactly this remapping — owned by nothing, so a file the sandbox writes with mode 0777 into a shared location is still owned by a host identity with no privileges anywhere, and root inside the container is not root outside it. Making the pip cache read-only removes the cross-run write channel at a cost measured in seconds: with the cache read-only, installs that hit it still take 6 s, and installs that need something absent fall back to the registry through the egress proxy from chapter 3, on default-deny egress.

Put that figure next to the one the cache was introduced for and the decision stops being a trade at all. The cache existed because installs took forty seconds without it and six with it. With the cache read-only they still take six. The whole of the benefit was in reading the cache, and none of it was in writing to it, so closing the write channel costs nothing measurable — the only thing lost is that a run cannot populate the cache for its successors, which a build-time step does better anyway. Security changes that cost nothing are rare enough to be worth recognising on sight, and the shape recurs: a resource is shared for the benefit of one direction of access, and the direction that creates the exposure is the other one. Ask of every shared mount which direction is paying for it, and remove the other.

The three mount flags repeated on every entry are not decoration. nosuid means a setuid binary — one carrying the bit that makes it run with its owner's privileges rather than the caller's — written into the workspace does not gain privileges when executed, which closes the classic path of dropping a setuid-root shell into a directory that some other, more privileged process will later scan or execute. nodev means a device node created inside the sandbox does not work, so mknod /workspace/mem c 1 1 gives an inert file rather than a window into physical memory. noexec on /tmp means downloaded binaries cannot be run from the one directory that has to stay writable, which is not a strong control on its own — an interpreter can still read and execute a script's contents — and it does raise the cost of the most common tooling used against sandboxes, which drops a compiled exploit into /tmp and runs it.

The overlay is what makes per-run workspaces affordable. Copying a 2 GB checkout for each execution costs about 3.5 s on an NVMe device sustaining 600 MB/s, plus 2 GB of disk per concurrent run. An overlay with the checkout as a read-only lower layer and a fresh upper layer costs about 12 ms to set up and only stores what the run modifies, which for a typical analysis job is a few hundred kilobytes. That difference is why shared writable directories keep getting proposed and why they stop being necessary once the overlay is in place.

Both halves of that comparison are worth doing on paper. Two gigabytes at six hundred megabytes a second is a little over three seconds of pure copying before the run has done anything, and the overlay's setup is measured in milliseconds, so the copy is on the order of three hundred times more expensive in time and several thousand times more expensive in disk — two gigabytes held per concurrent run against a few hundred kilobytes. The reason the overlay wins by that margin rather than by a few per cent is that it does not copy at all: the read-only lower layer is one set of blocks that every run reads from, and the upper layer stores only what a run changes. Cost therefore scales with what the run writes rather than with the size of the checkout, which is the property that makes per-run workspaces affordable at all. Before it, "give each run its own copy" was a proposal with a real price attached, and shared writable directories were how teams avoided paying it.

Having built it, attack it. The suite below runs inside the real sandbox image on every change to the image, the runtime configuration or the harness, on the same discipline as the egress canary in chapter 3, and each assertion is an escape that must fail.

import os, subprocess, pathlib, pytest

ALLOWED_MOUNTS = {"/", "/proc", "/sys/fs/cgroup", "/workspace", "/tmp", "/opt/cache"}

def test_no_unexpected_mounts():
    out = subprocess.run(["findmnt", "-rno", "TARGET"], capture_output=True, text=True)
    assert set(out.stdout.split()) <= ALLOWED_MOUNTS

def test_no_host_socket():
    assert not any(p.is_socket() for p in pathlib.Path("/var/run").glob("*.sock"))

def test_shared_cache_is_readonly():
    with pytest.raises(OSError):
        open("/opt/cache/poison.pth", "w").write("import os; os.system('id')")

def test_cannot_name_host_root():
    for probe in ("/proc/1/root/etc/shadow", "/proc/self/root/../../etc/shadow"):
        with pytest.raises(OSError):
            open(probe).read()

def test_tmp_is_noexec():
    pathlib.Path("/tmp/x.sh").write_text("#!/bin/sh\nid\n"); os.chmod("/tmp/x.sh", 0o755)
    assert subprocess.run(["/tmp/x.sh"], capture_output=True).returncode != 0

def test_image_carries_no_credentials():
    hits = subprocess.run(
        ["grep", "-rIl", "-E", r"AKIA[0-9A-Z]{16}|-----BEGIN .* PRIVATE KEY-----",
         "/etc", "/home", "/root", "/usr/local"], capture_output=True, text=True)
    assert hits.stdout == ""

Count ALLOWED_MOUNTS against the findmnt output at the top of the chapter and the two lists settle the question of what was actually decided. Nine targets were visible in the live sandbox; six are named in the assertion; three are gone. Two of the three get argued for in the pages above — the Docker socket, which is a host compromise with no exploit attached, and the credentials directory, which is a long-lived key. The third is the host's certificate-trust directory, and nothing anywhere makes a case for removing it. It is absent because the list is an allowlist, and an allowlist does not require an argument to drop something; it requires one to keep it. That inversion is the entire mechanical benefit of writing the set down. Under review-by-diff, every mount that exists stays until somebody objects to it, and objecting requires noticing; under an asserted set, every mount that is not written down fails the build, and keeping it requires a sentence.

One test belongs outside the sandbox, because it exercises the collector rather than the sandbox: create results/report.csv as a symlink to /etc/shadow and a directory entry named ../../../../etc/passwd, run the real collection path, and assert that the harness copies neither. That test is what catches an artefact-download endpoint that normalises paths with os.path.join and never calls realpath.

Confidentiality inside the scope, and the kernel underneath it#

Filesystem scoping controls reachability. It says nothing about the confidentiality of what you put inside the scope on purpose, so a run given a customer's dataset in /workspace can read every byte of it, which is the whole reason it was given the dataset. The controls that bound that are the credential design in the next chapter and the tool-level allowlists from topic 7.5, on tool execution safety.

It is also not a boundary against the kernel. A read-only root and a masked /proc do not change which syscalls are reachable, and the point from chapter 1 stands unchanged: a process that escapes through a kernel bug is outside every mount namespace you configured. Scoping and isolation technology solve adjacent problems and neither substitutes for the other: the first decides what a well-contained process is able to name, and the second decides whether the containment holds at all. A perfect mount table under a shared kernel and a perfect hypervisor with the Docker socket mounted are both real designs, and each one fails on the axis the other one covers. Nor is it the other kind of scope, the one a program is merely told about, which is the subject of topic 6.9, on roots and OS-level sandboxes.

So the property to write into the design document is not where the workspace is mounted but what resolves from inside it, stated as a list and asserted on every change to the image, the runtime configuration or the harness. findmnt from inside is the measurement; the suite is the evidence.

The last test in that suite points at the remaining gap. Scanning the assembled image for credentials passes today and says nothing about the design that put a credential there in the first place. A secret in an image layer is readable by every execution that image ever runs, by anyone who can pull it from the registry, and by anyone who can read a cached layer on any node that ever scheduled the workload — and it is readable precisely because the process can name it, which is the same property this chapter has been enforcing. The next chapter takes that credential out of the image and then keeps going, because getting it out of the image fixes the worst version and leaves the property that matters untouched.