A review describes the boundary; only a probe finds it detached

The write succeeded. At 06:14 on a Thursday, run run-a03c81 created /srv/agent/probe-canary/run-a03c81 — a file containing a UUID and a timestamp, on a tmpfs, a small filesystem held in memory, that exists for no other purpose — and the probe that was supposed to receive EROFS, the error a kernel returns for a write to a read-only filesystem, received a working file descriptor instead. The harness drained the queue and stopped scheduling within four seconds, which is what a failing probe is wired to do. A probe in this sense is a check built into every run that deliberately attempts something the sandbox is supposed to forbid, purely so that the refusal can be observed; the day the refusal does not arrive is the day the boundary is known to have come off.

The cause took an afternoon to find and none of it was a security change. Six weeks earlier the platform team had migrated the sandbox from a hand-rolled docker run invocation to a new runner that generated container specs from a template. The template was faithful about the image, the resource limits and the seccomp profile, and it dropped the :ro suffix on one bind mount because the field in the intermediate representation had been called readonly and nothing had populated it. Two weeks after that a base image bump changed the default user from agent to root, which nobody noticed because nothing in CI asserted on it. Both changes passed review. Neither review was a security review, because neither change was about security, and the security review that had approved the original configuration had happened five months before either of them was written.

What a sign-off can and cannot establish#

Security reviewed the container configuration and the review was correct. What it established is that a particular YAML document, read on a particular day, described a boundary with the properties the team wanted. The running system is a different object: a template, a runner, a base image, a network policy, a CI job that builds the image, a credential-minting service, and the interactions between all of them, each on its own change cadence and each maintained by people who are not thinking about the sandbox when they touch it. A review is a statement about a document. Every failure of this kind that gets written up afterwards has the same structure — a control that was right when it was inspected and quietly stopped being attached — and no volume of review prevents it, because the review happens before the changes that break it.

"No agent has ever done anything harmful" needs a more specific answer than a general appeal to caution, because as a factual claim it is probably true and it is being used to support a conclusion it cannot reach. The claim decomposes into two: no agent has attempted to cross the boundary, and no attempt has been detected. Over those six weeks the harness ran about 16,800 tasks, essentially all of them internally authored prompts asking for ordinary code changes, and not one of them had any reason to write outside its workspace. The boundary was open the entire time and the observation "nothing harmful happened" is exactly what you would expect to see, which means it distinguishes nothing. The second half is worse, because detection of an unattached control requires something in the system to try the forbidden thing, and until the probe existed nothing ever did. An outcome that would be identical whether the control is present or absent is not evidence about the control.

The absence of harm is also a statement about the population of tasks rather than about the sandbox. It was true in February when every prompt came from the platform team, and chapter 3's external intake changed the population without changing the sandbox — which is precisely the situation in which "it has always been fine" is least informative.

Two probes, and why each is harmless#

The demonstration has to prove a boundary exists without creating an incident while proving it. That constraint rules out anything using a real credential or a real destination, and it is satisfiable because the point of a probe is to observe a refusal, and a refusal is equally observable when the thing being refused is worthless.

The second row of "independent signal" is the design detail that makes probe 2 worth more than its cost. If a connection does get out, the record of it exists on a host in a different account that is not part of the system under test, so the probe still works on the day the sandbox's own log pipeline is the thing that broke. A check that depends entirely on the logs of the system it is checking has a shared failure mode with what it is checking.

PROBE_PATH = "/srv/agent/probe-canary/"
CANARY_URL = "https://canary.acme-lab.net/probe"
CANARY_TOKEN = "ghp_000000000000000000canaryNOSCOPE0000"   # zero scopes, own org

def run_probes(run_id: str) -> ProbeResult:
    results = {}

    try:
        with open(PROBE_PATH + run_id, "w") as fh:
            fh.write(f"{run_id} {time.time()}\n")
        os.unlink(PROBE_PATH + run_id)
        results["fs_confinement"] = "BREACHED"
    except (PermissionError, OSError) as exc:
        results["fs_confinement"] = "held" if exc.errno in (errno.EROFS, errno.EACCES) \
                                   else f"inconclusive:{exc.errno}"

    try:
        r = httpx.get(CANARY_URL, timeout=5.0,
                      headers={"Authorization": f"Bearer {CANARY_TOKEN}",
                               "X-Run-Id": run_id})
        results["egress_policy"] = "BREACHED" if r.status_code == 200 else "held"
    except httpx.ProxyError:
        results["egress_policy"] = "held"
    except httpx.HTTPError as exc:
        results["egress_policy"] = f"inconclusive:{type(exc).__name__}"

    return ProbeResult(run_id, results)   # anything not "held" blocks the run

Fail-closed is in the last line and in the word inconclusive. A probe that times out, throws an unexpected exception, or does not report at all is treated as a failure, because the states "the boundary held" and "the check did not run" are indistinguishable from the outside and the safe reading of an unknown is the bad one. A DNS failure resolving the canary host is inconclusive rather than held, since it means the probe never reached the layer it was testing. Teams that map unknown to pass end up with a probe that has been silently erroring since a library upgrade and a dashboard that has been green throughout.

{"ts":"2026-06-04T06:14:02Z","kind":"sandbox_probe","run_id":"run-a03c81",
 "fs_confinement":"BREACHED","egress_policy":"held",
 "action":"queue_drained","image":"agent-sbx@sha256:9c1e…",
 "runner":"spec-template-v4","policy_rev":"egress-2026-05-19"}

Recording the image digest, the runner version and the policy revision alongside the verdict is what turned an afternoon of investigation into a bisection, a search backwards through the sequence of builds for the first one that behaved differently: the first BREACHED line named an image digest, and the digest named a build, and the build named the pull request that dropped :ro.

Where the probes go, and where they stop#

They run at the start of every task, before the agent gets the prompt, and at 35 ms per run against a 14-minute median task the cost is not worth sampling to avoid. A nightly job in CI is the tempting alternative and it verifies the wrong environment, because CI builds its own container from its own spec and the thing you need to know about is the one the queue is using right now. The probe belongs in the run.

They will also never find a novel escape. Probe 1 checks one path, and a container that is confined against /srv/agent but not against /proc/sys passes it while being wide open; probe 2 checks one hostname, and a policy that leaks DNS or permits a neighbouring CDN passes it too. These are regression detectors for controls somebody already decided to build, in the same relationship to security work that a smoke alarm has to a fire safety programme. Adversarial testing is a different activity with different people, and the NVIDIA guide Practical Security Guidance for Sandboxing Agentic Workflows is a reasonable map of the control surface worth probing as the set grows.

The failure mode that destroys the value is easy to fall into and worth naming: someone adds canary.acme-lab.net to the egress allowlist to quiet the deny alerts, or the runner special-cases the probe path so the tests stay green. A probe that is exempted from the policy it tests is worse than no probe, because it produces a green signal that people rely on. The rule is that the probe uses the same path as the agent, with no exception anywhere in the stack, and that any change touching probe handling is reviewed as a change to the control itself.

The order to build this in#

The stronger version of the argument the VP is making is one worth conceding partly, because probes are cheapest when there is less to probe. An agent that holds no credentials, as in chapter 6, has a much smaller test surface — the only question left is whether mediation is in the path, and the broker's own denial log answers it continuously. The team that spends a week on probes for a design in which every run carries a production-adjacent token has chosen the harder problem; removing the token removes most of the probing.

For a team starting from a shared clone and an inherited shell, the sequence that gets the most per week of effort is: a worktree per run so parallel tasks stop corrupting each other, an explicitly constructed environment with minted development credentials, an egress proxy in report-only mode for two weeks and then enforcing, a reaper on a five-minute timer, and these two probes wired to drain the queue. The container-per-task path from chapter 2 arrives when external task text does, and the broker from chapter 6 arrives when the first privileged action does. A working example of most of this assembled in one place is vigilante, which wires per-task worktrees, credential scoping and an audit log together and is short enough to read in an afternoon.

The thing to run first is the probe, on the sandbox that already exists, before any of the rest is built. It costs an afternoon and it answers the only question that matters at the start, which is whether the boundary you already believe you have is attached today.