A deferred read resolves against the wrong directory in a clean container

You are asked to certify that deploy-diagnostics works from a clean environment before it ships to the on-call surface. On your machine the request why did the eu-west rollout stall at 40% selects the skill, loads the body, reads both bundled files, runs the packaged rollout_state.py, and returns an answer naming the internal code DPL-4471 and explaining that the rollout paused because two replicas reported clock skew beyond the ledger tolerance. That is correct, and it is the kind of answer nobody could produce without the reference file.

Your teammate runs the same request in a fresh container built from the published image. The agent selects the skill, loads the body, and returns:

The rollout appears to have stalled at 40%, which typically indicates that the deployment controller paused after a batch of pods failed their readiness probes. Check pod events for CrashLoopBackOff or failing health checks, and confirm the new image is present in the regional registry.

No error. No warning. A competent, well-organised paragraph about Kubernetes rollouts, and the actual cause is nowhere in it. The trace shows a Read tool call for ./reference/error_codes.md that returned ENOENT: no such file or directory, followed by a second attempt at reference/error_codes.md, also failing, followed by the model deciding it had enough general knowledge to answer. It was right that it had enough to produce a paragraph. The paragraph is the failure.

Why it works on exactly one machine#

The path in the body is relative, and a relative path resolves against the process working directory of whatever is executing the read. That is the agent runtime's cwd, which is normally the directory the user or the harness started in, and it has nothing to do with where SKILL.md lives. On your machine you had been editing the skill and your shell was sitting in ~/skills/deploy-diagnostics, so ./reference/error_codes.md happened to resolve. In the container the agent starts in /workspace, and the skill is installed at /opt/agent/skills/deploy-diagnostics, so it does not.

# your machine, while authoring
$ pwd
/Users/you/skills/deploy-diagnostics
$ cat ./reference/error_codes.md | head -1
# Deployment error codes

# the container the on-call surface actually runs
$ pwd
/workspace
$ cat ./reference/error_codes.md
cat: ./reference/error_codes.md: No such file or directory
$ ls /opt/agent/skills/deploy-diagnostics/reference/
error_codes.md  rollout_phases.md

This is the same class of bug as a cron entry that runs a script by relative path, works every time from your shell, and fails at 03:00 when cron starts it from /. The difference is what happens after the failure. Cron writes a non-zero exit status and an error to a mail spool; the agent writes an answer. A tool that reports a missing file to a language model has handed it a recoverable condition, and recovering from missing information by producing the most plausible available text is the behaviour the model was built for.

The fix is that a body must never name a path that depends on where the process started. Runtimes differ in what they offer, and it is worth knowing which of these your platform supports before writing the reference:

<!-- fragile: resolves against the agent's cwd -->
Consult `./reference/error_codes.md` for the meaning of DPL-* codes.

<!-- portable: the runtime substitutes the installed skill directory -->
Consult `${SKILL_DIR}/reference/error_codes.md` for the meaning of DPL-* codes.

<!-- also portable where the runtime resolves skill-relative paths itself -->
Consult `skill:reference/error_codes.md` for the meaning of DPL-* codes.

An absolute path hard-coded to /opt/agent/skills/... works in the container and breaks for anyone who installs the skill elsewhere, which includes every developer running it locally, so it trades one environment-specific failure for a different one.

The check has to prove the read happened#

The instinct at this point is to write an end-to-end test: send the request, capture the answer, compare it to a known-good answer. That is the weakest instrument available for this particular failure, and it is weak in two independent ways.

Model output varies run to run, so a strict comparison fails constantly for reasons that have nothing to do with the skill. The team's response to a check that is red four mornings out of five is to loosen it, first to a fuzzy similarity score, then to a keyword subset, and the loosening is not a lapse in discipline — it is the only way to get a build green when the check conflates every source of variation into one boolean. By the time the threshold is low enough to pass on a normal week, it is low enough to pass the container's Kubernetes paragraph, which shares most of its vocabulary with the correct answer. A golden-output diff on model-generated text specifically fails the case where the agent produces something plausible from general knowledge, and that is the case this chapter is about.

What you want to assert is narrower and mostly deterministic: that the file resolved, that its content entered context, and that the answer depended on it. Those are three separate checks with different costs and different flake profiles.

def assert_reference_used(trace, path_suffix: str, canary: str) -> None:
    """Three assertions over one run's trace, from cheapest to strongest."""
    reads = [b for b in trace.tool_uses
             if b.name == "Read" and b.input["file_path"].endswith(path_suffix)]

    assert reads, f"no Read issued for {path_suffix}"                    # step 5's failure
    result = trace.result_for(reads[-1])
    assert not result.is_error, f"read failed: {result.content[:120]}"   # this chapter's
    assert len(result.content) > 200, "read returned an empty file"      # packaging
    assert canary in trace.final_text, f"{canary!r} absent from answer"  # actually used

The canary is the part that makes the last assertion sound. DPL-4471 is an internal code that does not appear in any public corpus, so a model that never read the file cannot emit it, and a model that did read it will emit it when explaining the cause. Choosing the canary well matters more than the assertion around it: a sentinel that the model could plausibly guess, or that appears in the request, proves nothing. Codes, internal service names, an invented threshold value like tolerance 40 ms — those are safe. English phrases are not.

The first row is the one that should run on every commit, because it needs no model at all. Extract every path-shaped token from each SKILL.md, resolve it against the packaged skill directory, and stat it:

#!/usr/bin/env bash
set -euo pipefail
fail=0
for skill in "$1"/*/; do
  # any backticked path containing a slash, minus the ${SKILL_DIR} prefix
  grep -oE '`[^`]*/[^`]*`' "$skill/SKILL.md" | tr -d '`' \
    | sed 's|^\${SKILL_DIR}/||; s|^skill:||; s|^\./||' \
    | while read -r p; do
        [[ -s "$skill/$p" ]] || { echo "MISSING $skill$p"; fail=1; }
      done
done
exit $fail

Running that against the registry the first time found eleven broken references across 402 skills, seven of them ./-prefixed paths that had worked on their authors' machines. It costs no tokens and cannot flake, and it would have caught deploy-diagnostics before anyone opened a container.

The script is a third tier with its own resolution rules#

deploy-diagnostics bundles rollout_state.py, and a packaged script fails in more ways than a packaged document. The path has the same cwd problem, so the body must name it the same portable way. On top of that the file needs the executable bit, which several packaging pipelines drop because they build the archive with a tar step that normalises modes; it needs an interpreter that exists in the runtime image, which is where a script with a #!/usr/bin/env python3.11 shebang meets a container carrying 3.9; and it needs its own dependencies, which is where a script importing httpx meets an image that has requests and nothing else.

Each of those produces a different exit status and a different line on stderr, and the agent treats all of them the same way it treated ENOENT — as information, from which it will construct a paragraph. The container run in this chapter's opening had a second failure underneath the missing reference that nobody noticed for a week: rollout_state.py ran, exited 1 with ModuleNotFoundError: No module named httpx, and the model summarised that as "the diagnostic tooling is unavailable in this environment" and carried on. The static check therefore has to stat the script, assert the mode bit, and resolve the shebang against the image:

$ find "$skill" -name '*.py' -o -name '*.sh' | while read -r s; do
    [[ -x "$s" ]] || echo "NOT EXECUTABLE $s"
    interp=$(head -1 "$s" | sed -n 's|^#!\(/[^ ]*\).*|\1|p')
    [[ -z "$interp" || -x "$interp" ]] || echo "NO INTERPRETER $interp for $s"
  done

A dry invocation belongs in the certification job as well — running the script with --help or a no-op argument and asserting exit 0 catches the import errors before a model ever sees them, and costs a few hundred milliseconds rather than a full run.

Keeping it useful in CI#

The static check belongs on every pull request. The trace assertions are model calls, so they are slower, cost money, and have a genuine nondeterminism floor, and treating them like unit tests is what produces the red-build fatigue that ends with the checks disabled.

Three things keep them tolerable. Pin the model version in the certification job, because a model upgrade changes fetch rates and a check that moves under you is a check nobody trusts; upgrade it deliberately and expect to re-baseline. Run each request five times and assert on the aggregate rather than the individual: require 5/5 on the deterministic assertions — the read resolved and returned content — and 4/5 on the canary appearing in the answer, since paraphrase is a real and acceptable variation. And separate the two failure modes into separate jobs, so a red build tells you which one broke; a job named skill-paths failing means someone wrote a fragile reference, and a job named skill-fetch-rate dropping from 5/5 to 3/5 means the body's instruction to read has weakened, which is chapter five's problem and has a different fix.

The container itself is the other source of flake, and it is worth building the certification image from the published skill package rather than from the repository working tree. A test that runs against ./skills in the checkout resolves paths that the shipped package may not contain at all — a reference/ directory excluded by a packaging glob, a script that is present but not executable. Certifying the artifact you ship rather than the directory you edit is what makes "works from a clean environment" mean anything.

What this check does not tell you#

It says nothing about whether the file's contents are correct. A reference that resolves, loads, and contains an error-code table that went stale eight months ago will pass every assertion here and produce confidently wrong answers, and that is a maintenance problem rather than a boundary problem.

Nor does passing certification mean the problem cannot recur in production, where the environments are more varied than the one you certified. The same skill installed by a user into a home directory, mounted into a sandbox with a different root, or vendored into a repository by a team that copied the directory without reference/, all produce the container's behaviour again. The cheapest production signal is a counter on failed tool reads, tagged with the skill and the path, emitted by the runtime rather than inferred from answers — a read that returns an error is an event the runtime already knows about and usually discards, and turning it into a metric converts a silent class of failure into a graph with a name on it. On the platform team's registry that counter found four skills whose references broke only under one particular install path, none of which had failed any check.

It also says nothing about whether the file should have been deferred in the first place. The canary assertion proves the read happened on this request; it does not tell you what fraction of real traffic issues it, or what the runs that skip it produce. A skill can pass certification cleanly and still be deferring material that 90% of its runs need, paying a round trip each time to save tokens it should not have been saving.

Answering that requires stepping back from single runs to a comparison across many of them: what deferred loading actually costs and buys in prompt tokens, in latency, and in whether the answers are any good — measured against the alternative of loading every body upfront, and measured on a population that includes the requests where nothing was selected at all.