Measure the hook's latency and its failure direction

The release review for rolling the plan-attachment hook out to all 140 repositories lasted eleven minutes, and ten of them were spent on a single row of a table nobody had expected to argue about. The proposal came with an experiment: 2,000 archived infrastructure sessions replayed against a staging Terraform backend, 1,000 with the hook registered and 1,000 without, same tasks, same model, same seeds where seeds existed.

The 63 is the row that stopped the release, and it took a while to be noticed because everyone read it as a success. Violations down from 139 to 63 is a 55% reduction, which is a good result for most changes and a fatal one here, since the hook's entire claim is that the number on covered paths is zero. A control that reduces violations by half is an instruction with a slower runtime.

Where the 63 came from#

Splitting the 63 by timestamp resolved it in about a minute. Fifty-one of them fell inside a single 22-minute window on the second day of the replay, during which the change-request service — the thing the hook queries to find out whether a plan is attached — returned connection refused because it was being redeployed. The other twelve were spread thinly and turned out to be the make deploy-infra wrapper from chapter 1, which is a coverage problem rather than a failure-mode problem.

The 51 came from four characters of exception handling:

try:
    cr = subprocess.run(["change-request", "current", "--json"],
                        capture_output=True, text=True, timeout=4)
    attachments = json.loads(cr.stdout or "{}").get("attachments", [])
except Exception:
    sys.exit(0)          # <- the whole design decision, unlabelled

Written that way, the hook fails open, meaning it allows the call whenever it cannot determine the answer, which makes the invariant hold exactly when the change-request service is up. That is not a bug in the sense of a mistake in logic; the author considered the case and chose availability, and the choice is defensible for some rules. What is not defensible is that the choice was invisible — no comment, no metric, no line in the design document — so the platform team's claim to the compliance auditor that unplanned applies were impossible was a claim about the availability of an unrelated internal service, and nobody in the room knew it.

Do the arithmetic before choosing. The change-request service ran at 99.5% monthly availability, which is 3.6 hours per month of unenforced operation. At the team's observed rate of 4.5 applies per hour during working hours, that is about 16 unchecked applies a month, of which roughly one in ten would have been a violation under the old instruction-only regime. So the fail-open hook takes expected violations from about 14 a month to about 1.6 a month, which is a genuine improvement and is not the guarantee anyone was describing. Every sentence written about this control has to be conditioned on a second system being up.

Three directions to fail in, and the cost of each#

The Unknown state is the one every hook has and most hooks do not name. Which edge you take out of it is a property of the invariant, not of the hook.

The plan hook went to a degraded check, which took most of a sprint. The change-request attachment step was altered to also drop a signed receipt file into .terraform/plan-receipt.json containing the plan's SHA-256 and an expiry, so the hook's normal path became a local file read with a signature verification, and the remote call became a fallback used only when the receipt is missing. That change moved p95 added latency from 1,130 ms to 12 ms as a side effect, because the tail had been the service's own p95 all along, and it made Unknown rare enough that fail-closed became affordable on the remaining path.

Choosing fail-closed without that work would have been the correct decision anyway for this particular rule, and it is worth saying why the team resisted it. A fail-closed hook converts every dependency outage into an agent outage, which is a support load, and the first time it happens somebody proposes an environment variable to skip the check. That variable is the same escape hatch chapter 6 watched a false-positive rate produce, arriving by a different road. Fail-closed is only sustainable when the dependency is reliable enough that the block is rare, which is an argument for making the check local rather than for softening it.

Latency is measured on traffic that was never going to violate anything#

The p50 of +17 ms sounds harmless and the p95 of +1,130 ms is the number that matters, because it is paid by ordinary work. Of the 1,398 applies in the treatment arm, 1,335 were compliant and gained nothing from the hook except the wait. Every hook is a tax on the calls that were fine, and the tax is levied on the boundary's whole matcher rather than on the rule's actual scope — the plan hook matched every Bash call, so ls, git status and terraform fmt all paid the subprocess spawn.

Two cheap structural fixes recover most of it. Narrowing the matcher so the hook is not invoked for calls it will immediately exit on cuts the spawn cost entirely for those calls. Ordering the predicate so the cheapest discriminator runs first — the regex over the command string, before any I/O — means the expensive path is only taken for the calls that could actually violate the rule. Both were already in the chapter 1 script, which is why its p50 was 17 ms rather than 40, and neither helps the p95, because the p95 belongs to the calls the rule is about.

The 1.3 percentage points of lost task completion is the same tax in a different currency, and it decomposes into two unequal parts. Nine of the thirteen extra failures were sessions where the agent hit the block, generated a plan, attached it, retried, and ran out of its turn budget three steps later, so the hook cost them a completion by costing them four turns. The other four were the false positives: applies that had a valid plan attached under a different change-request ID because the session had been resumed, which the hook could not see. Those four are the ones worth arguing about at a release review, since nine sessions that ran out of budget while doing the right thing are a budget problem, and four sessions blocked from doing a legitimate thing are the hook being wrong.

Replaying archived sessions has a limit worth stating, which is that the agent in the treatment arm reacts to blocks and therefore diverges from the recorded trajectory almost immediately, so the two arms stop being the same experiment after the first rejection. That is unavoidable and it biases in a known direction: it undercounts the ways the treatment arm's agent finds new routes over a long horizon, because the replay ends before most of them. The number to trust from a replay is the violation count on covered paths; the numbers to treat as indicative are completion rate and anything downstream of the agent's reaction.

The third quantity worth recording is what the hook does to normal requests when it fails at something other than its dependency. During the replay 0.7% of Bash calls produced this:

Traceback (most recent call last):
  File ".claude/hooks/require_plan.py", line 12, in <module>
    cmd = event["tool_input"]["command"]
KeyError: 'command'
PreToolUse:Bash hook exited with code 1

The event was a Bash call with a description and no command, which happens for a shape of the tool input the author had not seen. Exit code 1 is non-blocking, so the call proceeded, and the traceback went into the transcript where the model read it. In nine sessions the model treated the traceback as a failure of its own command and retried it; in two it decided the environment was broken and asked the user to check their setup. A hook that crashes is not only unenforced, it is an input to the agent's reasoning, and an ugly one.

Measuring the hook with the hook is not a measurement#

The column labelled "applies without a fresh plan" cannot come from the hook's own logs, because a hook that fails to fire logs nothing and a hook that fails open logs nothing useful. The replay counted violations with an independent observer: a PostToolUse hook that recorded every completed apply and asked the change-request service afterwards whether a plan had been attached at that moment. That observer is exactly the detector chapter 2 warned against shipping as a control, and as an instrument it is the right tool, because it sits after the fact and sees what happened rather than what was proposed.

Keeping the observer running after the release is what turns the experiment into a monitor. The metric that matters in production is not how often the hook blocked something, which is a measure of the agent's behaviour, but how often an apply completed with no attached plan, which is a measure of whether the invariant held. Those two numbers diverging is the signal that a path exists which the hook does not cover, and it will diverge, because the twelve make deploy-infra cases were still in the treatment arm and no amount of measuring one hook against one action will find the next wrapper. Finding out what a whole fleet of hooks actually enforces takes a different method.