Activation up, compliance down: the edit that scored well

Twenty-two runs in the "never loaded" bucket was a clear enough mandate, so deploy-release got a new description. The old one named the procedure — release deployment workflow including preflight verification and tagging — and the new one named the situations people are actually in: shipping a build to an environment, cutting a release, promoting or rolling back a version, getting a fix out to production. Activation on the labelled set went from 40.4% to 85.1%. It was a good week.

Nine days later the compliance check from chapter three, running nightly, showed make preflight present in 68% of runs, down from 91%. Nobody had touched the body. The rule was in the same place, in the same words, in a version whose hash matched the one in the manifest, and it was being satisfied a quarter less often than before the description changed.

The two numbers move on different populations#

Stratifying compliance by whether the case had been in the corpus before the rewrite explains it in one line.

The requests in the second group are things like roll back to 4.1.8, restart the workers in staging, promote the canary to 25%, and get this config change out to prod. Every one of them is a deploy-shaped request and the new description matches them correctly by its own terms. The body is a different matter: it is written end to end for cutting a versioned release from main, and its first mandated step assumes there is a build to verify. Asked to roll back to an existing tag, an agent holding those instructions does something reasonable — it notices that preflight verification of a build that shipped last week is beside the point, and skips it. Compliance falls because the rule is being applied to work it was never written to govern, and the agent's judgement about that is arguably better than the rule's.

The description and the body are two halves of one artefact, and an edit to either changes what the other is being asked to do. Broadening a description without extending the body produces exactly this: a skill that fires on more work and governs less of it. The fix here was not to narrow the description back, because those requests genuinely wanted the skill; it was to add a branch to the body covering rollbacks and config-only changes, with its own verification step, which brought combined compliance to 86% at the same activation.

A baseline is three numbers, not one#

Both movements were visible only because all three numbers were recorded together against the same cases. Activation alone said the edit was a large success. Compliance alone, seen a week later without the activation context, looks like model drift. Task quality — whether the run produced a correct deployment, judged by the assertions in the case plus a small rubric for the parts that resist assertion — is the third, and it is the one that catches the opposite failure, where a skill fires correctly and is followed exactly and makes the output worse because its constraints are wrong for the case.

{
  "baseline_id": "deploy-release@2024-06-20",
  "skill_version": "git:4f1c9ab",
  "corpus_version": "corpus/deploy-release.jsonl@git:1a77e02",
  "model": "claude-sonnet-4-5-20250929",
  "runs_per_case": 5,
  "cases": {"positive": 94, "negative": 120},
  "metrics": {
    "activation":       {"value": 0.404, "ci95": [0.353, 0.457]},
    "false_activation": {"value": 0.033, "ci95": [0.019, 0.056]},
    "compliance":       {"value": 0.912, "ci95": [0.874, 0.941]},
    "task_quality":     {"value": 0.781, "ci95": [0.731, 0.826]}
  }
}

Each metric is stored with a ci95 alongside its value, which is the 95% confidence interval: the range the true rate plausibly occupies given how many cases and how many runs produced the point estimate. Activation of 0.404 with an interval running from 0.353 to 0.457 is a different claim from the same 0.404 measured to within half a point, and storing the interval is what stops a later comparison being read as though it were the latter. Four fields in that record do more work than the metrics themselves. The corpus version pins which cases produced the numbers, so a comparison against a different corpus is refused rather than silently made. The skill version ties the numbers to a hash the instrumentation from chapter four also records, which is what lets a production compliance drop be lined up against the edit that caused it. The model identifier prevents the most demoralising kind of investigation, where a metric moved because the provider shipped a new snapshot. Runs per case is there because a single run per case gives a number that moves on its own, which the next chapter deals with in full.

The comparison itself is short, and its shape matters more than its contents:

GATES = {
    "activation":       -0.05,   # may not fall more than 5 points
    "false_activation": +0.03,   # may not rise more than 3 points
    "compliance":       -0.05,
    "task_quality":     -0.05,
}

def compare(base: dict, new: dict) -> list[str]:
    breaches = []
    for metric, limit in GATES.items():
        delta = new["metrics"][metric]["value"] - base["metrics"][metric]["value"]
        if (limit < 0 and delta < limit) or (limit > 0 and delta > limit):
            breaches.append(f"{metric}: {delta:+.1%} (gate {limit:+.0%})")
    return breaches

Every metric is gated, including the ones the edit was not aiming at, because an edit is proposed to move one number and gets accepted on the strength of the others not moving. The gate that would have caught the deploy-release rewrite is the compliance gate, and it would have caught it before merge rather than nine days after — provided compliance was measured on the cases the new description newly matches, which means the corpus has to grow with the description.

A baseline is not a benchmark, and confusing the two produces some strange conversations. The absolute value of 78.1% task quality means nothing outside this corpus; it is not comparable to another team's skill, another repository's cases, or the same cases scored by a different rubric. Its only job is to be the thing today's run is subtracted from. That also means a baseline has an expiry: change the model, change the corpus, or change the rubric, and every stored number is void and has to be re-recorded before any comparison is legitimate. Teams get this wrong in a specific way, keeping a dashboard of skill scores over six months across three model upgrades and reading the line as skill quality. What that line shows is a mixture of skill edits, corpus growth and provider changes, in unknown proportions. The discipline is to re-record the baseline as its own commit whenever any of the three pinned fields changes, with the old numbers kept in git history rather than in the chart.

The skill nobody edited#

The same instrument answers a question that looks unrelated. Over one month, activation for db-migrations fell from 88% to 61%, and its git history is empty for that period — no change to the description, none to the body. db-performance was added to the repository three weeks in.

Two explanations fit. The new skill may be competing for the same requests, so a request that used to select db-migrations now selects its neighbour, which is a routing change. Or the production request mix may have moved — a quarter dominated by read-path work produces a stream of requests that genuinely should not select db-migrations, and if the falling number is being computed over live traffic rather than over fixed cases, the denominator changed underneath it.

The frozen set is the discriminator, and it works because it is the one thing in the system that did not change. Re-running the 240 cases from chapter one against today's repository took eleven minutes and returned 62.5%, which settles it: the cases are identical, the labels are identical, and the only difference is that another skill is now in the candidate set. Had it returned 87%, the answer would have been that nothing is wrong with the skill and the production number was measuring the market rather than the mechanism.

What to change, and what to refuse#

The tempting fix is a line in db-performance saying to defer to db-migrations when the request involves a schema change. It should be refused, for three reasons that are worth separating. The rule lives inside a body that has to be loaded before anyone can read it, so it can only apply on the runs that have already routed the wrong way — it is a correction issued after the decision it is trying to influence. It does not scale past the second collision, because the fourth database skill needs deference lines in three bodies and each of them needs updating whenever a fifth arrives. And it leaves the two descriptions overlapping, so the underlying ambiguity is still there for whoever adds the next skill to walk into.

The change that holds is at the boundary between the descriptions, edited as a pair and evaluated as a pair. db-migrations names schema change as its territory and read-path analysis as explicitly not its territory; db-performance does the reverse. Each skill's positive corpus goes into the other's negative corpus, which is the mechanism from chapter two applied across skills rather than within one, and both baselines are re-recorded in the same commit. The nineteen genuinely ambiguous cases from the original labelling pass are the ones to look at while writing those sentences, because they are precisely the requests where the boundary needs to be stated rather than assumed.

What should have been in place to catch this in week one is a scheduling rule rather than a new metric: adding a skill re-runs the baseline for every existing skill in the repository, and that run blocks the merge exactly as an edit to an existing skill would. Skills are not independent artefacts, and a repository that treats a new skill as an addition rather than as a change to a routing table finds out about collisions at month end.

The gates in the comparison function have been treated as given so far — five points here, three points there. Those numbers were made up, and a gate set too tight fires on nothing but noise while one set too loose never fires at all. Working out which movements mean anything, on a stochastic system evaluated over a few dozen cases, is the subject of the next chapter.