Compliance is a difference only the instructions explain

The compliance panel had read 100% for six weeks. Then a reviewer working through the previous week's merges found 20240612094411_add_orders_deleted_at.sql, 20240613141002_index_users_email.sql, and 20240614083318_status_enum.sql, all three with an up section and nothing underneath it. The skill's central rule is one sentence — every generated migration must include a rollback section that reverses the up migration — and three migrations in five working days had gone out without one, while the evaluation that existed to catch exactly this reported no failures at all.

The evaluation worked the way most first evaluations work. After each run it took the session transcript, pasted it into a second model call along with the skill body, and asked a question:

Below is the text of a skill that was active during an agent session, followed
by the full transcript of that session. Did the agent follow the skill's
instructions? Answer COMPLIANT or VIOLATION, then explain in one sentence.

<skill>{skill_body}</skill>
<transcript>{transcript}</transcript>

That grader answered COMPLIANT on all three of the offending runs. Its explanations were specific and confident: for the June 13 run it wrote "The agent created the migration with both up and down sections as required by the skill."

Two ways a transcript grader gets this wrong#

The first reason is that the grader is reading the agent's account of the work rather than the work. The transcript for June 13 contains this assistant message near the end:

I've created db/migrations/20240613141002_index_users_email.sql with the index
creation in the up section and the corresponding DROP INDEX in the rollback
section, following the repository's migration conventions.

That sentence is the strongest evidence in the transcript, it is directly responsive to the question the grader was asked, and it is false. The agent wrote the file in two steps — a create with the up section, then an edit intended to append the down section — and the second edit failed against a stale file hash, returned an error the agent did not surface, and the summary was composed from intent rather than from the file. No amount of prompt tuning fixes this, because the grader is being asked a question about a description of an artefact and is answering it correctly. Sharpening the question to "is a rollback section present?" makes the grader more confident about the same paragraph.

The second reason is that the rule is in the grader's context. The prompt supplies the skill body verbatim, so the grader reads a clear, well-formed requirement immediately before reading a transcript in which an agent talks fluently about migrations and rollbacks. Model graders shown a rule and a plausible-sounding narrative about that rule lean towards compliance; on a hand-built set of 40 runs with 12 known violations, this grader caught 3 of them, and its false-compliance rate was concentrated exactly on the runs where the agent's summary was articulate. Transcript truncation compounds it, because a 90k-token session gets middled-out — the head and tail kept, the middle dropped — to fit the grader's window and the part that gets dropped is the tool call payloads — the request, the rule, and the closing summary all survive, and those three are the parts that describe intentions.

The team tried the obvious fix first, and the result is worth recording because it is so nearly encouraging. They replaced the open question with a specific one — does the final version of the migration file contain a rollback section with at least one statement? — supplied the rule verbatim, and added two few-shot examples of violations — worked cases pasted into the grader prompt so it has seen what a failure looks like before it is asked to spot one. Detection went from 3 of 12 to 6 of 12. That is a genuine improvement and it leaves the check answering a question about a description of the file, so the ceiling is set by how faithfully the transcript happens to describe what was written, which is a property of the agent's prose style rather than of the grader. Another round of prompt work would buy another point or two of the same kind.

Assert on the file and on the calls#

The redesign replaces the question with a set of post-conditions — assertions about the state the run left behind, evaluated after it finishes rather than asked of anyone — checked against the workspace and the run's tool call log. The mechanics of writing assertions against parsed fields rather than against text blobs were covered in 3.3; what is specific here is choosing what to assert about a skill.

import re, pathlib

DOWN = re.compile(r"^--\s*\+migrate\s+Down\s*$", re.M)
DDL  = re.compile(r"\b(DROP|ALTER|CREATE|TRUNCATE)\b", re.I)

def check_rollback(run) -> list[str]:
    failures = []
    written = [p for p in run.files_written if "db/migrations/" in p]
    if not written:
        return ["no migration file produced"]
    for path in written:
        sql = pathlib.Path(run.workspace, path).read_text()
        m = DOWN.search(sql)
        if not m:
            failures.append(f"{path}: no rollback section")
            continue
        body = sql[m.end():]
        if not DDL.search(body):
            failures.append(f"{path}: rollback section has no DDL statement")
    return failures

def check_verify_ran(run) -> list[str]:
    calls = [c for c in run.tool_calls
             if c.name == "bash" and "db-verify" in c.input.get("command", "")]
    if not calls:
        return ["make db-verify never ran"]
    if all(c.result.exit_code != 0 for c in calls):
        return ["make db-verify ran but never succeeded"]
    return []

Re-run over the same 40 sessions, these two functions found 11 of the 12 known violations. The one they missed had a rollback section containing -- rollback not applicable for this backfill, which is a real violation of the spirit of the rule and passes a presence check; the DDL clause was added afterwards specifically because of it, and it is the reason the second regex is there. DDL is data definition language, the CREATE, ALTER and DROP statements that change schema as opposed to the rows inside it, so requiring one in the rollback section is a cheap way of asking whether the section does anything.

The same functions run over history rather than over a live run, which is what turns them into the check the reviewer was doing by hand:

git log --since="30 days ago" --diff-filter=A --name-only --format= \
  -- 'db/migrations/*.sql' \
| sort -u \
| while read -r f; do
    git show "HEAD:$f" | grep -q '^-- +migrate Down' || echo "MISSING ROLLBACK: $f"
  done

Nineteen seconds against the repository, and it would have found the three files six weeks before the reviewer did.

Post-conditions belong in the same versioned file as the request, keyed per case, because different requests warrant different assertions and a global list either becomes the intersection of what every case can satisfy or fails half the suite for the wrong reason. A case asking for an index gets the rollback and verify assertions; a case asking to split a column across three tables additionally gets an assertion that the down migration restores all three. Written this way, the corpus stops being a list of prompts and becomes a set of executable expectations, which is what makes the comparisons in the rest of this topic possible at all.

{"id": "log-0455", "request": "can we get an index on users.email, queries are timing out", "expect_skill": true, "assert": ["rollback_present", "rollback_has_ddl", "verify_ran"]}
{"id": "tkt-2210", "request": "split address into street/city/postcode across the customer tables", "expect_skill": true, "assert": ["rollback_present", "rollback_has_ddl", "verify_ran", "roundtrip_schema_identical"]}

The number that means the skill did something#

A compliance rate on its own can still mislead, because a rule the repository already enforces will show 100% compliance whether or not the skill exists. The migrations repo has a file template with a -- +migrate Down line already in it, so any agent that copies the template gets a rollback header for free.

Evidence that the agent followed the instructions is a difference in its tool calls or its output that only the instructions explain. Getting that difference means running the same cases twice, once with the skill available and once without, and comparing:

Two of those four assertions are nearly worthless as evidence about the skill. The header and the filename come from the template and would look identical if the skill file were deleted, so tracking them tells you the template is intact. The other two are where the skill earns its place, and they are the assertions whose movement should trip an alarm. The public harnesses that formalise this comparison — agent-skills-eval is the clearest small one — exist because the with-and-without run is the only way to attribute anything, and running only the "with" arm produces numbers that drift with the repository rather than with the skill.

Which claims survive as code, and which do not#

Not every requirement in a skill body reduces to a regex, and pretending otherwise produces a suite that measures the easy third of the skill and calls it coverage.

The rubric named in the fifth row is a written scoring guide handed to the grader alongside the run — the specific things a good warning about an exclusive lock has to say, and the score to award when it says some of them — which is what stops a graded check from being one model's unexamined opinion, and it is worth the effort only for the claims that genuinely have no fixed form. The middle row is worth noticing for the opposite reason, because it looks like it needs judgement and does not. Applying both directions against a throwaway Postgres container and diffing the schema dumps takes 6 to 9 seconds per migration and answers the question exactly, which makes it the single highest-value check on the list. Claims migrate up this table over time as someone works out how to observe them; the graded rows are the residue, not the default.

What the redesign still cannot see#

Three categories escape a post-condition suite, and it is worth naming them rather than discovering them the way the first grader was discovered.

Vacuous compliance is the largest. A rollback section reading SELECT 1; satisfies the presence check and, with a sufficiently literal regex, the DDL check too. Every deterministic check has a form that passes it without meaning it, and the defence is not a longer regex; it is sampling. Ten per cent of passing runs go to a graded check or a human each week, and what that sample is looking for is specifically the vacuous pass, which means the grader's job is now narrow enough to be reliable.

Runs that produce no artefact at all are the second. If the agent discusses a schema change and never writes a file, the file-based check has nothing to assert on, and returning "no migration file produced" as a failure is wrong for the many sessions where discussion was the correct outcome. This is why the check keys off the run having written under db/migrations/ rather than off the skill having loaded.

The third is semantic correctness of the rollback under conditions the check does not reproduce — the down migration that works on an empty test database and would take a 40-minute exclusive lock on production. That belongs to the graded row of the table above, and honestly to code review; a skill evaluation that claims to cover it is overreaching.

There is also a failure the whole chapter has been assuming away. Every check here starts from a run in which the skill was active. When compliance drops from 93% to 71%, the suite says the rule was violated more often, and it does not say whether the agent read the rule and ignored it, read a version of the rule from a stale cache, or never had the rule in front of it at all. Those three have different fixes, and from the output they are indistinguishable.