Behaviour changes need a release to correlate against

The data platform team maintains db-migrations, a skill that tells the agent how this company writes Postgres migrations: file naming, the review checklist to satisfy, which operations need CONCURRENTLY, and how to split a change so that the deploy and the migration can go out in either order. Nine teams have it. On 14 January someone on the data platform team rewrote a section of its body to make the generated files tidier, collapsing the pattern of one statement per file into a single file per logical change, wrapped in an explicit BEGIN/COMMIT.

On 28 January the ledger team's migration was rejected in review with a comment reading "you can't create an index concurrently inside a transaction." Building an index without locking out writes is one of a handful of Postgres operations that the database refuses to run inside a transaction block, so wrapping the file in BEGIN/COMMIT makes the statement illegal. They fixed that one by hand. Over the next nine days three more were rejected for the same reason, and someone asked in their channel when this had started happening. Nobody could answer. They had been writing migrations the same way for months, they had changed nothing on their side, and the only thing they could say with confidence was that the first rejection was on a Tuesday.

An investigation with nothing to line up against#

Here is what the ledger team had available, and what each source could actually tell them:

Every row is either too coarse or measures the wrong thing. The 23 commits include eight typo fixes, four additions of new examples, and the one edit that mattered, and reading them in the hope of spotting the culprit is a manual bisect over a repository whose history nobody on the ledger team knows. The mtime spread is the detail that made it worse than a simple time window: three of their engineers had reinstalled at different times, so the behaviour did not change for the team on a single date, it changed for each engineer whenever they last ran the install. Their nine-day window was really three overlapping windows, which is why the pattern looked intermittent and why the first two rejections were written off as reviewer preference.

It took eleven days and a conversation between two teams to find the 14 January commit. Nothing about that investigation was hard once the question was asked in the right place; what was missing was any artefact the ledger team could have named. The complaint they filed said "migrations started failing review around the end of January," and there was nothing on the maintainer's side to match that against, because the maintainer's side had no notion of a release either — it had a main branch that people installed from.

What a release supplies that a commit does not#

A commit records that the file changed. A release asserts that a particular state of the file was made available to consumers on a particular date, under an identifier that does not move. That combination is what makes correlation possible: the ledger team's report becomes "we are on 2.3.0, installed 21 January, and the failures started after that," which either matches a release or does not, and either way it narrows the search from 23 commits to one comparison.

The third component is the one teams forget: the consumer must be able to say which version they are running. A version number that exists only in the maintainer's repository is a label on a shelf nobody can read from where they are standing. Practically this means the installed package records its version somewhere the consumer can print, and — for a skill, where output is the only thing anyone sees — the generated artefacts carry it too. A migration file whose header reads -- generated with db-migrations 2.3.0 turns the eleven-day investigation into a two-minute one, because the rejected file and the working file from December carry different numbers and the diff between those two versions is a page long.

The changelog entry is what makes that diff legible without reading it. Most changelog entries fail at this, and the failure has a recognisable shape:

## 2.3.0 — 2026-01-14
- Improved migration file formatting for readability.
- Minor wording fixes.

That entry is not wrong, and it would not have helped the ledger team even if they had read it on the day, because it describes the maintainer's intent rather than the observable difference in output. An entry written for an operator trying to correlate a failure looks like this:

## 2.3.0 — 2026-01-14

### Changed
Generated migrations are now emitted as one file per logical change,
wrapped in an explicit transaction, instead of one statement per file.

Before (2.2.x):
    20260110_1_add_ledger_entries_index.sql
    CREATE INDEX CONCURRENTLY idx_ledger_entries_account ON ledger_entries (account_id);

After (2.3.0):
    20260114_add_ledger_entries_index.sql
    BEGIN;
    CREATE INDEX CONCURRENTLY idx_ledger_entries_account ON ledger_entries (account_id);
    COMMIT;

**Affects** any team whose review process rejects DDL inside a transaction,
and any operation that cannot run transactionally (CREATE INDEX CONCURRENTLY,
ALTER TYPE ... ADD VALUE, DROP INDEX CONCURRENTLY).

**Rollback**: pin to 2.2.4 until 2.4.0, which will suppress the transaction
wrapper when a non-transactional operation is present.

The difference is that the second entry contains a string the ledger team would have searched for. Someone whose migration was rejected for a transaction wrapper searches the changelog for "transaction" and lands on it; nothing in the first entry matches any word that appears in their problem. A changelog for a shared behavioural component is a search index for future incidents, and it should be written with the vocabulary of the failure rather than the vocabulary of the change.

Entries like that get written when the person who made the change writes them, at the time of the change, and they do not get written when someone assembles a release from a month of commits. Reconstruction after the fact produces the first entry rather than the second, because a reader of the diff can see that a BEGIN appeared and cannot see which team's review process cares. Putting a "consumer-visible effect" field in the pull request template, left blank only for changes the author claims have none, moves the writing to the one moment when the answer is known.

The staggered installs deserve one more note, since they are what turned a single dated change into three overlapping windows. Installing a shared skills repository by cloning main means every engineer is on a different revision determined by when they last pulled, and the correlation key you worked to create is then attached to something nobody is actually running. Pinning the consumer side to a tag — a plugin manifest referencing db-migrations@2.3.0, or an install step that checks out a tag rather than a branch — is what makes "which version are you on" a question with an answer. It also gives the rollback line in the changelog something to act on, since a team cannot pin to 2.2.4 if their install procedure has no place to put a version.

What the test asserts when the output is not deterministic#

The other half is catching this before it ships, which for a skill means running it and inspecting what comes out. The complication is that the output is generated text and will differ between runs — file names, column ordering, comment wording — so the assertion cannot be a fixture comparison. It has to be a property that holds regardless of surface variation, and the properties worth asserting are the ones a consumer's downstream process depends on.

# tests/test_migration_invariants.py
import re, pytest
from harness import run_skill          # runs the skill against a request, returns files

NON_TRANSACTIONAL = re.compile(
    r"\b(CREATE INDEX CONCURRENTLY|DROP INDEX CONCURRENTLY|ALTER TYPE \w+ ADD VALUE)\b",
    re.IGNORECASE,
)

CASES = [
    "add an index on ledger_entries.account_id for the balance query",
    "add a 'refunded' value to the payment_status enum",
    "add a nullable settled_at column to payments and backfill it",
]

@pytest.mark.parametrize("request_text", CASES)
@pytest.mark.parametrize("run", range(5))
def test_non_transactional_ddl_is_never_wrapped(request_text, run):
    for name, sql in run_skill("db-migrations", request_text).items():
        if NON_TRANSACTIONAL.search(sql):
            assert not re.search(r"^\s*BEGIN\s*;", sql, re.MULTILINE | re.IGNORECASE), (
                f"{name}: non-transactional DDL emitted inside a transaction"
            )

@pytest.mark.parametrize("request_text", CASES)
def test_backfill_is_separated_from_schema_change(request_text):
    files = run_skill("db-migrations", request_text)
    combined = "\n".join(files.values())
    if "UPDATE" in combined.upper() and "ADD COLUMN" in combined.upper():
        assert len(files) > 1, "schema change and backfill emitted in one file"

Three things about that suite are deliberate. The first test runs each case five times, because a generated-output check that passes once is a check on a single sample from a distribution, and the January change would have produced the transaction wrapper on most runs but not necessarily all of them. Five runs across three cases is fifteen samples per invariant, which took about 40 seconds and roughly 30,000 input tokens per full run on the data platform team's setup — cheap enough for every pull request, and the sort of number worth measuring before deciding it is not.

The second is that both assertions are conditional. They do not require the skill to produce an index or a backfill; they require that if it does, a specific relationship holds. Unconditional assertions on generated output produce flaky suites that teams learn to ignore, and an ignored suite is worse than no suite because it occupies the slot where a real check would go.

The third is that the cases come from the record of what consumers actually do, in the sense established in the previous chapter, rather than from the maintainer's memory of what the skill is for. ALTER TYPE ... ADD VALUE is in the list because a team hit it in production, not because anyone would have thought of it while writing tests.

Where this stops helping#

A regression test covers the behaviours somebody thought to name. The 14 January edit was made by an engineer who considered the change cosmetic, and if the suite had contained only the invariants the data platform team believed in at the time, the transaction wrapper would have sailed through, since nobody had written down that DDL — data-definition statements, the ones that alter the schema rather than the rows — must not be wrapped — it was a fact about the review process in one consuming team. Suites on shared components grow by incident, and the honest statement of their coverage is "the failures we have already had, plus the ones we guessed."

A changelog has a matching limit, which is that it is read after the fact. It does not prevent the incident; it collapses the investigation. The ledger team still had four rejected migrations, and the value of the entry would have been turning eleven days of cross-team archaeology into a search. That is a real saving and it should not be oversold as prevention.

Both of them also assume that the change passed through something. A test runs when a pull request opens, and a changelog entry is written when a release is cut, so both are attached to a process where a human is looking at a diff before it reaches nine teams. The 14 January edit went in without either, and the reason was not negligence about tests. It was that a one-paragraph edit to a markdown file did not look like the kind of change anybody reviews, which is a question about what review of shared agent configuration is supposed to be looking for in the first place.