Behaviour lives outside the code and changes without a commit

The billing assistant handled 8,412 sessions on Thursday, and 361 of them came back from the human support queue tagged answer was wrong. For the six weeks before that, the tag had run between 1.1% and 1.4% of sessions, so Thursday sat at 4.3%, roughly three and a half times the band the team had learned to treat as normal. The on-call engineer did the obvious thing at 18:00 and reverted the only change anyone could point at, a pull request merged on Monday afternoon that rewrote one paragraph of the system prompt covering refund eligibility. Friday came in at 3.8%.

A revert that moves the number from 4.3% to 3.8% is compatible with several stories. The prompt might have caused a fifth of the damage and something else the rest. The prompt might have caused none of it, while a second cause decayed on its own over the same twenty-four hours. Or Friday's difference might be ordinary day-to-day variance, since the tag rate has a standard error of about 0.2 percentage points at that volume and 3.8% is not a clean separation from 4.3%. The team spent the next week arguing about which story was true, and the argument was unresolvable because nothing in their system had recorded what was in force when each of those 361 answers was produced.

That argument is the ordinary condition of the work this topic covers. AI-specific technical leadership is the part of a technical lead's job that has no counterpart on a conventional service: setting the review standards, the ownership and the funded operational work for the artifacts that decide how a model-backed feature behaves. Those artifacts are the prompts, the evaluation suites that encode what a correct answer is, the skills and tool descriptions the model is handed, and the serving configuration — retrieval settings, decoding parameters, thresholds, and the model version in force. Everything else about running the team carries over unchanged from any other backend group, including estimation, on-call, design review and hiring. What is added is a small set of artifacts that sit beside the code, change the product whenever they move, and are routinely moved by people who do not work for you.

The practice exists because ordinary change control was built around a repository. On a payments service, the set of things that can alter behaviour is close to the set of things in your version control history, so review, CI and revert between them cover almost all of it. A model-backed feature breaks that assumption in three places at once, because the behaviour is written in prose and configuration rather than in compiled code, the model underneath is a dependency the vendor updates without a version bump anyone approved, and correctness is a human judgement about an output rather than an assertion that passes or fails. A change-management boundary drawn around the repository therefore covers a shrinking share of what determines the product.

You meet this in four ordinary places: a pull request that edits one sentence of a prompt or one integer in a YAML file and has to be told what evidence it owes, a CODEOWNERS entry deciding who may delete an evaluation case the system keeps failing, a quarterly plan where the hours for labelling data either carry someone's name or quietly disappear, and a Thursday like this one. The rest of this topic works through the first three. The incident comes first because it is where the shape of the problem is visible with none of the apparatus in place.

Three things changed that week and only one was in the repository#

Reconstructing the week afterwards took two engineers three days and mostly consisted of reading other teams' changelogs. Monday's prompt edit was in the repository, reviewed by one person, merged with a comment saying it read better. On Tuesday the provider rolled out a point release of the served model; the request still named provider/large-v4, the pricing page did not change, and the only external evidence was a line in a status feed nobody on the team subscribed to. On Wednesday a marketing campaign for the annual plan went live, and the share of sessions whose first message mentioned annual billing went from 6% to 22%, drawing questions into a corner of the documentation corpus that had four short pages covering it.

Each of those is a behaviour change to the running system. Exactly one of them went through code review, ran in CI, and produced an artifact somebody could revert. The other two arrived without a commit, without a ticket, and without anyone on the team being asked to approve them. Two of the week's three behaviour changes were invisible to every process the team ran, which is what the shrinking boundary looks like on a particular Thursday.

The third row is the one people underestimate, and it turns on a slice, which is a named subset of traffic — annual-plan sessions, questions in Portuguese, multi-document questions — whose quality you can measure separately from the whole. Nothing inside the system changed at all on Wednesday, and the quality metric still moved, because the metric is computed over whatever questions arrive and the questions stopped being drawn from the same distribution. A team that has convinced itself that quality regressions come from changes it made will look for a culprit inside its own repository and find one, which is how a prompt gets reverted for a corpus gap.

A dependency that ships without a version bump#

The model is worth treating separately, because engineers reason about it using habits built on package managers and those habits mislead. A pinned library version is an immutable artifact; the bytes you resolved in January are the bytes you resolve in July, and an upgrade is a line in a lockfile that somebody reviewed. A model identifier is an alias over an artifact the provider maintains, and several things move underneath it without any action on your side: point releases and safety updates to the weights, changes to the serving stack that alter tokenisation or sampling behaviour at the margins, capacity routing that puts your traffic on differently configured hardware, and default parameter changes on the API itself.

The pin you can actually get is narrower than a package pin and comes with a clock attached. Providers expose dated or fingerprinted builds so that you can hold one, and they publish deprecation windows, commonly six to twelve months, after which the pinned build stops being served and you are moving whether you have evidence or not. Your most consequential dependency has a forced upgrade on someone else's calendar, and the question you control is whether you find out by scheduling an acceptance run or by reading a Thursday support queue.

The provider's changelog is not a substitute for your own measurement, either. A release note saying improved instruction following is a statement about aggregate benchmark movement across everyone's workloads, and it carries no information about the four percent of your traffic that asks about annual-plan proration. Two of the three causes above were external, and neither announced itself in a way the team would have connected to its own quality number.

The measuring instrument has error too#

There is a further wrinkle in the Thursday data that took the team another day to notice. The answer was wrong tag is applied by human support agents when they pick up an escalation, using their own reading of what the assistant should have said, with no rubric. Two of the six agents on the queue had started three weeks earlier, and new agents escalate and tag more aggressively while still learning which answers are defensible, which puts an unknown amount of the Thursday movement in the measurement rather than in the system.

That is the third structural difference from conventional engineering work. An HTTP 500 is a fact about the system, and two engineers looking at the same trace agree it happened. A wrong answer is a judgement, so the signal you are steering by has its own reliability that has to be established rather than assumed, and it moves when the people making the judgements change. Chapter 4 is about what it costs to make that instrument trustworthy; the point for now is that a quality metric produced by unmeasured human judgement is a fourth thing that can move without a commit.

Quality is a distribution, so five spot checks are noise#

The second difference compounds the first. A change to a payments service either passes its tests or does not, and a single reproduction is proof of a defect. A change to a prompt shifts a distribution over outputs, and the effect on any individual output can be positive, negative, or invisible. A reviewer who runs the new prompt on five questions and reports that the answers look good has measured almost nothing, which is worth putting in numbers because the intuition here misleads badly.

Take the team's own base rate of 1.2% wrong answers, and suppose you want to catch a change that doubles it to 2.4%. Detecting that difference at 80% power and a 5% significance level takes roughly 1,900 sessions per arm under a two-proportion test, so a serious regression needs on the order of two thousand samples on each side before it is reliably visible. A reviewer's spot check of five outputs will report looks fine for a change that has doubled the failure rate more than nine times out of ten, and it will do so honestly, because five samples from a distribution where 97.6% of outputs are good will usually contain zero failures.

This is also why the wrong-answer tag itself, at 8,400 sessions a day, is a usable instrument for large moves and useless for small ones. The team could see a jump from 1.2% to 4.3% within hours. A change that took the rate from 1.2% to 1.5% would have been invisible for a week and then arguable for another, and most prompt and config changes are that size.

What has to be on the record for the question to be answerable#

The recording requirement follows directly. Every served request needs to carry, alongside the usual latency and token accounting, the identity of every artifact that determined its behaviour. Topic 4.10 established how prompts get versions and how a pinned revision makes rollback a configuration change; the point here is that the pinned revision has to appear on the request record itself, not only in the deployment manifest, because the deployment manifest tells you what was configured and the request record tells you what actually served.

{
  "request_id": "01JQ7F3M2ZK4T8",
  "ts": "2026-03-12T14:22:07.441Z",
  "surface": "billing-assistant",
  "prompt": { "id": "billing/system", "revision": "r-118", "sha256": "9f2c1a" },
  "model": {
    "id": "provider/large-v4",
    "build": "fp_0e91b7",
    "temperature": 0.2,
    "max_output_tokens": 800
  },
  "retrieval": {
    "k": 5,
    "reranker": "rr-2",
    "corpus_snapshot": "docs-2026-03-09",
    "doc_ids": ["kb-4412", "kb-0981", "kb-4402", "kb-1180", "kb-4415"]
  },
  "router": { "policy": "v3", "branch": "billing_faq" },
  "input_features": { "plan_tier": "annual", "locale": "en-GB", "turn": 1 },
  "outcome": { "tagged_wrong": true, "escalated": true }
}

The build field is the one teams leave out and the one that decides whether a week like this is diagnosable. Providers expose a build or fingerprint identifier precisely because the model id is a stable alias over a changing artifact, and the value is only available at the moment of the call. Nobody can reconstruct it afterwards, and the provider will not backfill it for you.

With those fields present, the week's question becomes a query rather than an argument.

SELECT
  prompt_revision,
  model_build,
  input_features ->> 'plan_tier' AS plan_tier,
  COUNT(*)                                        AS sessions,
  ROUND(100.0 * AVG((tagged_wrong)::int), 2)      AS pct_wrong
FROM served_requests
WHERE ts >= '2026-03-09' AND ts < '2026-03-14'
GROUP BY 1, 2, 3
HAVING COUNT(*) >= 300
ORDER BY pct_wrong DESC;

Run against the reconstructed data, that query produced a picture nobody had guessed from the incident channel.

The prompt revision barely moves the number; the annual-plan slice is eight times worse than the monthly one on the new model build and about two and a half times worse on the old one. The dominant effect is a corpus gap in annual-plan documentation that Wednesday's campaign exposed, with a smaller genuine interaction between the new build and that slice. Monday's prompt change was innocent and had been reverted, which also removed an improvement to refund wording that a different team had asked for six weeks earlier.

The fix is an inventory, not a logging ticket#

The tempting conclusion after an incident like this is that the team needs better logging, and the ticket gets written that way: add prompt version and model fingerprint to the request record, close, move on. That fixes the next incident's forensics and leaves the condition that produced this one untouched. Two of the three changes that week were behaviour changes that no process in the team recognised as changes at all. The provider's point release had no owner, no acceptance step, and no place where someone could have said hold it in the old build for a fortnight. The shift in input mix had no monitor, because nobody had written down that the distribution of incoming questions is an input to quality the same way the prompt is.

So the fix has three parts, and only the first is logging. Every serving artifact gets its identity onto every request record, which is what makes attribution possible after the fact. Every behaviour-affecting input gets a named owner with the authority to accept or refuse a change to it, including the model version, which chapter 3 works through. And every change to one of those inputs carries evidence before it merges rather than after complaints arrive, which is chapter 2. What the incident says about how the team works is that its change-management boundary is drawn around the repository, while most of what determines the product's behaviour sits outside that boundary and moves on other people's schedules.

None of this is an argument that AI systems are ungovernable or that ordinary engineering discipline does not apply. Staged rollout, pinned versions, reversible deploys and a real incident review all carry over unchanged and are worth more here than elsewhere. It is also not the same thing as observability in the operational sense. Every latency and error dashboard the team owned was green through the entire week, because the system was serving 200-millisecond responses with a 0.03% error rate the whole time and was simply wrong in 4% of them. Quality has no page, no threshold that fires on its own, and no meaning outside a sample large enough to measure.

Recording the versions makes the postmortem tractable. It does nothing about the change that has not shipped yet, sitting in a pull request that alters one sentence of a prompt, or one integer in a YAML file, waiting for someone to decide what evidence it owes before it goes anywhere near production.