A prompt change needs evidence, and so does retrieval_k

Two pull requests landed against the billing assistant on the same Tuesday morning. PR #482 changed one sentence of the system prompt, replacing "explain the refund window" with "state the refund window in days and then explain it". It collected eleven review comments over two days, three of them about whether state was the right verb, and merged on Thursday with two approvals. PR #486 changed a single line in config/retrieval.yaml, k: 5 to k: 8, with the description "improving recall, we were missing context on multi-doc questions". It was approved in four minutes by the first person who saw the notification and merged before lunch.

Nobody involved was being careless. The prompt PR looked like a behaviour change because it contained English sentences describing what the assistant should say, and the config PR looked like a tuning knob because it contained an integer. The review effort went where the diff looked like content. What the two changes did to the product ran the other way round.

What the integer actually did#

The team ran the retrieval change through the evaluation suite three weeks later, after a separate complaint about the assistant inventing policy details. The suite has 480 cases across five slices, each with a reference answer and a groundedness judgement over the retrieved documents.

The change did what its author intended on the slice they had in mind, and it broke the behaviour that mattered most to the support team, which is the assistant declining to answer when the corpus does not contain the answer. Three extra documents of marginal relevance give the model enough plausible-looking material to construct a confident answer out of, so the refusal rate collapsed. The overall accuracy number, the one a hurried reviewer would have looked at, moved by less than its own confidence interval and told you nothing.

Note that the prompt change and the config change were reviewed against completely different standards, and the config change was the one that altered cost by 41%, latency by half a second at p95, and the system's willingness to say I don't know by twenty points. Configuration is where behaviour changes hide, because a config file has no prose for a reviewer to argue with, the value is trivially reversible, and the word tuning carries an implication that the effect is a matter of degree.

One gate, defined by what a change can move#

The standard that fixes this does not distinguish prompts from config. It asks whether the change can alter the distribution of outputs, and if it can, the change carries evaluation results on the affected suite, before and after, attached to the pull request, with the movement accepted by the suite's owner. A prompt sentence qualifies. So do retrieval_k, the reranker version, temperature, max_output_tokens, a similarity threshold, the router policy, the corpus snapshot, the tool descriptions, the skill definitions the agent loads — named bundles of instructions, examples and tool permissions that the model pulls in for one kind of task — and the model build in force.

The mapping from paths to suites and owners belongs in the repository, so that neither the author nor the reviewer has to remember it.

# review-policy.yaml
classes:
  prompt:
    paths: ["prompts/**", "skills/**/SKILL.md"]
    suites: [billing_core, refusal, tone]
    evidence: [eval_run, slice_table]
    approver: prompt_owner
  serving_config:
    paths: ["config/retrieval.yaml", "config/decoding.yaml", "config/router/*.yaml"]
    suites: [billing_core, refusal, multi_doc]
    evidence: [eval_run, slice_table, cost_latency_delta]
    approver: serving_owner
  model_version:
    paths: ["config/models.lock"]
    suites: [billing_core, refusal, tone, multi_doc, safety]
    evidence: [eval_run, slice_table, cost_latency_delta, canary_plan]
    approver: serving_owner
  eval_data:
    paths: ["evals/cases/**", "evals/rubrics/**"]
    suites: []
    evidence: [case_provenance, rationale, agreement_check]
    approver: eval_owner

serving_config requires strictly more than prompt because a retrieval or decoding change moves cost and latency alongside quality, and those numbers are invisible in an accuracy table. model_version requires the most, which is the point chapter 1 left open: a provider's new build is a change to your product, and the only way to treat it as one is to pin the build in a lockfile so that adopting a new build is a pull request that a person opens deliberately and runs the suites against.

A CI job turns the policy into something that cannot be skipped by a fast approval.

#!/usr/bin/env python3
"""Fails a PR whose diff needs evaluation evidence it has not attached."""
import json, subprocess, sys, yaml
from fnmatch import fnmatch

policy = yaml.safe_load(open("review-policy.yaml"))["classes"]
base = sys.argv[1]
changed = subprocess.run(
    ["git", "diff", "--name-only", f"{base}...HEAD"],
    capture_output=True, text=True,
).stdout.split()

required = {
    name: spec
    for name, spec in policy.items()
    if any(fnmatch(f, p) for f in changed for p in spec["paths"])
}
if not required:
    sys.exit(0)

# Evidence is posted by the eval runner as evidence.json on the PR branch.
try:
    evidence = json.load(open("evidence.json"))
except FileNotFoundError:
    evidence = {}

problems = []
for name, spec in required.items():
    for suite in spec["suites"]:
        run = evidence.get("runs", {}).get(suite)
        if not run:
            problems.append(f"{name}: no run for suite '{suite}'")
        elif run["prompt_sha"] != evidence["head_sha"]:
            problems.append(f"{name}: run for '{suite}' is stale ({run['prompt_sha']})")
    for item in spec["evidence"]:
        if item not in evidence:
            problems.append(f"{name}: missing '{item}'")

if problems:
    print("Behaviour-affecting change without evidence:", file=sys.stderr)
    for p in problems:
        print("  -", p, file=sys.stderr)
    sys.exit(1)

The staleness check matters more than it looks. A run produced against an earlier commit on the branch is evidence about a system that no longer exists, and once a team is attaching eval runs at all, the most common way the gate rots is that the run stops corresponding to the diff.

Knowing which movements are real#

Attaching numbers to a pull request creates a new way to be wrong, which is treating every difference as a result. The suite has run-to-run variance of its own, and a team that has not measured its own noise floor will spend review cycles arguing about movements that are indistinguishable from rerunning the same code.

Measuring it takes one afternoon. Run the unchanged suite three times against the unchanged system and record the spread. On the billing suite at temperature 0.2, three consecutive runs of the same 480 cases scored 81.2%, 80.4% and 81.9%, so the observed range on the headline number is about 1.5 percentage points before anything has changed at all. The 60-case refusal slice was noisier in absolute terms, ranging over 4 points, because a slice of 60 cases moves by 1.7 points whenever a single case flips. Those two figures are what a reviewer needs in front of them to read the table, and they belong in the eval report rather than in someone's memory.

Two things shrink the floor without adding cases. Scoring both arms on the identical case list and comparing case-by-case rather than comparing two aggregates removes most of the sampling variance, since the same case either changed verdict or did not. Fixing decoding parameters for eval runs, typically temperature 0 for the system under test, removes the rest of the sampling noise while leaving the model's own nondeterminism. What remains after both is small enough that a nine-point move on the multi-document slice is unarguable and a 0.8-point move on the overall number is correctly read as nothing.

The report is also where the author says what they think the change does, in one sentence, before anyone else reads the numbers. A pull request that attaches a slice table and no interpretation pushes the whole cost of understanding the trade onto the reviewer, and in practice the reviewer scans the first row. The convention that works is a single line at the top of the PR body: this buys 9 points on multi-document questions and costs 20 points of correct refusal and 41% of the token bill, and I think that trade is wrong for us until the corpus covers annual plans. Authors who write that sentence catch their own bad trades roughly as often as reviewers do.

The model build needs a cadence, not just a lock#

Pinning the build in config/models.lock makes adoption a reviewable event, and it does not make the decision go away, because the pin expires. Providers serve a given build for a bounded period, so the lock buys you the right to choose when inside that window rather than the right to stay.

What that turns into operationally is a standing job rather than a reaction. The billing team runs the full suite against the current pin and against the newest available build on the first Monday of each month, at a cost of about $6 and twenty-two minutes of machine time, and the output is a diff table with the same shape as the retrieval table above. Most months it shows movement inside the noise floor and the job closes itself. The month it does not is the month the serving owner opens a pull request against the lock with a canary plan attached, on their own schedule, with the deprecation date still six months away. That is the difference between accepting a model change and receiving one, and it is the same gate applied to a change whose author happens to work at another company.

Why the gate sits before the merge rather than after it#

The competing approach is to ship the change behind a flag, route 5% of live traffic to it, and watch production metrics. That is a real control and larger teams run both, so the question is what each one buys.

An online canary measures the thing you actually care about on the traffic you actually get, and it catches interactions that no offline suite contains. It also needs volume and time. Chapter 1's arithmetic applies directly: at a 1.2% base rate, catching a doubling takes about 1,900 sessions per arm, so a 5% canary on 8,400 sessions a day accumulates 420 sessions a day in the treatment arm and needs four and a half days for that single comparison, before any slice breakdown. The retrieval change's worst effect was concentrated in a 60-case out-of-corpus slice that represents perhaps 3% of live traffic, which at canary volume is twelve sessions a day and would have taken months to surface as a signal.

The offline suite gives up realism and buys concentration. It is built so that the cases that matter are over-represented relative to production, which is what allows 480 cases to say something about a slice that live traffic delivers a dozen times a day. It runs in eleven minutes, costs about $3 per full pass at current pricing, and produces the same numbers for everyone who runs it. The gate sits before the merge because that is where the change is cheap to alter and where a named person is already looking at it, and the canary sits after, as a check on what the suite failed to represent.

What the evidence does not establish#

A passing suite is not a proof that the change is good. It is a measurement of what moved on the cases someone chose to include, and the honest limitation is that the cases were chosen before this change existed. The retrieval table above says nothing about behaviour on questions in Portuguese, on documents added to the corpus last week, or on the multi-turn sessions the suite does not contain. An owner who accepts a change on that evidence is accepting a specific, bounded piece of information, which is a different act from believing the change is safe.

The gate is also not a requirement that every number improve. Most useful changes trade something, and the retrieval change is a reasonable trade for a product where multi-document questions dominate and out-of-corpus questions are rare. What the gate refuses is the trade being made silently by someone who never saw both columns. The decision belongs to a person, and the point of the evidence is to put the trade in front of them.

Nor should the gate cover everything. Applying it to a logging change, a dependency bump with no behaviour surface, or a refactor that leaves the rendered prompt byte-identical produces eleven-minute waits for nothing and trains people to route around the check, which is how these regimes die. The review-policy.yaml file above is deliberately a small, explicit list of paths, and keeping it small is part of keeping it credible.

The remaining weakness is one the policy file quietly exposes. Every class names an approver, and those names are doing all the work: prompt_owner decides whether a three-point drop in groundedness is acceptable, serving_owner decides whether 41% more cost is worth nine points on multi-document questions, and eval_owner decides which cases count as the definition of correct in the first place. A gate whose approver is a team label rather than a person is a gate that anyone on the team can wave through in four minutes, which is where PR #486 started.