Thresholds set while excited are not really thresholds

The Friday memo went out and the decision was to stay. On 2 April someone forwarded a thread showing the candidate solving a hard case beautifully, and the question reopened. On 19 April a customer asked which model the product used, and it reopened again. On 6 May a competitor announced they had migrated, and it reopened a third time. Each reopening consumed roughly six engineer-hours between the re-running, the re-arguing and the meeting, so the rejection cost about 24 hours of senior time spread over six weeks, plus the ambient cost of a team that could not tell whether the decision was made or merely delayed. The migration itself had been estimated at three days.

The reason the no would not stick is visible in the memo. It said the trade did not seem worth it, which is a summary of a judgement rather than a statement of a rule, and a judgement can be re-examined by anyone at any time in good faith. Nothing in the memo said what would have made the answer yes, so every new piece of evidence looked potentially decisive, and there was no way to establish that a given piece of evidence was not.

Gates before candidates#

The fix borrows its structure from preregistration in clinical research, where the analysis plan — the primary endpoint, the effect size that counts, the statistical test — is filed before the data exists. The point of filing early is that it removes the researcher's freedom to choose, after seeing the results, the framing under which the results look best. That freedom is not usually exercised dishonestly; it is exercised by a person who genuinely believes the new endpoint is more appropriate, and who would not have thought of it if the data had come out the other way. Model evaluation has exactly the same degrees of freedom, and exactly the same difficulty noticing when they are being used, which is why the gates have to be written on an ordinary Tuesday when no release is pending and nobody has anything invested in the outcome. Most of the numbers in them are already committed to somewhere else in the business and are being restated in a form a candidate model can be measured against: a service level objective, or SLO, is the latency or availability figure the product is formally held to, and the cost ceiling comes from whatever margin the finance model assumes.

The distinction between a veto and a trade is what stops the table from becoming a wish list. A veto is a condition under which no amount of improvement elsewhere buys adoption, and it exists because some costs are not fungible with quality — a parser with no fallback path does not care that summaries got better. A trade is an axis you are willing to exchange against another at a rate stated in advance, and stating the rate in advance is what converts the Friday argument about whether 2.8 points is worth $4,800 a month into a lookup.

Where the table lives affects whether it holds. A gates file checked into the same repository as the suite, changed by pull request, with the rationale column mandatory and a required reviewer who is not the person proposing the change, behaves like configuration. The same table in a wiki page behaves like a suggestion, because a wiki edit leaves no diff anyone reads and no moment at which someone else has to agree. The diff is the mechanism: when the p95 ceiling moves from 3.00 to 5.50 seconds three days after a release that missed it at 5.10, the pull request makes the sequence visible to everyone, and visibility is usually enough to stop it without anybody having to accuse anybody of anything.

The exchange rate is the entry most teams have never written and can usually produce in half an hour. It comes from whatever model already justifies the feature's existence: if a point of weighted quality corresponds to roughly 900 fewer escalated tickets a year at £14 of handling time each, the annual value of five points is around £63,000, and a monthly premium of $6,000 sits inside that with room for error. Being approximately right in advance beats being precisely arbitrary in the moment.

The noise band is a gate, not a footnote#

The reason the quality gate is +4.0 points rather than +0.1 is that a suite of 90 cases cannot resolve a difference of 2.8 points. Treating a pass rate as a proportion over 90 independent cases, the standard error at a pass rate near 0.87 is about 3.5 percentage points — the standard error being the spread you expect between two measurements of the same underlying thing purely because you measured 90 cases rather than all possible ones — so two systems whose true quality is identical will routinely produce observed deltas of several points in either direction.

import math, random

n, p = 90, 0.87
se = math.sqrt(p * (1 - p) / n)
print(f"se = {se:.3f}")                       # se = 0.035

# empirical: two identical systems, 10_000 paired runs
def run(): return sum(random.random() < p for _ in range(n)) / n
deltas = sorted(abs(run() - run()) for _ in range(10_000))
print(f"p50 |delta| = {deltas[5000]:.3f}")    # 0.033
print(f"p95 |delta| = {deltas[9500]:.3f}")    # 0.096

Nine and a half percent of the time, two systems that are genuinely identical will differ by ten points or more on a 90-case suite. The observed +2.8 from the previous chapter is therefore consistent with no difference at all, and the correct description of it in a memo is that quality was indistinguishable, not that quality improved slightly. Averaging three runs per case tightens this a little by removing sampling variance within a case, and it does nothing about the variance that comes from having only 90 cases, which is the dominant term.

Two consequences follow. The first is that the quality gate has to be set above the noise band or it is not a gate, since a threshold of +1.0 point will be crossed by chance every second release. The second is that if you want to detect a genuine 2-point improvement, you need a bigger suite — roughly 1,200 cases to get the standard error down near 1 point — and the honest position is usually that you do not care about 2 points enough to pay for 1,200 cases. Deciding that in advance is what makes it a design choice rather than a discovery you make while trying to justify a conclusion.

A no that fits in one sentence#

Gates in a table are only half the mechanism. The other half is that the verdict is produced mechanically and stated as a number that was missed, because a rejection that names a number can be argued with only by attacking the number, and attacking the number means either re-running the measurement or proposing a change to the gate — both of which are bounded activities with an owner, unlike a general re-litigation of whether the new model feels better.

def verdict(c: Candidate, g: Gates) -> str:
    if c.incident_cases_passed < g.incident_cases_required:
        return (f"No: {c.incident_cases_passed}/{g.incident_cases_required} "
                f"incident cases pass (veto, all must pass). "
                f"Failing: {', '.join(c.incident_failures)}.")
    if c.ttft_p95_s > g.ttft_p95_s:
        return (f"No: TTFT p95 {c.ttft_p95_s:.2f}s exceeds the "
                f"{g.ttft_p95_s:.2f}s ceiling set {g.set_on}.")
    if c.cost_per_task > g.cost_per_task:
        return f"No: ${c.cost_per_task:.4f}/task exceeds ${g.cost_per_task:.4f}."
    if c.quality_delta_pts < g.quality_delta_pts:
        return (f"No: quality delta +{c.quality_delta_pts:.1f} pts is below the "
                f"+{g.quality_delta_pts:.1f} pt gate and inside the "
                f"±{c.noise_band_pts:.1f} pt noise band.")
    return f"Yes: clears all gates. Migration ticket {g.ticket_template}."

# 2026-07-24, candidate release:
# "No: 12/14 incident cases pass (veto, all must pass).
#  Failing: summarise-0047, extract-0112."

That sentence is what goes in the Slack reply, in the memo, and in the log, and it is the same sentence every time somebody reopens the question. When the thread arrives on 2 April showing the candidate solving a hard case, the response is that two incident cases still fail and the p95 is 5.1 seconds against a 3.0 second ceiling, and the conversation moves to whether either of those has changed since. Six weeks later, when the provider ships a latency improvement, the question genuinely has changed and the slice gets re-run, which takes eleven minutes rather than six hours because the gates specify exactly what would need to move.

Precommitment is not a lock#

Gates get reviewed on a schedule, and the review is deliberately scheduled away from any release — the same quarter's planning session, alongside the SLOs. A gate that has never been revisited is not evidence of discipline, because the 3-second TTFT ceiling was derived from a chat interface that may since have become an asynchronous batch job where three seconds is irrelevant and cost dominates. Reviewing gates on a cadence keeps them attached to the product; changing them in the presence of a specific candidate does not, and the way to hold that line is to require that any gate change be applied to the previous three candidates as well, which makes the cost of a self-serving adjustment immediate and visible.

The requirement to apply a gate change retroactively is worth stating as a rule because it is the cheapest available defence against motivated reasoning. When the TTFT ceiling was proposed to move from 3.00 to 4.00 seconds during the quarterly review, applying the new ceiling to the three previously rejected candidates showed that two of them would have been adopted, one of which had since been superseded and one of which had a serious structured-output problem. That reframed the proposal from is four seconds acceptable into were we wrong to reject those two, which is a much more concrete question and one the team could answer from its own support data. The ceiling moved to 3.50 with a note explaining the reasoning, which is a smaller change than proposed and one nobody has reopened.

There is also an override, and its existence is what keeps the system honest rather than what undermines it. A named person can adopt a model that fails a gate, provided the reason is written down, the failing number is stated, and the decision goes in the log with a review date. The one legitimate use of it in eighteen months was a forced migration: the incumbent model was given a six-month end-of-life notice, the only viable successor missed the cost gate by 9%, and there was no version of the world in which staying was an option. The override recorded exactly that — the gate missed, by how much, why the alternative was worse, and a review date at which the cost would be re-examined against whatever else had shipped by then. Systems without an override path get gamed, because a rule that cannot be broken openly will be broken quietly by someone who redefines an axis, and a quiet redefinition leaves no review date behind it.

What none of this addresses is the possibility that the gates are measuring the wrong thing. The mechanism above rejects a candidate that fails a number, with complete indifference to whether that number describes anything users care about, and it will keep doing so cleanly and defensibly while the suite quietly drifts away from the product. That failure mode announces itself in a specific way: engineers who have used the candidate for a week start saying it is obviously better, and the table says it is worse.