Eval gate overrides

An override is a recorded decision to merge past a gate that is doing its job correctly. It is not the same thing as a bypass, and the distinction is the subject here. Both let the change through; only one leaves behind an artefact that says who decided, why, over which cases, and until when.

The situation that produces a legitimate override is ordinary. Eval cases encode the behaviour a product wanted on the day they were written, and when a behaviour is retired on purpose the cases asserting it start failing correctly: the grader is right, the threshold is right, and the change is right too. Nothing about the gate is broken and its verdict is still wrong. How that gets handled decides whether the gate is still a gate in six months.

Why that happens by construction, rather than through anybody's neglect, is the part worth being able to rebuild. An eval case is an assertion about intended behaviour, frozen at the moment somebody wrote it down: given this input, an acceptable answer looks like that, because that is what the product was for on the day. Intended behaviour then changes, and it changes by decisions taken somewhere else entirely — in a product document, a legal review, a call with a large customer — none of which has any link back to the case file. No part of the eval pipeline connects a decision to the assertions that decision invalidates, and the case cannot notice on its own, because a case holds no representation of intent beyond the assertion it makes. So it goes on asserting an intent that has been revoked, and it goes on passing while the old behaviour is still shipping, which is why nothing looks wrong until somebody changes the behaviour to match the new intent. On that day the case fails, correctly, against a change that is also correct. A legitimate override is therefore ordinary rather than exceptional: it is what a working gate does the first time a deliberate product decision reaches code the case set has an opinion about.

Owner, reason, expiry — and scope#

Without an owner there is nobody to ask about it in three months. Without a reason the next person sees a bypass and cannot tell whether it was a considered judgement or a bad Friday. Without an expiry it never comes back, and the gate you thought you had is a gate with a permanent exemption you have forgotten about. Scope is the fourth field and it is nearly as important, because an override that suppresses the whole gate for one merge is much larger than one that suppresses three named cases.

# evals/overrides/2026-08-04-callback-flow-removal.yaml
owner: r.okafor                       # must be a current member of a CODEOWNERS team
reason: >
  Product retired the callback-request flow on 2026-07-31 (decision doc PROD-441).
  Cases callback_frustrated_01/02/03 assert the removed behaviour and now fail
  correctly. The PR's answers are right; the cases are stale.
pr: 1187
scope:
  case_ids: [callback_frustrated_01, callback_frustrated_02, callback_frustrated_03]
expires: 2026-08-25                   # 3 weeks; sweeper fails the nightly after this
tracking: EVAL-318                    # ticket to rewrite the three cases

The CI job reads any override files present, excludes the named cases from the blocking calculation, and reports both numbers so the raw result stays visible.

# evals/gate.py
def apply_overrides(results: dict, overrides: list[dict], today: date) -> dict:
    active = [o for o in overrides if o["expires"] >= today]
    excused = {cid for o in active for cid in o["scope"]["case_ids"]}
    scored = {c: p for c, p in results["per_case"].items() if c not in excused}
    return {
        "raw_score": results["score"],
        "gated_score": sum(scored.values()) / len(scored),
        "excused_cases": sorted(excused),
        "overrides": [{"file": o["_path"], "owner": o["owner"], "expires": str(o["expires"])}
                      for o in active],
    }

# and a scheduled job, separate from the PR path:
def sweep(overrides: list[dict], today: date) -> int:
    expired = [o for o in overrides if o["expires"] < today]
    for o in expired:
        print(f"EXPIRED override {o['_path']} owner={o['owner']} ticket={o['tracking']}")
    return 1 if expired else 0        # non-zero fails the nightly and pages the owner

The sweeper is what makes the expiry real, and the step underneath that sentence is worth taking slowly, because expires: 2026-08-25 looks like a deadline and is not one. A date written into a file is data. Data becomes a control only when something reads it and acts on what it reads, and until then it is a string that renders nicely in a diff and changes nothing at all. An expiry date that nothing enforces is a comment, and three weeks later the file is still there with the same date on it and the three cases are still excused.

apply_overrides does read the field, and reading it is not enough, because of when that function runs. It runs on the pull request path, and its only response to an expired file is to stop excusing the cases named in it — a change in what the next pull request sees, rather than a message to anybody. Where no pull request touches that area, there is no next pull request: the file sits in the repository, the date sits in the file, and nothing evaluates either. On a quiet branch that can be weeks, and the quiet corners of a codebase are exactly where a forgotten exemption survives longest. A scheduled job carries no such dependency, because it runs whether or not anybody pushed — which is the property an expiry needs, given that an expiry is an event on the calendar and a push is not.

The same reasoning is why sweep returns a non-zero exit code instead of printing its findings and returning zero. A job that logs a message is a warning addressed to nobody: there is no keyboard the log line arrives at, and the premise of the whole mechanism is that the person who granted the exception has stopped thinking about it. A failed nightly is an interruption somebody has to answer. Running it on a schedule rather than on the pull request path is the second decision in that line, and it settles whose interruption it is — an expired override interrupts whoever owns it, not whoever happens to push next. Had the date been enforced by the gate on pull requests instead, the person who met it would be whoever pushed that morning, someone with no context on the decision and with an obvious cheapest route to a green check, which is to edit the date. That is how a three-week exception becomes a permanent one without anybody deciding to grant one. Routing the failure to the owner named in the file keeps the cost of an expiry with the person who chose to take it on.

The mechanism you choose determines whether any of those fields can exist at all, which is why the informal options are worse than they look.

The last row is the only one where granting the exception is itself a reviewable change, which is the property that matters. The override directory sits under a path listed in CODEOWNERS, the file that maps repository paths to the teams whose approval a change there requires, so an override cannot land without the people who own the eval set reading the reason someone wrote. It also has a useful side effect: writing the reason field is enough friction that people who do not have a reason tend to go and fix the change instead.

The reason a file can carry four fields that none of the other mechanisms can is not that YAML is expressive; a pull request label could hold the same four strings. It is that granting the exception is itself a change to the repository, so it inherits, without anybody building it, everything the repository already does to changes. A change arrives as a diff, which is what puts the reason in front of somebody. A change has a reviewer, which is what makes the grant a decision two people took rather than one. A change enters the history, which is what lets the question "when did this start and who started it" be answered in six months. And a change touches a path, which is the only hook CODEOWNERS has to attach an approving team to. None of the informal mechanisms is a change to anything: an admin merge is an action taken once against the required check and recorded in an audit log nobody reads, a label is metadata hung on a pull request, and [skip-eval] is a string in a commit message that no reviewer is ever asked to approve. That is why none of them can be reviewed, scoped or expired. The four empty columns in that table are not four separate shortcomings — they are one missing property, showing up four times.

Scope is the column easiest to underrate, so it is worth converting into what it means on the day. An admin merge suppresses the required check as a whole, which is to say every case in the suite goes unchecked on exactly the change that was risky enough to be blocked. The override file suppresses three named case ids and leaves everything else blocking, so the change is still being examined by every case that was not the subject of the argument. The two options only differ when the author turns out to be wrong about something other than the thing they were arguing about — which is the situation a gate exists for, and the one an unscoped exception removes it from.

Eleven overrides in two months, every one of them correct#

Now the version of this that produces an argument. Over two months and 260 merged pull requests, one team's gate was overridden eleven times. A review of all eleven found that every one was justified — the change was right, the eval set was stale, the override was the correct call. Product's reading is that a gate which has never once been right about a blocked change is pure friction, and it should be removed.

Eleven is worth converting into a rate before anybody argues about it, because a bare count of exceptions carries no sense of scale and eleven of anything sounds like a lot. Two hundred and sixty merges divided by eleven overrides is a little under twenty-four, so the gate let a change through under protest on roughly one merge in twenty-four and held on the other twenty-three without an argument anybody wrote down. For a control that runs on every single change, that is a low exception rate rather than the signature of a broken mechanism, and it is the first thing to establish, because the meeting is being asked to remove the gate rather than to fix the eleven.

The sentence a reader will confidently write after that summary is the sentence product wrote down — every override was correct, so the gate never caught anything real. It is tempting for a good reason. A gate exists to block changes that should not ship, every blocked change the review inspected turned out to be one that should have shipped, and eleven out of eleven therefore looks like a hit rate of zero. The step that fails is the therefore. Eleven out of eleven is a statement about eleven pull requests, and those eleven were not drawn at random from the gate's verdicts; they are the subset somebody disputed and won. What the eleven are evidence of and what the gate is worth are two different questions, and the log answers only the first of them.

Start with what the log is a verdict on. Eleven correct overrides is eleven instances of the eval set being wrong, and the median age of the stale cases involved was seven months. The suite had no owner, no refresh cadence, and no relationship to the product decisions that kept invalidating it. Every one of the eleven followed the same shape as the callback flow: a deliberate product change landed, the cases asserting the old behaviour stayed, and the next person to touch that area met a gate defending a decision that had been reversed. That is a maintenance failure in the case set, and removing the gate does not fix it; it removes the only mechanism that was surfacing it.

The first change is to attach eval-set maintenance to the thing that invalidates it. A behaviour change that ships without touching the case file is the actual defect, so prompts/ and evals/ get CODEOWNERS entries pointing at the same team, and a cheap check warns on any pull request that modifies prompt or policy files without modifying cases. A quarterly review of every case older than six months, with a report on the age distribution, turns the seven-month median into a number somebody watches. None of that is exotic; it is the maintenance any test suite needs and rarely gets when it was born as a notebook.

The second problem with product's conclusion is evidential, and it is the part most worth being able to reconstruct, because the same mistake is available anywhere an exception log exists. Ask what has to be true of a pull request before it can appear in that log. It has to have reached the gate, which excludes every change nobody wrote. It has to have been blocked, which excludes everything the gate passed. And it has to have been merged anyway, which excludes everything that was blocked and then fixed. Three conditions, each of them removing one of the outcomes the gate produced, and what survives all three is by construction the set of occasions on which the gate was wrong. Eleven overrides is not a biased sample of the gate's verdicts. It is a complete census of a single verdict, and it contains no counterfactual.

The outcomes those conditions removed are not hypothetical, and three of them are ordinary. The changes that were never written because the author knew the gate would catch them do not appear. Neither do the changes fixed before the pull request was opened, because the developer ran the 95-second smoke tier locally first, nor the reviews that went differently because a three-axis table was sitting in the pull request comments. Each of those is the gate doing its job, and each of them leaves no override file behind, for the plain reason that an override file is the artefact of the gate being overruled rather than of the gate working.

Put in ordinary terms: a review conducted on the override log can only ever find overrides, so "we read all eleven and every one was justified" is not a finding about the gate at all. It is a restatement of what the log is. The harder version of the same point is that the log looks the same under two diagnoses that call for opposite decisions. A well-functioning gate sitting on top of a case set that has gone stale in a few corners produces exactly this log — eleven correct overrides and nothing else — and so does a gate that blocks nothing but noise. Separating those two requires evidence the log cannot contain, which is why the argument has to leave the log.

Part of that missing evidence is recoverable, and it is the number to bring to the meeting. Using the same CI history query that measures the re-run flip rate — the share of eval failures that pass when the identical commit is run a second time — separate the eval failures into three buckets: those followed by a re-run on the identical SHA, those followed by an override, and those followed by a code change that then passed.

What makes that split recover anything is that CI history is not filtered the way the override directory is. Every eval failure the gate ever produced left a record behind, whether or not anybody argued with it, so the population being divided up is the gate's verdicts rather than its defeats. What each bucket means then follows from what happened next, and the three are cleanly different. A failure followed by a re-run on the identical SHA — the commit hash, so the code and the prompts are byte for byte what they were — is a failure somebody did not believe, and because nothing about the change moved between the two runs, a pass on the second attempt is noise rather than a fix. A failure followed by an override is the gate being overruled, and that bucket is the eleven. A failure followed by a code change that then passed is the gate being obeyed: somebody read the red check, altered the change, and the altered change satisfied the same cases that had rejected the first version of it.

Over the same two months, that last bucket held 34 pull requests. Sampling ten of them and reading the diffs found six where the fix was substantive — a prompt change that had broken citation formatting, a retrieval parameter typo, a tool schema change that silently dropped an argument. Those 34 are the gate's actual yield, and none of them are in the override log, which is why an override review conducted on its own always concludes that the gate is worthless. Thirty-four against eleven is roughly threefold, over the same window and the same two hundred and sixty merges: the gate produced about three corrections for every exception it granted, and every one of those corrections sat in the bucket nobody had looked at.

The controls that would have to stand in its place#

Removal is a real option and it is not a taboo. Blocking pre-merge is one control among several, it is the most expensive one in developer time, and it is worth keeping only while it is the cheapest way to prevent the failures it prevents. The condition for retiring it is that other controls are demonstrably standing in its place, and "demonstrably" is the load-bearing word.

That word is carrying the whole condition, so it is worth saying what it rules out. Every row of the table below is a claim about what would happen during an incident that has not occurred yet: the nightly would see the drift, the canary would surface the drop, the rollback would finish inside the window. A claim of that shape is a plan, and a plan is indistinguishable from a control right up to the moment it is needed, which is the one moment the difference matters. Reading a plan cannot settle it either, because the failure modes all live in the parts nobody wrote down — the alert that routes to a rotation with nobody on it, the rollback that turns out to need a rebuild, the proxy metric that moves for six reasons other than quality. The only way to establish that a control is a control is to exercise it and watch what happens. That is why the third column exists at all, and why every entry in it names an exercise rather than a document.

The last row puts that condition into its own name: what the argument needs standing behind it is a rehearsed revert, not a documented one. Its game day is the rehearsal — a scheduled exercise in which the team performs the rollback against a real deployment while somebody holds a stopwatch — and the fifteen minutes in the middle column is a time somebody recorded during one rather than a figure somebody estimated before one, which is the entire difference between the two. A revert path nobody has walked is a plan and not yet a control. The same standard applies to every other row, which is why each one carries a way of showing the claim rather than asserting it.

The arithmetic that follows from those is what decides it. At 40,000 requests a day, a canary at 5% for a 30-minute detection window plus 15 minutes to revert exposes roughly 60 requests to the regression.

That sentence compresses two divisions and a multiplication, and each is worth doing rather than quoting, because everything the section concludes rests on the size of the answer. Start with forty thousand requests a day. A day is fourteen hundred and forty minutes, so forty thousand spread across one is forty thousand divided by fourteen hundred and forty, which comes to a bit under thirty requests a minute. Dividing a daily total by the minutes in a day is what "per minute" means, and the step is allowed for as long as the total really is spread across all of them. The canary takes five per cent of traffic, and five per cent is one twentieth, so a twentieth of those thirty a minute is a little under one and a half requests a minute reaching the new version. The exposure window is the detection window plus the revert: thirty minutes to notice plus fifteen minutes to roll back is three quarters of an hour, and nothing shorter is on offer, because the regression is live from the moment it is deployed until the moment the rollback finishes. A little under one and a half requests a minute, sustained for forty-five minutes, comes to a bit over sixty — which is the roughly sixty the estimate quotes.

The step to be honest about is the first one. Dividing a daily total by the minutes in a day assumes traffic arrives evenly through the day, and no real product's traffic does: an assistant serving one working population is close to idle at three in the morning and busy for a few hours after nine, so a flat per-minute rate is several times too low at the peak and several times too high overnight. That does not make the calculation useless, and it does change what kind of statement it is. Read it as an order of magnitude — tens of requests rather than thousands — and treat the time of day the deploy went out as part of the answer, since the same forty-five minutes costs far more at eleven in the morning than at midnight. The figure a reviewer should ask for is the peak-hour rate, because a blast-radius argument is only ever as strong as its worst case, and the worst case is a deploy that lands at the busiest hour there is.

If the product can absorb 60 degraded answers, and every row of that table has been demonstrated rather than asserted, then blocking pre-merge is buying a small reduction in an already small blast radius — the volume of traffic that meets a regression before it is pulled, sixty requests in this case — at the cost of 22 minutes on every developer's loop, and demoting it to report-only on pull requests while keeping it blocking on merge to main is a defensible position. If the canary has no proxy metric that correlates with quality, which is the common case, then nothing in that table stands and the pre-merge gate is the only control there is.

One part stays blocking regardless, and the reason is that both of the things that made the rest of this argument possible are missing from it. The deterministic assertions cost forty seconds and catch unambiguous breaks — unparseable responses, nonexistent citation IDs, PII in output. There are only two ways a gate earns the distrust that starts an argument like this one. Its verdict can be wrong about the change, which is what a stale eval case is, or its verdict can differ between two runs of the same commit, which is what noise is. Neither is available here. An unparseable response is unparseable whatever the product decided last week, so no product decision can turn a red assertion into a case that deserves excusing, and there is no override anybody could write that would be correct. And the assertion contributes no variance of its own: run it twice over the same output and it returns the same verdict both times, so a red result is not something a re-run can talk you out of. A failure there is a break, every time, and there is no version of the argument where those are worth removing.

An override, then, is a decision and it should look like one. Owner, reason, expiry, scope, in a reviewed file, with a scheduled job that will not let the date slide: that is the difference between an exception and a hole, and it costs about ten minutes at the moment somebody is impatient. What the accumulated log then tells you is how well the case set is being maintained, and reading it as a verdict on the gate is how a team talks itself into removing the only thing that was reporting the problem.

The order to build all of this in is roughly the order these chapters took. Bind the run to the commit so the number means something, measure the noise floor before choosing any threshold, check the re-run flip rate to find out whether the gate is actually functioning, split the suite so the check is cheap enough to survive, put cost and latency on the same run, and set up the override mechanism before you need it rather than during the incident where you do. Then review four numbers monthly: the flip rate, the override count with the age of the cases behind it, the tier costs, and the remaining headroom in each budget. Each of those moving is a different thing going wrong, and all four are cheap to compute from artefacts the pipeline is already producing.