Re-runs and gate precision

A developer opens a pull request and the eval check comes back red. Two responses are available and they are far apart in cost: read the diff, re-read the failing cases and work out what the change broke, which is an afternoon; or press the re-run button, which is free. Which of those is the right move depends entirely on one quantity, the fraction of red checks that mean something, and nothing else about the failure is visible from outside. Without a number for that fraction the developer is choosing blind, and a developer choosing blind takes the cheap option, because the cheap option is right often enough to feel justified.

That fraction has a name. Precision, in the sense a classifier uses the word, is the share of positive verdicts that correspond to a real problem. Applied to an eval gate it is the share of red checks that mean a regression rather than a bad draw, and it is the number that decides whether a developer can act on a failure at all. A gate whose failures are half noise gives a developer a signal that is right half the time, and the cheapest way to find out which half is to press re-run, which costs them nothing personally.

That is a rational response to the signal they are given, and it is also the same response a developer makes when the failure is genuine, because they cannot tell the two apart from the outside. This chapter is about what that does to the number the gate reports, how to measure the precision of your own failure signal from history rather than from a model, and which of the available fixes actually moves it.

Re-running to green replaces measurement with selection#

A green check licenses one sentence: the evaluated commit scored above the threshold on this dataset, with this grader, at this moment. Re-running until green breaks that sentence, because the reported score is no longer a measurement of the commit. It is the maximum over however many attempts the developer was willing to make, and the maximum of k draws is a biased estimate of the thing being drawn from.

That word biased is technical, and the mechanism under it carries the rest of the chapter. An estimate is unbiased when it is as likely to land above the quantity it is estimating as below it — the errors point both ways, and over enough repetitions they cancel. A maximum cannot behave that way. It is by construction the largest of the draws it was chosen from, so it can never come in below any of them, while each individual draw was already as likely to be high as low. An estimator with only one direction available to it is biased in that direction, and the direction available to a maximum is upward.

How far upward depends on how many draws it had to choose from, because every additional attempt is a fresh chance to beat the best score so far and no chance at all to lower it. A single draw lands above the true mean about half the time. Two draws land above it unless both come in low, which happens about a quarter of the time, so a best-of-two already sits above the mean three times in four. Push that to four attempts and the reported figure is the best of four independent tries, which lands above the centre unless all four come in low — an event that gets rarer as attempts accumulate. How far above is the part no argument here derives: for a distribution with the ordinary bell shape, the best of four sits on average about one standard deviation above the centre, and that is a property of the bell shape rather than something the counting above produces.

One standard deviation is an order of magnitude and not a constant, because the exact multiple depends on the shape of the distribution being drawn from: a long upper tail gives the maximum more room to run, and a tightly bounded distribution gives it less. The way to get the multiple for a real suite is to run an unchanged commit several times over and read the best-of-four off the results.

With a suite whose unchanged-commit distribution has a mean of 0.856 and a standard deviation of 0.018, the expected maximum of four draws sits about one standard deviation above the mean, at 0.875. Reporting a best-of-four rather than a single draw therefore adds about a standard deviation to every score, and adding that to the score has the same effect on what gets through as subtracting it from the threshold. A team with a re-run habit is effectively gating at a threshold two points looser than the one written in the config, and no one can read that fact anywhere.

The consequence for genuine regressions follows directly. Consider a change that really does hurt, dropping the true mean from 0.856 to 0.830 against a gate at 0.85. On any single run it scores under the gate with probability 87%, so it fails the first time, as designed. The developer re-runs.

The 87% is two steps from figures already on the page, and both are short enough to walk. The regressed commit produces scores centred on 0.830 while the gate sits at 0.85, so the gate is twenty thousandths above the centre of this commit's own distribution, and one standard deviation on this suite is eighteen thousandths — which puts the gate a little more than one standard deviation above the commit's mean. The second step asks how much of the distribution lies above a point that far up, and treating the run-to-run scatter as normal, the ordinary bell shape, the share sitting more than one standard deviation above the centre is about one in eight. So roughly one run in eight clears the gate and the other seven fall under it. Normality is an assumption rather than a fact about eval scores, and the way to check it is a histogram of repeated runs on an unchanged commit.

The table is one multiplication repeated. If a single attempt clears the gate thirteen times in a hundred, an attempt fails the other eighty-seven times in a hundred. Two attempts both fail only when the first fails and the second fails as well, and the chance that two independent things both happen is the product of their separate chances — so eighty-seven hundredths multiplied by itself, which comes to about three-quarters. The change is stopped by two attempts about three-quarters of the time and gets through on at least one of them the remaining quarter of the time, which is the second row. A third attempt multiplies the failing fraction by itself once more: three-quarters of eighty-seven hundredths is a little under two-thirds, leaving a bit over a third to pass, which is the third row. A fourth repeats the step, and the passing share climbs past four in ten.

What licenses that repeated multiplication is the assumption that the attempts are independent draws from the same distribution, and it holds here rather than being merely convenient. The commit has not changed between attempts, the case list has not changed, and the grader has not changed. Nothing carries from one attempt to the next except the parts of the system that were random to begin with — sampling in the model under test, sampling in the judge — so each attempt starts where the last one started and knows nothing about how it went. Same distribution, no memory: that is what independent and identically distributed means, and it is why multiplying is permitted.

The place the assumption breaks is worth knowing, because a CI system can break it by accident. If a retry reuses cached results for the cases that already passed and re-executes only the ones that went red — which is what an ordinary re-run failed jobs button does — then the second attempt is not another draw from the same distribution. It is the first attempt's good half held fixed while only its bad half is re-rolled, which is a strictly more favourable experiment than the one the multiplication describes, so the real pass probabilities sit above the table's. That is why the scenario further down disables that button for this workflow rather than treating it as a detail of pipeline configuration.

Three attempts is not an unusual norm on a check people already consider flaky, so roughly a third of genuine 2.6-point regressions merge with a green tick and a passing score in the artefact. The gate is not catching fewer regressions than it used to; it is producing an affirmative statement of quality about changes it had identified as problems.

This is the specific way the situation differs from the flaky integration test it gets filed next to, and the difference is not one of degree. Take the test first. When it fails because a sandbox API returned a 503, there is an underlying fact the test was built to report — does this code work — and that fact is deterministic: on this commit, with this input, the code either works or it does not, and running the test again cannot change which. The failure did not come from that fact. It came from the sandbox, which is orthogonal to the question being asked. Re-running resamples only the orthogonal thing, so if the sandbox is healthy the second time, what is left is the deterministic fact with the interference removed. The retry did exactly what it advertises.

An eval score has no deterministic fact underneath it. The quantity the suite reports is itself a distribution: a model that samples its own output, graded by a judge that samples its own, over a fixed set of cases, produces a score that scatters from run to run while nothing in the repository moves. Re-running does not resample something orthogonal to the thing being measured, because it resamples the thing being measured. The second attempt is therefore not a corrected version of the first — both are equally valid draws — and keeping the one that passed is not error correction but selection, which moves the number that gets reported upward by the amount the paragraphs above worked out.

In plain words: a retry is legitimate when the thing being re-rolled is the interference, and illegitimate when the thing being re-rolled is the signal. What sits between the two cases is not an analogy. It is one gesture — press the button, take the second answer — applied to two objects whose insides are shaped differently, which is precisely why it travels so smoothly out of ordinary CI and does so much more damage on arrival. A developer who has re-run a thousand flaky tests correctly has no experience that would warn them the thousand-and-first press is doing something else, because from the outside the two are identical: a red check, a button, a green check.

The precision of the failure signal#

The instinct is to measure quality, and the useful measurement is about the gate itself. Two numbers matter and both come out of data your CI system already stores.

The first is the noise floor — the spread of the score on an unchanged commit — which you now have a second reason to want: without it you cannot say what fraction of failures are expected. The second is the empirical re-run flip rate, the fraction of failed eval checks that passed on a later attempt against an identical commit. That is the precision of your failure signal read from history rather than modelled. Strictly it is precision's complement — the share of red checks that turned out not to mean anything — and precision is whatever share of the red checks is left over. It is the number to put in front of whoever thinks the gate is fine.

The link between a flip and precision is worth making explicit, because the query below measures one thing and the argument needs the other. A check that failed and then passed on an unchanged commit was, on the evidence available, a failure the commit did not cause: nothing about the code moved between the red and the green, so whatever moved was the measurement. Each flip is therefore a red check that did not mean a regression, which is what makes the flip rate an estimate of the noise share of failures. It estimates that share from above rather than exactly, and the table of attempt counts above says why — a genuine regression also passes on a later attempt a fair fraction of the time, so some flips are real problems re-rolled into a merge rather than bad draws.

# scripts/flip_rate.py — how often does a failed eval check pass on re-run, same SHA?
import collections, requests

REPO = "acme/assistant"
H = {"Authorization": f"token {TOKEN}", "Accept": "application/vnd.github+json"}

def eval_runs(days: int = 60):
    url = f"https://api.github.com/repos/{REPO}/actions/workflows/eval.yml/runs"
    page = 1
    while True:
        r = requests.get(url, headers=H, params={"per_page": 100, "page": page}).json()
        if not r["workflow_runs"]:
            return
        yield from r["workflow_runs"]
        page += 1

by_sha = collections.defaultdict(list)
for run in eval_runs():
    by_sha[run["head_sha"]].append(run["conclusion"])   # attempts, oldest first

retried = {sha: c for sha, c in by_sha.items() if len(c) > 1 and c[0] == "failure"}
flipped = {sha: c for sha, c in retried.items() if "success" in c[1:]}

print(f"commits whose first eval attempt failed and were re-run: {len(retried)}")
print(f"...of which a later attempt passed on the same SHA:      {len(flipped)}")
print(f"flip rate: {len(flipped) / max(len(retried), 1):.0%}")
# commits whose first eval attempt failed and were re-run: 88
# ...of which a later attempt passed on the same SHA:      47
# flip rate: 53%

Eighty-eight re-run failures over sixty days with 53% flipping is a much harder thing to argue with than a modelled standard deviation, and it comes with a list of SHAs somebody can go and look at. Join it against the per-case results in the stored artefacts and you get the third useful number, which is which cases are responsible: on this suite, the seven cases that flip verdict on an unchanged commit accounted for 41 of the 47 flips.

Shrinking the spread, and what each option costs#

Every fix works by reducing the standard deviation of the score, and they differ in what they cost and what they give up. Why the spread is the lever, rather than one lever among several, follows from how a red check on a healthy commit happens at all. It happens when a draw from the score distribution lands below the gate, and how often that occurs is governed by the distance between the gate and the centre of the distribution measured in standard deviations. Shrinking the spread increases that distance in those units without moving either the gate or the commit's true quality, so fewer healthy commits fail. Fewer false failures alongside the same number of real ones is precisely a rise in precision, and it is the only lever that raises precision without asking anybody to behave differently.

The first row is the one to do first because it is free and it is targeted. A case that returns a different verdict on identical input is not measuring anything; it is either ambiguously specified, graded by a rubric that does not discriminate, or genuinely dependent on something outside the commit. Reading those seven cases usually produces four repairs, two retirements and one genuine discovery about the product.

The second row deserves more use than it gets. A large fraction of what teams route through an LLM judge is mechanically checkable: the response parses as JSON against the declared schema, the cited document ID exists in the corpus, the tool name is one that exists, no PII pattern appears in the output — personally identifiable information, meaning an email address, a phone number or a card number matched by a regular expression — and the refusal string is absent. Those assertions have a standard deviation of zero and cost microseconds, and moving even a third of a suite onto them pulls the aggregate spread down while making the failures self-explanatory.

There is a structural option alongside those, borrowed intact from ordinary flaky-test practice, which is a quarantine lane. Cases identified as unstable move out of the blocking suite and into a job that runs on the same schedule, reports, and cannot fail the build. The blocking suite gets quieter immediately, the quarantined cases stay visible instead of being deleted, and quarantine carries an owner and a review date so the lane does not silently become the place where inconvenient cases go.

The one rule that makes it work is that quarantining is a decision recorded in the repository with a name attached, and the causal step there is the recording. A change to a file in the repository goes through review, lands in history with an author and a date, and can be counted — so the list has a size anybody can read off, a rate of growth that shows up in a diff, and a person per entry who can be asked why. None of that is true of a decision taken in a chat thread, and without it the lane cannot be audited, so nothing stops it absorbing the cases that fail for real reasons. A quarantine list that anyone can append to during a bad afternoon is a slower version of turning the gate off, slower only in that no single removal looks like a policy change.

The last row is the interesting one, because it takes the developers' re-run habit and makes it correct. If the team is going to run the suite three times anyway, have the pipeline do it and aggregate over the three rather than letting a human select the best. The mean of three runs has a standard deviation of 1.3 points against 2.2 for a single run, the cost is explicit and budgeted rather than hidden in individual developers' afternoons, and the reported number is an estimate of the commit rather than a high-water mark.

That 1.3 is one rule applied once, and the rule is the one every argument about repeated measurement rests on: averaging k independent draws from the same distribution does not divide the spread by k, it divides the spread by the square root of k. The square root of three is a bit over one and seven tenths, and a spread of 2.2 points divided by a bit over one and seven tenths lands right about 1.3 points, which is the table's figure. The same rule sets expectations for anyone tempted to go further, because running four times as many repeats would not quarter the remaining spread — it would halve it.

The honest thing to say about that row is that much of it is not new expenditure at all. On a commit that goes red — which is the only kind of commit anybody re-runs — the team was already paying for two or three runs of the suite, on the same commit, out of the same budget. What changes there is not the money but who does the aggregating and by which rule. A human selecting the best of three reports a maximum, and a pipeline averaging three reports a mean. One of those two summaries is biased upward by construction, for the reason worked out at the top of this chapter, and the other is not.

That is worth stating inside its condition, because the table's runtime cost of three times everything is the honest headline and it stays honest: a commit that would have gone green on the first attempt now costs three runs as well, and those are runs nobody was paying for before. The sharper point is the narrower one. On the commits people were already re-running, the money was being spent either way, and the only thing in question was who got to choose the summary — which is why this row buys most of its stability by taking that choice away from the person with an interest in the outcome.

Tightening the threshold moves the behaviour, not the quality#

The intuitive response to a gate people are routing around — that they are getting away with something and the line should be firmer — is worth working through, because it is the change most likely to be proposed in the meeting where the flip rate gets presented.

The sentence to watch for is the gate is too lenient, so we should raise it, and it deserves to have its appeal described before it is contradicted. It has the right shape: changes are getting through that should not, the threshold is the one knob that decides what gets through, and moving that knob is a one-line edit to a config file which costs nothing and needs nobody's cooperation. It also reads as the responsible option, since the people who propose it are the ones who care most about quality. The step it skips is the one the first half of this chapter spent its arithmetic on. Changes are not getting through because the bar is low; they are getting through because the measurement scatters and a developer is permitted to keep drawing until a draw lands on the right side of the bar. Raising the bar does nothing to the scatter, and it moves the bar further into the middle of it.

Raising the gate from 0.85 to 0.88 against a distribution with mean 0.856 and sd 0.018 means an unchanged commit fails 91% of the time. That figure is the same two steps as before with the sign reversed. A gate at 0.88 sits twenty-four thousandths above the unchanged-commit mean of 0.856, and one standard deviation is eighteen thousandths, so the new gate stands more than one and a third standard deviations above the centre of what a healthy commit produces — where the old gate at 0.85 stood just below that centre. The share of a bell-shaped distribution lying that far above its centre is under one in ten, so a healthy commit clears the new bar about one time in eleven and fails it the other ten, which is the 91%.

The failure rate goes from one pull request in four to nearly all of them, the noise share of those failures goes up rather than down, and the re-run norm goes from one or two attempts to as many as it takes. Expected attempts to clear 0.88 by chance is around eleven, at whatever the suite costs, and that figure follows from the one-in-eleven by a single step: when every try succeeds with the same chance and the tries are independent, the average number of tries before the first success is one divided by that chance. A bar you clear half the time takes about two tries on average, and a bar you clear one time in eleven takes about eleven. That is an average rather than a worst case, so about half the commits taking this route need more than eleven.

Long before anyone gets there the organisation finds the cheaper path: a [skip-eval] convention in commit messages, pull requests kept in draft, the required check quietly made optional for a release week and never made required again, or an approval-based bypass used routinely. None of those is announced as a change of policy, which is what makes them cheap.

What makes strictness self-defeating rather than merely ineffective is that the progression closes into a loop. A tighter gate fails more healthy commits. Every one of those extra failures is noise by construction, since the commits were healthy, so precision falls. Lower precision means a larger share of red checks that a re-run will clear, which makes re-running a better bet than reading the diff, so people re-run more often and with more attempts each. More attempts means a higher maximum, which means the effective threshold drifts further below the written one — which is the gap the tightening was meant to close.

That whole progression — a tighter line, a louder gate, and a quiet route around it — is the reason to state the diagnosis in the strong form rather than as a degradation. A gate whose failures are half noise has stopped separating good changes from bad ones, which describes something that is not functioning rather than something working less well than it did, and because it still emits a green check it is asserting something false about the changes that get through. Every strictness-first response makes the variance problem worse by increasing the number of times people have to fight the gate, and the levers that work all reduce the spread: fix the unstable cases, move what is mechanically checkable off the judge, quarantine with an owner, and aggregate repeated runs in the pipeline instead of letting a human pick the best one.

There is one honest limitation in all of this. Every lever in the table above buys stability with money or with wall clock: three runs instead of one, 300 cases instead of 120, three judge samples per case instead of one. A team that adopts all of them has a gate with a standard deviation under one point and a pull request check that takes twenty minutes and costs real money on every push, which is a different failure and the one the next chapter is about.