Removable flakiness and sampling variance

A flaky test is one that passes on some runs and fails on others with nothing in the repository changing in between. Most engineering organisations have a written policy for them, and it is a reasonable policy: quarantine the test — move it out of the blocking suite so its result no longer fails anybody's build — file a ticket, fix it next sprint. The policy was written for a world in which every intermittent failure has a root cause, because until recently every intermittent failure did.

Once a suite contains model calls, that stops being true, and two failures with identical dashboard rows can need opposite treatment. This chapter is about telling them apart, by a distinction that does not depend on the failure rate, the age of the test, or how the failure reads in the log.

Two red rows, two different questions#

Here are two rows from the flakiest-tests page of one continuous integration dashboard, sorted by failure rate:

test_invoice_total_parses                    5.0%   (7 / 140 runs)
test_summary_mentions_refund_window         15.0%  (21 / 140 runs)

test_invoice_total_parses calls no model. It reads fixtures/invoice_8812.json off disk, runs the parser, and asserts the total is 4238.50. When it fails, the total comes back as 0.00. The fixture has not changed since it was committed; the parser has not changed in two months.

Digging into it takes about forty minutes. The parser memoises exchange-rate lookups — caches each answer on disk so the second request for the same currency pair does no work — to /tmp/fx-cache.json, the path is a module constant with no worker suffix, and the suite runs under pytest -n 4. Four workers share one cache file. Most of the time the write that populates it completes before any other worker reads it; occasionally a worker reads the file while another is halfway through rewriting it, gets a truncated document, swallows the JSONDecodeError, and falls back to a rate table of zeros. The 5% is the probability that two specific workers interleave inside a window of a few milliseconds.

The mechanism under that sentence is one step and it is the step people skip. Rewriting a file in place is not one event. It is an open, some number of writes, and a close, and at every instant between the first write and the close the bytes on disk are neither the old document nor the new one — they are a prefix of the new one. A reader that opens the path during that interval gets exactly what is there, which is valid bytes forming an invalid JSON document. Nothing is corrupted and no error is raised by the filesystem, because everything behaved as specified; the reader simply asked a question during the one window in which the answer is meaningless. The size of that window, divided by the interval between reads, is where a number like 5% comes from, and it is why the rate moves with machine load and worker count rather than staying put.

That is a defect, in a specific and nameable sense: there is a piece of state — a shared path — whose removal makes the failure rate go to zero and stay there. Parameterise the cache path by PYTEST_XDIST_WORKER and the test passes 140 runs out of 140, forever, on any machine, at any load. The fix is four characters of an f-string. It is also, and this matters more than the size of the fix, a real bug in production code: the same race exists in the service, where four gunicorn workers share the same file, and the same truncated read produces invoices totalling zero for real customers.

test_summary_mentions_refund_window does call a model. It sends one support ticket, asks for a summary, and asserts that the string "14 days" appears in the result. In twenty-one runs out of a hundred and forty, the summary is a perfectly good summary that mentions the refund policy without naming the window: "customer is outside the refund period", or "refund window has closed". The requirement that an agent reading the summary can see the deadline is arguably violated in those runs. The requirement that the answer be sensible is not.

Ask the same question of this row and the answer comes back empty. Which piece of state, if removed, makes the failure rate go to zero? There is none. The test is fully hermetic, meaning nothing outside its own declared inputs can reach it: same fixture, same prompt bytes, same pinned model, no shared files, no clock, no ordering dependence. Running it alone on a quiet machine reproduces 15% just as reliably as running it in the suite.

That last observation is doing more work than its length suggests, and it is the cheapest experiment in this chapter. Every removable cause named so far — a shared cache path, a clock, an ordering dependence, contention between workers — is a piece of state that something else touches. Running the case alone, slowly, on an idle machine removes all of them at once, without anybody having to guess which one applied. So if the failure rate holds up under isolation, the whole family has been eliminated in a single run rather than one hypothesis at a time. If it collapses, you have not learned which shared thing was responsible, but you have learned that a shared thing was, which is enough to know the afternoon is worth spending. It costs the time of one suite run and it is worth doing before the first hypothesis is voiced.

For this row, isolation changes nothing. The variation comes from the sampler — the draw that the previous chapter measured directly, producing four distinct summaries in five runs of one prompt — and no amount of hygiene in the harness, the code around the assertion that loads the fixture, makes the call and records the result, touches it. Rerunning this test for a year will not make it deterministic.

The two rows therefore need opposite treatment. Quarantining the first one buys time on a live production race, and the "fix" is a genuine fix. Quarantining the second one removes the only thing in the delivery path that was looking at model behaviour, while the dashboard reports green.

What @retry(times=3) actually asserts#

The other way to make a red row go away is cheaper than quarantine and feels less like giving up, so it is what most teams reach for:

@retry(times=3)
def test_summary_mentions_refund_window():
    ticket = load_fixture("tickets/refund_window.json")
    summary = summarize_ticket(ticket)["summary"]
    assert "14 days" in summary

The decorator did not make the test more reliable. It changed the claim the test makes. Before, the assertion was "a draw from this distribution contains the refund window". After, it is "at least one of three draws from this distribution contains the refund window". Those are different statements about the world, and the second is far weaker than it looks.

Write out what the retry does to a per-sample pass rate. If a single call passes with probability p, three independent attempts produce a green run with probability 1 - (1 - p)³.

That expression is worth building rather than quoting, because every later chapter of this topic uses the same two moves. Start from the only way the decorated test can go red: all three attempts have to fail, since one success anywhere stops the loop and reports green. So work out the probability of three failures and subtract it from one, which is legitimate because "all three fail" and "at least one passes" are the only two outcomes and one of them must happen. A single attempt fails with probability 1 - p, written that way because p is the chance it passes and there is nothing else it can do. Three failures in a row is (1 - p) multiplied by itself three times, and multiplying is allowed here for a specific reason rather than by convention: the attempts are independent, meaning the outcome of one carries no information about the next. That independence is a real property of the situation and not an assumption of convenience — each attempt is a fresh draw from the same distribution, the model holds no memory of the previous call, and nothing about the first failure makes the second more or less likely.

Put one value through it before trusting the general form. At a per-sample rate of 0.60, each attempt fails four times in ten. Three independent failures happen four-tenths of four-tenths of four-tenths of the time, which is a little over six times in a hundred. Subtract that from one and the build comes back green about 0.936 of the time. Doing the same across a range of per-sample rates is what makes the problem visible:

Read the 0.60 row against the 0.99 row. A case whose per-sample rate collapses from 99% to 60% — two in five customers now getting a summary that omits the deadline — still shows green about 94 times in 100. The dashboard barely twitches. Noticing the difference at a level anyone would act on takes something like fifteen consecutive builds, and by then the regression is a month old.

Where that fifteen comes from is the practical form of the table and worth having, because it converts a probability into a wait. The 0.60 row puts about six red runs in every hundred builds, which is one red build roughly every sixteen. A single red build against a background of occasional reds tells nobody anything, so the signal a person would act on is a run of reds close enough together to look unusual — and at one in sixteen, a fortnight of builds can easily pass with one red in it, which reads exactly like the flakiness everyone has already agreed to tolerate. The regression is not hidden by anybody's carelessness. It is hidden by an event rate that is indistinguishable from the noise the team taught itself to ignore when they added the decorator.

There is a second cost, and it is the one worth internalising. Every run of that decorated test computes exactly the number you wanted — it takes samples, it observes how many passed — and then throws it away. Three samples went out, one or two came back acceptable, and the only thing recorded is a boolean. The information was in your hands and the harness discarded it before writing the result. A test that ran three draws and recorded "2/3" would be strictly more useful than the same test recording "pass", at identical cost in tokens and latency.

The retry is not senseless in every context. Wrapped around a call that fails because the provider returned a 429 or a connection reset, it is doing exactly its job: the underlying event is a transient infrastructure failure, retrying resolves it, and no information is lost because there was none to record. That is why the pattern is tempting here — engineers have used it correctly for years against network flakiness, and the syntax is identical. The difference is invisible in the decorator and lives entirely in what the failure means.

Two smaller costs are worth pricing while the decorator is on screen. On any run where the first attempt fails, the test makes two or three model calls instead of one, so the suite's token bill and its wall-clock time both scale with how badly it is doing — the worse the system gets, the slower and more expensive the green build becomes, which is a signal nobody is watching for. And the number three is a choice made by feel. Nothing about a codebase determines it; someone typed a number that made the build pass. times=5 would keep a case at a 30% per-sample rate green about five builds in six.

Four responses, and when each is right#

Laying the options side by side makes the mismatch obvious:

The diagonal is what makes the dashboard dangerous. The response that is correct for one row is actively harmful on the other, and a page that ranks rows by failure rate gives you no way to tell which row you are looking at.

Model tests get ordinary defects too#

The distinction is about causes, not about which tests call a model, and reading it as "model tests are flaky by design, so red is fine" costs you the ability to find real bugs. A test that calls a model can carry both kinds of flakiness at once, and the removable half is usually easier to spot than the general case suggests.

Three causes account for most of it. Provider errors — 429s, 529s, connection resets — that a harness records as case failures rather than as harness failures, which drags the measured pass rate down by however loaded the provider was that night. Shared fixture state, exactly as in the invoice parser, when a suite writes recorded responses or scored results to a common path. And a floating model alias in the config, where a name resolves to whatever version is current and the thing being measured changes underneath the measurement without a single commit.

The clean way to separate the two halves is to hold the model call fixed. Record the responses for a run, then replay them from disk with the network disabled and run the suite twenty times. Any variation that survives is yours: it is in the harness, the scoring code, or the fixtures, and it has a root cause you can remove. Any variation that disappears was the sampler. That replay is worth building before anyone spends an afternoon arguing about a rate, because it turns a judgement call into a ten-minute experiment.

The replay has one blind spot worth naming. It fixes the model's output, so it cannot distinguish a harness bug that only fires on outputs the recording does not contain — a parser that crashes on a response with a trailing comma, say, where the recorded run had none. Recording a wider set of responses is the only answer, and it is the same problem as choosing cases, which the fourth chapter of this topic takes up.

Sorting an intermittent failure by cause rather than by rate is a two-line diagnostic worth running before any policy is applied. Name the state whose removal drives the rate to zero: if you can name it, the fix is the work, and quarantine is at best a holding action on something also broken in production. If you cannot, and the rate reproduces in isolation on a quiet machine, the generator is the cause, and every response that treats the symptom either deletes the signal or converts it into a weaker claim while the dashboard turns green.

Neither response tells test_summary_mentions_refund_window what to assert instead. Exact-string equality pins one draw out of many, and neither retrying nor quarantining rescues it. The assertion itself has to change, and the shape it changes into — parse the output, check the properties every acceptable answer must have, then aggregate across a whole set of cases rather than demanding each one pass every time — is what the next chapter builds.