Intermittent failures and frame replay

A failure rate is a fingerprint that two suspects share. Take a tool that reached 214 agent runs and came back with an empty result on 11 of them, each time producing a model that told the user there were no invoices for a customer who has four hundred. Nobody has reproduced it by hand; an inspector has been driven at that tool perhaps sixty times across two days, with every combination of arguments anyone could think of, and it has never once returned an empty set for a customer with data.

The theory that forms from those facts is a race, and the reasoning behind it is that the failure is intermittent, it only appears under the agent, and concurrency is what makes things intermittent. Every clause of that is true and the conclusion does not follow, because the agent chooses new arguments on every run. An input shape the model produces roughly one time in twenty, hitting a handler that mishandles it every single time, produces exactly a 5% failure rate that never reproduces when a human types the arguments. Both hypotheses predict intermittency, both predict agent-only, and they call for opposite fixes.

Two suspects share a fingerprint here because the number on the board is a product rather than a reading. A failure rate observed across a run of agent calls is how often a run presents the condition that breaks the server, multiplied by how often the server breaks when it is presented. Write the deterministic handler defect in that form and the second factor is one, since the handler mishandles the shape every time it sees it, so the whole of the observed intermittency is carried by the first factor — the rate at which the model happens to write that shape. Write the race in the same form and the first factor is one, because any arguments at all will do, and the intermittency is carried entirely by the second — the chance that the window opens on a given call. Two different products can land on the same value, and one time in twenty is a value both of them can reach.

An experiment that separates them therefore has to pin one of the two factors rather than measure their product again. Pinning the first is much the cheaper: send the recorded arguments on every attempt and the rate at which the condition is presented becomes one by construction, so whatever failure rate comes back is the second factor read off directly. The instrument that does it is a replay — re-sending the exact bytes of a recorded request to the server, outside the run that produced them, as many times as the experiment needs. Fifty failures out of fifty attempts says the second factor is one and the recorded shape is sufficient on its own. Zero out of fifty says the shape is not sufficient, and whatever the failing run supplied on top of it is still missing.

The agent varies more than the arguments#

The inference that produced the race theory is worth naming, because it is tempting and its shape recurs. For a failure to be intermittent, something has to differ between runs; concurrency is one thing that can differ; so concurrency is a candidate. The argument then treats that candidate as the only one, which is a necessary condition being read as a sufficient one. Everything else an agent moves between runs qualifies on exactly the same grounds, and there is a good deal more of it than concurrency.

Each of the following is a variable that has to be held fixed before an experiment means anything, and the agent moves all of them at once.

Holding them fixed needs one piece of equipment that an inspector does not provide: a client that speaks the protocol and takes its instructions from a file. An inspector is built for a person issuing one call at a time, so it is the wrong shape for fifty repetitions, for eight simultaneous calls, or for reproducing a nine-call sequence with the original gaps between them. Forty lines over an SDK's client class covers it — perform the handshake, read a list of recorded frames, send them with whatever timing and concurrency the experiment calls for, and write every result to disk. Once that exists, the difference between a serial replay and a concurrent one is a flag, which is what makes the branch in the diagram below cheap enough to take.

The one variable this cannot hold fixed is the model itself, and it is the one most often changed by accident. Runs recorded last week and replayed this week may be running against a different checkpoint — the particular trained snapshot of the model that a provider happens to serve behind a given name — a different system prompt, or a different client build; staging and production frequently differ in all three. Recording the model identifier, the client name and version from the handshake, and the server build alongside every capture makes "did the population change" a query rather than a guess.

Every row in the right-hand column is available from a continuously running frame log of the kind chapter 3 builds, which is the reason to have it on all the time rather than switching it on when something breaks. What you need is not a description of the failing runs; it is the bytes of the failing requests.

Why the capture has to be continuous follows from a property of the evidence rather than from a preference about tooling. A frame is only recorded while a recorder is running. An intermittent failure gives no notice before it happens, so the run whose bytes you need is by definition one nobody was watching. Switching capture on after a report arrives therefore starts the clock at the wrong end: nothing can be replayed until the failure recurs, and at one run in twenty that wait is most of the investigation.

Replay separates the two hypotheses in fifteen minutes#

Pull the failing tools/call frames out of the capture, and pull a hundred successful ones for contrast. Then run each failing frame against the server directly, serially, fifty times, and count.

import json, collections, subprocess

frames = [json.loads(l) for l in open("capture.jsonl")]
calls  = [f for f in frames if f.get("method") == "tools/call"]
failed = {88, 91, 107, 133, 140, 151, 166, 178, 190, 201, 209}

def replay(params, n=50):
    hits = collections.Counter()
    for _ in range(n):
        out = subprocess.run(
            ["./mcp-call", "--tool", params["name"],
             "--args", json.dumps(params["arguments"])],
            capture_output=True, text=True).stdout
        hits[json.loads(out)["count"] == 0] += 1
    return hits

for c in calls:
    if c["id"] in failed:
        print(c["id"], json.dumps(c["params"]["arguments"]), replay(c["params"]))

Three outcomes are possible and each one closes a line of inquiry.

Each branch earns its conclusion from what the arm above it held still. The serial replay sends the recorded bytes and nothing else, one call at a time, so the arguments are fixed and no two calls overlap. A failure that survives those conditions needs neither a fresh sample from the model nor a second call to collide with. That is what makes the recorded shape sufficient on its own, and it is what condemns the mutex without anyone having to read the locking code: removing overlap cannot remove a failure that is already happening without any.

The concurrent arm differs from the serial one in exactly one respect, whether calls are in flight at the same time, and that single difference is what gives its result meaning. A failure that appears only there was caused by overlap. Overlap in turn can only cause a failure when two calls reach the same mutable state, since calls that share nothing cannot influence one another whatever order they run in. That is why the branch's instruction is to find which state rather than to guard it — a lock placed before the state is named is a guess about which of the server's shared things the two calls were fighting over, and a wrong guess leaves the failure in place while making the timing harder to reason about.

Fifty replays at roughly 200 milliseconds each is ten seconds per argument set, so eleven of them finish inside two minutes, and the hundred controls take another twenty. Set against two days already spent on a theory, the ratio is the whole argument for doing this first.

The controls are not optional, and they are the part people leave out. Replaying only the failing arguments tells you those arguments fail; replaying the successful ones tells you the difference between the two populations is in the arguments rather than in something you changed when you built the replay harness. A serial replay in which both populations fail is a broken harness, and finding that out after you have drawn a conclusion is worse than not running it.

Counting is only half of what a replay produces. The other half is an account of the failing population, and it has to balance. An argument shape that comes back fifty out of fifty explains exactly as many of the observed failures as there were calls carrying that shape, and any failure whose shape has not been replayed is still unexplained. Adding the explained groups back up and landing on the observed failure count, with nothing left over, is what licenses the sentence "we know why it fails". Landing short of it means a second defect is sitting inside the first one's symptom, and the shortfall is the part of the board where it is hiding.

The third outcome is the awkward one, and it is worth knowing what it costs before you reach it. When neither the serial replay nor the concurrent one reproduces the failure, the variable is somewhere else in the table above, and the remaining candidates are all about sequence rather than about arguments: the state a preceding call left behind, the position of the call within a long-lived session, or the gaps between calls. Each of those turns the experiment from replaying one frame into replaying the whole captured sequence — first with the original ordering, then in a fresh session against a session forty calls old, then with the captured inter-arrival times rather than back-to-back. That is an afternoon rather than fifteen minutes, which is why it is the third branch and not the first, and the capture is what makes it possible at all.

What a replay cannot settle is worth stating plainly, because a clean result is easy to over-read. It holds the request bytes fixed, so it says nothing about how often the model will write those bytes again; a defect that has been explained is not a defect that has stopped being reachable. It drives the server through a harness of your own, so it never exercises the production client and cannot reproduce a fault living there. And it reproduces a request rather than a moment, so any condition the server's own clock supplied when the failure happened — an entry that had just expired from a cache, a credential a second away from being refreshed — is simply absent from the replay, which is the deeper reason the third branch is expensive rather than merely slow.

A failure that disappears when you add logging has told you almost nothing#

Somebody will have already tried the obvious thing, and on this investigation somebody had: a branch that logs the arguments, the SQL, the row count and the elapsed time on every call, deployed to staging, after which the failure did not recur in ninety runs. The conclusion drawn at standup was that it must be a timing problem, since only a timing problem would care about logging.

That inference is unsafe because a logging statement changes at least three things at once. It adds latency, typically 50 to 500 microseconds for an in-process append and 1 to 3 milliseconds when the sink is synchronous and writes to a file or a socket, which is more than enough to close a race window measured in tens of microseconds. It usually takes a lock: most logging frameworks serialise writes to a handler, so concurrent tool invocations that previously interleaved now queue behind each other, and the concurrency the theory depends on has been quietly removed by the instrument measuring it. And on a stdio server, a log line written to the wrong stream corrupts the frame stream, which changes the failure into a different failure rather than removing it.

There is a fourth possibility that is easy to miss and was the real one here: the logging branch was deployed to staging, and staging's model routing was pinned to a different version than production's, which produced arguments in a slightly different style and stopped emitting explicit nulls. The variable that changed was not the logging.

Set the competing explanations aside for a moment and ask what the ninety clean runs are worth on their own, because the answer is neither nothing nor what the standup took it for. At a rate near one in twenty, a single clean run is what you would expect nineteen times out of twenty with the defect completely untouched, so a short stretch of them carries almost no weight — which is where most people go wrong, declaring a fix after a handful of runs and moving on. Length is what converts an absence into evidence, and the length has to be set against the observed base rate and stated before the runs are made rather than chosen afterwards from what happened. Ninety consecutive clean runs against a rate of one in twenty clears that bar comfortably, which is precisely why the conclusion felt solid. What it licenses is that the rate changed. Which of the things that changed alongside it moved the rate is a separate question, and a run of clean results carries nothing at all about that.

So the correct conclusion from a vanishing failure is that you have a fourth hypothesis, not a confirmation of the second. The response that keeps the investigation honest is to make observation non-perturbing rather than to accept the disappearance: keep the transport-layer frame capture always on, since it is outside the handler and adds a single append per message; buffer any in-handler diagnostics into a fixed-size in-memory ring, a buffer that overwrites its own oldest entries instead of growing, and flush it only on a failed result, so the hot path pays a memory write rather than a syscall; and do the analysis offline against the capture instead of adding statements to the code under suspicion. The replay experiment does not require the failure to be happening while you watch, which is the property that makes it usable at all.

The same reasoning condemns the mutex as a diagnostic. Merging it and observing the failure rate drop from 5% to 1% would have been read as confirmation, and a rate change is exactly what you get from any change that alters timing, including one that fixes nothing. A lock is a fix to deploy after the shared state has been named and the concurrent replay has produced the failure on demand; used as an experiment, it can only return "yes".

Underneath both of those cases is one rule about evidence. An observation separates two hypotheses only when the two disagree about what it will be, and here they agree: a deterministic input defect and a race both predict that the number moves when the timing moves. A number that moved is therefore consistent with both and discriminates between neither. The confirmation reading also discards the one part of that observation which did carry information. A fix that removes a cause predicts the failures stop, so a rate settling at 1% rather than at none is a rate with survivors in it, and the survivors are evidence against the mutex arriving inside the same measurement that was being read as proof of it.

The apparatus worth leaving running#

Almost all of it is cheap once written and none of it is available in the twenty minutes after something starts going wrong. Continuous frame capture with metadata always on and payloads sampled gives you the bytes of a failure you did not anticipate. Inspector fixtures give you a surface test that runs in a few seconds per call in CI. The raw conformance replays from chapter 6, on settling a dispute with a client vendor, defend the layers below your handlers, which is where the defects that cross vendor boundaries live. "How to Test MCP Servers: Tools, Debugging, and CI" is the reference for wiring the last two into a pipeline rather than a laptop.

The habit underneath all of it is narrower than a methodology and easier to apply: before running anything, say which two explanations the result would separate. A frame log separates a server that never answered from a client that never read. A direct call separates the server from everything above it. A replay separates an input shape from a race. A change that could not have come back the other way — a mutex, a global timeout, a logging branch — has not tested anything, however good it feels to have shipped it.

That leaves the honest limit. None of this reaches the failures that are properly the model's: a description that reads two ways, a tool the model declines to call, an argument style that changes when a provider ships a new checkpoint on a Tuesday. Those have no frames to read and no fixture to replay, and they are found by running the agent and watching, which is the loop this topic opened with and has been careful not to replace.