Testing a retry wrapper

A retry wrapper sits in front of a provider call and decides three things: which failures are worth another attempt, how long to wait before each one, and when to stop. All three are ordinary deterministic logic, and all three are awkward to test for the same reason — exercising them honestly means waiting. A wrapper configured for five attempts with exponential backoff, where each delay doubles the last, at a base of one second, spends 1 + 2 + 4 + 8 + 16 = 31 seconds getting to its last attempt, and a test that exhausts the retries spends those 31 seconds too if the wrapper sleeps by calling time.sleep.

That sum is worth checking rather than accepting, because the obvious way of counting gives a different total. Five attempts have four gaps between them, so adding up the waits between attempts gives 1 + 2 + 4 + 8, which is fifteen seconds and not thirty-one. The two figures differ by the last term, and the last term is a fact about where the wait sits in the loop. A wrapper that sleeps before each retry produces four delays; a wrapper that sleeps after each failed attempt and then decides whether to try again produces five, because the fifth failure is followed by its own delay before the wrapper concludes it has run out of attempts. The schedule in this chapter is the second kind, which is why the last delay in the sum has no attempt after it.

That final sleep is not a rounding detail. It is sixteen seconds a caller waits for an answer the wrapper has already decided not to pursue, and no test that counts calls can see it, because the call count is five under either arrangement. It shows up in exactly one place, which is the recorded list of delays — and this is the first of several times in this chapter that a list of sleeps turns out to distinguish two implementations that a call count treats as identical.

The obvious repair is to make the delays smaller under test. This chapter is about why that particular repair removes the case worth having, and what to do instead: take the clock away from the wrapper rather than shrinking its numbers, using the Clock protocol and FakeClock from the second chapter of this topic, where sleep() records the delay it was asked for and advances the fake's own now() by the same amount. What that buys is a file running in single-digit milliseconds while executing the exact schedule that ships, and — more usefully — the ability to write the one test that a shrunken-constants suite cannot express at all.

What shrinking the constants deletes#

A wrapper in front of a slow upstream normally enforces two limits at once. A per-attempt timeout of 10 seconds bounds how long any single request may take. An overall budget of 30 seconds bounds the whole operation including the waiting between attempts, because the caller — an HTTP handler with its own 35-second deadline — needs the operation to return one way or another before its own clock runs out.

With a base delay of one millisecond, the accumulated backoff across five attempts is 31 milliseconds against an overall budget of 30,000. No arrangement of a fake's responses can make the second limit fire, because the only way to consume the budget under test is through the delays, and the delays have been made a thousand times smaller than the thing they are supposed to interact with. Every test in the file passes. The code path that cuts a retry short because the budget is nearly gone is unreachable from the suite, in both the passing and the failing direction.

This is what production then looks like when that path is wrong. Three attempts, each taking 9.5 seconds and each therefore inside the per-attempt timeout, separated by 1 and 2 seconds of backoff:

Reading that table down the "Ends at" column is one addition per row and shows where the overrun comes from. The first attempt runs from zero to 9.5. One second of backoff carries it to 10.5. The second attempt adds 9.5 and lands at 20.0. Two seconds of backoff carry it to 22.0. The third attempt adds 9.5 and lands at 31.5. Every one of those additions is legal, in the sense that no limit in the wrapper's configuration was violated at the moment it was checked: each attempt was compared against the per-attempt timeout, each took 9.5 seconds, and 9.5 is under 10.

The budget column is the same arithmetic run in the other direction, starting at 30 and subtracting each row's duration, and it is the column that never gets consulted. What makes the overrun possible is not that any check failed but that the check which would have caught it was never made. Three quantities each individually inside their limit sum to a total outside a limit that nothing compares against, and the sum is where the caller lives. Testing a wrapper of this shape therefore means testing the two limits interacting, not each of them alone, and a suite that exercises them one at a time will report both as correct.

Every attempt satisfies the per-attempt timeout, and the operation returns at 31.5 seconds against a 30-second budget. Whether that is a bug depends on what the wrapper is supposed to do at 22 seconds with 8 seconds of budget remaining: it should either refuse to start attempt 3 or start it with an 8-second deadline rather than a 10-second one. A suite that only exercises the per-attempt timeout cannot tell which the code does, so a wrapper that ignores the budget entirely passes review, and what leaks into production is a handler that occasionally exceeds its own deadline, holds a worker thread past the point where the load balancer has given up, and returns a response to a connection that has already been closed.

Driving time instead of spending it#

The alternative keeps the shipped constants and takes the clock away. The fake client returns a programmed sequence of responses and exceptions and records what it was called with:

class FakeLLMClient:
    def __init__(self, responses: Sequence[LLMResponse | Exception]) -> None:
        self._responses = list(responses)
        self.calls: list[dict] = []

    def complete(self, **kwargs) -> LLMResponse:
        self.calls.append(kwargs)
        item = self._responses[min(len(self.calls) - 1, len(self._responses) - 1)]
        if isinstance(item, Exception):
            raise item
        return item


def test_retries_twice_on_429_then_succeeds():
    clock = FakeClock()
    inner = FakeLLMClient([
        RateLimited(status=429), RateLimited(status=429), ok_response(),
    ])
    client = RetryingClient(inner, clock=clock, base_delay=1.0, max_attempts=5)

    result = client.complete(model="claude-sonnet-4-5", prompt="hi", max_tokens=10)

    assert result.finish_reason == "stop"
    assert len(inner.calls) == 3
    assert clock.slept == [1.0, 2.0]


def test_gives_up_immediately_on_400():
    clock = FakeClock()
    inner = FakeLLMClient([InvalidRequest(status=400)])
    client = RetryingClient(inner, clock=clock, base_delay=1.0, max_attempts=5)

    with pytest.raises(InvalidRequest):
        client.complete(model="claude-sonnet-4-5", prompt="hi", max_tokens=10)

    assert len(inner.calls) == 1
    assert clock.slept == []

Two assertions carry each test. The call count says whether a retry happened, and clock.slept says what schedule was followed, which is a stronger statement than "it retried" — it catches a wrapper that retries with a constant delay when it is supposed to back off exponentially, and it catches an off-by-one where the first delay is 2 seconds because the exponent starts at 1. Both of those pass a test that only counts calls.

The 400 test is the other half of the specification. Retrying a request the server has already rejected as malformed adds latency for the caller and load for the provider and cannot succeed, so a wrapper that treats all failures alike is wrong in a way that only shows up as a slow error. Asserting clock.slept == [] pins that no backoff was even computed.

The other question these tests should settle is which failures are retried at all, and that is a table of status codes rather than a narrative, so it belongs in a parametrised test where the policy is visible in one screen:

@pytest.mark.parametrize("status,expected_calls", [
    (429, 3),   # rate limited — retry
    (500, 3),   # provider fault — retry
    (503, 3),   # overloaded — retry
    (529, 3),   # provider-specific overload — retry
    (400, 1),   # malformed request — never
    (401, 1),   # bad key — never
    (403, 1),   # permission — never
    (404, 1),   # unknown model — never
    (422, 1),   # schema rejected — never
])
def test_retry_policy_by_status(status, expected_calls):
    ...

Written this way the policy is reviewable by someone who has never read the wrapper, and adding a status code is a one-line diff a reviewer can argue with. The 404 row catches a specific production pattern: a model alias that has been retired returns 404, and a wrapper that retries it turns an instant, obvious configuration error into a 31-second timeout that looks like a provider outage.

Jitter — a small random offset added to each delay so that many clients retrying at once do not come back in lockstep — makes clock.slept unassertable, and randomness is a dependency in the same way time is. Pass a random.Random instance in from the composition root, seed it in the test, and assert against the sequence that seed produces; or assert bounds rather than exact values, checking that each delay falls within the jitter window around the nominal schedule. The version that does not work is leaving the module-level random in place and loosening the assertion until it always passes.

There is a lighter-weight alternative to injecting a clock, which is patching time.sleep for the duration of the test, or using a library like freezegun to control datetime.now(). Patching sleep alone makes the file fast and leaves the wrapper's own reading of the current time untouched, so a budget computed from time.monotonic() still sees zero elapsed time. Freezing the clock globally reaches every module in the process, including ones you did not intend to affect, and it has to be told when to advance. An injected clock is a slightly larger change to production code in exchange for a test that says what it means, and the deciding factor is usually that the wrapper needs now() and sleep() to agree with each other.

Writing the case that separates the two limits#

With a controllable clock, the interaction from the table above becomes an ordinary test. The fake needs to consume time, which means letting it advance the clock as part of responding:

def test_overall_budget_cuts_the_third_attempt_short():
    clock = FakeClock()
    inner = SlowFakeClient(clock, per_call_seconds=9.5,
                           responses=[RateLimited(429), RateLimited(429), ok_response()])
    client = RetryingClient(inner, clock=clock, base_delay=1.0,
                            attempt_timeout=10.0, overall_budget=30.0)

    with pytest.raises(BudgetExhausted):
        client.complete(model="claude-sonnet-4-5", prompt="hi", max_tokens=10)

    assert len(inner.calls) == 3
    assert inner.deadlines[-1] == pytest.approx(8.0)   # not 10.0
    assert clock.now() <= 30.0

inner.deadlines[-1] is the assertion that distinguishes the two limits. The third attempt begins at 22 seconds, and the wrapper must hand it the smaller of the per-attempt timeout and the remaining budget, so a correct implementation passes 8 seconds and an implementation that only knows about attempt_timeout passes 10. Both produce a failure; only one of them produces it before the caller's own deadline. This test is unwritable with millisecond constants, since at that scale the budget is never approached and min(10.0, remaining) always returns 10.

Both limits are readable from the clock the test controls, which is what makes this writable at all. FakeClock.now() advances by exactly the amounts the code under test asked for — the fake client's 9.5 seconds of work and the wrapper's own sleeps — so the deadline the wrapper computes is derived from those same amounts and can be asserted accordingly. Nothing in the test measures anything. Every quantity in it was put there by the test or computed from quantities the test put there, which is why the file can execute a schedule spanning half a minute of modelled time in single-digit milliseconds of real time.

The same technique covers the questions that follow: what happens when the budget is already exhausted before the first retry, whether the wrapper returns the last error or a wrapper-specific exception, and whether it counts the time spent sleeping against the budget or only the time spent in calls. Each is a couple of lines and each is a decision someone made implicitly.

Failures a fake cannot produce#

A fake returns the failures you thought to program, so the coverage it provides is bounded by your imagination about how a provider misbehaves. A 429 in the wild often carries a retry-after header, and a wrapper that ignores it in favour of its own exponential schedule will either retry too early and be rejected again or wait far longer than necessary. Testing that requires a response object with real headers, which points at a recorded cassette rather than a hand-built exception.

Connection resets, TLS handshake failures and DNS timeouts arrive as exception types from the HTTP stack rather than from the SDK, and whether your except clauses catch them is a question about class hierarchies you can only answer against the real stack. A fake raising RateLimited never exercises the path where httpx.ReadTimeout propagates past a handler written only to expect provider errors. Streaming introduces a failure with no equivalent in the request-response world, where the connection drops after 300 tokens of output have already been consumed by the caller, so retrying means the caller sees a prefix twice — and no fake returning whole responses can represent that.

The provider's own timeout semantics are the subtlest of these. A request your client abandons after 10 seconds may still be running on the provider's side, so a retry can produce two completions and you are billed for both. Whether that matters depends on the operation's idempotency — whether performing it twice leaves the world in the same state as performing it once — and it is not observable from any test that never touches the network.

There is a fifth item that is neither the fake's fault nor fixable by touching a provider, which is whether retrying is safe at all. A completion request is idempotent in the sense that repeating it does no damage, but the operation the wrapper sits inside frequently is not: an agent turn that retries after a timeout may execute the same tool twice, and if that tool issues a refund the second execution is a real refund. Deciding where the retry boundary sits — around the HTTP call, around the model turn, or around the whole agent loop — is a design question, and the test that pins it asserts that the tool handler was invoked exactly once across a run where the underlying call was made three times. That test is writable with the fake and is easy to forget, because the retry wrapper looks self-contained.

What is left in the fast offline suite after those four move elsewhere is everything about the wrapper's own logic, which is most of it, at a cost of milliseconds. The trade this chapter argues for is one parameter on a constructor in exchange for a suite that runs the numbers you ship: the call count says a retry happened, the recorded sleeps say on what schedule, and the recorded deadlines say whether the two limits agree when they disagree.

Before getting to how the offline and provider-touching suites are separated, there is one more class of failure that never reaches the network at all: a request the model answers perfectly and your code cannot use, because the tool schema you advertised and the arguments you parse have drifted apart.