Saturation tests and kill switches

A saturation test deliberately raises offered load against a model endpoint until the platform starts rejecting requests, and then keeps the pressure on long enough to watch what happens when it stops. It is the only way to observe the things a synthetic error cannot produce: the retry-after values the platform chooses under real pressure, how long a bucket takes to refill, and how a fleet of your own workers behaves against that refill. Every one of those is a property of the platform's implementation rather than of your client, and none of them can be guessed.

That last claim deserves its steps, because "cannot be guessed" is the kind of sentence a reader accepts and then quietly disbelieves. Three things decide what happens at the ceiling, and none of the three is visible from your side of the connection. The first is the refill schedule: a bucket's rate is published, but its maximum and the window it is enforced over usually are not, so how much budget exists one second after a rejection is an implementation detail of somebody else's admission controller. The second is the retry-after value, which the platform computes from its own view of when capacity will exist, and which may be a constant, a function of how far over the line you went, or a function of how many callers it is currently deferring. The third is the interaction, and it is the one that cannot be reasoned about from a single worker at all: what your fleet does at the ceiling is what many workers do against one refill schedule, and that is emergent behaviour of a system whose other half you did not write and cannot read.

A synthetic 429 injected by a local proxy supplies a status code you chose and a header you chose. It therefore exercises your side of all three and the platform's side of none of them, which is not an argument against using it — it is an argument for knowing which half of the question it answers.

The complication is that a quota bucket on a cloud model platform is scoped to an account or subscription as well as to a model, so the set of workloads a test can degrade is the set of workloads billing to that account rather than the set in your repository. This chapter is about the containment that makes the test repeatable — because the second and third runs are where the useful data comes from, and a first run that produces somebody else's outage rarely gets a second.

What the blast radius actually contains#

Draw the boundary honestly before deciding how to contain it. On a typical enterprise arrangement the account covers every team the cloud contract covers, and the list is usually longer than engineering believes, because internal tools, data pipelines and someone's evaluation harness are all on it. Blast radius here means exactly that set: everything whose requests draw on a bucket your test is emptying.

The scale is worth computing rather than hand-waving, because it decides how much containment is proportionate. A test that occupies 85% of an account's token budget for eleven minutes is not a small experiment; it is the dominant workload on that account for the duration, and any service sharing the account sees a provider-side error rate for as long as it runs. Services with a fallback path degrade quietly, which is worse for attribution than failing loudly: their own dashboards show a provider problem, and the platform's status page shows all green.

The status page is worth one sentence of mechanism, because engineers reach for it first and it cannot help. A provider status page reports the health of the provider's service across its tenants. An account driving its own quota to zero is not a fault in that service; it is the service enforcing a limit exactly as designed, for one customer, and there is no threshold of that behaviour at which a status page turns amber. The neighbouring team is therefore looking at a green page while receiving errors, and the correct conclusion from those two facts — that the problem is not the provider's — is available to them and is the one thing nobody believes when the errors are coming from a provider's hostname.

There is a second consequence with no dashboard at all. Sustained saturation is visible to the platform's own operations, and accounts that repeatedly pin their limits get flagged — not as abuse in the malicious sense, but as capacity risk, which is the same queue that quota-increase requests go through. A team that ran an uncontained saturation test twice in a quarter and then asked for a limit raise has made its own case harder to argue.

How much of that is automated and how much is a person reading a utilisation graph varies between platforms, and none of it is documented anywhere a customer can read. That is a reason to treat it as a cost rather than as a rule. You cannot verify the mechanism, and you equally cannot verify that it is absent, so the only position that survives either answer is to make the run look like what it is — announced, bounded in rate and in duration, and attributable to a project created for the purpose — rather than like an account behaving unpredictably.

Where the test runs decides who it can hurt#

Isolation does most of the work, and there are three ways to get it with quite different costs.

The cleanest is a separate project, subscription or account carrying its own quota. Where limits attach to a project or subscription id, a dedicated one gives you a bucket nobody else draws from, and saturating it is arithmetic rather than an outage. The cost is administrative — creating it, getting model access approved on it, getting a non-trivial quota assigned so the test measures something like production behaviour, and keeping its credentials out of the production configuration. Budget days rather than hours, because model access approval is frequently the long pole.

The second option is a non-production region. Quota is regional, so driving eu-west-3 to its ceiling leaves eu-central-1 untouched, and the isolation is real. What you get in exchange is a smaller bucket, since a region nobody runs production in has a default allocation rather than a negotiated one, and a smaller bucket saturates at a load that may not exercise your client's concurrency behaviour the way production would. It also tells you nothing about how your production region behaves at its own ceiling, which for most purposes is fine, because what is being tested is the client rather than the platform.

The third is running against production capacity in a window when nobody else needs it, which is what teams pick when the first two are blocked. It is the weakest containment and it is not worthless, provided the window is genuinely quiet and every other control below is present. Its failure mode is that "off-peak" gets defined by the person running the test rather than by the account, and on a subscription with teams in three time zones there may be no hour that is off-peak for everyone. Ask the account rather than your calendar: the platform's own usage metrics, grouped by project over a fortnight, show the real trough, and it is often not the one you would guess, because batch and reporting jobs cluster at night.

A bounded ramp and a kill switch make the test repeatable#

The load generator needs three properties a simple concurrency loop does not have. It increases offered load in defined steps rather than starting at full rate, it watches something other than its own success rate, and it can be stopped from outside the process. That last one is the kill switch, and it has to work when the person who started the test is not at the keyboard, which rules out relying on interrupting the terminal that launched it. The watching is done by a watchdog: a loop running beside the load generator whose only job is to decide when the run has to stop.

The Drain state is not decoration. The behaviour the next chapter is about only appears after offered load returns to normal, so a test that stops by killing the process loses the most interesting part of the trace.

The reason a trace's most interesting stretch comes after the load stops is that the interesting behaviour is not caused by the load. It is caused by what your own clients did with the rejections, and the rejections were handed out during the ramp while their consequences — queued retries, requeued tasks, workers sleeping on a delay somebody configured years ago — are still pending at the moment the generator goes quiet. Killing the process at the top of the ramp destroys nothing about where the ceiling is and everything about how long it takes to stop mattering. Recovery is the part that decides how long a neighbouring service stays broken, and it is the only part a synthetic error cannot manufacture, because it needs a real backlog produced by real rejections and then amplified by whatever your own retry layers do with them.

import time, threading, requests

STEPS_RPS = [2, 4, 8, 16, 32, 64, 128]   # bounded: the last step is the ceiling
STEP_SECONDS = 60
HOLD_SECONDS = 90

stop = threading.Event()

def watchdog():
    """Trips on signals the test itself cannot see: other teams' health."""
    while not stop.is_set():
        if requests.get(KILL_SWITCH_URL, timeout=2).text.strip() == "stop":
            stop.set(); return                       # operator, from anywhere
        if error_rate("payments-risk-scoring", window="60s") > 0.01:
            stop.set(); return                       # collateral damage
        if error_rate("our-service", window="60s") > 0.05:
            stop.set(); return                       # our own production
        time.sleep(5)

threading.Thread(target=watchdog, daemon=True).start()

for rps in STEPS_RPS:
    if stop.is_set():
        break
    run_at(rps, STEP_SECONDS, stop)
    if throttled_fraction_last(seconds=30) > 0.10:
        run_at(rps, HOLD_SECONDS, stop)              # hold at the ceiling
        break

drain_and_observe(seconds=900)                       # the part that matters

The two error-rate checks in the watchdog are the difference between an abort path that works and one that only looks like it does. A load generator that stops when its own requests fail will not stop, because at saturation its requests are supposed to fail; that is the experiment. The condition worth aborting on is a rise in errors somewhere the test was not meant to touch, which means wiring the watchdog to another team's service-level indicator — the metric their own alerting treats as the definition of healthy, usually an error rate or a latency percentile — and getting read access to that indicator is a conversation you have during planning rather than at 02:00.

The general form is worth extracting, because it applies to any test that degrades something on purpose. The signal an abort is wired to has to satisfy two conditions at once: it must move when the test causes harm, and it must sit outside the set of things the test is deliberately breaking. Your own error rate satisfies the first and fails the second, which is why wiring the abort to it produces a test that stops immediately and measures nothing. The provider's status page satisfies neither. Another team's service-level indicator satisfies both, and that is why a technical control ends up requiring a conversation with people who are not running the test.

The upper bound on the ramp matters as much as the abort. STEPS_RPS ends at 128 because that is roughly twice the rate at which throttling is expected to begin, and an unbounded ramp keeps climbing until something outside the plan breaks.

The steps double rather than climb evenly, and the reason is that the ceiling's location is unknown before the run. A doubling sequence covers a range in a number of steps proportional to the logarithm of the range rather than to the range itself, so seven entries span a rate ratio of sixty-four to one and find an unknown ceiling anywhere inside it. Doubling also bounds the run in time as well as in rate, which is the property the notification list depends on: seven steps of sixty seconds is seven minutes, plus ninety seconds at the ceiling, so the plan can state how long other teams are exposed before anybody starts.

Bounding the ramp also bounds the bill: requests admitted before the ceiling is reached are billed at full price, and a ramp with a runaway top step is the standard way a test costs four figures.

Telling people is a control, not a courtesy#

Write the notification list into the test plan and treat sending it as a precondition. It needs the on-call rotations for every service on the subscription, the account owner, and whoever fields the provider relationship, so that if the platform opens a question about the account's usage pattern, somebody can answer it with "that was our scheduled test on the 14th" rather than starting an investigation. The message needs the window, the region, the model, the project id being saturated, the expected symptom, the kill switch URL, and the name of the person who can be woken. Three lines is enough.

Isolation changes the numbers you measure#

None of this makes the test risk-free, and two limits deserve stating. Isolating into a separate project changes the numbers: a different bucket size means throttling begins at a different offered load, so the timings recorded are your client's behaviour at a ceiling rather than a prediction of when production will hit its own. That is the right trade, because the thing under test is the client's retry and recovery logic, which does not care where the ceiling sits.

The second limit is that a watchdog wired to another team's error rate detects damage after it has started. Ten to fifteen seconds of elevated errors will have happened before the abort fires and in-flight requests drain, which is acceptable for a scoring service with a fallback and is not acceptable for something without one. Where the downstream system has no fallback, the answer is isolation rather than a faster watchdog.

The belief worth blocking at this point is that isolation makes a test safe, rather than making one specific failure impossible. A dedicated project removes the shared bucket, which is the failure that was predicted. It does nothing about a credential still pointing at production, a client library with a region compiled into it, an environment variable inherited from whichever shell the run was launched in, or a service the test calls that itself calls the model on the shared account. That is exactly why the rerun above kept the watchdog wired to the payments team's indicator after the bucket had been separated. These controls are not alternatives to one another and they are not a checklist to choose from; they fail in different ways, which is the only reason to have more than one of them.

There is also a cheaper technique people reach for instead, and it is worth being precise about what it substitutes for. Injecting synthetic 429s with a local proxy in front of the client costs nothing, touches no shared quota, and is an excellent way to check that retry code compiles, honours retry-after, and gives up where you think it does. What it cannot produce is the platform's own behaviour at the ceiling: the retry-after values it chooses under load, whether it degrades before it rejects, how long the bucket takes to refill, and how a fleet of your own workers interacts with that refill. Run the proxy version first, because it is fast and catches the obvious defects for free; run the real one because the recovery shape is the part that cannot be synthesised.

Containment is what makes a saturation test a piece of engineering rather than an outage with a hypothesis attached. The five controls are cheap individually — a project id, a region, a window read off a usage graph, a top step in a list, a URL, and an email — and the reason to build all of them before the first run is that the failure they prevent is not yours to see. Your own dashboards stay green throughout.

With the containment in place, the test itself is short, and the trace it produces answers a specific question: which error arrives, what the SDK does with it on your behalf, and why the service stays broken for several minutes after the load stops.