Exercising the fallback path

A fallback path is the route a request takes when the provider above it in the chain is unavailable: a second vendor, or a smaller model you host yourself. Declaring one costs an adapter, a set of credentials and an entry in a configuration file, and none of those three things is exercised by being correct. This chapter is about the distance between a chain that is declared and a chain that works, which is made up entirely of properties that only exist when traffic is flowing.

The reason that distance exists at all is a distinction worth being precise about, because everything else in the chapter follows from it. Your configuration holds a set of claims about what your service will say: this hostname, this key, this model name. Whether any of those claims is accepted is decided by somebody else's system at the moment you say it. A credential is not a property of the string in your secret store; it is a row in the vendor's account database, which the vendor can revoke, rotate or migrate without your file changing a character. A quota is not a number you hold either; it is a counter maintained at their side and compared against a limit attached to your account. So the two halves of the arrangement live in different places, and nothing on your side can go stale, because nothing on your side is the part that has to still be true. That is why reading the configuration back and finding it correct tells you nothing: you are checking the half that cannot be wrong.

Two exercises close that distance, and they are not substitutes for one another. Routing a small fixed share of live traffic to the secondary permanently keeps its credentials, its quota and its output quality under continuous observation, and it tests none of the machinery that moves traffic during an incident. A game day — a scheduled exercise in which a production dependency is broken on purpose while the people who would handle the real incident watch it happen — tests the transition, the alerting and the humans.

Where you inject the failure decides what you are testing#

There are four points on the request path where you can make the primary unavailable, and they are not interchangeable, because each one leaves everything upstream of it untested.

The rule underneath that is simple enough to apply to any fault injection, not only this one. A request passes through a sequence of stages, and injecting a fault at one of them means every stage before it is skipped: the failure is manufactured downstream of them, so they never run, so nothing they might have got wrong can be observed. What the exercise proves is bounded by what sits between the injection point and the place you are watching. The consequence is that "we tested the fallback" is not a statement until somebody says where the failure was introduced, and the four points below produce four genuinely different claims.

Injecting at point 1, a flag in the gateway that makes the primary adapter throw, exercises your breaker logic, your classification table and your chain traversal. It runs in seconds, it is safe, and it proves nothing about credentials, DNS, TLS, quota or the vendor's behaviour, because no packet leaves the process. Point 2 is pinning the chain so the secondary is the head — the runtime chain configuration from chapter 3 — which sends genuine traffic to the secondary end to end and tests everything about that provider, while skipping the breaker entirely because there is nothing to fail. Point 3 blackholes egress to the primary's hostname, which produces real connection timeouts and resets and is the closest analogue to a vendor outage; it is also the one that reveals whether your timeouts are what you think they are, since a request that hangs behaves very differently from one that returns a fast 503. Point 4 is disabling the credential at the vendor, which is the only way to see how the system handles a hard 401 from the primary rather than from the secondary.

The difference between a hang and a fast 503 is the one that surprises people, and it is a difference in what the system is being asked to do rather than in how bad the outage is. A 503 is an event: it arrives, your classifier reads it, a counter moves, and the chain advances in milliseconds. A hang is the absence of an event, and the only thing that converts it into one is a timeout expiring. So an exercise that produces fast rejections runs the classification and the breaker; an exercise that produces hangs runs the timeouts first, and everything downstream of them happens only after they fire. If a per-attempt timeout is set higher than anybody remembers, that is the exercise which finds out, and it is the exercise a real vendor incident is more likely to resemble.

Most teams do point 1, call it chaos testing — the practice of injecting faults on purpose to observe how the system responds — and are surprised by point 3 during the real thing.

A small named slice, continuously#

The lowest-cost exercise that finds credential and quota problems is to route a fixed, small fraction of live traffic to the secondary permanently, using a deterministic hash so a given user's session stays consistent rather than flipping providers between turns of a conversation.

def choose_provider(request, chain, canary_pct=1.0):
    key = f"{request.session_id}:{CANARY_SALT}"
    bucket = int(hashlib.sha256(key.encode()).hexdigest()[:8], 16) % 10_000
    if bucket < canary_pct * 100 and request.route.canary_eligible:
        return chain[1], "canary"          # named, not anonymous
    return chain[0], "primary"

At 300 requests per minute, a 1% slice is 3 requests per minute — sixty minutes to the hour and twenty-four hours to the day makes that about 4,300 a day. The figure is worth computing rather than eyeballing, because the intuition that one per cent is a negligible sample is right about any one minute and wrong about a week. That volume finds an expired key within minutes of expiry, keeps the secondary's account continuously warm, and produces a rolling quality comparison against the primary on identical traffic — the same measurements as the offline provider-comparison suite from chapter 4, but on today's prompts rather than on a snapshot. The canary_eligible check keeps the slice off the routes that the degradation policy of chapter 5 marked as non-degradable, because a permanent 1% of refund decisions taken by the secondary is a policy change wearing the costume of a test.

The word named is doing real work in the second return value. A canary request that is not labelled shows up in aggregate quality metrics as unexplained variance, and someone eventually spends a week investigating a 0.3% regression that is the canary working as designed. Tag it at the point of routing, exclude it from headline dashboards, and chart it separately as the comparison it is.

A blackhole, on a calendar#

The slice does not test the transition. Nothing in it exercises the breaker, the alerting, the runbook or the humans, because there is no moment where behaviour changes. That is what a game day is for. Pick a Wednesday at 14:00, blackhole the primary for 15 minutes by dropping outbound traffic to its hostname at the network layer, so calls hang and time out the way they would in a genuine outage. Have the on-call rotation and one person from product watch it, and write down what actually happens against what you predicted.

The observations that matter are specific and mostly timing-based. How many seconds from the block to the breaker opening — predicted 20, and if it is 90 the window or minimum-volume floor is wrong for your per-pod traffic. What p99 latency does during the transition, which is the retry arithmetic from chapter 1 made visible. Whether the secondary's rate limit holds at full traffic, which is the only exercise on this list that can tell you. Whether any alert fired, and whether the person who received it could find the runbook. And how long the recovery takes when the block is lifted, since a breaker with a doubling cooldown that has reached a 300-second cap will keep serving from the secondary for five minutes after the primary is healthy.

Shadow mirroring deserves a note because it is the one people over-invest in. Sending every request to both providers and discarding the secondary's answer gives you a very large equivalence sample and doubles your inference bill to get it, which is why it is normally run for a week before a chain change rather than continuously.

Rules that make the blackhole safe enough to repeat#

A game day that goes wrong once will not be scheduled again, so the safety rails matter as much as the exercise. Write the predictions down before starting — breaker opens within 20 seconds, p99 rises to under 3.5 seconds, the tier-mix alert fires within 2 minutes — because an exercise with no prediction cannot be failed, and one that cannot be failed teaches nothing beyond a pleasant feeling. Define the abort criteria numerically in advance, in terms of things you can see on a graph within a minute: customer-visible error rate above 1% for 60 seconds, p99 above 6 seconds, or the secondary returning 429s on more than 5% of calls. Give one named person the authority to call the abort without discussion, and make the revert a single action you tested five minutes before the exercise started, which is what the named chain presets from chapter 3 are for.

The claim that an exercise without a prediction cannot be failed is about people rather than systems, and it is worth one more step because the failure is so easy to walk into. An observation made with no prior expectation arrives already accompanied by an explanation, because the person watching constructs one as they watch. A breaker that takes ninety seconds to open, observed cold, is explained on the spot as the window doing its job. The same ninety seconds set against a written prediction of twenty is a discrepancy of seventy, and a discrepancy has to be accounted for by somebody. Writing the number down first is what converts a graph you are looking at into a result, and it costs the five minutes before the exercise starts.

Then the scheduling constraints, which are mostly about who is awake. Run it in business hours with the on-call engineer, someone from product and someone who knows the vendor relationship all watching, because half of what you are testing is whether a human can tell what is happening from the tools available to them. Announce it internally, and to the vendor if you have a support contract, since a support ticket opened during your own exercise wastes everyone's time. Avoid change freezes and the day after a major release, when a second signal would be indistinguishable from your injected one.

The secondary is always cold#

There is a systematic reason the first minutes of a real failover are worse than any measurement predicts, and it is warm state. A primary that has been serving a 12,000-token system prompt and policy corpus all morning is serving it from the vendor's prompt cache, which keeps the processed form of a prefix it has seen recently so that a repeat of that prefix does not have to be recomputed, at roughly a tenth of the input token price and with time-to-first-token around 0.6 seconds instead of 2.1. The secondary starts with an empty cache on every one of those prefixes, so for the first minutes of a failover it pays full input price at full latency, and if the failover is short or your traffic is thin, its cache never warms at all.

The size of that discount follows from where the tokens are. A request on this route carries a 12,000-token system prompt and policy corpus that is identical on every call, plus whatever the user actually asked, which is small beside it. The shared prefix is therefore most of the input bill, and the cache prices that prefix at roughly a tenth. Removing nine tenths of the largest component of a bill is what takes a figure like $2.10 down to $0.95, and none of it is available on a provider that has never seen the prefix before.

That inverts a comparison people take from the eval suite. Measured serially and cold on both sides, the primary cost $2.10 per thousand requests and the secondary $0.83. In steady-state production with a warm cache the primary actually costs about $0.95 per thousand, so the failover is roughly break-even on cost while being materially worse on latency for the first stretch — the opposite of the story the suite table tells if you read it without this correction. The same asymmetry applies to connection pools, TLS session resumption, DNS caches and any provisioned-throughput arrangement that ramps with usage.

A continuous 1% slice keeps a little of this warm, which is one more argument for running one, though 3 requests per minute will not hold a prompt cache whose entries expire in five minutes. The reason is that a prompt cache is keyed on a prefix rather than on a provider: each distinct system prompt is its own entry, with its own expiry measured from the last time that particular prefix was seen. The rate that decides whether an entry survives is therefore the rate per prefix, not the rate at the vendor, and a slice of three requests a minute spread over a product's several routes gives any one of them a small fraction of those three. The aggregate looks continuous and every individual entry still expires between visits.

The practical consequence is to size your capacity headroom and your latency expectations for the cold case, and to treat the first two minutes of any failover as strictly worse than anything you have measured offline.

Capacity at full load is the largest gap#

A 1% slice tells you nothing about a rate limit, and a game day run at 14:00 on a Wednesday tests the load you have at 14:00 on a Wednesday, which for most products is well under peak. Nobody sensibly blackholes their primary during Black Friday, so the honest substitute is arithmetic: get the secondary's per-minute token limit and concurrency ceiling in writing, compare them against your measured peak with a margin, and re-check both after every traffic milestone. The 200,000-against-1,100,000 gap above was discoverable on a spreadsheet a year before it was discovered in an exercise.

The comparison really is that shallow, which is the uncomfortable part. The secondary would accept 200,000 input tokens a minute and the primary was carrying 1.1 million at peak, and 1.1 million divided by 200,000 is five and a half. So a complete failover at peak presents the secondary with five and a half times what it will accept, and the excess does not queue politely; it comes back as 429s, which your classifier reads as the secondary failing, which opens its breaker and ends the chain. Both numbers were published — one in a vendor console, one on a dashboard — and neither was ever put beside the other.

Two vendors can fail together#

Correlated failure is the second gap. Two vendors whose inference runs in the same cloud region, or who share a control-plane dependency — the management services that handle authentication, quota and routing, as against the data plane that serves inference — will fail together in the incident that matters, and no amount of exercising the path between them surfaces it. That is a question to ask during procurement and to re-ask when a vendor changes its hosting.

Fifteen minutes does not exercise hour three#

Duration is the third gap. Fifteen minutes exercises the transition; it does not exercise the daily token quota you will hit at hour three of an eight-hour outage, or the queue that has been draining slowly since the first hour, or the cache that was warm for the first ten minutes and is now cold.

There is a fourth gap, and it is the one that team hit on the game day itself. Everything worked: the breaker opened in 18 seconds, the secondary carried the load, p99 rose from 2.4 to 2.9 seconds and no user-visible error rate moved. Someone asked, near the end of the fifteen minutes, how anyone would have known this was happening if it had not been on the calendar. The error-rate graph was flat. The latency graph had moved half a second. No alert had fired, because nothing had failed.

The answers going out were being produced by a model that the eval suite said gets contractual facts wrong three times as often, and there was nothing anywhere — not on a dashboard, not in the response, not in the support tool — that said so. Exercising the path proves it works, and leaves it invisible while it carries traffic, which is the subject of the final chapter.