Escalated requests pay for both calls in sequence

The proposal had one line of latency analysis in it: the small model returns in about 400ms against 1.2 seconds for the large one, so 85% of requests get three times faster. Every number in that sentence was correct. The p95 on the dashboard went from 1.9 seconds to 3.1 seconds in the week after rollout, and stayed there.

Both facts are true at once, and reconciling them is the arithmetic of this chapter. It is the same arithmetic as a cache: adding a cache in front of a database makes the median dramatically faster and does nothing whatsoever for the miss path, and if the misses now have to pay a cache lookup before they pay the database, the tail gets slightly worse rather than better. A cascade is a cache with a 20x price ratio instead of a 200x one, and with a hit rate you are choosing rather than observing.

Where the percentile actually lands#

Take the mixture literally. Eighty-five percent of requests complete after one cheap call. Fifteen percent complete after a cheap call, a decision, and a strong call, in sequence, because the strong call cannot start until the cheap answer exists — that is what makes it a cascade rather than a race.

Sorting all requests by latency puts the entire cheap population below the entire escalated population, near enough, so the boundary between the two sits at the 85th percentile. Anything you ask for above that lands inside the escalated group, and the question becomes where inside it. The relationship is one line of algebra: for an overall percentile P above 1 − r, the corresponding percentile within the escalation path is

q = (P − (1 − r)) / r

At r = 0.15, the overall p95 corresponds to q = (0.95 − 0.85) / 0.15 = 0.667 — the 67th percentile of the escalation path. The overall p99 corresponds to q = 0.933, the escalation path's p93. Neither of those is "400ms plus 1,200ms". Adding two medians gives 1.6 seconds, which is a plausible first estimate and is also a statement about the median escalated request, sitting somewhere around the overall 92nd percentile. The p95 is further out than that by construction.

Then there is the correlation, which is what took the dashboard to 3.1 seconds. Requests escalate because they are hard, and hard requests in most workloads are also long: a 4,000-token support thread with three attachments and a nested billing question rather than a two-line password reset. Long inputs make the cheap model slower, because prefill — the pass over the input tokens that happens before the first output token can appear — scales with input length, and long answers make it slower still, because decode emits one token at a time. Both effects apply to the strong model as well. The escalated population is therefore not a random 15% sample of the latency distribution; it is drawn from the slow end of both distributions at once.

Measured over one week of production traffic, the four populations looked like this:

The escalated request's end-to-end latency is the sum of the second and fourth rows plus the escalation decision itself, which gives a p50 near 2.55 seconds for that path and a p95 near 4.8 seconds. Feeding those into the formula, the overall p95 sits at the escalation path's p67, which measured out at 2.95 seconds, and the overall p99 at its p93, around 4.5 seconds. The always-strong baseline had a p95 of 2.4 seconds. The cascade made 85% of requests roughly three times faster and made the number the SLO is written against 23% worse.

Reading it off the data rather than off the medians#

The estimate is not hard to compute properly, and doing it before rollout costs nothing beyond a replay you already have. Given per-request latencies from a shadow run — the same production traffic replayed through both models offline, with none of the results served to anyone — and the escalation decision for each request, the mixture is exact:

import numpy as np

def cascade_latencies(cheap_ms, strong_ms, escalated, decide_ms=0.0):
    """Per-request end-to-end latency for a sequential cascade.

    cheap_ms, strong_ms: arrays of measured latency per request, same order
    escalated: boolean array, True where the signal rejected the candidate
    decide_ms: cost of the escalation signal itself (verifier call, judge, parse)
    """
    total = cheap_ms + decide_ms
    total[escalated] += strong_ms[escalated]
    return total

def report(total, always_strong):
    for p in (50, 90, 95, 99):
        print(f"p{p:<3} cascade={np.percentile(total, p):7.0f}ms "
              f"always-strong={np.percentile(always_strong, p):7.0f}ms")

# one week of replayed traffic, n = 41,208
total = cascade_latencies(cheap_ms, strong_ms, escalated, decide_ms=140.0)
report(total, strong_ms)

p50  cascade=    418ms always-strong=   1150ms
p90  cascade=   2310ms always-strong=   1980ms
p95  cascade=   2951ms always-strong=   2412ms
p99  cascade=   4487ms always-strong=   3910ms

The crossover in this data is between p85 and p90, and its position is set entirely by the escalation rate. That is the property worth internalising: the cascade's tail percentiles are a function of the escalation rate, so they move when the escalation rate moves, with no component getting any slower. A prompt revision that makes the cheap model hedge more, a threshold nudged from 0.7 to 0.75, a shift in traffic mix after a marketing campaign — any of these can take escalation from 12% to 22%, and the p95 will move from the escalation path's p58 to its p77 without a single model changing behaviour. Chapter 15.3 established how to write objectives against indicators; the consequence here is that escalation rate belongs among your indicators rather than in a cost report, because it is the input that moves the latency objective.

The same table also shows why the objective cannot sensibly be written on the mixture alone. An SLO of "p95 under 2.5 seconds" on aggregate traffic is satisfied at 10% escalation and violated at 15%, and when it is violated the on-call engineer sees a latency page for a system in which nothing is slow. The useful form is two indicators with separate objectives — one for the direct path, one for the escalation path — plus an objective on the escalation rate itself, since it is the term that couples them. Then a page says which of the three moved.

What streaming does to the comparison#

There is a case where the cascade is slower for 100% of traffic, and it is the common case for chat surfaces.

If the response is streamed, the user's experience is dominated by time to first token — TTFT, the delay between the request arriving and the first word appearing on screen — rather than by completion time. The strong model in the table above has a TTFT of about 300 milliseconds; it then streams for another 850 milliseconds while the user reads. The cheap model has a TTFT of about 90 milliseconds. On those numbers the cascade should be a clear win, and it is not, because the cascade cannot show the user any of the cheap model's tokens until it has decided whether to keep them. Streaming a candidate you might discard means either retracting visible text or leaving a wrong answer on screen, so the decision has to happen on the complete candidate.

The effect is that a cascade converts a streaming interface into a blocking one on the cheap path. Perceived first-token latency becomes the cheap model's completion time plus the signal's cost — around 520 milliseconds here — against 300 milliseconds for always-strong, and on the 15% that escalate it becomes 520 milliseconds plus the strong model's TTFT, around 820 milliseconds. Every user waits longer to see the first word, and the money saved is real. Whether that trade is acceptable is a product decision, and it is one that the "85% of requests get three times faster" framing hides completely, because completion time was never the metric that surface was judged on.

Batch and asynchronous workloads have the opposite profile, which is why cascades tend to land well on document pipelines, offline enrichment, evaluation harnesses and anything behind a queue. When the consumer of the result is a job rather than a person, a p95 of 4.5 seconds against 2.4 is a throughput question rather than an experience question, and throughput improves because 85% of the work now occupies a cheap model for 380 milliseconds instead of a strong one for 1.15 seconds.

The tail moves with load as well as with the mix#

The percentile arithmetic above treats each request as independent, which holds while the system is comfortably under its capacity limits and stops holding at exactly the moment you would most like it to.

Two mechanisms couple escalation to load. The first is capacity: a cascade at 15% escalation consumes strong-model capacity equal to 15% of its request rate, so a traffic spike that doubles requests also doubles strong-model demand, and provider rate limits are usually expressed in tokens per minute against the strong model specifically. Hitting that limit produces 429s — the HTTP status a provider returns when you have asked for more than your allowance in the current window — and a 429 on the escalation path is a request that has already spent 620 milliseconds on a cheap call and now has to wait for a retry window. The second is composition: traffic spikes are rarely uniform. An incident that drives a support surge sends a burst of unusually long, unusually complicated tickets, which raises the escalation rate at the same time as it raises the request rate, so the strong model's share of demand grows faster than traffic does.

Measured during one such event, request volume rose 2.4x over fifteen minutes while escalation rate rose from 12% to 19%, which put strong-model token demand at 3.8x its baseline against a limit sized at 3x. The resulting queueing added 1.1 seconds to the escalation path's p50, and because the overall p95 already sat on that path, aggregate p95 went from 2.95 seconds to 4.3 seconds during the window. Nothing about the models changed. The capacity headroom to size for is the escalation path's peak, computed at the escalation rate you see under stress rather than the one you see on an average Tuesday, and that number is typically 40–60% above the naive product of peak traffic and mean escalation rate.

Bounding the path you decided to add#

Since the escalation path is where the tail lives, it is the path that needs limits. Three of them do most of the work.

A deadline on the cheap call, set low. The cheap model is on the critical path for every request including the ones that will escalate, so a cheap call that hangs for 8 seconds has already lost the request before the strong model starts. Setting its timeout near its own p99 — 900 milliseconds against a p95 of 710 — and treating expiry as an escalation keeps the worst case additive rather than open-ended.

A deadline on the escalation decision. If the signal is a verifier model, a judge, or anything else that makes a network call, it has its own tail, and it is unbounded by default. The 140 milliseconds in the code above is a budget, not an observation.

A single escalation. Two hops is a bounded path; a ladder of four models with a scored stopping rule at each step, of the kind the original FrugalGPT experiments used, is a research design whose worst case is the sum of every model in the chain. If you want more than one escalation step, the total budget has to be stated up front and enforced by a wall clock rather than by a hop count.

There is one design that escapes the sequential penalty, at the cost of the saving it exists to produce: speculative execution, where both models start at once and the strong response is cancelled when the cheap answer passes. That gives the strong model's latency profile on escalated requests with no stacking, and it pays for the strong model's prefill on every request whether or not it is used. On the numbers from chapter 1 — a 20x price ratio between the two models — speculating on 100% of traffic to save on 85% of it usually costs more than it saves, so it appears mainly on latency-critical surfaces where the budget makes a sequential cascade impossible and the saving is a secondary goal.

All of which assumes the escalation decision is at least deciding the right things, so that the 15% who wait are the 15% who needed to. The system in the opening story escalated on a number the cheap model wrote about its own answer, which is where the trouble actually was.