End-to-end success under a deadline

An offline evaluation harness measures one quantity: the share of a held-out set a design gets right, given as long as it needs and as many retries as it wants. A production service is judged on a different one, which is the share of requests that come back with a valid answer, inside the caller's timeout, that is also correct. That is the end-to-end success rate, and everything the harness reports is one factor in it, sitting alongside how often a request breaches its deadline, how often one of the calls it depends on fails, and what the design returns when it has run out of time to do the thing it was built to do.

The gap between the two quantities is widest for a design that fans out — issues N model calls concurrently for one request and aggregates the answers. Five samples of one prompt plus a judge call is six network calls where a single call to a stronger model is one, and each of the extra five is another chance to be slow and another chance to fail. The measurements below come from an underwriting service that classifies applications, scored on a 1,900-case held-out set and deployed behind a gateway with a 2.5-second timeout, where p95 means the latency ninety-five per cent of requests come in under, and an SLO is the latency or availability target a team publishes and is held to.

The p95 of a fan-out is a per-call p99#

Concurrent calls cost the maximum of N latencies rather than the sum, and the maximum grows slowly — a fact usually stated as reassurance and worth stating precisely instead, because the precise version is what decides this comparison. If the N calls are independent draws from a latency distribution whose cumulative distribution function F(t) gives the share of single calls that finish by time t, then the fan-out finishes by time t only when all of them do, so P(max ≤ t) = F(t)^N. The q-th quantile of the fan-out — the time that a share q of fan-outs comes in under — is therefore the per-call quantile at q^(1/N).

Both halves of that deserve their working. The exponent is there because "all of them do" is a conjunction: the fan-out is under t only if the first call is under t and the second is and the third, all the way to the N-th. When the draws are independent, the probability of a conjunction is the product of the probabilities, and here all N probabilities are the same number F(t), so the product is that number multiplied by itself N times. The consequence follows immediately from what kind of number F(t) is. It is a probability, so it is at most one, and multiplying a number below one by itself makes it smaller. A fan-out is therefore strictly slower than a single call at every quantile, and it gets slower as N grows, with no exception and no tuning available.

Turning that formula into the quantile statement is one rearrangement. The question a deadline asks is the reverse of the one the formula answers: not "what share of fan-outs beat this time", but "what time do a share q of fan-outs beat". So set the left-hand side to the share you want, F(t)^N = q, and solve for F(t) by taking the N-th root of both sides, which gives F(t) = q^(1/N). Read in words, that says the per-call reliability you need is the fan-out reliability you want, rooted N times — and rooting a number below one pushes it towards one. Taking the fifth root of the share a p95 target asks for lands on the 99.0 in the p95 row of the table below, which is the per-call percentile five concurrent calls must hit for their slowest to come in under a p95.

A median request now waits on what used to be a rare call, and a p95 request waits on the per-call p99, which is the part of the distribution nobody profiles and nobody writes an SLO against. The judge cannot start until the slowest candidate arrives, so its own latency stacks on the end: 0.6 seconds at p95 for a call with a long prompt and a 40-token output, bringing the design to about 2.6 seconds.

The belief that gets people into this is not that the maximum is the sum. Everyone knows concurrent calls overlap. The belief is that because the maximum grows slowly, a fan-out costs roughly what one call costs, and the table's own top row is what makes that tempting: a median fan-out lands at 1.24 seconds, which is an unremarkable number for a single call to this model. Read the middle column of that row and the comfort disappears. 1.24 seconds is the time a single call beats 87.1% of the time, so the typical fan-out is already waiting on a call that would sit in the slowest eighth of ordinary calls. The honest restatement is that issuing five calls is not doing one thing once. It is requiring five separate things all to go right, and a deadline is judged on the occasions when one of them does not.

The judge's 0.6 seconds is added rather than maximised for a structural reason worth being explicit about, because the same design has both kinds of composition in it. The five generation calls overlap because none of them needs anything from the others, so their cost is the slowest of the five. The judge needs every candidate to exist before it can rank them, so it cannot start until the fan-out has finished, and two stages in series cost their times added. One design, one addition and one maximum, decided entirely by which step needs the output of which. Adding the two p95 figures gives an estimate rather than a measurement, since a request sitting at the p95 of the whole is not required to sit at the p95 of both stages at once, and the figure it produces should be treated as a planning number until a real run replaces it.

The independence assumption behind P(max ≤ t) = F(t)^N is generous, and the measured numbers were worse than it predicted. Five concurrent calls go to the same provider, in the same region, through the same connection pool, and they queue behind the same capacity, so their latencies are positively correlated: when the provider is having a slow minute, all five are slow together rather than one of them. Under production load the ensemble's p95 measured 2.9 seconds against the 2.6 the fixture harness reported, and the gap widened at peak hours. The correlation that hurts a vote's accuracy has an operational twin that hurts its latency.

Six calls, six ways to not get an answer#

Per-call failures on this service ran at 0.4%, counting 5xx responses, connection resets, provider-side timeouts, and responses that failed schema validation. One call fails on 0.4% of requests. Six calls produce at least one failure on 1 - 0.996^6, which is 2.4% of requests, and the design needs an answer for every one of those cases before it goes near production.

Each piece of that expression is doing an identifiable job. The 0.996 is one minus the 0.4%, which is the chance a single call comes back at all. Raising it to the sixth power is the same conjunction as before: all six calls succeed only if the first does and the second does and so on, and independent probabilities compose by multiplication, so the chance of a clean run is that number multiplied by itself once per call. Subtracting from one is what turns "everything worked" into "something did not", and the subtraction is there to avoid the alternative, which is adding up the case of exactly one failure, plus exactly two, plus exactly three, and so on to six. The complement collapses all of those into one operation.

There is a shortcut that works here and is worth knowing along with the condition under which it stops working. Counting 0.4% once per call, six times over, comes to 2.4%, which is the same answer to the precision quoted. It agrees with the exact expression because the cases that plain counting double-counts are the ones where two calls fail at once, and at a rate this low those are rare enough to disappear below the second decimal place. Push the per-call rate up towards a few per cent and the shortcut starts overstating, because double failures stop being negligible. The useful reading of it is that at small per-call failure rates the risk of a request failing grows roughly in proportion to the number of calls it makes, which is the sentence that should be in a design review.

The redundancy in this design is also unevenly distributed, and the code below shows where. Losing one or two of the five generation calls is survivable, because that code proceeds on any three surviving candidates. The judge call has no redundancy at all: it happens once, it happens last, and it happens with the least remaining time. A failure there does not lose a fraction of the evidence, it loses the step that decides, which is why the fallback path has to exist rather than being a defensive nicety.

JUDGE_P95_S = 0.6

async def ensemble_answer(req, deadline_s: float = 2.5):
    """Deadline-aware fan-out with an explicit degradation path."""
    started = monotonic()
    tasks = [call(CHEAP, req, temperature=0.7, seed=i) for i in range(5)]
    done, pending = await wait(tasks, timeout=deadline_s - JUDGE_P95_S)
    for t in pending:
        t.cancel()                                   # no retry: no time for one
    cands = [t.result() for t in done if not t.exception()]

    if len(cands) < 3:
        return await call(STRONG, req)               # fall back, may still breach
    remaining = deadline_s - (monotonic() - started)
    if remaining < JUDGE_P95_S:
        return majority_or_longest(cands)            # skip the judge
    return await call(JUDGE, req, cands, timeout=remaining)

Three separate policies live in those fifteen lines, and each is a decision the offline score never had to make. Cancelling stragglers means some requests are judged over three or four candidates rather than five, which is the invisible-N problem: the request is aggregated at a smaller N and nothing in the metrics says so unless the count is logged. Skipping the judge under time pressure means the design silently degrades into a majority vote, which has nothing to count over free text, so the fallback here returns the longest surviving candidate — a heuristic that rewards padding. Retrying is the policy that looks most natural and fits worst, because a retry costs a full call inside a budget that is already 2.6 seconds against a 2.5-second deadline.

The last two lines of that diagram are the whole problem with judging a design on its offline score. The work completed, the answer was correct, and the person waiting saw an error page, because the gateway stopped waiting 400 milliseconds earlier. A breach is not a slightly worse answer; it is no answer, followed by a retry from the client, which arrives as a second full request and consumes another six calls.

Success measured end to end#

Everything above is one column of a single number, which is the fraction of requests that return a valid answer, within the caller's deadline, that is also correct.

Two of those rows check each other, and running the check is the quickest way to tell whether a latency table has been assembled honestly. A p95 is the time that ninety-five requests in a hundred come in under. The single strong call has a p95 of 1.9 seconds under production load, which sits below the 2.5-second deadline, so fewer than five requests in a hundred can possibly breach — and 1.6% duly does. The ensemble's p95 is 2.9 seconds, which sits above the deadline, so more than five in a hundred must breach, and 6.8% duly does. A p95 above the deadline is not a warning sign that needs interpreting. It is an arithmetic statement that the SLO is already missed.

The last row is a product of two of the rows above it, and knowing which two matters more than the multiplication. End-to-end success is the effective accuracy multiplied by the share of requests that arrive inside the deadline: for the single call, 87.4% correct across the requests that were not among the 1.6% that breached, which lands on 85.9%; for the ensemble, 89.0% correct across the requests that were not among the 6.8%, which lands on 82.6%. The call-failure row is deliberately not multiplied in on top of that. A request that lost a call either recovered inside the deadline, in which case its outcome is already inside the effective-accuracy row, or it did not, in which case it is already inside the breach row. Charging it twice would understate both designs, and the design making six calls would be understated more.

The effective-accuracy row is itself a step worth not skipping, because it is where the offline number stops being available. The ensemble scored 89.4% offline and delivers 89.0%, and the missing four tenths are the degradation policies: requests judged over three or four candidates because stragglers were cancelled, and requests that skipped the judge entirely and fell back to the longest surviving candidate. The single strong call has no equivalent step, and shows the same 87.4% in both rows, because a design with one call and no aggregation has nothing to degrade into.

Putting the two differences side by side explains the reversal without any further arithmetic. On the accuracy the caller can actually receive, the ensemble is ahead by a little over a point and a half. On the share of requests that arrive at all, it is behind by a little over five points. The second gap is larger than the first, so the product comes out the other way round, and the reversal needs no further explanation.

The design that is two points better offline is 3.3 points worse in production, and every part of the difference comes from a row the eval never had. That is not an argument that ensembles lose in general; it is an argument that they are evaluated on one axis and deployed on five. Had the same table come out with a 4-second deadline, the breach rates would have been 0.2% and 0.9%, the end-to-end numbers 87.2% and 88.6%, and the ensemble would have won.

That counterfactual is not a hopeful aside, and the mechanism behind it says exactly when an ensemble is the right design. Moving the deadline out shrinks both breach rates, but not by the same factor, because the two distributions sit differently relative to it. The single call's tail was already well inside 2.5 seconds, so there was little left for a longer deadline to recover. The ensemble's tail was straddling it, so most of its breaches were requests missing by a fraction of a second, and pushing the line out clears nearly all of them at once. Once the deliverability gap between the two designs falls below the accuracy gap, the product reverses, and the ensemble's two offline points start being worth what they say they are. The condition to state in a design review is that a fan-out is worth its extra accuracy only when the deadline sits comfortably beyond the fan-out's tail — not beyond its median, and not beyond its p95, which is where the deadline was.

Where the two points came from, by segment#

There is one more column the table cannot show, which is whether the offline gap survives contact with traffic the eval set does not contain. A judge drawn from the same family as the generator selects partly for fluency, so its advantage is a function of how much the inputs resemble the ones its rubric was tuned against. Splitting the offline gap by segment showed exactly that pattern.

The new channel submitted longer, more discursive applications, which produced longer candidates, which the judge preferred at a rate unrelated to whether they were right. Eight percent of traffic is small enough to lose in an aggregate and large enough to be the segment an account team asks about.

That sentence has three causal links folded into two commas, and each one is checkable on its own. A longer, more discursive application is a longer prompt with more material in it, and a model given more material to work with returns more of it, so the candidates on this segment are longer than the candidates elsewhere. A judge drawn from the same family scores partly on fluency and length, because those are properties its own training rewarded, so among five longer candidates it prefers the longest rather than the most accurate. Length on this segment is a property of the input channel and not of whether any particular candidate got the answer right. A selector driven by something uncorrelated with correctness is choosing close to at random among its candidates, which is why the ensemble's advantage here is not merely smaller but sits on the wrong side of zero.

The aggregate hid it by doing exactly what an average does. Weighting each production segment's gap by its share of traffic gives a little under two points, against the eval set's +2.0, and the two figures agree closely enough that nothing looked wrong. What the eval set could not report is that its own +2.0 was a weighted average over a traffic mix that contained no new-broker channel at all, because the channel postdated it. The number was never incorrect. It was answering a question about last year's traffic, and it was read as an answer about this quarter's.

Three strong calls, and the deadline again#

The obvious synthesis is the strong model at N=3, which on this task scored 88.6% offline and is the highest number anyone measured. Three things stand against it, and only one is cost.

The cost is straightforward: three strong calls is $0.0438 per request, or $3,723 a day against roughly $1,250, so the proposal adds about $2,470 a day and $900,000 a year. The marginal-cost arithmetic applies unchanged — 1.2 points for $2,470 a day is $2,058 per point per day, and whether that clears the bar depends entirely on what an error costs once it leaves this service, which for underwriting is genuinely expensive, since a wrong decline generates a complaint and a wrong approve generates a loss.

Each figure in that paragraph is derived from the one before it, and walking the chain is what makes the last number arguable rather than merely large. Three strong calls cost three times the $0.0146 that one costs, which is the $0.0438. Multiplying by the 85,000 applications a day gives the $3,723, and doing the same multiplication on the single call gives the roughly $1,250 it is being compared against. The difference between those two daily bills is the about $2,470, and a year of that is the $900,000. The last step is the one that turns money into a decision: dividing the $2,470 by the 1.2 points of offline accuracy it buys gives $2,058, and the unit is dollars per point per day — the daily price of holding one extra point. That unit exists so it can be set against the daily cost of the errors that point removes, which is a number the underwriting business can supply and the engineering team cannot.

The second is that N buys less on a stronger model, which is predictable rather than something to be discovered. A stronger model has a sharper answer distribution, so less of its error is dispersion that sampling can average away and more is displacement that no N will touch. On this task the cheap model gained 4.1 points from N=1 to N=3 and the strong model gained 1.2, the same effect that moves the plateau up and the optimal N down after a model upgrade.

Naming the two kinds of error makes that prediction checkable rather than a rule of thumb. Voting over N samples can only repair an error where the model's distribution already contains the right answer with real weight and a single draw happened to miss it; take more draws and the right answer wins the count. That is dispersion, and it is the only failure sampling addresses. Where the distribution sits on a wrong answer instead, every draw returns that answer, the vote is unanimous and nothing is repaired. That is displacement. Sharpening a distribution — which is most of what makes a stronger model stronger — removes dispersion specifically, since a sharper distribution is one where repeat draws land in the same place. So the errors a stronger model has left over are disproportionately displacement, and displacement is the kind N cannot reach. Gaining 4.1 points on the cheap model and 1.2 on the strong one is that composition shifting, not two unrelated measurements.

The third is the deadline again. Three strong calls have a p95 at the per-call 98.3rd percentile, which measured 2.3 seconds, giving a breach rate of 3.9% and an end-to-end success rate of 85.1% — below the single strong call it was supposed to improve on. That 98.3 is the same root as before with a smaller N: three calls need a less extreme per-call percentile than five do, which is why the requirement eased from 99.0. What undoes the saving is that the strong model's own per-call latencies are longer, and 2.3 seconds is where the two effects settle. The consistency check from the table above still applies and still passes: 2.3 seconds sits below the 2.5-second deadline, so the breach rate has to come in under five in a hundred, and 3.9% does. Spending $900,000 a year to lose 0.8 points of end-to-end success is the kind of outcome that only appears when the benchmark runs under the caller's constraints. Asking why not both is a reasonable question with a specific meaning, which is a request to raise the budget, and it deserves a straight answer with the marginal cost per point attached and the deadline arithmetic beside it rather than a defence of the current design.

Running the comparison on your own service#

Capture a week of real requests, including the segments that postdate your eval set, and run both designs against them through the actual gateway, with the actual timeout, the actual retry policy and the actual concurrency limits, on the same afternoon so that provider conditions are shared. Report end-to-end success rate, p95 and p99 under load, cost per thousand requests counting the judge and every retried or discarded sample, and the offline accuracy gap broken down by segment. Any design whose advantage lives entirely in the first of those numbers has not been compared yet.

A design that is right on the eval and late at the gateway has not improved anything. End-to-end success is the number that composes accuracy, breach rate, partial failure and whatever the design does when it degrades into the single quantity a caller experiences, and the only way to know it before shipping is to make the deadline part of the benchmark. The order in which to spend effort has been consistent across this topic and is worth carrying out of it. Read a hundred failures before adding a call, since a two-line prompt edit moved one slice 23 points for nothing. Check that the samples decorrelate before paying for them. Measure the accuracy-versus-N curve with error bars and stop where it flattens. Price a point and price an error before approving the spend. Prefer a check that consults the record over a model grading its own output. And run the final comparison under the constraints the service actually operates under.