Latency and cost of constrained decoding

The decision to keep a strict schema on is usually argued between two teams holding numbers that are both correct and not commensurable. One has a token bill that went down; the other has a tail latency that went up. Neither figure contains the information needed to trade it against the other, and the argument runs until somebody supplies a third number that neither team owns.

This chapter is about what actually moves when a constraint goes on, why each of the three movements has a different cause, how to convert the token column into money without overstating it, and what the deciding measurement is. The last part matters most: latency has no price on its own, and the number that gives it one is a timeout sitting in a config file three services away.

Three numbers, three separate mechanisms#

Output tokens fall because the model is no longer emitting anything except the object. A prompt-only arm averaging 340 output tokens included, on a typical response, four tokens of code fence, eleven tokens of "Here is the extracted data in the requested format:", a trailing sentence offering to clarify, and — on the roughly one call in nine where the model restated part of the input before answering — another forty or so. None of those are reachable states in a grammar whose first legal token is { and whose accepting state is the closing brace, so they are gone by construction rather than reduced by instruction. The saving is largest on small objects, where the wrapper was a large fraction of the response, and close to nil on a schema that generates a 900-token array.

The median holds because the mask is cheap relative to the thing it modifies. Producing one output token requires a full forward pass through the model, which on a mid-sized hosted model at this concurrency lands somewhere between 18 and 24 ms. Applying a compiled grammar means looking up the current parser state, intersecting its allowed set with the vocabulary, and zeroing the rest of the logit vector, which measures at roughly 0.2 to 0.6 ms per token on a warm cache. That is 1 to 3% added to each token, against 18% fewer tokens to produce, so the two effects roughly cancel in the middle of the distribution and the median moves by single-digit milliseconds in whichever direction the traffic mix happens to push it.

That cancellation needs one more term before it adds up, and leaving the term out is what makes the row look suspicious. If a call's wall time were nothing but decoding, eighteen per cent fewer tokens at one to three per cent more each would land as a double-digit percentage drop at the median rather than as four milliseconds. It does not, because a call is not all decoding. The network round trip happens once. The request waits in the provider's queue once. And the model has to read the input before it writes anything, which is a forward pass over the 1,150 input tokens that the constraint does not touch, since the constraint governs only what is emitted. The token saving therefore applies to one term of a sum whose other terms are unchanged, and it is diluted on its way to the number anybody reads off a dashboard.

The tail rises because the per-token cost is not the only cost, and the other costs are all bursty. Compiling a JSON Schema into a grammar happens the first time the provider sees a given schema, and it is not free: a flat six-field object compiles in tens of milliseconds, a 200-member enum takes longer because every member becomes a branch in the automaton, and a schema carrying pattern clauses is the expensive case because a regular expression has to be turned into a state machine and intersected with the tokenizer's vocabulary. On this pipeline the cold-compile cost measured between 40 ms and 310 ms depending on the schema. That cost is amortised across every request that reuses the compiled artefact, which is nearly all of them — and the exceptions land in the tail. A deploy that bumps schema_version invalidates the cache for every provider instance at once. So does generating a 90-member date enum per request to make a past date unreachable, which produces a cache miss on every single call and is the reason that particular schema's p99 sat 280 ms above its siblings.

The mechanism there inverts the usual reasoning about caches and is worth spelling out. A compiled grammar is stored against the schema that produced it, so the cache key is the schema document itself. A schema that is byte-identical on every request compiles once and is looked up thereafter, which is why the compile cost is normally invisible: one unlucky request pays it and nobody else does. A schema rebuilt from a database per request is a different document every time, so it never matches a key, and a cost designed to be spread across millions of calls is instead added to each of them. Nothing about the compilation got slower. What changed is how many requests share one, which went from all of them to one.

The p99.9 row is the one usually missing from both sides of the argument, and it is where the compile misses and the long completions stack on top of each other.

What 18% fewer output tokens is actually worth#

Converting the token column into money is arithmetic worth writing down explicitly, because "18%" invites people to apply it to the whole bill rather than to the part of the bill it touches. At 2 million calls a day with an average input of 1,150 tokens, input spend is unchanged at 2.3 billion tokens a day; at $2.50 per million that is $5,750 daily, and the constraint does not move a cent of it. Output falls from 680 million to 558 million tokens a day, and at $10 per million that is $6,800 against $5,580, so the saving is $1,220 a day, or about $445,000 a year. The 3% of calls that used to be retried through the repair path add 60,000 full calls a day at roughly $0.0063 each — input plus output, since a repair call resends the original context — which is another $377 a day and about $137,000 a year. The combined figure is close to $583,000 annually, against a bill running near $4.6 million.

Every step in that paragraph is one of two operations, and knowing which is which is what stops the figure being misapplied later. The first is calls multiplied by tokens per call, which turns a rate into a daily volume: two million calls at eleven hundred and fifty input tokens each is the 2.3 billion, and the same two million at 340 and then at 279 output tokens gives the 680 million and the 558 million. The second is a volume divided by a million and multiplied by a unit price, because provider pricing is quoted per million tokens; six hundred and eighty million output tokens at ten dollars a million is the $6,800, and five hundred and fifty-eight million is the $5,580. The saving is the difference between those two, and the annual figure is that difference taken across a year of days.

The input side is excluded from the subtraction on purpose, and it is the part the percentage gets misapplied to. The constraint acts on what the model emits and has no effect on what the model reads, so the 2.3 billion input tokens and their $5,750 are identical in both arms and cancel out of the difference entirely. Eighteen per cent is a saving on the output line alone, and on this workload the output line is the smaller of the two — which is why a number that sounds like a fifth of the bill turns out, once the repair calls are added and the total is set beside the $4.6 million it comes out of, to be closer to a tenth of it.

The $0.0063 for a repair call is the one figure in that paragraph that is a sum rather than a product, and opening it explains why a retry is expensive rather than incremental. A repair call resends the original context, so it pays the full input price — eleven hundred and fifty tokens at two dollars fifty a million, a little under three tenths of a cent — and then generates a complete fresh response, which at the prompt-only arm's 340 output tokens and ten dollars a million is a little over a third of a cent. Those two added together are the $0.0063, and the larger half is the output. Sixty thousand such calls a day is the $377.

That is a real saving on a real line item, and it is the strongest argument in the room until somebody asks what the alternative costs. The comparison implicitly being made is against reverting, and reverting also restores a 3% validation failure rate, which is not a free 3% — those failures land disproportionately on the hardest inputs, and scoring them as wrong rather than dropping them is what turns an apparent quality regression into a gain on a fixed denominator. Any argument about cost that leaves the quality column out has already conceded the wrong thing.

The one number that settles it#

Latency has no price on its own. A 120 ms increase at p99 is worth nothing at all if every consumer of a pipeline is a batch job with a two-minute budget, and it is worth an outage if something in the call chain gives up at a threshold the new distribution now crosses. The measurement that decides which world you are in is the tightest deadline anywhere downstream of the call, and the fraction of calls that exceeded it before and after.

Two readings come off that chain and neither of them is a percentile. The first is the headroom at the hop that binds. The orchestrator allows the enrichment step 1,600 ms and the model call's p99 is 1,540 ms, so one call in a hundred is finishing with about sixty milliseconds to spare. A budget with sixty milliseconds of slack at p99 is not a budget with room in it. It is a budget that will be crossed by any change at all, including changes the provider makes without telling anyone. The second reading is that the retry does not fit inside the hops above it: two attempts at 1,600 ms come to more than three seconds, which is more than the gateway's 2,500 ms and more than the client's 3,000 ms, so the retry the orchestrator is relying on to absorb its timeouts can itself be cut off from above. A chain of deadlines has to be read whole for exactly that reason. Each value was chosen to look reasonable next to the hop below it, and nobody added them up.

Every timed-out call is billed in full, because the provider generated the tokens whether or not the caller waited for them, and every one triggers a retry that is billed again. The retries do not arrive uniformly either — they cluster at peak, because that is when the tail is fattest — so they add load precisely when the provider queue is deepest, which pushes p99 up further and produces more timeouts. None of that appears in a percentile table, and all of it follows from one number that lives in somebody else's config.

Finding that deadline is usually the hard part, and it is an organisational problem rather than a technical one. The budget is spread across a chain of configs owned by different teams, each of which set its value against a latency distribution that no longer exists, and the tightest one is rarely the innermost. Read the chain from the client inward, write each timeout down next to who owns it, and look for the hop where the remaining budget is smallest relative to what the call actually takes. Where a hop has no explicit timeout, the effective deadline is whatever its own caller allows, which means an unset value is not a wide budget — it is a budget somebody else chose without being asked.

The measurement itself comes out of the same per-call records a paired eval harness is already collecting, with a deadline as a parameter rather than a percentile as the output:

def report(outcomes: list[Outcome], deadline_ms: float,
           in_price=2.50e-6, out_price=1.00e-5, in_tokens=1150) -> dict:
    lat = sorted(o.latency_ms for o in outcomes)
    n = len(lat)
    pct = lambda p: lat[min(n - 1, int(p * n))]
    over = sum(1 for x in lat if x > deadline_ms)
    return {
        "n": n,
        "p50": pct(0.50), "p99": pct(0.99), "p999": pct(0.999),
        "mean_out_tokens": sum(o.output_tokens for o in outcomes) / n,
        "invalid_rate": sum(1 for o in outcomes if o.failure) / n,
        "over_deadline_rate": over / n,
        "cost_per_call": in_tokens * in_price
                       + (sum(o.output_tokens for o in outcomes) / n) * out_price,
    }

for arm in ("prompt_only", "strict"):
    print(arm, report(by_arm[arm], deadline_ms=1600))

Printing over_deadline_rate next to cost_per_call for both arms puts the two teams' numbers on one row, which is the minimum needed for a decision. The deadline belongs in the config beside the timeout it mirrors, so that changing the timeout changes the report.

Buying the tail back without giving up the constraint#

Reverting is the expensive way to fix a tail regression, because it gives back the quality and reliability gains to solve a latency problem that has cheaper solutions. The levers below were measured on this pipeline over a week each, against the 1,540 ms p99 baseline.

The first four together took p99 to 1,285 ms, below the pre-constraint baseline, while keeping strict mode on everywhere the provider pair enforces it. Most of that is cache behaviour rather than anything intrinsic to grammar-constrained decoding, which is the usual finding when a tail regression is investigated rather than argued about.

One row of that table has a price that does not follow from its call count, and it is the hedge. Five per cent more calls on two million a day is a hundred thousand extra requests, and a hundred thousand full calls would come to roughly twice the $290 quoted. The figure is lower because a hedged request is abandoned the moment the first response lands, so it pays for the input it was handed and generates very little before being dropped — and a hundred thousand calls at the input price alone lands close to the number in the table. That is worth knowing before pricing a hedge on your own service, because the arithmetic depends entirely on the second call being cancelled rather than completed. A client that fires the hedge and then waits for both responses is paying full price twice and has bought nothing.

What one pipeline's numbers do not generalise to#

These numbers are a measurement of one pipeline, one provider, one week, and a specific set of schemas, and they do not generalise into a claim about constrained decoding. The per-token mask cost depends on the engine, and the engineering measurements in JSONSchemaBench show the spread across implementations is large enough that a figure from one is not evidence about another. The compile cost depends on your schema's shape far more than on its size. Tail latency at a hosted provider also carries co-tenancy noise — the variance introduced by other customers' traffic sharing the same hardware — which you do not control, so a p99 comparison taken from two different weeks is measuring the provider's traffic as much as your change; both arms have to run interleaved over the same window on the same inputs, which is what a paired harness makes possible.

Falling token counts are also not evidence of quality parity, and it is tempting to read them that way because they move in the direction everyone wants. The schema-ordering fix that puts an evidence and reasoning field before an enum deliberately spends about 60 extra output tokens per call, which shows up in the cost column as a regression and in macro-F1 as a 2.4-point recovery. A pipeline optimised on the token number alone will delete those fields at the next cost review and lose the accuracy quietly, which is why the three columns belong in one report rather than on three dashboards owned by three teams.

The general rule is that a constraint's cost is not one number and cannot be settled by one team. Tokens, tail latency and quality move for separate reasons, at separate magnitudes, and only become a single decision when a deadline somewhere downstream converts milliseconds into failures that can be counted alongside the others.

What this topic leaves you with is a default and a routine. The default is to constrain generation wherever the provider-and-model pair is verified to enforce it, keep the schema export honest and tested, validate every response in your own process anyway, encode absence so the model never has to invent it, mark each response with the mode that produced it, and keep a bounded repair path for the traffic that runs unconstrained. The routine is what to run whenever the schema, the model or the provider changes: both arms over one fixed input set, quality scored with failures counted rather than dropped, and latency read against the tightest deadline downstream. Two of the three numbers in this chapter were already on someone's dashboard; the third had to be looked up in an orchestrator config, and it was the one that mattered.