A day and reversible beats three weeks for the same money

Two proposals arrive in the same planning meeting for the eleven-route product from chapter 3, and both are estimated at about $4,000 a month.

The first is an output limit on assistant.turn and search.summarise, the two routes that between them carry 71.6% of the $74,114 bill. Not a bare truncation cap — the previous chapter settled that argument — but a brevity instruction plus a fixed response schema, with max_tokens parked above p99 as a rail. Output is 36% of the assistant route's spend and 33% of the search route's, so a 22% cut in generated tokens across both is $4,107 a month.

The second is moving a set of offline workloads to the provider's batch endpoint: the nightly report, the ticket backfill re-scoring, alt-text generation, UI string translation, and a nightly summarisation pass that runs inside the search route. Together they are $8,600 a month of spend at 50% off, or $4,300.

Both numbers are defensible. The proposals are not comparable, and the difference is not in the savings column.

The same money at two very different prices#

The batch figure erodes before it ships. A provider batch endpoint promises completion within a window — commonly 24 hours — with no guarantee of anything faster, so any item whose consumer has a deadline needs a synchronous fallback, and every fallback pays full price. In this product about 12% of batch items were expected to fall back, mostly the ticket backfill re-scoring when the queue was drained during business hours, which turns a 50% discount into a realised 44% and $4,300 into $3,784.

The effort figures diverge because of what each change touches. The output limit is two prompt edits and two config values behind a feature flag, deployable to 10% of traffic in an afternoon. The batch migration is a different integration shape: requests are serialised into a file and submitted as a job, results arrive asynchronously and have to be matched back to their originating rows, partial failures within a batch need their own retry path, and the fallback path is a second implementation of the same feature. Beyond the code, it changes what the product promises. The nightly report currently completes in about 40 seconds and users re-run it when a filter looks wrong; on a batch endpoint it completes some time before morning and cannot be re-run interactively at all.

Both estimates should be checked before either is scheduled, and for token-shaped levers the check is cheap. Take 5,000 requests from last week's ledger, replay them through the proposed prompt offline, and count tokens; you get the real distribution of the change rather than a guess, for an afternoon's work and about $40 of inference. The output-limit estimate here came from exactly that: replaying 5,000 assistant turns with the brevity instruction and schema produced a 22.4% reduction in generated tokens, against the 30% the proposal had assumed from a handful of manual trials. Replays cannot validate the batch estimate the same way, because what is uncertain there is not tokens but the fallback rate, and the only way to measure that is to ask the consumers of each workload what they do when a result is late.

# submission shape for a batch endpoint — note what is absent
batch = client.messages.batches.create(requests=[
    {"custom_id": row.request_id,
     "params": {"model": "claude-haiku-4-5", "max_tokens": 1024,
                "messages": [{"role": "user", "content": row.prompt}]}}
    for row in nightly_rows
])
# no stream, no per-item timeout, no retry hook, no ordering guarantee.
# results arrive as a file; matching them back is your problem.

The three-week figure also deserves the scepticism any integration estimate deserves. It was produced by the engineer who would do the work, counting the code: submission, polling, result matching, fallback. It does not count the operational surface that arrives with the code — a job that fails at 3am now needs an alert, an on-call runbook and a way to re-drive a partial batch — and it does not count the product work of telling users the report is now overnight, which involves at minimum a release note and realistically a conversation with two accounts. Integration estimates for asynchronous workflows run long more often than they run short, because the failure modes only become visible once the thing is live. Treating three weeks as a floor rather than a point estimate makes the comparison starker, not weaker.

Ranking by expected saving alone puts these two changes side by side. Ranking by saving per unit of effort separates them by a factor of sixteen, and adding reversibility separates them further, because the output limit can be undone by an on-call engineer at 2am and the batch migration cannot be undone at all in any timeframe that matters to a customer who has already been told the report now arrives overnight.

Shipping one changes the case for the next#

The stronger argument for going first with the cheap lever is that the second proposal's business case is computed against a workload that the first proposal changes.

$3,045 of the $8,600 batchable set is the nightly summarisation pass inside search.summarise, and the output limit cuts that route's generated tokens by 22%. After the limit ships, the batchable set is $7,930, its 44% realised discount is worth $3,489, and three weeks of work is now buying less than a day of work bought. Two other facts about that workload were also unknown at proposal time and became knowable within a month: the nightly report is scheduled for replacement by an incrementally-updated dashboard in the following quarter, which would strand the integration, and the ticket backfill's volume is a function of a migration that ends.

Saving divided by effort is a ranking heuristic rather than a rule, and it has two systematic blind spots worth naming so you can override it deliberately. The first is enabling work, which scores zero and is a prerequisite for everything else: the per-call ledger from chapter 2 saved nothing at all, took a week, and without it none of the estimates in this chapter exist. Work whose output is a number that other work depends on should be scheduled ahead of the ranking rather than inside it. The second is a lever whose value is mostly in the option it creates. Stabilising a prompt prefix so that caching can work is often worth little on its own and makes every future prompt change cheap, because a team that has to re-earn its hit rate after each edit will stop editing. Neither of these justifies reordering the whole list; both justify a short list of exceptions written down next to it, so the exceptions are argued once rather than every sprint.

The loop's value is entirely in the two decision nodes. Re-running the spend query after each change is what stops a team executing a ranking computed against a product that no longer exists, and holding 5% of traffic as a control — a slice deliberately left on the old configuration so the two can be compared side by side on the same days — is what makes the cost delta and the quality delta attributable to the change rather than to the week it shipped in.

Why four at once is not four times faster#

The temptation, given a deadline and four levers, is to ship them together. The savings do partly add, the work parallelises across a team, and everything lands before the quarter closes. What it costs is attribution, in both directions.

Suppose all four ship in the same week and the aggregate evaluation score drops 0.3 points on one task slice. Nothing in the data says whether that came from the output limit, the cheaper model tier, the deduplication or the batch path, and the cheapest remedy available is reverting all four, which returns all of the savings. Bisecting afterwards means four more rollout cycles with the evaluation re-run each time, which is more elapsed time than sequencing would have taken, spent under the pressure of a known regression rather than at planning pace.

Savings are equally unattributable in the good case. A bill that falls 31% after four simultaneous changes provides no evidence about which of the four to invest in next, and the estimates that produced the plan never get scored, so the team's ability to forecast the next round does not improve. The estimate that was wrong by 3× stays undetected because it was summed with three that were roughly right.

The levers are also not independent, which is the part the parallel plan implicitly assumes. Chapter 4 showed that tiering shrinks the dollar value of every other saving because it shrinks the base they discount. The interaction runs the other way too: shortening the output changes what the cheaper model is being asked to produce, and a small model asked for a four-sentence structured answer often passes an evaluation that the same model fails when asked for 900 tokens of prose. Ordering those two changes as shorten-then-tier can make the tiering lever viable on traffic where tier-then-shorten would have failed the quality gate and been abandoned.

What sequencing costs#

Going one at a time is slower to the target and that cost is real. Four levers at one to three weeks each, with a week of measurement between them, is a quarter; the same four built in parallel is perhaps five weeks. If the mandate has a date attached, strict serialisation may not fit.

The workable compromise is to serialise the rollout rather than the work. Build in parallel, put each lever behind its own flag, and stagger the ramps by enough time to read the evaluation between them — typically a week when you need a stable cost signal, since weekday and weekend traffic mixes differ enough to move spend per request by several percent on their own. Keep a permanent holdout of a few percent per lever, which costs a few percent of the saving and buys attribution that survives after the flags are cleaned up. On this product a 5% holdout on the output limit costs $205 a month and answers, at any point in the next year, the question of what the change is currently worth and what it is currently costing in quality — a question that otherwise requires reconstructing a baseline from an old month with a different traffic mix. The holdout has to be assigned by a stable hash of the tenant rather than per request, or the same user sees short and long answers alternately and reports it as a bug. That arrangement gets most of the parallel speed while keeping each change independently revertible, which is the property that actually matters at 2am.

Two things sequencing does not give you. It does not make a bad estimate good; it makes the error visible one lever at a time, which is a different service. And it says nothing about whether the savings should be accepted, because a 52% reduction achieved through three changes with unattributed quality effects is not yet a result — it is half of one.