The meter undercounts what it never saw finish

The support-experience team's budget dashboard said $2,900 of $4,000 on the 28th of the month, which is a comfortable place to be. The provider's invoice, arriving four days later, attributed $6,100 to the key that only the gateway holds. Nobody had misconfigured anything. No other service could use that key, because chapter 3's egress rules had been in place for two months and the flow logs showed no direct connections. The gateway's per-request records added up correctly to $2,900, and the vendor's per-key total added up correctly to $6,100.

Both numbers were right about different populations of requests. The gateway counts requests it saw finish. The provider counts tokens it generated.

Three ways a request ends without producing a usage record#

The support-experience product is a streaming chat assistant embedded in the agent console. When a support agent asks it a question, the answer streams token by token, and the console shows a stop button. Roughly 38% of generations are cancelled before completion, either because the agent has read enough or because they navigated to the next ticket. When the browser disconnects, the gateway's response writer fails, the handler unwinds, and the code path that reads the final message_delta event — the one carrying output_tokens — never runs. The provider, meanwhile, has generated tokens up to the moment the upstream connection was closed and bills for them. On a product where the median answer is 620 output tokens and the median cancellation happens around token 400, that is 400 tokens billed and zero tokens recorded, 38% of the time.

The second gap is the gateway's own retries. When an upstream call returns a 500 or times out, the gateway retries, and the usage record it eventually writes describes the attempt that succeeded. The failed attempt is not always free: a request that timed out at the gateway's 30-second deadline may have been generating for all 30 of those seconds and continued to completion at the provider, which bills for the full output. The gateway sees a timeout, records nothing, retries, and books one request's worth of tokens for what the vendor books as two.

The third is everything that completed upstream and failed on the way back. A gateway replica that gets OOM-killed mid-response, a connection reset between the gateway and the provider after the last chunk was generated, a 502 from an intermediary. From the vendor's side these are finished generations. From the gateway's side they are errors, and error paths in metering code are where zero-token records come from.

The direction is always the same, and that is the structural point rather than an accident of this codebase. Usage records get written on the success path because that is where the response object with the token counts lives. Every terminal state that is not a clean success needs its own line of code, and each one that is missing subtracts from the meter and never adds. A meter that is wrong in the other direction is a bug; a meter that is wrong in this direction is the default.

For a streaming product, cancellation dominates by a wide margin and it is worth checking that first. For a batch pipeline with no streaming and aggressive timeouts, the retry gap dominates instead. The mix tells you which line of code to write.

Counting what happened rather than what returned#

The repair is to accumulate usage as it passes through and to write the record on every exit, including the ones you did not plan for.

async def proxy_stream(req, principal):
    usage = {"input_tokens": 0, "output_tokens": 0}
    outcome = "unknown"
    try:
        async with upstream.stream(req) as up:
            async for event in up:
                if event.type == "message_start":
                    usage["input_tokens"] = event.message.usage.input_tokens
                elif event.type == "content_block_delta":
                    usage["output_tokens"] += 1      # approximate, corrected below
                elif event.type == "message_delta":
                    usage["output_tokens"] = event.usage.output_tokens
                yield event
        outcome = "complete"
    except ClientDisconnect:
        outcome = "client_cancelled"
        await upstream.cancel()          # stops further billing, does not refund
    except UpstreamError:
        outcome = "upstream_error"
    finally:
        meter.record(principal, req.model, usage, outcome)   # always runs

Two details in that block do most of the work. Incrementing output_tokens per delta gives a running count that survives a disconnect, and it is approximate because a delta is not exactly one token — measured against the authoritative count on completed requests, the per-delta approximation came in 3.1% low, which is a known bias you can correct rather than an unknown gap. Cancelling upstream on client disconnect matters because a stream you stop reading is not a stream the provider stops generating; without the explicit cancel, the vendor keeps producing to max_tokens and bills you for output nobody will ever see.

Reconciliation closes what remains. Pull the provider's per-key totals monthly, compare to the meter, and carry the ratio forward as a correction factor per model and per route. After the fix above, the gap at this company went from 52% to 4.2%, which is small enough to be a footnote in the chargeback — the internal bill that moves model spend onto the cost centre of the team that incurred it — instead of an argument.

Two details keep the corrected meter from drifting again. Usage records must be idempotent on the provider's request ID, which is to say that a record arriving twice for the same ID counts once, because a finally block that runs during a panic and again during a retry of the handler will double-count, and a meter that over-reports is politically worse than one that under-reports — a team that has been billed twice will never trust the number again. And the price table has to model what the vendor actually charges rather than a single input rate: cached input reads are typically around a tenth of the base rate while cache writes are around 1.25 times it, batch endpoints run at roughly half, and reasoning models bill their thinking tokens as output even though the caller never sees them. A gateway pricing everything at the base input and output rates on a prompt-caching-heavy workload can be 20% wrong in either direction with a perfectly correct token count, which is a different bug from the one this chapter opened with and produces the same argument in the same meeting.

Attribution is the other half of what makes the number usable. A per-principal total tells the recommendations team they spent $4,100; a per-feature breakdown tells them $3,600 of it was the product-summary route re-generating on every page view because the cache key included a timestamp. The gateway can produce the first from the token alone and the second only if callers label their requests, which is why chapter 1 put the metadata block in the client and made it required.

Enforcement means committing before you know#

Everything so far produces a better report. A report does not stop anybody from spending, because by the time a request has produced its usage record the money is gone. Refusing the request that would breach the budget means deciding at admission, when the only thing you have is an estimate.

The estimate is available. Input tokens can be counted exactly before the call, and the worst-case output is max_tokens, which the gateway can require callers to set. Pricing both at the model's published rates gives an upper bound on what this request can cost. Reserve that amount, run the call, then settle the reservation against actual usage.

-- Redis: reserve worst-case cost, refuse if it would breach the budget.
local key, cap, cost = KEYS[1], tonumber(ARGV[1]), tonumber(ARGV[2])
local used = tonumber(redis.call('GET', key) or '0')
if used + cost > cap then return {0, used} end
redis.call('INCRBYFLOAT', key, cost)
redis.call('EXPIREAT', key, tonumber(ARGV[3]))   -- month boundary
return {1, used + cost}

What that costs at request time comes in three parts. There is latency: one round trip to a co-located Redis, measured at 0.6 ms p50 and 3 ms p99, plus a second call to settle. There is availability, since the budget store is now on the critical path and you must decide in advance whether its outage fails open or closed — fail closed and a Redis blip takes down every model call in the company, fail open and the budget is advisory during exactly the incidents when something might be looping. There is capacity waste: reserving max_tokens when the median generation uses 12% of it means a team can be refused at $3,400 of real spend against a $4,000 cap. Settling promptly and reserving against a rolling estimate of that team's actual output ratio recovers most of it, at the cost of a bound that is no longer strictly an upper bound.

Reservations also leak. A gateway replica that dies between reserving and settling leaves the reservation held, so reservations need their own short TTL and a settlement that is idempotent by request ID.

The tempting alternative is to skip all of this and enforce on the provider's usage API, which is authoritative and already broken down by key. It is authoritative and it is late. Provider usage endpoints publish with a lag typically measured in hours, and a service stuck in a retry loop at 200 requests per second on a model costing $3 per million input tokens with 4,000-token prompts spends about $8,600 an hour. Enforcing on data that is four hours old means the budget is checked against a number from before the incident started. Provider usage is the right basis for reconciliation and chargeback, and the wrong basis for control.

What to offer a team that will not accept a hard cap#

The support-experience team's objection to enforcement was specific and fair: a hard cap means their product stops working mid-month for every support agent in the company, and the failure lands on customers who are already annoyed enough to be talking to support. A budget that can take the product down is a budget nobody will agree to, and a budget nobody agrees to gets exceptions until it is decoration.

The structure that makes this acceptable is that the hard ceiling exists at a level well above the working budget, and everything between the budget and the ceiling is degradation the team chose in advance. Publishing which of their routes are interactive and which are background is work only the team can do, and it is the thing that lets the gateway shed the right traffic at 90% instead of refusing whatever arrives next.

Two things this is not. It is not cost optimisation — the meter tells you where the money goes and enforcement stops the extreme cases, and neither makes a 12,000-token prompt cheaper. And it is not a billing system: the number you charge a cost centre should come from reconciled figures with the correction factor applied, not from the same counter you are enforcing on in real time.

The retry gap in the table above was worth $610 a month, which makes it look like the least interesting row. It is the one with a second life. Those retries exist because the gateway was configured to retry on 5xx, and the services calling the gateway were written before it existed, when retrying on 5xx was their own responsibility. Both layers are still doing it, and on an ordinary day that costs $610. On the day the provider has a brownout, it costs considerably more than money.