Cost and latency budgets in CI

A quality gate answers one question well: is this commit's answer quality at or above the line. It is silent on every other property of the change, and silence in a pull request reads as approval, because a green check is the only summary most reviewers see. The two properties most often bought with quality are latency and cost per request, and both of them move in exactly the changes that improve a score the most — more retrieval, a reranking pass, a bigger model, a second call.

The instrumentation gap is not the reason this happens. Every case in an eval suite runs the real pipeline, which means the harness has already issued every API call the product issues, and the token counts and stage timings are sitting in the response objects being discarded. What is missing is that nobody wrote them down next to the score. This chapter is about writing them down: what to record, where the budget numbers come from, and what a per-commit budget can and cannot substitute for.

The harness already knows the other two numbers#

That claim rests on what an eval case actually is, and the rest of the chapter depends on it being true. A case is not a simulation of the pipeline and not a stub standing in for it. It is a run of the pipeline: the same retrieval function, the same reranker, the same model, called the same way the product calls them, because a case that exercised anything else would be measuring something other than the product. By the time a case has been graded, every API call the product would make on that query has already been made, against the real provider, at the real prices.

Two of the three numbers then arrive without anybody having asked for them. A metered API has to tell you what it metered, since you are entitled to see what you are being charged for, so the provider returns the input and output token counts for a call inside the same response object that carries the answer — the usage field the code below reads. Converting those counts into money is a lookup against the rate card, which is arithmetic on values already sitting in the process. Neither step puts anything on the network.

A stage timing is cheaper still. It is two readings of a local clock, one taken before a call and one after, with the duration of the stage being the difference between them. Reading a clock does not touch the provider, does not add a request, and does not change what the pipeline does — the calls being timed are the calls the harness was going to make anyway. Ordinary instrumentation means adding something to a system in order to observe it, and this is the case where there is nothing to add.

That is why the distance between a one-axis report and a three-axis one is a reporting gap rather than an instrumentation gap, and the distinction decides how much work it is. An instrumentation gap is a project: a tracing library, a collector, a sampling policy, an argument about overhead in the hot path. A reporting gap is a change to what a job writes down at the end of a run it was already performing. Here is that change, at the level of a single case:

# evals/record.py — one case, all three axes
async def run_case(case: dict) -> dict:
    t0 = time.perf_counter()
    chunks = await retrieve(case["query"], k=20)
    t_retrieve = time.perf_counter()
    chunks = await rerank(case["query"], chunks)[:5]     # new in this PR
    t_rerank = time.perf_counter()
    resp = await client.messages.create(
        model=MODEL, max_tokens=1024,
        messages=[{"role": "user", "content": build_prompt(case["query"], chunks)}],
    )
    t_end = time.perf_counter()

    return {
        "case_id": case["id"],
        "passed": await grade(case, resp),
        "latency_ms": {
            "retrieve": round((t_retrieve - t0) * 1000),
            "rerank":   round((t_rerank - t_retrieve) * 1000),
            "generate": round((t_end - t_rerank) * 1000),
            "total":    round((t_end - t0) * 1000),
        },
        "tokens": {"in": resp.usage.input_tokens, "out": resp.usage.output_tokens},
        "cost_usd": price(MODEL, resp.usage) + RERANK_COST_PER_CALL,
    }

With that record, the aggregate report carries three numbers instead of one, and a change looks different when you can see all of them at once. Aggregating the per-case latencies takes one decision on the way, and the obvious choice is the wrong one. An average over the suite is held down by the many fast cases and says almost nothing about the slow ones: a change that leaves most requests alone and doubles the time on a minority of them moves the mean by a small fraction of what it did to those requests, and those requests are the ones people complain about. So two statistics get reported rather than one. The median is the middle value, with half the requests faster than it and half slower, and it describes the request nothing went wrong for. The other is the p95 figure: the latency 95% of requests come in under, so only the slowest one in twenty exceeds it, which is the usual way to talk about latency because the mean hides the tail users actually complain about.

Here is one such change — a cross-encoder reranker inserted into the request path, meaning a second, smaller model that reads the query and a candidate chunk together and scores the pair, which is more accurate than comparing precomputed embeddings and costs a model call per candidate:

The two latency rows rose by nearly the same number of milliseconds, which is the shape a step added to every request makes: it lands on the fast requests and the slow ones alike. A change that only hurt some requests would move the p95 and leave the median where it was. The per-stage split is what makes the result actionable: all of the added latency is in one hop that did not exist before, so there is no profiling exercise to run and no ambiguity about what to discuss.

Where that table appears matters as much as its contents. A staging environment carrying mirrored traffic will have the latency number days before a merge and nobody will read it, because a dashboard is a thing you go and look at and a pull request is a thing that comes to you. Posting the three-axis table as a comment on the pull request, updated on each push, is the difference between the information existing and the information being used, and it costs a few lines in the workflow. The same principle explains why a quality score alone is so persuasive: it is the only number on the page.

The sentence a reviewer is most likely to write, looking at a green pipeline, is that the eval gate passed so the change is fine to merge. It is tempting for reasons worth naming rather than dismissing. A gate is a thing that stops bad changes, so a gate that did not stop this one appears to be saying that the change is not bad. The check is called the eval gate rather than the answer-quality gate, which makes it sound like a verdict on the change rather than on one property of it. And it is the only summary on the page, with nothing beside it disagreeing. What is actually true is much narrower: green means the graded answers on this commit came in at or above the line, and that nothing else was measured. The commit in the table above is exactly that — passing on the one axis anybody checked, and over budget on the two nobody was reporting.

Where the budget numbers come from#

Both budgets are derived rather than chosen, and the derivations are short enough to put in a comment in the config file so that the next person to argue about them has the reasoning.

Latency starts from the product's stated target, which for one assistant was a service level objective — an SLO, the performance level the team has committed to in writing — of p95 2.0 seconds end to end, with main sitting at 900 milliseconds. Subtracting the second figure from the first gives what is left to spend, since two seconds is nine hundred milliseconds plus a bit over a second. That bit over a second is the 1.1 seconds of available headroom, and it is the whole of the latency every future change may add between them, for as long as the objective stands.

Two properties of that headroom decide the shape of the budget. It is shared, because there is only one request the user waits on however many pull requests contributed to it, so every change that adds latency draws on the same second. And it is finite, because an objective the team has committed to in writing is a commitment rather than an aspiration. A gate set at the SLO itself respects neither property. It asks each pull request one question — does this commit come in under two seconds — and the first change to arrive can answer yes while consuming all the headroom, at which point the second change to try is blocked for the first one's sins. Nothing about arriving first made that change worth the whole allowance.

Rationing the headroom per change is what removes the advantage of arriving first, and the ration written into the config is "no more than 10% slower than base, and no more than 1,200 ms absolute in the harness". That pair is a policy choice rather than a derived quantity, and it is worth being explicit about which of the numbers here is which. The 1.1 seconds follows from the objective and today's measurement, and anyone holding both can recompute it. How finely to slice it does not follow from anything: it is a judgement about how many changes the team expects to spend against the headroom before the next round of optimisation work, and a team expecting a dozen contenders slices more finely than one expecting two.

The reasoning has a familiar shape elsewhere, worth borrowing as a picture rather than as a fact about this system. A bundle-size budget in a web build rations a page-weight target the same way, and nobody sets that gate at the target and lets one commit spend it all, because the allowance belongs to the roadmap. The picture stops at what is being measured. A bundle's size is a property of the build output, counted exactly and identically on any machine that builds it, while a latency figure is a measurement of a run and moves between runs even when the code has not. That is why a latency budget leans on the relative form, and it is the reason for the caveat below.

Cost starts from unit economics, and the arithmetic is short once one term has been unpacked. That product bills usage at roughly $0.012 per request and the business plans on a 78% gross margin. Gross margin is the share of revenue left after the cost of serving what was sold, so a committed margin is already a ceiling on serving cost stated the other way round: the margin is the part that stays, and everything else is the part that may be spent. A seventy-eight percent margin therefore leaves a bit over a fifth of the revenue for serving.

Applying that share to the revenue gives the ceiling. Revenue is a little over a cent per request, and a bit over a fifth of a little over a cent is just over a quarter of a cent, which is the $0.0026 per request in the config. Nothing in that step is peculiar to model serving; it is the calculation a hosting bill or a card fee would go through unchanged, and the only reason it is worth writing out is that a ceiling without it invites an argument about the ceiling when the disagreement is really about the margin.

Main sits at $0.0021 against that ceiling, which leaves five ten-thousandths of a dollar to spend. The 24% of headroom quoted for it is that gap measured against what main already costs rather than against the ceiling: five ten-thousandths on top of twenty-one ten-thousandths is close to a quarter more than today's spend. Quoting it that way is deliberate, because the per-change rule below is also a percentage of the current position, and the two figures are then in the same units and can be compared without conversion.

The same rationing logic applies, and cost is where it becomes clear why the rationing takes two rules rather than one. The gate blocks on the absolute ceiling, and it blocks on any increase over 15% in a single change. Without the absolute ceiling, the percentage cap would still be satisfied by a sequence that walks the cost past the point where the business case holds, since each step is measured only against the step before it. Without the percentage cap, one change may take the position from where main sits to just under the ceiling in a single step, pass on the day it lands, and leave nothing for anything that follows. The compounding is the harder of the two to see, because a series of 14% increases is a problem that no per-change absolute ceiling will catch until it is already breached. Each rule catches what the other misses, which is why neither of them is decoration.

# evals/budgets.yaml — read by the same job that computes the score
quality:
  absolute_min: 0.79        # measured noise floor 1.4pp; 2.33 sd below the 30-day min
  max_regressed_cases: 8

latency:
  # product SLO is p95 2.0s end to end; main is at 900ms in production.
  # harness numbers run unbatched and warm, so treat them as relative.
  p95_ms_absolute: 1200
  median_delta_pct: 10

cost:
  # $0.012 revenue per request at a 78% gross margin target -> $0.0026 ceiling
  usd_per_request_absolute: 0.0026
  delta_pct: 15

on_breach: block            # per-axis; overridable, see the override policy

There is an honest limitation in using harness latency this way, and it should be written down where the numbers are. The eval run issues requests one at a time against a warm cache with no batch neighbours and no queueing. Each of those absences is a source of waiting that production has and the harness does not: a request arriving while another is being served waits behind it, a request batched with others waits for the batch to be assembled, and a cold cache turns a lookup into a fetch. Nothing in the harness waits behind anything else, so each of those terms is zero, which is why its absolute p95 is not the production p95 and will usually be optimistic.

What survives the move to the harness is the comparison. Both commits are measured by the same harness, over the same suite, under the same missing conditions, so whatever production would have added is missing from both measurements in the same amount. A quantity that sits in both numbers identically disappears when one is subtracted from the other, and that is why the difference between two commits is a defensible thing to set a budget on when the absolute figure is not. The absolute ceiling in that config is a coarse tripwire rather than an SLO check — it is there to catch a change that is wrong by an order of magnitude, not to verify the objective.

The cancellation holds while the missing conditions cost both commits the same, and there is a class of change for which they do not. A change whose cost depends on how many requests are in flight at once, or one that leans harder on a resource several requests share, adds something in production that is absent from the base measurement rather than present in both, so the harness difference understates it instead of cancelling it. Those are the changes something other than a budget has to find.

Load tests, SLOs and per-commit budgets#

A latency budget in the eval job is not a load test: it exercises one request at a time and will not find a connection pool that saturates at 200 concurrent requests or a reranker whose own p99 collapses under batch. And it is not an SLO, because an SLO is a statement about production over a window with an error budget attached, whereas this is a per-commit comparison against a base. The SLO itself is verified by a load test on the branch and by the canary, meaning the practice of routing a small share of real production traffic to the new version and watching its metrics before the rest follows. The three controls coexist, and the budget's specific contribution is that it is the only one of them that runs before the code merges, on a signal the harness produces for free.

Two other things the budget does not do are worth stating. It does not cap total spend, since a change that halves cost per request and triples usage will sail through while the monthly bill doubles. And it says nothing about the cost of the eval infrastructure itself, which the chapter on tiering handled separately and which is a different budget with a different owner.

Request-path latency and the cost of removing it later#

If the team had decided the full reranker was worth it anyway, that would have been a legitimate answer — a decision to spend half the latency headroom and a quarter of the margin headroom, made by someone who owns those budgets and recorded as a change to budgets.yaml in the same pull request. The gate's job is to make that a decision rather than a side effect.

What is not legitimate is deferring it, because "optimise later" describes an outcome that does not happen to latency introduced in the request path. The reason is in the word synchronous. A 540-millisecond synchronous hop is not a slow function waiting for a profiler; it is a network call to a model that has to complete before the prompt can be assembled. What makes "optimise later" a real plan for ordinary slow code is that slow code has waste in it and a profiler finds the waste. There is no waste in a call whose result the next step cannot start without. The time is not being spent badly. It is the price of the thing the change bought.

So the only way to remove the hop is to stop needing what it returns, and that means recovering the quality some other way — which is the original problem again, the one the reranker was proposed as the answer to. That is the position on the day it merges, and it only gets worse afterwards. The next three changes build on the reranker being there: the citation ordering starts assuming reranked order, a downstream prompt gets tuned against the reranked chunk distribution, an evaluation case gets written that only passes with reranking. Each of those is reasonable on its own terms, and each of them is a dependent, so by the time anyone tries to take the hop out it is a redesign with its own quality risk rather than an optimisation.

That argument is about inserting a call into the request path, and it does not carry to work that is merely expensive. Something sitting off the request path is a different case: a value precomputed at ingest, a cached result, a step that runs after the response has gone back. Work of that kind can be made faster or cheaper later, or dropped, without anyone having to recover a quality point first, because nothing the user is waiting on depends on when it finishes. The asynchronous reranking option in the table above is exactly that move, and the same table prices it — the latency stays where it was and most of the quality gain is lost, which is a decision somebody makes rather than one they defer.

Ratcheting from a baseline that already breaches#

Retro-blocking is not available for everything that merged under a quality-only gate, and it would not help; the code is in main and the customers are on it. The sequence that does work starts by measuring main against the new budgets and writing the result in as the starting position, so that if main already breaches the derived numbers, the gate goes in as a ratchet — nothing may be worse than today — with a dated ticket and a named owner for closing the gap to the real budget. A gate that blocks every pull request from day one because the baseline already fails is a gate that gets disabled in a week.

The reason a ratchet is the right instrument there, rather than a stricter gate with a grace period on it, is what an always-failing check is communicating. A check earns attention by discriminating: its verdict has to depend on what the change did, or reading it tells the reader nothing they did not know before opening the page. A gate set at a budget the baseline already breaches returns the same failure to every developer on every pull request, including the ones that touched nothing on that axis, so its output does not vary with its input. Everybody learns within a few days to merge past it, and switching it off is then the correct response to a signal carrying no information rather than a lapse in discipline. A ratchet discriminates by construction, because "nothing may be worse than today" is a comparison against what this change did, and a change that added nothing passes it.

The second step of that sequence is to backfill the history. Nightly infrastructure can run the three-axis measurement over the last 30 merge commits in one job, which produces a latency and cost trace for the past few weeks and usually identifies two or three changes responsible for most of the current position. On the codebase above that exercise found the reranker at 540 milliseconds and a retry-with-longer-context fallback added six weeks earlier that had gone unnoticed at 180 milliseconds of median latency for a 0.3-point quality gain, which nobody would have accepted as a proposal and which was removed the same afternoon.

The whole change is one of framing rather than of instrumentation. The harness already produces the timings and the token counts; putting them on the pull request beside the score, against budgets derived from the product's latency target and its margin, converts a green tick from a statement about one axis into a statement about three. What that buys is not mainly blocking — it is that somebody has to look at a change which trades a fifth of the latency headroom for five points of quality and say whether that is the trade they want.

Three gates on the same run means three independent ways for a pull request to be blocked, and the more axes you measure the more often you will meet a change that is genuinely correct and genuinely blocked. What you do at that moment, and how you record it, is the last thing this topic has to establish.