Fifty-two percent is half a result on its own

The slide said 52%. The mandate had been to halve the cost of a feature running at $40,400 a month, three levers had shipped over six weeks, and spend per request was down from $0.0238 to $0.0114. The first question from the review was two words: "And quality?" The answer offered was that the aggregate evaluation score had moved from 4.42 to 4.32 out of 5, which sounded like a small price and was, on the population that number described.

That population was not the one the product serves. Underneath the blended score, one task slice had dropped 0.60 points — a 15% relative fall — and it was the slice the enterprise tier uses almost exclusively. The blended number was arithmetically correct and answered a question nobody had asked.

What the number is a percentage of#

Before quality enters, the cost claim needs its own precision. Spend per request fell 52.2%. The invoice fell 47.9%, because request volume grew 9% over the same six weeks, from 1.70 million to 1.85 million a month, so the line item went from $40,433 to $21,050 rather than to $19,312. Both numbers are true and they answer different questions: the first says the engineering worked, the second says what finance will see. A writeup that quotes one without the other will be read as quoting whichever is more flattering.

The per-lever contribution, in the order the levers shipped:

The hit rate improvement came from moving a rendered timestamp out of the system block, which is the kind of change that produces $2,112 a month and appears in a diff as two lines. The output change is the largest single contributor. The routing change is nearly as large in dollars and, as the next table shows, carries most of the risk.

One score cannot answer the question#

The evaluation set is 600 items sampled from the previous quarter's production logs, stratified so that low-frequency task types are represented well enough to measure, graded against a rubric with three seeds per item. It is not a public benchmark and it is not the 40 hand-written examples that had been sitting in the repository since the feature launched, both of which are dominated by short, well-posed questions and would have shown nothing.

Three properties of that set are doing the work. It is fixed, so the same items are graded before and after and the comparison is paired — every item is its own baseline — rather than a comparison between two different samples that may have differed to begin with. It is stratified rather than proportional, which is why the 4% slice has 60 items instead of the 24 that random sampling from production would have given it — a slice sampled at its traffic frequency is a slice you cannot measure, and the whole point is to measure the ones a blended score hides. And it preserves the tail: questions that produced long answers, questions with retrieval misses, questions in the second and third languages the product supports. Sampling by taking the first 600 rows of a log export, which is what usually happens, gives you one hour of one day and over-represents whichever cohort was awake.

Grading method needs stating alongside the set. Where an item has a checkable answer — a policy limit, a date, an entitlement — exact match against a stored key is the strongest option and needs no judge. Where it does not, a rubric graded by a model introduces its own drift, so the judge model and its prompt are pinned by version alongside the eval set, and a 60-item human-graded subset is re-scored each quarter to confirm the judge still agrees with people. A quality delta of 0.10 points measured by a judge that was silently upgraded between the two runs is measuring the judge.

The blended row is what a single-number report shows, and every interesting fact in the table is invisible in it. Two slices covering 59% of traffic moved by 0.04 points or less, which is inside the ±0.04 interval the 600-item set supports — the band within which the score could move on sampling alone, with nothing having changed in the system — and should be reported as no change. Policy explanation lost 0.37, almost entirely from the output lever, which is the effect chapter 5 predicted and which the shipping team had accepted knowingly. Multi-document comparison lost 0.60, three quarters of it from routing, and at n = 60 the interval is roughly ±0.31, so the drop is real rather than sampling noise.

That last row is the one that decides whether the work is accepted. A 0.10-point blended cost for a 52% saving is a straightforward trade. A 0.60-point drop on the workflow that 4% of requests and most of the enterprise revenue depend on is a different decision, made by different people, and it is available only because the slices were measured separately with enough samples in the small ones to see anything.

The tool-driven action slice carries 6% of traffic and 20 evaluation items, which gives an interval of about ±0.44 — wide enough that the lever could have broken it in half without the number moving outside the band. Reporting it as unmeasured is the honest option and it is also an action item, because a slice that cannot be measured cannot be traded away deliberately.

Attribution has an ordering convention#

The three delta columns above are marginal effects in ship order: each lever measured against the state that existed when it shipped. That convention has to be stated, because the alternative — ablation, removing one lever at a time from the final configuration — produces different numbers whenever the levers interact.

Routing is the case where the two disagree materially here. Measured marginally, after the output limit had already shortened responses, routing cost 0.44 points on multi-document comparison. Measured by ablation from the final state, restoring the strong model while keeping the brevity instruction, it recovers 0.44. But routing evaluated against the original long-form outputs would have cost more, because a cheaper tier asked for 900 tokens of comparative prose degrades further than the same tier asked for a four-field structured comparison. Chapter 6's argument for shortening before tiering shows up here as a measurement artefact: the order the levers shipped in improved the second lever's score, and a reader who assumes ablation semantics will draw the wrong conclusion about what routing costs in general.

Neither convention is wrong. The requirement is that the writeup names which one it used and, where a pair of levers interacts strongly, reports both.

The record#

What gets stored is more than a number and more than a table, because the purpose of the artefact is to let somebody in nine months decide whether to revert one of these levers without redoing the work.

{
  "feature": "policy-assistant",
  "window": {"baseline": "2026-03", "final": "2026-05-15/2026-06-15"},
  "cost": {
    "spend_per_request_usd": {"baseline": 0.023784, "final": 0.011368},
    "spend_per_request_delta_pct": -52.2,
    "invoice_delta_pct": -47.9,
    "volume_delta_pct": 9.0,
    "source": "provider usage export, reconciled to invoice within 1.1%"
  },
  "levers": [
    {"id": "prefix-stability", "shipped": "2026-04-08",
     "saving_usd_month": 2112, "revert": "revert PR #4471, <1h, no data migration"},
    {"id": "output-shape", "shipped": "2026-04-29",
     "saving_usd_month": 9690, "revert": "flag output_shape=legacy, <5m"},
    {"id": "tier-routing", "shipped": "2026-05-20",
     "saving_usd_month": 9319, "revert": "flag router_policy=strong_only, <5m",
     "holdout_pct": 5}
  ],
  "quality": {
    "eval_set": "prod-sample-2026Q1-v3", "n": 600, "seeds": 3,
    "attribution": "marginal, ship order",
    "blended_delta": -0.10, "blended_ci": 0.04,
    "worst_slice": {"name": "multi-doc-comparison", "delta": -0.60,
                    "n": 60, "ci": 0.31, "traffic_pct": 4.0},
    "unmeasured_slices": ["tool-driven-action"]
  },
  "latency": {"p50_ms": {"baseline": 2900, "final": 1600},
              "p95_ms": {"baseline": 8400, "final": 5100}},
  "rejected": [
    {"lever": "prompt-compression",
     "reason": "net 7.5% after compressor cost; query-aware variant destroys prefix cache"},
    {"lever": "batch-endpoint",
     "reason": "3 weeks, forecloses re-run; target workload scheduled for replacement"}
  ]
}

Four fields in there do work that a prose summary would lose. The reconciliation tolerance says how much of the cost claim is measurement rather than truth, which matters when someone later finds a 3% discrepancy and wonders whether the whole result is suspect. The holdout on the routing lever means the quality comparison stays live after the flags are cleaned up, so a regression that appears in August can still be attributed. The revert entries state effort and blast radius while the people who know them are still on the team. And the rejected list stops the next person spending a fortnight rediscovering that compression nets 7.5%.

What the record does not settle is what happens between quarterly evaluations, and that gap needs its own instrumentation. Four signals are worth alerting on continuously because each corresponds to a lever silently failing: the cache hit rate per route, which falls the week someone adds a variable to the system block; the escalation rate, which drifts with query difficulty and quietly rewrites the routing lever's economics; the distribution of stop_reason, where a rising share of max_tokens means the rail has become a cap; and spend per request per route, which catches everything else including the changes nobody thought to mention. None of them measures quality. They measure whether the configuration the quality evaluation was run against is still the configuration in production, which is a cheaper question to answer daily and the one that goes wrong first.

The latency numbers are there because they moved in the product's favour and would otherwise go unclaimed: shorter responses reach the last token sooner and a cheaper tier is faster per token, so p50 fell 45%. Cost work often improves latency and occasionally destroys it, and either way it belongs in the same record as the money.

Saying what was given up#

The sentence the writeup owes the business is not "quality fell 0.10 points". It is that multi-document comparison — the workflow where an analyst asks the assistant to reconcile three policy documents — now produces materially worse answers, that this is 4% of requests and a large share of enterprise usage, that it is caused by the routing lever, and that exempting that task type from routing would return roughly $1,100 a month of the $21,100 saved. Stated that way it is a decision somebody can make in a meeting. Stated as a point on a five-point rubric it is a number that gets nodded at.

That is the shape of a defensible result for one feature owned by one team. A platform with six teams shipping independently has the same problem multiplied, and it acquires one the single-feature case does not have: before any of this can be ranked or assigned, everyone has to be reading the same numbers.