A new customer can take credit for your prefix reorder

The change was two hours of work and obviously correct: the per-user preference block had been sitting in front of the shared instruction block, which meant no two requests shared a prefix past the first few hundred tokens. Moving the variable part behind the stable one is the first thing anybody does with a prompt that is not caching. It shipped on a Monday. By the following Monday the agent-chat route showed hit rate up from 44% to 81% and cost per request down 30%, and the write-up was drafted before lunch.

On the Tuesday of that same week, a customer four times larger than any existing account finished onboarding and began sending about 40% of the platform's total volume on that route. Their workload is a single fixed policy prompt asked thousands of times an hour with two-sentence answers — close to the best case caching has. Nobody on the inference team was told, because onboarding a customer is not an inference-team event.

Both the deploy and the onboarding push hit rate up and cost per request down. The platform aggregate is the sum of the two, and there is no way to read the split out of it.

The aggregate cannot tell them apart#

The structure of the problem is the one an A/B test — two variants run side by side over the same period, so that everything except the variant is shared — exists to avoid. The treatment and a second cause both landed inside the measurement window, both push the observed quantity in the same direction, and the observed quantity is a single scalar with no memory of which requests contributed what. A before-and-after on a live platform is an observational study, meaning you watch what happened rather than controlling what varied, and an observational study with a known simultaneous confounder — a second cause that moved inside the same window and pushes the same number the same way — proves nothing at all.

The size of the confound here is not small. Work the mix arithmetic: in the after-window, the new tenant accounts for about 42% of eligible lookups on the route at a hit rate of roughly 97%. If the existing tenants had not moved at all, the platform figure would already have risen from 44% to about 66% on mix alone. The deploy is somewhere inside the remaining fifteen points, and so is anything else that changed that week.

Cost per request has the same problem with an extra twist, because the new tenant's requests are individually cheap — short questions, two-sentence answers — so their arrival drags the platform average down independently of any caching effect. A metric that is itself an average over a changing population moves whenever the population changes. That is true of cost per request, of hit rate, of p95 latency, and of every other per-request average on the dashboard.

Fix the cohort before you compare anything#

The repair is to compare a population that did not change. Take the set of tenants that were active on the route in both windows, and evaluate hit rate and cost per request over only those tenants, on only the route the change touched, with the model held fixed. Everything that entered or left the platform between the two windows is excluded from both sides.

Excluding the new customer alone is the version most people write and it is not sufficient. Tenants that churned out during the window distort in the opposite direction, and a tenant whose own volume tripled reweights the average without appearing as new. The filter that survives review is the intersection: tenants present in both windows, with each tenant's contribution weighted the same way on both sides.

-- Cohort-fixed before/after. `cache_events` carries one row per eligible
-- lookup, tagged as chapter 1 describes.
WITH windows AS (
  SELECT 'before' AS w, TIMESTAMP '2026-03-02' AS lo, TIMESTAMP '2026-03-09' AS hi
  UNION ALL
  SELECT 'after',       TIMESTAMP '2026-03-16',      TIMESTAMP '2026-03-23'
),
tagged AS (
  SELECT e.*, w.w
  FROM cache_events e
  JOIN windows w ON e.ts >= w.lo AND e.ts < w.hi
  WHERE e.route = 'agent-chat' AND e.model = 'claude-sonnet-5'
),
-- The cohort: tenants that sent traffic in BOTH windows. This drops the new
-- customer and anyone who churned, in one condition.
cohort AS (
  SELECT tenant_id FROM tagged WHERE w = 'before'
  INTERSECT
  SELECT tenant_id FROM tagged WHERE w = 'after'
)
SELECT
  t.w,
  COUNT(*)                                              AS eligible_lookups,
  ROUND(AVG(CASE WHEN t.outcome = 'hit' THEN 1.0 ELSE 0 END), 4) AS hit_rate,
  ROUND(SUM(t.cost_usd) / COUNT(*), 6)                  AS cost_per_request
FROM tagged t
JOIN cohort c USING (tenant_id)
GROUP BY t.w;

There is a subtler version of the same problem inside the retained cohort, and the query above does not handle it. Pooling every retained tenant's requests into one average still lets a tenant whose own volume tripled between the windows dominate the after-side. If that tenant happens to have a highly cacheable pattern, the cohort-fixed number inherits a smaller copy of the confound you just removed. The fix is to compute the hit rate per tenant and then average across tenants using each tenant's before-window share as the weight, so the after-side is evaluated on the same population proportions as the before-side. On most platforms the pooled and reweighted numbers land within a point of each other, and on the platforms where they do not, the difference is exactly the effect you were trying to exclude.

Two details in that query do real work. The model predicate is there because cache entries are model-scoped, so a partial model migration inside the window is its own confounder wearing a different hat. And the windows are the same length and the same days of the week, because a weekday-heavy window compared against one containing a bank holiday shifts the tenant mix even within a fixed cohort.

What the control route buys you#

The cohort-fixed comparison removes the confounder you know about. It does not remove the ones you do not, and on a platform of any size there is always something: a provider-side model update, a client release that changed how long user questions are, a retry policy that started double-sending. What rules those out is a route you did not touch, evaluated over the same cohort and the same windows. Subtracting that untouched route's movement from the treated route's movement is an estimator with a name, difference-in-differences, and the table below works it out on real figures.

Choosing the control is where this method is usually botched. A control route has to satisfy three conditions: it was untouched by the change, it shares the infrastructure and provider that any external shock would have arrived through, and it is served by the same tenants so that a shift in customer behaviour would show up on both sides. ticket-classify qualifies on all three. A route running on a different provider, or one serving an entirely different customer segment, is a control in name only — it would stay flat through a shock that hit the treated route, and its flatness would be read as evidence for your change.

A control also has to be plausibly stable, which is a property you check rather than assume. Pull four weeks of the candidate's cohort-fixed hit rate before doing anything; a control that wanders three points a week on its own cannot certify a treated route to any better precision than that. ticket-classify at 95% with a weekly standard deviation of 0.4 points can. This is the same discipline as picking a control arm in a product experiment, with the difference that you are picking it after the fact and therefore have to be honest about having done so.

The control route moving 0.3 points says that nothing platform-wide shifted between the windows, so the 24.5 points on the treated route is attributable to what happened on the treated route. Subtracting the control's drift from the treated route's change gives the difference-in-differences estimate, and 24.2 points is the number that belongs in the write-up. Thirty-seven points and 30% belong nowhere; they are the sum of your work and a sales team's, and claiming them costs you the credibility you will need the next time a number moves in the wrong direction and you want to argue it was not you.

Twenty-four points is still an excellent result. On a route where the prefix is most of the spend, a 17.6% cut in cost per request across the retained cohort is a real, defensible, repeatable saving, and it is the one you can quote when someone asks what the same change would be worth on the next route.

What this method still cannot rule out#

Difference-in-differences leans on an assumption worth stating rather than burying: that in the absence of your change, the treated and control routes would have drifted in parallel. That is plausible for two routes in the same service over two adjacent weeks and it is not guaranteed. If the retained cohort's questions on agent-chat genuinely got more repetitive that fortnight — a seasonal pattern, a customer's own product launch — some of your 24.2 points belongs to them. Nothing in the observational data can separate that, and the honest write-up says so in one sentence and moves on.

The version that removes the assumption is the one that has to be decided before the change ships rather than after. Put the reorder behind a flag, route a stable 10% of the cohort's traffic to the old layout for a week, and compare the two arms inside the same window with the same tenants and the same questions. That is a randomised holdback, it costs one config value and a slightly worse hit rate on a tenth of traffic, and it turns an argument into a measurement. Retrofitting it is impossible, which is why the decision belongs in the change's design and not in its write-up.

One detail in that holdback is specific to caching and gets people the first time. Assign the arms by a stable hash of tenant id, or of session id where prefixes are per-session — never per request. A per-request coin flip sends two prompt layouts from the same tenant to the same cache, so each layout gets roughly half the reuse it would have had alone, and both arms report a hit rate below what either would achieve at full traffic. The experiment then shows a real difference between the arms with both numbers biased downward, and the effect size it measures is not the effect size you would get on rollout. Arms that share a cache have to be separated at whatever granularity the cache key is defined on.

What the defensible number buys you is the next decision rather than the last one. Twenty-four points and 17.6% on this route, with the route's prefix share of spend known from chapter 3, gives a forecast for the four other routes with the same prompt layout defect: roughly the same points of hit rate, scaled by each route's own prefix share and volume. That forecast is what gets the follow-up work funded. A claim of thirty-seven points forecasts thirty-seven points on the next route, the next route delivers twenty-four, and the third request for time gets declined.

There is a second reason to want that flag, and it has nothing to do with attribution. A prefix layout is a thing that can be changed by accident as easily as on purpose, and when the accidental version ships, the graph moves in the opposite direction, at speed, with no ticket describing what was done.