Sampling records and cost attribution

Sampling spends your provider account on another party's behalf, which makes attribution a design problem rather than a reporting one. The provider sees an API key and a model and bills accordingly; the fact that one particular request was composed by a document-search server, part-way through serving a tool call, against context your client assembled, exists only inside your client and only for as long as the request does. Nothing downstream will reconstruct it.

A second consequence is easier to miss, and it is why a team can watch the right dashboards and see nothing at all. Per-user metrics are computed from the requests the application itself makes on a user's behalf, because that is the code path with a user attached to it. A sampling call does not travel that path: it originates in a server, is authorised by the client's request handler, and reaches the provider from somewhere else entirely. A median or a p95 over per-user token counts is therefore not an insensitive measure of sampling spend. It is a measure of a disjoint set of requests, and it would read exactly the same whether sampling spend were zero or ten times what it is.

That has a consequence teams meet at an awkward moment. Every interesting question about a fleet of connected servers is a group-by on a column that has to be written at request time or not at all. Which server spent the increase. Whether the increase came from more requests or larger ones. Whether a wiki server had started attaching conversation context it had not attached in February. Whether a ticket-triage server's completions had got worse when it silently began landing on a cheaper model. Whether anything a user typed had been placed into a prompt written by a third party, which is the question a privacy team asks and the one with a regulatory clock attached.

One record per request, written when the request resolves#

The record is written whatever the outcome, including the outcomes where nothing was spent, because a denial is as informative as a completion.

interface SamplingRecord {
  ts: string;                    // ISO 8601, request resolution
  requestIdentity: string;       // the digest from chapter 4
  representedCount: number;      // how many times this identity has arrived

  serverId: string;              // "doc-search"
  serverVersion: string;         // from initialize; catches "it changed in v2.3.0"
  sessionId: string;
  userIdHash: string;            // pseudonymous; enough to bound a privacy incident

  contextRequested: "none" | "thisServer" | "allServers";
  contextGranted: "none" | "own_resources" | "conversation";
  contextManifest: { source: string; items: number; tokens: number }[];
  redactionsApplied: { rule: string; count: number }[];

  promptDigest: string;          // sha256 of systemPrompt + messages
  systemPromptDigest: string;    // separately, so a changed system prompt is visible
  modelHints: string[];          // what the server asked for
  modelUsed: string | null;      // what actually ran, from the result

  inputTokens: number;
  outputTokens: number;
  costUsd: number;               // computed from a versioned price table
  latencyMs: number;

  outcome: "allowed" | "denied" | "throttled" | "timedout" | "replayed";
  reason: string | null;         // "policy.budget_exhausted", "user.denied", …
}

Two decisions inside that record are worth defending. The prompt is stored as a digest and not as text, because a telemetry store holding every prompt every server ever sent is a second copy of exactly the material the context ceiling exists to protect, sitting in a system with weaker access controls than the one it came from. The digest supports every question about repetition, drift and correlation that an investigation actually asks, and a system prompt digest kept separately turns "this server changed what it is instructing the model to do" into a query rather than an archaeology exercise. The other decision is contextManifest, which records what was attached by source and size without recording its content: eleven items from the conversation totalling 7,400 tokens is enough to bound a disclosure, and it is not itself a disclosure.

modelUsed has to come from the result rather than from the request, for the reason established in chapter 1, where the server's hints turn out to be advisory and the client selects: price per token differs across that selection by an order of magnitude. Attributing cost from what a server asked for rather than from what ran produces a report that is wrong in a direction nobody notices.

The volume is unremarkable. At 57,000 sampling requests a day, records of roughly 450 bytes come to 26 MB a day and under 10 GB a year, which is small enough that the retention argument should be about privacy rather than storage. Both steps are worth doing once rather than taking on trust: fifty-seven thousand records of four hundred and fifty bytes each is a little under twenty-six million bytes a day, and twenty-six million a day across three hundred and sixty-five days is a little over nine thousand million — under ten gigabytes, on the busiest single day described anywhere in this topic, repeated every day for a year.

With that column present, a month's spend is one query.

SELECT server_id,
       COUNT(*)                                   AS requests,
       COUNT(*) FILTER (WHERE outcome = 'denied') AS denied,
       SUM(input_tokens + output_tokens)          AS tokens,
       ROUND(SUM(cost_usd), 2)                    AS usd,
       ROUND(SUM(cost_usd) / COUNT(*), 4)         AS usd_per_request
FROM sampling_records
WHERE ts >= '2026-03-01' AND ts < '2026-04-01'
GROUP BY server_id
ORDER BY usd DESC;

The usd_per_request column is there because the shape of an attribution answer is rarely a large request. For one four-server fleet it read $0.0044 for the integration responsible for the bulk of a tripled bill — cheaper per call than any other integration on the same account. Nothing about any individual request was anomalous. The count was, and a report that only ranks by size would have pointed at the wrong server.

The costUsd in each record is computed locally from a price table, which means it will drift from the invoice and the drift has to be managed rather than ignored. Prices change, cached input is billed at a different rate from fresh input, and providers round in ways your arithmetic does not. Reconciling the sum of computed costs against the invoice once a month and expecting agreement within a few percent keeps the attribution honest; a gap that opens to twenty percent usually means the price table missed a model or a discount tier and every per-server figure derived from it is proportionally wrong. Versioning the price table and stamping its version on each record makes that correction retrospective rather than destructive, because you can recompute a past month instead of arguing about it.

Three alerts on the same records#

Detection is a query on those records, run continuously, per server rather than in aggregate. Three alerts cover most of the ground, and they are deliberately crude, because a sensitive alert on a signal nobody has calibrated is an alert that gets muted.

The first is a burn-rate alert: a server whose spend in the last hour exceeds three times its trailing seven-day p95 hourly spend, or exceeds ten percent of its 24-hour budget, pages the integration's owner. The second is a new-behaviour alert on the first sampling request from a server that has never sent one, or the first request from any server whose systemPromptDigest has not been seen before, which catches an integration that changes what it is asking the model to do in a version bump. The third is a volume-shape alert on representedCount, since an identity re-presented more than ten times in ten minutes is a loop and is worth knowing about before it is worth money.

Catching the next incident inside an hour rather than at the end of a month is the whole purpose, so the two thresholds in the first alert deserve a sanity check. A server burning its budget evenly would spend one twenty-fourth of it in an hour, a little over four percent, which puts a threshold of ten percent in an hour at roughly two and a half times the flat rate — loose enough that a busy afternoon pages nobody, tight enough that a runaway loop trips inside its first hour. The other clause, three times the trailing seven-day p95 hour, catches a different server: one whose budget is generous and whose behaviour has changed anyway. Each clause alone has a blind spot, which is why both are there. A multiple of a server's own history says nothing about a server that was always expensive, and a fraction of a budget says nothing about a server whose budget was set too high in the first place.

A global cap is a fuse#

The reflex when a bill triples is to ask the provider for a hard monthly spend limit with an alert at eighty percent. That is worth having as a backstop and it does not address any part of the problem above.

The blast-radius row is the one that decides it in practice. A fleet-wide cap that trips at midday on the 24th of the month stops the misbehaving server and also stops the support agent every user relies on, so the operational instinct is to raise the cap rather than let it trip, and a limit that is always raised is not a limit. A per-server budget that trips takes the offender offline and leaves the product working, which means it can be set at a level that actually bites.

The circuit breaker is the piece the earlier chapters did not cover, and it exists because a rolling budget has the same defect on a longer timescale. When a server crosses a threshold of denials or re-presentations in a window, sampling for that server switches off until a human turns it back on, and the switch is recorded with the reason. A budget that resets every 24 hours means an unfixed loop costs you the budget every day indefinitely; a breaker means it costs you once.

Aggregate totals and per-server rows still have to be reconciled, and doing it is what turned four rows into a conclusion rather than a suspicion. Ticket-triage's spend divided by its request count is a fraction over four tenths of a cent, which is the usd_per_request figure quoted earlier arrived at from the opposite direction — a sum and a count, computed without consulting the price table, landing where the price table said they would. The four rows do not add up to the invoice, and they should not. Eight thousand six hundred and ten plus two hundred and forty plus one hundred and seventy plus twelve is a little over nine thousand dollars against a bill of $12,940, and the missing four thousand is the application's own inference on the same API key, which no sampling record describes. What does close is the increase: the month rose by about eight thousand seven hundred and sixty dollars, ticket-triage accounts for eight thousand six hundred and ten of it, and the residue left over is smaller than four percent of the previous month's bill — which is to say smaller than the user growth alone would explain, and that is the sentence that lets a team stop looking.

Records also do something no alert does, which is make the postmortem short. An unexplained invoice became a named server, a named version, a mechanism and a fix in the time it took to run one query, and that is the difference the column makes. It is worth being precise about why: the column is not analysis, it is the join key that makes analysis possible, and it cannot be added retroactively to traffic that has already been served. A fleet running without it is not short of dashboards, it is short of one string per request.

What none of it establishes is that the controls work. Every record in that store describes a request the client actually handled, so the store is evidence about traffic that arrived, and it is silent on the traffic nobody has sent yet. Whether the context ceiling holds against a server that is trying, whether a denial survives an adversary who paraphrases, whether an extraction attempt produces an alert that reaches anyone — none of those are observable from production data, because a well-behaved fleet generates no examples of them. They have to be generated deliberately.