Share without its rate of change is a ranking, not a detector

At 14:02 on a Wednesday, p95 latency across every tenant on the platform went from 610 ms to 1,340 ms. At 14:03 the provider began returning 429s against the account's output-token ceiling, which the gateway retried into a growing queue. At 14:06 it stopped, because a customer's engineer had started a one-off backfill over their document archive — a batch run that pushes an accumulated history through a feature that normally sees one item at a time — and it had finished. Nobody was paged; the incident was reconstructed the following morning from a support ticket filed by a different customer.

The retrospective went looking for the dashboard that should have caught it and found two that had watched the whole thing without objecting. Aggregate token throughput across the platform, plotted in five-minute buckets against the previous week's range, showed 1.42 million tokens per minute during the incident window, comfortably inside a band that ran from 0.9 to 1.9 million and peaked on Mondays. Per-tenant request rates showed the backfilling tenant at 410 requests per minute, against their own ordinary range of 200 to 600. Both numbers were measured correctly and both were reported correctly.

Two honest dashboards with the same blind spot#

The aggregate view failed for reasons of resolution and reference. A four-minute excursion averaged into five-minute buckets is diluted by the minutes on either side of it, and comparing the result against a week-wide band means comparing it against a range that already contains Monday's legitimate peak. Any signal narrower than the bucket and smaller than the band's width is arithmetically incapable of standing out. The backfill added roughly 580,000 tokens per minute for four minutes, which is a large number in absolute terms and a small one relative to a band 1,000,000 tokens wide.

The per-tenant request rate failed for the reason chapter 1 established at the enforcement layer, now recurring at the observability layer. The backfill did not send unusually many requests; it sent ordinary numbers of unusually large ones. Their tokens per request went from 118 to 1,512 while the request count barely moved, so a chart of requests per minute per tenant is a chart of the one quantity that did not change.

tenant=globex   14:01  req/min 340   tok/min  40,100   share  3.3%
tenant=globex   14:02  req/min 372   tok/min 214,600   share 15.1%
tenant=globex   14:03  req/min 401   tok/min 486,900   share 28.4%
tenant=globex   14:04  req/min 410   tok/min 619,700   share 34.9%
tenant=globex   14:05  req/min 388   tok/min 601,200   share 34.1%
tenant=globex   14:06  req/min  95   tok/min  22,400   share  1.9%

Read the token column and the incident is unmissable; read the request column and there is nothing to see. Neither dashboard was lying, and the retrospective's conclusion — that the platform needed better alerting — was too vague to act on until somebody wrote down which quantities the alert would read.

How one tenant's tokens become everyone's latency

The coupling deserves to be spelled out, because "noisy neighbour" — the borrowed name for one tenant on shared infrastructure degrading service for the others without touching them directly — is often used as though the mechanism were obvious. The provider meters output tokens per minute against the account, not against a tenant, so globex's 620,000 tokens per minute pushed the account's output meter past its 400,000-per-minute ceiling and the provider began refusing whichever request arrived next. That refusal is indiscriminate: a tenant sending a 40-token classification got the 429 as readily as the backfill did, because the account is the unit the provider limits.

The gateway then made it worse in the ordinary way. Its retry policy re-sent 429s with backoff, which added roughly 18% more requests against a ceiling that was already the constraint, and each retry held a connection slot for the duration of its wait, so pool utilisation went from 62% to 97% and requests began queueing before they were even dispatched. Per-tenant in-flight caps did nothing, because each tenant was inside their own; the quantity that had no owner was the total. Latency for the other thirty-nine tenants was therefore a second-order effect of a first-order fact — one account's meter, one shared pool, and no aggregate control between them. That absence is the subject of chapter 7; what this chapter owes is noticing it within ninety seconds rather than the next morning.

Three signals, and why each is useless alone#

Detection of a noisy neighbour needs to answer three questions in sequence: who is taking the capacity, how fast their claim is growing, and how long before it runs out. Each question maps to one signal, and each signal on its own produces either a flood of alerts or a silence.

The composite is what makes each one meaningful. Share identifies a candidate, its derivative separates a step change from the ordinary daily shape, and headroom decides whether the step change is an incident or a capacity-planning note to be read on Monday. The same 30-point share increase is a page when the account has four minutes of headroom left and an email when it has 45% spare.

This is exactly where the obvious rule fails, and it fails in both directions at once. Alerting when any tenant exceeds a fixed 30% of platform tokens looks reasonable until you plot your actual customer distribution, which in most B2B platforms is heavily skewed.

The first row is why the fixed threshold gets muted within a week of being deployed, and the third row is why muting it matters. A tenant who goes from 2% to 20% in four minutes has multiplied their claim tenfold and consumed a fifth of the platform without ever approaching the line.

Writing the detector#

The signals are all derivable from one counter, provided it is labelled by tenant and incremented with settled tokens rather than requests. Recording rules — named expressions your metrics system evaluates on a fixed interval and stores as new series, so the alert reads a precomputed number rather than recomputing it — keep the alert expression readable and the query cheap.

groups:
  - name: llm_capacity
    interval: 15s
    rules:
      - record: tenant:tokens:rate1m
        expr: sum by (tenant) (rate(llm_tokens_settled_total[1m])) * 60

      - record: platform:tokens:rate1m
        expr: sum(tenant:tokens:rate1m)

      - record: tenant:token_share
        expr: tenant:tokens:rate1m / platform:tokens:rate1m

      # how far the share has moved in the last two minutes, in points
      - record: tenant:token_share:delta2m
        expr: tenant:token_share - (tenant:token_share offset 2m)

      # fraction of the provider account ceiling still unclaimed
      - record: platform:headroom
        expr: 1 - (platform:tokens:rate1m / 2000000)

      # minutes until the ceiling at the current first derivative of burn
      - record: platform:minutes_to_ceiling
        expr: |
          (2000000 - platform:tokens:rate1m)
          / clamp_min(deriv(platform:tokens:rate1m[3m]) * 60, 1)

The alert then joins the three, and the shape of the condition matters more than the constants in it.

- alert: TenantCapacitySurge
  expr: |
    tenant:token_share:delta2m > 0.08
    and tenant:token_share > 0.10
    and on() platform:headroom < 0.25
  for: 90s
  labels:
    severity: page
  annotations:
    summary: >-
      {{ $labels.tenant }} share {{ $value | humanizePercentage }},
      +{{ with printf "tenant:token_share:delta2m{tenant='%s'}" $labels.tenant
         | query }}{{ . | first | value | humanizePercentage }}{{ end }} in 2m;
      platform headroom
      {{ with query "platform:headroom" }}{{ . | first | value
         | humanizePercentage }}{{ end }}

Three constants there deserve to be argued about rather than copied. The 8-point delta over two minutes is set from your own distribution — plot the historical distribution of two-minute share deltas across all tenants and take a value above the 99.5th percentile, which for most platforms lands between 5 and 10 points. The 10% floor on share suppresses the small tenant who quadruples from 0.3% and is arithmetically dramatic and operationally irrelevant. The for: 90s is the flap guard, and it is also the reason this alert alone cannot be your only defence: ninety seconds of a 600,000-token-per-minute surge is 900,000 tokens spent before anybody is woken.

The counter feeding all of this has to be incremented at settlement with actual tokens rather than at admission with reserved ones. Reservations are inflated by whatever margin chapter 4's estimator carries, they are returned in a lump when the response lands, and a series built from them oscillates enough that a two-minute delta becomes noise. Actual settled tokens lag by one generation, which is around thirty seconds at the top of the range and is the same lag the incident itself has, so nothing is lost.

Cardinality is the other implementation constraint, meaning the number of distinct label combinations on a metric and therefore the number of separate time series the store has to keep in memory. Labelling the counter by tenant, route and model gives 40 × 12 × 4 series at this scale, which is negligible, and the same scheme at 4,000 tenants is 192,000 series on a single counter and will be felt. The usable split is to keep the share and delta series labelled by tenant alone, since those are what the alert reads, and to leave the route and model breakdown in the logs or in a lower-resolution rollup that only gets queried during an investigation. Scrape interval matters too — that is how often the monitoring system polls each process for its counters, and it sets the resolution of everything computed from them. At 15 seconds a two-minute delta rests on eight samples, which is enough to be stable, while a 60-second scrape gives it two and makes the derivative jump around. One more subtlety applies if you run more than one provider account — share has to be computed per account pool, because a tenant taking 40% of a pool they are alone on is not competing with anyone.

A second, slower rule belongs beside it with a page severity of ticket, firing when a tenant's share grows by more than 10 points over an hour with no headroom condition at all. That catches the customer who ramps a new workload deliberately over a week and needs a conversation about their contract long before they need a throttle.

Detection is not containment, and some of containment stays manual#

Ninety seconds of warning is only useful if something acts within it, and the actions divide cleanly into the ones a machine should take and the ones it should not.

The automatic branch is narrow on purpose. Clamping a tenant back to their contracted rate is safe because it takes nothing away that was sold to them — the extra throughput they were enjoying existed because the platform is oversubscribed and had spare room, and withdrawing it returns them to the number in their contract. Shedding non-interactive routes into a batch lane, and tightening burst capacity so the surge cannot re-form at full amplitude, are similarly reversible and similarly bounded. All three should emit an audit event and a notification to the customer, because a throttle that arrives without explanation generates a support escalation that costs more than the incident.

What should not be automatic is anything that cannot be undone by lifting a flag, and anything that changes the commercial relationship. Blocking a tenant to zero turns a latency incident into an outage for a paying customer on the strength of a two-minute derivative, which is a bad trade whenever the surge is a legitimate product launch. Automatically raising a limit in response to demand is worse, because it converts a detector into an amplifier and hands the account ceiling to whoever generates the most load. Permanent limit changes, contract renegotiations and provider quota increases all belong to a human with the tenant's history in front of them, and the automation's job is to buy that human the time to look.

The signals here also do not tell you whether the limits themselves were sensible. A platform whose per-tenant limits sum to several times the account ceiling will fire this alert constantly, correctly, and to no useful end, because the surge that triggers it is a tenant claiming exactly what they were granted. Chapter 7 takes a beta usage distribution and works out what the numbers should have been.