A suite that runs on your commits misses the provider's changes

Open the Actions history for the assistant service and the story of the previous chapter is written in it plainly. The evaluation workflow has 340 runs, all green, and the most recent one finished twenty-two days ago against the commit that changed a log line. Between that run and the support thread, the team shipped a design system upgrade to the web app, took a week of summer holiday, and spent nine days on a migration in a different repository. The evaluation suite did what it was configured to do throughout: nothing, because nothing triggered it.

This is not a gap in the suite's coverage. The 240 cases would have caught the change on any day of those twenty-two, as the three-arm run eventually demonstrated. The gap is in the trigger. A CI evaluation is edge-triggered on your merges — it runs at the moment a change of yours arrives and stays silent for as long as none does — which encodes an assumption that the system under test only changes when you change it, and that assumption held for every service any of us built before we started calling a vendor's inference endpoint in the request path. A pull request is a proposition about your code, and the suite exists to accept or reject it. Nobody opened a pull request when the alias started resolving somewhere new.

The provider's release calendar is exogenous. Snapshots ship, aliases are repointed, safety classifiers are retuned, and serving stacks are upgraded, all on a schedule set by commercial and research pressures inside another company and communicated, at best, in a changelog you are not subscribed to. Between your merges the system in production is drifting freely with respect to your test suite, and the length of that exposure is exactly the length of your quiet periods, which are longest in August and December and around any large migration — the same periods when your attention is furthest from this service.

Cadence has to come from the calendar#

The correction is to add triggers that do not depend on your merge activity, which in practice means three of them. Time is the obvious one, because a nightly or daily run bounds your exposure at the interval regardless of what the team is doing. Provider events are the second: a changelog entry, a deprecation notice, a status page incident, or an announcement that an alias will be repointed all justify a run within the hour rather than waiting for the next scheduled slot. The third is the cheapest and the most often skipped, which is a watcher on the resolved model identifier you started logging in the previous chapter.

That watcher costs nothing to run because it queries data you are already writing. Every production response names its concrete snapshot, so a distinct-value count over the last hour tells you the moment the alias moves, before any evaluation has run at all.

-- alert if today's traffic used a snapshot we have never seen before
SELECT response_model, COUNT(*) AS calls, MIN(created_at) AS first_seen
FROM llm_requests
WHERE created_at > now() - interval '24 hours'
GROUP BY response_model
HAVING MIN(created_at) > now() - interval '24 hours'
   AND COUNT(*) > 50;

It does not tell you whether the new snapshot is worse, only that the ground moved. That is enough to promote the next scheduled evaluation to right now, and it converts a class of incidents from "discovered by customers over nine days" to "discovered by a query within the hour". Not every provider change announces itself this way — a retuned safety classifier or a serving-stack change can leave the identifier untouched — so it is a high-precision, low-recall detector, valuable precisely because it is nearly free.

Provider events are worth wiring up concretely rather than relying on someone noticing. The changelog and status pages of every major provider are available as feeds, deprecation notices arrive by email to whichever address owns the account, and SDK release notes frequently mention a default that has moved. A small poller that turns any of these into a repository_dispatch — the GitHub Actions event that lets an outside system start a workflow over the API, with no commit involved — gives you a trigger with a latency of minutes, and the false-positive cost is one extra run of a 320-case set. The email is the one that goes missing, because it lands in a shared inbox that nobody reads; route it to the same channel as everything else.

# .github/workflows/drift.yml
on:
  schedule:
    - cron: "0 3 * * *"        # daily tripwire, 03:00 UTC
    - cron: "0 4 * * 0"        # full suite, Sundays
  repository_dispatch:
    types: [provider-event]     # fired by the changelog poller and the model-id watcher
  pull_request:                 # unchanged: the CI gate still guards our own changes

The nightly full suite is the wrong shape of expensive#

The natural first move is to point the scheduler at the whole suite and run it every night, and on a real suite the finance conversation arrives within a month. Take 5,000 cases where the median case is an agentic trace of four model calls, each re-reading a growing context of roughly 6,000 input tokens and producing 350 output tokens, plus a judge pass over the finished transcript. That works out around £0.08 per case, so the nightly run costs about £400, or £12,000 a month, which is a line item somebody will ask about.

The instinct when asked to cut it is to sample: run 1,500 of the 5,000 cases each night, chosen at random. That preserves the aggregate score's precision reasonably well and destroys the thing you actually need, because a uniform sample of a suite inherits the suite's composition. If 68% of your cases are English-language single-turn question answering, then 68% of your sample is too, and the Portuguese-language segment with 41 cases contributes twelve of them on a good night and four on a bad one. The regressions that hurt commercially are rarely uniform across segments; they land on long documents, or one language, or one tool, or one customer's document format.

Stratifying by purpose gets you more detection per pound because detection comes from covering the segments rather than from re-running every case. Enumerate the segments you would be embarrassed to regress on — say 40 of them, spanning languages, document types, tool paths, tenant tiers, and the three or four flows that carry contractual commitments — and take a fixed 8 cases from each, chosen to be the hardest ones that currently pass. That is 320 cases and about £26 a night. A set of that shape is a tripwire: it is sized to notice that something moved and to say roughly where, which is a weaker job than measuring how good the system is and is the only job a nightly run needs to do.

Eight cases per segment per night is far too few to judge a segment on, and the run is not asking you to. Scores accumulate into a rolling seven-day window, so each segment carries 56 fresh observations at all times, refreshed daily rather than weekly. The sensitivity is comparable to running the full suite once a week; the detection latency is a day. That difference — same statistical power, seventh of the delay — is what stratification actually buys, and it is why the daily lane is not simply a smaller version of the weekly one.

Total is roughly £780 of tripwire plus £1,600 of weekly runs a month, against £12,000 for nightly everything, and the stratified lane covers the Portuguese segment every single night rather than four cases at a time by accident.

Choosing the 40 segments is the part that takes judgement, and the useful question is not "what does our product do?" but "what would we have to explain?" Segments earn their place when a regression confined to them would be commercially or contractually visible: a language a customer contract names, a document type a large account depends on, a tool whose failure silently produces a wrong answer rather than an error, a flow that touches money or compliance. Enumerating those tends to produce between thirty and sixty for a mature product, and the list should be reviewed when the product changes shape rather than continuously. Within a segment, prefer cases that currently pass but only just, since a case that every model passes carries no information about degradation and a case that nothing passes carries none either.

The other operational detail that catches people is the scheduled job that quietly stops. A cron that fails to trigger produces no alert, because alerting is downstream of the run, and a suite that has not executed for nine days looks identical to a suite that has been green for nine days on every dashboard anybody built. A dead-man's switch fixes this: a separate check that fires when no completed run has been recorded in the last 30 hours. It is three lines and it is the difference between a monitoring system and the appearance of one.

What a schedule does not buy you#

A cadence bounds detection latency and nothing else. The exposure floor is the interval: a change that lands at 03:05 UTC is live for 23 hours and 55 minutes before the tripwire touches it, and during those hours the affected customers are getting the new behaviour. If a segment carries a commitment where a day of degradation is unacceptable, the answer is not a shorter interval — halving to twelve hours doubles the cost to halve an exposure that was already the wrong order of magnitude — but a guardrail in the request path, sampling live traffic against a check, which is a different mechanism with different economics.

The tripwire also does not attribute. When Tuesday's run comes back four points down on the invoice segment, the run has told you that something is wrong in a place, and the causes are not distinguished by the score: your retrieval index changed, the provider changed the model, the customer changed their inputs, or the prompt has quietly aged out of agreement with the product. Separating those requires a different manoeuvre, and running the tripwire more often will never do it.

The scheduled lanes do not replace the CI gate either, and the temptation to fold them together is worth resisting. The CI evaluation answers a question about a proposed change and blocks a merge; the scheduled evaluation answers a question about the world and pages a human. They share a harness and a case format and almost nothing else, including their thresholds, since a gate tuned to reject bad pull requests will be far too twitchy as a daily alarm — which is the subject two chapters from now.

The last thing worth being honest about is that a fixed set on a schedule is a fixed set. It ages: the 320 cases were selected against a product and a customer base that will not be the same in a year, and a segment that mattered in March may be a rounding error by autumn while a new one has appeared with no cases at all. Curation is recurring work, and a suite nobody has touched in eighteen months provides a comforting green tick over the wrong 40 segments.