Triggers and detection latency

Once a feature's checks are sorted into a deterministic pile and a sampled pile, the remaining decision is where each pile runs. That sounds like a scheduling detail and it is the decision that determines how long a quality regression sits in the main branch before anybody knows about it, how many commits are suspects when it is finally noticed, and whether the number an eval produces ever stops anything from reaching a customer.

One arrangement answers all three. A fast deterministic suite runs on every push and is allowed to block a branch, because it is cheap enough to run that often and its failures name a specific fault. A frozen-sample eval runs on every merge to main and is allowed to block nothing except the last step before production, because it reports a level rather than a fault and a level near a threshold is not evidence a particular merge is at fault. The two differ in what they are permitted to stop precisely because they differ in what their output means.

Detection latency is a property of the trigger#

A check tells you about the changes that landed since the last time it ran. If it runs nightly and a team merges thirty times a week, each run covers roughly six merges, so a red run identifies a set of six candidate commits rather than one. That six is worth deriving rather than accepting, because a reader dividing thirty by seven gets four and a third and concludes the chapter has slipped. The merges do not arrive across seven days. They arrive on the five days people are at work, and thirty spread over five working days is six a day, which is what one night's run covers.

Two separate quantities come out of a trigger, and separating them is the whole of this section. How long a regression hides is the time from the merge until the next run starts, plus the time that run takes to finish. How many commits are suspects when it goes red is the number of merges that landed inside that first interval. The check itself sets only one term in all of that — its own duration — and the trigger sets the rest.

With a clock-based trigger the first term follows from the interval. A merge lands at some point between one run and the next, with no relation to when the schedule fires, so its wait is anywhere from nothing at all to a full interval, and averages half of one. Nightly, that interval is a day, which puts the worst case at the twenty-four hours the table below prints and the typical case at half of it. Against that, the twenty-five minutes the run itself takes is a rounding error — which is the point. At a nightly cadence the speed of the check has stopped mattering at all.

Treating the landing point as uniform across the interval is generous rather than harsh here, and it is worth saying why. Merges cluster inside the working day and this run fires in the small hours, so almost no merge gets the short end of the interval; a change landing at any moment when people are actually working waits until the following night. The real average is therefore worse than half a day rather than better, and moving the schedule earlier only reshuffles which merges draw the long wait.

A trigger tied to the merge itself deletes that first term by construction. There is no interval to land inside, because the merge is the thing that starts the run, so the wait is zero and the latency is the run's own duration and nothing else.

Narrowing a set of candidate commits means bisecting — re-running the eval against candidate commits, in the manner of git bisect, until one reproduces the drop and the one before it does not. At 25 minutes and $6 a run, walking six candidates is two and a half hours and $36, assuming the first bisect ordering is lucky. Change the trigger to merge-to-main and each run covers exactly one merge, so a red run names the commit and the bisect cost drops to zero.

Walking the candidates one at a time is worth pausing on, because git bisect proper halves the range at each step and would need far fewer than six runs. Halving works when the verdict is a fact about the commit: green means clean, red means broken, and one run at the midpoint says which half to keep. A sampled score is not that. A run at a perfectly good commit can come back low from noise alone, and a halving decision taken on one such run sends the search into the half that does not contain the regression, with nothing later in the search to reveal the mistake. Running each candidate in turn, once, has no such failure mode, and six runs at 25 minutes and $6 apiece is what the nightly row prices.

Detection latency and bisect cost are the whole argument for the placement, and they are worth stating in units:

The pull-request row is the one teams reach for first, since it looks like the strictest option and therefore the safest. It is the next chapter's subject, and the short version is that its cost lands on the person least able to absorb it. Merge-to-main gets almost all of the detection benefit — a 25-minute delay rather than a day's — while the person waiting on the number is the team rather than an individual with a branch open.

The runs-per-week column shows the pull-request row's cost before any argument about who waits. The pull-request trigger fires on a push, not on a merge, and a branch takes several pushes before it merges — a review comment addressed, a rebase, a typo fixed — so the number of runs is the number of merges multiplied by the number of pushes per branch. Thirty merges become the ninety or more runs in the table, and the invoice multiplies alongside them, mostly to measure intermediate states of a branch that nobody will ever ship.

Two paths, and what each one is allowed to do#

The arrangement has a deterministic path allowed to block anything, and a sampled path allowed to block only the last step.

The shape is inherited rather than invented. Humble and Farley's deployment pipeline, where most modern continuous-integration conventions come from, splits checks into a commit stage fast enough to run on every change, an acceptance stage that is slower and runs on what the commit stage passed, and later stages for capacity and manual testing — with the explicit rule that a stage only ever runs on a binary the previous stage approved. Sampled evals slot into the acceptance stage almost exactly, including the awkward parts: acceptance tests were always the slow, flaky, expensive layer teams were tempted either to skip or to promote into the commit stage, and both temptations ended badly for the same reasons they do here. What is new with language models is that the slow layer now has a per-run invoice attached, which changes the arithmetic of how often it can be afforded but not the structure of where it belongs.

Three things in that shape do the work. The eval runs after the merge is allowed, so it never sits between an engineer and their branch. Its output is a number written to a durable place keyed by the commit sha — the hash naming exactly one commit — rather than a job status that exists only inside a continuous integration run and expires with the provider's log retention. And the release job does not run the eval; it reads the number that already exists for the commit it is about to promote, so a release costs nothing extra and takes no extra time while still refusing to ship a build whose measured quality is below the line.

Keying the number to the commit sha rather than to a job reads as bookkeeping and is not. A job status is a property of a run: it lives inside a continuous integration provider, it belongs to a workflow that can be renamed or deleted, and it expires on a retention schedule somebody else sets. The question asked at release time has a different subject — this build is about to go to customers, what did it score — and that is a question about a commit. If the score was only ever recorded as a job status, there is no way to answer it except by running the eval again, which is the twenty-five minutes and the six dollars the arrangement exists to avoid spending twice. Storing the score against the sha turns the answer into a lookup, and a lookup is what lets the release job be free.

Fresh in the release check is doing its own work, because a score can belong to the right commit and still be the wrong number to compare against a threshold. Two things age it. The suite may have changed since — a new case set, an edited judge prompt — in which case the stored figure came out of a different instrument and no longer means the same thing as the number in the condition. And the eval reads things the sha does not pin, such as a live index or a model alias, so a score can be an accurate record of what that commit scored last month and a poor prediction of what it would score today. Freshness is a claim about the measurement, not about the commit.

The release job consulting that stored number is the piece teams most often skip, and skipping it turns the whole arrangement into a dashboard. An eval that posts a number to a channel with no consequence attached anywhere is watched attentively for three weeks and then becomes wallpaper. Attaching the consequence to promotion — the step where a build that already exists is put in front of customers — rather than to merge keeps the consequence real while keeping the merge path fast, and it has the pleasant property that a regression landing on Monday is caught by the Tuesday release job even if every human missed the alert.

Why promotion is the cheap place to put the consequence comes down to what each step is holding when it is blocked. A blocked merge stops a person: their branch is open, the change is loaded in their head, and every minute the check takes is a minute of theirs. A blocked promotion stops an artefact that already exists and that nobody is currently waiting behind — the delay is absorbed by a release schedule rather than by an individual. The same refusal protects the same customers from the same build, and it costs a great deal at one step and very little at the other.

The sample has to be frozen for the number to mean anything#

The word doing quiet work in "frozen-sample eval" is frozen. Two runs are comparable only if they were scored on the same cases. A set regenerated from last week's production traffic before each run produces a score that moves for reasons having nothing to do with the code — one assistant's score dropped 5 points across a weekend purely because Saturday traffic skews toward returns questions, which it handled worse, and two engineers spent a day looking for a regression that did not exist.

That failure is the same elimination argument the first chapter of this topic made about a red X, applied to a number rather than to a binary. A difference between two eval scores has two candidate explanations: the system changed, or the cases changed. Freezing the cases removes one of them, so a difference has a single explanation left and the score becomes attributable to the code. Regenerate the set before every run and both terms move at once, after which no amount of care in reading the number can separate them, because the information needed to separate them was never recorded. The weekend drop is exactly that: five points of case-mix difference, indistinguishable at read time from five points of regression.

Freezing means the case list, the expected-behaviour labels, the judge prompt and the judge model are all pinned in the repository and versioned with the code:

# evals/order_assist/suite.yaml
version: 7
dataset: datasets/order_assist_v7.jsonl   # 400 cases, sha256 e3c1…9ab4
generation:
  model: gpt-4o-mini
  temperature: 0.2
  draws_per_case: 1
judge:
  model: claude-sonnet-4-5
  prompt: judges/order_answer_v3.md        # sha256 71fd…22c0
  temperature: 0
thresholds:
  block_release_below: 78.0
  alert_below: 78.0
  alert_on_drop_from_trailing_mean: 4.0

Changing any of those is a change to the measuring instrument, so it goes through review like a schema migration, and the run record stores the suite version alongside the score. Without that, a score of 74 six weeks from now is not comparable to the 80.6 being treated as the baseline, and nobody will be able to tell whether the assistant got worse or the rubric got stricter.

The wiring itself is unremarkable once the trigger is right:

# .github/workflows/eval-on-main.yml
name: eval-on-main
on:
  push:
    branches: [main]

jobs:
  frozen-sample-eval:
    runs-on: ubuntu-latest
    timeout-minutes: 45
    concurrency:
      group: eval-main          # one at a time; queue rather than fan out
      cancel-in-progress: false # never cancel: every merge needs its own number
    steps:
      - uses: actions/checkout@v4
      - run: uv run evals run evals/order_assist/suite.yaml --sha "$GITHUB_SHA"
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      - run: uv run evals publish --sha "$GITHUB_SHA" --channel orders-assist
        if: always()

cancel-in-progress: false matters more than it looks, and the default it overrides is not careless — it is correct for the question most continuous integration jobs answer. A build or a test suite is answering "is the current head of main good", and when a newer commit arrives, the answer being computed about the older one has been superseded and is worth nothing, so cancelling it saves money and loses no information. An eval on this trigger answers a different question: what did this commit score. That is a per-commit fact, it is the fact the release job will later look up by sha, and nothing else in the system will ever produce it again. Cancelling the run does not supersede that fact, it destroys it. Leaving the default in place therefore means that on a busy afternoon three merges share one score and you are back to bisecting.

The concurrency group keeps runs serialised so the queue drains at roughly one every 25 minutes, a little over two an hour. That is comfortable against six merges in a working day, which need two and a half hours of eval time out of a day that is longer than that, and it is a real ceiling rather than a figure of speech: a burst arriving faster than one merge every 25 minutes builds a queue, and the last merge in the burst waits behind every run in front of it before its own number exists.

Somebody specific has to find out#

A number posted where no one is accountable for it decays into background noise within a month, so the trigger question is only half the design. The trigger settles when the number comes into existence; it settles nothing about when anybody acts on it, and those are two different latencies that add up. The other half of the design is who is told, by what condition, and with what in their hands.

The condition is worth being precise about. Posting every run to a channel is reporting, and reporting is useful — the trailing series is how slow drift gets noticed — but it is not an alert, because a message arriving thirty times a week teaches people to scroll past it. The two alerting conditions in the suite file above are an absolute score under 78.0 and a drop of more than 4 points from the trailing mean of the last ten runs. The second catches an assistant degrading from 88 toward 80 without ever crossing the absolute line, which is what a slow retrieval regression looks like before it becomes obvious. Both page the owning team rather than a shared platform channel, because the merge that caused it belongs to that team and nobody else can act on it.

The two conditions catch different shapes because they compare against different things. An absolute threshold asks whether the system is acceptable right now, and it stays silent about everything above the line however fast the score is moving: the slide just described gives up most of the distance between where the assistant was and where the floor is, and trips nothing. A relative condition asks whether the system has changed, which is the question a regression detector was built for, and it fires part-way down.

Comparing against the trailing mean of the last ten runs, rather than against the previous run, is the detail that makes the relative condition usable. A single previous score carries the full run-to-run noise of the instrument, so comparing one noisy number against another noisy number widens the spread you are trying to see a signal through, and a threshold tight enough to catch a real slide would then fire constantly on nothing. Averaging ten runs gives a steadier reference line, so the comparison becomes a noisy number against a stable one. The cost is lag, and it is worth knowing about in advance: once a regression has been in main for ten runs it is inside the mean, the reference line has come down to meet the score, and the condition goes quiet on a system that never recovered. A relative alert detects transitions. It cannot tell you where you are.

What the alert carries decides whether it gets acted on. A message reading "eval score 71.2, threshold 78.0" gets acknowledged and deferred. One that names the merge commit and its author, shows the previous run's score for contrast, and links to the five cases whose verdict flipped between the two runs gives the author something to open. That last element is the expensive one to build and the one that makes the difference, since the difference between two runs is almost always concentrated in a handful of cases, and reading them takes four minutes where staring at a score takes an afternoon.

A merge-triggered eval does not block the merge#

The arrangement does not prevent a bad merge. A regression landing at 14:00 is in main at 14:00 and will still be in main at 14:25 when the eval reports. What changes is that somebody knows, the commit is named, and the release job will refuse to carry it forward. Teams that need the merge itself blocked are asking for the pull-request trigger, and they should read its cost honestly before choosing it.

It also does not remove the case for a nightly or weekly run. A frozen sample small enough to run thirty times a week is necessarily smaller than the set you would ideally measure against, and there is a larger set — 3,000 cases, an hour, $45 — worth running once a week to catch slower drifts and to check that the frozen sample has not diverged from the traffic it is supposed to represent. The merge-triggered run is a regression detector; the weekly run is a measurement.

The invoice and the wall-clock quoted for that larger set move differently, and noticing why is what makes the weekly run affordable at all. The larger set is seven and a half times the 400 cases, and it costs seven and a half times the $6 — the $45 is that multiplication and nothing more, because every case buys its own generation and judging tokens and a bigger set makes no case cheaper. Wall-clock does not follow the same line. The cases do not depend on one another, so they can be run many at a time, and the hour is a statement about how much concurrency the run is given rather than about how much work it contains. That is the general shape for anything sampled: the invoice is fixed by the number of cases, and the clock is a scheduling decision laid on top of it.

A merge-triggered eval also says nothing about a change that did not come through a merge. A prompt edited in a hosted console, a model alias resolving to a new version, a knowledge-base article rewritten by the support team: none of those produce a commit, so the first signal is whatever runs on a clock, which is the strongest argument for keeping a scheduled run alongside the triggered one.

That follows from what a trigger is. A trigger is a subscription to an event, and the event this one subscribes to is a commit landing on main. A change producing no commit produces no event, so no run fires, and the detection latency for that entire class of change is neither twenty-five minutes nor a day. It is however long it takes until something fires for an unrelated reason — the next merge, perhaps, or nothing at all in a quiet week. A run on a clock has the opposite property. It fires whether or not anything happened, which is wasteful when measured against commits and is exactly what is wanted against changes nobody committed.

What the placement buys is bounded. Detection drops from a day or more to roughly the length of one run, attribution drops from a window of merges to a single named commit, and the number acquires a consequence at promotion without ever standing between an engineer and a branch. None of that comes from a better eval; all of it comes from running the same eval on a different trigger and storing its output somewhere durable.

What the arrangement still rests on is an unexamined 78.0 and an alert on a 4-point drop. Where those numbers come from, and whether a 400-case sample can even resolve a 4-point drop, are questions this chapter has deferred. Before them comes the option set aside above: running the eval on every pull request, which is what most teams build first, and what happens to it about three weeks later.