A 25-minute suite that is red one run in twelve gets switched off

The proposal is to run the skill evaluation on every pull request. It is 200 agent invocations, takes 25 minutes wall clock, costs about $18 in model calls, and roughly one run in twelve comes back red for a reason nobody can reproduce on a second attempt. The team merges around 40 pull requests a week.

Work forward from those numbers. Forty runs a week at a one-in-twelve spurious failure rate is between three and four red builds a week that mean nothing, each of which costs somebody a 25-minute re-run and a context switch, and roughly one in three of which will need two re-runs. By the third week the pattern is legible to everyone and the phrase "just re-run the evals" is in the team's vocabulary. Somewhere around week five a release is blocked on a Friday afternoon by a red eval on a change to a CSS file, and a skip-evals label appears. By week eight the label is on most pull requests, or the job has continue-on-error: true, or it has been moved out of the required checks and nobody looks at its output.

That prediction is the design input. A suite that gets disabled has exactly zero value no matter how well its assertions are written, and the ten weeks it survives are worse than useless because during those ten weeks the team believes it is protected. Any design that does not survive the arithmetic above should be discarded before it is built.

Split by what each stage can actually decide#

The suite contains checks with very different cost and reliability profiles, and running them as one job means the whole thing inherits the cost of the slowest and the flakiness of the least stable.

The first row is the one most teams skip and it carries more weight than its runtime suggests. The post-condition functions from chapter three are ordinary code, and they can be unit-tested against recorded run fixtures — a saved workspace and tool call log from a run known to violate the rule, and one from a run known to satisfy it — with no model invocation at all. That test catches the regression where somebody loosens a regex and the compliance number jumps six points for reasons that have nothing to do with the skill. It runs in under a second and it can never be flaky.

The second row is deliberately weak, and that is the point. Forty cases resolve differences of about fifteen points, so its gate is set at fifteen points and it exists to catch the catastrophic edit — a description accidentally emptied, an exclusion clause that swallows the whole domain — rather than to adjudicate a two-point movement. A blocking check whose job is to catch disasters can be blocking, because it almost never fires. A blocking check whose job is to adjudicate small differences fires all the time and gets removed.

name: skills
on: pull_request
jobs:
  static:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pytest tests/skills/ -q          # 8s, no model calls

  routing-smoke:
    if: contains(github.event.pull_request.changed_files, 'skills/')
    runs-on: ubuntu-latest
    timeout-minutes: 8
    steps:
      - uses: actions/checkout@v4
      - run: python -m evals.run --subset smoke --runs 1 --gate wide

The nightly job is where the real measurement lives, and it is not a build status. It writes its four metrics and their deltas against the recorded baseline into a file, posts the deltas to the team channel, and opens an issue assigned to the skill's owner when a gate from chapter six is breached. Nobody's merge is blocked by it, which means nobody has an incentive to make it stop happening.

Retrying until it passes is not a fix#

The proposal that always arrives at this point is to retry failing cases twice and pass the case if any attempt passes. It removes the red builds, and it does so by changing what the number means.

If a case passes with probability p on any given run, best-of-three reports it as passing with probability 1 − (1 − p)³.

A case that genuinely works two times in five reports as passing four times in five. The metric has stopped estimating how often the skill works and started estimating whether it is capable of working at all, and those diverge exactly where the interesting cases live — the mixed cases from the previous chapter, sitting at 2/5 and 3/5, are the ones best-of-three flatters most. It also hides regressions in the range that matters most: a change that drops a case from 1.0 to 0.7 is a serious regression and reports as 97.3%.

The legitimate version of the same idea is to run each case a fixed number of times and score the proportion, which is the repeats discipline from chapter six. The difference is not cosmetic. Best-of-three stops at the first success and discards the failures; fixed repeats count every run, so a case at 3/5 reports as 3/5 and shows up as unstable rather than as green.

Genuinely unreproducible failures still need somewhere to go, and the answer is a quarantine list rather than a retry. A case that produces mixed results across three consecutive nightly runs moves to a quarantined set: still executed, still reported, excluded from the gate, with an issue attached. The size of the quarantine is itself a metric, and a quarantine that grows month over month is a suite decaying in slow motion. Fourteen quarantined cases out of 214 is a suite with a known weak spot; sixty is a suite nobody should be reading.

Zero temperature does not make an agent loop deterministic#

The other standing proposal is to set temperature to zero and pin the model version, on the theory that this removes the variance and lets the full suite stay on every pull request. Greedy decoding, which is what temperature 0 amounts to — always taking the highest-probability next token rather than sampling one from the distribution — does reduce variance, since re-running the A/A pairs from the previous chapter at temperature 0 moved the largest observed activation delta from 2.7 points to 1.6. It does not remove that variance, because most of what varies in an agent run is not the sampler.

Tool results differ between runs: a timestamp in a filename, a row count that moved, a git log with one more commit, a container that resolved a hostname on the second attempt. Timing differs, so a step that takes 51 seconds on one run and 62 on another crosses a timeout on one of them and the agent takes a different branch. Retries and any concurrency change the order in which results arrive, which changes the context the next decision is made from. And the model itself is not bit-identical across identical inputs at the provider level, because batching and kernel selection vary with load; greedy decoding makes divergence unlikely per token and an agent run is tens of thousands of tokens with branch points throughout.

Pinning the model version has a second problem beyond not working. It converts a continuous, small stream of variance into a single cliff on the day the pin is lifted, when the suite moves by an unknown amount for reasons entangled with every skill edit made since the pin. Baselines are already tied to a model identifier for exactly this reason, and the useful practice is to re-record against a new model deliberately, with no skill changes in the same commit, so the model's contribution is isolated.

The worst effect of the temperature-zero theory is what it does to the team's reading of red builds. Told that runs are now deterministic, people treat every red as real, investigate each one, find nothing about half the time, and arrive at the disabled job by the same route as before with more effort spent on the way.

Starting from where you are#

Nothing in this topic requires the full apparatus on day one, and building it in order is how it stays alive. An afternoon gets a long way: pull sixty logged requests for one existing skill and label them, pull forty requests that should not trigger it from the neighbouring skills' traffic, write one deterministic post-condition against the artefact the skill is supposed to produce, run it, and commit the resulting numbers as a baseline file with the corpus hash and model identifier in it. That is a real instrument, and it will tell you something uncomfortable about a skill you currently believe works.

What comes after is the loop rather than a project. Every edit to a description or a body is proposed with a delta against that baseline on all the metrics rather than the one being tuned, every new skill re-runs the baselines of the skills it might collide with, and the nightly run reports deltas to an owner who is expected to look. For a fuller treatment of scoring methodology than this topic has room for, the Practical Guide to Evaluating and Testing Agent Skills covers prompt-set design and harness structure in more depth, and SkillsBench is the closest thing to a published methodology for measuring whether a skill changes the output at all. Both assume the thing this topic has been building: a set of cases you did not write yourself, labelled in both directions, checked against artefacts rather than narratives, and compared to a number you wrote down before you started editing.