Binding an eval score to a commit

Continuous integration, universally shortened to CI, is the shared automation attached to a repository: whenever code is pushed, a hosted machine checks out that exact commit, installs a declared set of dependencies, runs whatever jobs the repository asks for, and reports each job back onto the pull request as a check that is either green or red. Most backend teams already have a pipeline of this kind running their unit tests and their linter, which is the mental model to carry in. Evaluation in CI means putting the eval suite — the cases, the model configuration, the grader, the whole apparatus the previous topic was about selecting — into that same automation, so that the score is produced by the machine against the commit under review rather than by a person on a laptop against whatever happened to be on it.

When the pipeline goes one step further and compares that score against a number committed to the repository, exiting non-zero when it falls short, the check becomes an eval gate: a required check that has to be green before the merge button is available, in the same way a failing test suite blocks a merge today. The number being compared against is the pass threshold, and it lives in a file rather than in a reviewer's memory. Where all of this shows up in practice is the checks list at the bottom of a pull request: alongside build, test and lint there is a check named something like eval; it takes a few minutes, it posts a score and a link to a stored result file, and while it is red the merge button is disabled. Everything in the rest of this topic is a decision hiding inside that one line — which number the check compares against, what happens when it goes red for a reason that is not a regression, what it costs to run, and what else it ought to be measuring while it has the machine running anyway.

The inputs a score is relative to#

Every measurement in an eval run depends on inputs that can each move independently:

• the code under test, which is the only one anybody thinks about
• the dataset — which cases, at which version, with which expected outputs
• the model, including the exact version string rather than a floating alias
• the grader, whether that is a set of assertions, a scoring function or a judge model with its own prompt and its own model version
• the run configuration — temperature, sample count, retries, concurrency, timeouts, and whether skipped cases fail loudly or quietly

A score is only interpretable relative to all five. When any of them is unpinned, two runs that report the same number can be measuring different things, and two runs that report different numbers can be measuring the same thing badly. Those two failures run in opposite directions, and a reviewer reads both of them as facts about the code.

The first happens when something other than the code moves between two runs. A colleague adds cases to the case file in the morning; the run from before their commit and the run from after come back at the same score, and the pair reads as a change that made no difference. What it says instead is that one average was taken over one set of cases and the other over a larger set containing cases nobody had scored before, and two averages over different populations are not comparable in either direction. Their being equal is a coincidence of arithmetic. The comparison the reviewer thinks they are making needs the two runs to differ in the code and in nothing else, and a bare score records nothing about whether that held.

The second failure wastes days rather than hiding a problem. The code is identical across two runs and the numbers differ, because a model alias resolved to a newer version overnight, or the judge was called at a different temperature, or a retry setting changed which responses got a grade at all. The score drops, the only variable visible in the pull request is the change under review, and the change is blamed and reworked. Nothing in the number separates that from a real regression, since a mean over graded cases has the same shape whichever of the five inputs produced it.

The property being lost has a name in ordinary build engineering, which is provenance: the ability to point at an output and reconstruct exactly which inputs produced it. It prevents neither failure. What it changes is that both become detectable afterwards — the two runs that agreed can be shown to have read different case files, and the two runs that disagreed can be compared input by input until the one that moved is found. A score carrying no provenance offers nothing to compare, which is why disagreements about what caused it get settled by argument rather than by evidence.

The reason this feels unfamiliar in eval work is that most teams arrive at it from the notebook side rather than the build side. The eval started as a script somebody ran to answer a question, and it kept the ergonomics of a script long after it had been promoted to a merge criterion.

The last row is worth reading carefully, because the point of moving the run into CI is not that developers are suspected of lying. The point is that CI produces the number from the same inputs every time without anyone having to remember to, and it produces evidence alongside it. A typed number stops being the only copy of the result, and a claim that can be set beside an independently produced one needs nobody's honesty to be assessed.

There is a second effect that matters as much and gets less attention. A local run happens on a machine that has been accumulating state for months — a Python environment upgraded piecemeal, an API key with a different rate limit tier, a retrieval index pointed at a development bucket somebody reindexed by hand in June, a proxy that quietly caches. Each of those can shift a score by several points without anyone touching the product code. A CI runner starts from a declared image with a declared dependency set, so when the number moves, the set of things that could have moved it is small enough to enumerate.

What binding actually looks like#

The mechanism is unremarkable, which is part of why it gets skipped. A workflow triggered on pull requests checks out the code, runs the suite, writes a machine-readable result, and exits non-zero if the score is under the threshold.

# .github/workflows/eval.yml
name: eval
on:
  pull_request:
    branches: [main]

jobs:
  eval:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements-eval.txt

      - name: Run eval suite
        env:
          ANTHROPIC_API_KEY: ${{ secrets.EVAL_API_KEY }}
          EVAL_COMMIT: ${{ github.event.pull_request.head.sha }}
        run: |
          python -m evals.run \
            --config evals/config.yaml \
            --cases evals/cases.jsonl \
            --out results.json \
            --fail-under 0.80

      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: eval-results-${{ github.event.pull_request.head.sha }}
          path: results.json
          retention-days: 90

Two details in that file do most of the work. --fail-under 0.80 puts the threshold in the pipeline rather than in a reviewer's head, so the pass criterion is version-controlled and changing it is a reviewable diff. if: always() on the upload means the results survive a failing run, which is the only case where anyone urgently wants to read them. The flag is needed because a step following a failed one is skipped by default, there being no point continuing a broken build; the upload is the exception, since the results file is the only record of which cases moved and the default behaviour would throw it away at exactly the moment it became evidence.

A third line does quieter work. EVAL_COMMIT passes the head SHA of the pull request into the harness, which is how a commit identifier reaches the results file without anyone typing it: the runner is the only party that knows for certain what it checked out. Asking the clone it is standing in would give a different answer.

That answer follows from what a pull request is asking. A pull request proposes a change to a branch — main, in the workflow above — and the question a reviewer needs settled is how main will behave once the change is in it. The branch head does not hold that state, since it is missing whatever else has landed on main since the branch was cut. The base does not hold it either, since the base is missing the change. The state being asked about belongs to no commit on either side, and the only way to obtain it is to perform the merge.

So the forge performs it: when a pull request is opened or updated, the server merges the head into the current base and keeps the result on a reference of its own, which is what actions/checkout fetches by default on a pull_request trigger. On a pull request, the default checkout is therefore not the branch head but a merge commit of the head into the current base, which is usually what you want, because it is the code that would exist after merging. Inside that checkout, git rev-parse HEAD answers with the merge commit rather than with the head of the branch under review, which is why the head SHA is passed in explicitly rather than read back out.

That commit was constructed by the server and lives there. It is also not a commit that exists in anyone's local clone: it is not on the branch, it does not appear in git log on a developer's machine, and an ordinary fetch of the branch does not bring it down. Two things follow. The merge commit is not stable, because it is recomputed whenever either side moves, so the SHA one run recorded last week need not be what that reference points at today. And a developer reproducing a CI failure by running the suite on their branch is running a slightly different experiment whenever the base has moved.

Developers report that a failure does not reproduce locally, and on a branch open for any length of time that report is true and uninformative at once: the two trees differ by exactly the commits that landed on the base in between, a difference nothing in the working copy displays. Recording both the head SHA and the base SHA, as the manifest below does, is what lets someone recreate the exact tree later. Neither alone would do it, because the head SHA names the change without the code it was merged into and the base SHA names that code without the change. The pair determines the merge, so anyone holding both can redo it and get the tree the run scored — provided the merge was clean, which it must have been for CI to have had a tree to check out at all.

The results file matters more than the workflow. A single float is not evidence; a run manifest is.

{
  "score": 0.87,
  "threshold": 0.80,
  "passed": true,
  "run": {
    "commit": "a3f91c2e5b7d4408f1c9e2a6b3d5f7091c4e8a2b",
    "base_commit": "7d1e4a09bb2c3f5580a6e1d4c7b9f2038e5a1c6d",
    "workflow_run": "https://github.com/acme/assistant/actions/runs/8823410551",
    "started_at": "2026-08-08T09:14:02Z",
    "duration_s": 412
  },
  "dataset": {
    "path": "evals/cases.jsonl",
    "sha256": "9f2c...b41e",
    "case_count": 120,
    "cases_run": 120,
    "cases_skipped": 0
  },
  "model": {
    "id": "claude-sonnet-4-5-20250929",
    "temperature": 0.0,
    "max_tokens": 1024
  },
  "grader": {
    "kind": "llm_judge",
    "prompt_sha256": "4d81...77aa",
    "model": "claude-sonnet-4-5-20250929"
  },
  "results_sha256": "e70a...92c1"
}

The run configuration bullet above hides a policy decision that this manifest forces into the open, which is what the harness does when a case cannot be loaded or cannot be run. A missing fixture, an unreachable retrieval index or a malformed line in the case file are all conditions a script written for interactive use handles by printing a warning and carrying on, because the person at the keyboard will see it. That is the whole of a warning's power: it does nothing on its own, and everything it achieves it achieves by being read by somebody who then declines to trust the number under it. Nobody is at the keyboard in CI. The log there is opened once a check has gone red, which is the one case in which the score was not going to be believed anyway, so the warning has no reader at the moment it would have mattered. The only channel that reaches a person unconditionally is the exit code, because the forge turns a non-zero exit into a red check and a red check into a disabled merge button.

What the warning would have been warning about is a change to the denominator. With nobody reading it, the same warning becomes an average over a quietly smaller denominator that still reports itself as the suite. The score is a mean over the cases that produced a grade, and a case that fails to load contributes nothing to the top of that fraction and is not counted at the bottom either, so the mean carries on over whatever is left. A case that did not run is not a pass and not a failure; its result is unknown, and averaging over the cases that did run puts the average of the others in place of that unknown. The rule that holds up is that a case which does not run is a failure of the job rather than a failure of the case: the process exits non-zero, the score is not published, and the reason is in the log. Since a warning is not a control in an empty room, the choice was never between warning and failing; it was between no control and failing.

cases_run and cases_skipped as separate fields beside case_count are the readable form of that rule, and they turn a silent skip into an arithmetic contradiction that a human notices immediately. What makes them necessary is that a silent skip changes nothing on the page: the score is still a plausible-looking number, the run still finishes, the check still goes green, and no field has a value that looks wrong, since a mean over the cases that ran is a perfectly well-formed mean. Recording how many cases the file holds beside how many were graded creates a pair of fields that must be equal, and an inequality between two numbers that must be equal is a contradiction a reader can see without knowing anything about the suite, what the cases test, or what a good score would be. A control that needs no domain knowledge to exercise still works late on a Thursday afternoon.

Three fields in that manifest are digests, and much of what the manifest buys rests on two properties of them. sha256 is a function that reads every byte of a file and returns a short string of fixed length. Its output depends on the whole of its input, so changing one character anywhere in the file changes the string, and changes it beyond recognition rather than slightly. The function is also fixed and public, so two parties holding the same bytes compute the same string without sending each other the file. A digest is therefore a name for a file's contents that anyone can compute independently and compare in one line.

That turns each file the manifest names from something the run asserts into something a reader can check: the path says which file was meant to be read, the digest says which bytes were read. The dataset digest means a rebase that changes the case file changes the digest, so two scores from different dataset versions cannot be compared by accident — the comparison stops at a mismatch between two strings rather than proceeding into an arithmetic that has no meaning.

The grader prompt digest catches an uncommitted local edit directly, and catches is worth unpacking. The prompt is a file in the tree, so the repository determines its digest at any named SHA, and anyone with repository access can recompute it. The manifest records the digest of the prompt the run actually used. When a developer edits that prompt in their working copy and never commits it, the bytes their run reads exist in exactly one place on earth, so the digest it records matches nothing anybody else can produce — an edit that never reaches the repository produces a digest nobody can reproduce. Without the digest that edit is invisible rather than merely unproven, since the committed prompt looks untouched to everyone else and the only trace is a working copy on one laptop. With it, the mismatch is arithmetic rather than an accusation.

None of this is specific to GitHub Actions. Promptfoo ships a config for wiring the same run into GitHub Actions, GitLab and Jenkins, and DeepEval takes the alternative route of expressing each case as a pytest test so that a failing metric fails the build through the test runner you already have. The choice between those shapes is a tooling question of the sort the previous topic covered. What they share is that the run is executed by the pipeline against a named commit, and that is the property being bought.

The property is worth stating carefully because it is easy to buy the ceremony without it. A CI job that pulls its cases from a mutable S3 prefix has reintroduced the stale-cases problem with more steps. A job that pins the model to an alias like latest or a major-version-only string has reintroduced the model-drift problem and will produce an unexplained score change on a morning when nobody touched the repository. Running in CI is the delivery mechanism; the thing that makes the score mean something is that every input is addressed by content or by version, and the addresses are recorded in the output.

The sentence a passing gate supports#

A passing gate supports one sentence, and it is narrower than the one people hear.

The evaluated commit scored above the threshold on this dataset, with this grader, at this moment.

Each clause is doing work. This dataset means the 120 cases someone wrote, which are a sample of the input distribution chosen by the people who happened to be thinking about it, weighted toward the failures they already knew about. This grader means the score is a measurement of agreement with a judge that has its own biases and its own error rate. At this moment means the model version, the retrieval index, the tools it called and any upstream service were all in the state they were in during those seven minutes, and a rerun tomorrow against a re-embedded index is a different experiment.

The sentence a reader is most likely to write after all of the above is that the eval ran in CI, so the score can be trusted, and it fuses two properties that came from different places. The temptation is honest, because everything the pipeline added is genuine evidence: the declared image, the recorded SHAs, the digests, the job that fails when a case will not load. What that evidence establishes is that the number is the output of a computation whose inputs can all be named, which is attributability and nothing more. Whether the computation measures anything worth measuring is a property of the cases and of the grader, and the pipeline has touched neither. A suite of easy cases scored by a lenient judge, run against a named commit, produces a number attributable to the last byte that informs nobody of anything.

Attributability is a precondition for validity rather than a substitute for it. Until you can say which cases, which model and which grader produced a number, the question of whether the measurement is any good cannot be put at all, because there is no fixed object to put it about. Once you can, the question is open, and none of the work in this chapter answers it.

Reading the per-case deltas rather than the aggregate#

What a green check does not license is the broader reading a reviewer naturally takes from it, which is that the change is safe. It says nothing about inputs outside the dataset, and a 120-case suite against a product taking forty thousand requests a day is a very thin sample. It says nothing about behaviours nobody wrote a case for, which is where new regressions concentrate, because the cases you have are mostly derived from failures you have already seen. It says nothing about latency, cost, or anything else the harness did not measure. And it does not say the eval set still encodes what the product wants, since a case written in March asserts March's intended behaviour and will keep asserting it long after the product has moved.

"Thin" understates the first of those, since thinness is what more cases would fix. The suite was never a random draw from those forty thousand daily requests, so it is a census of the failures somebody wrote down rather than a sample of production, and more cases make it a larger census of the same thing.

The practical version of this is what you do when the check goes green on a change you are uneasy about. The gate has done its whole job and it has not answered your question, so the answer has to come from reading the diff, from the per-case output in the artefact — which cases moved, and in which direction — and from whatever the change does that no case covers. Reviewing the per-case deltas rather than the aggregate is the cheapest defence, because a change that improves the mean by moving twelve cases up and four cases down is a different object from one that moves twelve up and nothing down, and the aggregate cannot tell you which you have. It cannot because a mean is a sum divided by a count, and addition discards where its terms came from: four cases falling and four rising cancel exactly, whatever those eight were about. The per-case output has not discarded it, which is why the upload is worth having on a green run too.

Teams that treat the tick as the end of review converge on a suite that is optimised against, in the sense that changes are shaped until the number clears rather than until the behaviour is right, and the suite slowly stops correlating with anything a user would notice. That is a slower failure than #412 and a harder one to reverse, because by the time it is visible the suite has been green for months.

Binding the run to the commit is therefore a narrow win and a load-bearing one. It does not improve the cases, the grader or the coverage; those stay exactly as good or as bad as they were. What it changes is that the score becomes attributable to a named commit with its inputs addressed by content or version, and that it is produced on every change rather than on the changes somebody remembered to check. Everything later in this topic — the threshold, the re-runs, the cost, the extra axes — assumes that property and is meaningless without it.

There is a further limit that this chapter deliberately leaves standing. Everything above treats the score as a stable property of the commit, as though running the same suite twice on the same SHA would produce the same number. It generally will not. The same 120 cases against the same pinned model with the same grader, run three times an hour apart, can easily return 0.87, 0.83 and 0.86, because judge models are not deterministic, because retries change which responses get scored, and because 120 cases is a small sample of a noisy process. That spread interacts with the threshold in a way that decides whether the gate is protecting anything, and putting a defensible number on the threshold is the subject of the next chapter.