An optimiser is a proposer, not a stage in your pipeline

With prompt versioning, review and CI evaluation already in place, an optimiser needs no path of its own. Its winner enters the path that exists, as a candidate carrying the numbers that justify it. That sounds like a small architectural point and it is the difference between a programme that produces reviewable improvements and one that quietly destroys the team's ability to measure anything.

The design that invites the mistake is clean by analogy: the team already runs tests automatically, already runs evaluation automatically, and already deploys automatically, so a manual step in the middle of prompt improvement looks like the odd one out. Wire the optimiser in as a CI stage on every merge to main, have it commit and deploy the winner if it beats the current prompt, and the loop closes. The analogy fails because CI is a deterministic function of its input and a search is not, and because the resource a search consumes is not compute.

Three things a search-in-CI design breaks#

CI produces the same artefact from the same input, which is what makes a green build mean anything and a bisect — walking backwards through the commit history, rebuilding at each step, to find the change that broke something — possible. A search is stochastic in three independent places, so a search stage makes every build emit a different prompt, and a build re-run after an infrastructure blip deploys different behaviour than the one it replaced. Nothing downstream can reason about what is running.

That chain is worth walking, because "the build is not reproducible" sounds like a purity complaint until you follow what rests on it. A bisect rebuilds the tree at each commit and asks whether the symptom is present, and it assumes that rebuilding a commit produces the artefact that commit produced the first time. Put a search in the build and the artefact becomes a draw, so rebuilding an old commit yields a prompt nobody has ever run, and the answer at each step is about that new prompt rather than about the commit. A rollback rests on the same assumption from the other end: reverting a merge restores the input to the build, not the prompt the build emitted. The two operations a team reaches for when production is misbehaving are therefore precisely the two this design disables, and it disables them without any error being raised.

Nobody reads the prompt. Under that design the text reaching production has been read by no human at any point, and a candidate can contain instructions that move the grader without helping anyone. Constraint filters catch the categories you thought of; the value of a reject log is that it reveals categories nobody anticipated. Removing the reader removes the mechanism by which the filter list grows.

The holdout is consumed, and this is the one that cannot be undone. A holdout scored 38 times a week, with the highest-scoring candidate kept each time, has been used for selection 152 times in a month, and by then its reported margins are inflated in exactly the way a tuning score is. Rebuilding it means commissioning several hundred fresh expert-labelled cases, on the order of 40 person-hours for 300 cases, and until that lands the team has no unbiased estimate of anything. A design that spends the holdout at CI frequency destroys the team's ability to measure prompts at all, which is a strictly larger loss than any prompt improvement it could produce.

Both of those figures are one multiplication away from the merge volume, and doing them is what moves the design from plausible to obviously wrong. A search scores the holdout once, so 38 searches a week is 152 scorings in a month, which is the count in the table below; the provider cost scales off the same number, as does the machine time. The replacement cost is the only one that does not scale with anything the team controls. Three hundred cases at something like eight minutes of a domain expert's attention each is roughly two and a half thousand minutes, which is where the 40 person-hours comes from, and no amount of money shortens it, because the input being spent is a specialist's attention rather than a budget line.

The third column is the shape that works, and its distinguishing property is a line drawn between generating a candidate and promoting one. Generation is stochastic, expensive, unattended and frequently wrong. Promotion is deterministic, cheap, reviewed and gated. Those are different activities with different failure modes.

The winner enters the path that already exists#

Topics 4.10 and 3.16 supply the whole promotion path: prompts are files under review with pinned revisions, and CI scores a merged prompt against a fixed eval suite. A search adds nothing to it. The optimiser runs on a schedule outside CI, and its output is a pull request.

CI never runs the search. It evaluates the merged prompt exactly as it would evaluate a prompt someone typed by hand, which keeps the build deterministic and keeps one holdout scoring per merged candidate rather than one per attempt.

What makes the pull request reviewable is the manifest that travels with it. A reviewer cannot verify a quality claim by reading English, so the manifest has to carry everything the claim depends on in a form that can be checked.

{
  "run_id": "opt-2024-11-18-a",
  "seeds": [7, 13, 29],
  "proposer": {"model": "gpt-4.1-2025-04-14", "temperature": 1.0},
  "judge": {"model": "gpt-4.1-mini-2025-04-14", "temperature": 0.0, "n": 1},
  "tuning_cases": "evalset@a41f2c9#tuning",
  "holdout_cases": "evalset@a41f2c9#holdout",
  "holdout_scorings_used": 4,
  "scores": {
    "baseline": 0.840,
    "winners": {"seed7": 0.858, "seed13": 0.861, "seed29": 0.856},
    "proposed": "seed13",
    "paired_ci_vs_baseline": [0.004, 0.038]
  },
  "constraints": {"passed": true, "rejected_candidates": 137,
                  "reject_reasons": {"length": 96, "policy": 11, "latency": 30}},
  "convergent_instructions": [
    "quote clause numbers exactly as they appear in the source",
    "answer in the language of the question"
  ],
  "budget": {"tokens": 588, "added_p95_ms": 42, "cost_per_1k_usd": 1.47},
  "pipeline_sha": "9c14b3f",
  "selection_axis": "shortest candidate within noise of the best"
}

Opening the pull request is the last thing the job does, and it is worth writing out because the shape of the call is the shape of the boundary. The job has write access to a branch and to the GitHub API, and no access at all to the deploy pointer.

git switch -c "opt/2024-11-18-a"
cp out/winner.txt      prompts/drafting/v14-optimised.txt
cp out/manifest.json   prompts/drafting/v14-manifest.json
git commit -am "prompt(drafting): candidate from opt-2024-11-18-a"
git push -u origin HEAD

gh pr create \
  --base main \
  --title "prompt(drafting): candidate opt-2024-11-18-a (seed 13, +2.1 pts)" \
  --body-file out/pr_body.md \
  --label prompt-candidate \
  --reviewer priya

The body is generated from the manifest rather than written by the search, so it says the same things the numbers say and cannot drift from them.

## Proposed: seed 13 winner, 588 tokens

| | baseline v11 | proposed | delta |
| --- | --- | --- | --- |
| Holdout (86 cases) | 0.840 | 0.861 | +2.1 pts |
| Paired CI vs baseline | — | [+0.004, +0.038] | excludes zero |
| Prompt tokens | 480 | 588 | +108 |
| Added p95 | — | +42 ms | within the +60 ms budget |

Selection axis: shortest candidate within noise of the best.
Seeds scored 0.858 / 0.861 / 0.856 — all three inside the holdout's resolution.
Constraints: passed. 137 candidates rejected (96 length, 30 latency, 11 policy).
Pipeline SHA at search time: 9c14b3f.

pipeline_sha is there because a manifest whose pipeline SHA is behind main describes a search run against a system that no longer exists, and the correct response to that pull request is to close it and re-run rather than to review it. holdout_scorings_used is there because the holdout is a budgeted resource and the number needs to be visible to the person spending it.

# .github/workflows/prompt-search.yml
on:
  schedule: [{cron: "0 2 1 * *"}]      # 02:00 on the 1st
  workflow_dispatch:                   # or on demand after a pipeline fix
jobs:
  search:
    runs-on: ubuntu-latest
    timeout-minutes: 480
    steps:
      - uses: actions/checkout@v4
      - run: python -m promptsearch.run --config search.yaml --seeds 7,13,29
      - run: python -m promptsearch.open_pr --manifest out/manifest.json

Which of the manifest's claims CI is entitled to believe#

Everything in that manifest was written by the job that produced the candidate, which is the same as saying the candidate is grading its own homework. A bug in the search harness that shuffles the holdout in with the tuning cases, or an off-by-one in the token counter, will be reported confidently and will look exactly like a good run. Some of the claims are cheap to recompute at merge time and should be, and a few are expensive enough that the answer is to verify them against a source outside the job rather than to redo them.

The scoring log is the piece that makes the last three rows possible, and it is a small thing to build: an append-only file, written by the only code path allowed to load the holdout, recording a timestamp, the prompt revision, the eval set digest, the judge config and the resulting score. CI checks that the manifest's holdout number appears in that log under the run's own identifier and that the log's total for the quarter matches what the manifest claims to have spent. A candidate whose holdout score does not appear in the log was scored by something other than the sanctioned path, and the correct response is to close the pull request rather than to investigate the discrepancy under time pressure.

The inference in that last sentence deserves to be made explicit, because it is the only place in this design where a check on a self-reported number has any force at all. The log is written by the one code path permitted to load the holdout. A score that appears in the log therefore came through that path. A score that does not appear means either the holdout was loaded by code that should not have been able to load it, or the number was never measured — and those two possibilities are indistinguishable from the outside, because both produce exactly the same artefact: a manifest asserting a figure with nothing behind it. That is why the response is to close rather than to investigate. The pull request has already established that its numbers cannot be traced, and nothing else in it is evidence of anything.

The reviewer is not checking whether the prompt is good, because reading English cannot establish that and the holdout number already speaks to it. Their job is a different list, and stating it explicitly is what stops prompt-candidate pull requests from being rubber-stamped. They check that the manifest is complete and internally consistent, and that pipeline_sha matches the current pipeline. They read the prompt for the banned content categories, since the filters catch known categories and a human catches the next one. They check that the convergent instructions actually appear in the proposed candidate, because a candidate that shares nothing with the other seeds' winners is a draw from the plateau with no corroboration. They confirm the selection axis was declared and is defensible — "shortest within noise of the best" is a reason, "highest score" is not. And they write the block-by-block annotation the rollback rule requires; being unable to write it is itself the finding, and the correct outcome is to close the pull request and hand-write the convergent instructions instead.

Stale candidates, closed candidates, and the neighbouring design#

Review latency is the failure mode this design owns, and it is worth naming because the table above lists it as the price of the whole approach. A candidate pull request that waits nine days for attention is frequently invalid by the time it gets it, since main has moved and the manifest's pipeline_sha no longer matches. On the team this chapter draws from, median time to first review was four days and two of the seven runs were closed for exactly this reason. The fix is not to review faster, which nobody can commit to, but to make staleness cheap and automatic: a scheduled job closes any prompt-candidate pull request older than seven days with a comment naming the SHA that moved, and the next monthly run proposes again from the current pipeline. Closing costs nothing because the candidate was never scarce, and a stale pull request left open costs a reviewer the twenty minutes it takes to discover it is worthless.

Closing needs to be a normal, frequent outcome for any of this to be real. Four of the first seven monthly runs were closed without merging: two because the pipeline had moved underneath them, one because no candidate cleared the 1.0-point margin, and one because nobody could write the annotation. A programme where every run merges is not a programme with a review step.

None of this says the running cannot be automated. The search runs on a cron, opens its own pull request and needs no human until the pull request exists — the automation is complete right up to the point where an unreviewed artefact would reach users, and that is the only place the line goes.

It also is not online optimisation. There is a family of techniques that skips offline holdouts and searches against live traffic directly, allocating requests between prompt variants with a bandit — an allocator that keeps shifting live traffic towards whichever variant is winning as the outcomes accumulate — and letting production outcomes supply the objective. That trades the holdout problem for a much sharper one, since every candidate is experienced by real users before it is known to be safe, and it needs traffic volumes and outcome signals most internal systems do not have. It is where a good deal of the current research energy sits and it is a different risk posture, worth knowing exists and worth keeping out of the same conversation as a monthly pull request.

The boundary that all of this draws is between a machine that may propose anything and a path that promotes only what a person has read. Everything on the proposing side can be automated to whatever degree the budget allows, and nothing on the promoting side changes because a machine wrote the candidate rather than an engineer — same review, same CI evaluation, same pinned revision, same rollback rule.

Once the shape is settled the programme becomes routine, which is when the harder question arrives. After six months of monthly runs the team has about 3 cumulative points on the holdout, and it also has two reverts and an incident that ran a day long. Only the first of those three numbers has ever appeared in a report.