Automated prompt optimization

Read these first

• Prompt versioning, storage and review
• Evaluation in CI

What you will know

• Run an automated prompt optimiser against a real eval set

  Use separate tuning and holdout cases, because search can overfit the visible examples while appearing to improve quality.

• Compare its gain against the gain from an hour of error analysis

  Measure both changes on the same held-out slice and include engineer time, since a small metric gain may not repay the investigation cost.

• Re-run the search with fixed seeds and a held-out set, then report whether the winner survives

  Search methods are stochastic and can select brittle wording, so reproducibility and out-of-sample performance matter more than one winning run.

• Add constraints for prompt length, latency, and forbidden content, and reject candidates that violate them

  An apparent quality improvement can consume context budget, increase serving cost, or introduce unsafe instructions.

• Write a rollback rule that selects between the baseline and optimized prompt using quality, cost, and maintainability thresholds

  A prompt that wins offline may be unacceptable after model changes, incident review, or a small regression in operational limits.

Chapters

1. 1

   An optimiser proposes candidates, scores them, and keeps the winners

   Every other chapter here reads the output of a machine, so start with the machine. A prompt search loop proposes candidates — an LLM rewriting the instruction, mutation and crossover over a population, bootstrapped selection of few-shot examples — scores each one against an objective your grader computes over a set of cases, and keeps the best to seed the next round. Knowing where the randomness lives in that loop is what makes a fixed seed meaningful: it governs candidate sampling, example ordering and the grader's own sampling, and pinning one of the three is not pinning the run.

   Read chapter
2. 2

   0.91 on the searched set is a fitted score, not an estimate

   An optimiser that scores 400 candidates against your 200-case set and reports 0.79 rising to 0.91 has produced a training number: the winner was chosen against those exact cases and that exact grader, so the figure is guaranteed optimistic and grows more so with every candidate tried. Here you split tuning cases from a holdout the search never sees, and work out what a holdout equal to baseline would have meant.

   Read chapter
3. 3

   Three seeds, three different winners, one plateau

   Re-running the same search under different seeds yields prompts that read nothing alike and score 0.831, 0.828 and 0.835 against a baseline of 0.800. That spread sits inside the holdout's own noise, so shipping the highest is selecting noise while believing you are selecting quality, and each such choice erodes the holdout into a tuning set. Decide on length, latency and editability instead, and report whether the winner reproduces.

   Read chapter
4. 4

   Constraints are filters in the search, not a review afterwards

   A candidate worth 2.1 points that runs 2.4 times longer, adds 180ms to p95 and tells the model never to say it is unsure was selected partly because suppressing hedging moves your grader. Deleting that sentence before deploying leaves a prompt whose score you no longer have, since any edit invalidates the measurement. Learn to encode length, latency and forbidden content as rejection rules inside the loop.

   Read chapter
5. 5

   The optimiser searches over your bugs as well as your prompt

   Six hours of search bought 1.2 points on the holdout; an hour spent reading thirty failures found a retrieval bug worth 4. The consequence runs deeper than the comparison, because the winner's wording partly compensates for the broken retrieval and the run has to be repeated once the fix lands. This chapter puts error analysis and search on the same held-out slice with engineer time priced in, and orders them.

   Read chapter
6. 6

   A rollback rule names a threshold on every axis, not just quality

   An optimised prompt worth 2.1 points that nobody on the team could confidently edit, with a new model version due in three weeks, needs a written rule rather than a judgement call at the time. Keep it only while it holds its margin inside agreed token and latency limits, re-evaluate both prompts on the same holdout whenever the model or pipeline moves, and keep the baseline beside it so reverting is a switch.

   Read chapter
7. 7

   An optimiser is a proposer, not a stage in your pipeline

   With prompt versioning, review and CI evaluation already in place, a search needs no new path of its own: its winner enters the existing one as a candidate carrying its holdout numbers and constraint checks. A human reviews and merges it, and CI evaluates the merged prompt rather than running the search. This chapter draws the line between generating candidates and promoting them.

   Read chapter
8. 8

   Counting reverts and lost incident hours on the same ledger

   Six months of monthly runs, roughly 3 cumulative points on the holdout, two prompts reverted after model upgrades, and one incident that ran a day longer because the on-call engineer could not safely edit generated text. Reporting benefit in points and cost in nothing makes the programme unfalsifiable. Here you build the comparison that could actually conclude it is not worth keeping.

   Read chapter