Running an eval tool bake-off

A bake-off is a fixed-length trial in which every candidate tool is put through the same work on the same data, so that the comparison is between what they do rather than between what they say. It is the standard answer to a shortlist, and it is easy to run badly, because the artefact it produces — a grid of options against criteria — looks the same whether the cells were measured or copied out of a pricing page.

The five criteria that matter for an eval tool are integration effort, reproducibility, latency, cost, and failure visibility. Every vendor claims all five, in language chosen so that the claim is technically true under some configuration. Reproducibility means seeded runs at one vendor, immutable experiment records at another, and pinned case-set versions at a third, and only one of those has anything to do with whether the same commit scores the same twice. A grid assembled by reading looks like this, and it contains no information:

The reason that grid contains no information is worth stating as a mechanism rather than as a complaint about marketing. Each cell in it is the output of a process: somebody read a page and copied a phrase. That process is a measurement of what the vendor chose to write, and what the vendor chose to write is a decision made by their marketing team about how to describe a product they control the description of. Nothing in the chain from their product's behaviour to that cell is constrained by their product's behaviour, which is why a tool that flips eleven cases across identical runs and a tool that flips none can both truthfully print "reproducible by design". Filling the same grid by running the tools replaces that chain with one that starts at the code.

The word "reproducibility" is where this bites hardest, and the fix generalises. A criterion stated as a word is filled in by each party with whichever meaning flatters them, because the word does not say what would count as satisfying it. Stated as a procedure — run the identical suite on the identical commit five times and count the cases that did not return the same verdict every time — it has one outcome per tool, and no vendor gets to supply a different definition afterwards. Every criterion in this chapter that carries a number was converted that way, and the one that resisted conversion, failure visibility, got a written scale instead.

Replacing the grid takes a fixed case set, a deliberately hostile addition to that case set, and a rule about which cells are allowed to be filled in from reading.

One frozen case set, plus six cases that go wrong#

All candidates run the identical suite: 200 real cases pinned at a case-set SHA — the content hash of the case file, which names those exact 200 cases and changes the moment one of them is edited — so that nobody can quietly improve their numbers by curating. On top of those go six injected cases, each engineered to break in a way that is confusing rather than obviously invalid, because the confusing failure is what separates the options.

# evals/cases/bakeoff-injected.v1.yaml
- case_id: brk-01
  intent: prompt renders to ~240k tokens (a 90-page policy PDF pasted into context)
  reveals: does the run report context_length_exceeded, or silently score 0?
- case_id: brk-02
  intent: judge is asked a question its rubric does not cover, so it answers in prose
  reveals: unparseable judge output — retried, errored, or coerced to fail?
- case_id: brk-03
  intent: routed through a proxy that returns one 500 on the first attempt only
  reveals: is a transient error retried, and is it distinguishable from a failure?
- case_id: brk-04
  intent: reference answer deliberately contradicts current policy (bad gold label)
  reveals: can a human see the grader's reasoning and blame the label, not the model?
- case_id: brk-05
  intent: right-to-left Arabic with emoji and a zero-width joiner in the output
  reveals: does display, diffing and CSV export survive it intact?
- case_id: brk-06
  intent: a second case submitted with case_id brk-01
  reveals: rejected, silently deduped, or does it overwrite the first?

Then a scoring rubric for the one criterion that everybody claims and nobody defines. Failure visibility is scored 0 to 3 on each injected case, and the definition is stated before the trial starts so that nobody negotiates it afterwards:

• 0 — the case shows as failed with no indication that anything unusual happened.
• 1 — an error is distinguished from a failure, with no detail about which error.
• 2 — the error class and the raw provider response are visible in the UI or the run file.
• 3 — the raw model output, the rendered prompt, the grader's own input and output, and the error class are all reachable within two clicks or one jq command.

brk-04 is the case worth watching most closely, because a wrong gold label — the reference answer a case is graded against, which the grader treats as the truth and never questions — is the most common real defect in an eval suite and the one that wastes the most engineer time. A tool scoring 0 on it shows a red case that stays red, an engineer spends half a day trying to make the assistant produce an answer that is in fact wrong, and the defect survives. A tool scoring 3 puts the grader's rationale — the answer redirects the customer to the marketplace, which the reference answer requires — right next to the reference answer, and the defect is obvious in ten seconds.

brk-03 and brk-02 split the field almost as sharply, for the reason the previous chapter gave in pricing a hand-built harness: a runner that cannot distinguish an error from a failure will report both as red, and a tool that scores the single injected 500 as a failed case is telling you that its nightly numbers include however many provider hiccups happened that night. brk-02 catches the same defect on the grader side, where it is more common and less noticed, because a judge that answers in prose instead of the expected verdict token is not a rare event on a rubric that has any ambiguity in it.

Which cells may be filled in from reading#

List prices can be read. Everything else has to be observed, because every one of the remaining criteria is defined by behaviour that a quickstart does not exercise.

Reproducibility is measured by running the identical suite on the identical commit five times per tool and counting per-case flips, exactly as in the first chapter of this topic, on repeatability in a tool trial. The number that goes in the cell is the count of cases that did not return the same verdict all five times, and the spread of the aggregate.

Two numbers rather than one, because they answer different questions. The flip count says how much of the suite is unreliable and therefore how much rubric work is waiting. The aggregate spread says where a gate threshold would have to sit, which is the question that decides whether the tool can block a merge at all: a threshold has to sit below everything an unchanged commit can produce, or it fires on nothing that happened. A tool whose aggregate wanders over six points therefore needs its threshold at least six points under the current score, and a regression smaller than six points then passes the gate unnoticed. That is the trade the spread column is pricing, and it is why a fast, cheap tool with a wide spread is ruled out by one row regardless of the rest of its column.

Integration effort is measured by actually merging a CI job for each option and logging the hours, including the hours spent on things the quickstart skips: getting credentials into the runner, making the process exit non-zero on failure, publishing the run artifact where a reviewer can reach it, and making the job's output legible in a pull request. The quickstart number and the merged-job number differ by roughly an order of magnitude, consistently.

Latency is measured on your quota with your concurrency, on the full 206 cases, three times, reporting the median wall-clock. A vendor's parallelism claim is about their side; your rate limit is on your side, and it is usually the binding constraint.

Cost has a readable component and an unreadable one. The unreadable part is the unit — a platform priced per trace, where a trace is one recorded model call with its prompt, response and timings attached, bills each grader call as a separate trace on some plans, so a 200-case suite with three graders and five repeats is 3,000 billable traces per run rather than 200.

That multiplier is worth building up in the order the calls actually happen, because it is the only part of a price sheet that can be off by an order of magnitude without anybody misquoting anything. Start with 200 cases. Each case is graded by three graders, and if every grader call is its own trace, that is six hundred traces before repeats. Run each case five times and the whole structure repeats five times over, giving 3,000. The list price per trace did not change and the invoice is fifteen times the one anybody estimated from the case count. The lesson is not that per-trace pricing is a trick; it is that a unit price is meaningless until you know what the vendor counts as a unit, and the count is a property of their implementation rather than of your suite.

That multiplier only shows up on an invoice, which is why the trial has to run long enough to produce one, or at least long enough for the usage dashboard to show the count.

Failure visibility is the injected cases, scored against the rubric above by two people independently.

Two weeks is four engineer-weeks if two people run it, and that budget only stretches across four options with a hard timebox: a day and a half per tool to reach a merged CI job, and whatever is unfinished at that point becomes the integration-effort score rather than a reason to extend. Overrunning on one candidate is the most common way a bake-off degenerates, because the sunk effort makes that candidate feel closer to done than the ones nobody has fought with yet. The same discipline applies to who does the driving: one engineer taking all four is faster and produces a comparison contaminated by the order they were attempted in, so it is worth having each person do two, swapping which two, and comparing notes on the injected cases separately before any numbers are shared.

Three results that pick the hand-built option#

Three outcomes make the harness the answer, and they are worth committing to before the trial rather than after, because a two-week trial generates enough ambiguity to rationalise anything.

The first is no tool clearing the reproducibility bar. If every candidate flips cases across identical runs and none exposes seeds, judge sampling settings or repeats, then no candidate can gate a merge, and the harness at least can.

The second is dataset-shape rejection. If getting your cases into a tool's schema costs the fidelity that made them useful — the multi-turn history collapsed into one string, the retrieved chunks no longer addressable by the grader — then the tool is measuring something adjacent to what you meant to measure.

The third is a failure-visibility score below the harness. A tool whose interface tells you less about a confusing failure than jq over a directory of JSON is charging for a downgrade, and no amount of dashboard compensates.

The axes a two-week trial never touches#

A bake-off is a measurement of four options at your current size, on your current suite, with two engineers driving it. It does not exercise the review queue with eight annotators and 4,000 cases, it does not exercise a provider outage in the middle of a nightly run, and it says nothing at all about the eighteen-month question from the chapter on artifact ownership, because every candidate looks fine on that during a trial where you have not accumulated anything yet. The way to be honest about it is to write down, in the decision record, the conditions under which the decision is revisited: a suite above roughly 1,500 cases, more than three people labelling, or the first time a grader change needs to be applied retroactively to six months of stored outputs.

It is also worth spanning shapes rather than sampling one. A CLI-and-YAML tool that lives in your repository and a hosted platform with a database behind it are different bets about where the artifacts sit and who does the labelling — Braintrust vs. Promptfoo: LLM evaluation comparison draws that contrast directly, and Langfuse vs Arize Phoenix vs LangSmith is a hands-on account of the hosted end. A shortlist of three tools that are all the same shape has already made the decision that matters and skipped the argument.

The reason shape matters more than features is that the two shapes fail in different places, and a trial only exercises the failures its shape has. Hosted platforms concentrate the risk in artifact ownership, which a two-week trial cannot reach because nothing has accumulated yet. Repository-resident tools concentrate it in the human surface, which a trial run by two engineers cannot reach either, because neither of them is the support colleague who will be labelling next year. Putting one of each on the shortlist does not remove those blind spots. It puts them in different places, so that at least one candidate in the comparison is strong exactly where the other is weakest, and the argument gets had rather than assumed.

What makes the exercise work is that its evidence is unfakeable. A vendor cannot write documentation that changes what their product does with a duplicated case ID, an unparseable judge response, or a reference answer that is simply wrong, and those three events happen weekly in a real suite. Pin the cases, inject the six, state the rubric in advance, read the price and observe the rest, and two weeks produces a grid whose cells disagree with each other — which is the sign that it was measured.

The trial ends with a choice, and choosing means the 2,400 cases and labels that accumulated during four months of using the old setup have to move. That move is the part of this whole exercise that is genuinely irreversible.