A suite grown around one agent scores fit, not capability

The vendor's deck opens with their public score, offers a two-week trial with a support engineer attached, and quotes a seat price that would cost about $140,000 a year across the platform group. The platform lead's proposal is to run both agents through the 187-task suite from the last chapter and compare. The VP wants a decision in three weeks and considers the benchmark plus a trial to be enough evidence already. Three weeks later, the head-to-head comes back at 71% for the incumbent and 54% for the challenger, and somebody writes in the doc that the vendor's agent is seventeen points worse.

That sentence is not supported by the experiment that was run. The suite is a shared component, and almost nothing else in the pipeline is. What the two agents were actually run through is the incumbent's harness with the challenger's model dropped into it, and the seventeen points contain an unknown mixture of capability difference and fit difference, with no way to separate them after the fact.

Everything the 71% has been trained against#

The incumbent has been in production for fourteen months. In that time the harness accumulated 63 commits, and the interesting thing about them is that almost none were model changes.

Drop another agent into that and it inherits a retrieval ranker tuned to compensate for a different model's weaknesses, a summariser trained on a different model's traces, and a diff repair pass for malformed hunks it does not produce. The challenger may also arrive with its own versions of half those components, which the harness will simply prevent it from using. A score gap measured this way is real in the sense that it predicts what would happen if you deployed the challenger tomorrow without changing anything — and that is a fit measurement, which is a legitimate thing to want and a different question from "which agent is better".

Separating the two requires holding constant only the parts that define the task: the statements, the oracles, the protected paths, and the grading rule. Everything downstream of "here is the problem, here is the repository" belongs to the agent under test, including how it retrieves, how it edits, how many attempts it takes, and how it decides it is done. The comparison then has one more term that is easy to forget, which is the tuning budget. Fourteen months of harness work went into the incumbent's number, so a challenger given four days of integration and no iteration is not being compared on equal footing even if it brings its own scaffold. The defensible version allocates the vendor a fixed budget — say eight engineer-days across their support engineer and one of yours — and gives them a development split of tasks to iterate against, with the rest never shown.

Handing a vendor a development split makes people uncomfortable, and the discomfort is worth examining because it points at what the suite is for. The objection is that they will tune to the tasks, which is true and is the entire point: the incumbent's harness was tuned to your repository over fourteen months, and the question on the table is what the challenger looks like after comparable investment, not what it looks like cold. The protection is structural rather than moral — the development split is a disjoint set of tasks, the evaluation split is never shown, and the vendor never sees an oracle from it. Sixty tasks for iteration and 127 held back is a reasonable division on a suite of this size, and it converts "did they cheat" into a question you can answer by checking which task ids left your network.

Fit is also worth measuring on its own terms, as long as it is labelled. If the realistic deployment is "swap the model, keep our harness", then running the challenger through your pipeline unchanged answers exactly the deployment question, and the seventeen-point gap is the right number for that decision. What makes the original doc wrong is the claim it attaches to the number. "The challenger is worse" is a capability claim; "the challenger is worse in our current harness with four days of integration" is the measurement, and the second one invites the obvious follow-up about what a month of integration would do.

Three weeks buys three separable answers#

Three weeks does not buy that experiment. It does buy several smaller ones whose results do not depend on the harness being fair, and the discipline that makes them worth running is deciding the decision rule before the data arrives.

The first gate is cheap and often decisive. Take five tasks, hand the vendor the sandbox image, and see whether their agent runs inside it with no network, whether it can be pinned to a specific model version so that a rerun next month is comparable, and whether it will accept the repository as a local checkout rather than requiring an upload. This is not a capability test and it does not need statistics; a binary answer within a few days is what makes the rest of the schedule possible.

The second gate is about the cost of the output rather than its correctness. Pick twenty tasks, run both agents, strip the patches of any identifying formatting, and have three engineers review the forty diffs blind, recording time spent and a verdict. The number this produces is noisy and the sample is small, and that is fine, because the question it answers is coarse. If reviewers are consistently spending 35 minutes on the challenger's patches against 18 on the incumbent's, the resolution rate is close to irrelevant, and if reviewers cannot tell the two apart, that is a genuine finding worth writing down.

The third gate is the only one that touches capability, and it is deliberately narrow. Mine 40 fresh tasks from the last two months of history — recent enough that neither harness has ever been tuned against them, and, if your repository is public, recent enough to sit past both models' training cutoffs. Run each agent under its own harness, with the vendor's scaffold doing whatever it does. Report resolution with a confidence interval — the band of underlying rates that the observed result is consistent with — and treat anything inside it as a null result, which means the experiment did not detect a difference rather than that there is none to find. Forty tasks gives roughly a ±13-point interval on a rate near 60%, so this gate can only detect a large difference, and knowing that in advance is what stops the outcome being over-read.

Deciding in advance what nothing means#

The failure mode of a three-week trial is that it produces an ambiguous number and the ambiguity gets resolved by whoever is most senior in the room. Writing the decision rule down first is a cheap defence, and it is short:

gate_3:
  tasks: 40            # mined from the last 8 weeks, never used for tuning
  runs_per_task: 3     # per chapter 5, single runs cannot support this
  decision:
    challenger_wins: "resolution advantage > 13 points AND review time not worse"
    incumbent_holds: "incumbent advantage > 13 points"
    null_result: "otherwise"
  null_result_means: >
    We did not measure a capability difference. This does NOT mean the agents
    are equivalent; it means 40 tasks at 3 runs cannot see a difference this
    small. Renewing the incumbent is the default because switching costs
    ~8 engineer-weeks of harness work, not because the challenger lost.

That last field is what a null result settles, and it is the field people skip. A trial that finds nothing has still eliminated the possibility of a large difference, which is worth knowing when the pitch implied one, and it has left the decision to be made on the grounds it was always going to be made on: switching cost, support, data policy, and price.

When there is no experiment to run#

Some conditions end the evaluation before any of this, because they make a capability number unobtainable rather than merely expensive.

An agent that cannot execute inside your sandbox cannot be measured against your suite at all. The oracle is a test run against a specific tree in a specific image, and if the vendor's product only runs in their cloud against a repository you upload, then whatever it produces is being graded in an environment you cannot pin, with a toolchain you cannot see, at a model version they can change mid-trial. You can still form an impression of the patches; you cannot produce a number that means anything next quarter.

Source egress — your code leaving your own network, to a vendor's servers or to whichever model provider sits behind them — is the second, and it is a policy question rather than an engineering one. If the agent requires uploading the repository, and your customer contracts, regulatory posture or licensing terms forbid it, the correct response is to decline the evaluation rather than to run it and treat the result as information. Running it establishes a fact about a system you cannot deploy, and it puts the code somewhere it should not be in order to establish that fact. The same applies when the vendor's terms reserve the right to train on submitted code, or when they will not commit to a model version for the length of a contract, which makes every future rerun of your suite incomparable to the last.

The third is subtler and worth naming because it looks like a technicality. If the vendor's agent cannot be run non-interactively — no batch API, no way to submit 187 tasks and collect 187 patches — then you cannot run a suite against it, only a demo. Demos measure the tasks somebody chose to demonstrate.

Every gate above eventually produces a resolution rate, and there is a decision to make about how that rate is computed before it goes in a deck, because an agent that samples is not a function and a single attempt is not the whole story.