The same agent, both ways, under one stated workload

Two builds of one agent can be compared, and almost every attempt to do it produces a number nobody can defend. The reasons are ordinary and they are worth naming, because avoiding them is most of the work: the two arms run at different times of day and measure the provider rather than themselves; the failure modes everybody cares about occur too rarely in a short window to be counted; and the two builds turn out not to be the same agent, because one of them quietly lacks argument validation or carries a different retry policy.

Fixing all three requires deciding what the agent is supposed to survive before any measurement starts, and writing that down converts the comparison from an argument into a procedure. What follows is one such comparison — a billing agent built on a graph runtime and built again as a small loop — with the specification first, the results second, and the reasoning that connects them in between.

Writing the workload down first#

Both builds of the agent below were finished on a Wednesday, and on Thursday morning the team spent an hour arguing about which was faster before anyone had measured either. The framework build had been running in staging for three weeks; the hand-written loop had been assembled in two days from the existing tool functions and a version of the loop that already knew about derived idempotency keys and shielded cancellation. The arguments were the ordinary ones — the framework must have overhead, the loop must be missing something — and they were unresolvable because nobody had written down what the agent was supposed to survive. The first genuinely useful thing that happened that week was somebody opening a file and describing the workload.

name: billing-agent-bakeoff
traffic:
  source: replayed production transcripts, 2025-05-01..2025-05-07
  runs_per_hour: 1700
  concurrency: 24
  mix: {status_lookup: 0.62, refund: 0.21, plan_change: 0.11, escalate: 0.06}
  steps_per_run: {p50: 4, p95: 11}
targets:
  end_to_end_p95_ms: 9000
  duplicate_effects: 0
  runs_lost_per_day: 0
faults:
  - {tool: search_orders, mode: timeout, after_ms: 30000, rate: 0.05}
  - {tool: charge_subscription, mode: connection_reset, rate: 0.02}
  - {kind: provider_overloaded_529, rate: 0.03}
  - {kind: pod_kill, every_minutes: 15}
duration_hours: 6
schedule: A/B interleaved in 5-minute blocks

Writing that file took forty minutes and made the rest of the comparison mechanical, which is the ratio worth noticing. Every argument that would otherwise have happened during the analysis — whether pod kills count, whether 529s are the framework's problem, what "fast enough" means — is settled in advance by a line in a file, and settled in a form both sides agreed to before either knew which way the numbers would go.

The targets block is doing something specific and it is the part most often omitted. A measurement without a target produces a comparison of two numbers against each other, which is always winnable by whichever is smaller. A measurement against a stated target produces a verdict about whether either build is acceptable, and it makes a difference that falls well inside the target irrelevant by construction rather than by argument afterwards.

Making the comparison fair is most of the work#

Three properties of that specification are doing more than they look like, and skipping any of them produces a number that cannot be defended in a review.

Interleaving the two builds in five-minute blocks controls for the provider, whose latency drifts by a factor of two over the course of a day and occasionally by more during a regional incident. The mechanism is worth stating because it generalises to every measurement made against a shared dependency. Provider latency is a factor that applies to both arms and varies with time, so any schedule that gives the two arms different distributions of time gives them different amounts of that factor. Running the framework build for three hours and then the loop for three hours measures the provider's afternoon as much as it measures either build. Alternating them in blocks short relative to the drift means both sides experience the same weather, the common factor cancels between them, and the difference that remains survives the comparison.

Injecting faults deliberately is what makes failure behaviour observable at all, because at 1,700 runs an hour the natural rate of interesting failures is too low to characterise anything in six hours. The arithmetic is uncomfortable: six hours at that rate is around ten thousand runs, and a failure mode occurring at its natural production rate contributes a handful of events to that, from which no rate can be estimated and no behaviour characterised. The rates in the file are calibrated to the previous week's production numbers, roughly doubled, which keeps the failure modes realistic while producing enough events to count. The pod_kill entry is the important one: a kill every fifteen minutes across a six-hour run is 24 events, which is a normal day of rolling deploys and evictions compressed into an afternoon.

Holding the agent's behaviour identical is the property that is easiest to lose. Both builds used the same model version, the same system prompt byte for byte, the same six tool schemas, the same 30-second tool timeout, and the same derived idempotency keys. A comparison in which the loop quietly lacks argument validation, or the framework build has a different retry policy, is a comparison of two different agents. Verifying it took a test that ran both builds against the same recorded transport and asserted the request bodies matched, which is the same wire-level assertion that catches adapter changes on upgrade.

What the numbers said#

6h, 10,214 runs per build, 24 pod kills per build, interleaved
                                     framework        loop
end-to-end p50                          5.9 s         5.6 s
end-to-end p95                         14.8 s        14.1 s
end-to-end p99                         28.3 s        27.6 s
runtime overhead per step, p50           34 ms         3 ms
runtime overhead per step, p95          210 ms         6 ms
duplicate effects                           0             0
runs interrupted by pod kills             412           409
  of those, resumed and completed         412             0
  of those, lost                            0           409
median time to resume after kill          1.9 s         n/a
529 handling                        absorbed      absorbed

The latency result was the one that surprised the people who had spent an hour arguing about it. The framework's overhead is real and measurable — 34 ms per step at p50, dominated by the checkpoint write, rising to 210 ms at p95 when Postgres is busy — and at four steps a run that is roughly 140 ms of a 5.9-second request.

Work that fraction out and the reason for it becomes portable to other agents. Runtime overhead is charged per step, so a run pays it as many times as it has steps, and the p50 run has four. With model calls averaging 1.4 seconds each, also charged per step, the two quantities scale together and their ratio is a property of the runtime and the model rather than of the run length. That ratio is what makes the overhead small here, and it is what would make it large on an agent whose model calls were fast. Against model calls of that size the runtime is not where the time goes, and end-to-end p95 differed by 0.7 seconds, which is inside the noise the interleaving was designed to expose.

Both builds absorbed provider overload identically, because both used the same client library and the retry that mattered for 529s was the SDK's. Both produced zero duplicate effects, which is only true because the hand-written loop inherited the derived-key scheme; the first version of it, written before that, produced 3 duplicate charges in a two-hour pilot, and that number is worth recording as the cost of building durability yourself rather than as evidence against the loop.

The recovery row is where the two builds are not comparable at all. Interrupting 412 runs and completing 412 of them, against interrupting 409 and completing none, is the difference between a rolling deploy being invisible and a rolling deploy generating support tickets.

How many runs a kill can destroy follows from two numbers in the workload file. At any moment the count of runs in flight is the rate they arrive multiplied by how long each one lasts, so at 1,700 runs an hour with a mean run length of 24 seconds, roughly 11 runs are in flight at any instant. That is the population a single instantaneous kill exposes. A real deploy is not instantaneous, so it also catches runs that start while it is progressing: a deploy that kills all pods over ninety seconds destroys somewhere between 60 and 70 of them. Both figures scale with run length, which is why an agent that gets slower gets more exposed to this without anything else changing.

Building the harness raised one problem worth describing, because everyone who runs this experiment meets it. The two builds cannot be compared step by step, since the model does not produce identical tool calls on identical inputs even at temperature zero, the setting that tells it to take its highest-probability continuation at every step rather than sample among the plausible ones — sampling is only part of it, and provider-side batching and model routing contribute the rest. Observed on this workload, replaying the same 200 transcripts twice through one build produced a different tool-call sequence in 6% of runs.

That rules out assertions of the form "both builds took the same path" and forces the comparison to be statistical: fix the traffic, run enough of it that a 6% behavioural jitter is noise against the effect you are measuring, and report distributions rather than individual runs. The relationship between those two percentages is the whole sizing rule — the run count has to be large enough that the effect you care about is distinguishable from the jitter, and the jitter here is larger than the effect. Ten thousand runs per build was chosen so that a 5% difference in p95 would be distinguishable from that jitter, which is also why six hours was the duration rather than one.

A week of operations, measured#

Latency and failure behaviour are one axis; the work each build demands from the team is the other, and it is the one that gets asserted rather than measured. Logging it for a week is cheap and tends to produce a result nobody predicted.

The totals being within an hour of each other is the useful part, because it contradicts both of the positions people bring to this argument. What that licenses is narrow: over one week, on this agent, neither approach was meaningfully cheaper to operate. What it does not license is any statement about what the work was, because two totals an hour apart are compatible with every possible composition, and these two have no row in common. The framework did not remove the operational work; it changed what the work was about, from building observability and recovering lost runs to absorbing upgrades and operating a state store. Which of those two a team would rather do is a legitimate input to the decision, and it is a different question from which is cheaper.

The number that decided it#

For this agent the decision was made by a single row: 412 recovered against 409 lost. The workload has runs that last long enough to be interrupted, deploys frequent enough to interrupt them, and effects that make an abandoned run a customer-visible event rather than a retry. Nothing in the latency table was close enough to the 9-second target to matter, and the operational totals were a wash. The framework was adopted with the exit section of the decision record written the same week.

The belief that had made the Thursday argument feel important is worth naming now that the table has settled it. Both sides had assumed the comparison was about speed, and the overhead figures are real — a framework step genuinely costs more than a loop step, by an order of magnitude, and that shows up in every measurement of it. What the overhead cannot vary with is whether an interrupted run survives, since it is the same number on a run that completes and on a run that dies. A comparison conducted entirely in those units is constant across the only difference that turned out to matter, which is why an hour of arguing about it produced nothing.

What six hours cannot tell you#

Three things this experiment does not measure are worth naming, because a result presented as complete invites someone to over-trust it.

It says nothing about whether either build answers customers correctly, since both ran the same prompts and tools and the comparison held behaviour constant by design; agent quality is an eval question and lives on a separate axis entirely. Holding behaviour constant is what made the latency comparison meaningful and it is exactly what makes the quality question unanswerable from these runs — the same design decision does both.

It cannot measure upgrade burden, which is the framework's largest recurring cost and only shows up over months. The 3.5 hours in the operations table is one week's sample of a quantity that is lumpy, with most weeks at zero and the occasional week at two days, so a single week's observation is roughly as likely to understate it as to overstate it and carries no information about the mean either way.

And it says little about how quickly a new contributor becomes productive, which matters more than any latency number on a team that is about to double, and which favours the framework in most cases because its concepts are documented and shared while yours are in one engineer's head.

The cost of running it is worth stating plainly, since it is the main objection. Two builds, a replay harness, a fault injector, and a day of analysis came to roughly nine engineer-days here, most of it in the second build and the harness. That is expensive for a decision someone would rather make in a meeting and cheap against the fourteen months the earlier chapters describe, and the harness is reusable: the same replay and fault-injection setup became the load test that runs before each release, which is work the team would have done anyway.

The result that would justify choosing the small loop despite its operational work is exactly the combination the neighbouring team had: framework overhead consuming a material share of a latency budget you cannot renegotiate, plus a recovery column that is worth nothing because interrupted runs are free to replay. Simplicity on its own does not survive contact with the first durability requirement, which is what the 3 duplicate charges in the pilot demonstrated and what the nine days of hand-built checkpointing priced.

Whichever way the comparison goes, the first week afterwards looks the same, and it is the work this topic has been describing. Instrument the thing before it misbehaves, including the error paths, so that a duplicate has a fingerprint rather than a rumour. Derive idempotency keys from the model's tool call identifiers so that every layer below you can retry without consequence. Write the exit section of the decision record while the answers are still cheap, with a measurement under each heading and a CI job that keeps the portability claim honest. Put an eval suite around behaviour, because no runtime has an opinion about whether the agent refunded the right order.

If you have not yet built the loop by hand, the Anthropic cookbook's basic agent workflows implement every core pattern with plain API calls, and Simon Willison's You should write an agent makes the case for doing it once regardless of what you ship. Then run your own version of the bake-off above, with your own workload written down first, and name the number that decided it — because the next person to ask why you chose this will deserve better than a preference.