Lead · 7 chapters
Evaluating an architecture
Running the same task through competing topologies and comparing success, cost and latency rather than assuming
Without this discipline, multi-agent architectures become expensive preferences that survive despite losing to a simpler loop.
Read these first
What you will know
Run one task through two topologies and compare success, cost, and latency
Use matched inputs and repeated trials, then include token or tool spend and a latency statistic such as p95 rather than a single best run.
Write an architecture decision record that selects a topology from measured success, cost, and latency, including thresholds and trade-offs
The record should make the choice falsifiable by naming the constraints, acceptable regressions, and conditions that would trigger a reversal.
Run a controlled case in which a simpler loop beats a multi-agent topology, and identify the measured cause
Extra coordination can add failure surfaces, context overhead, and tail latency even when the design appears more capable on paper.
Build the harness that makes this comparison repeatable
Version prompts, models, tools, fixtures, and scoring code; capture traces and costs so later results are attributable rather than anecdotal.
Patch the production design to use the winning topology and verify the change on a held-out task set
A topology that wins only on the tuning set is overfit, and an unshipped result has no operational value.
Chapters
- 1
One trial per task cannot separate four points from noise
A result arrives: 150 tasks, one run of each, 84% against 88%, plus a note that the pipeline's fastest run was quicker. More tasks do not repair one trial per task against a stochastic system, and a best-case latency is the number least related to what any user experiences. Repeat trials per task, report a tail statistic such as p95, and put token and tool spend beside both arms.
- 2
A rerun that reverses itself measured a different system
Three months after picking topology B on measured results, the same fixtures now favour A, and neither topology's code has changed. Prompts live in the application repo, the model is whatever the provider alias points at, and the rubric was revised in month two, so the two runs measured different systems against different criteria. Pin prompts, model snapshot, tool definitions, fixtures and scoring code, and capture traces and cost per run.
- 3
Six weeks of tuning fits the arm with more places to patch
Both arms were iterated against the same 120 tasks and the multi-agent one now leads by five points. Equal effort is not equal exposure to overfitting: more prompts and more boundaries mean more places to lodge a fix for one failing case, so that arm is fitted to those examples more thoroughly than the loop is. Re-run both on tasks neither has ever been tuned against before the result decides anything.
- 4
A loss concentrated at one handoff is a contract defect
The single loop wins on success rate, and the traces show the multi-agent failures clustering at one boundary where a constraint from the original request goes missing. That is a payload to fix and re-run, not a verdict on topologies. Recording it as it stands turns one lossy handoff into standing evidence against multi-agent designs, which is how a team acquires an architectural belief from a bug.
- 5
Coordination loses through failure surface, duplicated context and hops
A topology can also lose on its merits, and the traces look different: failures spread across every boundary rather than clustering at one, context re-sent at each handoff inflating tokens per task, and each serialized hop adding to the tail. To run a controlled case where the simpler loop wins and name the cause, measure failure rate per boundary, duplicated input tokens, and latency added per hop, then check whether the gap scales with the number of boundaries. Without those numbers every loop win is discountable as somebody's broken payload, which is the mirror of the mistake the previous chapter prevents.
- 6
A decision record names the conditions that would reverse it
Writing down which topology won is only useful if a reader in six months can falsify it rather than re-argue it from scratch. Include the measured success, cost and latency per arm, the thresholds that decided the choice, the regressions accepted deliberately, and the conditions under which you would switch back. That last part is what separates a decision from a preference with numbers attached.
- 7
A study nobody shipped is worth nothing and decays
Four months ago the loop beat the multi-agent design on success, cost and p95; production still runs the multi-agent version and the harness has not been run since. Both measured systems have moved, so migrating on those numbers is as unevidenced as the topology being replaced. Re-run first, then patch production to the winning topology and verify it on a held-out task set before calling anything done.
External resources
- Demystifying evals for AI agentsAnthropic Engineering
Anthropic’s practical guide to building eval harnesses for comparing AI agent architectures.
- Beyond Self-Talk: A Communication-Centric Survey of LLM-Based Multi-Agent Systemsarxiv.org
Catalogues the competing topologies so you know what you're A/B testing.
- Evaluating AI Agents: A Production Blueprint with Strands and AgentCoreAWS ML Blog
End-to-end code for scoring agent runs on success, cost and latency.
- LLM evaluation: a beginner's guideevidentlyai.com
Shows how to compare architectures by measuring task quality, cost, latency, and reliability.
- How to evaluate agents in practiceGoogle Cloud Tech
Practical trajectory and outcome metrics for comparing agent designs.
- Agentic Evaluations Workshop - Deep Dive on the Future on Evals for Agents.Hugging Face
Long-form workshop on building the benchmark you'll run designs through.
- Welcome to the LLM evaluation courseEvidently AI
Walks through evaluating LLM architectures by comparing task success, cost, and latency.