The simplest design that passes evaluation is the one to build first

It is Wednesday of the first week. The project is an agent that answers questions about a company's internal policies by reading a document corpus, four engineers are assigned, and the engineering manager wants the architecture decided by Friday so the work can be split three ways and run in parallel. Nobody has run a single query yet. There is no evaluation set, no baseline number, and no trace of any real user question, because the product surface that would produce those questions does not exist either. The architecture proposal on the table is an orchestrator that decomposes each question into sub-questions, dispatches a retrieval worker per sub-question — a model call that searches the document corpus and brings back the passages it believes answer that piece — and merges the results — a design that three people can build simultaneously, which is exactly why it is being proposed on Wednesday of the first week.

The scheduling pressure is real and the answer to it is not "think about it longer". It is to notice that committing to a topology now means the first numbers this project ever produces will describe the orchestrator, and a number with nothing to compare it against does not tell you whether the orchestrator was worth building. If the merged answers come back at 78% on whatever evaluation eventually exists, you will not know whether a single loop would have scored 77% for a third of the calls or 60% and vindicated the whole design. Recovering that comparison in month three costs a rebuild of the thing you skipped in week one, and by then the prompts have diverged and the tools have grown orchestrator-shaped assumptions, so the rebuild is not cheap and its result is not clean.

Failures in a single loop have one address#

The operational argument for building the loop first is about triage speed rather than elegance. When a coordinated design returns a wrong answer, the wrongness has at least three plausible homes: the decomposition was bad, so the workers answered the wrong sub-questions; the decomposition was fine but a worker retrieved poorly; or both were fine and the aggregation step dropped or contradicted something. Distinguishing those requires reading the plan, then each worker's trace, then the merge input, and forming a judgement about which link broke first. Nothing about that is impossible, and it costs perhaps ten to fifteen minutes per failing case once you are practised at it.

That cost has a shape worth naming, because it is what makes the difference between the two designs grow rather than stay fixed. Locating a fault means reading every artefact that could contain it, so the time to triage one failing case is the number of candidate links multiplied by the time to read one of them. An orchestrator with five workers has a plan, five worker traces and a merge input, so the count of things to read rises with the number of workers, and it rises for every failing case, every week, for as long as the design is in production. A single loop's count is one, whatever the question was. The ten-to-fifteen minutes is not a statement about how difficult coordination is to understand; it is that multiplication, done once per case.

In a single loop, the same investigation is one trace read top to bottom:

run 41ba  question: "how many days of carryover leave can a UK contractor take?"
  turn 1  tool=search_docs  q="carryover leave policy"        -> 4 hits, top score 0.71
  turn 2  tool=read_doc     id=hr-leave-v3 §4                 -> 1,180 tokens
  turn 3  answer: "5 days, per HR-Leave v3 §4"
  grade:  WRONG (expected: contractors are out of scope of HR-Leave; see CONTRACTOR-HB §2)

The failure has one address. The retrieval returned the employee handbook and never surfaced the contractor handbook, because the query the model wrote did not contain the word "contractor". That is a retrieval problem with a specific fix — query expansion, or a metadata filter on worker type — and you found it by reading eight lines. Across a first triage session on this project, 18 failing cases took about 40 minutes to categorise into four buckets, and three of the buckets turned out to be the same retrieval issue.

It is worth following the same failure through the coordinated design, because the reason it becomes hard to see is mechanical rather than mysterious. The worker handling the sub-question about carryover would run the same search, get the same employee handbook, and return a confident passage for its slot. Its slot is now filled, so nothing in the plan or the merge has any signal that the retrieval was wrong. The merge step receives several filled slots, one of which contains a rule from a document that does not apply to contractors, and it writes a fluent answer combining them. The symptom therefore appears at the merge, which is the last place the fault is, and the visible artefact is a blended answer rather than a search that returned the wrong handbook. Each of those links is checkable in a trace, and every one of them has to be checked before the retrieval fix becomes obvious.

That clean attribution is not a permanent property; it is a property of having few moving parts, and it is most valuable in exactly the period when you understand the task least. Later, when the retrieval is good and the prompt is stable and the remaining failures are genuinely about coordination, a richer design becomes a reasonable thing to test — with a baseline to test it against.

The smallest evaluation set that is worth having#

"Build an eval set" sounds like a quarter of work, which is why teams postpone it and then have no numbers for months. The version that fits in a day is much smaller than the version people imagine, and it is useful the moment it exists.

Forty cases is a workable first target: twenty drawn from whatever real questions you can scrape together — support tickets, Slack searches, the questions the internal helpdesk gets most — twelve chosen because you already suspect they are hard, and eight written to be adversarial in a way you specifically care about, such as questions whose answer differs by jurisdiction or by employment type. Each case is a question, an expected answer or a set of facts the answer must contain, and the document that supports it.

{"id": "leave-uk-contractor", "q": "How many days of carryover leave can a UK contractor take?", "must_cite": ["CONTRACTOR-HB"], "must_contain": ["not eligible"], "must_not_contain": ["5 days"]}
{"id": "expense-alcohol", "q": "Can I expense a client dinner including wine?", "must_cite": ["EXPENSE-POL-v7"], "must_contain": ["pre-approval", "£75"]}
{"id": "notice-probation", "q": "What notice do I give during probation in Germany?", "must_cite": ["DE-EMP-ADD"], "must_contain": ["two weeks"]}

The harness that runs them is smaller still, and the point of showing it is that nothing here is a platform investment:

def evaluate(agent, cases: list[Case]) -> Report:
    rows = []
    for case in cases:
        t0 = time.monotonic()
        result = agent.run(case.q)                      # returns .text, .citations, .n_calls
        rows.append(Row(
            id=case.id,
            passed=(
                all(c in result.citations for c in case.must_cite)
                and all(s.lower() in result.text.lower() for s in case.must_contain)
                and not any(s.lower() in result.text.lower() for s in case.must_not_contain)
            ),
            calls=result.n_calls,
            seconds=time.monotonic() - t0,
            text=result.text,
        ))
    return Report(rows)

String containment and citation checks cover more than they look like they will, because most failures in a retrieval-backed agent are the agent citing the wrong document or confidently stating a number from the wrong policy. Cases whose correctness genuinely needs judgement can be graded later by a model against a rubric, which is a written list of what a correct answer has to contain and what it may not claim; the point of Friday's version is that it produces a number, a per-case pass list, and a calls-per-question figure that will matter in chapter 6.

What "passes evaluation" means has to be written down alongside the set, because otherwise the threshold moves to wherever the current build happens to land. For this project it was: 85% of cases pass, no case in the adversarial eight returns a confidently wrong answer without a citation, and median calls per question under 5. Those numbers were agreed with the person who would field complaints about the feature, which is what makes them a threshold rather than a preference.

What the team does on Thursday#

The parallelisation argument deserves a real answer rather than a deferral, and the answer is that most of the work in an agentic project does not depend on topology at all.

Five of six streams are unblocked, which is enough for four engineers, and the sixth is the only one that would have been wasted if the shape turns out to be wrong. The trace schema row carries a condition worth honouring from day one: if your spans — the individual records that make up a trace, one per model call or tool call — record run_id, step, tool, tokens_in, tokens_out and outcome without assuming a particular arrangement of agents, then a later migration to a coordinated design does not invalidate your dashboards, and comparing the two designs is a query rather than a project. Chapter 7 leans on that property.

The column that decides each row is the middle one, and it is answerable without any judgement about the project. A workstream depends on the topology when its interface changes if the arrangement of model calls changes. An index does not know how many calls will query it. A tool is a function with a schema, and the same function is callable by one loop or by five workers. A trace schema depends on the topology only if it names the roles — a column called worker_index bakes in a fan-out — which is why the row carries a condition rather than a flat "no". Reading the table that way turns it from a list of reassurances into a test you can apply to a workstream it does not mention.

Where this advice is wrong#

The rule is to start from the simplest design that passes evaluation, and there are jobs where the simple design cannot pass by construction. A market-scan feature that must research eleven named competitors inside a 30-second budget is one: a single loop researching them sequentially takes eleven times the per-vendor latency no matter how good the prompt is, so the fan-out is not an optimisation, it is the only design in the feasible region. Recognising that requires the shape analysis from chapter 1, and where the shape is unambiguous and the constraint is hard, building a loop first to prove it fails is theatre.

The advice is also not a claim that evaluation sets are reliable. Forty cases give you a standard error of roughly five percentage points around a score near 90%, which means an observed gap of five points between two designs on forty cases is inside the noise. That figure is not a rule of thumb and it is worth being able to reproduce, since it decides how much of your week to spend on a gap. Each case either passes or fails, so a score is a proportion, and the standard error of a proportion is the square root of the score times one minus the score, divided by the number of cases. Near nine tenths that product is nine tenths times one tenth, which is nine hundredths; divided by forty it is a little over two thousandths; and the square root of that is a shade under five hundredths, which is the five percentage points. The formula also says where the number comes from: it falls as the square root of the case count, so quadrupling the set halves the error bar, and adding ten cases to forty barely moves it.

Two residuals come with that closure, and both change what you are entitled to say. The first is that it is the error on one score. Comparing two independently drawn scores puts two such errors together, and the spread of the difference is wider than the spread of either score — which is why an observed gap has to clear more than five points before it means anything. The second is that the arithmetic assumes the forty cases are an independent sample from the traffic you care about, and a set built from twenty scraped questions, twelve suspected-hard ones and eight adversarial ones is not that. It is deliberately enriched for difficulty, so the score it reports is not an estimate of production quality at all, and the error bar describes sampling noise inside the set rather than uncertainty about the field.

What partially rescues small sets is pairing: run both designs on the same cases and compare per-case outcomes, so that the variance from case difficulty cancels and you are only estimating the disagreement rate. The reason the cancellation works is that a case both designs get right and a case both designs get wrong carry no information about which design is better — they would appear in both columns whatever you built — so the comparison rests on the cases where the two disagree, and the effective sample size is the count of those. If design B fixes seven cases that A failed and breaks two that A passed, that is a claim about nine specific cases you can read, and it is far more informative than "93 versus 92". Chapter 6 has an uncomfortable example of exactly that gap.

Finally, none of this says the loop is the design you ship. It says the loop is the design whose number you own by Friday of week two, and that owning that number is what converts every later architecture argument from a matter of taste into a matter of arithmetic. The tempting move at this point is to accept all that and still build the orchestrator first, on the grounds that a simpler path can always be assembled later if the complicated one disappoints. What that reasoning holds constant is the code — the workers really are still callable, the tools really do still exist — while the property that decides whether a comparison is possible is not the code but whether the two designs are still comparable, and that drifts from the first sprint. Prompts get tuned against orchestrator traces, tools grow arguments that only a planner supplies, and the eval set fills with cases discovered by watching the orchestrator fail. The option to simplify survives; the option to compare is what quietly expires.

The argument you will have next is which alternative to build and measure, since there is always more than one candidate and each one costs a sprint to try properly. Deciding that well means writing down, before you start, the single question whose answer would settle it — which is a discipline with a specific form, and it is easy to write a question that sounds decisive and discriminates nothing.