The question that decides between two architectures, written before you choose

Week five, and the policy assistant's single loop — one model answering staff questions against a corpus of internal policy documents, choosing its own searches until it has an answer — passes 86% of the evaluation set at a median of four model calls per question. Someone proposes replacing it with an orchestrator that splits each question into sub-questions and runs a retrieval worker per sub-question. The room produces four arguments for the change within about a minute: internal questions are complex and often have several parts; the orchestrator scales better as the corpus grows; parallel workers will be faster; and orchestrator–worker is the standard pattern for research tasks.

None of those four arguments discriminates, and the reason is worth being able to apply rather than recognise. A discriminating question is one whose two possible answers point at two different designs, so the test is one you can run on a sentence in a meeting without knowing anything about the project: imagine both answers, write down what each one implies, and see whether the two differ.

Four arguments, each answered both ways#

Running the test on the room's four takes less time than making them. If you imagine both answers to "are internal questions complex?" you get the same design either way — complex questions are also what a longer loop is for, so "yes" buys a higher iteration cap and "no" buys a lower one, and neither buys an orchestrator. "Scales better" is not answerable without saying which quantity is growing and what breaks when it does, and following that through is what shows it to be beside the point: what grows is the document corpus, what that changes is the index and the recall of a search against it, and both designs issue searches against the same index with the same retrieval code. A corpus ten times larger degrades both arms in the same way, so there is no answer to "will it grow" that separates them. "Parallel is faster" is a claim about wall-clock latency that has to be weighed against a budget nobody in the room had stated. And "it is the standard pattern" is a description of what other people built for other tasks.

The discipline is to write the question down before choosing, in the same spirit as writing an assertion before the code that satisfies it: you commit to what would count as an answer while you still have no stake in which answer arrives. A question that passes the test has three properties. Its two answers imply different designs. It can be answered from data you already have or can get in under a day. And it is about the work rather than about the technology.

Sorting the candidates#

Putting the room's four arguments through that filter, alongside two better ones, makes the difference visible:

The verdict column has three values rather than two, and the gap between "Partly" and "Yes" is the part that repays reading. The latency row is not vague; it is conditional on a fact nobody has supplied, so it becomes a discriminator the moment somebody writes down the product's budget and measures against it, and stays useless until then. The three "No" rows cannot be rescued that way, because no additional fact would make their two answers point at different designs.

The last two are the pair worth carrying around, because together they are the definition of the shape a fan-out serves. Enumerability alone is not enough: you can list "find the vendor's pricing" and "compare it to ours" up front, but the second cannot start until the first has returned, so dispatching them in parallel buys nothing and forces the aggregation step to reconcile a comparison made against a placeholder.

The two conditions do different jobs, which is why both are needed and why neither alone will do. Enumerability says the work can be named before it runs, which is what an orchestrator has to do in its first call. Independence says the work can be finished in any order, which is what parallel dispatch assumes. A job that is enumerable and dependent can still be written down in advance — as a sequence of stages, not as a fan-out. A job that is independent but not enumerable cannot be dispatched at all, since you have nothing to dispatch until something has run. So the pair closes the question of whether a fan-out is the right shape, and it closes nothing else: both answers can be yes and the fan-out still lose on latency budget, on cost, or on the operational obligations of chapter 5. Availability and correctness of a shape are not the same as it being the shape to build.

The independence half of the pair deserves its own treatment, because it is the one teams wave through. "Can each subtask answer without reading what another found?" is a question about data dependencies between the outputs, and it fails in a way that looks like success: dispatch a dependent subtask in parallel anyway and it will return something, since a language model asked to compare against a figure it has not been given will produce a comparison against an assumption. The result is not an error, it is a plausible answer built on a placeholder, and it surfaces weeks later as a quality complaint rather than as an exception.

That is the stumble worth naming in advance, because the belief underneath it is a belief that a defence exists. The reasoning goes: if a worker were missing something it needed, the run would fail, and the run did not fail, so nothing was missing. What the successful run establishes is that every worker returned a well-formed answer, and a well-formed answer is exactly what a model produces when it is asked to compare against a figure it does not have. The quantity being watched — did the run complete, did the schema validate, was there an exception — is constant across both the healthy case and the broken one, while the property that matters, whether the comparison had its input, varies underneath it. No amount of monitoring the constant will reveal the variable.

Checking the property before you build costs an hour with a sample of real questions and a pen: for each one, write the subtask list, then draw an arrow wherever subtask B's prompt would need to contain subtask A's output. If the arrows exist, the honest options are sequential stages or a loop, and the fan-out only applies within a stage.

On a project with no traces to read, the same question can be answered with a paper exercise that takes an afternoon. Sample thirty inputs, have someone who knows the domain write the subtask list for each one before seeing any results, and then check the lists against what a working run actually needs. A cheaper variant asks a model to produce the subtask list five times per input at temperature 0.7 — high enough that the model's wording and choices vary from run to run — and measures how stable the lists are: inputs whose decomposition is genuinely determined up front produce nearly identical lists, while inputs where the decomposition depends on what you find produce five different lists, and that instability is evidence rather than noise. Neither method is as good as reading traces from a running baseline, which is the strongest reason to have one.

The horizon question, and why it is separate#

A third discriminator matters when the two designs differ in how open-ended they are: is the horizon bounded, and by what? A job with a natural terminal condition — the invoice is either validated or it is not — has a bounded horizon that the code can enforce. A job whose stopping condition is "when the answer is good enough" is bounded only by a cap you invented, and a cap you invented is a place where runs go to produce undefined output.

The horizon question does not always point the same way as the other two. Consider an agent that resolves failing CI builds: read the failure, find the responsible change, propose a fix, open a PR. Median resolution by a human today is about 20 minutes, the latency budget is generous, and the cost budget is not. Two designs are on the table, a single tool-using loop and a planner–executor split, in which one model call writes out the list of steps before any of them runs and a second agent carries them out in the order the plan gives.

The one question that decides it is whether the steps can be written before the first tool call, or whether each step is chosen from what the last one returned. For CI failures it is emphatically the latter: what you do after reading the log depends on whether it is a compile error, a flaky integration test, or a dependency bump, and what you do after finding the responsible change depends on whether the change is one line in a config or a refactor across nine files. That is the step-chosen-from-result shape from chapter 1, which a single loop already handles, and the planner in a planner–executor split would be producing a plan it has to discard as soon as the first tool returns.

The other candidate questions look plausible and fail. "Is the task multi-step?" is true of both designs and therefore decides nothing — multi-step is not the same property as steps-known-up-front. "Is latency acceptable?" is generous here by assumption, so it cannot separate anything. "Is the domain complex?" is about the tools and the prompt rather than the arrangement.

What the pilot's traces would have to show, for the loop to have been the wrong call, is specific enough to query. If a substantial fraction of runs contains a fixed prelude — every single run reading the failure log, then fetching the diff of the last three commits, then running the failing test locally, in that order — then those three steps were never chosen from results and belong in a chain ahead of the loop. If runs regularly exceed twelve iterations while making no progress, repeating the same tool call with the same arguments three or more times, the loop is thrashing in a way that a plan with explicit stages would prevent. And if the cost per resolved build is dominated by re-sending a context that grows every turn — say 38,000 input tokens on the last iteration of a fifteen-turn run against 4,000 on the first — then a design that gives each stage a fresh, smaller context becomes a cost argument rather than a quality one.

The generous latency budget is the part that most often gets misread. Slack latency makes the extra planner call feel free, and it is free in wall-clock only. Every run still pays for the plan's input and output tokens, and in a split design the executor typically re-receives the plan plus the original failure context on each step, so the same tokens are billed two or three times. At roughly 6,000 extra input tokens and 400 output tokens per run, over 900 builds a month, that is on the order of a few million tokens a month bought with no measured quality gain. The arithmetic is one multiplication and it is worth doing rather than accepting: six thousand four hundred tokens a run, nine hundred runs, is between five and six million tokens. What that closure licenses is a comparison between designs on the same billing unit, and its residual is that a token count is not a bill — input and output tokens carry different prices, so the split has to be kept when the figure meets a budget. It also buys an operational surface that has to be maintained forever, which is the subject of chapter 5.

Writing it down so it can be checked later#

The output of this exercise is a paragraph, not a document, and its value comes from being written before the choice rather than after it. Keeping it in the repository next to the design makes it reviewable:

# decisions/2024-11-ci-agent-topology.yaml
question: >
  Can the sequence of steps be written before the first tool call, or is each
  step chosen from what the previous one returned?
evidence:
  source: 120 CI failures sampled from the last 30 days
  finding: >
    3 failure classes (compile, flaky test, dep bump) require disjoint step
    sequences; within a class the 2nd step depends on the 1st step's output in
    71/120 cases.
answer: chosen-from-result
decision: single tool-using loop, iteration cap 15, cost ceiling $0.35/run
flip_if:
  - ">=40% of runs open with the same three steps in the same order"
  - "cap-hit rate exceeds 5% of runs over a two-week window"
  - "median input tokens on the final turn exceed 30k"
review: 2025-01-15

The flip_if block is the part people leave out, and it is what turns a decision into something falsifiable. Each of its three lines is a query somebody can run against the trace store without asking anyone what they think, which is the property to aim for when writing one: a threshold, a window, and a quantity the system already records. For the invoice job from chapter 1, the same field would read: flip when validation can require fetching an attachment and re-extracting, because at that point the operation order stops being fixed. Notice how much narrower that is than "flip if invoices get more complicated" — a condition you could argue about forever, and which no trace can settle.

Two details about the timing matter as much as the content. The question is written before the prototype exists, because a prototype creates an interest: once three days have gone into a planner, the question that gets asked is whether the planner works rather than whether it was needed. And the question is written by whoever will own the thing in production, since that is the person who will be paged when the answer turns out to be wrong, and who therefore has the incentive to write a condition that can actually be checked rather than one that can be argued.

What a written question cannot rescue#

A written question is not a guarantee of a right answer. It cannot rescue you from sampling the wrong 50 runs, and the failure is quiet: a fortnight of pilot traffic is whoever was in the pilot, and if that was one department with one kind of question, the three-fifths-to-two-fifths split is a fact about them rather than about the feature. It also says nothing about whether the design you chose is well implemented, since a correct shape built badly and an incorrect shape built well both score poorly and the number does not distinguish them.

It is also possible to write a question that discriminates beautifully and cannot be answered at any reasonable cost, such as whether next year's traffic mix will still look like this one; when that happens the honest move is to pick the design that is cheaper to abandon rather than to guess at the answer. Which design that is has an answer independent of the traffic: the arrangement with fewer parts to unwind, fewer places where other teams have taken a dependency, and no persisted per-subtask state to migrate.

What the written question does guarantee is that the eventual experiment has a hypothesis attached. That matters more than it sounds, because the experiment itself is where most architecture comparisons go wrong — not in the question, but in a run where the shape was one of several things that changed at once.