A misroute is decided before the specialist can help

Routing is the design in which a cheap first step reads an incoming request, assigns it to one of a fixed set of classes, and hands it to a path built for that class alone. The component that performs the assignment is the router, also called the dispatcher, and the paths it chooses among are specialists. Each specialist is an ordinary model call with three things settled in advance: a system prompt, which is the standing instruction and reference text the model is given before it ever sees the customer's message; a set of tools, the functions it is permitted to call to look something up or change something in another system; and a context budget, a ceiling on how much text the whole request may occupy, measured in tokens, the sub-word chunks a model reads and writes. This topic works through routing on one running example, a support assistant with five such paths sitting behind the same front door, and it returns to that assistant in every chapter because the design decisions only become visible against a fixed taxonomy and a fixed set of measurements.

The piece of machinery that makes the decision has a name worth attaching to it before any story about a routed system finishes, because the property to carry through all of this is that classification is a commitment rather than a suggestion. A specialist that receives a request outside its class is not merely a slightly worse answerer of the question in front of it. It is answering with the wrong instruments, and it usually does so confidently, because nothing in its prompt suggests the question might not be its own. That is why the accuracy of the step in front of the specialists governs the value of the specialists themselves, and why a routed system has a quality ceiling that no amount of work on the prompts behind the router can lift.

Why one prompt for everything gets worse as it gets broader#

Routing exists because the alternative is a single prompt that has to be good at everything, and it is worth being precise about how that prompt degrades rather than asserting that it does. Put the pricing rules, the webhook retry semantics, the sign-on configuration and the documentation style guide into one system prompt with all nine tools attached. Three separate costs follow, and each one follows from a property of how the call is executed.

The first is attention. Everything in the system prompt is in context on every turn, and the model has to decide, for the message in front of it, which parts of that material are relevant. Material that has nothing to do with the question is not inert — it is competing text that the model has been told to treat as instruction. The second is tool selection. The model chooses a tool by matching the request against the tool descriptions it has been given, so nine similar-looking tools with overlapping surfaces present a harder discrimination than three distinct tools with disjoint ones, and a wrong first call is the ordinary outcome of a hard discrimination. The third is billing and latency. A prompt is processed in full before any output appears, so the whole of it is paid for whether the customer asked about an invoice or an outage.

Cutting that one prompt into four narrower ones removes all three costs at a stroke, and creates an obligation that did not exist before. Each narrow prompt carries only its own domain material, offers only its own tools, and is only as long as its own subject requires. But a narrow prompt is only correct for one kind of request, so something now has to decide which of the four a given message belongs to. That decision is the router, and it is a component the single-prompt design did not need to have. Naming it that way is the important part: the router is not a feature of the specialists, it is the price of having them.

The pattern turns up wherever one entry point serves visibly different kinds of request. Support triage of the kind this topic uses is one instance. Coding assistants that send "explain this file" and "rename this symbol across the repository" down different paths are another, since the two need different tools and different amounts of the repository in context. The cost-driven version, in which a first stage sends routine requests to a small model and difficult ones to a large one, is a third. In every one of them the value of the specialists depends on the accuracy of the step in front of them, which is the same statement made three times about three different systems.

The dispatch itself is small#

The mechanism is one extra model call in front of the one you were already making. The router receives the user's message, returns a label constrained to a known set, and the label indexes a table of handlers.

import json
from anthropic import Anthropic

client = Anthropic()

ROUTES = ["billing", "api_integration", "account_access", "how_to", "incident_status"]

ROUTE_SCHEMA = {
    "type": "object",
    "properties": {"route": {"type": "string", "enum": ROUTES}},
    "required": ["route"],
    "additionalProperties": False,
}

def classify(message: str) -> str:
    response = client.messages.create(
        model="claude-opus-5",
        max_tokens=64,
        system=ROUTER_SYSTEM_PROMPT,       # class definitions + boundary examples
        output_config={"format": {"type": "json_schema", "schema": ROUTE_SCHEMA}},
        messages=[{"role": "user", "content": message}],
    )
    return json.loads(response.content[0].text)["route"]

def handle(message: str) -> str:
    route = classify(message)
    return SPECIALISTS[route].run(message)  # each entry: prompt + tools + budget

Two details in that listing carry more weight than their size suggests. The first is the enum, a fixed list of permitted values that the model's output is required to be one of. The route is constrained to the five strings the dispatch table knows about, and the reason that matters is a mechanical one: SPECIALISTS[route] is a dictionary lookup, a dictionary lookup on a key that is not present raises KeyError, and the constraint on the output is the only thing standing between that exception and a label the model invented. Routers written without it fail in a characteristic way. They return "billing_question" or "Billing" or a short sentence explaining the choice, and the dispatch either crashes or falls through to whatever the else branch does, which is usually the most general path, which is usually the wrong one.

The second detail is that classify and run are separate calls with separate prompts. The router's prompt contains the class definitions and nothing about how to answer anything; the specialist's prompt contains everything about how to answer and nothing about the other four classes. Neither one holds the whole system in its head, which is the point of the design and also the source of the failure the rest of this chapter is about, since a prompt that has never been told the other classes exist cannot notice that it is holding one of them.

The incident_status branch is worth noticing because it breaks the symmetry. Messages asking whether the API is down do not need a specialist prompt at all; they need the current status-page state and a templated reply, which is ordinary code. Routing does not oblige you to send every class to a model, and one of the cheapest wins in a dispatcher is discovering that a slice of your traffic has a deterministic answer. That is a design decision rather than a requirement: nothing about the routing pattern forces it, and it is available because somebody looked at what a class of messages actually needed rather than at what the other four branches happened to do.

What each specialist inherits at the moment of the decision#

Read that table as a description of what a misroute costs rather than as configuration. Three things are fixed at the moment the router emits a label, and each is fixed for a reason that can be checked.

The knowledge is fixed because the specialist's system prompt is the only domain material in context, and a model cannot cite or apply a document it was not given. The action space is fixed because the tool schemas were chosen per-route, and a model can only call a tool that was declared to it in that call — the billing prompt does not have a degraded version of search_delivery_log, it has no representation of it at all. The context budget is fixed because each path allocates its window differently, and the allocation was sized against the class the path expects: the billing specialist spends most of its window on policy text, the integration specialist spends most of its window on log output.

Follow those three through one wrong label. A message about a duplicated webhook delivery that lands on billing is handled by a prompt carrying 3,000 tokens of refund policy and no way to see a delivery attempt. A message about a refund that lands on api_integration is handled by a prompt that will happily search the delivery log for a payment that was never an event. The context budgets differ by a factor of three between paths, which matters because the integration path was sized for verbose log output and the billing path was not, so a misrouted integration question can hit the billing path's tighter budget even if it somehow got the right tool. These differences are the entire reason to build specialists. They are also the entire reason a misroute is unrecoverable.

The step most people skip is the last one, and it is where the belief that a defence works does its damage. The intuition is that a specialist handed the wrong request will notice, hedge, or say it cannot help, because a competent human on the billing desk would. It does not notice, and the mechanism is visible in the three fixings above. The specialist has no representation of the alternative paths, so there is nothing for it to compare its own fit against. Its prompt tells it that it is the billing specialist and that billing questions arrive here, so the arrival of a message is itself evidence, inside the prompt's own frame, that the message is billing. Its tools return real data — a real ledger, a real charge history — so nothing in the tool results contradicts the frame either. The output is therefore fluent, specific and sourced, and fluency is the quantity that stays exactly constant whether the route was right or wrong. The dangerous property varies; the signal a reviewer instinctively reads does not.

Contrast the routed design with the single general prompt it replaces: one system prompt carrying all four knowledge bases, all nine tools, and a context budget large enough for the worst case. It answers the webhook question adequately because search_delivery_log is in its tool list and the retry semantics are somewhere in its prompt. It is worse on average, for the three reasons the previous section gave. What it never does is become structurally incapable of answering a question it was given, and that asymmetry — worse on average, never impossible — is what makes the comparison between the two designs an arithmetic question rather than a matter of taste.

The condition routing has to satisfy#

Because the two designs fail in different ways, comparing them requires more than "specialists are more focused." Write it as expected quality, which means the average score over the traffic, with each outcome weighted by how often it happens.

Let p be the router's accuracy, meaning the share of requests it assigns to the class they belong to. Let S be the specialists' average score on requests they should have received, M the average score on misrouted requests, and G the general prompt's score. A routed request has exactly two fates: it reaches the right specialist, which happens a share p of the time and scores S, or it does not, which happens the remaining share and scores M. Those two fates are exhaustive and mutually exclusive, so the expected score is the sum of each outcome's score times its share, and routing is worth building when

p·S + (1 − p)·M > G

For the support assistant, measured over a month, G was 87%, S was 93%, and M — the score on the requests that landed on the wrong specialist — was 55%, which is higher than zero because some misroutes are near-misses that a neighbouring specialist can still half-answer. At the router's measured accuracy of 94%, the routed design predicts 0.94 × 93 + 0.06 × 55 ≈ 90.7%, comfortably above 87%.

The more useful thing to do with that inequality is to solve it for p, because p is the term you can change by working on the router. Set the two sides equal and rearrange. Expanding the left side gives p·S + M − p·M, so collecting the p terms turns the equation into p·(S − M) + M = G. Subtract M from both sides and divide by S − M, which is legal here because S − M is positive whenever the specialists are better on their own traffic than on somebody else's, and if that difference were zero there would be nothing to route for. The break-even accuracy is therefore

p* = (G − M) / (S − M)

which reads as the share of the gap between misrouted and correctly routed performance that the router has to cover in order to reach the general prompt. Substituting the measured figures gives the same thing as the arithmetic in the original terms: 55 + 38p = 87, so p ≈ 0.842. A router below roughly 84% accuracy makes the whole system worse than the prompt it replaced, and it does so while adding a model call, four prompts, and a class taxonomy to maintain.

That break-even is the argument the "specialised prompts are obviously better" position leaves out. Its premise is true — each specialist prompt is shorter, has fewer distractor instructions, and does not have to work out which product area it is in — and every one of those benefits is contained in S. The comparison it performs is S against G, 93 against 87, which is a comparison between the specialists and the general prompt on the assumption that the specialists receive their own traffic. The router is the thing that makes that assumption true or false, and it appears nowhere in the argument. Six points of specialist advantage sounds decisive until you notice that it is fully consumed by a router 16 points short of perfect.

The formula also says which lever is worth pulling when the condition fails. Because p* falls as M rises, a system whose misroutes land somewhere survivable needs a less accurate router than one whose misroutes fall off a cliff, and raising M is often cheaper than raising p. That is not an aside; it is the whole of the redesign the last chapter of this topic arrives at.

Two things about that arithmetic are worth flagging as simplifications. It treats S and M as single numbers when in fact they vary by class, and a system whose aggregate looks like 90.7% can contain one class where the specialist is worse than the general prompt and another where it is far better. It also treats p as a property of the router rather than of each boundary, when in practice the accuracy on incident_status and the accuracy on the line between billing and api_integration are not remotely the same number. Both simplifications hide real failures, and both get taken apart later.

Routing is not one agent choosing a tool#

The neighbouring idea, and the one it gets confused with, is tool selection inside a single agent. Both involve a model picking among options, and both can be described as "the model decides which capability to use." The difference is recoverability, and it comes from what the model sees after a wrong choice. When an agent calls search_docs and gets nothing useful, the empty result is returned to it as part of the conversation, so the wrong choice is observable from inside the same turn and the agent calls search_delivery_log next; the cost is a round trip. When a router picks billing, the billing specialist does not see an empty result — it sees a plausible ledger and produces a plausible answer, and the alternatives were never declared to it. The routing decision is irreversible within the turn precisely because the alternative paths are invisible from inside the chosen one.

Routing is also not orchestration. A dispatcher performs no decomposition: it does not split a request into parts, it does not sequence specialists, and it does not combine their outputs. One request in, one label, one path, one answer. If a support message genuinely contains a billing question and an integration question — and about 4% of this traffic does — routing handles it by picking one, which means it handles it badly. That limitation is structural and no amount of router accuracy addresses it, because there is no single correct label to be accurate about. A request with two correct answers cannot be scored against a file that records one.

And routing does not repair a weak specialist. If the how_to prompt is worse at how-to questions than the general prompt was, sending it exactly the right traffic makes that class worse than before while the aggregate stays flat, because the classes that improved absorb the difference. Routing changes which prompt sees which input; it has no opinion about whether that prompt is any good. Those two failures — a class with no correct label, and a class whose specialist is a regression — are invisible in the inequality above, since both are hidden inside averages that the inequality treats as constants.

Everything in this chapter assumed two things you do not yet have. It assumed the five classes are the right five, when they were chosen by looking at the support team's queue names, and nothing has yet tested whether those names describe boundaries a classifier can find or boundaries that change what happens next. It also assumed "94% accurate" is a number someone measured, which requires a set of real requests with known correct routes — a different artifact from the fixtures the specialists are scored on, and the thing every measurement in this topic consumes.

Both of those gaps sit underneath the condition this chapter derived, since p, S, M and G are all figures that have to come from somewhere before the inequality means anything. The next chapter builds both: the taxonomy, derived from traffic rather than from the org chart, and the labelled route set that makes any claim about router accuracy checkable.