Routing / dispatcher

Read these first

• Choosing an architecture

What you will know

• Classify an input and route it to a specialist path

  A wrong classification commits the request to an unsuitable prompt, tool set, or context budget before the specialist can compensate.

• Use a cheap model or a classifier for the routing decision

  The router should cost and take far less than the selected path, with confidence thresholds preventing an expensive model from deciding every case.

• Measure routing accuracy separately from downstream quality

  A strong final-answer score can hide systematic misroutes that happen to recover, while a correct route can still expose a weak specialist.

• Handle an input that matches no route

  Unknown or overlapping classes need an explicit fallback, rather than silently selecting the nearest specialist or failing closed.

• Compare a router against a single-path baseline on cost, latency, and end-to-end quality

  Specialist branches add classification overhead and operational surface area, so routing is justified only by a measured improvement over the simpler design.

Chapters

1. 1

   A misroute is decided before the specialist can help

   Classification is a commitment: once a support message lands on the billing path it is answered by a prompt with the wrong knowledge, the wrong tools and the wrong context budget, and no amount of specialist focus recovers it. Here you build the classify-then-delegate path and state the condition routing must satisfy — specialists beating one general prompt by enough to survive the router's own errors.

   Read chapter
2. 2

   Your route classes are a design you derive from real traffic

   Every measurement in this topic consumes something the routing diagram never shows: a fixed set of classes, and a set of real requests labelled against them. Derive the taxonomy from traffic rather than from the org chart, and test each candidate boundary twice — can anything separate the two classes from wording alone, and do they genuinely need different knowledge or tools, or should they be one class. Then build the labelled route set: who labels, how many requests, and what you do with the ones two labellers assign differently, because those disagreements are exactly the boundaries a router will get wrong and the reason 'routing is easy' is a claim about one boundary rather than about routing.

   Read chapter
3. 3

   Routing accuracy is a number you have to measure on its own

   A router that is 96% accurate feeding specialists scoring 93% on correctly routed inputs cannot deliver much above 89% end to end, and the aggregate hides errors in both directions. Score the routing decision against labelled routes and keep a per-class confusion matrix, and you can tell a weak specialist from one that rarely receives its own traffic. Specialist fixtures silently assume a correct route.

   Read chapter
4. 4

   The router should cost a fraction of the path it picks

   Using the same large model to choose between four paths can add 700ms to every request and 30% of spend for a decision a small model or an embedding classifier makes nearly as well. Move the decision down, re-measure accuracy against the same labelled routes, and escalate to the expensive model only below a confidence threshold chosen from data — not from the belief that routing is inherently easy.

   Read chapter
5. 5

   Specialist handoff hides the decision nobody is scoring

   The standing counter-proposal is to delete the dispatcher and let each specialist hand off when a request turns out to belong to someone else. That relocates the same classification into a specialist prompt where nobody scores it, pays two full specialist calls every time it fires — worse than the router latency it was supposed to save — and needs a loop guard so two specialists do not bounce a request between them. Weigh it against the one thing a central dispatcher buys you: a single place where the routing decision exists, and therefore a single place it can be measured against labelled routes.

   Read chapter
6. 6

   Every dispatcher needs a path for the input that fits nothing

   Some requests belong to none of your five classes and some sit between two. With no explicit destination they are handed to the nearest specialist, which answers confidently from the wrong playbook. This chapter adds the low-confidence fallback — a general prompt or a human queue — plus the logging that turns unmatched cases into a reviewable group instead of a scatter of one-off complaints.

   Read chapter
7. 7

   Two points spread thin usually means an under-tuned baseline

   Four specialists beat the general prompt by two points, at 300ms per request and five prompts to maintain. The decisive breakdown is where those points live: concentrated in one class argues for a single specialist plus a general default, while an even spread usually means the baseline was simply never tuned. Run that comparison, and name the measured result that would have you delete the router.

   Read chapter