Routing accuracy is a number you have to measure on its own

A routed system produces two things a grader could look at. One is the answer the customer receives, scored against the end-to-end rubric, which is the written scoring guide a grader applies to a finished answer to decide whether it was any good. The other is the label the router emitted on the way, which is a prediction with a right and a wrong answer and which usually never gets extracted as an event at all. Teams measure the first because it is the thing users see. The second is what this chapter is about, and the reason to insist on it is not tidiness — it is that the first number cannot be interpreted without it.

Two claims follow from the structure of the design, and both are arithmetic rather than opinion. Routing accuracy sets a hard ceiling on end-to-end quality, so it is not optional information: it is the term that determines how much headroom specialist work can possibly recover. And the aggregate score is built from two independent factors, which means a single number cannot tell you which one moved. Everything below is a consequence of those two.

The ceiling the router puts on everything behind it#

Start with the ceiling, because it is the fastest way to see why the aggregate cannot answer the question. Take a router measuring 96% against labelled routes and specialists scoring 93% on inputs that were routed correctly.

A request is answered well by a correctly routed specialist only if two things both happen: the router sends it to the right place, and the specialist then answers it well. Those are two separate conditions, and the share of traffic satisfying both is the product of the two shares rather than the smaller of them, in the same way that the share of a population that is both left-handed and left-footed is smaller than either share on its own. So the requests that reach the right specialist are 96% of traffic, 93% of those are answered well, and that path contributes 0.96 × 0.93 ≈ 89.3%.

The remaining 4% are answered by the wrong prompt with the wrong tools. They are not automatically failures — some misroutes are near-misses that a neighbouring specialist half-answers — so whatever they contribute is a bonus on top of 89, and the true figure is 89% plus some fraction of 4%, not 89% flat. That fraction is bounded by 4%, since a share of traffic cannot contribute more than itself.

Both bounds are now available, and they are what makes the end-to-end number readable. The figure cannot be 93%, because 93% assumes every request reaches its specialist and 4% of them do not. It cannot be much above 90%, because the whole of the misrouted slice is only 4 points wide even if every one of those requests is answered perfectly. If the end-to-end number comes back at 91.5%, that is consistent with this system. If it comes back at 94%, one of the three numbers is wrong and the most likely candidate is the rubric, because a rubric is the only one of the three that can drift without anybody editing a prompt.

The aggregate conceals errors running in both directions#

The first direction is a misroute that gets answered well anyway. A message about a duplicated webhook lands on billing; the billing specialist happens to notice the word webhook, hedges, and says something about delivery retries that is close enough for the grader to pass it. The end-to-end score records a success. What actually happened is a routing failure that got lucky, and it will not get lucky on the next one, and there is no record anywhere that the request was misrouted at all. A system can accumulate a great deal of this and look healthy, because the only trace it leaves is a slightly worse answer on a scale that has no place to record why.

The second direction is more expensive: a correctly routed message sitting on a specialist that is worse than the general prompt was. The how_to prompt was written to be lean — 600 tokens of style and scope rules plus search_docs — on the theory that documentation questions do not need domain material in context. If that theory is wrong for a slice of how-to questions, those requests are now answered worse than they were before the router existed.

The reason that failure hides is a matter of weights. A class contributes to the aggregate in proportion to its traffic share, so a small regression on a large class and a large gain on a small class can cancel exactly. Because how_to is 38% of traffic, a one-point regression there cancels a five-point gain in a class that is 19% of traffic, and the aggregate moves up because a different class improved. Nothing in the aggregate says which.

Both directions are invisible for the same reason: the routing decision was never scored against anything. The fix is to treat it as its own prediction with its own ground truth — the independently established correct answer a prediction gets compared against — which the labelled route set from the previous chapter provides.

from collections import Counter

def route_report(items, classify):
    """items: the labelled route set. classify: message -> route label."""
    matrix = Counter()                       # (true, predicted) -> count
    for item in items:
        matrix[(item["route"], classify(item["text"]))] += 1

    routes = sorted({r for r, _ in matrix} | {p for _, p in matrix})
    for route in routes:
        actual = sum(n for (t, _), n in matrix.items() if t == route)
        served = sum(n for (_, p), n in matrix.items() if p == route)
        hit = matrix[(route, route)]
        print(f"{route:18} recall {hit/actual:5.1%}  precision {hit/served:5.1%}"
              f"  n={actual}")

The two sums in that loop are the whole idea. actual adds up every cell whose true route is this class, which is a row of the matrix; served adds up every cell whose predicted route is this class, which is a column. The same diagonal cell, hit, is divided by each in turn, which is why one class yields two different percentages.

Running that offline costs one router call per labelled item and nothing else — the specialists are never invoked, so the whole report on 800 items is a few minutes and a couple of dollars. That cheapness matters, because it means routing accuracy can be a pre-merge check on every change to the router prompt rather than a quarterly exercise.

What the confusion matrix says that the accuracy number does not#

Here is the confusion matrix for the support assistant's router, run over the 800-item labelled set. Rows are the true route, columns are what the router predicted, and each cell holds the number of requests that fell into that combination, so a perfect router puts every count on the diagonal and zero everywhere else.

The two per-class figures it yields answer different questions. Recall for a class is the share of the requests that genuinely belong to it which the router actually sent there, read along the row. Precision is the share of the requests the router sent to that class which genuinely belonged, read down the column. A specialist can be starved by poor recall and polluted by poor precision at the same time, and the two are repaired by different work.

Work one row and one column by hand, because the two directions are easy to confuse until you have done each once. The api_integration row totals 120, which is the parenthesised figure next to the row label and also the sum of the cells along it. Of those, 94 landed on the diagonal, so recall is 94 divided by 120, which is 78.3%. Now the billing column: adding the cells down it gives 235, of which 214 sit on the diagonal, so precision is 214 divided by 235, which is 91.1%. The billing specialist therefore receives 235 requests of which 21 belong somewhere else, 17 of them integration questions, which is to say it is being graded in production on a stream that is 9% requests it has no tools for. Its end-to-end score is depressed by traffic that is not its fault, at the same time as the integration specialist's score is depressed by traffic it never sees. Recall and precision are the two directions, and a matrix gives both per class where an accuracy figure gives neither.

Raw accuracy on this set is 758/800 = 94.8%, which is the sum of the diagonal over the total. Weighted by real traffic share it is 93.8%, and the two differ because the set was stratified — the small classes were topped up, so the set over-represents incident_status relative to production, at twice its production share, and under-represents api_integration. Weighting to traffic therefore takes weight away from a class the router almost never gets wrong and gives it to the class the router gets wrong most, and the figure falls. Report both, and use the traffic-weighted figure whenever you are reasoning about end-to-end impact. A single "94%" that does not say which of the two it is will be compared against a number computed the other way within about a month.

Now read the matrix rather than the summary. One row is nothing like the others: 22% of integration traffic never reaches the integration specialist, and it is not scattered — 17 of the 26 losses go to billing, the exact boundary where the human labellers disagreed most in the previous chapter. That row is completely absent from 94%, and the reason is a multiplication. A class contributes its own error rate to the aggregate only in proportion to its traffic share, so a 22-point hole in the 19% of traffic that api_integration carries moves the aggregate by about four points — which looks like noise next to a headline that went up. The aggregate is not lying. It is averaging, and averaging is exactly the operation that turns a hole into a ripple.

When complaints cluster in one class, look at the matrix before the specialist#

The integration specialist scores 94% on its own fixture set — 35 hand-written traces with expected diagnoses, refreshed twice, passing consistently. The reading that follows naturally from that is that the specialist has been ruled out, and it is exactly backwards, in a way worth naming because it is a belief that a defence works. Every one of those 35 fixtures is an integration question sitting in an integration fixture file, which is to say every one of them was routed correctly by construction. The fixture set measures the specialist's performance conditional on a correct route. It cannot detect that a fifth of the real traffic never arrives, because the arrival is the assumption the file is built on.

That is the quantity that stays reassuringly constant while the dangerous one moves. A fixture score is computed over a population the router cannot touch, so it holds steady at 94% whether the specialist receives all of its traffic or none of it. A passing fixture run is therefore evidence about the prompt and no evidence at all about the system, and the two get conflated because the same specialist's name is attached to both numbers.

So the first hypothesis, when complaints concentrate in one class whose specialist passes its fixtures, is that the requests generating those complaints are not reaching that specialist at all. The test is cheap and needs no rebuild. Take 200 recent complaints from customers in that class, have someone label the true route for each — an hour of work against the definitions already written — and join against the routing log to see where each one actually went. If most of them were routed correctly, the specialist genuinely is the problem and the fixtures are unrepresentative. If a large share landed on billing, the specialist was never on the path and its prompt is not the thing to change.

The offline matrix goes stale, and a macro average is not a diagnosis#

Everything above is computed against a fixed file, which means it describes the router's behaviour on a sample of traffic from whenever that file was built. Traffic moves: a new product surface ships and generates a class of question nobody labelled, a marketing campaign brings a wave of trial accounts asking things the taxonomy has never seen, month-end shifts the mix toward billing by six points. The offline number can be excellent while the deployed router degrades, and the gap between them widens quietly.

The cheap countermeasure is to emit the routing decision as a structured event on every production request — the chosen route, the runner-up, whatever confidence the classifier exposed, and the request identifier — and then to watch two things that need no labels at all. The first is the predicted class distribution against the labelled set's traffic weights: if how_to was 38% of traffic in March and the router is assigning it 51% in May, something changed, and it is either the traffic or the router. The second is the rate at which requests get corrected downstream, whether by a specialist escalating or by a human reopening the ticket, grouped by predicted class. Neither is a substitute for the labelled comparison, and both fire before the quarterly relabelling would have caught the drift.

The other trap is the average over classes. Macro-averaging the five recalls, which counts each class once regardless of how much traffic it carries, gives 93.7%, and traffic-weighting gives 93.8%. The two land within a rounding of each other here for a reason that is easy to check and easy to misread: macro weighting gives the weak class a full fifth of the total, traffic weighting gives it slightly less, and the difference between those two weights is far too small to show. Both round to the same reassuring figure while the underlying distribution is four classes above 96% and one at 78%. Averages of a bimodal distribution, one whose values bunch at two separated levels with nothing in between, describe nothing. The per-class rows are the deliverable; the summary line is a convenience.

And a perfect router in front of bad specialists produces bad answers, with a routing report that looks immaculate throughout. Nothing in this chapter argues that routing accuracy is the number to optimise. It argues that it is the number that has to exist separately so that the end-to-end number becomes interpretable — two measurements, one of which sets the ceiling and one of which says how much of the ceiling you reached.

Both of the numbers this chapter produced were bought with a model call in front of every request, and that call is not free. The router in this system runs on the same large model as the specialists, adds about 700ms to every turn, and accounts for roughly 30% of total spend — for a decision that is a five-way classification of a hundred-odd tokens of text. The next chapter moves that decision down the price curve, re-measures it against this same labelled set, and works out what confidence threshold sends the genuinely hard cases back up.