The limits of exclusion clauses

An exclusion clause is a sentence in a tool description saying which neighbouring cases belong somewhere else — for ticket history, use search_tickets. Chapter 3 established that it only holds when the receiving tool claims the handoff, so a properly written boundary is two clauses, one on each side. That makes the unit of work the pair rather than the tool, and it is the fact this chapter turns on.

Consider a triage board for an internal operations agent showing 61 bugs closed last quarter, 23 of them sharing the label wrong tool. Every one of the 23 was fixed the same way, by adding a sentence to a description. The result is visible in the repository. get_account_summary now carries a 211-word description of which 140 words are prohibitions, naming five other tools it must not be confused with. Two of its neighbours have paragraphs of similar length. Each of those 23 fixes was locally correct — the description is the only text the model sees when it chooses, so when it chooses wrong the description is where the intervention has to land, and there is no other string with the same leverage. What that reasoning misses is that being the only available input does not make it the right place to intervene.

That reasoning is worth taking apart rather than waving away, because its first two steps are true. The description really is the only text that takes part in the comparison, and an intervention really does have to land there in order to affect a single decision. The step that does not follow is the third one. A fix landing in the only available place is still a fix aimed at a symptom, when what produced the misroute is the arrangement of the tools rather than the wording of one of them — and the tell for that is invisible inside any one bug, because inside one bug the arrangement is a constant. Twenty-three tickets, each correctly diagnosed and each correctly fixed, is what a structural problem looks like when it is met one ticket at a time.

Clauses grow with pairs, not with tools#

Twenty-four tools have 276 pairs; twenty-five have 300. The twenty-fifth tool does not add one tool's worth of boundary work, it adds twenty-four new relationships, and every one of them is a place where a request could plausibly land in either of two tools.

Both counts come from the same small formula: every tool pairs with every other, which is twenty-four choices times twenty-three on a twenty-four-tool surface, halved because each pair has been counted once from each end. The difference between the two totals is the more useful quantity, and it is no coincidence that it equals the old tool count. A new tool pairs with each existing tool exactly once, so the number of new relationships it creates is the number of tools that were already there. The work of adding a tool therefore grows every time a tool is added, while the work of writing a tool stays where it was — which is how a surface goes from comfortable to unmanageable with nobody having done anything differently.

Most of those pairs are not contested, which is what makes large tool surfaces workable at all. send_slack_message and get_disk_usage will never compete, and no clause is needed. The pairs that are contested cluster in domains: five tools that all read customer state, four that all export something. Within a cluster the pair count is what matters, and a cluster of six tools has fifteen internal pairs. Once four or five of those are contested, each description in the cluster is carrying clauses about three or four siblings, and the paragraph appears, because a description in a dense cluster is being asked to encode a boundary that has more sides than a sentence has room for. Nobody had to write a bad description to get there. A 140-word exclusion paragraph is a measurement of cluster density that happens to be stored as prose.

Follow the counting into a single cluster and the paragraph stops being a mystery. Six tools give fifteen internal pairs. Each contested pair needs a clause on both sides, because an exclusion holds only when the receiving tool claims the handoff, so five contested pairs is ten clauses spread across six descriptions. A tool sitting in the middle of the cluster is on one side of three or four of those pairs, which puts three or four clauses in its own description, each naming a different sibling and each needing enough words to say what is being handed over and what is being kept. That arithmetic produces a hundred and forty words with nobody writing badly at any point, and it produces them in whichever tool is most central to the cluster rather than in whichever tool is worst described.

Telling a description bug from an architecture bug#

The general claim is not much use during triage, where the question is always about one misroute. Three checks separate the two diagnoses, and they run in order of how cheap they are.

The first check costs nothing and rules out the largest class. Write the boundary as a sentence and ask whether the model could evaluate it from the request and the transcript, which is chapter 4's evaluable-at-selection-time test applied to a rule you are about to write rather than one you already wrote. Use this for balances and that for contact details passes. Use this for enterprise accounts fails, because the tier is not knowable until something has been looked up, and no amount of rewriting converts an unevaluable rule into an evaluable one. A boundary that fails this check is an architecture finding on its own: either the two tools should be one tool that branches internally, or the tier lookup needs to happen before the choice is offered.

The second check is a replay with the competitor removed — an ablation, meaning you rerun the same cases with one part of the system taken away and read off how much that part was contributing — and it is worth automating because it answers a question the raw logs cannot.

def ablate(cases, tools, hide: str, route):
    """Replay misrouted cases with one rival tool removed from the block."""
    reduced = [t for t in tools if t["name"] != hide]
    correct = sum(1 for c in cases if route(c.request, reduced) == c.expected)
    return correct / len(cases)


misrouted = [c for c in contested if route(c.request, TOOLS) != c.expected]
print(f"with rival present: 0.0%  (n={len(misrouted)})")
print(f"with rival hidden:  {ablate(misrouted, TOOLS, 'get_account_summary', route):.1%}")

If the requests that were losing to get_account_summary route correctly once it is hidden, then the request contained enough signal all along and the two descriptions simply failed to divide it — a description bug, fixable by writing both sides. If they are still wrong with the rival gone, landing on a third tool or producing invented arguments, then the signal was never in the request, and rewriting prose is going to move the errors around rather than remove them. On the operations agent, the get_account_summary versus get_billing_state pair ablated to 96%, and the get_account_summary versus search_tickets pair ablated to 41%.

Read those two figures as answers to the question the ablation actually asks, which is whether the information was ever in the request. Ninety-six percent means that with the rival gone, nearly every previously misrouted request went where it belonged — so the request had always carried enough to decide it, and two descriptions had failed to divide something divisible. Forty-one percent means the reverse: with the rival gone, most of those requests still landed somewhere wrong, and that cannot be the rival's doing, because the rival was no longer there. Whatever was missing was missing from the request. The distance between the two numbers is why the ablation is worth automating, since both pairs looked identical on the triage board and only one of them has a fix that involves writing anything.

The third check is about shape. Tabulate misroutes as ordered pairs of chosen and correct, as in chapter 3, and look at how concentrated the distribution is.

The second row is the architecture finding that hides most successfully as a description problem, because from inside any single bug it looks exactly like the first row. It only becomes visible when you stop triaging bugs one at a time and look at a quarter of them together, which is the practical reason the standing fix survived fifteen iterations.

What actually breaks first#

The intuition is that clauses stop working when the model starts ignoring them, and that is not what happens first, or at all in any clean way. The first failure is the opposite of the one you were guarding against. A description that is 140 words of prohibition and 20 words of positive claim is a weak positive match in a comparison that ranks candidates, so the tool starts losing requests that genuinely belong to it. On the operations agent, get_account_summary's accuracy on requests it should own fell from 91% to 78% over two quarters while the exclusion paragraph grew from nothing to 140 words. The exclusions did their job — it stopped stealing from its neighbours — and it also stopped winning its own traffic, and because the eval reported overall accuracy the two effects partly cancelled and the aggregate barely moved.

Subtract the two figures in the repository and seventy-one words are left in get_account_summary for everything that is not a prohibition: the positive claim, whatever it says about its arguments, whatever it says about what comes back. Two thirds of the string is about what the tool is not for. A ranking has no slot for that. When the model weighs this description against a request the tool genuinely owns, most of what it is weighing is text about other tools, and the part arguing that this is the right choice has been squeezed into a third of the space it used to have. The exclusions did exactly what they were written to do, and the cost landed on an axis nobody was measuring.

The second failure is staleness, arriving through the mechanism chapter 2 described but at the level of the surface rather than the argument. A clause reading for ticket history, use search_tickets is a claim about another tool, and nothing in your build breaks when search_tickets is renamed, rescoped, or removed. In a cluster of six tools with cross-references in every description, a single rescope invalidates clauses in five other files that no test covers and no reviewer of that pull request will open.

The third is asymmetry compounding. Symmetric coverage of a six-tool cluster with five contested internal pairs takes ten clauses; teams write five, because each one is added in response to one bug and the bug names one direction. The unclaimed side of each boundary keeps competing at full confidence, which is the ten-point shortfall chapter 3 measured, now present in five places at once and interacting.

Underneath all three sits chapter 5's bill. Those three paragraphs total around 900 tokens, re-sent on every turn of every session, and every edit to any of them invalidates the cached prefix for every conversation in flight.

Nine hundred tokens is the block cost of the boundary work alone, before any tool has said what it does, and the multiplier from the pricing chapter applies to it unchanged — it is paid once per turn and not once per session. The second half of that sentence is the sharper one for a team that edits descriptions often. Exclusion paragraphs are the part of a tool block most likely to be revised, because they change whenever a neighbour changes, so the least stable text in the prompt is sitting at the position where instability costs the most.

What to require before the twenty-fifth tool#

The intervention that does not work is a review checklist for new tools, which is the natural-seeming response and quietly makes things worse: it institutionalises adding tools by giving the process a green path, without ever putting the alternative on the table. Four requirements do better, and each of them is a gate rather than a form.

The first is a written partition statement — one sentence naming the axis that separates the new tool from its nearest existing neighbour, evaluable from the request alone. If nobody can write that sentence, the tool does not ship, because the model is going to be asked to apply a rule that its authors could not state. Writing it is also the cheapest way to discover that the new tool is the old tool with a different backend.

The second is eval coverage for the pairs the tool creates, not for the tool. That means contested-slice cases for each existing tool it plausibly competes with, gated on per-pair accuracy in the shape chapter 6 built, plus cases in the no-tool slice. A new tool that passes an aggregate and degrades one neighbour by eight points has shipped a regression nobody will find for a quarter.

The third is a token budget entry. The block has a cap; the new definition has to fit inside it; when it does not, something is cut or merged, and that conversation happens before the merge rather than during the next cost review.

The fourth is a question the review must answer rather than a rule it must satisfy: which existing tool does this make redundant? Sometimes the honest answer is none. When the answer is it partly overlaps these two, you have found the merge before it becomes three exclusion paragraphs.

Where this stops being a solved problem#

None of this says twenty-four tools is too many. The count is a bad metric on its own, because an agent with sixty tools spread across disjoint domains routes cleanly while nine tools over a single domain can be unusable. Contested pairs are the thing to count, and they are countable from your own logs rather than from a rule of thumb.

The token half of the problem has a real mitigation that the selection half does not. Tool search and deferred loading keep definitions out of the prompt until the model asks for them, which removes most of chapter 5's per-turn bill and preserves the cached prefix because schemas are appended rather than swapped. What they do not do is remove the comparison; they relocate it into a retrieval step, where a search over descriptions now decides which candidates the model gets to see. A pair that two descriptions could not separate is a pair that a retriever will not separate either, and the failure moves from a wrong call to a missing candidate, which is harder to see in a trace. That is roughly where the state of the art sits, and it is an honest limitation rather than a temporary one: nobody has a general answer for the contested middle, only better places to put the ambiguity.

Exclusions are the cheapest intervention in the system and the one to reach for first. Their exhaustion is the most reliable signal you will get that the problem has moved somewhere else, and reading that signal correctly is a matter of three checks — is the boundary evaluable, does the rival's removal fix it, and is the damage in one pair or spread across a cluster — rather than of writing the sentence better.

Which leaves the practice, and it is small. Keep a routing log with labelled correctness so the ordered-pair table can be built on demand rather than reconstructed during an incident. Audit contested pairs on a fixed cadence instead of when a bug arrives, because the concentration signal only appears in aggregate. Put the partition statement in the pull-request template for new tools, where it costs a sentence and occasionally saves a quarter. And when the checks say the problem is the surface rather than the string, the answer is in topic 5.10, on tool granularity and surface — while what the tool sends back once it has been chosen correctly is the subject of topic 5.12, on tool results.