Fault injection at the boundary

Happy-path coverage answers whether the tools work. It does not answer the question a boundary owes before it ships, which is what the agent does when the boundary is having a bad day — and a boundary has four distinct kinds of bad day that a direct function call would never have made anyone think about.

Answering that is a matter of fault injection, which means deliberately breaking a dependency under controlled conditions and asserting on what the caller does. The caller here has two halves worth separating: the model, and the orchestration around it — the code that decides which tools the model is offered, assembles their results, and constrains what the turn is permitted to say. Most of the useful assertions land on the second half, because it is the half whose behaviour is deterministic.

Four faults, three of which raise something#

These four are the list because they are the distinct ways the boundary can misbehave as seen from the caller's side, which is a smaller set than the seven added failure modes counted in chapter 1, on measuring a function against a server. A rolled-back server, a client configuration that was never updated, and a tool that has been removed all present to the agent as either unavailability or drift, so they are covered by injecting those two rather than by enumerating causes. The equivalent list for the direct function call is empty. Functions do fail — the database can still be down — but those failures are in-process exceptions raised by code the same team wrote, tested and deploys, and they land in a traceback rather than in a well-formed response.

Running that mapping out in full is what shows the four are a partition rather than a shortlist. Two of chapter 1's seven — the server process not running, and a connection refused by a closed port, a name that does not resolve or a certificate that has expired — reach the caller as the same observation: nothing arrives inside the deadline. One is the refused credential, which arrives as a rejection of the token rather than of the request. Three more reach the caller as a tool definition that does not match the one the client holds, which is drift in the terms of the table above: client and server built against incompatible schemas, a server deployed while the client's configuration was never updated, and a server rolled back while the client still calls a tool that no longer exists. That accounts for six. The seventh is the caller mapped to the wrong tenant, which returns a well-formed successful response containing somebody else's rows — the same signature as the partial result, and the reason the fourth class has to exist at all.

The mapping is many-to-one, and in one direction it is one-to-many: a rollback that also takes the process away presents as unavailability rather than as drift, and both readings are already covered. That is the property the partition is built on. The seven are named for what went wrong and the four are named for what the caller observes, and there are only four things the caller can observe — nothing, a rejection, a shape that does not fit, or a well-formed answer. A cause nobody has enumerated yet, a proxy that closes idle connections or a certificate rotated on the wrong host, still lands in one of the four, because it has nowhere else to land. What that licenses is a claim about coverage rather than about causes, and it is the difference between a gate and a checklist: a checklist of known causes grows every time somebody finds a new one, and a partition of what the caller can see does not.

In one respect the two lists do not correspond at all, and the gap is worth stating rather than smoothing over. The partial downstream result is not one of the seven added failure modes, since a function running in-process can just as easily return three rows when nine exist. What the boundary changes is the form the incompleteness arrives in. In-process, a failed page is an exception in code the same team wrote and can see in a traceback; across the boundary, it has been serialised into a successful response whose status line says that nothing went wrong.

The first three produce something to assert on, and that makes them ordinary tests. The harness is a shim between the client and the transport, and it does not need to be elaborate:

# tests/conftest.py
import pytest, httpx

class FaultyTransport(httpx.AsyncBaseTransport):
    """Wraps the real transport and injects one fault class per test."""
    def __init__(self, inner, mode=None):
        self.inner, self.mode = inner, mode

    async def handle_async_request(self, request):
        if self.mode == "unavailable":
            raise httpx.ConnectError("connection refused")
        if self.mode == "denied":
            return httpx.Response(403, json={"error": "token scope revoked"})
        resp = await self.inner.handle_async_request(request)
        if self.mode == "partial":
            body = resp.json()
            body["invoices"] = body["invoices"][:3]   # provider dropped page 2+
            return httpx.Response(200, json=body)      # note: still a 200
        return resp

@pytest.fixture
def mcp_client(request):
    mode = getattr(request, "param", None)
    return build_client(transport=FaultyTransport(real_transport(), mode))

@pytest.mark.parametrize("mcp_client", ["unavailable"], indirect=True)
async def test_unavailable_fails_fast_and_marks_capability_absent(mcp_client, agent):
    turn = await agent.run("What does ACME owe us?", timeout=10)
    assert turn.tool_state["billing.invoices"] == "unavailable"
    assert turn.elapsed_s < 4          # tool timeout is 3s, not the turn's 30s
    assert turn.answer_policy == "refuse_definitive_claims"

Three things in that last assertion block are worth naming. The elapsed-time assertion exists because the default timeout on most clients is longer than the turn budget, so an unreachable server does not present as a failed tool call; it presents as a slow agent, and then as a turn that dies with no partial output. The tool_state assertion exists because the orchestration has to know a capability is missing, which is information the model does not have and cannot infer. The answer_policy assertion is the one that matters most and is covered further below.

Schema drift is the fault that is cheapest to make safe and most often left to chance. A client that pins the tool schema it was built against and compares it at connect time turns a runtime error into a startup error, which is where you want it — the connection either establishes with the tools you expect or refuses and pages you, rather than degrading into a model that keeps trying to call customer_id on a server that now wants account_id. Version skew between a deployed server and a client that was never updated is exactly this, and the mitigation is a hash of the tool definitions checked at registration.

Why moving the check earlier helps is worth deriving, because the reason is not that a startup error is more polite than a runtime one. A connection establishes once and then carries many calls, so a mismatch detected at call time is detected once per call, by a model that cannot distinguish a permanently wrong argument name from a transient failure and will therefore reformulate and try again. Detected at registration, the same mismatch is a single event, before any call exists, in a code path whose available responses include refusing to start. The check has not become more sensitive by being moved. It has been placed at the one point in the lifecycle where a single failure can stop everything downstream of it, which is what makes a hash of the tool definitions a sufficient instrument rather than a crude one.

The credential refusal is answered by a person#

A 403 from the boundary is the one fault where every available technical response is worse than doing nothing. Retrying does not help, because the scope was revoked and retries against an auth path will get the caller rate-limited or the account locked. Falling back to a second credential is a privilege escalation performed automatically during an incident, which is how a boundary built to keep a credential away from the caller quietly stops being a boundary. Degrading to a cached answer means serving billing data whose authorisation has just been withdrawn, and the reason for the withdrawal may be that the caller is no longer permitted to see it.

The correct response is that a named human with authority to grant the scope has to act, and the design work is making sure the system knows who that is and gets to them quickly. That means the tool's error path carries the identity of the caller, the scope that was refused, and the queue that owns the grant, and that the agent surfaces something the user can act on rather than an apology. It also means the alert on this failure is routed to the team that administers the credential rather than to the team that owns the server, since the server is behaving correctly.

This matters beyond tidiness because 403 is the fault most likely to be "fixed" by a well-meaning engineer at two in the morning with a broader token, and the fix will outlive the incident. Making the correct answer explicit in the record, in the runbook and in the test — assert that no alternative credential is attempted — is what prevents the boundary from being unwound by the incident it was designed for.

The fourth fault has nothing to assert on#

Nothing in that exchange is an error from the agent's point of view. It asked a question, received a well-formed successful response, and reasoned correctly from what it was given. The defect is that the response says the same thing whether the customer has three invoices or has nine and the server could only see three, so the model is being asked to distinguish between two situations that are identical in its input.

One property separates this fault from the other three, and almost everything difficult about it follows from that property alone. Unavailability, a refused credential and a drifted schema each change the type of what comes back: an exception where a response was expected, a rejection where a result was expected, a shape that fails to match the definition the client holds. A type is something a test can assert on without knowing anything about the data, which is why those three are ordinary tests. A dropped page changes only the contents, and a shorter list is a perfectly valid list. There is no property of the response that distinguishes it from a correct one, so there is nothing for an assertion to hold on to.

Several of the complaints filed against this fault separately are that same fact showing up in different layers. No exception is caught, because nothing raised. No alert fires, because alerts are configured on the things that raise. The success rate stays clean, because a response was returned and it was well-formed. And the test needs a model call, because the only place the defect becomes visible is the sentence an agent writes from a response that looked complete. None of the four is repaired by better error handling, since each of them is the absence of a distinction rather than the mishandling of one.

The fix is a contract change rather than error handling. The response has to carry its own completeness, and the server has to be able to tell the difference between a query that returned nothing and a query that could not be completed:

def get_invoices(customer_id: str) -> dict:
    pages, complete, missing = [], True, []
    for page in provider.paginate(customer_id):
        if page.error:
            complete = False
            missing.append(f"invoices after {page.cursor_date}")
            break
        pages.extend(page.items)
    return {
        "invoices": pages,
        "complete": complete,          # false means "do not treat as the full set"
        "missing": missing,            # what is known to be absent
        "as_of": now().isoformat(),
    }

The support-ticket function in chapter 1 returned a truncated flag for the same reason, which was a local decision about a 20-row limit; here it is the difference between a correct answer and a collections notice. The flag alone is not enough, because a field in a JSON payload is a suggestion to a model rather than a constraint on it. The orchestration has to act on it: when complete is false, the turn is put into a policy where a definitive claim about the set is not permitted, and the user sees the coverage gap stated in the answer. Putting that rule in the system prompt is weaker than putting it in the code path that assembles the tool result, because the prompt is advice and the code path is not.

Testing this costs more than the other three. There is no exception to catch, so the assertion is on the envelope and then on the agent's final answer, which requires a model call per case. One billing team's happy-path suite ran in 40 seconds; twelve partial-result cases with an assertion that the answer contains an explicit coverage statement took about 90 seconds and were nondeterministic enough to need a retry policy in CI. That is the price of testing the only fault class where the failure is a plausible sentence rather than a stack trace.

Those two timings are not comparable as they stand, and setting them side by side makes the partial-result cases look cheaper than they are. Divide each by the number of cases it covers and they sit an order of magnitude apart — comfortably under a second for a happy-path assertion, several seconds for a partial-result one. None of that gap is the cases being harder to write. It is the model call: a test that asserts on an envelope runs at the speed of the process, and a test that asserts on a sentence runs at the speed of an inference and inherits its variance, which is where the retry policy comes from rather than from any flakiness in the server.

The belief worth dislodging before any of this reaches production is that a healthy success rate is evidence of a healthy boundary. It is tempting because the metric is neither lying nor miscomputed: a success rate counts responses returned without error, every response in a partial-result failure is returned without error, and the objective reports exactly what it was defined to report. It is false as a statement about the boundary because what is being measured is the transport and what is at risk is the answer. A server that replies well-formedly to every request holds a perfect transport record while being wrong about every customer it was asked about. A metric defined over the delivery of responses is blind by construction to a defect living in their contents, which is why the counter that catches this one has to be defined over the completeness field rather than over the status line.

Every fallback breaks something#

The fallback that gets chosen by default for unavailability is to omit the tool from the model's toolset and let the turn proceed. It is attractive because it degrades gracefully, keeps the assistant responsive, and produces no visible errors during an outage.

Omission converts an outage into a quality regression, which is the worse of the two because it is invisible. A model given no billing tool does not experience an absence; it experiences a question about invoices and produces the most plausible response available, which will be drawn from whatever the conversation already contains and from its priors about how invoices work. It cannot signal reduced coverage to anyone, because from inside the turn there is nothing to signal. Meanwhile the server's dashboards show no failed calls, since no calls were made, and the only trace is in the quality of answers during the window.

That generalises into a rule about where a fallback is allowed to be implemented. Any fallback that works by removing something from the model's input — a tool withheld, a field stripped, a list truncated — cannot be noticed by the model, because the model's only evidence about the world is the input it was handed and the removal is precisely what is not in it. Detection has to sit in the layer that performed the removal, which is the orchestration, since that is the only participant that saw both states. A fallback implemented in the orchestration and reported nowhere is the partial-result defect one level up: the system knows something that the answer does not.

The version worth defending combines the second and last rows: the tool stays registered so its failure is visible in the trace, the error is structured so the orchestration can classify it, and the constraint on what the turn may claim is applied in code rather than left to the model's discretion. That still leaves an honest gap. None of it makes the model's judgement deterministic, so what you can promise before launch is that missing data is detectable, that the system knows when a capability is degraded, and that a person is told — rather than that the answer will always be right.

Four fault classes, four demonstrated fallbacks, and a completeness contract are what turn "we chose this boundary" into "we can show this boundary survives". The distinction that makes the exercise worth the fortnight is the one between a fault that raises and a fault that returns: everything that raises is caught by ordinary testing discipline eventually, and the fault that returns successfully with less data than it should is caught by nothing you already have, because every layer between the provider and the user regards it as a success.

Being able to show that is also what makes it possible to approve a proposal you are not sure about. A reviewer who cannot predict whether a boundary will be worth it can still require that its bad days be demonstrated before launch, which converts an argument about the future into a checklist. The next chapter is about that conversation, where the boundary is not yet chosen and the reviewer who says no has to hand back something more useful than a no.