Direct calls and agent runs

Calling the server from a script means starting the process, completing the initialize handshake that opens every MCP session, listing the tools and invoking them with arguments you wrote yourself — usually over stdio, the transport where the client runs the server as a subprocess and exchanges JSON-RPC messages on its standard input and output. An agent run means giving a model a task in natural language, letting it choose tools and arguments, and scoring whether the task got done.

The two are routinely compared as though the second contained the first, on the grounds that it exercises the same server code plus the planner's behaviour on top, the planner being the model in its role of choosing which tool to call next and with what arguments. It does not contain the first, and the practical consequences of believing otherwise are a green suite that hides nine real defects and a green agent run that hides a schema bug in every tool the planner never selected.

What a direct call is an instrument for#

Direct calls cover four families of defect, and the value of the instrument is that when one fires you know it was the server, because there was no other component in the loop capable of being wrong.

Protocol defects are the ones that make a server unusable rather than incorrect. A server that answers tools/call before initialize has completed works fine against a client that always initialises first and hangs against one that pipelines. A server that sends a response to a notification — a message with no id — violates JSON-RPC, and some clients treat the unexpected response as a protocol error and drop the connection. These never reproduce on the developer's machine because the developer's client is the one the server was written against.

Schema defects are the ones where the declaration and the implementation disagree, and the sharpest of them has a blast radius out of proportion to its size.

# One tool's inputSchema names a required field that does not exist in properties.
{
  "name": "get_page",
  "inputSchema": {
    "type": "object",
    "properties": {"url": {"type": "string"}},
    "required": ["url", "format"],
  },
}

A strict client validating the tools/list payload rejects the whole document, so all fourteen tools disappear and the agent reports that the server exposes nothing. A lenient client ignores it and the tool works. Whether this is a catastrophic bug or a no-op is a property of somebody else's client, which is exactly the kind of thing to find in 3.2 seconds rather than in a customer's environment.

Serialisation defects are where the values are right and their representation is not. A Python handler that returns a datetime through a permissive encoder can emit "datetime.datetime(2026, 8, 1, 14, 30)", which a model will cheerfully copy into the next call's since field. A float that becomes NaN produces JSON no strict parser accepts. Issue ids above 2^53, the largest integer a JavaScript number represents exactly, arrive in a JavaScript client with the last digits altered, so 9007199254740993 becomes 9007199254740992 and the lookup that follows returns the wrong record with no error anywhere.

Lifecycle defects are the ones that only appear on the second or hundredth interaction: a transport that drops and reconnects without resuming the session, so the client loses the tool list mid-run; two concurrent calls sharing a cursor stored on the server instead of in the response; a cancelled call that leaves an advisory lock held until the process restarts. A script can force all of these deliberately, and an agent will encounter them by accident on a Tuesday.

Two of those four families hide a step that is worth taking slowly, because in both cases the consequence is much larger than the cause and the reason is not obvious. Start with the schema one: why does a single bad required entry remove fourteen tools rather than one? Because tools/list answers with one JSON document containing all of them. A client that validates validates the document. There is no partial-success path in that operation, and nothing in the response format lets a client say I accepted twelve of these — so the only two behaviours available are accept it whole or reject it whole, and a strict validator does the second. The size of the failure is set by the granularity of the message, not by the size of the mistake.

The integer case has a step under it too. A JavaScript number is a double-precision float, which carries a fifty-three-bit significand, so every integer up to two to the fifty-three is representable exactly and above that the representable values start to thin out. Just past that boundary they sit two apart: the even ones survive and the odd ones have no representation at all, so an odd value is rounded to the nearest available neighbour, which is the even number below it. That is precisely what the example shows — a value ending in three arriving as one ending in two — and it explains the two properties that make the bug nasty. Only the low digits change, so the id still looks like an id, and the substitution lands on another value that is a perfectly legal id, so the lookup succeeds and returns the wrong record with nothing anywhere reporting an error.

The error paths from the previous chapter belong here too, since an agent run reaches authorization_expired only when a token happens to expire mid-run. A direct call can fire it on demand.

async def test_expired_token_response(server):
    async with stdio_client(server, env={"DATASTORE_TOKEN": EXPIRED_FIXTURE}) as s:
        result = await s.call_tool("list_datasets", {"workspace": "wk_1"})
        assert result.isError
        body = json.loads(result.content[0].text)
        assert body["code"] == "authorization_expired"
        assert body["retryable"] == "never"
        assert body["user_action_required"] is True
        assert "Settings > Integrations" in body["recover"]
        assert "DATASTORE_TOKEN" not in result.content[0].text  # no secret echoed

That test asserts on the contract the agent depends on, deterministically, in milliseconds, on a branch that might otherwise render for the first time in production.

Selection, description and output size#

Every call in such a file specifies a tool name, and a person wrote that name. The instrument therefore has nothing to say about the step that precedes every real call, which is the choice of which tool to invoke — the script never chose, because you chose when you wrote the call. Whether search_docs is the obvious tool for a documentation question, given its name and description and the thirteen others sitting next to it, is unobservable from a test that hardcodes search_docs.

Two related properties travel with it. A description problem is invisible for the same reason: a test asserting that a one-word query returns three results confirms the tool works on the input its author already knew to send, and says nothing about whether a caller who has not read the source would send that input. And a result can be correct and unusable — no assertion about JSON structure catches "this consumes 23,000 tokens of a context window", because the property is not about the response, it is about what the response does to the caller downstream.

The script attaches at the client and drives everything to the right of it. The two edges to the left — a task becoming a selection, and a result becoming the model's next move — are the ones it cannot reach, and they are where selection legibility and output legibility live.

Two instruments over different populations#

An agent run is not a superset of the script. It is a noisier instrument over a smaller sample.

The coverage row is the one that undoes the superset claim. On a twenty-task set the agent called six tools; eight were never invoked, so a schema defect on any of them — including the required-field bug that hides the entire tool list from strict clients — passes an agent run silently. The attribution row is the one that makes agent failures expensive to act on: a failed task produces a transcript, and a transcript has to be read before anyone can say which component was wrong, whereas each direct-test failure names a file and a line.

The superset belief is worth dismantling rather than contradicting, because the reasoning behind it is nearly right. An agent run does execute the server's code, and it does add a planner on top, so it looks like the same coverage plus something. The word doing the damage is same. The code an agent run executes is the code the planner chose to reach, and the planner reached six of fourteen tools — so the run covers a subset of the surface and adds a component, which is not a superset of anything. Worse, the eight it skipped are not a random eight. A tool goes unselected either because no task in the set needed it, or because its name and description did not make it the obvious answer to a task that did, and the second reason is a defect in exactly the property the agent run is meant to be measuring. The unexercised set is therefore enriched in the tools most likely to be wrong, which is the opposite of the property you want from a sample.

The runtime and cost columns are worth converting before treating them as a reason to prefer one. Eleven minutes is six hundred and sixty seconds, which against the script's three point two is a factor of a bit over two hundred, and the money column moves from nothing to a couple of dollars a run. That ratio is what makes the two instruments schedule differently rather than compete: something two hundred times cheaper can run on every commit, and something that costs dollars and minutes runs when a change could plausibly have moved what it measures.

Determinism deserves its own note, because it changes what a green run means. A direct suite passing means the assertions hold. An agent run scoring 20/20 means that sample, at that temperature, with that model version, scored 20/20; the same set scored 18/20 an hour earlier. Neither number is wrong, and treating the higher one as the state of the world is how a regression gets shipped.

The three reruns give the number that makes this operational, and it is the number to write on the dashboard. Identical server, identical task set, identical prompts: eighteen, twenty, nineteen. The spread is two tasks, and two tasks out of twenty is ten points. So ten points is the resolution of this instrument at this sample size — anything smaller than that is not a signal, whichever direction it moves. A change that takes a suite from nineteen to twenty has moved by one task, which is inside that spread, and the honest report is a range with the number of runs beside it. The consequence for the harder direction is the uncomfortable one: a genuine regression costing one task in twenty is not detectable here at all, and the way to detect it is more tasks or more runs rather than more attention paid to the score.

The classification in that account closes, and the closing is what makes it usable. Twenty tasks with eleven completed leaves nine failures; five selection, three description and one payload size is nine, with no fourth group and no leftover. That absence licenses the sentence that follows it. Had the groups summed to fewer than nine, "none of these is a defect the test file was capable of finding" would be a statement about the failures somebody managed to classify, and the unclassified remainder would be exactly where a defect the script could have caught would hide. Because the three groups exhaust the nine, the claim covers every failure in the run, and the afternoon of transcript reading bought a complete partition rather than a sample of one.

The payload failure can also be checked against a figure given earlier in the chapter, which is worth doing because the two came from different places. A hundred results at roughly ninety kilobytes is a little under a kilobyte of JSON per result. Converting bytes to context is approximate — as a rough rule, English-heavy JSON runs a few characters to the token, and the ratio varies with the content — but it puts ninety kilobytes in the low tens of thousands of tokens, which is the same order as the twenty-three thousand named earlier as the thing no structural assertion catches. The point of the check is not the precision. It is that the response was well-formed, correctly ordered, within its declared limit, and asserted green, and its defect was a quantity that lives on the other side of the client from every assertion in the file.

The night before a demo#

Given time for one, the useful rule is to run the instrument that covers what changed, and to weight it by which failure you cannot recover from while people are watching.

For a server that is new or has not been exercised since a dependency bump, the direct suite is the one to run. Its failure modes are total rather than partial: a lifecycle or schema defect does not cost you one task out of twenty, it costs you every task, and it fails in a way that looks to the audience like the whole thing is broken. It also costs almost nothing, which means the real question is what to do with the remaining time rather than which to sacrifice. What you accept by stopping there is complete ignorance of selection: you will not know whether the model reaches for the tool you are about to demonstrate, and if the demo has a tool set where two things overlap, you will find out in front of the room.

The rule inverts when the change under test is a consolidation, a rename or a description rewrite. Those changes cannot break the protocol and can only be validated by something that chooses, so the agent run over a task set is the instrument, and what you accept there is that an untouched tool the planner never selects could have been broken by the same commit.

The narrow mitigation, when the schedule is genuinely one evening, is to run the direct suite and then run the agent exactly once on the three tasks the demo will use. That is not an evaluation and should not be recorded as one, since three tasks at one sample each has no statistical content at all. It answers a smaller and more urgent question, which is whether the specific path you are about to walk down in public works tonight.

The reason to keep both instruments rather than choosing is that each one's blind spot is the other's subject. Protocol, schema, serialisation and lifecycle defects are total, deterministic and attributable, and they are invisible to a run that never invokes the tool carrying them. Selection, description legibility and output size are none of those things, and they are invisible to a script that was told which tool to call. A server that has only ever been checked by one of the two has a known category of defect nobody has looked for.

Both instruments also share a blind spot that no amount of running either one closes. You wrote the assertions and you wrote the task set, so both are shaped by knowing what since expects, that ids come from list_projects, and that the search index wants keywords. A person who has never seen the server has none of that, and neither does a model.