Diagnosing a failure with no deploy

An MCP integration has two participants that can change, and only one of them changes through your repository. A server — the process or hosted endpoint that advertises tools and performs them — publishes its contract at connection time, so what it will accept today is something you can ask it rather than something you have to infer. That makes evidence ordering during an incident a real choice rather than a matter of taste: the question "has the thing on the other end changed" is cheap to answer, and answering it first eliminates or confirms an entire class of cause before anybody reads a line of source.

The instinct runs the other way, and it is worth understanding rather than scolding, because it is produced by every other debugging experience an engineer has had. In ordinary systems the code you can edit is the code most likely to be wrong, the error message points into a stack you own, and reading source is the only investigative tool that is always available. An MCP dependency inverts two of those: interrogating the other side is available and fast, and the other side is the only participant that changed without a commit.

Diffing the far side first#

The snapshot job from chapter 4, where a scheduled connection records the contract and diffs it against a committed copy, already produces the artefact needed; during an incident it is run by hand, against the same endpoint, and diffed against the committed file from the last known-good day.

$ node record-contract.js > /tmp/crm.now.json
$ jq -S '.tools[] | select(.name=="crm_search")' /tmp/crm.now.json > /tmp/a.json
$ jq -S '.tools[] | select(.name=="crm_search")' contracts/crm.json > /tmp/b.json
$ diff /tmp/b.json /tmp/a.json
   "inputSchema": {
     "type": "object",
     "properties": {
       "query": { "type": "string" },
-      "include_archived": { "type": "boolean" },
       "limit": { "type": "integer", "minimum": 1 }
     },
-    "required": ["query"]
+    "required": ["query"],
+    "additionalProperties": false
   }
$ jq -r '.serverInfo' /tmp/crm.now.json
{ "name": "crm-mcp", "version": "4.2.0" }
$ jq -r '.serverInfo' contracts/crm.json
{ "name": "crm-mcp", "version": "4.1.6" }

Two facts come out of that, and both of them are about direction. The server removed include_archived and simultaneously started rejecting unknown keys, which means the client's behaviour did not change and the acceptable input did. And the build answering now reports 4.2.0 where the snapshot recorded 4.1.6, which names the change as an upgrade that happened on the server side and gives the vendor's changelog a specific range to be read across. Neither fact is available from inside a client's argument builder, and both of them determine what the correct fix is: if the argument was load-bearing, the fix is a conversation with the vendor about the removed capability rather than a quiet deletion of the parameter.

The error text aims the reader in the wrong direction, which is the other half of why the instinct wins. -32602 Invalid arguments for tool crm_search: unexpected key "include_archived" — -32602 being the JSON-RPC code for parameters a server will not accept — names a key the client sends and says nothing about the schema having moved, so it reads as an accusation against the argument builder. Forty lines of conditionals that assemble an argument object always contain something that looks wrong, and changing it may well make the agent work again while teaching the team nothing about what happened and leaving a client that no longer sends a parameter somebody added for a reason.

"Conclusive in one direction" is the property that fixes the ordering, and it is worth unpacking rather than asserting. A non-empty diff establishes that the acceptable input changed, because both documents being compared were produced by the server describing itself and the client had no hand in either; nothing on your side could have made them differ, so the cause is on theirs. An empty diff establishes only that the schemas match, which leaves every hypothesis about arguments, about content and about your own configuration exactly where it was. Evidence shaped like that belongs early whenever it is cheap: in the good case it removes a whole class of cause outright, and in the bad case it costs ninety seconds and removes nothing, so the gain is large and the loss is bounded by the price of running it. Reading the client's argument builder has the opposite shape — it is slow, it always turns up something that looks wrong, and finding something that looks wrong is not evidence that it is the cause.

Recording which build answered#

The version string is the second half of the first move and gets skipped more often than the diff, because it feels like paperwork until an incident review asks when the change landed. Where you read it depends on how the server is deployed, and each source has a different reliability.

serverInfo from the handshake is the server's self-report, which chapter 1 already showed can disagree with the version a registry listing and its package both declare. It is still the most useful single field, because it is what the running process believes about itself, and a change in it is nearly conclusive. For a container, the ground truth is the image digest the runtime actually resolved, which survives any lie the process tells about itself:

$ docker inspect --format '{{index .RepoDigests 0}}' ghcr.io/vendor/crm-mcp:4.2
ghcr.io/vendor/crm-mcp@sha256:9c1a...e77b     # yesterday: sha256:41ff...02d9

For a hosted endpoint there is often nothing but serverInfo and the response headers, so it is worth capturing the whole header block on one call during an incident; vendors commonly emit a build or release identifier there even when the protocol response carries none. Whatever the source, write it into the incident notes at the time. The value of a version number is almost entirely in being able to pair it with a timestamp later: a first failing run, a vendor status entry, and a version change line up into a single account that survives the review meeting. Without the version, the same facts reduce to a coincidence in time, and coincidences invite the counter-argument that something in the consuming system must also have changed. Capturing it costs one field in the session log and turns the post-incident question from a debate into a lookup.

Five servers, ordered by how easily they move#

An agent that has run cleanly for months and now fails about one run in five depends on five servers: one launched with @latest, two pinned by exact version with a lockfile, one container referenced by the tag :2.1, and one vendor-hosted endpoint. The repository has no commits in nine days. The obvious suspect is the @latest server, and the reasoning behind that instinct is sound while the conclusion is incomplete.

The table carries four rows for five servers, and the collapse is deliberate rather than a slip: the two lockfile-pinned servers share every column, so they are one case appearing twice. A disagreement between a count in the prose and a count of rows is always worth a look, because the same shape conceals a real error about as often as it saves a line. Two entries merged because they behave identically is a summary. Two entries merged because nobody noticed they differ is a hypothesis that has been deleted before it was tested, and the only way to tell the two apart is to read the columns rather than the count.

Two of those five look pinned in the configuration file and are not. The container tag reads as a version to everyone who reviews it, and it is a mutable pointer that resolves at pull time, so :2.1 today and :2.1 last month can be different images with the same label. The hosted endpoint has no pin available at all, as chapter 4 established, and it is the one that changes most often because the vendor's release cadence is the only thing gating it. The @latest server, counter-intuitively, drops to third for this specific incident: it can only change at process start, so if the pods have not restarted in eleven days it has been serving the same resolved version for eleven days, and the restart log settles that in one query. The two lockfile-pinned servers are last, and the reason to keep them on the list at all is that a pin does not cover the vendor API the server calls out to.

Nothing in that ordering is about which vendor is more trustworthy. It is entirely about which references are mutable and what event causes them to resolve, which is a property of the configuration you wrote rather than of the software you consume.

What 20% is telling you before you gather anything#

An intermittent failure rate that sits near a round fraction is evidence in its own right, and the most useful thing it rules out is a clean cutover. If a single process had upgraded and the contract had changed, every run would fail, because every run would meet the same schema. Failing about one in five means requests are meeting two different behaviours, so something is split.

There are four splits that produce this shape and they are distinguishable. A fleet split is the common one: five replicas behind a load balancer where one is running a different build, whether that is your own rolling restart part-way through or the vendor's canary, a new build put in front of a fraction of traffic before full rollout, sitting at 20%. A data split gives the same arithmetic for a different reason — every replica is identical and 20% of the requests carry the argument that the new schema rejects, which fits neatly when the failing parameter is optional and only some queries set it. A tenancy or credential split shows up when the vendor enables a flag per account and your agents run under more than one identity. A time split, such as a token or session expiring on an hourly boundary, concentrates failures in a window rather than spreading them, so a histogram over the hour separates it from the others in one glance.

The single most valuable piece of evidence is therefore the raw exchange from a set of runs, both failing and passing, with the server's reported version attached to each session:

$ jq -r 'select(.event=="mcp_session") | [.run_id, .server, .server_version, .outcome] | @tsv' \
    agent-logs-2026-02-17.jsonl | sort | uniq -c | sort -rn | head
 213  crm  4.1.6  ok
  54  crm  4.2.0  error
  49  search  1.8.2  ok

That split — 54 failures out of 267 sessions, 20.2%, and every failure on 4.2.0 while every success is on 4.1.6 — eliminates the data split, the tenancy split and the time split in one table, and names a fleet running two builds. Had the version been identical across both outcomes, the same table would have eliminated the fleet split instead and pointed at the arguments, at which point grouping the failures by which optional parameters were present is the next query rather than the first one. The discipline is that each piece of evidence is chosen for what it eliminates, and a version-tagged session log eliminates three of four candidates at once, which nothing in the client's source code can do.

The denominator behind that failure rate is worth saying out loud, because it is the commonest way a figure like it gets quoted wrongly. Two hundred and thirteen successes plus fifty-four failures is two hundred and sixty-seven crm sessions, and the fifty-four failures are counted against those and only those. The forty-nine search sessions are not in the denominator and should not be: they are a different server, they were never going to meet this schema, and folding them in would divide the same fifty-four failures by a larger number and make the problem read as smaller than it is. The command hides a second trap in plain sight. It ends in head, so what is printed is the top of a longer listing, and any total obtained by adding up the rows on screen is a total over whatever happened to be shown. Both crm rows are present here, which is why the sum is safe — and confirming that they are both present is the step, rather than assuming it.

Each of the three splits that were eliminated predicts something the same table would have shown, which is why one query settles four hypotheses instead of one. A data split has every replica identical and only the requests carrying the offending argument failing, so it predicts failures and successes appearing against the same server version; the table separates them cleanly by version, so that hypothesis is out. A tenancy or credential split predicts the separation falling along a client id or an account and the version column staying constant, which is not what is on the page. A time split predicts failures clustering inside a window rather than spreading across the day, which is a property of the timestamps rather than of the versions, and is why a histogram over the hour is its one-glance test. A fleet split is the only one of the four that predicts exactly what is there: two versions answering, one of them failing, both present throughout. The log line earns its place because it carries the single field all four hypotheses disagree about.

Why a fleet split should land near one run in five, rather than anywhere else, is a proportion rather than a coincidence. If requests are spread evenly across replicas and every session makes the offending call, the share of sessions that fail is the share of replicas running the new build — the same ratio, arrived at twice. It composes as a share and not as a sum: a second faulty replica does not add a second failure rate, it moves the one rate up. Two assumptions hold that equality together and both are worth checking rather than believing. Even spreading fails under sticky routing, where each user is pinned to one replica, and the symptom becomes a subset of users failing every time instead of all users failing sometimes. A canary weighted by traffic share rather than by replica count breaks the equality outright, which is why the vendor's rollout percentage is a more useful thing to ask them for than a replica count.

Where this reasoning stops being right#

Reaching for the server first is a rule about evidence ordering and not a claim about fault. Plenty of failures that arrive with no commits are genuinely yours: a feature flag that rolled to 20% of users on your side, a cache expiring, a config map updated out-of-band, a dependency of your own that resolved differently in a rebuilt image. The diff is worth running first because it is fast, cheap and conclusive in one direction; when it comes back empty, the server has been eliminated for schema-level change and the investigation moves to your side with one hypothesis already closed.

Empty is also not a full exoneration, since chapter 4's shared blind spot is exactly this: a server can keep every schema stable and change what it returns, and no contract diff will show it. If the diff is empty and the failure is about content rather than about arguments — a field that is now always null, a result set that is suddenly empty for known-good inputs — the next probe is a fixed call with known inputs compared against a recorded response, which is an acceptance test rather than a contract check.

The other limit is that all of this depends on having a snapshot from when it worked. Without one, the diff has nothing on the left-hand side, and the investigation degrades into reading a changelog that may not mention the change and asking a vendor who may take a day to answer. Teams that adopt one habit from this topic should adopt the committed snapshot, because it is the artefact that makes every later incident cheap.

The order, then, is fixed and short. Pull the current contract and diff it against the last known-good snapshot; read the version of the build that answered and write it down beside the timestamp; rank the servers by which references are mutable rather than by which vendor feels risky; and read the failure rate before gathering anything, because a rate near a round fraction already says two behaviours are in play.

Knowing that the server tightened its schema still leaves the question of why the client sent a call the server was always going to reject. The server's schema was available to the client at connection time, in the same tools/list response the diff was run against, which the next chapter turns into a guard that fails before the request leaves the process.