Agent transcripts as debugging evidence

A screenshot of an agent transcript is a record of one run of a system with at least five moving parts, and the server you own is one of them. Somebody typed a question into a chat application. A language model inside that application decided the question needed data it did not have, picked one of the tools it had been offered, and wrote the arguments for it. The application then sent that call to a program you wrote, running in another process or on another host, over the Model Context Protocol — MCP, an open standard for connecting an AI application to the tools, files and prompts it is allowed to use. It exists because the alternative is one bespoke integration per assistant per system: five assistants and eight internal services is forty pieces of glue, each with its own authentication, its own payload shape and its own maintainer. MCP replaces that grid with a single wire format, in which a server — the kind of program built in topic 6.10, on writing an MCP server — advertises what it can do, a client embedded in the assistant connects to it and calls it, and the two exchange JSON-RPC messages: small JSON objects that each carry a method name, an id, and either parameters or a result. One such message, as it crosses the transport, is a frame. You meet all of this wherever a coding assistant reads your repository, a chat client queries an internal database, or an agent files a ticket on somebody's behalf.

Debugging in this setting is the work of deciding which of those participants misbehaved, using evidence that has passed through all of them. A transcript is the least direct evidence available: the model wrote half of it, the client rendered the other half, and none of the calls in it are calls you specified. It is still worth reading closely, and the difference between reading it as a bug report and reading it as evidence is the difference between an afternoon spent in the right file and an afternoon spent in the wrong one.

Reading one transcript twice#

Here is such a transcript, with the client's rendering stripped back to the fields that matter. The → lines are calls the client sent to your server as tools/call, the protocol's one method for invoking a tool, and the ← lines are what came back:

[14:02:11] assistant: I'll look up this customer's recent orders.
[14:02:11] → tools/call search_orders
             {"customer": "Acme Corp", "since": "last month"}
[14:02:12] ← result {"orders": [], "count": 0}
[14:02:13] assistant: Nothing there. Let me widen the window.
[14:02:13] → tools/call search_orders
             {"customer": "acme", "since": "2026-01-01", "limit": 500}
[14:02:14] ← result {"orders": [ …20 items… ], "count": 20,
                     "truncated": true}
[14:02:16] assistant: I found some older orders but nothing from the last
             few weeks. The order search seems unreliable — you may want
             to check the admin console directly.

Read that as a bug report and it says the tool returned nothing, then returned the wrong twenty things. Read it as evidence and it says something narrower: two different calls were made, with three arguments that differ between them, and the server answered both in under a second. Whether either answer was wrong is a question the transcript cannot settle, because the transcript does not contain a single call anybody specified.

The word doing the work in that sentence is reproduction, and a transcript is easy to mistake for one. It has the inputs, the outputs and the timestamps. It is a genuine record of the failure, and it was produced by the system under investigation. None of those properties is what makes something a reproduction. A reproduction is a procedure that causes the failure again, on demand, from a state you can restore, and the only procedure this artefact comes with is "ask the assistant the same question" — which does not re-issue the same call. Re-running it draws fresh arguments from the model. A run whose inputs cannot be held still can neither confirm nor refute a claim about what those inputs produce, so the transcript is a record and not a procedure, which is exactly why it can suggest a fault and cannot establish one.

Three defects are visible here, and none of them is a broken handler#

Take the first call. The since argument is the string last month, which is not a date. Somebody's server accepted it, did not raise, and returned an empty result set. That is a real fault and it is worth fixing, but it lives in the tool's input schema rather than in the query handler — a schema that types since as a bare string will accept anything the model writes there, and a handler that parses failed dates into a null lower bound will happily return nothing forever. The defect is that the interface permitted a call nobody would have made deliberately.

The second call differs from the first in every argument. customer went from Acme Corp to acme, since moved back seven months, and limit appeared for the first time with the value 500. If the server matches customer names exactly and case-sensitively, then the two calls were searching two different customers, and comparing their results tells you nothing about consistency. If the server caps limit at 20 — which the truncated: true field in the response strongly implies — then twenty results out of five hundred requested is the server behaving as designed and saying so in the payload, while the model ignored the flag it was told about and reported the truncated set as though it were complete.

Both of those readings are conditionals, and they stay conditionals because of how the second call was built. Attributing a change in output to a change in input requires that exactly one input changed; then the difference in the result has one candidate cause and can be assigned to it. Here three inputs moved at once, and each of the three is on its own enough to turn an empty result into twenty rows. A different customer string would do it under exact matching. A lower bound seven months earlier would do it if the orders are old. A limit that was absent the first time would do it if the absent case defaults to something small. One observation, three sufficient causes, and no way in the transcript to separate them. Widening several things at once is a reasonable thing for anyone hunting data to do, and it is also what makes this particular pair of calls worthless as a controlled comparison.

The truncation reading needs one more step, because the flag is the least ambiguous field in the whole exchange. A count of twenty is compatible with two different worlds: twenty matching orders existed, or many more existed and twenty came back. Nothing in a bare count separates them. A handler that found twenty rows has no reason to announce that anything was cut, so truncated: true is the server distinguishing the second world from the first — a statement about rows that existed and were withheld. Three things follow from that one property. The flag says nothing about which rows were selected, so it is not evidence about the query. It says the set is incomplete, which directly contradicts the model's closing claim that there was nothing from the last few weeks. And it points at the server's ceiling rather than at the server's matching as the thing that produced the shape the reporter saw. A server can answer correctly and label its answer accurately and still leave a user with a false conclusion, when nothing downstream reads the label.

So the transcript contains three plausible faults: a schema too loose to reject a non-date, a matching rule the description never explained, and a truncation signal the model did not act on. Not one of them is "the handler returns wrong rows", which is the defect the ticket implies and the one you would have gone looking for. Fixing the handler would have taken an afternoon and changed nothing about the next run.

What the agent adds between you and the server#

The gap between a run and a defect is filled by machinery, and it is worth naming the pieces because each of them is a variable you did not set.

The model chooses the arguments. It writes them from your tool's name, description and input schema — the JSON Schema document your server publishes to say which arguments the tool takes and what type each one is — plus whatever is in the conversation, and it does this fresh on every attempt with a sampler that is usually not deterministic. Two calls in a row are two draws, so a failure that appears in one and not the other may be a difference in the draw rather than a difference in the server.

The model also chooses whether to call your tool at all, and which of your tools to call. A report that search_orders is broken sometimes turns out to be a report that the model kept reaching for list_customers first and giving up when that came back empty. The transcript in front of you is the model's plan as much as it is your server's behaviour.

The client sits between the two and does work of its own. It validates the model's arguments against your input schema before sending anything, which means a call can fail without ever reaching you — chapter 5 goes into that case in detail, because it produces a server log with nothing in it. It applies its own timeout. It may retry an errored call silently. It serialises your content blocks into whatever shape it feeds the model, and it decides how much of a large result survives that step.

And the transcript itself is a rendering. Most agent UIs pretty-print the tool result, collapse long arrays, and drop fields the display does not use. The JSON in a screenshot is not the JSON that crossed the wire; it is the client's summary of it. When someone tells you the response was missing a field, the first question is whether it was missing from the payload or from the viewer.

Timing is the last one and the least visible. Agents issue tool calls in parallel where the client permits it, reuse a single session across many calls, and interleave your server with other servers on the same connection pool. A run that fails one time in twenty under those conditions is the subject of chapter 7, on failures that appear only under the agent, and it is exactly where the difference between "the agent varied something" and "the server has a race" gets decided.

The two paths, side by side#

The cost row is the one that quietly decides how a week goes. A debugging loop through the agent means editing the handler, restarting the server, restarting the client so it re-runs initialize and picks up the new tool definitions, re-typing the prompt that provoked the failure, and waiting out however many model turns it takes to get back to the call — twenty seconds on a good day, closer to a minute when the model decides to explore first, and no guarantee it makes the same call twice. Ten iterations of that is fifteen or twenty minutes of wall clock, and roughly a third of those iterations produce a run that never reached the code you changed. The same ten iterations against the raw protocol are three minutes and every one of them exercises the line you edited. Over a two-day investigation that difference is not a convenience; it is the reason one team converges and another accumulates theories.

Two quantities are being multiplied in that paragraph and only one of them is wall clock. The other is the fraction of iterations that reach the code under test, and through the agent it is well below one, because the model may explore a different tool first or write arguments that take a different branch. What the loop costs is the elapsed time divided by the iterations that landed, not by the iterations that ran — and on the figures above, where about two in three attempts count, ten of them buy nearer six or seven observations. The direct loop has no second factor at all. Every call carries the arguments you typed, so the number of calls made and the number of observations obtained are the same number, and that identity is the whole of the advantage.

The two columns are not ranked. The right-hand one is where you go to establish that a defect exists and that a change removed it, because it is the only version of the exchange whose inputs you control. The left-hand one is where a whole class of defect is visible and nowhere else: a description that reads as though since accepts natural language, a truncated flag no model has ever acted on, a tool whose name is close enough to a neighbouring tool that the model picks the wrong one half the time. Those are faults in your server's interface, they are just as real as a null-pointer in the handler, and no hand-written protocol call will ever surface them because you already know what you meant.

There is a tidier way to see what the right-hand column actually is. The section above named five variables the agent contributes: which arguments get written, which tool gets picked, whether a failed call is quietly retried, how the result is rendered before anyone sees it, and what else is in flight on the same session at the same moment. A direct protocol call sets each one to a constant. You write the arguments, you name the tool, you make a single attempt, you read the frames rather than a summary of them, and you issue the call on its own. Nothing is left over. That is why a failure in the right-hand column can be attributed and a failure in the left-hand one cannot: attribution needs a fixed background, and the right-hand column is the same exchange with every one of the five held still.

The reverse error costs a second week#

Having reproduced all three claims directly, tightened the schema so since requires a date, documented the matching rule and made truncation explicit in the text the model reads, it is tempting to close the ticket on the strength of three green protocol calls. Those calls prove the server now behaves correctly on inputs you chose. They say nothing about whether the model, reading the new description, will now produce a well-formed since, or whether it will start refusing to call the tool at all because the tightened schema made the required arguments look expensive to satisfy. Schema and description changes are changes to the model's input, and the only place their effect is observable is a run. Reproducing directly is how you establish and remove a defect; going back through the agent afterwards is how you find out whether the fix landed where it was aimed.

That last claim rests on where a schema is read, and the path is short. The schema your server publishes is not consumed by your handler alone. The client fetches it in tools/list and serialises it, together with the tool's name and description, into the text the model sees before it decides anything at all. Editing the schema therefore edits two inputs in one commit: the validator's, and the model's. A direct call exercises the first, because you are the one supplying arguments to be validated. It cannot exercise the second, because the instrument works by removing the model, and the model is the only thing that reads what you just rewrote. So a tightened since is verifiable in a quarter of a second and a clearer description is not verifiable there at all, and no quantity of green direct calls converts into evidence about the second. Two kinds of change shipped together, and they need two instruments between them.

There is also a limit to how much of the ticket the transcript can be blamed for. Someone who reports "the tool doesn't work" after watching an assistant give up is describing a genuine failure of the product, and answering them with "your evidence is inadmissible" is both unhelpful and, in this case, wrong — three of the server's behaviours really were at fault. The distinction being drawn is about what you do next with the evidence, not about whether the report deserves attention. A reporter who is told their transcript was the reason three defects were found sends you the next one; a reporter who is told their transcript was not a reproduction stops sending them.

Turning three claims into three calls needs something that speaks the protocol without an agent in front of it, and that requirement is less trivial than it sounds. You cannot get there by curling an endpoint, because a server of the kind built in topic 6.10, on writing an MCP server, does not answer a bare request — it expects an initialize exchange, an initialized notification, and a session that persists across calls, and only then will tools/call mean anything. What you need is a client that performs the handshake, lets you type the arguments yourself, and shows you the frames.

The mistake worth avoiding is not trusting transcripts. It is treating a transcript as the reproduction rather than as the source of the hypothesis, and the tell is that the resulting investigation goes straight to the code that produces the output rather than to the interface that produced the call. Every variable in a transcript that you did not set — the arguments, the tool selected, the retries, the rendering, the timing — is a variable that has to be pinned before the run means anything, and pinning them is what a direct call does.

That is what an inspector is for, and it does more than save you typing. Because it removes both the model and the production client from the exchange, the result of an inspector run carries information in a direction most tools do not: a server that behaves perfectly under the inspector and fails the moment a real client connects has told you something specific about where the fault cannot be.