Stack traces in a tool result

A tool with no failure path still returns something when it fails. The handler wraps a repository method, whatever the library raises propagates, the framework serialises it into the tool result, and the model gets the frames. Nobody decides this; it is what happens in the absence of a decision, which is why it is so common in tools that were otherwise carefully built.

Here is what a support agent received on turn 4 of a ticket workflow, before anyone had written a failure path for update_ticket at all:

Traceback (most recent call last):
  File "/srv/agent-gateway/dispatch.py", line 218, in call_tool
    return await handler(**arguments)
  File "/srv/agent-gateway/tools/tickets.py", line 74, in update_ticket
    ticket = await repo.set_status(ticket_id, status)
  File "/srv/agent-gateway/repo/tickets.py", line 133, in set_status
    await conn.execute(_SET_STATUS, status, ticket_id)
  File "/opt/venv/lib/python3.12/site-packages/asyncpg/connection.py", line 350, in execute
    return await self._protocol.query(query, timeout)
asyncpg.exceptions.InvalidTextRepresentationError: invalid input value for enum ticket_status: "OPEN"
DETAIL:  unnamed portal parameter $1 = 'OPEN'
CONTEXT:  UPDATE tickets SET status = $1::ticket_status WHERE id = $2
          AND tenant_id = current_setting('app.tenant')::uuid

That block is 280 tokens, and it is a short one; a trace with a chained During handling of the above exception, another exception occurred section and a dozen frames of framework middleware runs 800 to 1,100. At a 4% failure rate across 120,000 tool calls a month, the shorter version alone is about 1.3 million tokens of frames per month that exist only because nobody wrote an except clause. The token arithmetic is the least interesting objection to it, and it is the one that gets raised first because it is the one that shows up on an invoice.

That monthly figure is two multiplications, and the term people leave out is the middle one. A 4% failure rate across 120,000 tool calls is a little under five thousand failures a month, and a little under five thousand failures at 280 tokens of frames apiece is about 1.3 million tokens. The 280 is the short trace; the chained version at 800 to 1,100 puts the same failure count into the millions. Neither figure includes the part that makes it a floor rather than a total: a trace is a tool result like any other, so it stays in the conversation and is re-read as input on every turn that follows, and frames delivered on turn four are paid again on turn five and turn six. The bill is therefore larger than the multiplication says, and it is still the objection that matters least.

Three things go wrong that a token count does not capture#

The first is that the trace publishes the inside of your service to a component whose job is to paraphrase things into user-facing text. Those nine lines disclose the deployment path, the module layout, the Python and asyncpg versions, the exact SQL of the update, and the fact that tenant isolation is enforced by a Postgres session variable named app.tenant. A model that has been asked to explain what went wrong will summarise what it was given, and summaries of tool results reach end users routinely — in a chat reply, in a ticket comment, in an email the agent drafts. The DETAIL line is the sharper edge, because exception details frequently carry the offending row rather than a placeholder: a duplicate-key violation on an email index prints the email address, and a check-constraint violation on a payments table prints the amount and the account. None of that is deliberate disclosure and all of it is disclosure.

The second is that there is no identifier in that response that anything can rely on. The nearest candidate is the final line, and it is produced by a library rather than by you. Upgrading asyncpg rewords it, migrating to psycopg changes the exception class as well as the text, adding a CTE — a common table expression, the WITH clause in front of a query — changes the CONTEXT block, and renaming the enum type changes the substring your alerting rule matches on. An alerting rule keyed on invalid input value for enum is a rule that will stop firing during a dependency bump, without failing, and the first sign will be a dashboard that goes quiet. The same problem applies to the model's own behaviour: whatever recovery strategy it settles on is a strategy conditioned on the wording of a string that a library maintainer owns.

The third is that the frames are worse than useless for recovery and are occasionally worse than absent. A trace describes how the failure travelled through your process, which is precisely the information the caller cannot act on, and it omits the permitted values, which is the information it needs. What it does contain is a rich set of internal names, and models will act on those: agents given traces attempt calls to functions they saw in a frame, report to the user that "the database rejected the query" when the actual fault was an argument they could have fixed, and in one measured run spent two turns trying to work out whether current_setting was a tool they had access to. A vague error leaves the model to guess; a trace gives it a set of confident wrong directions to guess in.

Why a model treats those names as usable is worth following, because "models will act on those" is an assertion resting on a mechanism. Everything in a model's context has the same standing: it is text that arrived, and nothing marks a region of it as internal. A tool result is specifically the place where the model has learned that facts about the world arrive. So a name that propagated into one — current_setting, OrderRepo.findByCustomer, pool.js — reads as a name that exists and is relevant, because it turned up where relevant names turn up. There is no separate channel saying which identifiers may be called and which merely passed through a stack unwinding, and a frame carrying a plausible function name contradicts nothing in the tool schema explicitly; the schema lists what exists, not what does not. Two turns spent working out whether current_setting was available are therefore not a lapse in the agent. They are what a caller that reasons from its context does with context nobody intended it to reason from.

The habit has a respectable origin, which is why it survives code review. Web frameworks have shipped debug pages that render the full trace, the local variables in each frame and the SQL log since Django and Rails made it standard practice in the mid-2000s, and the guard against disclosure was a single deployment flag that turned the whole thing off in production. That arrangement works because the two audiences are cleanly separated by environment: a developer on localhost sees everything, an end user on the internet sees a generic page. A tool result has no equivalent boundary, because the model is on the inside of the system in the sense that it receives internal data, and on the outside in the sense that it composes text for users. The flag that made debug output safe does not have a setting for a caller that is both.

The fix that is usually mistaken for the whole fix#

The natural response, once the trace is noticed in a context window, is to wrap the handler and return the message.

try:
    return await handler(**arguments)
except Exception as exc:
    return {"error": str(exc)}

That produces invalid input value for enum ticket_status: "OPEN", which is 15 tokens instead of 280, carries no file paths and no SQL, and takes an afternoon to apply across every tool. It is a real improvement on two of the three problems and it is where a great many teams stop, because the remaining problem is the one that does not show up in any artefact you look at afterwards.

That last clause explains the stopping better than any judgement about the team does, and it generalises. Two of the three problems leave evidence somebody already goes and looks at: the token cost is on an invoice, and the disclosure is visible by reading a single tool result. The third is a property of what the agent does next. It appears only in traces nobody opens by default, and only in aggregate, because any individual run that recovered looks entirely fine. So the fix ships, the two visible signals move, and the work closes. Nobody decided the third problem was unimportant; nothing in the review loop was ever pointed at it. A change that improves what is measured and leaves what is not measured untouched will look finished every time.

What it does not change is the model's next move. The message names the offending value and the internal name of a Postgres type, and says nothing about what is permitted, so the caller is back to sampling Open, ACTIVE, and then a read call to infer the enum from data, at a lower token cost per attempt. The string is also still unstable, since str(exc) is the library's text with the same lifetime as before, so there is still nothing to alert on, branch on, or count. And str(exc) leaks more often than it looks like it does: httpx request errors stringify to the full URL including query parameters, which is where API keys and signed tokens live, and driver errors stringify with the constraint name and often the conflicting value.

The last row is the one worth dwelling on, because the objection to removing traces is usually that debugging gets harder. Nothing in this removes a single frame from your logging; the trace is logged in full, with the tool name, the arguments, the tenant and a correlation identifier. What changes is the audience. Frames go to the engineer who can read them, and a code goes to the caller who can act on it, and the correlation identifier is what lets one be found from the other.

A code you own is also the only string in this arrangement that can stabilise, and stability is not a nicety here — it is what everything downstream is built on. An alerting rule, a dashboard series, a branch in the harness and a line in a runbook each key on some token, and if that token belongs to a library then every one of them carries a maintenance dependency on another project's release notes. Owning the string collects that dependency into one place: the driver's wording changes, the mapping changes with it in the same commit, and invalid_status goes on meaning what it meant. That is the row's real content. Not that a mapping is tidier, but that it is the only arrangement in which a dependency upgrade cannot quietly turn an alert off.

Building the replacement for status: "OPEN"#

The mapping belongs at the tool boundary, in one place, as a translation from exceptions to an error body carrying a stable code, the permitted values and a retry class. The important design property is that the permitted values come from the same definition the validator uses, so the message cannot drift away from the constraint it describes.

TICKET_STATUSES = ("open", "pending", "closed")

class ToolError(Exception):
    def __init__(self, code: str, *, cls: str, message: str, **extra: object) -> None:
        super().__init__(message)
        self.body = {"error": code, "class": cls, "message": message, **extra}

def validate_status(received: str) -> str:
    if received not in TICKET_STATUSES:
        raise ToolError(
            "invalid_status",
            cls="invalid_request",
            field="status",
            message=f"must be one of: {', '.join(TICKET_STATUSES)}",
            received=received,
            retryable=False,
        )
    return received

Raising at validation, before the statement is sent, is what makes the enum values available to the message at all — by the time Postgres has rejected the cast, the permitted set exists only inside the database's type catalogue. The driver exception then becomes a backstop for the cases validation missed, and it gets mapped too, because a backstop that returns frames is not a backstop.

async def call_tool(name: str, arguments: dict[str, object]) -> dict[str, object]:
    reference = new_reference()          # e.g. "err_7c1f9a"
    try:
        return await HANDLERS[name](**arguments)
    except ToolError as exc:
        log.info("tool_error", extra={"code": exc.body["error"], "tool": name,
                                      "reference": reference})
        return exc.body | {"reference": reference}
    except InvalidTextRepresentationError as exc:
        # Validation should have caught this; map it rather than leak the driver text.
        log.warning("unvalidated_enum", exc_info=exc,
                    extra={"tool": name, "reference": reference})
        return {"error": "invalid_argument", "class": "invalid_request",
                "message": "An argument did not match the permitted values for its field.",
                "retryable": False, "reference": reference}
    except Exception as exc:
        log.exception("unmapped_tool_error",
                      extra={"tool": name, "arguments": redact(arguments),
                             "reference": reference})
        return {"error": "internal_error", "class": "transient", "retryable": True,
                "retry_after": 2,
                "message": ("The ticket service failed to process this request. "
                            "Retry once; if it fails again, report the reference "
                            "to the user and stop."),
                "reference": reference}

The final branch is where the honesty of the whole scheme is decided. An unmapped exception is one you have not thought about, which means you do not know whether repeating the call could succeed, and any class you assign to it is a guess. Marking it transient with a small retry_after and an instruction to stop after one repeat is the least damaging guess for a read-only tool, because the common causes of unmapped exceptions genuinely are transient. On a tool with side effects the same guess is a bet that an idempotency key is doing its job, and it is only defensible because that key exists.

The condition in that last sentence is the part to carry, because the branch is often copied without it. Marking an unmapped exception transient on a read-only tool costs a repeated call and nothing else, so the guess is cheap even when it is wrong. On a tool that writes, the same marking instructs the agent to repeat an operation whose commit status is unknown — the corner of the grid where a repeat may do something — and the only thing standing between that instruction and a second committed effect is a derived key doing its work on every attempt. So the branch is not universally correct. It is correct in a service that has built the other mechanism, and a service that has not should mark its unmapped failures terminal and let the agent surface the reference instead of trying again.

The reference field is a small thing that changes what happens when recovery is impossible. It gives the model something specific and true to hand the user, so the terminal report is "the ticket service failed; reference err_7c1f9a" rather than a paraphrase of an internal error, and it gives support a key that joins straight to the frames in the log. Ten tokens buys the entire link between the two audiences.

What that counter measures deserves stating exactly, because it is easy to read as a reliability figure and it is not one. It is the share of failures that fell through every named case, so what it reads is how much of the failure surface has been thought about. Starting at 46% means a little under half of all failures arrived as a body saying only that something had gone wrong — well-formed, mapped in the sense of having a shape, and carrying no more information than the bare string it replaced. Working it to 4% did not make the service fail less often. It moved failures out of the unnamed bucket and into bodies naming a code, a class and a correction. It is also the one number in the design that a broader except cannot improve, since a wider catch that still returns internal_error leaves the counter exactly where it was.

Saying less is the other failure#

Mapping exceptions is not a licence to say less. There is a failure mode on the other side, where a team reads the security argument, replaces every error path with {"error": "internal error"} and considers the matter closed. That is a hammering loop with better hygiene: the agent has no code to branch on, no signal that repetition is futile, and no correction to make, so it retries until the harness stops it and then reports an outage. Withholding the offending value and the permitted set does not protect anything, since the model supplied the value itself and the permitted set is in the schema you published to it.

Nor is the mapping layer a substitute for the schema doing its work up front, and it is not the same artefact as the copy a user eventually sees. Three audiences read three different renderings of the same failure: the model gets the code, the class and the correction; the log gets the frames, the arguments and the tenant; the human gets a sentence with the reference in it. Trying to serve all three from one string is how the raw trace ended up in the model's context in the first place.

A tool result is a message to a caller who cannot read your source, so frames belong in your log and codes belong in the response. The mapping that makes that split is small — a validation error type, a dispatch-level translation, a reference id shared between the two — and what it buys is a body the model can branch on, an identifier your alerting can key on that survives a dependency upgrade, and no deployment paths or SQL in text that will be paraphrased to a user.

The claim that remains unproven is the one the whole chapter rests on. The trace version of update_ticket also completed its tasks — the model guessed, read the ticket, and got there — so a review that asks only whether the workflow succeeded will find no difference between the two implementations. Demonstrating that the rewrite helped means measuring how many attempts recovery takes, before and after, across enough runs that the tail is visible, which is a test rather than an inspection.