Tracing every exit in an agent you did not write

Approving an agent for production is a different job when you did not write it and cannot run it. Its tools point at systems you have no credentials for, its loop belongs to a framework somebody else chose, and what you actually hold is source code and a handful of trajectories — recorded runs, each one a file containing the full message list a turn produced and the timings for it. Out of those two things you can build an exit inventory: one row for every way control can leave every loop in the system, naming who owns that exit, whether anything bounds it, and what evidence exists that it has ever fired. The inventory is the deliverable, and the sign-off is a judgement about it.

The answer that arrives first in a review thread is a configuration value. The framework's executor takes a max_iterations argument, somebody set it, and therefore the system is bounded. That claim deserves to be taken seriously, because it is true and it is checkable, and it is insufficient in three separate ways: the number's meaning is not fixed across libraries, its scope is one loop out of several, and nothing in the recorded runs shows it has ever executed. Those are three different kinds of gap, and each has a different repair.

Reading a default is not the same as reading the loop#

The first problem is that the default's semantics are not obvious from its name, and they differ between libraries. Some count model requests, some count completed tool executions, some count "steps" where a step is a model call plus all the tool calls in it — which means a turn that issues four parallel calls per iteration burns one unit or four depending on which library you are holding. Some raise on the limit, some return a sentinel string — a fixed placeholder value standing in for a real result — like Agent stopped due to iteration limit, as though it were the model's answer, and one of those behaviours turns a bound into a silent corruption of your output. Ten minutes reading the executor's source settles all of this and no amount of reading the config does.

The shape of that mistake is worth naming, because it is not carelessness and it recurs in every review of this kind. A configured number is a constant, and it is reassuring precisely because it is a constant: the value is the same whatever ticket arrives. What varies underneath it are the three properties that decide whether the value means anything — the unit the number counts, the behaviour when the count is reached, and the set of loops the count applies to. A quantity held fixed while the dangerous properties move is the general form of a defence that does not defend, and the repair is always the same: check that the constant is attached to the thing you are actually worried about.

In this case the framework counted model requests, and on the fifteenth it returned an object whose output field was the sentinel string. Downstream, svc-triage wrote that field into the ticket as the triage note. So the bound existed, fired correctly, and produced a ticket note reading "Agent stopped due to iteration limit" that a support agent would read as a system error and escalate. That is a finding, and it took reading two files.

That damage is worth following one step further than the code goes, because it happens downstream of everything the framework promises. The executor's contract is that it returns an object with an output field. The calling code's contract is that it writes output into the ticket. Neither contract says anything about whether output came from the model, so a sentinel — a value that satisfies the type and violates the meaning — passes every check between the two. The consequence is that the only place the failure becomes visible is the ticket a support agent opens, hours later, with no way to tell it from a crash. A bound that raises is loud, and a bound that returns a string shaped like an answer is silent, and silence is what makes this the more expensive of the two behaviours.

The second problem is bigger: a max_iterations on the executor bounds the executor. It says nothing about code paths that reach a model without going through it. The way to find those is mechanical:

# every place this repo talks to a model
rg -n 'messages\.create|\.invoke\(|\.run\(|chat\.completions' --type py

# every loop and recursion in the files those appear in
rg -n '^\s*(while|for)\b|def .*\(.*\).*:.*# recurses' --type py svc_triage/

Three call sites came back. The executor, which was the one everybody knew about. A summariser that compressed the message list when it grew past a token threshold, called from inside the executor's loop — a single request, no loop of its own, but it meant the real worst case was 15 executor requests plus up to 15 summarisation requests, and the cost estimate in the design doc had counted the first number only. And a third, inside a tool.

That estimate is a residual worth naming rather than an error worth correcting. It was not wrong about the executor; it was silent about the rest of the system, having counted one call site of the three the search found. A figure that is silent about most of its own subject reads as complete to everyone who has not run the search, which is what makes this class of residual expensive: nobody disputes it, because nobody can see what it left out.

Composition is what turns three call sites into a worst case, and the two additions behave differently. The summariser sits inside the executor's loop and issues at most one request per iteration, so its count is driven entirely by the loop above it and the two counts add — fifteen and up to fifteen more. Adding is the correct operation only because the summariser has no loop of its own. The third call site does have one, and it does not add.

The loop inside the tool#

The tool was called apply_refund and it was 90 lines. Most of it was a retry: the payments API occasionally rejected a refund with a validation error naming the offending field, and rather than surface that to the outer loop, the author had the tool call the model again to repair its own arguments and resubmit.

def apply_refund(charge_id: str, amount_cents: int, reason: str) -> str:
    args = {"charge_id": charge_id, "amount_cents": amount_cents, "reason": reason}
    while True:                                  # no counter
        result = payments.refund(**args)
        if result.ok:
            return json.dumps({"status": "refunded", "refund_id": result.id})
        if result.error_type != "validation_error":
            return json.dumps({"status": "failed", "error": result.error_type})
        args = repair_arguments(args, result.message)   # a model call

Two things about this are worth separating. The unbounded while True is the obvious one, and it is reachable whenever the payments API returns a validation error the repair step cannot fix — a currency mismatch, say, where no rewriting of the arguments will help. The 40 staging trajectories contained no such case, because staging pointed at a sandbox that accepted everything.

The subtler one is that this loop is nested inside the outer loop. The outer bound of 15 multiplies rather than caps: fifteen iterations, each of which may call a tool containing an unbounded inner loop, is not a bounded system, and the arithmetic that makes it look bounded — "at most 15 model calls" — is wrong by an unbounded factor. Any exit inventory has to be built per loop and then composed, and a nested loop with no bound of its own poisons every bound above it.

The algebra behind that sentence settles every nesting rather than this one, so it is worth writing out. Give the outer loop a bound of n iterations and the inner loop a bound of m calls per invocation. The composed system performs at most n times m model calls, and that product is why nested loops with bounds are unremarkable — both factors are finite, so the product is finite. Now remove the inner bound. There is no m to multiply by, so there is no product to be finite, and the composed system is unbounded for every value of n, including one. A single iteration of the outer loop is enough to reach a loop that need never return. That is what "multiplies rather than caps" means, and it is also the reason an inventory composed by taking the largest bound would be wrong: the composition is a product, and one unbounded factor sends the product with it.

There is a third property here that matters more than either: payments.refund is not idempotent in this code. It takes no idempotency key — an identifier the caller sends so the provider can recognise a repeat of a request it already carried out and return the original result instead of doing the work twice — so a retry that the tool believes failed but which actually committed server-side issues a second refund. That turns an unbounded loop into an unbounded loop over an irreversible side effect, which is a different category of finding from "this might be slow".

Keeping that category separate is worth the effort, because the two failures have different worst cases and different remedies. An unbounded loop over a read spends time and money, and an infrastructure timeout eventually stops it; the damage is a slow request and a bill, and both are recoverable by waiting. An unbounded loop over a non-idempotent write commits one irreversible effect per pass, and a timeout that kills the worker halfway through stops the loop without undoing anything it already did. The bound is missing in both cases. Only in the second does the missing bound leave a permanent mark on a customer's account, which is why it belongs in a different section of the memo from anything about latency.

The inventory#

With the three call sites found, the audit is a table you can put in the sign-off memo, one row per exit rather than one row per component:

The continue in row three is the second bug from chapter 2 arriving in someone else's repository: the handler logs the tool failure and moves on without appending a result, so the next request carries a tool_use id with no matching tool_result and is rejected by the provider. Both trajectories where a tool raised ended in a 400 twelve milliseconds later, and both had been recorded as "agent error, investigating" in the staging notes. That is the value of reading trajectories rather than summary metrics — the failure was in the sample the whole time, labelled as something else.

Two counts in that table close on the sample, and closing is what makes a zero in it mean anything. Take the executor's three exit rows: thirty-eight trajectories left through the model-controlled path, none through the cap, and two through a raising tool. Those add to forty, which is every file in the set. Now take the iteration distribution, read from a different field of the same forty files — thirty-four finishing in one or two iterations, five in three to five, and one in nine — which also adds to forty. Two instruments sharing no step, built from different parts of the record, arriving at the same total.

Each closure licenses a claim that an open column could not support. Because the exit rows account for all forty runs, no fourth executor exit is hiding in the sample: whatever else that loop is capable of, it did not do it here, and the inventory's list of executor exits is complete with respect to the evidence. Because the same rows close, the zero recorded against max_iterations is a genuine absence rather than an unlisted category — the cap was never exercised, as opposed to exercised and not recorded. A column adding to thirty-six would license neither claim, since the four unaccounted runs could be a fourth exit or a gap in the recording, and nothing in the table would say which. The habit worth taking from this is to add the column up before quoting any zero in it.

The right-hand columns also show what the trajectories cannot tell you. Forty runs, thirty-eight of which left through the model-controlled path and two through a provider rejection, are forty pieces of evidence about paths the model or an accident chose, and zero pieces of evidence about any bound. A sample that never exercises an exit tells you nothing about that exit, and the correct response is not to ask for more trajectories — a hundred more staging runs against a sandbox that accepts everything will still be a hundred more happy paths. The correct response is to ask for a test that forces the exit, which is exactly the scripted-model harness from chapter 3 and takes an afternoon to build against code you did not write.

The general claim underneath that is worth having in plain form, because it decides what a reviewer should ask for next. Evidence about an exit comes only from runs that took it. A sample in which an exit never fired constrains how often it fires and says nothing at all about what happens when it does, and those are two separate questions with two separate repairs. Drawing more samples from the same source moves the first and cannot move the second, however many are drawn, which is why "send us more staging runs" is the wrong request and "write a test that trips the cap" is the right one.

Where trajectories are genuinely irreplaceable is the iteration distribution. Thirty-four of the forty finished in one or two iterations, five in three to five, and one in nine — a customer with 41 charges in the window, where the model paged through them. That one run is the most informative file in the set, because it says the tail is real and a cap of 15 leaves about six iterations of headroom over the worst observed case, which is a defensible number rather than a round one.

The headroom is the whole of that defence, and the arithmetic is one subtraction anyone in the review can repeat: the cap minus the worst observed iteration count. Nine against fifteen leaves six. Six is defensible in a way that fifteen on its own is not, because fifteen is a number somebody typed and six is a distance from something that actually happened. The distribution supplies the worst observed case, so a cap chosen without a distribution is a cap chosen without a measurement, and the honest way to write it in a memo is as a margin over an observed maximum with the sample size printed beside it — a maximum drawn from forty runs is a weaker claim than the same maximum drawn from a hundred times as many, and the margin should be read accordingly.

The adversarial pass, done on paper#

The last step needs no access to anything. For each tool, ask what it returns in the case that is not an error and not a useful answer, and whether that return gives the model a reason to call it again. This is the same question chapter 5 asked from the inside, and it is the fastest way to find the exit that is missing.

get_charges returns {"charges": []} for a customer with no charges in the window. Empty, valid, and a perfectly good reason to try a wider window — that is the archived-repository failure with a different name. check_refund_status returns {"status": "pending"} while the provider settles, which can take up to four hours; the model has every reason to check again, and there is nothing in the loop that says a pending status is a reason to stop rather than to poll. search_similar_tickets returns the top five by embedding distance and never returns fewer, so it always looks like progress. Three tools, two of them capable of removing the model-controlled exit on ordinary production data, and none of that requires running the code.

What those three have in common is worth stating, because it is the property to look for in a tool list you have never seen. Each returns a well-formed value in a case where nothing useful happened, and none of them can express "there is nothing here and there will not be". A tool that cannot say that leaves the model with a valid answer and an open question every time, which is the condition under which the model-controlled exit recedes indefinitely. The adversarial pass is really a search for tools with no terminal value in their range.

What justifies refusing to sign off#

Five things, in the order they should appear in the memo. An unbounded loop you can point at in the source. A bound whose behaviour on tripping is untested or wrong, which covers both the sentinel string written into the ticket and the more common raise. A tool with an irreversible side effect reachable inside a loop with no idempotency key and no per-tool call budget. A conversation-invalidating error path, such as the continue that drops a result. And the absence of any test or trajectory exercising a bound you are being asked to trust — the bound is not presumed broken, but "we believe it works" is not a thing to sign your name to.

What does not justify refusing is worth writing down too, because a reviewer who blocks on the wrong things stops being invited. Not using a framework is not a finding. Using one you dislike is not a finding. The absence of a planner, a memory layer or a supervisor is not a finding, for the reasons chapter 3 laid out. Verbose prompts, unusual file layout and disagreement about where the tools live are all comments, not blocks. The blocking list is short on purpose: it is the set of defects that cost money or do damage without anyone noticing, and every one of them can be established from source and recorded runs alone.

The line between those two lists is what the whole method is for, and it is drawn by a single question rather than by taste: does the defect produce loss that nobody observes at the time it happens. An unbounded loop over a write does, because the effect lands on an account and the loop looks like latency. A sentinel written into a ticket does, because it reads as a system error to the only person who sees it. A framework choice does not, because everyone who dislikes it can see it and say so. Reviewers who cannot state that test tend to block on the visible things, which are the ones least in need of a block.

Chip Huyen's long-form Agents piece is a good companion for the failure taxonomy behind that list — it catalogues where these systems go wrong at a level of detail no design review has time for. What the inventory adds is an order of operations: build one row per exit per loop, compose the loops by multiplication rather than by taking the largest bound, and write in the evidence column what actually fired rather than what the configuration promises. A row whose evidence column is a zero is not a passing row; it is a row with nothing behind it, and saying so in the memo is most of the job.

Whichever findings you end up writing, the memo goes to people who do not read code, and the word at the top of it is doing more work than the whole audit.