A stubbed model and one tool are enough to run an agent

Every component that gets added to an agent arrives with a claim attached. The planner is there so the work is coherent rather than opportunistic; the memory layer is there because the agent has to recall what it learned; the supervisor is there to catch wrong answers before a user sees them. Each claim is plausible, and none of them is falsifiable while the component is present, because there is nothing to compare against. That is the practical problem this chapter solves, and the solution is an old one from experimental work: build the version with none of the components in it, run your own task through it, and see what the result actually loses.

That version is smaller than people expect, and it is easier to reason about than the description suggests, because two of the four parts of an agent can be replaced by things you fully control. The tool can be a single real function. The model can be a stub — a stand-in object that returns a fixed sequence of prepared responses and ignores what it was asked. Once those two substitutions are made, every remaining line is code you wrote, and any difference between two runs is caused by something you can point at. The rest of this chapter builds that program, reads what it settles, and marks the line where it stops being evidence.

The whole thing, with the model taken out#

The stub is the interesting move. Replacing the model with a scripted sequence of responses means the loop becomes deterministic and can be run a hundred times in a second, and it means every remaining line is code you own and can therefore be held responsible for. The tool stays real, because a fake tool would make the exercise a test of nothing.

Both halves of that are worth unpacking, because they are what makes the result trustworthy rather than merely cheap. Determinism is the important one: the script is fixed, so the sequence of turns is identical on every run. Two runs that differ therefore differ because of your code, and a failure can be attributed rather than guessed at. Cheapness follows from the same substitution — no network call means no key, no bill and no waiting — which is what turns the exercise into something you can run in a test suite rather than an experiment you schedule. The real tool is the other side of the trade. The filesystem supplies the cases the happy path never produces, a missing file being the obvious one, and a fake tool that always returns a tidy string would remove exactly the conditions the loop needs to survive.

# ablation.py — a complete agent, with the model replaced by a script.
import json
import pathlib
from dataclasses import dataclass, field

TOOLS = [{
    "name": "read_file",
    "description": "Read a UTF-8 text file from the repository and return its contents.",
    "input_schema": {
        "type": "object",
        "properties": {"path": {"type": "string"}},
        "required": ["path"],
    },
}]

def read_file(path: str) -> str:
    p = pathlib.Path(path)
    if not p.is_file():
        return json.dumps({"error": "not_found", "path": path})
    return p.read_text(encoding="utf-8")[:2000]

@dataclass
class Block:
    type: str
    id: str = ""
    name: str = ""
    input: dict = field(default_factory=dict)
    text: str = ""

@dataclass
class Turn:
    stop_reason: str
    content: list

class StubModel:
    """Returns canned turns in order. Ignores the messages it is given."""
    def __init__(self, script):
        self.script = list(script)
        self.requests = 0

    def create(self, *, messages, tools):
        self.requests += 1
        if not self.script:
            return Turn("end_turn", [Block("text", text="(script exhausted)")])
        return self.script.pop(0)

SCRIPT = [
    Turn("tool_use", [Block("tool_use", id="t1", name="read_file",
                            input={"path": "pyproject.toml"})]),
    Turn("tool_use", [Block("tool_use", id="t2", name="read_file",
                            input={"path": "src/service/version.py"})]),
    Turn("end_turn", [Block("text", text="Version 2.4.0, set in src/service/version.py.")]),
]

Two of those definitions exist only to describe the shape of what a model returns. A Turn carries a stop reason and a list of content blocks. A Block carries a type, an identifier, a tool name, an argument object and a text field. Those are the fields the real provider's response objects carry, which is why the stub is a structural replica rather than a rough imitation, and that property is what makes the swap at the end of this section as small as it is.

The loop is the same shape as chapter 1's, with a counter and two exits:

def run(model, user_input, max_iterations=5):
    messages = [{"role": "user", "content": user_input}]

    for i in range(max_iterations):
        turn = model.create(messages=messages, tools=TOOLS)
        messages.append({"role": "assistant", "content": turn.content})
        print(f"iter {i}: stop_reason={turn.stop_reason} blocks={len(turn.content)}")

        if turn.stop_reason != "tool_use":
            return next(b.text for b in turn.content if b.type == "text")

        results = []
        for b in turn.content:
            if b.type != "tool_use":
                continue
            out = read_file(**b.input) if b.name == "read_file" \
                  else json.dumps({"error": "unknown_tool", "name": b.name})
            print(f"        {b.name}({b.input}) -> {len(out)} chars")
            results.append({"type": "tool_result", "tool_use_id": b.id, "content": out})
        messages.append({"role": "user", "content": results})

    return f"Stopped after {max_iterations} iterations with no answer."

print(run(StubModel(SCRIPT), "What version is this service?"))

Running it prints the trace of an agent doing agent things:

iter 0: stop_reason=tool_use blocks=1
        read_file({'path': 'pyproject.toml'}) -> 1180 chars
iter 1: stop_reason=tool_use blocks=1
        read_file({'path': 'src/service/version.py'}) -> 96 chars
iter 2: stop_reason=end_turn blocks=1
Version 2.4.0, set in src/service/version.py.

Two tool executions against the real filesystem, results fed back into a growing message list, a terminal answer, and a bound that never fired. Around sixty lines including the dataclasses, and roughly forty once the stub is swapped for the real client — which is a one-line change, because client.messages.create(model=..., max_tokens=..., tools=TOOLS, messages=messages) returns an object with the same two attributes the loop reads.

Both of those claims can be checked against the listing rather than taken on trust, and checking the second one is what explains the first. Read the loop body and note every attribute it touches on the object the model hands back: turn.stop_reason and turn.content, and nothing else. Then note every field it touches on the blocks inside that content: a block's type, its name, its input and its id. The real client's response exposes that same pair of attributes, and its tool-use blocks carry those same four fields. So the only expression in the program that has to change is the call itself, and every line that reads from the result keeps working unaltered. That is the one line.

The line counts follow from the same observation about which definitions exist only to fake the model. Counting non-blank lines, the two listings together come to around sixty. What disappears on the swap is Block, Turn, StubModel and SCRIPT, since a real client supplies its own response objects and the sequence is no longer scripted, and that removal takes out about a third of the program — leaving roughly forty. Nothing in the loop is touched, which is the point being made: the machinery under test is the same machinery in both versions.

Two details in that listing are doing more work than their size suggests. The else json.dumps({"error": "unknown_tool", ...}) branch exists because a scripted model can request a tool that does not exist, and so can a real one; without the branch the loop raises a KeyError and the turn dies with no result appended, which is the second bug from the previous chapter arriving by a different road. And read_file returns a JSON error object rather than raising when the path is missing, because a tool that raises leaves the loop to invent a result and a tool that returns an error string lets the model read what went wrong and try a different path. Both choices are about keeping the message list well-formed under conditions the happy path never produces, and both are cheaper to get right at sixty lines than at six hundred.

What an afternoon of this settles#

The point of the exercise is not that the sixty lines are production-ready. It is that they let you make the claims about the missing components falsifiable, one at a time, on your own task rather than in the abstract. That is an ablation, in the sense the word carries in experimental work: take a part out, hold everything else fixed, run the thing again, and record what the result loses.

The clause carrying the weight there is hold everything else fixed, because that is what makes the difference attributable to the part you removed rather than to anything else. It is also the clause that is hardest to honour in software, and the borrowed word quietly promises more than the exercise delivers. In a laboratory the parts being removed are physically separate. Here they are not: taking out the planner changes the prompt the model is working from, and taking out the framework changes how you observe everything else. What the afternoon actually produces is the version with all of them removed at once, and the rows below are five separate claims each evaluated against that single stripped baseline. That is a weaker design than five one-at-a-time removals, and it is the one that fits in an afternoon.

Take the planner. The claim for it is that the model needs to decide the sequence of steps before executing any of them, so that the work is coherent rather than opportunistic. Put the real model behind that loop, run it against thirty configuration questions, and count how many of them need a plan that a step-by-step loop cannot discover — on this task, the honest answer was zero, because every question was answerable by reading a file, seeing what it pointed at, and reading that. Take the memory layer. Its claim is that the agent needs to recall things across the turn, and the message list already does that; the question is whether it needs to recall things across conversations, which is a different feature with a different cost, and which the example task did not require at all.

Read that table across rather than down, because the gap between the two right-hand columns is a different kind of gap in every row. For the planner the claim is contradicted outright: the sequencing it was meant to supply was discoverable one step at a time. For memory the gap is a confusion inside the claim, which runs together recall within a turn — already provided — with recall across conversations, which nobody had asked for. For the router the gap is that the premise went untested: one prompt covered all three question types, so the claim that they need different prompts was never put in a position to be right. The framework row is two substitutions, each of which needs its own reason.

The durability half of that row is worth showing rather than stating, because the reasoning transfers. Checkpointing exists so that a run can resume after a crash without redoing the work it had already done, which means its value is bounded above by the length of the run. A turn that finishes in about twelve seconds can be started again from the beginning for about twelve seconds, so the most that checkpointing can save here is twelve seconds of work, set against the cost of writing state at every step and the bugs that come with restoring it. The comparison flips the moment turns get long, or the tools do things that cannot be repeated safely — and neither was true of a file read.

The supervisor row is the only one where the claim survived, and it is the one worth doing arithmetic on. Both of its numbers are rates over the same denominator, which is the thirty configuration questions the whole exercise was run against. One of the thirty had a wrong answer the supervisor caught. Thirty of the thirty were charged an extra model call. Nothing sits between those two figures: there is no outcome in which the supervisor ran without costing a call, and none in which it caught something without running, so the row closes on itself and no further cause is hiding inside it. Dividing one into the other gives the thing the review actually needed, which is an exchange rate — thirty extra model calls buy one wrong answer prevented.

That rate is what the ablation licenses, and it is worth being precise about what it does not. It prices the component; it does not decide it. The decision is a comparison, and the comparison has a threshold you can state in advance: the supervisor pays whenever a single undetected wrong answer costs more than thirty model calls do. Written in the general form the same statement covers every case rather than the one that was measured. If a checking component catches some number of errors across some number of attempts, and costs one extra call on each attempt, then the price of a catch is the attempts divided by the catches, and the component pays when the cost of a missed error exceeds that. Nothing has to be re-run when the cost of an error changes, which is the property that makes the form worth having.

Put the measured numbers into it and both directions fall out without further measurement. On this task a model call is a small fraction of a penny and a wrong answer is a wrong sentence in a summary of a service's configuration, so paying thirty calls a catch is a poor trade. If the supervisor catches one wrong answer in thirty and the wrong answer would have been a quoted price, the second model call is obviously worth it, and the ablation has told you the price rather than the answer. Running the experiment converts an architecture argument into a cost-benefit one, which is the useful move.

The right-hand column is the part that does not transfer. Those are findings about one task with four tools and turns that finish in three iterations, and the components in the left column exist because they earn their keep somewhere. A planner starts to pay when the task has a shape that a greedy loop, one that takes the best-looking step at each point without looking ahead, reliably gets wrong — an itinerary where the flight has to be chosen before the hotel because of the dates, and the model, choosing one step at a time, books the hotel first. Memory pays the moment "remember I prefer aisle seats" is a requirement. A framework pays when you want distributed checkpointing, or when six teams need to build agents the same way and the framework is the shared vocabulary. The ablation does not tell you those are wrong; it tells you which ones you are currently paying for without evidence.

Three of those five results are absences — nothing needed a plan the loop could not find, nothing needed to survive past the turn, nothing needed a second prompt — and an absence measured on one task is the weakest evidence there is to carry anywhere else. The supervisor row is the only one that records an occurrence rather than an absence, which makes it the most portable of the five and still leaves it resting on a single event. One catch is one catch. It is enough to establish a price and not enough to establish a rate, and a review that treats it as a rate has read more out of the row than the row contains.

The permanent test a scripted model buys#

The stub makes the loop deterministic, and that is both its whole value and its whole limit. Because the script is fixed, running the program tests the harness: that the assistant turn is appended before dispatch, that every call id gets a result, that the terminal turn is recognised, that the counter fires. Those are the properties that break in production and they can now be tested in milliseconds, with no API key and no bill. That is worth having as a permanent test, not just an afternoon's exercise — a scripted model is the cheapest way to assert that a tool error still produces a valid request, or that iteration twelve returns the thing it is supposed to return.

In practice that turns into a handful of scripted conversations kept alongside the unit tests. One where the model requests two tools in a single turn, to check that both results come back in one user turn. One where the tool raises, to check that an error result is appended rather than skipped. One where the script never returns a terminal turn, to check that the counter fires and that the string it returns is the string the caller expects rather than an exception. One where the script requests a tool name that is not in TOOLS. Each is four lines of script and runs in under a millisecond, and together they cover the failure modes that produce provider 400s and runaway loops in production. The team in the story kept theirs; it outlived the framework decision by a year.

Those four scripts are not an arbitrary selection, and the rule that generates them is worth carrying to other projects. Each one forces a path that a working system never takes: a turn with more than one call in it, a tool that fails, a conversation that does not end, and a name that is not in the toolset. A real model reaches those paths rarely and unpredictably, which is exactly why they are the paths that break. A script reaches any of them on demand, because the script is the thing deciding, and that is the one respect in which a stub is strictly better than the real thing.

Where a stubbed model stops being evidence#

The exercise tells you nothing about whether the agent works. The stub reads messages and ignores it entirely, which is exactly the capability the real model supplies: choosing src/service/version.py on iteration two because pyproject.toml turned out to point there. With a script, the sequence is a property of the script. Every interesting question about an agent — does it pick the right tool, does it recover when a tool returns nothing, does it stop when it has enough, does it hallucinate a file path — is a question about that choosing, and none of them can be asked here. An ablation with a stubbed model proves your machinery is complete. Whether the model can use the machinery is a separate exercise requiring a real model, real questions, and someone reading transcripts.

The mistake this invites is worth naming carefully, because it is not a slip and nobody makes it out of carelessness. The belief is that a green ablation has defended you against having chosen the wrong architecture. What makes it tempting is that the evidence looks complete: the trace printed above is indistinguishable from the trace a real agent prints. Two tool executions, results fed back into a growing list, a terminal answer, a counter that never tripped. Every observable quantity is the same in both worlds. The property that differs is the one that produces none of those observables — whether anything was choosing — and it is invisible precisely because the stub was built to be a faithful replica of everything else.

Say it as a rule and it becomes easy to apply elsewhere. A test that holds the visible output constant while the dangerous property varies underneath it is not evidence about that property, however green it runs. Here the dangerous property is model behaviour, and the stub's create method pops the next scripted turn regardless of what it was passed, so no amount of running it can say anything about how a model would behave. The defence the ablation genuinely provides is narrower and still worth having: it protects you from paying for a component that has no evidence behind it. It does not protect you from an agent that picks the wrong tool.

The second limit is that a passing ablation is not an argument against the components, only against adopting them by default. The supervisor row is the demonstration of that, since it did catch something, and the exercise turned a disagreement about whether to have one into a number a room can act on. Nothing in the table says a planner is a bad idea. The table says which components this task was paying for without evidence, on the day the experiment was run, and that is a smaller and more useful claim than the one people reach for when the results come back mostly empty.

Once the machinery is proven and the components have been justified or dropped, the next step is to put the real model behind the same loop and read what it does. Anthropic's cookbook has a customer service agent that is exactly this shape — model, tools, loop, stop — at a size you can read in one sitting, and it is a good reference for what the sixty lines look like when the tools are real endpoints rather than a file read.

That swap is worth being precise about, because it is the only substantive difference between the program above and a production agent. The script determined the sequence of tool calls before the program ran; the model determines it while the program is running, from what the previous results contained. Which raises the question of where the line actually falls between this and a pipeline that calls a model at each of three fixed steps — and control flow, not the presence of a model, is what answers it.