Two implementations that both work are not equally cheap

The same argument had run three times in six weeks, with different requirements and identical structure: someone proposes a mechanism, someone else proposes a different one, both demo successfully, and the meeting resolves by seniority or by who is willing to build it that sprint. The third time, the requirements were written on the board first and the mechanisms deliberately left blank, and the discussion took eleven minutes because most of the disagreement turned out to be about which column people were weighing rather than about the facts.

The three requirements were these, taken verbatim from the planning doc.

R1. Every SQL migration the agent writes must be reversible, must not take an ACCESS EXCLUSIVE lock for longer than two seconds on a table over ten million rows, and must add indexes concurrently. Proposals: put it in the system prompt; write a skill; build a validate_migration(sql) tool.

R2. The agent must be able to answer whether a feature flag is currently on for a specific account. The flag service is owned by our team and sits in the same VPC. Proposals: in-process tool; MCP server run by the platform group, since "flags are a platform concern".

R3. When a customer disputes an invoice, the agent must reconcile the invoice against usage records for the disputed period across the metering store, the entitlements record and the billing ledger, then explain the difference. Four agent products owned by four teams read entitlements; the entitlements team already runs the service and has offered to expose it over MCP. Proposals: a sub-agent; a set of tools; a server; some combination.

The five columns, filled in once#

Before running requirements through it, the matrix is worth writing down as a reference. Each column has been established by an earlier chapter, and the point of the table is that no mechanism dominates: every row wins somewhere.

The columns are not commensurable and should not be scored into a total. What they do is force the argument onto a named axis: when two people disagree about R2, the table makes them say whether they are disagreeing about ownership or about latency, and those have different evidence.

Running the three requirements#

R1 is standing knowledge, by the test from chapter 1 — nothing leaves the process, and the correct migration is fully determined by rules that can be written down. That eliminates the tool as the primary mechanism and leaves prompt against skill, which chapter 2 settles on session mix. Migrations appear in roughly 8% of this agent's sessions, so the rules do not belong resident; they belong in a skill whose description names the situations (writing, reviewing or editing a database migration, schema change, ALTER TABLE or index creation).

The interesting part of R1 is what happens to the rejected tool. validate_migration(sql) is a real and useful piece of code, and the mistake is only in making it a thing the model may choose to call. Run as a CI check on the pull request, the same function has the correctness of its own tests, because nothing can route around it. R1 resolves into a skill that shapes generation plus a validator in the pipeline that gates it, and neither of those is the mechanism anyone proposed.

R2 is a deterministic external action with one consumer, which is the case chapter 3 rules on. The flag service is ours, on our release train, reachable from our process, and the credential is one our agent may legitimately hold. Nothing about "flags are a platform concern" changes any column: platform ownership of the service is already true and is not affected by whether our agent calls it through a library or through a second process. The in-process tool wins on cost, owner and latency, and ties on containment. This is the requirement instrumented below, because it is the one where both implementations demonstrably work and the argument has to be settled on something other than whether it works.

R3 is two requirements wearing one coat, which is the most common reason these discussions stall. The entitlements read has four consumers on four release trains and a team willing to operate it, so it is the server case from chapter 3 — three facts at once, all present. The reconciliation is something else: an open-ended walk across three stores whose shape depends on what it finds, generating intermediate results nobody wants afterwards, which is the delegation profile from chapter 4. Splitting them gives a sub-agent that holds the reconciliation loop and calls three tools, one of which happens to be backed by a server.

Chapter 4's warning applies to the last row and was checked before committing to it: an aggregate-returning usage_summary(account, from, to) was built first, and it removed enough of the walk that the sub-agent's remaining job was three calls rather than eleven. The delegation survived the cheaper-repair test, which is the only way it should ever survive.

Instrumenting R2, and what the harness has to record#

Both implementations of the flag check were built and run against the same 500 replayed agent sessions, 140 of which involved a flag question. The harness wrapped the dispatch point so that every call produced one record regardless of which side of the boundary the body lived on:

@dataclass
class CallRecord:
    session_id: str
    tool_name: str
    implementation: str        # "in_process" | "mcp"
    latency_ms: float
    outcome: str               # "ok" | "error" | "timeout"
    error_class: str | None
    result_tokens: int         # tokenized length of what entered context
    result_bytes: int

def instrumented(name, impl, fn):
    def call(**kwargs):
        t0 = time.perf_counter()
        try:
            result = fn(**kwargs)
            outcome, err = "ok", None
        except Exception as exc:
            result, outcome, err = error_payload(exc), "error", type(exc).__name__
        rec = CallRecord(
            session_id=current_session(), tool_name=name, implementation=impl,
            latency_ms=(time.perf_counter() - t0) * 1000, outcome=outcome,
            error_class=err, result_tokens=count_tokens(result),
            result_bytes=len(json.dumps(result)),
        )
        emit(rec)
        return result
    return call

Three fields there are the ones teams leave out. result_tokens is the cost the capability imposes on the conversation, which is invisible if you only time the call. error_class is what makes a reliability number readable — 31 timeouts and 31 auth failures are the same rate and completely different problems. And recording implementation on every record rather than running the two variants in separate experiments is what allows a paired comparison on identical sessions, which matters because session difficulty varies more than the implementations do.

Two things the harness cannot record were measured by hand. Operational burden was logged by the two engineers involved as hours spent over six weeks on anything that would not have existed under the other implementation. Behaviour under unsafe input came from a fixed adversarial suite of 40 sessions in which the flag name, the account identifier or the returned flag description carried injected instructions.

Latency is the smallest difference in that table and the only one anyone had argued about. The resident schema cost is nine times larger for the server, for a reason that has nothing to do with the flag check: the platform group's server exposes nine methods, and the agent pays for all nine schemas on every turn of every session, whether or not flags come up. That is chapter 2's residency argument reappearing on the action side of the matrix, and it is a real reason to prefer narrow servers over broad ones if you take a server at all.

The adversarial results being identical is the finding worth carrying forward. Both implementations returned the same attacker-influenceable description field into the same context beside the same tool list, so the injection surface is a property of the capability and the arrangement rather than of where the body runs. Choosing a server over a tool bought nothing there, and expecting it to is a common misreading of what a process boundary does.

What the numbers do not settle#

The measurement establishes that the two R2 implementations are functionally equivalent and unequally expensive, and it does not establish which to ship, because the largest column in the table — 19 hours against 1.5 — is a cost someone might reasonably choose to pay if a second and third consumer were coming. The instrumentation prices the options; the decision needs a statement about the future that no benchmark can supply.

That is also the point at which a recommendation stops being reportable and starts being arguable. A reviewer reading "we chose the in-process tool, here are the numbers" will ask what would have to change for the answer to be different, and a write-up without an answer to that question is a preference with a table attached. The next chapter writes the recommendation for a case with a stated budget, names the thresholds that flip it, and shows why crossing one of them is a relocation rather than a rewrite.