A shadowed agent really does send the email

The shadow run was switched on at 10:05 and switched off at 10:52. In those forty-seven minutes the candidate agent sent 18 emails to real customers, four of which apologised for a delay on an order that had shipped, and issued four refunds totalling €96 against real payments. The configuration had been reviewed. CRM_BASE_URL pointed at the staging instance, EMAIL_DOMAIN was set to shadow-test.internal, and both had been checked in the pull request by two people.

Shadow traffic is a copy of live production requests sent to a candidate configuration alongside the real one. Production answers the user as it always did, the candidate answers into a store, and what the candidate produced is read by the team and by nobody else, which is how it supplies behaviour on genuine inputs at zero exposure for the people sending them. That property is the one a canary from chapter 4 cannot have, since every request in a canary arm is a real answer going to a real person. What shadowing costs is a duplicate inference for every request it copies, and what it does not remove on its own is everything the candidate does besides producing text.

The email tool did not use EMAIL_DOMAIN. It called the company's notification gateway, which honours a staging flag for templated campaign sends and ignores it on the transactional /v2/send endpoint the agent uses, because that endpoint was built for password resets and nobody had considered that a non-production caller might reach it. The refund tool never had a staging concept at all: it holds the payment provider's SDK client, constructed from the same secret the production service uses, because refunds in this system are a direct provider call with no intermediate service to point somewhere else.

Nothing malfunctioned, which is the part worth sitting with: an agent with tools is a program that calls functions, the functions were the real ones, and no component in the path had any notion that this run was hypothetical.

The redirect is the wrong boundary#

Configuring each integration to point at a test environment is a list, and it is only correct while the list is complete. New tools get added by people who are not thinking about shadow mode. Existing tools acquire second code paths. Tools reach shared internal gateways whose staging semantics belong to another team and change without notice. A retry wrapper can bypass the redirected client entirely if it was written against the lower-level HTTP layer. Every one of those is a way for the list to fall out of date silently, and you find out by reading the outbox.

The boundary that holds is the point where the agent runtime dispatches a tool call, because every tool goes through it by construction. In shadow mode that dispatcher never calls an implementation.

class InertDispatcher:
    """Records intended calls. Never invokes a tool implementation."""

    def __init__(self, prod_trace: Trace, recorder: Recorder) -> None:
        self._prod_trace = prod_trace      # the production run for this request
        self._recorder = recorder

    def dispatch(self, call: ToolCall) -> ToolResult:
        replay = self._prod_trace.find_result(call.name, canonical(call.args))
        if replay is not None:
            source = "replayed"
            result = replay
        elif call.name in SAFE_READ_ONLY and call.name in SYNTHESISABLE:
            source = "synthesised"
            result = synthesise_from_schema(call)
        else:
            source = "unavailable"
            result = ToolResult(error="tool_unavailable_in_shadow")

        self._recorder.record(
            request_id=self._prod_trace.request_id,
            call=call, result=result, result_source=source,
        )
        return result

Two properties of this are worth being deliberate about. It is default-deny: a tool the shadow harness has never heard of falls through to unavailable rather than to an implementation, so adding a tool cannot silently add a side effect. And it records the intended call with its arguments, which is the actual product of a shadow run — issue_refund(order_id="A-88213", amount_eur=45.00) is the finding, whether or not it would have succeeded.

What the inert call returns back to the model is a design decision with consequences, because the result becomes context and steers everything after it. Returning an error to every call produces a candidate that spends the run apologising for system problems, which tells you nothing about its judgement. Replaying the production run's actual tool results, matched on tool name and canonicalised arguments, keeps the candidate on realistic ground for as long as it is asking for the same things. In a week-long run on this support assistant, 63% of candidate tool calls matched a recorded production result, 22% were synthesised for read-only tools, and 15% had no basis — and those 15% are not noise to be smoothed over, because a run that hits unavailable is a run whose later turns are reacting to an artefact of the harness.

One layer is not enough for something that sends email on behalf of your company, so the dispatcher is backed up by controls that fail closed independently of it. The shadow worker runs with credentials that are revoked or scoped to nothing, so a call that somehow escapes the dispatcher gets a 401 rather than a delivery. Its network policy denies egress to everything except the model provider and the comparison store, so a call that somehow has credentials cannot reach the provider holding them. And the first shadow run is twenty requests, stopped, with a human reading every recorded intended call and checking the outbox, the refund ledger and the CRM audit log for anything that moved. Forty-seven minutes is a long time to find out; twenty requests is not.

Where the traffic goes#

The copy is enqueued and consumed asynchronously, off the request path, so that a slow or failing candidate cannot add latency to the response the user is waiting for. Running the candidate inline and discarding its output is the version people build first, and it couples production latency to an unproven configuration for no benefit, since nothing about the comparison needs to happen within the request.

Rate limits deserve a line of their own. The candidate consumes provider quota, and if it draws on the same key as production then a week of shadowing doubles the request rate against a limit that was sized for one. A separate key with its own quota, or at minimum a token bucket — a local rate limiter holding a refilling allowance of requests — that yields to production, is what keeps a shadow run from throttling the service it is meant to validate.

Money: 100,000 requests a day for seven days is 700,000 additional inferences, and at the candidate's $0.021 per request that is about $14,700 for the week. Sampling to 25% of traffic brings it to $3,675 and still yields 175,000 comparisons, which is far more than the analysis will use.

Comparing decisions rather than wording#

The comparison store held 8,400 shadowed conversations after the first day. Cosine similarity on the final response text, which measures how closely two pieces of text sit once each has been turned into an embedding vector, was above 0.9 for 71% of them, which reads as reassuring and is close to meaningless: both agents are writing polite support replies in the same house style, so text similarity mostly measures the style. It is highest, in the worst cases, exactly where the decisions differ — "I've processed your refund, it will appear in 3–5 working days" is nearly identical whether the refund was €5 or €45.

The comparison that carries information is structured over the decisions:

• Did the candidate call the same set of tools, and in what order?
• On the tools that move money or state, are the arguments equal — order_id, amount_eur, case_status?
• Did it reach the same terminal decision: refund, no refund, escalate to a human, ask a clarifying question?
• Did it cite the same policy, and does that policy support the claim it made?
• Did it ask for information the customer had already provided, or act without information it needed?

On this run, decisions diverged in 4.1% of conversations, 344 of them. Of those, 61 were refund-versus-no-refund, and 12 issued an amount differing by more than €20 from production's. Those 73 conversations are a day's reading for one person and they are the entire value of the exercise. The other 271 were mostly escalation-versus-clarifying-question, which is a product judgement rather than a defect.

Trajectories, meaning the ordered sequence of model steps, tool calls and results that made up one run, diverge irreversibly once the candidate makes a different tool call, because from that point it is in a state production never occupied and the replayed results stop matching. Comparison therefore has to be prefix-oriented: identical up to step k, first divergence at step k, classification of that divergence. Trying to align full trajectories after they have separated produces diffs that grow without meaning anything.

Shadowing next to its neighbours#

The distinctions are practical. Shadowing cannot tell you whether the candidate is better, because there is no user on the other end to react to it — no reply, no follow-up question, no thumbs-down, no escalation. Grading shadow output offline gives you a judge's opinion of it, which is worth having and is not the same thing. Shadowing also cannot validate anything whose behaviour depends on side effects landing, so a refund flow's downstream consequences — the ledger entry, the confirmation email, the customer's next message — are outside what a shadow run can observe by definition. And the inert layer is itself a source of false divergence: a candidate that behaves correctly against a synthesised tool result that was wrong will look like it made a mistake.

The week of output is now yours to look after#

At the end of the run the comparison store holds a week of production requests with customer names, order numbers and free-text complaints, plus two sets of model outputs about them, plus every intended tool call with its arguments. That is a new copy of production personal data, created for debugging, living in a system that was built last month by two engineers.

Reducing what you store is cheaper than governing it. Redaction at capture — replacing customer names and contact details with stable pseudonyms before the row is written, keeping order numbers because the comparison needs them — removes most of the sensitivity while leaving the decision comparison intact, and it has to happen on the write path because a redaction job that runs nightly means the raw content existed for a day. Sampling helps too, since 175,000 comparisons support the same conclusions as 700,000 and represent a quarter of the exposure.

The obligations arrive with it. The store needs the same classification and access controls as the production data it copies, which usually means it is not a bucket anybody on the team can list. It needs a retention period that somebody has decided — thirty days is a defensible default for a promotion decision that will be made within one — and enforcement of that period rather than an intention. It has to be reachable by erasure requests, because a customer's deletion propagating through production and not through the shadow store leaves you with data you have told a regulator you deleted. It belongs in the record of processing activities, the internal register of what personal data the company handles and for what purpose, which a regulator can ask to see. If it lives in a different region than the production store, residency commitments may already be broken. And if the candidate runs on a provider the production configuration does not use, a week of customer content has been sent to a processor that may not be covered by the existing agreement or listed in the subprocessor notice, the published list of outside parties you have told customers may handle their data.

The same reasoning condemns the original plan of pointing the candidate at the staging CRM. Staging is where access is broad, where retention is nobody's job, where snapshots get copied to laptops, and where the data is supposed to be synthetic. Filling it with real complaints from real customers moves production data into weaker controls, which is the outcome the redirect was supposed to avoid.

All of this — the shadow comparisons, the canary arms, the flag values, the versions — assumes that each recorded row can be tied to what was actually running when it was produced. The next chapter takes a request from nine days ago and asks precisely that question, with a regulated customer waiting for the answer.