A simulator sharing the agent's model shares its blind spots

A simulated user is a language model given a persona and a goal and pointed at the agent under test, so that a multi-turn evaluation — one scoring a whole conversation rather than a single reply — runs without a person on the other end. What it reports is a task-completion rate: the fraction of scenarios whose goal was satisfied by the end of the session. Two models produce that number, and only one of them is under test.

The slide says Multi-turn eval: 87% task completion (30 scenarios), and underneath it a line that reads "simulated user: mid-tier model, persona + goal prompt". The proposal attached to it is to ship a change to the memory strategy on the strength of that number, against a previous run that scored 80 percent. Everything in the harness is built the way chapter 2, on making a simulated user reproducible, describes — frozen scripts, cassettes (recorded tool responses replayed instead of live calls), pinned model, reproducible to within two points. The number is real in the sense that re-running produces it again. It still does not support the conclusion, for three reasons that stack.

Twenty-six out of thirty#

The first problem is arithmetic. Eighty-seven percent of 30 scenarios is 26 successes and 4 failures. That is a count rather than a percentage: a rate over 30 trials can only land on multiples of one thirtieth, and no value in between exists for it to take.

Around any rate computed from a small number of trials there is a confidence interval, which is the range of underlying success rates that would plausibly have produced the result you observed; the Wilson method is one way of computing it, preferred here because it stays sensible at small samples and near the ends of the scale where the textbook normal approximation does not. That approximation lays a symmetric band around the observed proportion, which is fine in the middle of the scale and breaks near the ends. At twenty-nine out of thirty it puts the upper end above certainty, which is not a rate anything can have, and at thirty out of thirty the width it computes is zero, so it returns a single point and claims no uncertainty at all. A suite reporting 87 percent operates in exactly that regime.

Wilson asks the question the other way round. Rather than banding the observation, it solves for the set of underlying rates that would plausibly have produced the count observed, and since every candidate it considers is itself a rate, no arithmetic can push the answer past either end of the scale. A 95 percent Wilson interval on 26/30 runs from about 70 percent to about 95 percent.

from math import sqrt

def wilson(successes: int, n: int, z: float = 1.96) -> tuple[float, float]:
    p = successes / n
    denom = 1 + z**2 / n
    centre = (p + z**2 / (2 * n)) / denom
    half = (z / denom) * sqrt(p * (1 - p) / n + z**2 / (4 * n**2))
    return centre - half, centre + half

wilson(26, 30)   # (0.703, 0.947)
wilson(24, 30)   # (0.634, 0.907)   <- the "80%" baseline

Every symbol in that function does one job. successes is the number of scenarios that passed and n the number of trials, so p = successes / n is the observed proportion — the 87 percent itself. z sets the confidence level, and the default 1.96 is the number of standard deviations enclosing 95 percent of a normal distribution. denom, which is 1 + z**2 / n, sits slightly above one and shrinks whatever is divided by it; the amount by which it exceeds one falls away as n grows, which is the term making more trials buy a narrower interval. centre is deliberately not p: adding z**2 / (2 * n) before dividing by denom pulls the centre away from whichever end of the scale p is near, and that is what keeps the answer inside the scale. half is the half-width, and the two returned values sit either side of the shifted centre rather than either side of p.

Putting twenty-six and thirty through it by hand reproduces the stated band. p is twenty-six over thirty, the 87 percent on the slide. z**2 is 1.96 squared, a shade under four, so z**2 / n is near enough an eighth and denom is about one and an eighth. The shift added to p is half that eighth, a bit over six points, taking the numerator of centre to about ninety-three percent; dividing by one and an eighth leaves a centre a little over eighty-two percent. Inside the square root, p * (1 - p) / n is 87 percent times the thirteen percent left over, a bit above eleven percent, divided by thirty — about four thousandths. The other term is a shade under four divided by four times thirty squared, about one thousandth. Those two sum to roughly five thousandths, whose square root is about seven points, and multiplying by z / denom — a shade under one and three quarters — gives a half a little over twelve points. Taking that off the centre gives 0.703 and adding it gives 0.947.

The second call in the code does the same work for the baseline run's twenty-four successes. The two intervals overlap across nearly their whole width, which means the run that scored 80 percent and the run that scored 87 percent are consistent with the same underlying quality. What makes them overlap is the number of trials rather than the closeness of the two results. Each interval spans about a quarter of the whole scale because thirty is a small number of trials, and the seven points under argument are a small fraction of that width, so a gap twice the size would disappear inside it too. The same fact is available as a count: twenty-six and twenty-four differ by two scenarios, and one scenario in thirty moves the rate by a bit over three points. Two scenarios flipping the other way would have reversed the ranking.

To distinguish 87 percent from 78 percent with 80 percent power — an 80 percent chance of detecting a difference of that size when it is genuinely there — and a two-sided test at α = 0.05, which tolerates a 5 percent chance of announcing a difference that is not, takes roughly 280 scenarios per arm (one arm per variant under comparison, so one for the current memory strategy and one for the proposed change) when the arms are independent, which nobody has. Pairing helps a great deal here — running both variants on the identical scenario set and testing the per-scenario differences rather than the two rates removes most of the between-scenario variance, and for a suite where the two arms agree on the large majority of scenarios it can bring the requirement down by an order of magnitude. It works because a scenario both variants pass, and one both variants fail, contribute nothing to the per-scenario difference. That is a real technique and it is worth using, and it still does not rescue 30 scenarios with a 7-point gap.

There is a second statistical problem underneath the first, which is that the interval assumes the 30 scenarios are a sample from the population you care about. They are not a sample of anything; they are 30 scenarios someone wrote, which makes the interval a lower bound on the uncertainty rather than an estimate of it. If all 30 were written by the engineer who also wrote the agent's prompt, they encode that engineer's model of what users do, and the population they generalise to is that model.

The simulator says "great, thanks"#

The second problem is that the simulated user is not a hard grader. Given a persona and a goal, a helpfulness-tuned model — one whose post-training rewarded agreeable, cooperative replies — is disposed to cooperate with its interlocutor, and it will treat a partial answer as a complete one far more often than a person with money on the line would. The bias is structural, because the verdict and the dialogue come out of the same place: the judgement that a goal is satisfied is produced by the same policy that produces the next utterance, and that policy was trained to make agreeable continuations likely. Accepting is the agreeable continuation and challenging is not, so the probability of an accept is raised by something with no bearing on whether the goal was met.

Measuring the size of that bias means handing sessions the simulator has already passed to people and asking whether the task was done. On this suite eleven of thirty such sessions were rejected by at least two of three raters, and the arithmetic that follows is where most readers go wrong.

The eleven are a share of the sessions the simulator passed, which is not the denominator the run reports against. Eleven out of thirty rated sessions is an over-acceptance rate of about a third, and the thing it is a rate of is passes rather than scenarios. The trap is that the rated set and the scenario set both happen to number thirty, so the wrong division produces the same fraction and looks like the right one. Read as a share of scenarios, the eleven rejections would come straight off the twenty-six passes and leave fifteen out of thirty, or half. That is a different claim and a wrong one: the run produced twenty-six passes rather than thirty, so thirty passed sessions had to be drawn from a wider pool than one run supplies.

Against the right denominator the correction is a multiplication rather than a subtraction. About a third of the twenty-six passes do not survive a human read, which leaves roughly seventeen standing, and seventeen over the full scenario count of thirty is a little under sixty percent. The immediate consequence is that 87 percent is not 87 percent even within its own frame. If the simulator over-accepts on something like a third of the sessions it passes, the human-anchored completion rate for that run is nearer 60 percent, and the interval around it is no narrower. The correction moves the point estimate and leaves the thirty trials that set the width where they were.

The tempting belief at this point is that reproducibility is a defence. The freezing work described in chapter 2 — frozen scripts, cassettes, a pinned model — is real and correct, and without it the completion rate is an impression that drifts rather than a number. What that work pins is the rate. It does not pin the simulator's willingness to accept a partial answer, and that willingness decides what the rate means. The same 87 percent comes out of a suite whose simulator refuses every confirmation missing a required detail and out of one that thanks the agent for an incomplete confirmation, because a rate counts verdicts and carries nothing about how they were reached. A quantity that holds still while the dangerous property moves underneath it is the shape to watch for.

Correlated errors#

The third problem is the one specific to running the simulator on the same model family as the agent, and it is the least visible because it produces no anomaly in the logs. Two instances of the same model, or two models from the same family, do not fail independently. They share pretraining data, tokenisation, and a great deal of preference tuning, which means they share priors — the default expectations a model brings to a situation before it has looked at the specifics in front of it — about what a booking flow looks like, what a confirmation message must contain, and which details are worth asking about. When the agent omits something because its prior says that detail is unimportant, the simulator has the same prior and does not notice the omission.

The chain from a shared prior to a missed failure has four links and none of them is optional. The agent writes a confirmation and leaves a detail out, because its prior says a confirmation of this kind does not carry that detail. The simulator then decides whether its goal is satisfied, which amounts to asking whether anything is missing, and it answers from the same prior. Nothing is missing according to it, so the session is recorded as goal-satisfied. No log shows an anomaly, because no disagreement occurred for a log to record. A longer run only draws more samples from the prior generating the error.

The Orbit team's flight tool holds a fare for two hours. The agent's system prompt says so; the agent nevertheless told users at several points that the hold was good "for 24 hours", which is the industry-typical figure and is what a model trained on the open web would guess. The simulated user, running on the same family, never once questioned it across 30 scenarios and 340 turns. A rules-based checker caught it in an afternoon; a same-family simulator would not have caught it in a year, because the error is in the prior rather than in the reasoning.

That is the same shape as the reproducible rate and a sharper instance of it. The constant quantity was the simulator's agreement rate, pinned at the top of its scale across every scenario and every turn in the run, while the property varying underneath was the agent's factual accuracy — correct about the hold in its system prompt, wrong about it whenever it spoke. The agreement rate could not move, because a shared default is not a conclusion either model reasoned its way to, and so not something the second model can disagree with by thinking harder. A simulator drawn from the agent's own family cannot detect an error whose cause is a prior the two share, however long the run.

Swapping the driver to a different provider's model, holding everything else fixed, moved reported completion from 87 percent to 71 percent on the same 30 scenarios. That gap is not a measurement of which simulator is right. It is a measurement of how much of the original number was the simulator agreeing with itself. Holding everything else fixed is what licenses that reading: the agent, the scenarios, the tools and the success criteria did not move between the runs, so the sixteen points cannot be a property of the agent, which behaved identically in both. Two things it does not license: it settles nothing about which figure is closer to the truth, since neither driver is an oracle and a lower number is not automatically the honest one, and it is not an estimate of the correlation, because the two drivers still share most of a pretraining corpus and a similar tuning objective, so the swap removes only the part of the shared prior they do not have in common — which makes the sixteen points a lower bound on the correlated component rather than a measurement of it.

No row in that table is the answer on its own. The scripted user cannot react, so it exercises none of the recovery behaviour that matters; the live human costs three orders of magnitude more per session and cannot be re-run against tomorrow's commit. The working arrangement is a different-family model driving the user, calibrated against a small number of human-rated sessions, with a fully scripted spine underneath it for the constraints and interruptions.

The table teaches through the differences between rows rather than through any row's best cell, and the sharpest difference runs between its top three rows and its bottom three. Every row carrying no correlated-blind-spot risk at all is a row where no model generates the user's turns at test time, and each pays for that in one of two ways: the scripted user and the recorded transcript are frozen and cannot react, while the live human reacts as well as anything can and cannot be re-run. That is not a coincidence. What creates the correlated blind spot is a model generating the user's side from priors it shares with the agent, and that same generation is what makes the user responsive in the first place, so independence from the agent's priors is bought by removing the generation and the reactivity leaves with it. The trade is forced; the particular arrangement recommended above is not, and a team able to spend on live sessions would land elsewhere for defensible reasons.

Anchoring, calibration, a second family and adversarial variants#

Four changes turn the number into evidence, in rough order of how much they buy per hour spent.

Anchor the scenarios in real sessions. Sample production transcripts — stratified, so that long sessions and abandoned sessions are represented rather than only the tidy ones — and derive scenario goals, constraints and interruptions from what those users actually did. This is what converts the interval from "uncertainty about 30 things I invented" into "uncertainty about a sample of the traffic". Thirty human-anchored scenarios are worth more than 200 invented ones, and the scenario count still has to grow.

Calibrate the automatic verdict against human raters on a subsample. Double-rate 60 sessions, compute agreement, and report it alongside the completion number. Orbit's first calibration gave Cohen's κ of 0.48 between the simulator's own goal-satisfaction verdict and the human majority, which is poor; κ is an agreement score discounted for the agreement two raters would reach by guessing, so 0 is chance, 1 is perfect, and values under about 0.6 mean the two are agreeing rather less than the raw percentage suggests. After replacing the simulator's verdict with a programmatic check against the structured goal — the machine-readable list of fields a finished session must contain, declared per scenario in the scenario file that chapter 2, on making a simulated user reproducible, describes — plus a judge with explicit anchors, meaning a model scoring the transcript against written descriptions of what each score is for, κ came to 0.79. A completion rate reported without a calibration figure is asserting agreement rather than showing it.

Drive the user with a different model family from the agent, and pin both. This is the cheapest of the four to implement and its effect is immediate. It does not deliver independence — the two models still share most of a pretraining corpus and a similar tuning objective, so treat it as reducing the correlation rather than removing it — and the honest way to report it is to run both and show both numbers, because the gap between them is itself informative about how much of your result is model agreement.

Add adversarial variants of each scenario. Not a hostile user, but a normal one who is harder to satisfy: re-asks when an answer is vague, supplies an incorrect fact at turn 6 to see whether the agent corrects it or adopts it, changes their mind about a date, and refuses to accept a confirmation that omits a detail the scenario declared required. Run these as separate scenario ids rather than as randomness inside the existing ones, so that the cooperative and adversarial rates are reported separately. Orbit's adversarial set scored 44 percent against 71 percent for the cooperative set on the same underlying tasks, and the 27-point gap is a more useful piece of information than either number alone. The gap outranks the levels because the two sets are built on identical tasks, so task difficulty is held fixed and cannot be what moved: the 27 points are attributable to a user who does not accept, which is the quantity neither rate reports on its own.

ADVERSARIAL = Persona(
    id="persistent-checker",
    system=(
        "You are the traveller. Do not accept a confirmation that omits "
        "any item in `required_details`. If a detail is missing, ask for "
        "it once, plainly. At turn 6 state one incorrect fact about the "
        "itinerary drawn from `planted_error` and see whether it is "
        "corrected. Never thank the agent for an incomplete answer."
    ),
    required_details=["confirmation number", "fare per leg",
                      "return date", "seat assignment"],
    planted_error="I think we said the 15th for the return, right?",
)

A calibrated simulator is an instrument, not a user#

Doing all four does not make the simulated user a user. It makes the number a measurement with a known and stated uncertainty, produced by an instrument whose disagreement with human judgement you have quantified on a subsample. That is the most that is available at this cost, and it is enough to support a claim of the form "this change moved per-turn adherence at turn 8 by 0.6 points on 40 paired scenarios, with human calibration κ = 0.79" — which is a claim someone can act on, and is a very different object from "87 percent".

Each change leaves a residue worth knowing before its numbers get quoted. A different-family driver reduces the shared prior without eliminating it, so a wrong default most of the open web also holds — the class the fare-hold mistake belongs to — survives the swap and still needs a rules-based check against the system prompt. A κ computed on a double-rated subsample is itself an estimate from a sample, and over-acceptance concentrated in a rare shape of session is under-represented there by the same mechanism that made that shape rare.

The slide reports the output of two models with one of them under test, and the three problems above are three ways of saying so: thirty trials are too few for a seven-point gap to be a gap, a helpfulness-tuned grader passes sessions a person with money on the line would reject, and a grader from the agent's own family cannot see an omission both were trained to find unremarkable. Re-running the suite surfaces none of the three, which is what makes reproducibility so comfortable a thing to mistake for validity.

There is also a limit that no amount of simulator quality removes. Task completion is an end-state verdict: the goal is satisfied or it is not, and it is only knowable when the session ends. The 87 percent slide has nothing to say about what happened at turn 4 of a session that eventually succeeded, or about a session that is still running. Finding where in a session quality collapses requires a number per turn, and turn 4 of 12 has no end state to assert against.