A simulated user is only useful when it is reproducible

A simulated user is a program that plays the human side of a conversation with an agent under test, so a long session can be run without a person sitting in it. The Orbit harness — the rig that drives those sessions and scores them — holds 40 of them as scenarios, each a goal, a persona and a script for one traveller booking one trip. Its headline figure is end-to-end completion: the fraction of scenarios in which the agent fully satisfied the structured goal before the session ended. Driving such a conversation takes an afternoon. Driving the same conversation twice is the actual requirement, and the first version of the harness missed it — four runs against one unchanged commit returned 70 percent, then 82.5 percent, then 72.5 percent, then 80 percent.

Every claim an evaluation suite makes is a comparison of two runs, so the apparatus rests on the assumption that a number which did not move means nothing moved. An instrument whose own readings wander by more than the effect you are hunting cannot report that effect, and will report effects that are not there. Reproducibility, throughout this chapter, means the mechanical property that the same suite run twice against the same agent produces the same conversations, and therefore the same score, to a spread you have measured rather than hoped for.

The instrument has to be a fixture, not a personality#

The shift that fixes a wandering instrument is to stop thinking of the simulated user as a character and start thinking of it as a test fixture that happens to be written in English. A fixture is allowed to be complicated, and it is allowed to contain a language model, but it has to produce the same conversation on Tuesday that it produced on Monday, so that when the number moves you know it was the agent that moved.

Concretely, a scenario in the Orbit harness is a file, version-controlled, whose content hash goes into the run identifier.

from dataclasses import dataclass, field

@dataclass(frozen=True)
class Constraint:
    id: str
    utterance: str          # exactly what the user says, verbatim
    state_at_turn: int      # when they say it
    checker: str            # name of a programmatic assertion

@dataclass(frozen=True)
class Interruption:
    at_turn: int
    utterance: str
    resumes_goal: bool      # does the user return to the main task after?

@dataclass(frozen=True)
class Scenario:
    id: str
    persona: str            # "impatient, terse, replies in <15 words"
    goal: dict              # structured, not prose — see below
    opening_utterance: str  # turn 1 is fixed text, never generated
    constraints: list[Constraint]
    interruptions: list[Interruption]
    max_turns: int = 12
    seed: int = 0

TRIP_LISBON = Scenario(
    id="book-042-budget-ceiling",
    persona="Frankfurt-based sales engineer, terse, answers in one line.",
    goal={
        "book_flight": {"from": "FRA", "to": "LIS",
                        "depart": "2026-04-14", "return": "2026-04-16"},
        "book_hotel": {"city": "LIS", "nights": 2},
        "add_traveller": {"name": "Ana Reis", "to": "hotel"},
    },
    opening_utterance="I need to get to Lisbon on the 14th for a workshop.",
    constraints=[
        Constraint(id="budget-450", state_at_turn=2,
                   utterance="Anything over EUR 450 per leg needs my "
                             "manager's sign-off, so keep it under that "
                             "or flag it to me first.",
                   checker="no_leg_over_450"),
    ],
    interruptions=[
        Interruption(at_turn=5, resumes_goal=True,
                     utterance="Wait — what expense code do I use for "
                               "customer workshops?"),
    ],
)

Three of those fields carry most of the reproducibility. The goal is structured data rather than a sentence, which means the driver can decide when the goal is satisfied without asking a model, and the same structure is what chapter 4 — where a turn is scored while its task is still half finished — scores partial progress against. The opening_utterance is literal text, because turn 1 sets the trajectory of everything after it and generating it is the single largest source of run-to-run drift. And constraints and interruptions carry explicit turn indices, so the budget line lands at turn 2 in every run and the expense-code question lands at turn 5 in every run, regardless of what the agent said at turn 4.

Turn 1 earns that treatment for a structural reason. Every turn the agent produces is conditioned on the whole history above it, and turn 1 sits in all of those histories. A different opening sends the agent to a different tool call, which returns different options, which is what the user model then has to choose between. Variance introduced at turn 1 is therefore multiplied through the session rather than added to it, so what comes out is a different conversation and not one turn of noise. Variance at the last turn can only affect the last turn.

The interruptions field deserves more than its one line in the dataclass, because a simulated user that pursues its goal monotonically from turn 1 to turn 12 is the easiest possible conversation partner and no real session looks like that. People change their minds, ask an unrelated question in the middle, correct something the agent got wrong four turns ago, and abandon a sub-task without saying so. Each of those does specific damage: a topic change pushes the original goal further back in the context and makes it a candidate for eviction, a correction creates two contradictory facts in the history and tests which one the agent honours, and an abandoned sub-task leaves a dangling commitment that a competent agent should either close or surface. Orbit's scenarios carry between one and three interruptions each, drawn from a small library of shapes, placed at turn indices chosen once from a seeded random number generator — seeded meaning it is started from a fixed integer, so it emits the same sequence of draws on every run — and then written into the file as literal integers. Generating the placement fresh from the seed on each run would also be reproducible, and writing it down instead means a reader of the scenario file can see what the test does without executing anything.

Two kinds of decision are stacked in that last sentence. The seed is forced — without it an interruption lands at turn 5 on Monday and turn 7 on Tuesday, which is a change to the test wearing the costume of a change to the agent — while writing the drawn integers into the file is a choice somebody made and could have made otherwise.

Sessions also need an ending, and letting the model decide when to say goodbye reintroduces the variance you just removed. The driver terminates on three conditions, checked in order: the structured goal is satisfied, max_turns is reached, or the agent has produced two consecutive turns that make no tool call and add no new information, which is the harness's definition of a stall. Recording which of the three fired is itself a signal, because a suite in which the stall condition fires on 8 of 40 scenarios has a different problem from one where 8 scenarios hit the turn cap. Sessions that end early for goal satisfaction are also the reason per-turn numbers need careful denominators, and that, in a chapter about how one aggregate number hides everything happening late in a session, is chapter 7's problem.

The user model still generates the connective tissue: acknowledgements, answers to the agent's clarifying questions, choices between the three hotels it was offered. That generation is where the remaining variance lives, and it gets pinned in two stages, the second of which leans on a cassette: a stored record of replies the user model has already produced, filed under the conversation state that produced each one and played back whenever the harness reaches that state again. The name is borrowed from the HTTP-testing libraries that record real responses to a file and replay them in later runs, and the idea is the same one — a live call the first time, a recording afterwards.

def user_turn(scenario, history, turn_idx, cassette):
    # 1. Scripted content always wins over generation.
    for c in scenario.constraints:
        if c.state_at_turn == turn_idx:
            return c.utterance
    for i in scenario.interruptions:
        if i.at_turn == turn_idx:
            return i.utterance
    if turn_idx == 1:
        return scenario.opening_utterance

    # 2. Otherwise generate — but replay if we have seen this state before.
    key = cassette.key(scenario.id, turn_idx, history)
    if key in cassette:
        return cassette[key]

    reply = user_model.complete(
        system=USER_SYSTEM_PROMPT.format(persona=scenario.persona,
                                         goal=json.dumps(scenario.goal)),
        messages=history,
        temperature=0.0,
        seed=scenario.seed,
        max_tokens=60,
    )
    cassette[key] = reply
    return reply

The cassette is keyed on the conversation state, which matters more than it looks. Keying on turn index alone would replay the user's turn 6 even when the agent said something completely different at turn 5, producing a transcript in which the two sides are talking past each other: change a prompt so the agent offers two hotels where it offered three, and a stored "the second one, please" fires anyway, and whatever score attaches to that transcript scores a conversation nobody would have had.

Keying on the hash of the full history — a short fingerprint of the entire message list, identical only for identical histories — means a replay only fires when the conversation genuinely reached the same place, and a real agent change causes a cassette miss and a fresh generation. That turns a miss into a signal. A miss rate near zero tells you the agent behaved identically. A miss rate of 60 percent on a run tells you something substantive changed, which is useful before you have looked at a single score, because the miss rate falls out of the run itself and needs no scorer and no judge.

The borrowed word stops carrying at that point. An HTTP cassette replays a server's deterministic response to an identical request, so a hit stands in faithfully for the live call and a miss means only that the client asked for something new. A conversation cassette replays a generated reply: a hit guarantees the conversation state matched and guarantees nothing about whether the stored reply is still what the current user model would produce, so pinning a newer model version turns every recording into a fossil. A miss, conversely, carries information the HTTP case never has, because what moved the key was the agent under test.

The shape of one turn#

Everything crossing a boundary in that diagram is a place where entropy gets in, and the tool layer is the one teams underestimate. Orbit's search_flights hits a live sandbox of a GDS — a global distribution system, the shared airline inventory and fare database that every travel tool queries — and the fares in that sandbox move daily. A run in which the cheapest Thursday return is €412 and a run in which it is €610 are testing different things, and the budget constraint is only exercised in the second. Under the ceiling the agent can take the cheapest option it is offered and satisfy the EUR 450 rule without ever noticing the rule exists; over it, the agent has to detect a breach and either flag it or look elsewhere. The pass rate on that constraint is therefore part agent and part price feed, and a week of cheap fares reads as an improvement nobody shipped.

The fix is the same as for the user: record the tool responses once, key them on the normalised call arguments, and check the cassette into the repository next to the scenario. That also freezes now(), which otherwise makes "next Tuesday" mean a different date every week and quietly invalidates every date assertion in the suite. The chain there is the quietest failure of the set. A relative date is not a fixed instruction but one resolved against whatever now() returns, while the assertions were written against the date it resolved to when the scenario was authored — the trip above asserts a departure of 2026-04-14, which does not move. Once the two drift apart, the suite fails an agent that did the right thing, on a schedule set by the calendar rather than by the codebase.

Every row names a source of variance, the mechanism that pins it, and the damage when it is left moving. Seven of the eight share that shape, so reading them for their levels teaches little — they are seven instances of one instruction. The finding is the gap between the last row and those seven, and it is a difference in kind: its middle column says the thing cannot be pinned, and its right-hand column says nothing breaks anyway.

The last row is the honest one. Setting temperature to 0 — the sampling setting at which the model takes its highest-probability continuation at every step instead of drawing from the distribution — and pinning a seed reduces variance substantially and does not eliminate it, because hosted inference batches your request with other people's, and floating-point reduction order inside a batched matrix multiply is not stable across batch compositions. Mixture-of-experts routing, where each token is passed through a subset of the model's sub-networks selected at run time, adds more of the same.

Both are structural rather than defects anyone could fix on request. Batching is how a hosted endpoint pays for its hardware, so the batch your call lands in depends on who else is calling at that instant, which is not a property of your run and is invisible from it. Floating-point addition is not associative, so a changed reduction order shifts the scores assigned to candidate next tokens and a nearly tied token can flip. Expert routing stacks a second copy of the effect one layer up, since which sub-networks a token traverses is itself decided from those scores.

Measuring a floor is nonetheless a complete answer to variance you cannot remove. What the suite is asked for is one decision rule: whether a difference between two runs belongs to the agent or to the instrument. A pinned source settles that by being absent from both runs, and an unpinned one settles it by being bounded, which costs a few repeat runs and licenses exactly the same conclusion.

In practice Orbit's suite, with the agent's own sampling as the only unpinned source, reproduces end-to-end completion within about ±2.5 points across five identical runs of 40 scenarios. That residual is the noise floor, it is a property of the suite that you measure rather than assume, and every claim the suite makes has to clear it. Before the cassettes and the fixed scripts, the same measurement was ±6 points.

The suite's resolution is one scenario#

End-to-end completion is not a continuous quantity, and treating it as one is what makes a difference of a point or two look discussable. The suite holds 40 scenarios, each of which either finished with its structured goal satisfied or did not, so the numerator of the rate is a count of whole scenarios and the denominator is 40 in every run. Both are properties of the suite rather than of the agent: the denominator is fixed because the scenario files are checked in, and the numerator is an integer because the end-to-end measure awards no partial credit.

That settles which values the rate can take. Write the number of satisfied scenarios as k, so the reported rate is k divided by 40. Let one scenario flip from failing to passing: the count becomes k + 1 and the rate becomes (k + 1) divided by 40. The two fractions may be subtracted directly, because they share a denominator and nothing moved except the count, which leaves 1 divided by 40 — and 1 divided by 40 is 2.5 percent. Every reachable value of the rate is a multiple of 2.5 points, so the resolution of the instrument is fixed by the suite's size before a single run happens. Set that against the floor: five identical runs of the frozen suite reproduce completion within about ±2.5 points, which is exactly one scenario, so resolution and repeatability have landed at the same size and a finer floor would need the agent to be steadier than the arithmetic.

The four early figures confirm they are counts wearing percentage signs. Seventy percent of 40 scenarios is twenty-eight of them, 82.5 percent is thirty-three, 72.5 percent is twenty-nine, and 80 percent is thirty-two — every one a whole scenario, as every reachable value must be. Read that way the 70-to-82 excursion stops being twelve and a half mysterious points and becomes five scenarios that passed in one run and failed in another with nothing in the agent changed.

The same conversion prices the old floor. A spread of ±6 points is between two and three scenarios in each direction, a band a little under five scenarios wide, and that band is what made the excursion possible. An agent change worth five points of completion is two scenarios, which sits comfortably inside it. The rule this licenses is blunt: no claim smaller than one scenario is measurable on a suite of this size, whatever a dashboard prints to a decimal place.

An identity is hiding in the first paragraph of this section. The numerator is a count and the denominator is fixed, so the rate and the count are one fact written two ways: given either, plus the suite size, you recover the other exactly. That licenses reporting the count. Nothing is lost by saying thirty-three of 40 scenarios instead of 82.5 percent, and something is gained, because the count puts the granularity on the face of the number while the percentage hides it behind a decimal point implying a precision the suite does not have.

The residual is the half that gets assumed. The identity fixes what the instrument can resolve, meaning the smallest difference it is able to express, and says nothing about whether two runs of one commit land within a single step of each other. That is a separate empirical fact, obtained by running the suite against itself and reading the spread, and it came out at one scenario only after every other source of variance had been pinned. Resolution is arithmetic and repeatability is a measurement, and a suite can have a fine resolution and a floor several steps wide.

Reproducible is not the same as right#

It is worth being clear about the alternative that this replaces. Before the simulator, the team's approach to long sessions was to replay real production transcripts: take a session that went badly, feed the user's messages back to the agent in order, and see whether it did better. That works while the agent's behaviour is close to the behaviour that produced the transcript, and it degrades the moment the agent diverges, because the user's turn 5 in the recording was a reply to a message the new agent no longer sends. Past a certain amount of drift you are feeding the agent non-sequiturs and scoring it on how well it copes. Production transcripts remain the best source of scenario material — the goals, the constraints, the interruptions worth scripting — and a live simulator is what turns that material into something that survives a prompt change.

A frozen scenario file gives you an instrument whose readings you can compare. It says nothing about whether the readings correspond to anything, and the distance between those two sentences is where the expensive mistake lives. Once the scripts are frozen and the cassettes checked in, the reported number becomes extremely stable, and a stable number reads as a correct one. That reading is tempting for a good reason: every source of variance the team could name has been eliminated, and each elimination was individually correct. But the list was a list of things that made the number move, and no item on it was a check on what the number was about.

The shape to recognise is a quantity pinned while the dangerous property ranges freely. You can build a perfectly deterministic simulated user that says "great, thanks!" to every answer the agent gives, replay it a thousand times, and get an extremely stable 100 percent completion rate that describes no human being. Swap it for a user that argues, corrects and abandons sub-tasks and the stability does not move — both are equally frozen — while the validity of the measurement travels the whole way from worthless to excellent. Reproducibility is a property of the instrument's repeatability and carries no information about what the instrument is pointed at, which is why watching it feels like a defence and is not one. Reproducibility is a precondition for measurement, and validity is a separate argument you have to make separately.

The freezing also has a cost that shows up on a calendar rather than a bill. Verbatim utterances at fixed turn indices rot: rename a tool, change the wording of a confirmation prompt, add a required field to the booking flow, and scenarios start failing for reasons that have nothing to do with the behaviour you meant to test. Orbit's team measured 9 of 40 scripts needing repair in one quarter, at roughly 25 minutes each. That is the standing tax on this approach, and chapter 8, which totals a suite's cost as scenarios times turns times tool calls, puts it next to the token bill where it belongs.

What the apparatus buys is a trade. A scenario file that fixes the opening utterance, the constraints, the interruptions, the tool responses and the clock cannot surprise you with a failure mode nobody scripted, and it charges a repair bill every quarter. In return it yields a number whose movement means something, and an advance statement of how small a movement may mean anything: one scenario, by arithmetic and by measurement alike.

There is a third limit worth naming now because it shapes the next chapter. That chapter is about what a simulator inherits when it runs on the same model as the agent it tests. A cassette can only replay conversations that were generated in the first place, and what generated them was a language model given a persona and a goal — which is a model's idea of a user, complete with a model's idea of when an answer is good enough. Freezing that idea makes it repeatable. It does not make it a user.