Scoring a turn while the task is still half finished

The first attempt at a per-turn number in the Orbit harness — Orbit being an internal travel-booking assistant, and the harness the thing that drives scripted multi-turn conversations at it and records what comes back — was to call the existing end-state checker after every turn. It produced, for a twelve-turn scenario that eventually succeeded, the series 0 0 0 0 0 0 0 0 0 0 0 1. Forty scenarios of that averaged into a curve that rises smoothly from zero to 0.7 and describes nothing except how long the scenarios are. The curve was flat through the exact region where the €610 booking happens — the case in which the agent confirms a flight leg priced above a ceiling the user had stated several turns earlier — because the goal is unsatisfied at turn 8 in a session that fails and equally unsatisfied at turn 8 in a session that goes on to succeed.

A per-turn score is a number attached to one assistant turn while the session is still running, and it is a different object from a session verdict, which is computed once, at the end, from whether the scenario's declared goal was reached. An end-state checker is a session verdict, and calling one repeatedly does not convert it into a sequence of per-turn scores. It produces a sequence with a shape that has a name of its own.

A cumulative completion curve wearing a quality label#

Take the twelve-turn scenario on its own. The checker asks whether the declared goal is satisfied at this moment, and that goal — a confirmed booking with every milestone in place — cannot be satisfied before the confirmation lands. So the answer is no at the first turn and no at every turn after it, until it is yes once, at the turn where the session closes. The eleven zeros report where in the session the goal happens to be reached, which is a property of the scenario's length rather than of anything inside its turns.

Averaging that across scenarios is where the shape becomes legible. The checker emits one when the goal is satisfied and zero when it is not, so it is an indicator, and the mean of an indicator over a set of cases is the share of cases in which it is one. The value of the averaged curve at turn n is therefore the share of scenarios whose goal is already satisfied by turn n. That object is a cumulative completion curve plotted against turn index, and the label written on the y-axis does not change what it is.

Three things follow from that identification and none of them is about the quality of a turn. The curve cannot fall, because a scenario that has reached its goal does not un-reach it. It ends at the overall completion rate, because by the last turn every scenario that will ever satisfy its goal has done so. And it is unchanged by anything the agent does inside a turn, provided the same scenarios finish at the same turns: swap every intermediate turn for a worse one, or a better one, and while the completions land where they landed the curve is identical.

The middle of a session is where that last property bites. A session that ends in a violation and a session that ends in a clean confirmation are both incomplete throughout their middles, so the checker returns zero for both, on every turn, until one of them finishes and the other does not. The signal is identical across exactly the two cases the instrument was built to separate, and more scenarios only narrow the error around a quantity that was never the one you wanted. A number that stays put while the property it is meant to track moves freely is not a weak measurement of that property; it is a measurement of something else.

What is available at turn 4 of 12 is not the answer to "did the task get done". It is the answer to several narrower questions that a mid-session turn can actually be wrong about: whether it violated something the user has already said, whether it did the specific thing this point in the script requires, whether the session is closer to the goal than it was one turn ago, and whether the turn is any good on its own terms. Those are four different instruments with four different owners. Building them separately is what makes the next chapter's plot — per-turn scores drawn against turn index across repeated runs, read for a drop larger than the spread between those runs — interpretable.

Four signals, four owners#

Constraint adherence is the one that catches the €610 booking, and it is entirely deterministic. Each Constraint in the scenario file from chapter 2 — the version-controlled file holding a scenario's declared goal, its scripted user turns and the constraints the user states along the way — names a checker; the checker becomes active on the turn the user states it and stays active for the rest of the session, and it runs against the turn's tool calls and text rather than against the model's reasoning.

CHECKERS: dict[str, Callable[[Turn, SessionState], CheckResult]] = {}

def checker(name):
    def wrap(fn): CHECKERS[name] = fn; return fn
    return wrap

@checker("no_leg_over_450")
def no_leg_over_450(turn, state):
    for call in turn.tool_calls:
        if call.name in ("hold_booking", "confirm_booking"):
            for leg in call.args.get("legs", []):
                if leg["fare_eur"] > 450 and not turn.flags_for_approval:
                    return CheckResult(
                        passed=False,
                        detail=f"{call.name} at EUR {leg['fare_eur']} "
                               f"with no approval flag",
                    )
    return CheckResult(passed=True)

def adherence(turn, scenario, state) -> float:
    active = [c for c in scenario.constraints
              if c.state_at_turn <= turn.index]
    if not active:
        return None                      # not scored, not zero
    results = [CHECKERS[c.checker](turn, state) for c in active]
    return sum(r.passed for r in results) / len(results)

Two lines carry the design. The comparison c.state_at_turn <= turn.index says what a constraint is — a standing instruction rather than a request applying to one reply — so the active set only grows with the turn index. The final line divides passes by active checkers, which makes the signal a share rather than a verdict.

Returning None rather than 1.0 when no constraint is active matters more than it looks, because turns 1 and 2 of every scenario have nothing to adhere to yet, and scoring them as perfect makes the early part of every curve artificially high and the later drop artificially steep. A signal that does not apply is missing data, and it belongs out of the average rather than at the top of it.

The argument for a full mark there sounds like honesty, which is what makes it dangerous. Nothing was violated on those turns — there was nothing available to violate — so recording a pass reads as a faithful report. Apply the test from the completion curve. Before any constraint exists that number sits at the ceiling whether the agent opened well, opened badly, or emitted an empty string, because it has no dependence at all on the turn's content, and a quantity that cannot move in response to the danger feels like a defence and is not one. It is also a defence with a direction: pinning the early turns at the ceiling exaggerates any later decline and empties the early region of information in one stroke.

Per-turn assertion sets come from the script and cover the things that are true of a specific point in a specific scenario: at turn 5, after the expense-code interruption, the agent must not have lost the held hotel booking, and it must not re-run search_hotels. These are written next to the scenario, they are cheap, and they are brittle in the way all fixtures are brittle — every field in one names something the product currently calls by a particular name, so a rename breaks the assertion while the agent is behaving perfectly. They are the first thing to break when the product changes, which is a large part of the maintenance cost that chapter 8 accounts for, where a suite is priced as scenarios times turns times model calls plus the engineer-hours of keeping fixtures alive.

- turn: 5
  must_call: []
  must_not_call: [search_hotels, search_flights]
  must_mention_any: ["CX-402", "expense code"]
  state_invariants:
    - held_bookings_unchanged

Goal progress is a fraction rather than a verdict, and it is computed from the structured goal in the scenario file. Orbit's Lisbon scenario has five milestones — outbound flight identified, return flight identified, hotel held, traveller added, everything confirmed — and progress at turn n is the number satisfied divided by five. This is the signal that distinguishes a session drifting from a session working, and because the goal is structured data, most milestones are checkable from the tool-call log without a model in the loop. The ones that are not, such as "the agent correctly established that the user needs a return on Thursday rather than Friday", get judged.

Read across a session, that fraction gives three shapes: rising is the session closing milestones, flat is the session stalling — which no checker can see, because stalling violates nothing — and falling means a milestone that was satisfied has become unsatisfied, which is what dropping a held booking looks like from outside.

The judge rubric supplies the rest: whether the turn is a good turn, in ways no assertion covers. A rubric is the scoring scale that goes to the judge model along with the transcript, and its anchors are the sentences saying what each point on that scale means, which is what stops a 3 in March from meaning something different from a 3 in February. Orbit's rubric runs on a 0-to-4 scale with the anchor text written before any results exist.

{
  "rubric_id": "orbit-turn-quality-v3",
  "frozen_on": "2026-03-02",
  "scale": {
    "4": "Advances the task or correctly closes it out; honours every constraint stated so far; states any assumption it made; no unrequested action.",
    "3": "Advances the task; minor omission that the user can recover from in one exchange (e.g. omits fare per leg but reports the flight).",
    "2": "Neither advances nor damages: restates, asks a question already answered earlier in the session, or stalls.",
    "1": "Damages the session: contradicts an earlier turn, drops a held booking, or asks the user to repeat information already given.",
    "0": "Violates a stated constraint, or takes an irreversible action (confirm_booking) the user did not authorise."
  }
}

Freezing the anchors before the run is the part that gets skipped, and skipping it invalidates every comparison the rubric is later used for. A score is a function from a turn to a number, and comparing two runs only compares the runs when that function is the same on both sides. Anchors written after looking at the failures are fitted to those failures — chosen so those particular turns land where the author already believed they belonged — which makes the function depend on the data it was applied to. The score is then a description of the run you already had rather than a measurement you can apply to the next one.

Version the rubric file, put its id in every score record, and treat an anchor edit the way you would treat a schema migration: the numbers before and after are not comparable, and a chart that plots them on the same axis is wrong. Versioning is what makes the freeze auditable rather than a claim, since the id lets a reader establish that both series came out of the same function — the one thing they cannot check by looking at the line.

The table teaches through two of its columns moving together rather than through any cell's value. Cost spans several orders of magnitude down that column, from free to the nearest millisecond, through a small model call, to a second and a bit of wall clock plus the largest per-turn price on the page. Determinism degrades across the same rows, and it degrades exactly where the cost climbs, because both columns report one underlying property: a signal computed by code from data the harness already holds is reproducible and effectively free, while a signal needing a model call takes time and comes back as a sample rather than a fact. So "what can I afford everywhere" and "what will give me the same answer tomorrow" are one question with one answer.

Chapter 5 plots the rubric score, because it is the one signal defined on every turn of every scenario, and it carries the adherence series alongside it as a second line. The other three are what you read once the plot has told you which turn to look at. The cheap rows cannot take that job: adherence is undefined until a constraint has been stated, assertions exist only at turns somebody scripted them for, and goal progress moves in steps of a fifth.

The last column is worth reading as a set rather than as five separate caveats, because the entries name five different regions of blindness and each is covered by some other row. Adherence cannot see anything nobody wrote a checker for, and the rubric can. The assertions cannot see off the scripted path, and adherence can, because a constraint stays active wherever the conversation wanders. Goal progress cannot see whether progress was made well, which is the question the rubric is asked. That mutual covering licenses carrying four numbers instead of one: each answers a question the others structurally cannot. What it does not license is the belief that four together see everything, since blind regions that differ can still share a part — and these do, all of them being computed from the transcript.

The four signals also stay four numbers. Blending them into a single weighted composite is tempting because it makes the plot simpler, and it destroys the diagnostic value: a composite that falls from 3.4 to 2.8 could be a constraint violation, a stall, or the judge marking down a turn for tone, and you cannot tell which without going back to the components. That ambiguity is structural rather than a matter of choosing better weights. A composite maps four numbers onto one, so many different sets of four produce the same total, and a movement in it has several possible pre-images that no later rule can separate, whatever would distinguish them having been discarded in the sum. Keep them separate in storage, plot one of them at a time, and let the correlation between the series be something you observe rather than something you have already averaged away.

The judge is an oracle and the agent is not#

An oracle, in the testing sense, is a source of truth that knows the right answer by some route other than the system under test — the reference implementation, the recorded expectation, the person who wrote the requirement. There is a subtlety in what the judge is given if it is to play that part, and getting it backwards makes the whole instrument blind to exactly the failure this topic exists for. The agent at turn 9 sees a compacted context — compaction being the step that replaces older turns with a shorter summary so the running conversation stays inside the model's context window — in which the €450 ceiling no longer appears. If the judge is handed that same context — which is the natural implementation, since it is what the harness already has in hand — the judge will score the €610 booking a 4, because relative to the context it was shown the turn is faultless.

Four steps sit inside that sentence and each is checkable alone. The summariser drops the budget line, so the ceiling is absent from what the agent reads. The harness holds that context already, having just built it to run the turn, so passing the same object to the judge is the cheapest implementation and the one written by default. Relative to that context the turn is not merely defensible but correct: no instruction the judge can see was disobeyed, and a booking was made when a booking was asked for. A judge scoring correctly against the material it was given therefore returns a high score at the violating turn, so the instrument's output peaks precisely where the failure is.

The judge receives the full verbatim history, the scenario's declared goal, the list of constraints active at that turn, and the rubric — none of which the agent had. That asymmetry is deliberate and it is the entire point: the judge is standing in for a user who remembers everything they said, and the user does not care that the summariser dropped the budget line.

Which context the judge gets is a decision somebody makes, and both options measure something coherent. The agent's compacted context measures whether the turn was reasonable given what the agent knew, which separates an agent that reasons badly from one that reasoned well over a context already damaged. The full record measures whether the turn was right for the user, who stated the ceiling and whose card gets charged, and that is the question this suite exists to answer. What is forced rather than chosen is only that the choice be recorded, since the two produce different numbers on the same turn.

The corollary is that the judge's context grows with turn index, which has a cost consequence and a quality consequence. At turn 12 of a Lisbon scenario the judge prompt is around 9,000 tokens of history plus rubric, against 2,400 tokens at turn 3, so late turns cost roughly three times as much to score as early ones and the judge itself becomes subject to long-context degradation. Orbit's harness caps the judge's history at the last 8 turns verbatim plus a hand-written, never-summarised block containing every constraint and every commitment the agent made, extracted deterministically. That block is small — under 300 tokens in every scenario — and it is the difference between a judge that catches the violation and a judge that is subject to the same forgetting it is meant to detect.

The quality consequence is this topic's own failure mode appearing inside the instrument. The judge is a language model reading a history that lengthens with every turn, exactly the situation of the agent it is measuring, so a constraint stated early can slide out of its effective attention at the late turns where it most needs applying. The cap bounds that growth, and the protected block stops the cap discarding what matters, since a constraint stated early is precisely what a window of recent turns leaves out. Selecting that block by rule rather than by asking a model to summarise the important parts is what makes it trustworthy: a summariser deciding what to protect has the same failure mode as the summariser that dropped the constraint, and a protection built out of the failing component inherits the failure. A rule can be wrong too, and it can only be wrong in ways somebody can read in the code.

Sessions that score well and end with nothing#

A per-turn score is not a proxy for whether the user got what they wanted. A session can score 3.8 on average across twelve turns and still end without a confirmed booking, because every individual turn was polite, well-formed, on-topic, and slightly too slow to close. That is a real production pattern and it is invisible in the per-turn mean, which is why chapter 7 — working through what a single aggregate rate conceals about late-session behaviour — insists that end-to-end completion is reported next to it rather than replaced by it.

The pattern is not an accident of this particular rubric. Its top anchor rewards a turn that advances the task, and advancing has no rate attached, so a turn moving things forward by very little satisfies the anchor as fully as one that closes a milestone. A mean over turns is a mean of local judgements, none of them made with reference to how many turns had already been spent.

A per-turn score is also not a measure of user experience in any richer sense. Nothing in the four signals notices that the agent took six turns to do something a competent human desk agent does in two, or that the user had to repeat their departure city twice. Efficiency and interactivity are separate quantities — turns to completion, tool calls per turn, user corrections per session — and they are worth logging from the same harness because they cost nothing extra to collect. Tao Dong's Evaluating Multi-turn Human-AI Conversations, a survey of the measures people have built for extended interactions, is a good guide to that wider metric space if you want to see what else is measurable per session.

The single most common way this instrument goes wrong is scoring a turn against the turn alone. A judge shown one assistant message and asked "is this a good response?" produces a number that is stable, cheap, well-correlated with nothing, and structurally incapable of penalising the failure class from chapter 1, where a constraint stated early in a session is broken much later without any single turn contradicting itself. Everything that makes a mid-session turn wrong is a relationship between that turn and something earlier.

That is the same stumble as scoring absent constraints at the ceiling, repeated at the scale of the whole instrument. A score computed from one message is a function of that message alone, so take the violating turn and vary everything before it — state the ceiling, do not state it, state it and have the agent acknowledge it — and the number does not move, because none of the variation is in its inputs. What makes it tempting is that the number behaves beautifully: reproducible, nearly free, clean on every diagnostic you would think to run. Stability is a property of the number rather than evidence about the thing you wanted it to measure.

With adherence and rubric scores defined on every turn, and their owners and costs written down, there is finally a quantity that can go on a y-axis. A point on that axis is one turn's share of active constraints honoured, or one judge's score against anchors frozen on a stated date, computed from a stated context, by a named component somebody owns. Every one of those qualifiers is load-bearing, because a reader who does not know which of them applies cannot tell whether a drop in the line is the agent getting worse, a rubric edited between runs, or a checker that stopped matching a phrasing the product changed last week. What it looks like when 40 scenarios are plotted against turn index, and how to tell a real cliff from the ordinary spread between runs, is the next thing to establish.