Sorting a feature's checks

Classifying a feature's checks is a paper exercise that takes an afternoon and determines what the delivery process can do for the next year. The rule from the previous chapter is one question — does the same commit produce the same verdict, in minutes — and applying it to obviously deterministic and obviously sampled checks settles nothing anyone was arguing about. The argument is always in the middle.

This chapter works four checks through, drawn from an order-support assistant that answers questions about deliveries from a retrieved knowledge base. Given a fixed retrieval fixture, does the rendered prompt contain the three chunks in rank order, carry the user's locale, and have no unresolved template variables? When the provider returns 429 — the HTTP status meaning too many requests, sent when a rate limit is exceeded — with a retry-after header, does the client back off, retry at most three times, and surface a typed error? Does the model's response validate against the order-answer JSON schema? And are the answers good? The first two sort themselves; the third and fourth are the exercise.

Rendering is a pure function that happens to produce a prompt#

A prompt assembler takes a user question, a locale, a retrieved chunk list and a system template, and returns a string. Nothing in that is stochastic. Freeze the retrieval output as a fixture and the assembler becomes an ordinary function with an ordinary expected output.

"Pure function" in that heading is exact rather than decorative, and unpacking it is what shows where the repeatability comes from. The assembler's output is settled entirely by its arguments, and it consults nothing else — no clock, no environment variable, no network, no random source, no module-level state an earlier test could have changed. Run it twice with the same arguments and the same string comes back. That is not carefulness on the day; there is simply no channel through which anything else could reach the result.

Where the chunk list comes from is the part that was not like that. A retriever queries an index, and an index is a store outside the process that gets rebuilt, reranked and topped up on a schedule of its own. Let the test call the retriever and every one of those movements arrives inside the assembler's output, and the check stops being about the assembler at all. Freezing the retrieval result as a fixture is therefore not a convenience for avoiding a network call. It is the step that puts the last unpinned input under the test's control, and it converts the question from "does the whole retrieve-then-render path behave" into "does this function, given these three chunks, produce this string" — which is a question with one right answer.

# tests/test_prompt_rendering.py
import re
import json
from pathlib import Path

from assistant.prompt import render_order_prompt
from assistant.retrieval import Chunk

FIXTURE = json.loads(
    (Path(__file__).parent / "fixtures" / "retrieval_so449120.json").read_text()
)
CHUNKS = [Chunk(**c) for c in FIXTURE["chunks"]]  # 3 chunks, scores .81 .74 .62

UNRESOLVED = re.compile(r"\{\{\s*[a-z_]+\s*\}\}")


def test_chunks_appear_in_rank_order():
    prompt = render_order_prompt(
        question="where is my order",
        locale="nl-NL",
        chunks=CHUNKS,
    )
    positions = [prompt.index(f"[chunk:{c.id}]") for c in CHUNKS]
    assert positions == sorted(positions)
    assert [c.id for c in CHUNKS] == ["kb_7741", "kb_2210", "kb_9084"]


def test_locale_reaches_the_system_block():
    prompt = render_order_prompt("where is my order", "nl-NL", CHUNKS)
    assert "Respond in nl-NL." in prompt.split("<context>")[0]


def test_no_unresolved_template_variables():
    prompt = render_order_prompt("where is my order", "nl-NL", CHUNKS)
    assert UNRESOLVED.search(prompt) is None, UNRESOLVED.search(prompt).group()

Twelve milliseconds, no network, identical verdict every run. The third assertion earns its keep most often in practice: an unresolved {{ order_history }} in a shipped prompt is invisible in a diff review, costs nothing to detect, and degrades answers in a way that looks like a model problem for as long as nobody thinks to print the rendered prompt. A regression in chunk ordering behaves similarly — the answers get subtly worse because the most relevant passage is now third, and every instinct on the team points at the model.

Targeted assertions are not the only deterministic option here, and the competing one is worth naming because it is the default in most snapshot-testing cultures: store the entire rendered prompt as a golden file and diff against it. That version is also deterministic and also fast, so it passes the repeatability criterion, and it fails for a different reason. A prompt template gets edited constantly — a sentence about tone, a reordered instruction, a new section for order history — and every one of those edits reddens a golden-file test that is asserting nothing anybody meant to assert. Golden files are the right tool where the artefact is genuinely supposed to be stable, such as a serialised tool-call schema published to clients; a prompt under active iteration is not that.

The mechanism underneath that is worth following, because a golden file over a moving artefact does not merely become weak — it inverts into the opposite of a check. Equality on the whole file asserts every property the artefact has at once, including the hundreds nobody chose to assert. That breadth is the appeal and it is also the arithmetic: the surface on which the check can catch a real regression is the entire file, and so is the surface on which a deliberate edit sets it off. What it returns is one bit, differs or does not, and working out which of the differences mattered is left to a person reading a diff. Now add the rate at which the artefact moves. A prompt under iteration is edited often, so the false alarms arrive often, and the fix for each one is a single flag: regenerate the file, commit it, move on. Regenerating accepts every difference in the diff rather than the one that was intended, so each cheap fix quietly ratifies whatever else moved with it. After a few rounds the check no longer says the prompt is correct. It says the prompt is whatever it was the last time somebody pressed the button.

Retries are a state machine you wrote#

A retry ladder is code the team owns entirely. Its inputs are HTTP status codes, headers and a clock; its outputs are a sequence of attempts and either a result or a typed error. Substituting a fake transport for the provider makes the whole thing deterministic, and freezing the clock makes the sleep intervals assertable rather than merely observable.

# tests/test_retry_ladder.py
from assistant.client import OrderModelClient, RateLimited
from tests.fakes import FakeTransport, FrozenClock

def test_backs_off_on_429_then_succeeds():
    transport = FakeTransport([
        (429, {"retry-after": "2"}, ""),
        (429, {"retry-after": "4"}, ""),
        (200, {}, '{"order_id":"SO-449120","status":"in_transit"}'),
    ])
    clock = FrozenClock()
    client = OrderModelClient(transport=transport, clock=clock, max_retries=3)

    result = client.answer("where is my order")

    assert result.status == "in_transit"
    assert transport.attempts == 3
    assert clock.sleeps == [2.0, 4.0]          # honours retry-after, not jitter-only
    assert clock.total_elapsed <= 30.0


def test_gives_up_with_a_typed_error():
    transport = FakeTransport([(429, {"retry-after": "1"}, "")] * 4)
    client = OrderModelClient(transport=transport, clock=FrozenClock(), max_retries=3)

    try:
        client.answer("where is my order")
    except RateLimited as exc:
        assert exc.attempts == 4
        assert exc.provider_request_ids == ["req_a", "req_b", "req_c", "req_d"]
    else:
        raise AssertionError("expected RateLimited")

The frozen clock is the piece most often left out, and leaving it out is what turns this into a flaky check rather than a slow one. A retry ladder sleeps, and a sleep is an effect on real time. With a real clock the test waits the six seconds the ladder asked for, and any assertion about elapsed time becomes a measurement of the machine the test ran on rather than of the decision the code made. A loaded build agent stretches a two-second sleep, the assertion on total elapsed time goes red, and the red carries no information about the commit — which is exactly the failure the previous chapter's criterion exists to keep out of a merge gate. Injecting a clock converts the sleep from an effect into a recorded value, which is why asserting clock.sleeps == [2.0, 4.0] is both faster than a stopwatch and a stronger statement than one: it checks what the ladder decided to wait, and the decision is the thing under test.

The mechanics of substituting the provider are topic 2.17's subject, on testing code that calls a model, so what matters in this listing is not the fake. It is that the property under test has exactly one right answer. There is no version of this where the correct behaviour is "usually three attempts". A ladder that retries four times on a 429 is wrong on every run, forever, and a check that says so is a test in the plainest sense.

"One right answer" is a claim about the shape of the specification rather than a statement of confidence, and it can be checked rather than asserted. Write the ladder down as a mapping: from a sequence of status codes and headers, to a number of attempts, a list of waits, and one outcome. Then look at where each input on the left comes from in the listing above. The statuses come from the fake transport, the wait lengths come from its retry-after headers, the clock is handed in as an argument. Nothing else is consulted, and there is no input to that mapping the test does not supply. A second run therefore has no freedom to answer differently, and it has none for a structural reason rather than because the ladder happens to be well behaved.

Schema validation is two checks wearing one name#

"The model's response validates against the order-answer JSON schema" is ambiguous in a way that matters, and pulled apart it is two questions.

The first is whether the validation and repair machinery behaves correctly on a given response. Feed it a recorded completion with a trailing prose paragraph after the closing brace and it should strip and parse; feed it one with "quantity": "two" and it should reject with a field-level error naming quantity; feed it one missing carrier_ref and it should apply the documented default rather than throwing. Every one of those has a single correct outcome, the inputs are files on disk, and the verdict repeats. That is a test.

The move that made it a test generalises, and it is the only lever available on the deterministic side, so it is worth stating on its own. Any stochastic process can be cut into two halves: a draw, and a deterministic function of that draw. Persist the draw — write the completion to a file — and everything downstream of the file is a function of a fixed input, so it repeats. Nothing about the model became more predictable in the process. The randomness was moved out of the check and into a one-off act of recording.

That has a price, and the price is invisible from inside the check, which is what makes it worth writing down beside it. A frozen draw stands in for the distribution it came from only for as long as the distribution holds still. The day the model version changes, or the prompt changes, or user questions start arriving in a shape nobody recorded, the fixtures go on passing and go on saying nothing about live behaviour. The check cannot report its own staleness and no assertion inside it could, because staleness is a fact about the world outside the fixture. So the honest scope of these tests is narrower than it looks: they show the repair machinery is correct on the responses it has already seen, and they are silent about the responses it has not.

The second of the two questions is what fraction of live responses validate on the first attempt. That number is a property of the model, the prompt, the temperature and the distribution of user questions, and it moves when any of those move. Measured over 200 sampled production questions for this assistant it came out at 97.2% first-attempt validity, with the failures concentrated in multi-order questions where the model emitted an array where the schema wanted an object. Measure it again next week on a different 200 and it will read 96.4% or 98.1%, and neither number is wrong.

Why it moves while nothing is broken is worth making concrete, because "it is a sample" is a sentence a reader can repeat without being able to predict the next case. The figure is a proportion: how many sampled responses parsed, divided by how many were sampled. With two hundred cases each individual case is worth one two-hundredth of the reported rate, so a handful of cases landing the other way moves the headline by a point on its own. Two separate things decide which way those cases land. The model draws differently on the same question, which is the generator's own variance arriving one case at a time. And the two hundred questions are themselves a fresh draw from live traffic, so next week's set carries a different mix of the question shapes the assistant handles badly — a few more multi-order questions in the sample lowers the rate with no component of the system having changed at all.

That second source is what puts the check on the eval side rather than making it a slow test, and it is also what makes the number worth having. The question mix is not in the repository and never will be, so no amount of pinning brings it under the check's control. A rate that depended only on committed code would repeat, and it would also be blind to the thing teams actually get hurt by, which is traffic changing shape underneath a prompt that was tuned for the old shape.

So the validity rate is an eval, and a valuable one: a drop from 97% to 84% after a prompt edit is a real regression the deterministic parser tests cannot see, because the parser is behaving perfectly on the fixtures it was given.

The two checks fail in opposite situations. The parser test goes red when somebody breaks the repair logic; the validity rate drops when nobody touched the parser at all and the prompt or the model changed underneath it. Keeping only the first leaves a team blind to prompt regressions; keeping only the second tells them something is wrong across a whole subsystem without telling them what.

Answer quality carries two kinds of variance#

"The answers are good" resists this treatment because there is no artefact that can be frozen to make it repeat. The generator is stochastic, so the same question produces different text; the grader is a judgment, so the same text produces different scores. Both are true before anyone asks the harder question of which questions to ask, because the quality that matters is an average over a population of real user questions that can only ever be sampled.

The two sources are worth keeping apart rather than lumping together, because they are attacked differently and only one of them can be held still. Generator variance is removed by recording, exactly as it was for the parser tests: run the questions once, keep the answers on disk, score those. Grader variance survives that treatment, because the grader has to run again in order to produce a score at all. Every move available on the grader side reduces its noise rather than removing it.

The scale of the grader's contribution surprises people who have not measured it, and the way to measure it is to hold everything else still. Scoring the same 60 recorded answers twice with the same judge prompt and the same model, 4 of the 60 flipped between pass and fail on the second pass — a 6.7% disagreement rate with itself — and the aggregate score moved by 1.7 points. Nothing changed except the sampling inside the judge.

Recording the answers first is what licenses reading that movement as the grader's. The text being scored was byte for byte identical on both passes, the judge prompt was the same file, the judge model was the same model, so the movement has exactly one place left to have come from. Run the same experiment without freezing the answers and the identical result would have been equally consistent with a shaky generator, a shaky judge, or both, and it would have told nobody which one to attack.

Read that movement as a property of the instrument and it settles what the check can be used for. A regression smaller than a couple of points is invisible to a single run of this check no matter how carefully the generation side is controlled, which is a fact about the measuring instrument rather than about the assistant. The ways past that floor are purchases rather than fixes. Score each case several times and average, which cuts the judge's contribution and multiplies the bill by the number of draws. Enlarge the case set, which narrows the spread of the aggregate for the same reason a larger poll has a narrower margin, and costs in proportion. Or replace the judge with a deterministic rule wherever a rule can be written, which removes its variance completely for the narrow slice of quality a rule can express. Chapter 5, which sizes an eval set against the smallest regression worth catching, does the arithmetic on the first two. What matters here is that none of them is free, and that none of them is a matter of writing the judge prompt more carefully.

Writing the classification down, with the reason next to each row, is the artefact worth keeping in the repository:

Three of the five are infrastructure invariants, and that is the general shape. The deterministic pile is everything between the user's request and the provider's boundary, plus everything between the provider's response and your API's response. The eval pile is what happens inside the provider, measured through your prompt.

Where the index moves underneath the check#

Retrieval sits on the boundary and is worth placing explicitly, since most teams building over a knowledge base hit the same question. Whether the retriever returns kb_7741 in the top three for a fixed query against a fixed index is deterministic given a pinned index snapshot, so recall@3 — the fraction of labelled queries whose known-correct chunk appears somewhere in the top three results — over a hand-labelled query set is a test, as long as the index does not move underneath it.

It stops being one the moment the index is rebuilt nightly from live content, because then the same commit gives different recall on Tuesday and Thursday, and the check has become a measurement of a moving system rather than an assertion about code. The usual resolution is to run both: pin a frozen index snapshot for the fast job, and run recall against the live index on the same cadence as the rest of the eval pile. The same shape applies to anything else that is deterministic given a pinned artefact and sampled without one — a tool registry loaded from a live service, a feature flag read at request time, a prompt fetched from a remote store.

Stated generally, that is the whole classification in a line: a check is deterministic with respect to a boundary, and whatever sits outside the boundary is what makes it sampled. Rendering repeats because a fixture pins retrieval. The parser tests repeat because a file pins the completion. Recall repeats exactly as far as the index snapshot is pinned and no further. So the question to put to any check that looks ambiguous is not whether it is stochastic, but what it reads that nobody pinned, and whether that thing can be pinned without the check ceasing to measure what it was for. Pinning the index makes recall repeat and simultaneously stops it noticing that live retrieval got worse, which is why the resolution above is to run both versions rather than to pick one.

The plumbing is not the product#

The three tests in the fast job cover the plumbing exactly and cover the product not at all. Every one of them can be green while the assistant tells a Dutch customer their parcel arrives Thursday when the carrier has not scanned it since Rotterdam. That is the honest limitation of the deterministic pile, and it is why the eval pile is not optional. There is no way to reach answer quality with fixtures, and a team that responds by filling the fast job with more and more assertions is buying diminishing returns against a risk the assertions were never pointed at.

The classification is also silent about consequences. Five checks now sit in two piles, and the piles want different things. One wants to run in ninety seconds on every push and go red loudly on a commit that broke something. The other wants a stable sample, a budget, an owner, and a place where its number can stop a bad build from reaching customers without stopping an engineer from committing.

The value of doing this on paper is that it makes those needs visible before any job is written. Rendering, retries and parsing are pure functions of frozen inputs and one right output; a validity rate and an answer-quality score are estimates over a sample that move without a commit. The reason column is what keeps that true a year later, when somebody proposes moving the validity rate into the fast job because it is only one number, and the row says why it cannot go there and what would have to change for it to.

Deciding where each pile runs, and on what trigger, is what turns this classification into a delivery pipeline, and it is the next chapter's subject.