Testing a function that calls a model

A function that calls a large language model, usually written LLM, breaks the assumption every other unit test in a backend codebase rests on. An LLM is a service, running on somebody else's hardware, that takes a block of text and returns text it generates one word-fragment at a time, sampling each fragment from a probability distribution instead of looking it up. Your code reaches it over HTTP through a provider — Anthropic, OpenAI, Google, or a model your own company hosts — and the request is billed by the token, the sub-word chunk that models count text in, at a latency of roughly one to three seconds for a short completion. The same request sent twice can come back as two different strings, both of them correct. Same input, same output, free and offline: all four properties stop holding at the moment a function calls a model, and this topic is about which of them can be recovered and how.

The recovery starts by noticing that such a function is doing two separable jobs. The deterministic job is code you wrote: how the request is assembled, how the response is parsed, which branch runs when the answer comes back truncated, what happens when parsing fails, what a retry wrapper does with a rate-limit error. The other job belongs to the model, and the question about it — is this summary accurate, is this classification right — is answered by an evaluation, a scored measurement over a dataset of cases, compared against a baseline. This chapter is about testing the first job honestly and moving the second somewhere it can be measured properly. The mechanisms that make the rest of the topic possible follow from it: injecting a stand-in for the provider, replaying recorded responses, pinning the prompt string, and keeping the expensive measurements off the merge path.

What a substring assertion is measuring#

Take a test that reads an API key from a continuous-integration secret store, sends a fixed support-ticket transcript to a provider, and checks that the returned summary contains the word refund. When it fails, it fails like this:

E   AssertionError: assert 'refund' in summary
E    +  where summary = 'The customer contacted support twice about a charge
E       they did not recognise, and asked to have the money returned to the
E       card it was taken from.'

Nothing is broken. The summary is accurate, it is arguably better than the one the test was written against, and the ticket it describes is unmistakably about giving a customer their money back. The assertion failed because the model chose the phrase "have the money returned" instead of the word "refund", which is the kind of choice a model makes differently on different days and which no part of the application actually depends on.

Three separate sources of variation feed into whether that check passes, and only one of them is under the repository's control. Sampling is the obvious one: the same request sent twice can produce different token sequences, and pinning temperature to zero narrows the spread without eliminating it. Model versions are the second, since a stable model alias points at a build that the provider updates, so a request that produced "refund" in March can produce a paraphrase in June with no change on your side at all. The third is the one people forget, which is that the assertion was never precise about what it wanted. "Contains the string refund" is a proxy the author reached for because the real property — that the summary conveys the customer wants their money back — is not expressible as a substring check.

The first of those three deserves taking apart, because "narrows the spread without eliminating it" invites an obvious reply — pin the temperature to zero and be done — and a team that takes that reply will write the same substring assertion again with a configuration flag in front of it. Temperature zero changes the picking step at the end of generation: instead of drawing a candidate at random in proportion to its score, the code takes the highest-scoring candidate outright. That step is genuinely deterministic. Given one fixed column of scores it makes one fixed choice, every time, forever.

What it cannot do is fix the column of scores. Those are computed on the provider's hardware, where your request is batched together with whatever other requests arrived in the same window, and the arithmetic producing them is floating-point addition over very large sums. Floating-point addition is not associative: adding the same collection of numbers in a different grouping can change the last bits of the total, because each intermediate result is rounded to a fixed width before the next one is added to it. A different batch means a different grouping, so the same request can produce scores that differ in their final bits between one call and the next. Almost always that is invisible, because the leading candidate leads by far more than the wobble. It becomes visible at a near-tie, where two candidates sit closer together than the last bits do, and there the ranking flips and a different word comes out. Temperature zero therefore removes one source of divergence and leaves the test depending on the model never being nearly undecided about its wording — which is the exact situation that "have the money returned" and "refund" describe.

Each of the three has a tempting local fix, and each fix makes the test worse in a way that is easy to miss. Loosening the assertion to any(w in summary for w in ('refund', 'money back', 'reimburse', 'chargeback')) reduces the failure rate and simultaneously means the test now passes on a summary saying the customer was denied a refund. Adding retries around the call converts a signal into silence, since a genuine regression that happens to produce the right word on the second attempt goes green. Pinning to a dated model snapshot buys real stability and postpones the problem to the day the snapshot is retired, which providers do on a published schedule.

Underneath all three is a measurement problem. A test that fails one run in ten is telling you it disagrees with the model about wording roughly ten percent of the time; it says nothing about whether the other ninety percent were summaries you would have shipped. Its green runs get read as evidence of a quality nobody has checked, which is why deleting it eventually feels harmless. A test in that state has been performing reassurance rather than verification.

Asserting on the code you wrote#

The function under test does a great deal that has nothing to do with model prose. Here is a representative version, stripped of logging:

JSON_BLOCK = re.compile(r"```json\s*(\{.*?\})\s*```", re.DOTALL)

def summarize(client: LLMClient, transcript: str) -> Summary:
    resp = client.complete(
        model="claude-sonnet-4-5",
        messages=[{"role": "user", "content": render_summary_prompt(transcript)}],
        max_tokens=512,
    )

    if resp.finish_reason == "length":
        raise SummaryTruncated(tokens=resp.usage.output_tokens)

    match = JSON_BLOCK.search(resp.text)
    payload = json.loads(match.group(1)) if match else json.loads(resp.text)

    return Summary(
        text=payload["summary"],
        intent=payload.get("intent", "unknown"),
        needs_human=payload.get("confidence", 1.0) < 0.6,
    )

There are at least six behaviours in those fifteen lines, and every one of them is a deterministic function of the response object. The truncation branch fires on finish_reason == "length". The regex handles the case where the model wraps its JSON in a fenced block, which some prompts provoke and others do not. The bare json.loads handles the unfenced case. A missing intent key defaults rather than raising. A confidence below 0.6 routes the ticket to a human, and 0.6 exactly does not. A malformed payload raises JSONDecodeError and whatever your error handling from topic 2.11 does with it.

"A deterministic function of the response object" is the load-bearing claim in that paragraph, and it is worth walking rather than accepting, because the same walk shows where it stops holding. The function's only contact with anything outside itself is the call to client.complete. Every line below that call reads a field of the object the call returned — finish_reason, then text, then usage.output_tokens — and reads nothing else. No clock, no random draw, no environment variable, no second request. Because those fields are the only inputs the lower half of the function has, they fully determine which branch it takes and what it returns. Because they fully determine it, any object carrying the same fields drives the code down the same path, whatever produced that object. And because the provider was the only non-deterministic thing in the function, replacing the object it would have returned replaces the whole of the non-determinism, leaving ordinary code that ordinary tests can pin down.

The condition sitting inside that chain is the phrase "reads nothing else", and it is worth checking rather than assuming, because it fails quietly. A version of summarize() that stamped datetime.now() onto its result, or that chose one of three prompt variants at random, or that read a feature flag from an environment variable, would still be a deterministic function of its inputs — it would simply have inputs that the test cannot see or set. The remedy is the same one this chapter applies to the client: arranging for the hidden input to arrive as a parameter. Which inputs are hidden in a given function is the first thing to work out before writing a test for it, and the answer is usually one or two more than expected.

None of the six behaviours needs a network, then. Given a response object the outcome is fixed, and a test that supplies the response object directly through a fake — a small stand-in class written by hand, satisfying the same interface as the real client and returning whatever response the test hands it — runs in under a millisecond, costs nothing, and fails only when the code changes:

def test_confidence_below_threshold_routes_to_human():
    client = FakeLLMClient(response=Response(
        text='{"summary": "Customer wants a refund.", "intent": "refund",'
             ' "confidence": 0.41}',
        finish_reason="stop",
        usage=Usage(input_tokens=812, output_tokens=44),
    ))

    result = summarize(client, transcript="...")

    assert result.needs_human is True
    assert result.intent == "refund"

def test_truncated_response_raises_rather_than_parsing_garbage():
    client = FakeLLMClient(response=Response(
        text='{"summary": "Customer wants a ref',
        finish_reason="length",
        usage=Usage(input_tokens=812, output_tokens=512),
    ))

    with pytest.raises(SummaryTruncated) as exc:
        summarize(client, transcript="...")
    assert exc.value.tokens == 512

That truncation test is one a live suite has no practical way to write. Provoking a genuine truncation from a provider means finding an input long enough to exhaust max_tokens, which changes every time the model's verbosity shifts, so in practice nobody does it and the truncation branch ships unexercised. It is also exactly the branch that produces a JSONDecodeError in production on the day somebody lowers max_tokens from 512 to 256 to save money.

A fake is also the only practical way to assert on the request rather than the response. A live test can see what came back and has no visibility into what went out, so a change that silently dropped max_tokens or switched the model identifier would pass, and would show up as a cost anomaly a week later. A fake that records its arguments turns that into an ordinary assertion:

def test_summarize_requests_a_bounded_completion():
    client = FakeLLMClient(response=ok_response())

    summarize(client, transcript="...")

    call = client.calls[-1]
    assert call.model == "claude-sonnet-4-5"
    assert call.max_tokens == 512
    assert len(call.messages) == 1

That is a weak test in isolation and a useful one in a codebase where model identifiers get changed by find-and-replace. It is also the first hint of a stronger technique, since the interesting thing about the outgoing request is not max_tokens but the several thousand characters of prompt the model receives, and asserting on those needs a different mechanism than an equality check on a field.

Two questions, two places to answer them#

The distinction that keeps this straight is between whether your code handles a response correctly and whether the response was any good. Both are worth knowing, and they behave nothing alike as measurements.

The row about provider updates changes how you read a green pipeline. A correctness suite built on fixed response objects will stay green through a model deprecation, a pricing change, a new refusal format and an outage, because none of those alter the code under test. That is the correct behaviour for a merge gate, which exists to answer whether this diff is safe to land, and it means a green run is not a claim that the system works today. Something else has to make that claim, and it runs on a schedule rather than on a branch.

The belief that arrangement has to displace is that a passing suite means the prompt is right. It is tempting because everywhere else in the codebase it is very nearly true: a suite that covers a module's behaviour does go red when that behaviour breaks, so green really does carry information about correctness, and engineers have years of evidence for that reading. What breaks the inference at a model call is that the inputs to these tests are fixtures somebody wrote. A prompt that has drifted into asking for the wrong thing changes what the provider sends back, and changes nothing whatsoever about what a fake sends back, because the fake was never shown the prompt. The suite is green on the day the prompt is exactly right and green on the day half of it is deleted by a bad merge, and from outside the two runs are indistinguishable. That is not a defect to be fixed by writing more tests of the same kind, since no arrangement of hand-written response objects can see it. The instruments that can see it are what the rest of this topic is for.

An evaluation is a measurement with a confidence interval, so its natural verdict is "3 points below last week's baseline, worth a look" rather than "fail". Putting a measurement like that on a merge gate produces either a threshold so loose it never fires or one that blocks pull requests for statistical noise, and the team learns to override it. Where evaluations run and what gates on them is the subject of the last chapter of this topic; the load-bearing point here is that moving the quality question out of the unit suite is not a decision to stop caring about quality.

The failure modes a fake will not show you#

A test that hands summarize a hand-written Response object has strong opinions about your code and no opinions at all about the provider. If the API starts returning finish_reason: "max_tokens" instead of "length", every test above stays green and production starts parsing truncated JSON. If a field you never populate in your fake — a stop_sequence, a refusal flag, a content block of a type you do not handle — starts appearing, the fake will not tell you. The fake's response shape is a claim you are making about the wire format, and nothing in the test verifies that claim.

That gap has a specific remedy, which is recording a response that genuinely came off the wire and replaying it, with all the aging problems that implies. There is also a category of behaviour that no response object can express, such as a connection reset halfway through a stream or a rate-limit response carrying a Retry-After header, and those get their own treatment later.

It is worth being blunt about how far the fake gets you, because "we replaced the live call with a fake" is sometimes reported as though the testing problem were solved. What the two tests above establish is that given a response of a shape you believe in, your parsing and branching behave as specified. Whether that shape is real, whether the prompt that produced it is the prompt you think you are sending, whether the wrapper around the call retries the right failures, and whether the tool schema you advertise matches the arguments you parse are four separate questions, each with a different mechanism behind it. They are the next four chapters.

Where there is nowhere to put a fake#

Before any of that, there is a mechanical obstacle. The version of summarize above takes client as a parameter, which is what made both tests possible, and the version most codebases actually have does not:

def summarize(transcript: str) -> Summary:
    client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
    resp = client.messages.create(...)

There is nowhere to put a fake in that function. The client is constructed inside it, from a name the caller cannot reach and a key read straight from the process environment, so there is no argument a test could pass and no attribute it could set that would change which object client ends up bound to.

You can reach for unittest.mock.patch and monkeypatch the constructor, which works, and the cost of it follows from how that tool identifies what to replace. patch takes a dotted string naming a module attribute and swaps the object that attribute points at. The string is a path through the provider SDK's module layout, and no SDK promises that its module layout is stable — the promise is about the classes and methods you call, not about which file they live in. So a version bump that moves a class between modules, or that inserts a wrapper around the constructor, invalidates the string. The test then fails at patch time with an attribute error, in a test module that never called the moved class, for a reason that has nothing to do with the code under test. That is how an SDK upgrade comes to break tests in modules with no business knowing the SDK exists.

The split this chapter argues for is cheap to state and costs one structural change to obtain. Assert on your own parsing, branching and request-building by supplying the response, where the outcome is fixed and a failure means somebody edited the code; measure the model's output on a dataset, on a schedule, where a two-point move is a conversation rather than a red build. A suite arranged that way runs in milliseconds, costs nothing, needs no key, and keeps its meaning for years, which is precisely what the substring assertion could not do for a single quarter.

The structural change is substitution, and substitution needs somewhere to substitute. Arranging that — for the client, for the clock, and for the prompt string itself — is what the next chapter is about, because nothing else in this topic can be written until it is done.