Testing code that calls an LLM

Read these first

• Error handling

What you will know

• Write a unit test for LLM-calling code that runs offline and free

  Tests that call a real provider are slow, flaky, and expensive — so they get skipped, and then deleted.

• Record a real provider response and replay it as a fixture

  The captured artifact should preserve the request conditions that affect branching, while redacting secrets and accepting that it will age as the API changes.

• Snapshot-test the rendered prompt string so template changes are visible in review

  A small interpolation or ordering change can alter model behavior without changing application code, so reviewers need a concrete diff.

• Test that the retry and timeout wrappers actually retry and time out

  Test that the wrapper retries on 429 and gives up on 400, using a fake that fails on demand.

• Contract-test tool schemas so incompatible changes fail before an LLM call

  Schema drift can turn a valid model decision into an unusable tool invocation, and provider validation may report the failure only after deployment.

Chapters

1. 1

   Testing a function that calls a model

   A function that calls a language model does two separable things: your own assembling, parsing and branching, which is deterministic, and the model's wording, which is not. This chapter shows what happens when a test asserts on the second — a substring check that fails one push in ten and gets deleted within a quarter — and how to test the first instead, by handing the function a response object you wrote and asserting on the parse, the truncation branch and the outgoing request. The quality question moves to a scored evaluation that runs on a schedule.

   Read chapter
2. 2

   Seams, ports and the composition root

   Every technique in this topic starts by substituting something — a fake client, a replayed cassette, a clock you control — and none of them are available to code that constructs the provider client at the call site and reads the key from the environment while it does. This chapter defines the seam, weighs patching against a narrow port your own code owns, and moves construction to a composition root, so the client, the clock and the rendered prompt all become things a test can supply.

   Read chapter
3. 3

   Snapshotting the rendered prompt

   The string a model receives is the return value of a rendering function with six or seven inputs, and the template file is only one of them — a shared helper, a separator or a truncation limit changes the wire string while every file under prompts/ stays untouched. This chapter stores the rendered output in the repository so the change appears in the diff, states precisely what that assertion does and does not claim, adds a token-count bound beside it, and shows how the technique degrades when snapshots get large.

   Read chapter
4. 4

   Fakes, cassettes and prompt snapshots

   A hand-written fake exercises your branching, a recorded cassette pins the wire shape you parse, and a stored rendered prompt pins what you send — three artifacts routinely called mocks in review, protecting three different things. This chapter separates their guarantees, works through what a recording must preserve and what to redact, covers the replay mechanics that decide whether a miss fails loudly or silently reaches the network, and shows why the cassette is the only one of the three that can go stale while staying green.

   Read chapter
5. 5

   Testing a retry wrapper

   Drive a fake that returns 429 twice then 200, and another that returns 400, and assert the call counts; drive time through an injected clock so the file runs in milliseconds while executing the delays you actually ship. Shrinking the backoff in test configuration makes the file fast and deletes the only case worth having — the one where an overall budget cuts a retry short — and this chapter writes that case, tabulates the retry policy by status code, and names the failures a fake can never produce.

   Read chapter
6. 6

   Contract tests for a tool schema

   A hand-written tool schema in the prompt module and a handler type declared elsewhere diverge the day someone adds a required field, and the failure lands after every metric anyone thought to watch. This chapter writes the contract test that catches it in both directions before a model is called, then shows what generating the schema from the handler's annotations does and does not close — a tightened constraint the model cannot satisfy, prompt examples showing the old shape, and tool calls already stored on disk.

   Read chapter
7. 7

   Merge gates and scheduled suites

   Tests that touch a provider cost minutes and dollars per run and go red for reasons the author did not cause, which is why a pull-request pipeline built around them gets ignored and then gutted. This chapter sorts the tests by what a failure means and who has to react — offline gate, nightly provider contact, evaluation, production canary — enforces the split with a missing credential and a socket ban rather than discipline, and gives the suites that cannot block a way to be heard.

   Read chapter