Deterministic and code-based checks

Read these first

• Why traditional testing fails

What you will know

• Assert on schema validity, required fields, and forbidden strings

  Malformed or disallowed output should fail before downstream code turns it into a misleading success.

• Check that the right tool was called with the right arguments

  A plausible response can still cause the wrong side effect when tool identity or serialized parameters drift.

• Add latency and cost assertions alongside correctness ones

  A response can remain correct while exceeding an API budget or making a user-visible path too slow.

• Trace a failed evaluation to the deterministic assertion that rejected it before invoking an LLM judge

  Running cheap gates first preserves the concrete failure signal and avoids paying for a judge to rediscover it indirectly.

• Catch a real regression with a check that costs nothing to run

  A fixture that once passed and now fails proves the assertion protects behavior rather than merely exercising code.

Chapters

1. 1

   Asserting on a parsed field

   A substring check cannot tell an answer stating the order number from an answer quoting it inside an apology for not finding one. This chapter moves the assertion to a named location in a parsed structure — field present, non-empty, matching a format — and sets out the limits of forbidden-string checks, where the absence of \"guarantee\" says nothing about \"you will definitely get your money back\".

   Read chapter
2. 2

   The constrained decoding guarantee

   When a provider constrains generation to a JSON schema, the schema assertions in an eval suite start to look like dead code. The guarantee delivers types, required keys and enum membership on one path; it does not deliver that order_id names a real order, that refund_cents is inside policy, or that citations is non-empty, and it does not cover fallback models, retries, replayed fixtures or your own post-processing.

   Read chapter
3. 3

   The run record

   Almost every assertion worth writing about an agent reads something an eval loop does not keep by default. Record the whole run instead: each message, each tool call with the arguments exactly as the model emitted them — before any handler coerced them — each tool result, the usage block the API returns, and the per-call timings. That capture is what lets you assert amount_cents was the integer 4200 rather than trusting a handler that swallowed 42 without complaint.

   Read chapter
4. 4

   Tool-call assertions

   An agent that says \"I've refunded $42.00\" while passing amount_cents=42 satisfies every check written against its answer text. This chapter writes the assertion that reads the call instead — exactly one side-effecting call, named issue_refund, order_id matching the case, amount_cents equal to the integer 4200 — and covers the data format and startup contract check that keep two hundred such cases honest while the tool signature moves.

   Read chapter
5. 5

   Per-case cost ceilings

   Six added few-shot examples can more than double the unit cost while every correctness check stays green. This chapter reads prompt and completion tokens from the usage figures the API returns, computes cost per case, and fails against a budget stored beside it — then deals with the harder half, which is governance: a ceiling anchored to today's average with no owner gets raised on the first legitimate improvement and then protects nothing.

   Read chapter
6. 6

   Latency assertions

   Fifty cases fired concurrently at a provider from one CI runner produce timing failures on Tuesday afternoon and none on Wednesday morning, with no prompt change in between. This chapter takes a wall-clock measurement apart into the components you control and the ones you do not, caps parallelism for timing cases, excludes rate-limit sleeps, and moves most of the value onto token counts, which cannot be contaminated by your own concurrency.

   Read chapter
7. 7

   Regression fixtures from an incident

   An assertion that has never rejected anything is exercising code rather than protecting behaviour, and nothing in a green suite distinguishes the two. Freeze a real failing run as a fixture, confirm the new check rejects it for the stated reason before the fix lands and accepts the post-fix run after, and manufacture the missing cases by mutating a good record where no outage exists to inherit one from.

   Read chapter
8. 8

   Deterministic gates and an LLM judge

   Thirty percent of cases scoring 2 out of 5 on helpfulness sent one team chasing prompt quality for two days; the outputs were truncated mid-JSON. Ordering deterministic gates ahead of any judge turns that result into \"invalid JSON: unterminated object\" and costs nothing. The chapter sets out which properties a judge cannot see at all, and gives the maintenance argument for keeping both a straight answer.

   Read chapter