Contract tests for a tool schema

A JSON Schema for a tool is the machine-readable list of its parameters, their types and which of them are required, sent alongside the prompt so the model knows how the tool may be called. The handler that receives the resulting tool call — the model's structured request that your code run a named function with the arguments it supplies — has its own description of the same parameters, in whatever type system your language provides. Those are two descriptions of one contract, and unless something forces them to agree they will not.

A contract test checks that two independently maintained descriptions of the same interface still agree. This chapter writes one for a tool schema, which takes about fifteen lines and no network, then examines what happens when a team removes the duplication entirely by generating the schema from the handler's annotations. Generation is the right move and it closes exactly one of four ways these definitions can disagree; the other three are comparisons against something outside the generator — a body of arguments observed in production, the examples in the rendered prompt, and the tool calls already written to a database by conversations that happened last week.

Two definitions, one fixture#

The check that catches drift compares the two definitions against each other rather than each against itself, and the cheapest way to do that is a set of example argument objects that both sides must accept:

VALID_ARGS = [
    {"order_id": "A-9931", "amount_cents": 4200, "reason": "duplicate_charge"},
    {"order_id": "B-0001", "amount_cents": 1, "reason": "goodwill"},
]

def test_published_schema_accepts_what_the_handler_accepts():
    schema = CREATE_REFUND_TOOL["input_schema"]
    for args in VALID_ARGS:
        jsonschema.validate(args, schema)      # the model is allowed to send this
        RefundArgs(**args)                     # and the handler can consume it

def test_handler_requires_nothing_the_schema_omits():
    required = set(CREATE_REFUND_TOOL["input_schema"].get("required", []))
    handler_required = {
        f.name for f in fields(RefundArgs)
        if f.default is MISSING and f.default_factory is MISSING
    }
    assert handler_required <= required

The second test is the one that fires the moment somebody adds a required field to RefundArgs and not to the published schema. The new field lands in handler_required, the schema's required list still contains the old two names, and the subset assertion fails with both sets printed. No model is called, nothing is billed, and the failure names the exact field.

The direction of that subset is the part to get right, and it is the part people reverse. handler_required <= required says every field the handler insists on must also be a field the schema insists on. It has to run that way round because the two sets play different roles: the schema is a promise made to the model about what it will be asked for, and the handler is a demand made of whatever arrives. Anything the handler demands and the schema does not mention is a field the model has no reason to send and will not send, so the demand can never be met — which is the outage. The opposite case is not symmetrical. A field the schema requires and the handler does not is merely a value the model is compelled to produce and the code then ignores, which costs tokens and no correctness, and that is why the assertion is a subset rather than an equality. Writing == there looks tidier and turns an ordinary wasteful state into a build failure.

The reverse direction matters too and is easier to forget. A schema that marks a field required while the handler treats it as optional is not an outage, and it does constrain the model unnecessarily; more usefully, a schema advertising a field the handler ignores entirely means the model is spending tokens producing a value that goes nowhere, which is the kind of thing that survives for a year because nothing breaks.

The duplication that makes this necessary usually exists for an ordinary reason. A tool schema often lives in the prompt module because its description fields are prompt engineering — they are read by the model, they get reworded when the agent picks the wrong tool, and the person doing that rewording is thinking about model behaviour rather than about the handler. Keeping them next to the prompt is a deliberate choice with good reasons behind it.

Generation closes exactly one of these gaps#

The obvious conclusion is that two hand-maintained definitions are the problem, and that is right. A decorator deriving the schema from the handler's annotations makes structural disagreement impossible:

@tool(description="Refund an order to the original payment method.")
def create_refund(order_id: str, amount_cents: int, reason: RefundReason) -> RefundResult:
    ...

# create_refund.schema is now derived, and cannot disagree with the signature

Generation moves the prompt-engineering problem rather than solving it. The description string and the per-parameter descriptions are now attached to the handler, so tuning the wording that makes the agent choose create_refund over escalate_to_human means editing a file full of business logic, and those edits will be reviewed by people looking for logic bugs. Teams handle this by keeping the descriptions in a separate mapping the decorator reads, or by accepting the trade and putting the tool descriptions in the rendered-prompt snapshot from chapter 3 so a reviewer still sees the wording change as text. What does not work is treating the description as documentation, because it is an input to the model with measurable effects on tool selection.

With nine tools defined this way, the class of bug that adds a required field to one side only is gone from the codebase, and a test asserting that the generated schema matches the annotations really is a tautology — it tests the generator, which is a library concern rather than yours. A lead who concludes that contract tests are now unnecessary is right about structural drift and wrong about the category, because "schema" and "handler type" are only two of the four things that have to agree.

The last four rows all have the same structure: they compare the generated schema against something the generator has never seen.

A perfectly generated schema that breaks production#

Take the tightened constraint first, because it is the one that feels safest. Somebody notices that refunds above £500 should not be automatic and expresses that in the type:

AmountCents = Annotated[int, Field(ge=1, le=50_000)]

@tool(description="Refund an order to the original payment method.")
def create_refund(order_id: str, amount_cents: AmountCents, reason: RefundReason) -> RefundResult:
    ...

The generated schema now carries "maximum": 50000. The change is intentional, the generation is correct, the handler and schema agree completely, and production breaks for a specific slice of traffic: orders over £500 that previously produced a tool call the handler could reject with a clear message now produce either a tool call the provider refuses to accept, or — depending on how strictly the provider enforces input_schema — a call the model declines to make at all, so the agent responds with prose apologising that it cannot help. The failure mode moved from a structured rejection to an unstructured one, and the code path that used to handle "refund too large" is now unreachable.

Why that move is a loss rather than an equivalent outcome is worth being exact about, since in both worlds the customer does not get an automatic refund. In the old world the oversized amount arrived as a tool call, which is a value in your process. The handler could compare it against the threshold, return a specific error, route the case to a human, count it in a metric named after it, and phrase the message to the customer. Every one of those is a branch, and every branch exists because there was something to branch on. In the new world the amount never becomes a value at all: either the provider refuses the call before it is emitted, or the model produces prose instead, and prose is the one output shape your tool-handling code has no case for. The constraint did not change what is permitted. It changed which side of the boundary the decision is made on, and decisions made on the far side of that boundary come back as text rather than as data.

The same shape appears with narrowed enums and patterns. Changing order_id: str to Annotated[str, Field(pattern=r"^A-\d{4}$")] is a reasonable tightening on the day every order identifier starts with A-, and it silently excludes the legacy B- orders that make up 3% of the estate.

What catches this is a body of observed arguments rather than a type. Keep a file of tool-call arguments seen in production — a few hundred rows sampled from logs, with identifiers scrubbed — and validate every one against the current generated schema in the test suite:

def test_historical_tool_calls_still_validate():
    failures = []
    for row in load_jsonl("fixtures/observed_tool_calls.jsonl"):
        schema = TOOLS[row["name"]].schema
        try:
            jsonschema.validate(row["input"], schema)
        except jsonschema.ValidationError as e:
            failures.append((row["id"], e.message))
    assert not failures, failures[:5]

That test fails on the maximum change with a list of real refunds that would no longer be expressible, which turns an abstract argument in review into five concrete order identifiers. It is the same discipline as backward-compatibility checking for an Avro or Protobuf schema, where adding a required field is a breaking change because old writers exist. Here the old writers are the model's previous behaviour and the conversation histories on disk.

The prompt's examples are part of the contract#

Few-shot examples showing a tool call are copies of the schema written in prose, and they drift like any copy. An example demonstrating {"order_id": "A-9931", "amount_cents": 4200} after a reason field became required teaches the model to omit the field, which the model will do reliably, because an example outranks a schema description in practice. The check is mechanical:

def test_prompt_examples_validate_against_current_schemas():
    for example in extract_tool_examples(render_answer_prompt(**STANDARD_INPUTS)):
        jsonschema.validate(example.arguments, TOOLS[example.name].schema)

"An example outranks a schema description" is the sort of claim that should not be left as folklore, and the reason it holds follows from what the model is doing. Generation extends the text it has been given, one fragment at a time, with each fragment chosen on the evidence of everything above it. A schema is a description of a constraint written in a different form from the thing being produced: it says a field named reason is required. An example is the thing being produced, already produced once — a tool call, laid out exactly as the next tool call will be laid out, in the same conversation, a few hundred tokens earlier. Continuing text that matches the shape of nearby text is the operation the model performs best, and nothing in the loop weighs a stated rule against a demonstrated pattern and prefers the rule. The example is not competing with the schema on authority. It is competing on similarity to the thing being written, and it wins that comparison every time.

This runs on the rendered prompt rather than the template, for the reason established in chapter 3: the examples that reach the model may come from a YAML file, a database of curated cases, or a retrieval step, and only the rendered string contains what was actually sent.

Stored arguments are the third comparison. An agent that persists conversation state writes tool_use blocks to a database, and resuming a conversation replays them into the handler. A schema change is a migration on that data whether or not anyone treats it as one, so the validation above should run against a sample of persisted rows and not only against logged traffic. The practical consequence is that adding a required field to a tool is a two-step change in any system with persisted conversations or cached prompts, exactly as adding a non-null column to a table is: ship the field as optional with a default, let the model start supplying it, confirm from the observed arguments that it does, and only then tighten.

Grammar is all a schema describes#

None of this says anything about whether the model will choose the right tool, fill in a sensible amount, or decline when it should. A schema is a grammar, and every test in this chapter checks that two pieces of code agree about the grammar. Whether the sentences are any good is the evaluation question again.

It is also worth separating these tests from the provider's own validation, which happens when you send the tools and is real but late. Providers accept a subset of JSON Schema — support for $ref, oneOf, format and additionalProperties varies, and a construct outside the subset is sometimes rejected with a 400 and sometimes accepted and ignored, which is worse. A generated schema from a rich type system will happily emit $ref for a nested model, so a check that walks the generated schema and asserts it uses only the keywords your provider documents is worth the twenty lines, and it runs offline. What that buys is a verifiable statement made at build time about a property you would otherwise learn from a 400 in an environment somebody was deploying to.

That keyword check is a static approximation of a subset that changes without much announcement. The complement is a smoke test that sends the tool definitions to the real endpoint with a one-token completion request and asserts a 200, which costs a fraction of a cent and answers the question definitively for today's API. That test makes a network call, so it lives somewhere other than the fast suite.

The contract to defend, then, is wider than the pair of definitions people usually mean by the word. Generation makes the schema and the handler one artifact, which is worth doing and is the smaller half. The larger half is that a schema is also a promise to arguments the model produced yesterday, to examples sitting inside the rendered prompt, and to tool_use blocks already written to disk — and each of those needs a comparison against a sample of the real thing, all of which run offline in milliseconds.

Every test in this chapter, and every test in the four before it, runs without a network, without a key and in milliseconds. That was the point of the exercise, and it leaves an unanswered question about the other tests — the ones that touch a provider, cost money and take minutes, along with the evaluations that take longer still. Deciding which of those may block a merge is the last thing standing between this suite and a build a team keeps.