Asserting on a parsed field

An eval case holds three things together: an input, the assistant's response to that input, and a short list of claims about the response that a function can settle. Those claims are deterministic checks, also called code-based checks — ordinary code, a parse or an equality or a regular expression or a membership test in a set, which reads what the model produced and returns pass or fail without consulting a second model. The same stored response gets the same verdict this afternoon, tomorrow morning and in a year, which is what makes such a check usable as a gate on a merge. They carry weight because of the difficulty an earlier topic established: a model's output is a draw rather than a lookup, so a response cannot be compared byte for byte against a stored expected answer, and what remains is asserting properties that hold for every acceptable draw.

Deterministic checks assert the cheapest and most modest of those properties. They settle whether the response parses, whether the field the caller will read is present and correctly shaped, whether a banned phrase is absent, whether the agent invoked the right tool with the right arguments, and whether the case still costs what it cost last week. Whether the answer is good — helpful, faithful, appropriately hedged — is a separate question, handed by a later chapter to a grader that costs money per case and is capable of returning a different number on a second run.

This chapter is about the most basic of those properties — that a particular value is present, and correctly shaped — and about the two ways of asserting it that look interchangeable and are not. One reads the response as one long string and asks whether a shorter string occurs anywhere inside it. The other parses the response into a structure and asks about the value at one key. The chapter works through what each can be wrong about, what has to change upstream for the second to be available at all, and the mirror-image case of a check on content that must not appear, where the gap between characters and meaning is widest. Where the assistant's output is free prose with no structure to address, the weak check is the only one on offer, and the last section is about being precise rather than apologetic about that.

What each check can be wrong about#

Put a substring check beside its stronger sibling and the difference stops being about strictness. The second check parses the response into a structure and then asserts about one named location inside it:

import json, re

# Check A — characters somewhere in a blob
assert "order number" in answer.lower()

# Check B — a named field in a parsed structure
payload = json.loads(answer)
assert "order_number" in payload
assert payload["order_number"]
assert re.fullmatch(r"ORD-\d{6}", payload["order_number"])

Check A reads a flat sequence of characters and asks whether a shorter sequence occurs anywhere inside it. It has no notion of where the match landed, of who is speaking at that point in the text, or of whether the surrounding sentence asserts the thing or denies it. Every one of those is a way for A to be wrong while reporting success.

Why the last of the three is unreachable is worth working out rather than asserting, because it decides which checks are worth writing at all. English marks a denial with words placed somewhere else in the sentence — couldn't, never, failed to — and leaves the noun phrase being denied exactly as it stands in an assertion. The characters in here is the order number and in I couldn't find an order number are identical across the stretch the check looks at. What the check reports is one fact and only one: that a given run of characters begins at some position in the response. It never reads leftwards from that position, and there is nothing in the operation that could, because what it was handed is a sequence of characters rather than a sentence with a subject and a verb in it. The word that reverses the meaning therefore sits outside everything the check inspects, and the verdict comes back the same either way. The same argument settles the other two: who is speaking, and whether the text is a quote of something the customer sent, are both carried by structure a flat sequence does not record.

Here is the response that made the point on one support assistant, against a case whose job was to prove that a customer asking where their parcel is gets an answer containing the order number:

I'm sorry — I couldn't find an order number matching
lucia.f@example.com. Could you check the confirmation email
and send me the reference from the subject line?

The phrase is present, check A passes, and the response is the precise failure the case exists to catch. Check B reads a value at a key, so the match has an address; the assertion is about the field the downstream code will read, in the same place the downstream code will read it.

That does not make B correct, only wrong about different things. B can be wrong about truth: ORD-000000 satisfies the pattern, and so does the order number of a different customer. B is also brittle in a way A is not — the day someone renames the key to orderNumber, B fails on a response that is perfectly good, and A keeps passing. Both are worth knowing before you argue for one in review.

The last row is the honest reason teams drift towards A. A substring check never fails spuriously, so it never interrupts anyone, and an assertion that never interrupts anyone eventually stops meaning anything. Chapter 7, on freezing a failing run as a fixture, comes back to that with a way to tell the two apart.

The usual one-sentence justification for preferring B is that a regex is stricter than a substring, and it survives about ten seconds of pressure. A regex applied to the whole response is still a check on characters somewhere in a blob: re.search(r"ORD-\d{6}", answer) matches the apology above just as happily if the apology quotes a reference the customer supplied, and it would match a number appearing inside an example, a quoted previous message, or a URL. Strictness of the pattern is orthogonal to whether the match has an address. What changes the class of the check is parsing first, so that the pattern is applied to a value the response designates as the order number rather than to every position in the text at once.

Asserting at a named location#

The move that makes B possible is upstream of the assertion: the response has to have a location to name. For anything an eval suite grades, that usually means the model emits a JSON object with a defined shape and the human-readable sentence is one field inside it, rather than the sentence being the whole response with data sprinkled through it.

{
  "order_number": "ORD-448192",
  "status": "in_transit",
  "eta_date": "2026-08-14",
  "message": "Your order ORD-448192 is in transit and should arrive on 14 August."
}

With that shape, a case's assertions become a short list of independent claims, each of which fails with a distinct message:

import json, re
from datetime import date

def check_order_lookup(raw: str, expected_order: str) -> None:
    payload = json.loads(raw)                       # 1. parses at all

    for field in ("order_number", "status", "eta_date", "message"):
        assert field in payload, f"missing field: {field}"
        assert payload[field] not in (None, ""), f"empty field: {field}"

    assert re.fullmatch(r"ORD-\d{6}", payload["order_number"]), \
        f"malformed order number: {payload['order_number']!r}"
    assert payload["order_number"] == expected_order
    assert payload["status"] in {"in_transit", "delivered", "processing", "lost"}
    date.fromisoformat(payload["eta_date"])          # raises on 2026-13-45

Four kinds of claim are stacked there, and it is worth separating them because they fail for different reasons and belong to different owners. Parsing is a claim about the model's ability to close a brace, and it is the one that fails when a response is truncated. Presence and non-emptiness are claims about the contract — {"order_number": ""} is valid JSON with the right keys and is useless to the caller, which is why the empty check is separate from the presence check rather than folded into a truthiness test that would also reject the integer 0 elsewhere. Format is a claim about the shape of a value, and membership in {"in_transit", ...} is the same claim spelled as a set. Equality with expected_order is the only line that knows anything about one particular case.

Being exact about what those claims are evidence of matters more than the list of them, because the belief a reader is most likely to leave with — that parsing a field and asserting on it checks the answer — is the one that has to go. Follow a single order number through the machinery. The model generates a sequence of characters, and among them are the ones that spell order_number inside a pair of quotes, then a colon, then another quoted string. json.loads reads that sequence against JSON's grammar and hands back a mapping in which the quoted string sits at that key. The assertion reads the mapping at the key and tests the string against a pattern. Three steps, and the claim that survives all three is this: the model designated this value as the order number, and the value is shaped like one.

Neither half of that is a claim about the parcel. The designation is the model's own. Emitting "order_number": "ORD-448192" is the same act as writing the sentence your order number is ORD-448192, differing only in sitting somewhere code can find, so parsing recovers what the model asserted and never what is true. The pattern test is narrower still: it sorts every possible string into two piles, the ones matching ORD-\d{6} and the ones not, and a real order number and an invented one fall in the same pile. That is where meaning drops out of the chain. Nothing between the model's first token and the final assert consulted an order, so nothing anywhere in the chain could have known the difference.

The exception is one line, and it is worth counting the others against it. Of the claims in check_order_lookup — the parse, four presence checks, four non-emptiness checks, the format pattern, the status set, the date parse and the equality — exactly one compares the response against something the model did not produce, and it is payload["order_number"] == expected_order, where expected_order was carried in by the case. Every other line is a consistency check on the model's own output. Those are worth having: they catch a truncated response, a key the model stopped emitting, an empty field, a status the code downstream cannot switch on. What they cannot catch is a confident, well-formed, wrong answer, and how well a suite does against wrong answers is roughly how many of its lines reach outside the response for something to compare against.

Two details in the order-number regex are the ones that quietly go wrong. re.match(r"ORD-\d{6}", value) — the form most people write from memory — anchors only at the start, so ORD-4481929999 passes; re.fullmatch, or explicit ^...$, is what you meant. And in Python \d matches any Unicode decimal digit unless you pass re.ASCII, so ORD-٤٤٨١٩٢ in Arabic-Indic digits satisfies the pattern and then fails in your database lookup with a message that has nothing to do with digits. Neither of these will bite on the day you write the check. They bite eighteen months later on one case, and the debugging cost is disproportionate to how boring the cause is.

There is an over-correction on the far side of this, which is to compare the whole parsed object against a stored expected object. Exact-match golden comparison looks like the strongest possible assertion and behaves like the weakest, because a generated message field is different prose on every run and the case fails on wording nobody cares about. Within a week somebody has added a normaliser that strips the message field before comparing, and the golden file is now an object equality over the three fields that were worth asserting individually — with the difference that when it fails, the report is a diff of two dictionaries rather than a sentence naming what went wrong. Assert field by field, on the fields the case is about, and leave the free-text field to the graders of a later topic.

Fail messages deserve the same care. assert re.fullmatch(...) with no message produces AssertionError and a line number, and when a suite of 240 cases reports twelve of those you are opening twelve transcripts by hand. Including the offending value in the message — malformed order number: 'ORD-44819' — turns triage into reading the summary. That is not style; it is the difference between a failing suite you act on within the hour and one you postpone.

A forbidden string checks one word, not one promise#

The mirror image of a required-field check is a forbidden-content check, and it is where the gap between characters and meaning gets widest. Suppose legal states the assistant must never guarantee an outcome. The obvious implementation is one line, applied to every case:

assert "guarantee" not in answer.lower()

Run it across a 240-case suite and it passes on all 240. The number is real and the interpretation almost everyone puts on it is not. What has been measured is that a particular nine-letter sequence does not occur in 240 generated strings. What legal asked for was that the assistant never commits the company to an outcome, and these are only loosely related, because English has an unbounded number of ways to make a commitment:

You will definitely get your money back.
I can assure you the refund will be processed today.
100% you'll be refunded — no question.
We always approve claims like this one.

Every one of those passes the check. The check also has a failure mode in the other direction: "I can't guarantee a delivery date, only an estimate" is exactly the hedging legal wants and the assertion rejects it, so the first person to hit that case adds "can't guarantee" to an allowlist, and the allowlist grows until nobody can say what the check enforces.

One property sits underneath both failure directions, and naming it says exactly when a check of this kind is the right instrument. A substring test is a test on a string, and it is a faithful test of a requirement only when the requirement is itself about a string — when the set of character sequences that violate the rule is small, known, and writable down in advance. A competitor's trademark has that property. There is one spelling and perhaps a handful of variants, enumerating them is a morning's work, and the work stays done. A commitment does not have it. Committing is something a sentence does rather than a sequence a sentence contains, and the set of English sentences that do it has no boundary anybody can write out, which is why the four lines above are a sample rather than a list. The check is not weak because it is simple. It is weak because the requirement it was aimed at is not a property of characters at all, and adding literals to it never converts the one into the other.

The right way to report the green result is therefore to state its scope with the number: the literal token "guarantee" is absent from all 240 responses; this is not evidence that the assistant made no guarantees. Written that way it is still useful. A forbidden-string check is a cheap, deterministic, auditable gate on specific banned literals — a competitor's trademark, an internal codename, a deprecated support email address, a discount code that must never be uttered. Those are jobs where the string genuinely is the thing being forbidden, and where a stochastic grader would be a worse tool because you need the same verdict on every run and a log you can show an auditor. Policy adherence in general is not that job, and pretending a substring check covers it is how a suite acquires the appearance of a control.

Where the weak check is the only one available#

None of this makes the substring check obsolete, because parsing requires something to parse. When the assistant's output is free prose delivered straight to a human — a chat reply streamed into a widget, an email body, a summary paragraph on a dashboard — there is no schema, no key, and no location to name. The response is the blob, and the only assertions available are about characters in it.

That situation is more common than the structured case in exactly the places that matter most, and it has an escape route: change the contract rather than the check. If a field of the answer needs grading, ask the model to emit it as a field. The customer-facing text can stay in message while order_number, status and eta_date sit beside it, and your renderer shows only message. Doing that is usually a smaller change than the eval suite it unlocks.

Where you genuinely cannot change the contract — a third-party endpoint, a legacy path, a model whose output you do not control — keep the substring checks and be precise about what they are. Anchor them where you can: answer.strip().startswith("ORD-") is a claim about position, which is more than in gives you. Count occurrences when the count matters. Write the negative cases too, so that a check which cannot be strong is at least known to be weak by everyone reading the report.

An assertion earns its place by being able to reject something, and where it lands decides what it can reject. A check on characters somewhere in a response can be defeated by a sentence that mentions the thing while denying it; a check on a value at a key cannot, because the response has told you which value is the order number. Neither is a claim about whether the answer was any good, and neither is a claim that the value is true — ORD-000000 and a different customer's order both pass a format check. What parsing buys is that the assertion and the downstream code are looking at the same place, which is the minimum condition for a suite to fail when the product is wrong.

There is a third possibility this chapter has been quietly assuming away. If the model reliably emitted well-formed JSON matching the schema, the parse step would never fail and the presence checks would be dead weight — and providers now offer exactly that guarantee, enforced during decoding. That is the argument you will meet in review the week after you write these assertions, phrased as a proposal to delete half of them.