Reproducing a hallucinated argument

The first question anybody asks about an invented tool argument is how often it happens, and the usual next move — grep the last thirty days of logs — produces nothing, because there is nothing to find. A refunds service that accepts any string for reason_code and maps unrecognised values onto OTHER writes its normal audit row and returns 200. Application logs record that refund_order was called and that it succeeded. The evidence of the failure was destroyed at the moment the failure occurred, so there is no stack trace to start from, which is where a backend engineer normally begins.

So the reproduction is constructed rather than found. That has two consequences worth stating before any code appears. The conversations have to be written to create the condition under which the failure occurs, which is not the condition your average traffic creates. And the measurement at the end has to be one that does not require anybody to say what the right answer was, or the cost of the exercise scales with the number of cases and stops being affordable at the size where it would tell you something.

Historical payloads, and what they cannot tell you#

If your agent framework persists full tool-call payloads, a SELECT over ninety days of stored arguments is cheaper than anything else in this chapter and is worth running first. Two things usually go wrong with it. Retention on trace payloads is often seven or fourteen days because they are large, which is a shorter window than the drift you are trying to observe. And production traffic is unlabelled, so a value like NOT_AS_DESCRIBED appearing 3,000 times tells you nothing about whether it was the right code for those 3,000 refunds.

What historical payloads do give you cheaply is the out-of-set count. If that count is already alarming you can skip ahead and start fixing. When it comes back at eleven occurrences out of forty thousand, you still do not know whether the problem is rare or whether your traffic simply has not exercised the conditions that produce it, and distinguishing those two is the entire reason the rest of this chapter exists.

The reason a small count carries so little is that it is a product of two quantities and reports only their result. One is how often the model invents a value when it is in the condition that provokes invention. The other is how much of your traffic was in that condition over the period you looked at. A tiny product is consistent with a tiny first factor and a large second one, and equally consistent with the reverse, and the two situations call for completely different responses: the first says leave it alone, the second says you have a serious defect that this quarter's traffic happened to miss. Nothing in a count over unlabelled production separates them, because production does not record which condition each conversation was in. Constructing the corpus is a way of fixing one factor at a known value so that the other one can be read.

The condition you have to construct#

A hallucinated argument is not produced uniformly across inputs. When a customer writes "it turned up with a cracked lid and packing peanuts everywhere", the mapping onto DAMAGED_ON_ARRIVAL is nearly determined by the text, and the model gets it right at a rate that makes the failure invisible in aggregate. The failures concentrate where the conversation does not contain enough information to determine the value, and the model has to produce something anyway, because the field is required and the tool is about to be called. Under those conditions it produces the most plausible-sounding string, which is a description of the situation rather than a member of your set.

So the reproduction corpus is not a random sample of conversations. It is stratified deliberately — split into groups chosen in advance, each filled to a size you decide rather than to the share it holds in real traffic — with most of its mass in the region where the true value is unrecoverable:

The unrecoverable stratum is the one that requires thought to write, because it means enumerating the reasons a customer might give that your six codes do not cover: changed their mind, found it cheaper elsewhere, ordered the wrong size themselves, a gift recipient did not want it, the item was fine but arrived after the event it was bought for and they are not framing that as lateness. Each of those is a real refund reason and none of them is DAMAGED_ON_ARRIVAL, WRONG_ITEM_SHIPPED, LATE_DELIVERY, NOT_AS_DESCRIBED or DUPLICATE_CHARGE. OTHER is the correct answer to all of them, and OTHER is the answer the model is least likely to reach for, because it is the least descriptive token available and the model is optimising for a plausible continuation rather than for conformance to a set it was never shown.

The sizes in that table are chosen to make each stratum readable on its own rather than to mirror traffic, which is the point of stratifying at all. Ninety unrecoverable conversations exist because that is where the behaviour under study lives and a rate over ten of them would be unreadable; forty determined ones exist because that stratum is a control, and a control only has to be large enough to notice if it moves. What you give up is any ability to add the four numbers together — a corpus built this way has no overall rate, only four conditional ones, and reporting a single figure across it is the one mistake the design cannot survive.

The four stratum sizes do have to add up to the corpus, and checking that they do is worth the ten seconds. Forty determined, sixty ambiguous, ninety unrecoverable and fifty absent is two hundred and forty conversations, with no fifth group and no leftovers. That matters less as arithmetic than as a statement about coverage: every conversation in the corpus was written into exactly one condition, on purpose, by somebody who knew which condition they were writing. A corpus with a residual "other" bucket has conversations whose condition nobody decided, and a rate computed over that bucket is a rate over whatever happened to fall in it.

Writing 240 conversations by hand takes about a day. Generating them with a second model and hand-checking is faster and is fine here, since the corpus does not need labels for correctness — it needs to reliably reach the tool call, which you can verify by running it once.

An entry is a fixture rather than a transcript — a stored input written to be replayed unchanged against whatever version of the agent is current, rather than a record of something that happened — because it has to be replayable against a changing agent and it has to carry the small amount of annotation the analysis reads:

{
  "id": "unrec-034",
  "stratum": "unrecoverable",
  "true_code": "OTHER",
  "must_call": "refund_order",
  "messages": [
    {"role": "user",
     "content": "I bought the stand mixer for my sister's wedding on the 12th and the wedding is off. Order 90418, box never opened. Can I send it back?"},
    {"role": "assistant",
     "content": "I can help with that. Let me pull up the order."},
    {"role": "user",
     "content": "Thanks — it's the 5-quart one, $219.99."}
  ]
}

The stratum field is the only annotation the set-difference measurement needs, and it describes how the conversation was written rather than what the right answer is, so it can be assigned by whoever wrote the entry without an adjudication. true_code is filled in where a defensible answer exists, which here is OTHER, because a cancelled wedding has no member in the set; it is left null across the ambiguous stratum, where two codes genuinely fit and picking one would be inventing a ground truth to grade against. must_call records which tool the run is meant to reach, and it is there because a fixture that never gets that far contributes nothing to the measurement while looking, in a results table, exactly like a fixture that did.

Driving the agent without issuing 240 refunds#

The harness — the script that walks the corpus, drives the real agent over each fixture and writes down what it did — runs against a recording stub in place of the refunds service, meaning a stand-in object that implements the same method signatures, keeps every argument it is handed, and returns a canned success without touching the real system. The stub matters for two reasons: it stops the reproduction from moving money, and it lets you return a fixed plausible success so the model's downstream behaviour stays on the path you are trying to observe.

from dataclasses import dataclass, field

REAL_REASON_CODES = {
    "DAMAGED_ON_ARRIVAL", "WRONG_ITEM_SHIPPED", "LATE_DELIVERY",
    "NOT_AS_DESCRIBED", "DUPLICATE_CHARGE", "OTHER",
}

@dataclass
class Recorder:
    calls: list = field(default_factory=list)

    def refund_order(self, order_id, reason_code, amount_cents):
        self.calls.append(
            {"tool": "refund_order", "order_id": order_id,
             "reason_code": reason_code, "amount_cents": amount_cents}
        )
        return {"refund_id": "rf_stub", "state": "pending"}

def run_corpus(conversations, agent, stratum_of):
    rec = Recorder()
    rows = []
    for conv in conversations:
        rec.calls.clear()
        agent.run(conv.messages, tools=bind(rec))
        for call in rec.calls:
            if call["tool"] != "refund_order":
                continue
            rows.append({
                "conversation_id": conv.id,
                "stratum": stratum_of(conv),
                "emitted": call["reason_code"],
                "in_set": call["reason_code"] in REAL_REASON_CODES,
            })
    return rows

Two details in that loop are load-bearing. The stub returns success rather than an error, because an error would push the model into the retry behaviour that chapter 8 deals with and contaminate the measurement with a different failure mode. And the emitted value is recorded verbatim before any normalisation — no upper-casing, no stripping, no trimming — since damaged on arrival and DAMAGED_ON_ARRIVAL are different findings, and a helpful normalisation step in the middle of your harness will erase the second-largest class of argument failure you have.

What the run came back with#

Running that corpus at temperature 0.7 — the sampling setting that governs how much the model varies its answer between identical runs, kept at the production value so the spread of emitted values is the spread real customers get — with three samples per conversation produces 720 calls in about eleven minutes and costs a few dollars. Sampling three times rather than pinning temperature to zero is deliberate here, and it is the opposite of the choice a selection eval makes: the quantity of interest is the spread of strings a given conversation can produce, not the single most likely one, and a temperature-zero run reports the modal answer while hiding how wide the distribution around it is. Collapsed by emitted value:

Two hundred and twenty-three of 720 calls, 31%, carried a value that does not exist in the refunds service, and every one of them would have returned 200 in production. Within the unrecoverable and absent strata the rate is 78%. Within the determined stratum it is 1.4%, which is the four defective calls where the model produced a lower-case description instead of a code. A single anecdote about CUSTOMER_UNHAPPY was not a rare event; it was the visible corner of the dominant behaviour under a condition production traffic hits constantly, because customers asking for refunds frequently do not give a reason that fits your taxonomy.

Two checks tie that headline to the table above it, and both are worth running on any result of this shape before quoting it. The run count is the corpus multiplied by the samples per conversation: two hundred and forty conversations sampled three times each is the seven hundred and twenty. And the headline figure is not a separate measurement from the table — add the calls on the rows whose third column reads no, and they come to exactly the two hundred and twenty-three, so the collapsed view and the summary are the same quantity counted twice. When those two checks pass, the remaining division is straightforward: two hundred and twenty-three out of seven hundred and twenty is a little under a third of every call the corpus produced.

The comparison is a set difference, which is why it is cheap#

Nothing in that table required a human to say what the right answer was. The finding is emitted ∉ REAL_REASON_CODES, computed against the authoritative set, and the authoritative set is not a document — it is whatever the refunds service will actually branch on, which means reading the enum in its source or querying the reference table it validates against. Getting that set from a wiki page is how you end up measuring against a taxonomy that was renamed eight months ago.

The set difference is a different measurement from accuracy, and confusing them costs you the cheap one. Accuracy asks whether NOT_AS_DESCRIBED was the correct code for this particular conversation, which requires a labelled answer per conversation and, for ambiguous cases, an adjudication about what "correct" even means when two codes both fit. Existence asks only whether the emitted string is a member of a set the backend can act on, needs no labels, runs on any argument with an authoritative value set, and scales to the whole tool surface in an afternoon. Run existence everywhere and accuracy where it matters most.

Where an authoritative set exists, and where a corpus stops#

This generalises to any parameter whose legal values live in a system rather than in the schema. A component_id drawn from nine hundred rows in a service catalogue, a warehouse_code, a sku, an assignee that has to be a real user handle, a currency restricted to the eleven your payment processor is configured for. Those cannot be static enums in a tool definition that was serialised at deploy time — the catalogue changes weekly, and a 900-member enum is 4,000 tokens of every request. The reproduction is identical in shape: construct conversations where the value is not recoverable, capture what the model emits, and check membership against the live set.

What the corpus cannot do is give you a production rate. It is made of cases someone imagined, so the 78% in the unrecoverable stratum is a fact about the conversations that were written. It is strong evidence that the failure mode is real and common, and it says nothing about what fraction of real refund conversations fall into that stratum. A team that writes an unrepresentative corpus can produce a terrifying number for a failure that occurs on 0.1% of live traffic, or a reassuring one for a failure that dominates a customer segment nobody on the team resembles.

The reproduction also freezes. It was written against six reason codes, one prompt version and one model. When refunds adds a seventh code next quarter, when someone rewrites the tool description, or when the model behind the agent is upgraded, the numbers in that table become historical and the corpus has to be rerun before anyone can say whether the change helped. That is a scheduled job at best, and scheduled jobs measure the corpus rather than the traffic.

What survives both problems is the predicate itself. emitted ∉ REAL_REASON_CODES does not care whether the conversation came from a corpus or a customer, it needs no labels, and there is nothing about it that requires a harness — it can run at call time, on every request the agent makes, against the set the owning service is holding right now.