Checking arguments against a value set

A reproduction corpus is a test suite, and a test suite runs when somebody runs it. The same failure is running in production right now on cases nobody constructed, and it writes no error line, so the only way to know its current rate is to compute the same predicate on live traffic. That predicate — is this emitted string a member of the set the owning service will actually branch on — is cheap enough to run on every call and needs no labels, which makes it one of the few checks in this topic that can be shipped in an afternoon.

What makes it worth a chapter rather than a paragraph is that almost every team's first instinct is to make it a guardrail: reject the bad value, protect the backend, move on. The guardrail is the version that destroys the measurement while appearing to solve the problem, and the order in which you ship the two behaviours decides whether you end up with a number that responds to prompt changes or one that is permanently pinned at zero.

Where the check sits#

The dispatcher already sits between the model's tool-call message and the service client, because that is where argument deserialisation and auth already happen. The check goes there, before the call, with a value-set provider that reads from the authoritative source rather than from a constant in your repository.

from time import monotonic

class ValueSets:
    """Authoritative sets, pulled from the owning service, cached briefly."""

    def __init__(self, client, ttl_s=300):
        self._client, self._ttl, self._cache = client, ttl_s, {}

    def get(self, name):
        hit = self._cache.get(name)
        if hit and monotonic() - hit[0] < self._ttl:
            return hit[1]
        values = self._client.fetch_value_set(name)   # e.g. refunds.reason_codes
        self._cache[name] = (monotonic(), values)
        return values

CHECKED_ARGS = {
    ("refund_order", "reason_code"): "refunds.reason_codes",
    ("create_ticket", "component_id"): "catalog.components",
    ("create_ticket", "assignee"): "identity.active_handles",
    ("transfer_stock", "warehouse_code"): "wms.warehouses",
}

def check_arguments(tool, args, value_sets, metrics):
    mismatches = []
    for arg_name, raw in args.items():
        set_name = CHECKED_ARGS.get((tool, arg_name))
        if set_name is None:
            continue
        legal = value_sets.get(set_name)
        in_set = raw in legal
        metrics.increment(
            "tool_argument_value_check",
            tags={"tool": tool, "argument": arg_name,
                  "set": set_name, "in_set": str(in_set)},
        )
        if not in_set:
            mismatches.append((arg_name, raw, set_name))
    return mismatches

The metric carries no emitted value in its tags, which is deliberate. reason_code has fourteen observed values and would be safe; component_id has an unbounded space of invented strings, and a counter labelled by user-influenced free text is the standard way to take down a metrics backend. The values themselves go to a sampled structured log — one line in fifty, with the emitted string, the conversation id and the tool — which is enough to reconstruct a table of emitted values from production and nowhere near enough to hurt cardinality, which is the count of distinct label combinations a metrics backend has to hold a time series for.

The step under that warning is what makes it a rule rather than a superstition. A metrics backend does not store a counter. It stores one time series per distinct combination of label values, each with its own index entry and its own retained history, so what a metric costs is set by how many combinations can exist rather than by how many events arrive. A label drawn from a closed set of a dozen members multiplies that product by a dozen and stops there. A label holding a string the model composed makes the possible values the set of all strings, and a label space with no ceiling gives the backend no ceiling either. The same value is safe in a sampled log line, where it is one field of one record that ages out, and unsafe as a label, where it creates a series that persists.

Comparing that to a foreign-key constraint is close, and instructive where it breaks. A foreign key rejects the write, and rejecting is exactly what you may not want to do here, at least not first.

Observe before you enforce#

There are three things the dispatcher can do with a mismatch, and they produce three different systems.

Shipping observe-only for two weeks before deciding is worth the delay, because the production number is frequently not the corpus number and the gap changes what you do next. On one deployment production ran 41,200 refund_order calls in the first fourteen days and 6.2% carried an out-of-set reason_code — a fifth of the corpus rate, because most real refund conversations are in the determined stratum where the text pins the code down. That number is high enough to matter to refund reporting and low enough that reject-and-repair, at roughly a second of added latency on 6.2% of refunds, is clearly affordable. Had it come back at 31%, a repair turn on nearly a third of refunds would have been a different conversation.

Those two figures stand in a clean ratio and the ratio is the finding, not a coincidence. Six point two against thirty-one is a fifth, and the reason it is smaller rather than wrong is that the two numbers were computed over different populations. The corpus is deliberately weighted toward conversations where the true value cannot be recovered from the text, because that is the condition the failure needs. Production is weighted the way customers happen to write, which is mostly toward conversations that pin the code down. The corpus figure is a rate given a condition; the production figure is that rate multiplied by how much traffic is in the condition, summed over the conditions. Reading it the other way round gives you the share of live refund conversations sitting in the invention-prone region — divide the production rate by the corpus's conditional rate for that region and it comes out at well under a tenth of them, which is a statement about your customers rather than about your model, and one no amount of prompt work will move.

The failure mode of jumping straight to reject is subtle. Once the dispatcher returns an error to the model instead of calling the service, the metric you were about to build measures something else: how often the model invents a value on its first attempt, given that it will be corrected. That is a fine number to have, and it is not the number that tells you whether a prompt change helped, because the repair loop absorbs regressions. A model that got 30% worse at reason codes and a model that got 2% worse both show 0% bad values reaching the backend. Record the pre-repair emission as its own metric and the enforcement stops blinding you:

mismatches = check_arguments(tool, args, value_sets, metrics)
if mismatches and mode == "reject":
    metrics.increment("tool_argument_repair_started", tags={"tool": tool})
    return tool_error(
        f"{mismatches[0][0]} must be one of: "
        f"{', '.join(sorted(value_sets.get(mismatches[0][2])))}"
    )

The error text matters more than it looks. Returning "invalid reason_code" gives the model nothing to work with and it will guess again from the same distribution; returning the legal set turns the second attempt into a selection from six options, which succeeds on the first retry in the large majority of cases. Chapter 8 is about what happens when a tool error does not carry enough information for the model to make progress, and this is the small, well-behaved version of that.

Not every mismatch is an invention

The 6.2% is one rate over four different things, and the remediation differs for each. Splitting two weeks of sampled logs by the shape of the mismatch:

Two things about that Share column decide whether it can be used. It sums to a hundred with nothing left over, so the four shapes are a partition of the mismatches rather than four interesting examples, and any mismatch that arrives has to be one of them. And the shares are shares of the mismatches, not of the calls, which is the conversion that gets skipped: a shape's contribution to the overall rate is its share multiplied by the mismatch rate, so a fifth of the mismatches is nothing like a fifth of the traffic. Removing a shape works the same way in reverse: take out the shares that normalisation fixes, keep the remaining fraction of the mismatches, and multiply that fraction by the rate you started from. What comes back is the part of the original rate that survives the cheap fix.

The first two shapes are 27% of the mismatch rate and are not a model problem at all — the model chose the right code and formatted it differently, and two lines of normalisation in the dispatcher turn those calls into correct ones. Chapter 2's harness recorded values raw on purpose, because normalising there would have hidden this row entirely; the production check records both, tagging each call with in_set and in_set_normalised, so the two series can be graphed against each other and the gap between them is the formatting tax. What remains after normalisation is 4.5% of calls rather than 6.2%, and that is the number a description rewrite has any chance of moving.

What a trend line shows that a corpus run cannot#

With observe mode running continuously, CUSTOMER_UNHAPPY becoming OTHER stops being an anecdote and becomes a series. A description rewrite that the corpus scored at 31% → 4% showed up in production as 6.2% → 1.1% over the two days after deploy, which both confirms the fix and rescales it. A provider migration three weeks later showed up the same evening as 1.1% → 3.4%, visible without anyone deciding to look, because the check runs on traffic rather than on a schedule. When refunds adds a seventh code, the mismatch rate spikes for as long as the cached set is stale and then settles, which is a five-minute artefact with an obvious cause and a useful side effect: it proves the value-set provider is reading from the live source rather than from a constant somebody vendored.

A trend line only manages anything if something is watching it, and the definition is short enough to write out.

- alert: ToolArgumentInventionRate
  expr: |
    sum by (tool, argument) (rate(tool_argument_value_check{in_set="false"}[1h]))
      /
    sum by (tool, argument) (rate(tool_argument_value_check[1h]))
      > 0.03
  for: 2h
  annotations:
    summary: "{{ $labels.tool }}.{{ $labels.argument }} out-of-set rate above 3%"

Three percent is the post-fix baseline plus roughly twice its observed day-to-day variation, which on this metric was about ±0.6 points. The two-hour window is there because a deploy that rewrites a description produces a step change rather than a spike, and a five-minute window on a step change either fires on noise or misses the step. Alerting on the absolute count of out-of-set calls rather than on the ratio fires every time refund volume doubles, which is the usual reason these end up permanently silenced.

The check is an availability dependency now#

A membership test that reads from the owning service has a network call inside it, and the first version of this shipped without anyone deciding what happens when that call fails. Six weeks in, an identity service's read path was degraded for forty minutes during a migration, value_sets.get("identity.active_handles") raised, the exception propagated out of check_arguments, and every create_ticket in that window failed in front of a ticket service that was perfectly healthy. An instrument built to observe a silent failure had made itself an availability dependency of the tools it was watching.

def check_arguments(tool, args, value_sets, metrics):
    mismatches = []
    for arg_name, raw in args.items():
        set_name = CHECKED_ARGS.get((tool, arg_name))
        if set_name is None:
            continue
        try:
            legal = value_sets.get(set_name)
        except ValueSetUnavailable:
            metrics.increment("tool_argument_check_unavailable",
                              tags={"tool": tool, "set": set_name})
            continue                       # never block a call on the observer
        ...

Failing open — letting the call through untouched whenever the check itself cannot run, rather than failing the call closed as a security control would — is the right default for an observer, and it costs something, because a check that has quietly stopped checking looks exactly like a check that is finding nothing. The tool_argument_check_unavailable counter is what tells those two states apart, and it needs an alert of its own, since a value-set provider that has been failing open for three days produces a flat zero on the mismatch graph that reads as good news. The TTL, the time to live on a cached set before the provider goes back to the owning service for it, deserves the same scrutiny: five minutes means the first calls after refunds adds a code are scored against a stale set, which is tolerable and self-correcting, while a cache with no expiry — because someone made the set a module-level constant during a refactor — means the check has been measuring the day it deployed ever since.

The placement decision changes when you do not own the tool. An MCP server on the other side of a process boundary has no dispatcher of yours inside it, so the check has to live in the client, which is where it belonged anyway: the client is the last place that sees the arguments before they leave, and keeping the instrument there means it survives the server being replaced, upgraded, or operated by another team.

Three failures no value set constrains#

The first is any argument without an authoritative set. A ticket summary is free text, a search query is free text, and a model that writes a summary describing symptoms the customer never reported has hallucinated an argument in every sense that matters, with no set to check it against. Those need an entirely different instrument — a sampled judge or human review over a slice of traffic — and it is worth being explicit that the check covers the enumerable fraction of your argument surface rather than the argument surface.

The second is the value that exists and is wrong. NOT_AS_DESCRIBED on a refund that was actually a duplicate charge passes membership perfectly, reaches the backend, and corrupts refund reporting exactly as badly as CUSTOMER_UNHAPPY did. Membership is a necessary condition and nothing more, and closing that gap requires labelled data, which is expensive, which is why the cheap check goes everywhere and the expensive one goes where the cost of being wrong is highest. The honest position is that the in-set-but-wrong rate on most agent deployments is unmeasured, and where teams have measured it on refund-style taxonomies it tends to sit in the range of a few percent to low double digits depending on how ambiguous the taxonomy is.

The third is the selection failure from chapter 1. No value set constrains which tool gets chosen, so a perfectly well-formed search_orders call passes every check in this chapter and returns an empty list, and the dashboard you have just built stays green through it.

Nothing about the underlying model behaviour changed when the check was added, and no user had a better day. What changed is that a silent, well-formed, 200-returning failure acquired a number that responds to descriptions, models and catalogues, which is the precondition for anyone being able to say whether a change to any of those three helped.

The argument half of the problem now has a detector, a metric and a trend line. The selection half has none of those, and the standard advice about it — that models start degrading somewhere past twenty tools — turns out to be a claim about someone else's catalogue rather than yours.