One remaining reason to hand-parse, and what it costs you

Hand-parsing means running a regular expression over a model's completion to recover a tool name and an argument string from labels the model was prompted to write — the Thought:/Action: protocol from earlier in this topic. Every argument against it in the previous chapters assumed a provider's tool-calling API was available as the alternative. One situation removes that assumption, and it is the one remaining legitimate reason anyone reaches for this design. A model served on hardware you control, behind an inference server that exposes a completions endpoint and nothing else, genuinely cannot call a provider's tools endpoint. Data residency, air-gapped deployments, a fine-tuned open-weights model whose behaviour you depend on, an inference stack a platform team chose two years ago: none of these are excuses.

What the constraint does not establish is the conclusion. The absence of a tool-calling API rules out one of three options, and the argument for a parser is usually made as though it ruled out two. The third is constrained decoding — forcing the model's output to conform to a grammar as it is being generated — which runs on your own hardware, against the same model, in the same serving stack, and hands your code a validated object. The cost of that option is measurable and worth knowing before the decision, and so is the inventory of solved problems a parser re-opens in your repository.

The option between prose and a provider API#

Constrained decoding makes the model's output conform to a grammar by masking the token distribution at each step, so that any token which would make the output unparseable has probability zero. Applied to a JSON Schema, it produces output that is valid JSON conforming to that schema by construction — not checked afterwards, but impossible to violate during sampling. This runs entirely on your own hardware, against the same model in the same serving stack, and it has been available in the mainstream local stacks for years: llama.cpp has GBNF grammars, a BNF-style notation describing the exact strings the sampler will allow; vLLM exposes guided decoding backed by xgrammar or outlines; TGI, which is Hugging Face's Text Generation Inference server, and Ollama have equivalents.

"Masking the token distribution" is the step that carries the guarantee, and it is worth doing at the size of a few characters before believing it at the size of a schema. A model generating text produces, at every position, a score for every token in its vocabulary, and the sampler picks one according to those scores. The grammar is consulted between those two events. Given everything emitted so far, it computes the set of tokens that could legally come next, and every token outside that set has its score replaced by negative infinity, which makes its probability exactly zero rather than merely small.

Walk one field of the declaration below. After the model has emitted "tool": ", the grammar knows the next characters must begin one of three literal strings, so every vocabulary token that does not start search_trials, get_arm or get_document is masked out. Suppose the model, left free, would have written search_documents, a name it saw in a prompt exemplar and misremembered. It emits search_, which is legal because one alternative still matches. At the next position the only legal continuation is trials", because search_ has already eliminated the other two alternatives and no other completion of the first one exists. documents is not sampled with low probability; it is not in the candidate set. The model's preference has not changed, and the reachable output space has.

Two consequences follow from that, and only the first is usually noticed. An invalid tool name cannot appear, which is what the feature is sold on. The second is that a recovery path becomes unnecessary rather than better: there is no branch for "the model named a tool that does not exist", because the state the branch would handle cannot be entered. That is the same deletion the previous chapter's migration produced, arrived at from a completely different direction, and it is why constrained decoding belongs in the same category as a provider's tool API rather than in the same category as a stricter parser.

Take an agent answering questions about clinical trial documents from a model served inside your own building. The tool declaration from earlier chapters becomes a decoding constraint rather than a prompt convention:

POST /v1/chat/completions
{
  "model": "local-14b",
  "messages": [{"role": "user", "content": "Which arms of NCT04123456 were terminated?"}],
  "guided_json": {
    "type": "object",
    "properties": {
      "reasoning": {"type": "string"},
      "tool": {"type": "string", "enum": ["search_trials", "get_arm", "get_document"]},
      "arguments": {
        "type": "object",
        "properties": {
          "trial_id": {"type": "string", "pattern": "^NCT[0-9]{8}$"},
          "arm_label": {"type": "string"}
        },
        "additionalProperties": false
      }
    },
    "required": ["reasoning", "tool", "arguments"],
    "additionalProperties": false
  }
}

The response body's content is then a JSON object your code loads and validates in one line, with the tool name drawn from an enum the model could not have stepped outside of and arguments in named fields where a comma is a character in a string. On llama.cpp the same constraint is a grammar file, and for a fixed set of tools it is short enough to generate from your schemas at startup:

root        ::= "{" ws "\"reasoning\":" ws string "," ws "\"tool\":" ws tool "," ws
                "\"arguments\":" ws object ws "}"
tool        ::= "\"search_trials\"" | "\"get_arm\"" | "\"get_document\""
object      ::= "{" ws ( pair ( "," ws pair )* )? ws "}"
pair        ::= string ws ":" ws value
value       ::= string | number | "true" | "false" | "null"
string      ::= "\"" ( [^"\\] | "\\" ["\\/bfnrt] | "\\u" [0-9a-fA-F]{4} )* "\""
number      ::= "-"? [0-9]+ ( "." [0-9]+ )?
ws          ::= [ \t\n]*

There is a nuance here that matters when someone says a local stack "supports tool calling". Several of them implement that support by parsing the model's own output on the server — vLLM's --tool-call-parser hermes and similar flags select a parser that pulls a call back out of text the model generated in whatever format its fine-tune was trained on. That is the same recovery-from-prose design, moved across a process boundary and maintained by someone else, which is an improvement in who fixes the bugs and not a change in kind. Combining a tool-call parser with guided decoding, so the shape is guaranteed rather than recovered, is the configuration worth insisting on, and it is a serving flag rather than a project.

The sentence a reader is likely to write after skimming a serving stack's release notes is "our stack supports tool calling, so the shape is handled", and it is worth taking that belief apart because it is the most expensive one available here. The belief is tempting because the observable evidence for it is genuine: calls do arrive as structured objects, the flag really is called a tool-call parser, and on well-formed generations the traffic is indistinguishable from a provider's. What holds still under all of that is the rate at which structured objects come back on the traffic you have tried. What varies is whether a generation the model could produce is capable of yielding a call your code will act on, and a server-side parser leaves that varying, because parsing is recovery and recovery has a failure mode by definition. The flag moved the parser across a process boundary. Nothing about it made the shape unreachable, which is the property the first row of the table is measuring.

Read the table across rather than down. The first row states a mechanism and every row beneath it is that same mechanism surfacing somewhere else in the system, which is why the left column differs from both of the others in every single row while the right two give identical answers on escaping, on injection, on what a trace carries and on the code you end up owning. Those matching rows are the load-bearing ones, because they are the things a parser actually costs, and they say the middle column is not a compromise version of the right one. Where the right two do differ — how the guarantee is produced, how a failure is reported back, which stacks they run in, what they cost — only the last of those is a judgement rather than a fact about your infrastructure. And the row that decides which columns are in play at all is the plainest one on the sheet. "Works with a completions-only stack" reads No for the provider column, which removes it, and Yes for both of the others, which leaves a choice between two designs that agree on nothing else.

Constrained decoding is not free, and the honest entries in that column are the throughput and the quality question. Compiling a grammar costs tens of milliseconds once per distinct schema, and masking adds per-token overhead that measured between five and fifteen percent on the clinical-trials setup depending on how restrictive the schema was. There is also a real effect where forcing a model into a rigid structure degrades its answers relative to letting it write freely, which is why the schema above keeps a reasoning string as the first property: the model gets to produce text before it commits to a tool, in the same order the label protocol gave it, without that text being load-bearing for dispatch.

Be careful what that measurement licenses. It came from one model on one serving stack with one family of schemas, and what it supports is the claim that masking costs a bounded single-digit-to-low-double-digit fraction of throughput rather than a multiple — which is the claim that matters, because the fear it answers is that constrained decoding halves your tokens per second. What it does not support is a number you can put in your own capacity plan. The residual is stated in the measurement itself: the spread came from varying how restrictive the schema was, so the figure is a property of a schema family and not of the technique, and a different set of tool declarations can land outside the range in either direction. Measure it on your own schemas before anybody promises latency to a stakeholder.

What you are agreeing to own#

If the parser wins the argument anyway, the thing to be clear-eyed about is that the forty-five lines in the demo are not the deliverable. The demo's parser handled twelve questions written by the person who wrote the prompt, none of which contained a comma in a value, an argument longer than one line, a document quoting the word "Observation", or a tool with a side effect. The version that survives contact with a real corpus accumulates the parts the protocol left out. On the acquired service from the previous chapters, the equivalent code had grown from an initial 45-line parser to 380 lines across the parser, a result sanitiser, an argument coercion layer and their tests, and three of the four incidents in its first year were parse-related rather than model-related.

Neither of those figures is interesting as a level, and both are interesting as gaps. Three hundred and eighty lines is not a large module by any standard, and a team told "your parser will reach 380 lines" will reasonably answer that they have larger files than that. The number to look at is the difference: the growth is code that did not exist when the design was chosen, was not estimated, and does nothing a user would recognise. The same reading applies to the incidents. Three of four is not a statement that the system had many incidents; it is a statement about where the incidents came from. A system built to answer questions about clinical trials spent most of its incident budget on the layer that turns a substring into a call, which is the layer that was supposed to be the cheap part.

The observability difference is the one that is easiest to underrate at the whiteboard and hardest to live with afterwards. A step in the typed design emits a span with tool.name, tool.arguments as structured attributes, a duration, a retry count and an error flag, which means the questions an on-call engineer asks at two in the morning are queries: which tool started failing, did its argument shape change, is one customer id responsible for the spike. A step in the parsed design emits a span with one long string attribute, so the same questions require exporting transcripts and writing throwaway regular expressions against them, in the middle of an incident, using the same technique that caused it.

The specific things that transfer to your repository are worth listing, because each of them is a solved problem you are re-opening. Escaping is yours: you decide what happens when a value contains your delimiter, and every answer is a partial one. Argument validation is yours, and since you have no declaration, it lives as per-tool ad hoc checks that drift from the function signatures they guard. Retry semantics are yours, and as chapter two showed, the channel available for reporting a failure is the same channel used for reporting facts. The injection surface is yours, in the sense that you own both the vulnerability and the escalating list of spellings that patch it. Observability is yours in the worst way: your traces carry a span with a text attribute containing an entire transcript, so you cannot chart calls per tool, cannot alert on an argument distribution shifting after a prompt change, and cannot diff last week's arguments against this week's, because none of those are fields.

Twelve more questions, chosen to break it#

Since the demo is what the decision will actually turn on, it is worth being specific about what a demo of this kind does not exercise, so that the request is for evidence rather than for caution. Run the twelve questions again with a document title containing a comma and a semicolon. Run one where the user's own text contains the word Observation: followed by a colon and a newline. Run one against a document that contains the four labels in a footnote, since clinical protocols quote instructions constantly. Run two calls that should happen in the same step and see how the flat transcript represents them. Then take the parse-failure counter, which the demo does not have, add it, and run two hundred real questions to get a rate. If that afternoon produces a parse-failure rate under a percent and no silent argument corruption, the argument for shipping is much stronger than it was in the meeting room, and if it does not, nobody has to have an opinion about protocols.

Where the pro-parser case is strongest#

One local-model wrinkle deserves naming because it is the strongest technical version of the pro-parser case. Open-weights models are frequently fine-tuned to emit tool calls in a house format — Hermes-style XML tags, named for a widely copied family of fine-tunes, a Python-ish func(arg=1) call, or a bare JSON object — and a model is measurably better at the format it was trained on than at an arbitrary one you impose. That is a real argument for matching your schema's surface form to the model's training format, and it is not an argument for recovering calls from free prose, since you can compile a grammar that accepts exactly the model's native format and still guarantee the shape. The work is generating the grammar from your schemas once, and the payoff is that the model writes what it is fluent in while your code receives something it can validate.

The counter-argument that "we will only use it for read-only tools" is worth taking seriously and holds for exactly as long as that stays true. Tools grow. The research assistant in chapter three acquired its shell tool eleven months after launch, added by someone who needed to fetch one URL the fetcher could not reach, in a system whose parser had been written under the assumption that everything was read-only.

None of this is an argument that ReAct was a mistake, and it is worth ending on that. The paper's claim — that interleaving reasoning with acting beats either alone — is the foundation of every agent in production today, including the ones in this topic that never parse a character. What has been superseded is a transport that existed because, for about eight months, there was nothing else. Recognising the difference is what lets you keep the idea and decline the implementation, which is the position to argue from when a working demo is on the table and shipping it looks like the obvious next step.

The way to make that concrete on your own system is to take the three abuses from chapter two — a value containing your delimiter, a missing required argument, an argument nobody declared — and run them against whatever agent you currently operate, then look at where each one surfaced. If it surfaced inside a tool, you have the parsed design regardless of what the code is called, and the mapping from chapter five tells you what the migration consists of. Writing the schemas for the tools you already have is usually an afternoon, and it is the same afternoon whether you send them to a provider or compile them into a grammar for a model running in your own building.