A complete(prompt) helper abstracts the transport, not the task

The trial was scheduled for a Tuesday afternoon and budgeted at half a day, because the team had been careful. Every model call in the platform went through one function, llm.complete(prompt: str) -> str, and swapping the provider behind it meant changing the client construction and the request body inside that function — thirty-one lines of diff, all of it in one file. The unit tests passed. The integration suite, which stubs the model, passed. The change went out behind a flag at five per cent of traffic at 14:20.

By 15:05 there were three separate incidents. The invoice extractor's JSON parse failure rate went from 0.3% to 11.4%, because the new model liked to preface its object with a sentence of explanation. The ticket router began emitting a category label that was not in the enum — billing_and_payments instead of billing — roughly one request in forty. And the chat endpoint, the most visible surface in the product, was entirely unaffected, which was the most alarming finding of the three, because it meant the endpoint was not going through llm.complete at all. It had needed streaming, llm.complete could not stream, so eight months earlier someone had imported the provider SDK directly and moved on.

None of that is a criticism of the engineer who wrote the helper. It is a description of what the helper's signature is a boundary around, which is the transport: URL, auth header, retry policy, timeout, JSON envelope. That work is real, and centralising it is right. It is also the part of a provider swap that costs half a day. Everything that made Tuesday afternoon run into Thursday lived somewhere else.

What that Tuesday was, in the vocabulary of this topic, is a migration: moving a working system off the model it was built against and onto a different one. Here the move was to another provider, meaning the company that serves a model over an API and bills for the tokens going through it, and the same word covers the smaller move to a newer version of the same model from the same vendor. These are ordinary events rather than rare ones, because a model you depend on will eventually be retired on a published schedule, a rival's model will get better at the one job your product is judged on, prices change, and a capability you want arrives first somewhere you are not. Portability is the property of your own code that decides what such a move costs. A swap is always possible in the narrow sense that both vendors accept HTTP requests; portability is how much work sits between the decision to move and the point where you trust the result on live traffic. Engineers meet it in three ordinary places: a deprecation email with a date on it, a pull request that edits one model identifier in a config file, and a spreadsheet comparing two vendors' price per million tokens. The rest of this topic is about where that cost accumulates and how to move it somewhere you can see it before you commit to a date. The half-day estimate above was an estimate of this platform's portability, and it was wrong because the code that tied the system to one particular model was not in the file anyone was looking at.

What the signature promises and what it holds#

Here is the helper, close to how these are usually written, with the retry and timeout handling that makes it worth having in the first place.

# llm/client.py
_client = openai.OpenAI(api_key=settings.OPENAI_KEY)

@retry(stop=stop_after_attempt(3), wait=wait_exponential(max=8))
def complete(prompt: str, *, temperature: float = 0.0, max_tokens: int = 1024) -> str:
    resp = _client.chat.completions.create(
        model=settings.MODEL_ID,
        messages=[{"role": "user", "content": prompt}],
        temperature=temperature,
        max_tokens=max_tokens,
        timeout=30.0,
    )
    _metrics.observe(resp.usage.prompt_tokens, resp.usage.completion_tokens)
    return resp.choices[0].message.content

The type of the boundary is str -> str. Whatever the model actually returns — a stop reason, a refusal, a tool call, a usage record, a list of content blocks, a cached-prefix hit — has to be collapsed into a string before it crosses, or discarded. The interesting consequence is not what happens inside complete. It is what the callers had to do to work around that, because a caller that needs anything richer than a string is forced to encode its needs in the only channel available, which is the prompt.

The prompts are the provider-specific part#

Consider two of the forty call sites as they existed on the day of the trial.

# billing/extract.py
INVOICE_PROMPT = """Extract the invoice fields from the document below.

Return ONLY a JSON object. Do not write any preamble. Do not wrap the JSON in
markdown fences. Do not explain your reasoning. Begin your response with the
character { and end it with the character }.

Fields: invoice_number, issue_date (ISO 8601), currency (ISO 4217),
total_ex_tax (decimal string), line_items (array).

Document:
{document}
"""

def extract(document: str) -> Invoice:
    raw = llm.complete(INVOICE_PROMPT.format(document=document))
    return Invoice.model_validate_json(_strip_fences(raw))

# support/route.py
ROUTE_PROMPT = """Classify the ticket into exactly one category.

Categories: billing, technical, account, other

Answer with the category word alone, lowercase, no punctuation, no explanation.

Ticket: {ticket}
"""

def route(ticket: str) -> str:
    return llm.complete(ROUTE_PROMPT.format(ticket=ticket)).strip().lower()

Every one of those formatting instructions exists because of an observed behaviour of one specific model. "Do not wrap the JSON in markdown fences" is in there because a model wrapped JSON in markdown fences. _strip_fences is a function that exists because the instruction was not reliably obeyed. "Answer with the category word alone" is calibrated against a model that had a tendency to add a sentence of justification. These are not general prompt-engineering hygiene; they are patches for particular failure modes of a particular checkpoint — one frozen set of model weights that a vendor serves under a dated identifier, such as the February release of a family as against the May one — accumulated over months of production incidents, and they are sitting in forty different modules owned by six different teams.

When the provider changed, those patches did not become wrong so much as irrelevant, and a new and disjoint set of patches became necessary. The new model did not wrap in fences; it prefixed with prose. It did not add justifications to the classification; it occasionally answered with a two-word label that seemed more accurate to it than the four offered. The helper's signature gave the migration no place to stand, because the thing that needed changing was distributed across the call sites and had never been named as provider-specific anything. It looked like ordinary application code.

What escapes around the boundary#

The other half of the problem is the traffic that never went through the helper at all. Fifteen minutes with grep produced the inventory nobody had.

$ rg -c "llm\.complete\(" --type py | wc -l
      40
$ rg -l "from openai import|import openai|OpenAI\(" --type py | grep -v "^llm/"
chat/stream.py
agents/researcher.py
agents/tool_loop.py
search/rerank.py
ingest/embed.py
support/summarise_batch.py
eval/harness.py

Seven modules held a direct dependency on the provider SDK. The reasons are individually reasonable. chat/stream.py needed token-by-token streaming, which a function returning str cannot express. agents/tool_loop.py needed tool definitions, a tool-call response shape, and a way to feed results back as a subsequent message, none of which fit through a single string. ingest/embed.py used the embeddings endpoint, which is not a completion at all and so was never in scope for the helper. eval/harness.py needed logprobs and per-call latency.

So the true blast radius of the swap was forty prompt-carrying call sites plus seven modules with an unmediated SDK dependency, and the second group contained the highest-value surfaces in the product. The abstraction had produced a reassuring number — one file to change — that was measuring the wrong thing. Anything the boundary could not express had simply been built beside it, which is the normal fate of a boundary too narrow for the work.

Three categories of leak, and what a swap does to each#

It helps to separate what leaked, because the three kinds behave differently and are fixed differently.

Only the last row is what the helper abstracts. It is also the only row where the work is proportional to the size of the diff, which is why the estimate was wrong by a factor of six: the estimate was made by looking at the diff.

The fourth and fifth rows deserve a second look, because they are a different kind of failure from the first three. Structured output, meaning the provider itself constraining the response to a declared JSON schema so that valid output is a property of the request rather than a hope about the prompt, was never adopted at this company. The provider had supported a response_format with a JSON schema for over a year, which would have made _strip_fences unnecessary and driven the parse failure rate to approximately zero. Nobody used it, because the only way to reach the model was a function whose parameters were a string, a temperature and a token cap. The abstraction did not merely fail to protect the migration; it also quietly set the ceiling on what the platform could do with any model, for as long as it stood. That cost was paid every day for eight months, and it never appeared on a ticket.

The driver analogy, and where it stops holding#

The complete(prompt) -> str shape is not an accident of one team's taste. It is a direct import of a pattern that works extremely well elsewhere: the database driver. An application writes SQL against a connection interface, a driver translates it for Postgres or MySQL, and swapping the driver is close to a configuration change. Engineers who have done that swap successfully arrive at model providers expecting the same structure, with the model as the engine and the prompt as the query.

The analogy holds for one layer and fails at the next. What makes the database case work is that SQL is a specification with a published grammar, and both engines have committed to implementing it, so the query text is portable by contract. Where that contract runs out — window function syntax, upsert semantics, JSON operators, index hints — portability runs out too, which is why every ORM has a per-dialect module and why teams that write raw SQL against dialect-specific features accept that a migration means rewriting those queries.

There is no equivalent specification for prompts. The instruction "return only a JSON object with no preamble" is not implemented to a standard by anybody; it is a request whose success rate is a property of one checkpoint's post-training, measurable but not guaranteed, and drifting between versions of the same model. The nearest analogue to prompt text is not SQL. It is the collection of dialect-specific queries an ORM keeps in separate files precisely because it knows they are not portable, and the mistake in the complete(prompt) design is that it treats those queries as if they were the portable part while carefully abstracting the connection string.

The consequence is a boundary that has its layers inverted. A database driver abstracts the wire protocol and stands beneath a portable query language; the helper abstracts the wire protocol and stands beneath forty non-portable queries with nothing recording that they are non-portable. When someone asks what the swap costs, the visible artefact is the driver-shaped file, and the invisible artefact is the dialect layer nobody named.

That framing also answers what to say when the design is defended in review as provider-agnostic. The claim to test is not whether the code compiles against a second provider, which it will, but whether anything in the codebase records how this system's behaviour depends on the specific model behind it, and whether there is a test that fails when that dependence is violated. In this platform there was no such record and no such test, which is why the failure surfaced as a production incident rather than as a red build.

The tempting fix that reproduces the problem#

The obvious response to Tuesday is to widen the interface: add complete_streaming, add complete_with_tools, add a schema= parameter, add embed. That produces a larger transport shim, and the second migration goes better than the first in the sense that the seven modules with direct SDK imports now go through the shim too. It does not touch the first three rows of the table, which is where the schedule actually went. The prompts are still at the call sites, still tuned to a checkpoint, still with no test that says whether they are tuned to the current checkpoint. The parsing repairs are still local. The next provider swap still means forty separate acts of judgement by people who did not write the prompts, with no artefact that tells them whether they got it right.

There is a related instinct worth naming, which is to reach for a multi-provider library and treat the problem as solved by adopting someone else's shim. Those libraries are useful and they do genuinely centralise the transport, including the parts that are tedious to get right, such as streaming reassembly and error taxonomy. What they hand you is a wider version of the same boundary. The prompt that says "do not wrap the JSON in markdown fences" does not become portable because the HTTP call did.

The claim here is not that llm.complete was a mistake. Something has to own the base URL, the key, the retry budget and the timeout, and having that in one place is why Tuesday's transport work took half a day rather than a week. The mistake is in what the team believed the boundary bought them. It bought a cheap transport swap and was read as insurance against a provider change, and those differ by everything in the table above.

What the incident actually established is a shape for the fix. The things that moved on Tuesday — the prompt, the schema the output must satisfy, the parsing and repair, the retry rule specific to that call, and some evidence about whether it still works — always travel together. In the invoice extractor they are four artefacts scattered across two files with no name binding them. The next chapter puts a name and a boundary around exactly that group, and shows what a swap looks like when the unit you migrate is a task rather than a call site.