Model aliases and build identifiers

Every hosted model call names a model, and the string you write is not the thing that answers. It is a friendly name that resolves, at request time, to a specific set of weights running on a specific serving configuration, and a provider is free to change what it resolves to. This is a deliberate feature — it is how improvements reach every customer without a migration — and it means the one field most services record about a model call is the one field guaranteed not to change when the model does.

That guarantee is not an accident of implementation, and following why makes the rest of the chapter mechanical. A provider wanting to move every customer onto a better build without every customer editing a configuration file needs the name in the request to keep working while pointing somewhere new. Resolution therefore has to happen on the provider's side, at request time, after your process has finished assembling the body and put it on the wire. So the request body cannot possibly contain the resolved identity: at the moment it was written, the resolution had not happened, and the string in it is the string you chose. There is exactly one place in the exchange where the answer exists, and it is the reply.

The consequence is a question that cannot be answered after the fact unless somebody prepared for it: a quality number moved overnight with no deploy behind it, and was that the weights, the serving stack, or run-to-run variance? This chapter separates the three identifiers involved, works out which of them are readable and where, covers what pinning a dated snapshot buys and costs, and gives the logging and the query that turn the question into a lookup.

The name, the build, and the stack#

There are three distinct things a provider can identify, and they change on different schedules.

The friendly name is what you write in the request: a short, memorable, marketing-stable string. For a hosted provider it is usually an alias, and aliases are how you get improvements without a migration — the vendor points the alias at a newer build and everybody's requests move over. On self-hosted infrastructure the equivalent is a floating tag like qwen3-8b:latest, which behaves identically and for the same reason.

The build or snapshot is the specific set of weights behind that name at a moment in time. Providers surface this differently. OpenAI resolves the alias and returns the concrete snapshot in the response's model field, so a request for gpt-4o comes back saying gpt-4o-2024-08-06; if the alias is repointed, that field changes and your request body does not. Anthropic's Messages API also echoes the served model in model, with current model IDs being fixed strings like claude-opus-5 and older ones carrying explicit date suffixes like claude-haiku-4-5-20251001 — which is why pinning a dated ID where one exists is the difference between choosing your migration date and having it chosen. Self-hosted, the build is a weights file digest or a Hugging Face revision SHA, and nothing reports it to you unless you make it.

The three are worth keeping apart because a change in any one of them produces the same symptom on your dashboard and requires a different response, and only two of the three leave a trace. The serving stack is everything else that produces the numbers: kernel library version, driver, accelerator model, quantisation scheme — how far the weights have been compressed from their trained precision to a smaller number format for speed — tensor-parallel degree, and whatever routing decides which of those you get. Chapters 2 and 3 established that all of these change the arithmetic. Some providers expose a coarse handle on it — OpenAI's system_fingerprint is documented as an identifier for the backend configuration, intended to be read alongside seed so you can tell when the backend moved. Others expose nothing, and self-hosted you have the container digest and the node's driver version if you bothered to record them.

{
  "id": "chatcmpl-9tGx2K1QpVm4nDyRb8sLwZfEa",
  "object": "chat.completion",
  "created": 1739812655,
  "model": "gpt-4o-2024-08-06",
  "system_fingerprint": "fp_a7d06e42cc",
  "choices": [{ "index": 0, "message": { "role": "assistant", "content": "…" },
                "finish_reason": "stop" }],
  "usage": { "prompt_tokens": 1841, "completion_tokens": 96, "total_tokens": 1937 }
}

Everything you need to answer "did the model change?" is in the response and none of it is in your config.

The rows shade from "you control it" at the top to "nobody logs it" at the bottom, and the middle two rows are the ones worth capturing, because they are the only place where "the model changed" leaves a fingerprint you can diff.

Pinning moves the risk rather than removing it#

Once you can see that aliases move, the obvious response is to stop using them: request the dated snapshot, and the weights under you never change. That is the right default for anything whose output feeds an evaluation baseline, a cached artefact, or a contractual commitment, and it costs nothing at request time.

What it costs is later. Snapshots are retired on a published schedule, and a retired model ID does not degrade gracefully — it returns a 404, and every request that names it fails.

The hard failure is the correct behaviour rather than a rough edge, and seeing why is what stops it being reported as one. The value a pin buys is the promise that these exact weights answered your request. A provider that quietly served the nearest surviving build when the pinned one retired would be honouring the request and breaking the promise, and it would do so invisibly — reproducing precisely the silent substitution the pin was purchased to prevent. Refusing is the only response that keeps the guarantee intact. The cost of a guarantee is that it has to be enforceable, and here enforcement looks like an outage on a date somebody could have read.

Pinning therefore converts a silent quality change into a scheduled migration, which is a much better failure mode and still a piece of work somebody has to own. Teams that pin and then never look at the deprecation notices get the worst of both: no free improvements, and an outage on retirement day.

The workable arrangement is to pin deliberately and asymmetrically. Pin the snapshot in anything whose value depends on stability — evaluation harnesses, cached responses, recorded cassettes, regression baselines — and record the pin somewhere a dependency-update job can see. Leave the alias in production if you would rather take improvements as they come, and accept that you are then obliged to detect the change rather than approve it, which is what the logging below is for. What does not work is pinning in some places and not others without writing down which is which, because the day an evaluation and production disagree, nobody will be able to say whether that is a bug or the configuration working as designed.

There is a check worth running before trusting a pin, because "we pinned it" is a belief about a configuration file and the file cannot confirm itself. Some strings that look like snapshots are aliases, and an alias accepted without complaint is indistinguishable from a snapshot accepted without complaint until something moves. What settles it is one request: send the string you pinned and compare it against the model field of the reply. If they match exactly, the name you wrote resolved to itself and the pin is real. If the reply names something else, you pinned a pointer and the guarantee you thought you had was never in place. That check costs a single call and it is the only evidence available, since the resolution happens on the far side of the network by construction.

Self-hosting does not exempt you, and for exactly the same reason rather than a parallel one. A floating tag on a model repository resolves to whatever the latest commit is at pull time, and model repositories do get updated in place — a corrected tokenizer config, a re-quantised weight file, a fixed chat template. Pin the repository revision SHA rather than the branch, record the digest of the weight files at startup, and emit that digest as part of the service's identity alongside the container digest. Recording it once at process start and attaching it to every request record is enough; it cannot change without a restart.

Log it on the response path#

The instrumentation is small and the one detail that matters is where it reads from. Recording the model name from your configuration object tells you what you asked for, which you already knew. The record has to come off the response.

import time, logging, json

log = logging.getLogger("model.calls")

def call_model(client, request_id: str, body: dict) -> dict:
    started = time.time()
    response = client.chat.completions.create(**body)
    elapsed_ms = int((time.time() - started) * 1000)

    log.info(json.dumps({
        "request_id": request_id,
        "ts": time.time(),
        # what we asked for
        "model_requested": body["model"],
        "params": {k: body.get(k) for k in
                   ("temperature", "top_p", "seed", "max_tokens", "stop")},
        # what actually served it — read off the response, never off config
        "model_served": response.model,
        "system_fingerprint": getattr(response, "system_fingerprint", None),
        # our own deployment identity, for the self-hosted / self-inflicted causes
        "service_image": os.environ["IMAGE_DIGEST"],
        "prompt_template_version": TEMPLATE_VERSION,
        # the response, and enough to compare two of them
        "finish_reason": response.choices[0].finish_reason,
        "usage": response.usage.model_dump(),
        "output_sha256": sha256(response.choices[0].message.content.encode()).hexdigest(),
        "latency_ms": elapsed_ms,
    }))
    return response

model_requested and model_served are both present on purpose. Their divergence is the whole signal: the day they stop agreeing in the way they used to is the day the alias moved. Logging only the served value works too, but keeping both makes the alias indirection visible in the data rather than something you have to remember.

output_sha256 rather than the output itself keeps the log cheap and avoids putting generated customer content in a logging path that may have a different retention and access policy than your primary store. A hash is enough to answer "were these two runs identical", which is chapter 4's first and cheapest metric. It is enough because that question only ever needed equality, and a hash preserves equality exactly: identical inputs give identical digests, and two different digests prove two different outputs. What it does not preserve is anything else — a hash cannot tell you where two outputs diverged, so the first-divergence index of chapter 4 needs the text and this field does not supply it. Storing the digest by default and the text deliberately is the split that follows from that. When you need the text, you need it under whatever privacy regime applies, which is a decision to make deliberately rather than by defaulting a log field.

The same values belong on a metric as well as in a log line, because a log answers a question you thought to ask and a metric notices something you did not. Emitting a counter labelled by model_served and system_fingerprint costs nothing at the cardinality these fields have — a handful of distinct values at any time — and lets you alert on the appearance of a new one. The cardinality caveat is doing real work in that sentence. A metrics system stores one time series per distinct combination of label values, so labelling a counter by a field with thousands of values creates thousands of series and the cost is paid forever, which is why request IDs and user IDs never belong on a metric. A resolved build identifier and a backend-configuration string take a handful of values across a whole fleet, so the same instrumentation that would be reckless with one field is free with these two. "A model revision we have never seen before is now serving 4% of traffic" is a page-worthy event on some systems and a Slack notification on most, and either is better than finding out from a quality graph two days later.

With that record in place, the question becomes a query rather than a debate:

SELECT date_trunc('day', to_timestamp(ts)) AS day,
       model_served,
       system_fingerprint,
       count(*)
FROM model_calls
WHERE ts > extract(epoch FROM now() - interval '14 days')
GROUP BY 1, 2, 3
ORDER BY 1;

Three outcomes, and there are exactly three because two binary fields with one impossible combination leave three. Either the weights moved, or the weights held while the configuration around them moved, or both held. The third is a residual rather than a diagnosis: it says the cause is not on the provider's side of the boundary these two fields describe, which leaves your own inputs and the per-request arithmetic that nothing reports. That is a narrower answer than the first two and it is still a direction, which is the whole of what forty minutes of speculation failed to produce.

Each outcome sends you somewhere different. If model_served changed overnight, the weights moved and your quality delta is a model change — go read the provider's release notes, pin the previous snapshot if one is still available, and re-baseline. If model_served held and system_fingerprint moved, the weights are the same and the serving stack is not, which is chapter 2's mechanism at a coarser granularity than per-request. If both held, no model change occurred and the delta came from your side or from run-to-run variance, which is where the last chapter of this topic goes.

A matching revision is weaker evidence than it looks#

An identical model_served on both days means the same weights answered both calls. It does not mean the two calls were processed identically.

Providers layer things around the model that are not the model: safety classifiers that can decline a request, routing that sends you to a different region or hardware generation, quantisation changes on the serving side, and default parameter values that are documented but not frozen. A build identifier typically covers the weights and says nothing about any of that.

The boundary that field draws is worth making explicit, because it explains both what the log entitles you to conclude and where the conclusion stops. A build identifier names an artefact — a specific set of trained numbers — and an artefact identifier can only report on the artefact. Everything sitting in front of the weights or behind them is a different component with a different release cycle, and none of it is described by a name for the weights. So a stable build identifier is a strong claim about one layer and no claim at all about the request path around it, and a behavioural change with an unmoved identifier is not a contradiction to be explained away. It is the field working correctly and reporting on the only thing it was ever about. Anthropic's newest models illustrate the sharp end of this — sampling parameters that older models accepted are now rejected outright with a 400, which is a behavioural change to the API surface itself rather than to the weights.

The system_fingerprint-style field is coarser than people assume in the other direction too. It groups configurations rather than identifying a machine, so two requests can share a fingerprint and still have run on different GPUs in different batches, which is exactly the variation chapter 2 measured. That follows from what the field is for: a provider runs many machines on one configuration, so a value naming the configuration necessarily covers all of them, and the batch your request landed in is not part of a configuration at all — it is decided per step by a scheduler. A stable fingerprint is not a promise of reproducibility; it is the absence of one specific kind of change.

And a changed revision is not by itself a regression. Alias repointing usually moves you to a better model on average, and a 3-point drop on one graded set can coexist with improvements everywhere else. What the log entitles you to say is "the weights changed on Thursday night", which is a fact. Whether that was bad for you is a separate question your own graded set answers.

Two fields on the response path, then, and a query over them, convert the most common overnight mystery in this area into one of three answers. None of the three is free — a moved build means a re-baselining project, a moved fingerprint means the serving arithmetic changed underneath a stable model, and an unchanged pair means the cause is on your side — but each of them is a direction, which is what forty minutes of speculation in a channel never produces.

The good day is the one where model_served moved, the release notes confirm it, everyone re-baselines and goes to lunch. The bad day is the one where every identifier matches, the parameters match, the timestamps are three days apart, and the answer a customer is complaining about still cannot be reproduced by anyone on the team. At that point the metadata has ruled out the model and the parameters, and there is exactly one input left that nobody has actually compared, because everybody believes they already logged it. They logged the user's question. The model was never sent the user's question.