Silent updates behind a model alias

Model configuration is usually one line, and the string in it is usually an alias: a name like family-a-latest that resolves, on the provider's side, to one specific set of weights and one serving configuration. Providers publish dated snapshot identifiers alongside the alias precisely because the pointer moves, and it moves on their release schedule rather than yours, announced in a changelog nobody on your team is subscribed to.

Everything the earlier chapters of this topic treated as a fixed target — instruction authority, refusal thresholds, how a system turn is weighted against later content, how reliably a schema holds — belongs to that set of post-training decisions rather than to the alias. So a roll produces the same class of divergence a migration produces, arriving without a deploy, without a diff and without a before. This chapter is about detecting it, measuring what moved, and buying time in a way that has an end date attached rather than an assumption.

Nobody deployed anything, on your side#

Some changes do not even get a new snapshot identifier. A routing change that sends your traffic to a different serving stack, a safety classifier updated in front of the model, a quantised variant — the same weights stored at lower numeric precision so each server can carry more traffic — brought in under load: from your side these are indistinguishable from a version bump, and they produce the same behaviour shift.

Quantisation earns one extra step, because "lower precision" sounds as though it ought to be invisible and it is not quite. The model's choice at each position is a ranking of scores, and storing the weights with fewer bits shifts every score by a small amount. Where the leading token has a wide lead over the second, that perturbation changes nothing at all. Where two tokens sit close to tied, it can reverse them, and one reversed token early in a response carries the rest of the response somewhere else. So the effect is not a uniform blurring of quality. It is a small number of decisions flipping, concentrated on the inputs the model was least sure about — which are the inputs you care most about.

The one signal that is cheap and reliable is the model identity the API echoes back in the response, which most SDKs surface and most services throw away. Two responses to the same request, five days apart:

{"id": "msg_01H...", "model": "family-a-2024-11-06",
 "usage": {"input_tokens": 1412, "output_tokens": 96}}

{"id": "msg_01J...", "model": "family-a-2025-02-04",
 "usage": {"input_tokens": 1412, "output_tokens": 141}}

Logging that field as a dimension on every request costs nothing and turns "when did this start" from a forensic exercise into a query. Where a provider exposes a system_fingerprint, an opaque identifier for the serving configuration that produced the response, it captures the backend changes that arrive with no new dated identifier, which is exactly the class the model field alone will miss.

def call(messages, **params):
    resp = client.messages.create(model=CONFIG.model, messages=messages, **params)
    served = getattr(resp, "model", None)
    fingerprint = getattr(resp, "system_fingerprint", None)
    metrics.incr("llm.calls", tags={"alias": CONFIG.model, "served": served})
    if (served, fingerprint) != STATE.last_seen:
        events.emit("llm.identity_changed",
                    alias=CONFIG.model, previous=STATE.last_seen,
                    current=(served, fingerprint))
        STATE.last_seen = (served, fingerprint)
    return resp

An identity-changed event is the cheapest artefact in this chapter and the one that collapses an investigation, because it puts a timestamp on the provider's action next to the timestamps on your own metrics. Without it, the first hypothesis is always something you did, and the first thing anyone checks is the deploy log.

That is not a failure of imagination. It is a property of the evidence available. Debugging is a search over things that changed, ordered by what has a timestamp attached. Your deploys have timestamps. Your commits have timestamps. The provider's action, until something on your side records it, has none — so it is not merely low on the list of hypotheses, it is absent from the list, and no amount of care puts it there. Logging the served identity makes nobody smarter. It adds one timestamped event to the search space, and that event happens to be the one that matches.

The same change, arriving on the provider's schedule#

A cross-provider port and a silent alias roll are the same mechanism: a byte-identical prompt meets a different set of post-training decisions, and tool-call compliance, schema adherence, refusal thresholds and answer calibration each move independently. Everything established earlier in this topic applies without modification. What differs is the conditions under which you meet it.

The middle row is the expensive one. A port gives you both arms at once, so the comparison is a matter of running them. An alias roll gives you the new arm and whatever you happened to record about the old one, and if the frozen evaluation set was scored once at migration time and never since, your baseline is a number from eight months ago on a set that may have been edited in the meantime.

The last row deserves attention too, because prompt patches are model-specific by construction. A rule restated near the end of the input, a prefill, a stripped reasoning block — each was written against a measured divergence on a particular snapshot, and after a roll each is wording tuned for weights that are no longer being served. Some become unnecessary and cost tokens; a few become actively wrong.

Two snapshots, one frozen set#

Establishing what moved is the two-arm run from the start of this topic with the arms redefined: instead of provider A against provider B, it is the pinned old snapshot against the new one, on the same frozen sets, with the same decoding intent and the same number of repeats.

PARAMS = {"temperature": 0, "max_tokens": 900}

before = run_arm(client, "family-a-2024-11-06", PARAMS, repeats=5)
after  = run_arm(client, "family-a-2025-02-04", PARAMS, repeats=5)
policy_before = run_arm(client, "family-a-2024-11-06", PARAMS, repeats=5, evalset=POLICY_60)
policy_after  = run_arm(client, "family-a-2025-02-04", PARAMS, repeats=5, evalset=POLICY_60)

For a support-triage prompt, that is 260 tickets plus a 60-case policy set, five repeats, two arms: 3,200 calls, about $11 at that prompt's token profile, and twenty-five minutes with eight-way concurrency. The result is the diff the changelog did not give you.

The 3,200 is three multiplications stacked. The two sets together are 260 tickets plus 60 policy cases, which is three hundred and twenty cases. Five repeats makes it that many five times over. Two arms doubles it again, because each snapshot has to see every case on every repeat — the two arms being otherwise identical is the entire point of running them. None of that arithmetic is specific to a provider rolling an alias. It is the same shape as costing a two-provider comparison, with the arms relabelled.

Read as a whole, the February snapshot is a better model on the axis the product is named for and has its safety boundary drawn in a different place. The refusal shift is not diffuse: it is concentrated in in-policy medical-adjacent inputs, which is exactly the category a deliberately constructed policy set exists to hold at a proportion traffic never would. The 260-case set sampled from real tickets contained four such cases and showed a two-point wobble that would have been dismissed as noise.

The distance between the twenty points on the policy set and the smaller move visible in production is not a contradiction, and the explanation is what each population contains. The policy set holds thirty cases chosen to be in-policy medical-adjacent, so a shift in how the model treats that category lands on every one of them. A production tag is a mixture: coverage questions, genuine clinical questions the model is supposed to decline, and whatever else a tagging rule swept in. The same shift applied to a mixture moves the mixture's rate by the shift multiplied by the in-policy share of it, which is smaller by construction. That dilution is the whole reason to build a policy set. Not because it is more realistic than production — it is markedly less so — but because it is undiluted, and an undiluted category reads a change several times louder.

The output-token increase is worth pricing rather than noting. At 1.4 million triage calls a month, 47% more output is roughly 63 million extra output tokens, which at $15 per million is $945 a month arriving with no change on your side. That is a smaller effect than the refusals and it is the kind that never gets attributed, because it shows up on an invoice a month later next to everything else.

The chain is short. Mean output went from 96 tokens to 141, which is 45 more per call, and 45 on a base of 96 is the 47%. Multiplying the 45 by 1.4 million calls gives roughly 63 million extra output tokens; dividing by a million and multiplying by the $15 rate gives $945. The same answer is reachable from the percentage or from the per-call difference, and the difference is the safer route — a percentage of a mean is easy to apply to the wrong base, and it was the extra tokens per call that the invoice was actually built out of.

When more than one snapshot sits between your last good baseline and now, the same run bisects: score the intermediate snapshots on the policy set only, since it is the small set and the one carrying the signal, and the point where the refusal rate steps is the release that did it. That also tells you which changelog entry to read, and provider notes about a revised safety policy are much easier to act on once you know they are the cause rather than a candidate.

Pinning buys months, and the bill has a date on it#

Pinning is the correct immediate move and the wrong resolution, and the gap between those two is where teams lose the following year.

The belief that makes an alias attractive is worth stating plainly so it can be examined: the alias gives you the newest model, and the newest model is better. It is half true, and the half that holds is narrower than it sounds. A provider ships a snapshot because it is better on the axes they measured, which are broad benchmarks averaged over many tasks. Your product runs one prompt over one distribution and leans on a handful of behaviours, and "better on average" carries no commitment about any of them — the February snapshot in the table above is genuinely better at triage and has moved its safety boundary through the middle of a category this product is sold on. The other half of the belief does not survive at all. Even where a new model is better for you, an alias means the change lands on a date you did not choose, behind an announcement nobody read, with no baseline on the old side, and those three facts routinely cost more than the improvement was worth.

Dated snapshots retire. The typical shape is a deprecation notice followed by a retirement date somewhere between six and twelve months out, after which requests to that identifier fail or are silently served by a successor, and the notice arrives by email to whoever created the API key. A pin is therefore a dated loan: you have bought time, the size of the loan is written on the provider's deprecation page, and the interest is that every roll you sit out accumulates into the jump you eventually have to make.

The version that works is a pin with a ticket attached, containing the harness output, the patch that closes the gap on the newer snapshot, and a date before the retirement date. Moving to the February snapshot deliberately in March, with the coverage-question examples added and the frozen sets re-scored, is two days of work. The same move forced in November, across four accumulated rolls, is a three-week migration with a deadline on it.

Four rolls at once costing more than four times one roll is not a matter of volume. It is attribution. One roll gives two arms separated by a single provider action, so a delta on an axis was caused by that action, and a patch written to close it can be measured against it. Four rolls give two arms separated by four actions, and a delta might belong to any of them or to a combination. Patching then stops being measurement and becomes search: try wording, re-score, and be unable to say which of the four changes the wording was addressing. Bisecting on the policy set recovers part of that, and only for as long as the intermediate snapshots are still being served — which is precisely what a retirement deadline is taking away.

The harness that catches the next one first#

By this point in the topic the infrastructure already exists — the frozen sets, the policy set, the four-axis scorer, the per-error-type comparison, the thresholds in the abandon criteria file. Standing it up as a monitor means running it on a trigger rather than on a migration.

Two triggers rather than one, because the identity event catches rolls that come with a new identifier and the weekly run catches the changes that do not. The alias arm runs against family-a-latest deliberately, since the point is to observe what the pointer is doing while production continues on a pinned identifier.

# monitors/triage-model-drift.yaml
alias: "family-a-latest"
pinned_in_production: "family-a-2024-11-06"
sets:
  - evalsets/triage-260.json@a91f3c7d02b1
  - evalsets/policy-60.json@5e0c11ba9f34
repeats: 5
triggers: [identity_changed, "weekly:mon-03:00"]
alert_if:
  - {metric: refusal_rate_in_policy, threshold: "> +2 pt vs baseline"}
  - {metric: schema_valid_first_attempt, threshold: "< -0.1 pt vs baseline"}
  - {metric: tool_compliance, threshold: "< -1 pt vs baseline"}
  - {metric: mean_output_tokens, threshold: "> +15% vs baseline"}
owner: "support-platform"

Cost is the objection that gets raised and it is small: 3,200 calls weekly is about $48 a month, against an incident that consumed a day and a half of two engineers plus five days of elevated refusals on a category the product is sold on. The output-token line alone was running at $945 a month undetected.

That $48 is the same 3,200 calls as the one-off run, at the $11 the two-arm comparison cost, repeated once a week — a month holding four and a bit weeks is where the figure comes from. Only one of the three lines on the other side of that comparison is denominated in the same units, and it alone is close to twenty times the monitor's cost. The other two are the ones that would decide it anyway.

Those thresholds are worth checking against the incident that has already happened, because a threshold never tested against a real event is a guess dressed as a commitment. The output-token move was 47% against a threshold of +15%, so it fires with room to spare. The in-policy refusal move was twenty points against a threshold of two, which is ten times over. The schema move was just over a third of a point against a threshold of a tenth. All three would have paged on the Tuesday morning. That is what makes the file worth committing — not that the numbers look reasonable in the abstract, but that the one incident anybody has lived through trips every one of them.

The offline harness pairs with a production signal that is cheaper still, since the instrumentation from earlier chapters is already emitting it. Refusal rate by ticket tag, validator rejection rate, repair rate and mean output tokens, each tracked per served snapshot identifier, give a canary for free: route 5% of traffic to the alias and the rest to the pinned identifier, and compare those four series across the split. The offline sets tell you what changed on inputs you chose; the canary tells you what changed on the traffic you actually get, including the categories nobody thought to put in a set.

The categories a frozen set does not contain#

A frozen set is a fixed sample and a roll may land somewhere it does not reach. February's refusal shift was caught because someone had built a policy set that oversampled medical-adjacent inputs, and a shift in, say, how the model handles tickets written in Portuguese would have passed every threshold in that file. The monitor narrows the blind spot rather than closing it, and the honest maintenance task is adding a category to the policy set each time production surfaces one.

Alert thresholds also cut both ways. A roll that improves refusal calibration by three points trips the same alarm as one that degrades it, which is correct — an unrequested behaviour change is worth knowing about in either direction — but it means the response to an alert is a run of the comparison rather than an automatic rollback, and a team that automates the rollback will pin itself onto an old snapshot for a change that was an improvement.

And some of what looks like a model update is not one. Load-driven routing, a provider-side safety filter, a regional endpoint serving a different build: each produces axis deltas with no snapshot identifier change, and the fix is a support ticket to the provider rather than a prompt patch. The fingerprint field distinguishes them where a provider exposes it, and where none does, a run of the same comparison against a pinned identifier is the test — if the pinned arm has moved too, the change is not in the weights.

An alias is a convenience that trades a decision about when you absorb change for never having to make it, and the trade is only safe if you can see the change arrive and measure it against something. The smallest useful version of that is a week's work and it starts from whatever you are already calling. Find the aliases in your configuration and write down which dated snapshot each currently resolves to. Log the served identity on every response. Freeze the evaluation set you already have, hash it, and build the thirty-to-sixty-case policy set beside it for the categories where a refusal shift would hurt your product. Score your current model on both, with repeats, and store the result as a baseline with the snapshot identifier attached.

At that point the next roll, or the next migration, has a before to be compared against — which is the thing the February team did not have and spent a day and a half constructing under pressure. Then pick the migration you have been putting off, write the abandon criteria before you run it, and find out what actually breaks.