Pinning trades continuous small exposure for one forced migration

The platform team brings a one-page proposal to the architecture review: every model call across all fourteen services moves from a floating alias to a dated snapshot, enforced by a lint rule that rejects any model string without a date suffix. The compliance escalation is cited on the first line, and everybody in the room agrees that it should not have been unanswerable.

The objection comes from the product engineering lead and it is not a reflex. Her team's tool-calling agent got measurably better in April when the alias moved: lookup_eccn argument errors fell from 4.1% to 1.4% over a weekend, with no work from anyone, and they only noticed because the tripwire flagged the segment moving in the good direction. Pinning gives that up. She also points at the deprecation notice already in the vendor's changelog for an older snapshot and asks who is going to do fourteen migrations in the eight weeks before a retirement date that nobody in the room chose.

Both positions are correct about the thing they are looking at, and the proposal as written is wrong in a specific way: it treats pinning as the removal of a risk when it is a change in the risk's shape.

The exposure does not go away, it changes distribution#

With a floating alias you are exposed continuously and in small increments. The provider repoints the alias, some fraction of your behaviour moves, your tripwire finds it within a day if you have one, and you deal with a bounded problem while it is small — sometimes an improvement you did nothing to earn. You are never surprised by a migration because you never do one, and you can never demonstrate what the system did last month.

With a pin you are exposed on one date, all at once. Behaviour is stable until the vendor retires the snapshot, at which point every service on it has to move, and the qualification work you deferred for eighteen months arrives as a single project. The exposure has been concentrated rather than removed, and concentrated exposure has a worse tail: fourteen simultaneous migrations under time pressure is exactly the situation in which someone ships an unqualified prompt change to make a date. A third arrangement sits between the two and appears in the last column below: a canary lane keeps the pin in force for almost all traffic while a small slice of requests goes to the candidate snapshot, so a new version is measured on real work before it becomes everybody's.

The answer differs by surface, and one surface is not negotiable#

Pin where you would have to demonstrate behaviour after the fact, or where a silent change costs more than a scheduled migration. The compliance assistant qualifies on both counts: its answers get quoted to other people's lawyers, and the previous chapter's escalation is what an unpinned version of it costs. Anything under a regulatory obligation, anything whose output is contractually specified, and anything a customer can screenshot and send to counsel belongs in the same category.

Leave the alias on surfaces where the cost of a bad week is a support ticket. Internal tooling, draft generation that a human always edits, exploratory features with a hundred users a day — these get the improvements for free, and the tripwire covers them. The product lead's tool-calling agent sits in an interesting middle: high volume, real customer impact, and a genuine record of unearned gains from alias movement, which argues for the canary arrangement rather than for either extreme.

The evaluation harness is the one place where there is nothing to weigh. It always pins, because an instrument that drifts with the thing it measures cannot attribute anything. If your judge model floats and your production model floats, a moved score has four possible causes instead of two, and the replay ordering from earlier in this topic loses its first step entirely — you can no longer hold the model fixed, because you have no fixed model to hold. Pinning the judge is cheap and it is what makes every other measurement in this topic mean something.

# config/models.yaml — policy is a property of the surface, with an owner and a date
surfaces:
  compliance_assistant:
    policy: pinned
    model: claude-sonnet-4-5-20250929
    retires_on: 2027-03-01
    owner: team-compliance
  ticket_router:
    policy: canary
    model: claude-sonnet-4-5-20250929
    candidate: claude-sonnet-4-6-20260714
    candidate_traffic: 0.05
    owner: team-support
  internal_notes:
    policy: alias
    model: claude-sonnet-4-5
    owner: team-internal
evaluation:
  policy: pinned          # judge and system-under-test both, always
  judge_model: claude-opus-4-1-20250805

What has to exist before a pin is a safety measure#

A pin is a decision to migrate later, so the machinery that makes migrations routine has to exist before the pin, not after the deprecation notice.

The qualification pipeline is the fixed set and the stratified segments already described, run as a paired comparison: the same inputs through the incumbent snapshot and the candidate, scored by the same pinned judge, reported per segment rather than in aggregate. Retained request bodies make this dramatically better, because you can qualify against several thousand replayed real requests rather than against 5,000 curated ones, and the segments that matter commercially are represented in their true proportions. Expect a qualification to take one to three weeks of elapsed time per surface, most of it waiting for enough replay volume and for someone to adjudicate the cases where the two snapshots disagree and the judge is unsure.

The deprecation calendar is unglamorous and it is the part that actually fails. Every pinned surface carries a retirement date, that date lives in the same tracker as everything else the team plans, and the migration is scheduled at the start of the window rather than the end. Fourteen services on one snapshot with a twelve-month window is not fourteen problems in month eleven if migrations are started in month three and staggered one per fortnight.

Adjudication is the part of qualification that people underestimate, so it is worth sizing. A paired run over 3,000 replayed requests typically produces disagreement between the two snapshots on 10% to 20% of cases, most of which are stylistic and irrelevant; the judge resolves the great majority, and somewhere between 60 and 150 cases end up needing a human to say which output is better. At three minutes each that is a day of somebody's attention per surface, which is affordable when it is scheduled and impossible when it lands in the last fortnight before a retirement date.

The canary lane is what converts a migration from an event into a rollout. Route 5% of traffic to the candidate, tag every evaluation record with the model version it used, and run the alerting rule with model version as a segment dimension — which the rule from earlier already supports, since it treats segments generically. A candidate that regresses shows up on 5% of traffic in a day rather than on 100% at cutover.

Written down, the migration for one surface is short enough to be a checklist rather than a project plan: replay a few thousand retained requests through both snapshots, diff per segment, adjudicate the disagreements, fix or accept what the diff shows, run the candidate at 5% for a week with the alert rule segmented by model version, promote, and update the retirement date in the tracker. Every one of those steps exists already if the earlier parts of this topic are in place, which is the real reason a pin is safe or unsafe. The proposal in the architecture review is not "pin or do not pin"; it is whether the team is prepared to own that checklist fourteen times over the next two years.

The failures pinning adds#

Staleness is the obvious one and it is not only about quality. Improvements you decline include safety behaviour, price reductions, latency work, and context-window increases, and older snapshots are frequently served on less-favoured capacity, so a two-year-old pin can drift upward in latency and downward in availability without any change in what it says.

The subtler failure is that prompts overfit to a pinned model. Every workaround your team adds for a quirk — the phrasing that stops it padding, the example that fixes its date format, the instruction not to hedge — is calibrated against one snapshot, and the longer the pin holds the more of these accumulate. Migration cost therefore grows with pin duration in a way that is invisible until you try, which is a good argument for migrating on a rhythm even when nothing forces you to. A team that moves every six months does small migrations forever; a team that moves every three years does one enormous one.

Environment divergence is easy to create and annoying to debug: developers on the alias, production pinned, and a behaviour that reproduces on nobody's laptop. Whatever the policy, it should be the same policy in every environment, with the model string coming from configuration rather than from a default in a client library.

Pinning also promises more than it delivers. A dated snapshot fixes the weights; it does not fix the safety classifiers layered around them, the serving stack, the tokeniser, the default sampling parameters, or the system-level instructions a provider may apply. Behaviour behind a dated snapshot can and does move, less than behind an alias, and the tripwire stays on either way. Anyone who reads a pin as permission to stop scheduled evaluation has swapped one silent failure for another.

The direction of travel is mildly encouraging: deprecation windows have lengthened, providers publish behavioural diffs between snapshots more often than they did two years ago, and there is early tooling for automated dual-inference comparison over live traffic. None of that removes the decision, and a proposal that pins fourteen services without an owner, a date, and a qualification lane per surface is a promise to have this argument again under time pressure.

Running it#

The apparatus in this topic is small enough to describe as a week. Every production response writes its resolved model identifier, prompt version and hash, and — on the surfaces where it is worth the legal work — its inputs, into a log with a retention you can name. A watcher query alerts when a new snapshot appears in traffic. At 03:00 a stratified tripwire covers every critical segment; on Sunday the full suite runs; both run again within the hour after any provider event. Verdicts go through a rule with a minimum sample, a severity floor, and a persistence requirement, and confirmed regressions page while everything else lands in a digest. When one fires, the frozen replay runs first, then the input slices, then the prompt's commit history. Pinned surfaces have owners and retirement dates in the tracker, and one migration is always in progress.

Start with the log field. Capturing model_resolved on every request costs an afternoon and thirty bytes, and it is the difference between dating the next incident to an hour and arguing about which week it began.