Vaguer answers behind green dashboards is a measurable event

On a Thursday afternoon the support lead pastes four customer threads into the engineering channel with one line above them: "is something up with the assistant, it's gone woolly." The threads span nine days. Two customers use the phrase "generic", one says the answers "read like a disclaimer", and one asks whether they have been moved onto a cheaper tier. Nobody on the engineering side has any idea what they are talking about, because every instrument says the service is healthy.

The dashboards are worth reading carefully, because they are all telling the truth. Median latency is 1.9 seconds against 1.8 a fortnight ago, p95 is 4.1 against 4.0. The 5xx rate is 0.03%. JSON schema validation on the structured portion of every response passes at 99.7%, which is where it has sat since March. The last merge to the service was twelve days ago and it changed a log line. Every graph is flat, every alert is quiet, and four customers independently think the product got worse.

What the support lead has run into has a name. Drift is the gap that opens between the behaviour a system had when you last checked it and the behaviour it has today, in a week when nobody on your team changed anything: the model behind the name your code calls can be replaced by the company that serves it, the documents and questions arriving from real users can stop resembling the ones you built against, and a prompt written six months ago can fall out of step with a product that has kept moving. Drift detection is the standing practice of noticing that gap on purpose and on a schedule, rather than learning about it from four customers nine days after it started. It exists because the most consequential dependency in a system like this one has no version number in your lockfile; you can freeze your own code and pin every library you import, and the function on the other side of the vendor's API can still be swapped out behind a name that never changes. Where an engineer meets it in practice is a small permanent apparatus bolted to a service: a frozen list of test inputs re-run on a timer instead of on your merges, the outputs of every run kept rather than reduced to a score, a record of which model version actually served each request, and a rule that decides when the difference between two runs deserves somebody's attention. This topic assembles that apparatus a piece at a time, beginning with the run that turns a Thursday support thread into evidence.

That combination — quality complaints with no operational signal and no deploy of your own — is the shape of a provider-side model change, and it is a shape you can confirm rather than argue about. What the dashboards measure is whether the system responded: fast enough, without erroring, in the agreed format. A model that hedges every answer, opens with two sentences of caveat, and declines a question it used to answer is fully available, entirely schema-valid, and slightly faster than usual because it is producing less substance per token. None of the service level indicators — the SLIs, the small set of measured quantities an operations team promises to keep inside a bound — has a term for the content of the text.

The instrument is a fixed set whose outputs you kept#

The thing that turns "customers say it's worse" into evidence is a fixed evaluation set: a frozen list of inputs that you re-run on demand, where the inputs never change, so any difference in the results came from somewhere else in the stack. The evaluation harness from your CI pipeline already has most of this. What it usually lacks is the property that matters here, which is that every run archived its outputs verbatim and not only its scores.

Storing outputs sounds like a detail of the storage layer and it is the whole difference between being able to investigate this and not. A rubric grader compresses a 300-token answer into a number between zero and one, and that compression is designed to discard exactly the information you now need. A grader asked "does this answer the user's question correctly?" will score a crisp two-sentence answer and a correct-but-hedged nine-sentence answer identically, because both answer the question. The customers are not complaining that the answers are wrong. They are complaining about everything the rubric threw away.

So the run has three arms, and only the third is unusual. You re-run the set through the alias your production code calls — the undated model name such as claude-sonnet-4-5, which the provider keeps pointed at whichever concrete version it currently considers best — you re-run it against a pinned snapshot, one of those concrete versions named with its release date and frozen from then on, taken from before the complaint window, and you pull the archived outputs from a run you did before the window opened. The alias arm tells you what production does today. The pinned arm tells you what the old model does today, which controls for everything on your side that might have shifted — a library upgrade, a changed default parameter, a retrieval index that has grown. The archive tells you what the old model did back then, which controls for drift in the harness itself.

# evals/drift_probe.py — three arms over one frozen input set
import json, pathlib
from harness import run_set, load_archived   # run_set -> list[dict] with "output" kept

CASES = pathlib.Path("evals/fixed_set_v7.jsonl")   # 240 cases, unchanged since March

arms = {
    "alias_today":  run_set(CASES, model="claude-sonnet-4-5"),
    "pinned_today": run_set(CASES, model="claude-sonnet-4-5-20250929"),
    "archived_jun": load_archived(run_id="2026-06-14T02:11Z"),
}

for name, rows in arms.items():
    pathlib.Path(f"out/{name}.jsonl").write_text(
        "\n".join(json.dumps(r) for r in rows)
    )
    print(name, "resolved_model:", {r["response_model"] for r in rows})

That last print line is the cheapest diagnostic in the whole exercise and most teams do not capture the field it reads. When you send a request naming an alias, the response body names the concrete snapshot that actually served it. Ask for claude-sonnet-4-5 in June and the response says "model": "claude-sonnet-4-5-20250929". Ask for the same alias in August and it may say something else entirely, and the difference is sitting in a field you have been discarding since the day you wrote the client wrapper.

{
  "id": "msg_01Qk3vRy8fT2mNxA",
  "type": "message",
  "role": "assistant",
  "model": "claude-sonnet-4-5-20250929",
  "stop_reason": "end_turn",
  "usage": { "input_tokens": 1412, "output_tokens": 233 }
}

Read the shape of the text, not the score#

With 240 cases in three arms you have 720 stored outputs, and the comparison that settles the question is a handful of cheap textual statistics computed over each arm. None of them requires a model to compute, which matters because the point of the exercise is to have a measurement that does not itself depend on the thing under suspicion.

The two left-hand columns agreeing is what makes the right-hand column mean anything. The old snapshot produces today what it produced in June, so the harness, the retrieval layer, the prompt, and the grader are all where you left them. The alias arm has diverged from both, which places the change on the provider's side of the boundary, behind a name that did not change.

Notice the bottom row. The rubric moved two points, which on a 240-case set is inside the run-to-run spread you would have measured when you set up the CI gate, so a monitor watching the score alone would have seen nothing worth reporting. Meanwhile hedging roughly tripled, refusals went from one in 240 to nine, and the proportion of answers leading with a direct response fell 23 points. Every one of those is a plain description of what four customers said in their own words. The score was not wrong so much as blind: the answers still answer, at greater length, with more caveats and fewer numbers.

These particular signals are not arbitrary. Length, hedging, refusal, and specificity are the four axes along which a provider's tuning changes most visibly, because they are the axes providers deliberately tune: helpfulness training, safety calibration, and the general effort to reduce confidently wrong answers all push in the direction of more caveats, more qualification, and more declining of questions near a policy boundary. A model that has been made more careful in general will look, on a corpus of specific product questions, exactly like a model that has become vaguer. The signals also share the property of being computable without judgement, which means the numbers do not change if you re-run the analysis next week with a different grader.

Computing these is unglamorous and the crudeness is a feature, because you want a signal you can explain to the support lead in one sentence.

HEDGES = ("it depends", "generally", "typically", "may vary", "i'd recommend consulting",
          "in most cases", "it's worth noting", "cannot guarantee", "please verify")

def shape(rows):
    out = [r["output"] for r in rows]
    sents = sum(t.count(".") + t.count("?") for t in out) or 1
    return {
        "median_tokens": sorted(r["usage"]["output_tokens"] for r in rows)[len(rows)//2],
        "hedges_per_100_sent": 100 * sum(t.lower().count(h) for t in out for h in HEDGES) / sents,
        "refusals": sum(1 for t in out if "i can't help" in t.lower() or "i'm unable" in t.lower()),
        "has_figure": sum(1 for t in out if any(c.isdigit() for c in t)) / len(rows),
    }

A hedge list of nine phrases will miss hedging it has never seen and will fire on a case where "it depends" is the correct answer. That is tolerable because you are not scoring individual responses with it; you are comparing three populations of 240 responses each, where the same crude rule is applied identically to all three, and a systematic miss affects every arm equally. The comparison is what carries the meaning, which is also why the archived run has to have been produced by the same code. If you improve the hedge list, re-run it over the stored outputs of all three arms rather than comparing new numbers against old ones.

Two things you can do the same afternoon#

Because the old snapshot answered the fixed set the way it always did, it is still being served, and your production code can name it directly. Changing one string in configuration puts customers back on the behaviour they had in June, usually within a deploy cycle, and that is available as an immediate mitigation whether or not you intend to stay there. The window in which it remains available is set by the provider's retirement policy for that snapshot rather than by you, which is the subject of an argument this topic returns to at the end.

The other thing is the vendor conversation, and the evidence you have makes it a different conversation from the one most teams have. "Our users say the model got worse" produces a request for examples and a long thread. A ticket containing 240 paired outputs from the same inputs, the resolved model identifiers from both arms, and a table showing refusals moving from 1 to 9 and direct-answer rate falling 23 points is a reproducible report against their system, and it is the kind of thing that gets routed to someone who can check whether an alias was repointed or a classifier retuned in that window. Attach the raw outputs. The aggregate numbers persuade a support engineer; the paired texts persuade whoever they escalate to.

What this does not establish#

The fixed set tells you that behaviour behind the alias changed. It does not tell you why, and the candidate mechanisms are more varied than "they shipped a new model". An alias can be repointed to a newer snapshot, which is the case where the resolved identifier changes and the diagnosis is trivial. A safety classifier sitting in front of or behind the model can be retuned, which raises refusals and hedging while the identifier stays exactly the same. A provider-side system instruction can be added or reworded. A serving stack change — different quantisation, different speculative decoding, a different batching regime — can shift outputs without any of the labels moving at all. Your three-arm run distinguishes none of these from each other, and it does not tell you the change was a regression in any absolute sense. Longer, more hedged answers with fewer specifics may score better on somebody else's evaluation, and your rubric holding steady at 0.86 is consistent with the new behaviour being defensible in general and wrong for your product in particular. What you have is a difference with a location, which is enough to open a support case with the vendor and enough to justify pinning while you decide.

It also says nothing about your real traffic. Your 240 cases were chosen months ago and they represent what you thought mattered then. If the change hits a segment your fixed set covers thinly — long documents, one language, a rare tool call — the arms may agree while a quarter of your customers see something worse. The production feedback loops you already have are the other half of this: they tell you that users are unhappy without telling you what moved, and the fixed set tells you what moved without telling you who is affected. Neither substitutes for the other, and running the fixed set is not monitoring your product, it is interrogating one component of it under controlled conditions.

The distinction worth holding onto is between this and a benchmark. A published benchmark score for a model is a property of the model; the numbers in that table are properties of your model, your prompt, your retrieval context, and your grader together. When the provider says the new snapshot improved on their evaluations, both statements can be true at once, and only one of them is about your customers.

The uncomfortable part of this chapter is the date on the archived run. The change happened somewhere in a nine-day window, four customers noticed before you did, and the only reason anyone ran the fixed set at all was a message in a support channel on a Thursday. The instrument worked. It simply was not switched on.