Senior · 6 chapters
Drift detection
Provider model updates, data drift, prompt rot; scheduled re-evaluation
Without versioned evaluations and attribution signals, silent provider changes become customer-reported incidents that are difficult to roll back or explain.
Read these first
What you will know
Detect a quality regression caused by a provider-side model update
A fixed evaluation set with versioned outputs can expose changed behavior even when schemas, latency, and error rates remain normal.
Schedule re-evaluation rather than waiting for a complaint
A cadence tied to traffic and model-release risk catches degradation early without requiring every production request to be manually reviewed.
Distinguish data drift from prompt rot from a model change
Each cause demands different evidence: input-slice statistics, prompt-version comparisons, or replayed evaluations against pinned model identifiers.
Alert on the drift before users report it
Thresholds need sample-size and severity controls so natural variance does not create alert fatigue while meaningful segment regressions remain visible.
Reproduce a no-deploy regression using versioned prompts, inputs, model identifiers, and output traces
Provider aliases, mutable prompts, and missing request context can otherwise make the original behavior impossible to recover or compare.
Chapters
- 1
Vaguer answers behind green dashboards is a measurable event
Support reports the assistant has become vaguer over a week; latency, error rate, and schema validity are all normal and your team shipped nothing. Establishing a provider-side change means re-running a fixed evaluation set against the alias production calls and against a pinned snapshot from before, then comparing the stored outputs rather than the scores alone — an update shows up as shifts in length, hedging, and refusals.
- 2
A suite that runs on your commits misses the provider's changes
Evaluation runs in CI on every pull request and nowhere else, so three quiet weeks hide a model swap behind the alias: CI fires when you change something, and this changed on somebody else's schedule. Running 5,000 cases nightly at £400 is not the answer either. Covers a stratified tripwire set that covers every critical segment daily, with the full suite weekly and after any provider event.
- 3
Data drift, prompt rot, and a new model need different evidence
Invoice extraction drops after a customer's new billing system emits differently formatted PDFs; a summariser turns longer and hedgier over a weekend; a classifier misfiles cases after marketing renames two product tiers. Only the first is data drift. Learn the replay that separates them — frozen inputs against a pinned snapshot and against today's production call — and which of the three a nightly fixed set structurally cannot catch.
- 4
A two-point drop on thirty cases is noise, not a regression
Any of forty segments falling two points fires the alert, it fires most days, the channel has been muted for a month, and a genuine 9-point regression on a 60-case segment ran unnoticed for eleven days. Both symptoms come from one error: a rule stated in raw score points treats a 30-case segment like a 4,000-case one. Covers minimum sample, severity floors, and why moving the threshold to six points does not fix it.
- 5
A floating alias makes last month's answer unreproducible
A customer escalates an answer from the 3rd. You call the model through a floating alias, prompts are edited in place in an internal tool, and request bodies were never retained — only response text and latency. Work out what can be established, what can only be speculated, why a correct answer today confirms nothing, and which four changes — pinned snapshot, logged resolution, immutable prompt version, retained inputs — make the next one answerable.
- 6
Pinning trades continuous small exposure for one forced migration
Platform wants every call pinned to a dated snapshot; product engineering objects that it means missing improvements and migrating on the vendor's deprecation schedule. Pinning without a qualification pipeline and a deprecation calendar does not deliver predictability — it converts a continuous, small exposure into a fleet-wide move on a date nobody chose. Covers what each side gains and gives up, and whether the answer changes by surface.
External resources
- How is ChatGPT's behavior changing over time?arxiv.org
The reference measurement that provider updates silently move behaviour.
- Detecting Model Drift with Red Teaming | Promptfoopromptfoo.dev
This is a substantive walkthrough of drift detection using repeatable red-team baselines, scheduled CI scans, regression thresholds, and custom behavioral tests, despite being implemented with Promptfoo.
- Arize AI Docsarize.com
Shows how to monitor provider updates, data drift, prompt rot, and schedule re-evaluations.
- Quality Monitoring: Drift Detection and Regression Alerts for LLMsmbrenndoerfer.com
Concrete detectors and alert thresholds for scheduled re-evaluation.
- Why Monitor Model Drift in Production: A Practical Guide | MLflowmlflow.org
This is a substantive explainer covering data, concept, prediction, schema and training-serving drift, with concrete statistical tests, thresholds, layered monitoring, and operational alerting guidance beyond merely promoting MLflow.
- Evidently AI Tutorial-Open Source ML Models Monitoring and ObservabilityKrish Naik
Hands-on drift reports you can schedule against a golden set.
- ML Drift: Identifying Issues Before You Have a ProblemFiddler AI
Taxonomy of drift types and how each shows up in metrics.
- How to Evaluate Tool-Calling AgentsArize AI
Walks through evaluating tool-calling agents and detecting regressions after model, data, and prompt changes.