"A model's accuracy on launch day is its best day — unless you watch it. The world changes, and silent decay is the danger."
Level: Advanced · Time: ~12 min · Prerequisites: Module 42
Learning Objectives
By the end of this module, you will be able to:
- Explain why deployed models need monitoring
- Understand data and concept drift
- Know what metrics to watch
- Set up alerts and a response plan
1. Models Decay Silently
Unlike a bug that crashes loudly, a model usually fails quietly — still returning answers, just increasingly wrong ones — because the world drifted away from its training data. Monitoring catches this before it hurts users. This is what makes ML monitoring different from ordinary software monitoring. A crashed web server throws a 500 error and pages someone within seconds. A decaying model throws nothing: every request returns a perfectly-formatted, confident prediction — that happens to be wrong more and more often. The system looks 100% healthy on a traditional dashboard while its actual value quietly erodes.
Explain like I'm new: A deployed model is like milk in the fridge. It looks fine long after it's gone off. Monitoring is checking the date and giving it a sniff — so you catch the problem before someone drinks it.
2. Drift: The Core Threat
| Type | What changes | Example |
|---|---|---|
| Data drift | The inputs shift | New user demographics |
| Concept drift | The relationship shifts | Fraud patterns evolve |
Both mean the model's learned patterns no longer match reality, so accuracy slips. A spam filter, for instance, meets data drift when a new style of newsletter floods inboxes, and concept drift when spammers deliberately change tactics so yesterday's tell-tale words are now used by legitimate senders too — the same features, a changed meaning. The two often arrive together, which is exactly why catching either one early is worth the effort.
Key idea: You often can't measure accuracy live (the true answer arrives later or never). So you watch proxies: is the input data distribution changing? are predictions shifting oddly? Drift detection is early warning before accuracy is confirmed to have dropped.
3. What to Monitor
- Model metrics: accuracy/error when true labels become available
- Data drift: are incoming feature distributions changing?
- Prediction drift: is the mix of predictions shifting?
- System health: latency, errors, throughput, cost
- Business metrics: is the model actually helping (clicks, revenue)?
The layers matter because they catch different failures. System health catches a crashed or slow service. Data and prediction drift catch a world that has changed. Business metrics catch the subtle case where the model is technically fine but no longer helping the product. A mature setup watches all three.
Real-world use case: A demand-forecasting model quietly degrades after a market shift. Monitoring flags that recent inputs look different from training data (data drift) and error is creeping up. The team is alerted, retrains on fresh data, and redeploys — before the bad forecasts cause a costly over-order.
Try this: For a model you've built, pick three numbers you could log on every prediction — say, one input feature's average, the predicted label, and the response time. Charting those over a week is already a working monitoring dashboard, and it costs almost nothing to stand up.
4. Alerts & Response
Monitoring is only useful if it triggers action: set thresholds, alert the team, and have a plan — retrain, roll back, or investigate. This closes the loop with CI/CD's Continuous Training (Module 45).
Concept: Beware alert fatigue. Thresholds set too tight cry wolf constantly, and people start ignoring them — so the real alert gets muted too. Tune thresholds so an alert means "a human should genuinely look at this now," and route different severities differently (a dashboard note vs. a 3 a.m. page).
Common mistake: "Deploy and forget." An unmonitored model can be wrong for months before anyone notices — often hurting some groups more than others. Set up monitoring and alerts before you ship, not after an incident.
✅ Checkpoint
- Why is model failure often "silent"?
- What's the difference between data drift and concept drift?
- Why monitor proxies instead of just accuracy?
Answers: 1) The model keeps returning answers, just increasingly wrong, as the world drifts. 2) Data drift = inputs change; concept drift = the input→output relationship changes. 3) True accuracy often isn't available live, so drift proxies give early warning.
Key Takeaway: Deployed models decay silently as the world drifts, so monitoring is essential. Watch for data drift (inputs change) and concept drift (relationships change), plus model metrics, prediction drift, system health, and business impact. Since live accuracy is often unavailable, track drift proxies, set alerts, and have a response plan (retrain/roll back) — set up before launch.
Further Learning
Part of "Zero to AI Engineer." Simplified from the AI Engineer curriculum.