"Which change actually made the model better? Without tracking, you're guessing. With it, you know — and can reproduce your best result."
Level: Intermediate · Time: ~11 min · Prerequisites: Module 5
Learning Objectives
By the end of this module, you will be able to:
- Explain why ML needs experiment tracking
- Know what to record for each run
- Understand model registries
- Recognize the tools
1. ML Is a Science of Experiments
Building a model means trying many variations — different features, hyperparameters, and data. Experiment tracking records each run's settings and results so you can compare them and reproduce the winner. It is genuinely easy to run fifty experiments over a week and, by Friday, have no idea which combination produced Tuesday's promising result. Manual notes in a spreadsheet break down fast: people forget to log a run, mistype a learning rate, or overwrite a file. Automated tracking removes that fragility by capturing everything as the code runs.
Explain like I'm new: It's a lab notebook for ML. A scientist writes down every experiment's setup and outcome so they can tell what worked and repeat it. Without notes, you'd forget which recipe made the best cake. Tracking is those notes, automated.
2. What to Track
For every run, record:
- Parameters: hyperparameters, model type, data version
- Metrics: accuracy, loss, and other scores
- Artifacts: the trained model file, plots
- Environment: code version, library versions
- The data version used
Key idea: Track enough to reproduce any run exactly — parameters + code + data + environment. The whole point is that six months later you can say "run #47 was best, and here's precisely how to recreate it."
Notice that these four things answer four different questions. Parameters and metrics tell you what you tried and how well it did; the artifact is the result you can reuse; the code and data versions are what let you rebuild it from scratch. Drop any one and reproducibility quietly breaks — the classic trap is logging a great accuracy number but forgetting which data snapshot produced it, leaving you with a score you can admire but never recreate.
Concept: The real payoff isn't the log itself — it's comparison. With every run recorded identically, you can sort by accuracy, filter to one data version, or plot how a metric moved as you changed the learning rate. Decisions stop being "I think this helped" and become "the numbers show this helped."
3. Model Registry
Beyond tracking experiments, a model registry stores your approved models with versions and stages (e.g., "staging," "production"). It answers: which model is live? which was live last month? how do we roll back? Think of tracking and the registry as two connected shelves. The tracking shelf holds every experiment — messy, exploratory, most of them dead ends. The registry shelf holds only the chosen models, each stamped with a version and a stage, ready to promote or retire. Together they give you a clean line from "we tried a hundred things" to "here is exactly what's running in production, and here's how to undo it."
Real-world use case: A team runs 60 experiments tuning a recommender. Tracking shows run #52 (a specific feature set + learning rate) scored best. They register that model as v3, promote it to production, and — because everything's versioned — could roll back to v2 in minutes if v3 misbehaves.
4. The Tools
| Tool | For |
|---|---|
| MLflow | Open-source tracking + registry |
| Weights & Biases | Popular tracking + dashboards |
| Neptune, Comet | Alternatives |
| Cloud ML platforms | Built-in tracking |
A few lines of code log each run automatically.
Try this: In your next training script, wrap it with MLflow: call mlflow.start_run(), then log_param for each hyperparameter and log_metric for each score. Re-run it three times with different settings and open the UI — you'll instantly see the three runs side by side, sorted by accuracy. That single habit pays off for years.
Common mistake: Tracking metrics but not the data and code version. Then you can't reproduce a great result because you don't know which data/commit produced it. Track all four: params, metrics, code, data.
✅ Checkpoint
- Why track experiments?
- What four things let you reproduce a run?
- What does a model registry manage?
Answers: 1) To compare many runs and reproduce the best. 2) Parameters, metrics, code version, and data version (plus environment). 3) Approved model versions and their stages (staging/production), enabling rollback.
Key Takeaway: Experiment tracking is a lab notebook for ML — logging each run's parameters, metrics, artifacts, code, and data version so you can compare runs and reproduce the winner exactly. A model registry versions approved models and their stages for promotion and rollback. Tools (MLflow, W&B) automate it — just remember to track code and data, not only metrics.
Further Learning
Part of "Zero to AI Engineer." Simplified from the AI Engineer curriculum.