"A good model finds the real pattern — not too simple to miss it, not so complex it memorizes noise. It's a Goldilocks problem."
Learning Objectives
By the end of this module, you will be able to:
- Define overfitting and underfitting
- Recognize the symptoms of each
- Aim for models that generalize
1. The Goldilocks Problem
| Underfitting | Just Right | Overfitting | |
|---|---|---|---|
| What happens | Too simple; misses the pattern | Captures the real pattern | Too complex; memorizes noise |
| Student analogy | Didn't study | Understood the ideas | Memorized answers, not concepts |
| On new data | Poor | Good | Poor |
Key idea: Both extremes fail on new data — for opposite reasons. Underfitting never learned enough; overfitting learned too much, including the random noise. The sweet spot in the middle is what we want.
Picture drawing a line through a scatter of dots. Underfitting is a flat, lazy line that ignores the obvious upward slope. Overfitting is a wild, wiggly curve that snakes through every single dot, including the ones that are just measurement errors. The "just right" model is a smooth curve that follows the real trend while ignoring the random jitter — the shape you'd draw by eye if you squinted.
Explain like I'm new: "Noise" is the random, meaningless part of data — a sensor hiccup, a typo, one unusually rainy day. The real "signal" is the pattern that repeats. Overfitting is memorizing the noise as if it mattered; when new data arrives with different noise, the model is lost.
2. Spotting Each
- Underfitting: poor on both training and test data → model too simple
- Overfitting: great on training, poor on test → model memorized
Simple example: Studying for a test: underfitting is barely reading the material; overfitting is memorizing last year's exact answers. Neither helps when the real questions are new — only genuine understanding does.
The tell is the gap between training and test scores. A tiny gap with both scores low means underfitting. A big gap — say 99% on training but 70% on test — screams overfitting. A small gap with both scores high is the goal. Watching these two numbers side by side is the single most useful diagnostic habit in machine learning.
Try this: After training any model, print its training score and its test score together. If training is much higher, it's overfitting. If both are low, it's underfitting. That two-number check tells you what to fix next.
3. Aiming for Generalization
Generalization — performing well on unseen data — is the whole goal. Ways to help:
- Get more (and cleaner) data
- Keep the model as simple as it can be
- Use cross-validation (Module 30)
- Stop training before it starts memorizing
More data is often the strongest fix for overfitting: with thousands of examples, the random noise averages out and the true pattern shines through, so the model can't memorize its way to a good score. Keeping the model simple (fewer features, less flexibility) also helps — a technique called regularization gently penalizes complexity so the model prefers the simplest explanation that fits.
Early stopping is another simple, powerful trick. As a model trains, its performance on held-out data improves for a while, then starts getting worse as it begins memorizing. Early stopping watches that held-out score and halts training right at the turning point — before overfitting sets in. It's like taking a cake out of the oven the moment it's done, rather than leaving it until it burns.
Real-world use case: A bank builds a loan-default predictor that scores 98% on its training customers but only 72% on new applicants — classic overfitting. The team gathers more historical data, drops a few noisy features, and adds regularization. The training score drops to 84%, but the test score rises to 82% — and that is the model that actually helps in the real world.
Data scientist tip: If your model is perfect on training data, be suspicious, not thrilled. Real data has noise; a flawless training score usually means overfitting. A slightly lower, honest test score is far more valuable.
Key Takeaway: Underfitting (too simple) and overfitting (memorizes noise) both fail on new data. Diagnose by comparing training vs. test performance — the gap between them is the tell. Aim for generalization — good results on unseen data — with more/cleaner data, simpler models, regularization, and cross-validation. A perfect training score is a warning sign, not a win.
Further Learning
Part of the "Data Science and Data Scientist" course.