"A model that aces data it studied but flops on new data is useless. Honest testing is how we know a model actually learned."
Learning Objectives
By the end of this module, you will be able to:
- Split data into training and testing sets
- Explain why we never test on training data
- Read common evaluation metrics
1. Study Set vs. Exam Set
We split data into two parts:
- Training set (~80%) — the model learns from this
- Testing set (~20%) — unseen data to check if it truly learned
Key idea: Never test a model on the same data it trained on. That's like grading a student on the exact questions they memorized — they'd score perfectly without understanding anything. The test set must be new to the model.
Think of it like a driving test. You practice on quiet streets you know well (training), but the examiner takes you on roads you've never driven (testing). Only the unfamiliar roads prove you can actually drive, not just repeat a memorized route. The 80/20 split is a common starting point, but with very large datasets you might use 90/10, and with tiny datasets you lean harder on cross-validation (below) so you don't waste precious examples.
Explain like I'm new: Imagine you have 1,000 labeled photos of cats and dogs. You hide 200 of them in a drawer. The model studies the other 800 until it can tell them apart. Then you pull the 200 out and ask, "What are these?" Since the model has never seen them, its score is an honest measure of real learning — not memorization.
2. Evaluation Metrics
How well did it do? Depends on the task:
| Task | Common metric | Meaning |
|---|---|---|
| Regression | RMSE / R² | How close predictions are to reality |
| Classification | Accuracy | Share of correct predictions |
| Classification | Precision / Recall | Quality vs. coverage of a class |
The confusion matrix shows a classifier's right and wrong calls in a simple grid.
Simple example: Precision asks "when the model says spam, how often is it right?" Recall asks "of all real spam, how much did it catch?" A spam filter needs both — high precision (few good emails lost) and decent recall (little spam slips through).
Accuracy alone can lie. Suppose 99% of transactions are legitimate and only 1% are fraud. A lazy model that guesses "legitimate" for everything scores 99% accuracy — while catching zero fraud. That's why precision and recall matter: they focus on the rare, important class instead of hiding failures in a big pile of easy correct answers.
Common mistake: Reporting only accuracy on imbalanced data (where one class is rare). It can look impressive while the model is useless at the thing you actually care about. Always check precision and recall for the rare class.
3. Cross-Validation
To trust results more, cross-validation rotates which slice is the test set several times and averages the scores — a sturdier estimate than a single split.
In common 5-fold cross-validation, you cut the data into 5 equal slices. You train on 4 slices and test on the 5th, then repeat so each slice gets a turn as the test set. Averaging the 5 scores smooths out the luck of any single split — maybe one random split happened to put all the easy cases in the test set. It's like asking five different examiners instead of trusting one, then taking their average verdict. This is especially valuable when you have limited data, because every example gets used for both training and testing across the rounds — nothing is wasted.
Try this: In scikit-learn, cross_val_score(model, X, y, cv=5) returns five scores. Look at both their average and how spread out they are. A wide spread warns that your model's performance depends heavily on which data it sees — a sign it may not be stable yet.
Common mistake: Peeking at the test set while building the model (tuning until the test score looks good). That leaks the answers and inflates your score. Lock the test set away until the very end.
Key Takeaway: Split data into a training set (to learn) and a testing set (unseen, to judge) — never test on training data. Evaluate with task-appropriate metrics (RMSE/R² for regression; accuracy, precision, recall, confusion matrix for classification), and remember accuracy alone can mislead on imbalanced data. Use cross-validation for sturdier estimates, and never peek at the test set early.
Further Learning
Part of the "Data Science and Data Scientist" course.