"Two sneaky problems can quietly ruin an analysis: gaps where data is missing, and extreme values that pull everything off course."
Learning Objectives
By the end of this module, you will be able to:
- Handle missing values sensibly
- Recognize and investigate outliers
- Understand why both distort results
- Judge data quality
1. Missing Values
Real datasets are full of blanks — a survey left unanswered, a sensor that dropped out. You have a few choices:
| Approach | When to use |
|---|---|
| Remove the rows | Few are missing, and randomly |
| Fill with a value | Use the mean/median, or a sensible default |
| Flag them | When "missing" itself is meaningful |
Key idea: There's no single right answer for missing data — it depends on why it's missing. Deleting too much loses information; filling carelessly invents fake data. Think before you patch.
The why is everything. If a temperature sensor failed at random, filling the gap with the average is reasonable. But if people skipped an income question because their income was high, those blanks aren't random — filling them with the average would quietly erase the very pattern you care about. Ask what the missingness itself is telling you before you decide.
Explain like I'm new: "Flagging" means adding a little marker column that says "this value was missing." You're not guessing the value — you're honestly recording that it was absent, which sometimes turns out to be the most useful signal of all (a customer who skips a field may behave differently).
Real-world use case: A hospital dataset had many blank "blood pressure" readings. Deleting those rows would have thrown away sick patients whose readings were skipped in an emergency — quietly biasing the data toward healthier people. The team instead flagged the missing values and discovered that "reading skipped" was itself a strong clue about how urgent the case was. What looked like a gap became a signal.
2. Outliers
An outlier is a value far outside the normal range — a $5,000 grocery bill, a 200-year-old age. Some are errors; some are real and important.
Common mistake: Deleting outliers automatically. Sometimes the outlier is the story — the fraud, the breakthrough, the rare event. Investigate first: is it a mistake, or a genuine signal?
Remember from Module 5: a single outlier can drag the mean far from reality, which is why the median is often safer.
How do you actually find outliers? The fastest way is to look. A histogram or a box plot makes extreme values jump out as lone dots far from the crowd. A quick numeric check helps too: sort the column and eyeball the very top and bottom values. A "maximum age" of 200 or a "minimum price" of -50 announces itself the moment you glance at the extremes.
Concept: Not every unusual value is an outlier, and not every outlier is an error. Think of three buckets: a typo (fix or remove it), a rare-but-real value like a genuine big spender (keep it), and a different unit sneaking in (convert it). Deciding which bucket a value falls into is the real skill.
Real-world use case: In fraud detection, the whole goal is to find the outliers — the one transaction among millions that doesn't fit the pattern. Blindly deleting outliers there would delete the fraud you were hired to catch. Context decides whether an outlier is noise or the entire point.
3. Judging Data Quality
Ask of any dataset:
- Complete? How much is missing?
- Accurate? Do values look believable?
- Consistent? Same formats and units throughout?
- Timely? Is it recent enough to matter?
These four questions take a minute and save hours. "Consistent" catches the sneaky unit mix-ups — some rows in kilometers, others in miles — that produce numbers which look fine but are silently wrong. "Timely" reminds you that even perfect data goes stale: last year's prices or an old customer list can quietly mislead a decision made today.
Quality also depends on fitness for purpose — data can be good enough for one question and useless for another. A rough headcount is fine for "roughly how many people came?" but hopeless for "who exactly should we email?" Before trusting any dataset, picture the decision it will drive, then ask whether it's complete, accurate, consistent, and timely enough for that specific decision. Good enough beats perfect, but only you can define "enough."
Data scientist tip: Always chart your data before modeling. A quick histogram or scatter plot reveals missing chunks and outliers instantly — problems that are invisible in a giant table of numbers.
Try this: Take any dataset and run the four quality questions like a checklist — complete, accurate, consistent, timely. Even a two-minute pass will surface at least one problem you'd otherwise have carried, unnoticed, into your results.
Key Takeaway: Missing values and outliers quietly distort results. Handle missing data by removing, filling, or flagging — based on why it's missing. Investigate outliers before deleting (they may be the real story), and remember they skew the mean. Judge quality by completeness, accuracy, consistency, and timeliness — and always chart data first.
Further Learning
Adapted from Microsoft's Data Science for Beginners (MIT License).