"A clever idea proves nothing without a fair test. Experiment design is how you turn a hunch into evidence people trust."
Level: Intermediate · Time: ~4 days · Prerequisites: Module 4
Learning Objectives
By the end of this module, you will be able to:
- Form a clear, testable hypothesis
- Choose fair baselines and controls
- Use ablations to isolate what really matters
- Report results honestly with proper measures
1. Start With a Testable Hypothesis
A good experiment answers a sharp question. "Our model is better" is vague. "Adding component X improves accuracy on dataset Y by a meaningful margin" is testable — you know exactly what would confirm or refute it.
Sharpen it further and it becomes almost a bet: "If I add attention pooling, top-1 accuracy on CIFAR-10 will rise by at least 1 point, averaged over 3 seeds." Now the experiment has a clear pass/fail line drawn before you run it. Vague hypotheses tempt you to move the goalposts after seeing the data ("well, it helped on this subset"); sharp ones keep you honest.
Concept: A hypothesis you can't imagine being wrong isn't science. State what result would prove you wrong before you run anything.
2. Fair Baselines Are Everything
You compare your idea against a baseline — the current best or a sensible standard. The catch: the baseline must be tuned as carefully as your method. A crippled baseline makes anything look great, and that's the most common way results mislead.
Common mistake: Spending weeks tuning your new method and an afternoon on the baseline. The comparison is then meaningless. Give both equal effort — or your "win" is an illusion.
3. Control Your Variables
Change one thing at a time. If you swap the model and the dataset and the learning rate at once, you can't tell which caused the difference. Keep everything fixed except the variable you're studying.
Explain like I'm new: It's like testing whether a plant grows better with new fertilizer. If you also move it to a sunnier window and water it more, you'll never know what actually helped. Change only the fertilizer.
4. Ablations: Remove-One-Piece Tests
An ablation turns off one part of your method to see if it still works. If accuracy barely drops without your "key" component, then that component wasn't the reason for success. Ablations are how you prove which idea is doing the work.
| Version tested | Accuracy | Tells you… |
|---|---|---|
| Full method | 92% | the headline result |
| Without component X | 91% | X barely matters |
| Without component Z | 78% | Z is the real driver |
Read that table like a detective. The paper's title probably brags about component X — but the ablation reveals X adds only 1 point, while removing Z costs 14. The real story is Z. Good reviewers head straight for the ablation table, because it exposes which part of a method is doing the actual work versus which part is just marketing.
5. Measure and Report Honestly
- Run with several random seeds and report the average and spread — one run can be a fluke.
- Report compute and data so others can judge fairly.
- Don't hide failure cases; they're informative.
Why seeds matter so much: neural network training involves randomness (initial weights, data shuffling, dropout). Two runs of the same code can differ by a point or more purely by luck. If your method scores 91.2% and the baseline 90.8%, but each has a ±0.6% swing across seeds, your "win" is inside the noise — it means nothing. Reporting "91.2 ± 0.6" instead of a lone "91.2" is the difference between a trustworthy result and a coin flip dressed up as science.
Real-world use case: A team once reported a shiny new reinforcement-learning result that others couldn't reproduce. The culprit? A single lucky seed. When re-run across many seeds, the "improvement" vanished. This exact episode is why the field now expects seed-averaged results — and why the classic paper linked below became required reading.
Try this: Re-run a small experiment with three different random seeds. If the results swing wildly, a single-seed "improvement" you saw earlier may have been noise, not signal.
✅ Checkpoint
- What makes a hypothesis testable?
- Why must a baseline be tuned as hard as your new method?
- What does an ablation study reveal?
Answers: 1) You can state in advance what result would confirm or refute it. 2) An under-tuned baseline makes any method look good, so the comparison becomes meaningless. 3) Which specific component is actually responsible for the method's success.
Key Takeaway: Good experiments start with a testable hypothesis, compare against fairly-tuned baselines, change one variable at a time, use ablations to isolate what truly helps, and report results honestly across multiple seeds. This rigor is what separates a trustworthy finding from a lucky number.
Further Learning
- Henderson et al., "Deep RL That Matters" — a classic on fair evaluation
- Google's ML testing best practices
Part of "Research & Advanced AI." Original content for this learning platform.