"You don't read a paper like a novel, front to back. You read it like a detective — in passes, chasing the claim and its evidence."
Level: Intermediate · Time: ~3 days · Prerequisites: Module 1
Learning Objectives
By the end of this module, you will be able to:
- Read a paper in three efficient passes
- Separate a paper's claims from its evidence
- Spot common weak spots in experiments
- Keep useful notes you can return to later
1. A Paper Is a Claim + Evidence
Every paper argues: "We did X, and it works better than Y." Your job isn't to admire it — it's to ask, "If I removed the new idea and tuned everything fairly, would it still win?" That mindset keeps you from over-trusting a confident abstract.
Picture a paper claiming "our new layer boosts accuracy by 3%." Before you believe it, you separate the two things it's really saying: the claim ("this layer helps") and the evidence ("+3% on these datasets, these baselines, these seeds"). A strong claim resting on one lucky run against a weak competitor is worth less than a modest claim backed by many careful runs. Great readers instinctively split every sentence into "what they assert" versus "what they showed."
Concept: The abstract is marketing. The experiments section is where claims live or die. Always check whether the evidence really supports the headline.
2. The Three-Pass Method
Reading everything word-by-word wastes time. Use three passes:
- Pass 1 (5–10 min): Title, abstract, figures, headings, conclusion. Decide: read deeply, skim later, or drop.
- Pass 2 (~1 hour): Read the method and experiments. Note the key idea and how it's tested. Skip heavy proofs.
- Pass 3 (as needed): Deep read to actually reproduce it — cross-check the code against the text.
Most papers only ever need passes 1 and 2 — you reserve the deep third pass for the few you truly want to build on.
Try this: Set a 10-minute timer on any recent paper and write one sentence each for its problem, idea, and type of evidence. That's a complete Pass 1 — and often enough to decide if it's worth more time.
3. Anatomy of a Paper
| Section | Grab on Pass 1 | Read skeptically because… |
|---|---|---|
| Abstract | Problem + promise | it's optimistic by design |
| Intro | The contributions | it's positioning, not proof |
| Related work | Key prior papers | coverage can be biased |
| Method | The new idea | code may differ from the text |
| Experiments | Baselines, datasets | tuning may be unfair |
| Limitations | Honesty check | often thin — add your own |
4. Reading Experiments Like a Reviewer
The experiments decide everything. Ask in plain English:
- Are the baselines strong and tuned as carefully as the new method?
- Were results run with multiple random seeds (not one lucky run)?
- Is the compute and dataset reported honestly?
- Do the ablations (remove-one-piece tests) show the new idea is what actually helps?
Real-world use case: Suppose a paper reports 88.0% accuracy versus a baseline's 85.0%. Before celebrating, you check the appendix: the new method used a bigger model and ten times the training data. Now the "3-point win" might just be "more compute wins" — a completely different (and less interesting) claim. Reading like a reviewer means finding these hidden differences.
Common mistake: Accepting a headline number because it's bold and bigger. A weak or under-tuned baseline can make almost anything "state of the art." Fair comparison is everything.
5. Notes That Scale
Keep a tiny, consistent note per paper: one-sentence claim, the evidence (datasets/metrics), the limitations, and a link to code. Future-you will re-enter the paper in two minutes instead of thirty.
A note template that works well: Problem (one line), Key idea (one line), Evidence (datasets + main number), Weak spot (what you'd challenge), Link to code. After twenty papers you'll have a searchable personal map of a subfield — far more useful than twenty PDFs you half-remember. The consistency is the point: the same five fields every time so you can compare papers at a glance.
✅ Checkpoint
- What are the three passes, and what's the goal of each?
- Why is the experiments section more important than the abstract?
- Name two things that make an experimental comparison unfair.
Answers: 1) Pass 1 (skim to triage), Pass 2 (method + experiments), Pass 3 (deep read to reproduce). 2) Claims are proven or disproven there; the abstract only promises. 3) Any two: weak/under-tuned baselines, a single seed, unreported compute/data, or missing ablations.
Key Takeaway: Read papers as a detective, not a fan: use three passes (triage → method/experiments → deep read), separate claims from evidence, and scrutinize experiments for strong baselines, multiple seeds, and honest reporting. Keep short, structured notes so every paper you read compounds into lasting knowledge.
Further Learning
- S. Keshav, "How to Read a Paper"
- OpenReview — read real reviewer debates
Part of "Research & Advanced AI." Original content for this learning platform.