"A neural network can be right for reasons we don't understand. Interpretability is the science of opening the black box."
Level: Intermediate · Time: ~3 days · Prerequisites: Module 6
Learning Objectives
By the end of this module, you will be able to:
- Explain why interpretability matters
- Distinguish "explaining a decision" from "understanding the mechanism"
- Describe common interpretability techniques
- Recognize the limits of current methods
1. The Black Box Problem
Deep models make excellent predictions, but their reasoning is spread across millions (or billions) of numbers. When a model denies a loan or flags a tumor, "trust me" isn't good enough. Interpretability aims to make model behavior understandable to humans.
The stakes make this concrete. In medicine, a model might flag a scan as cancerous — but if it's secretly keying on a hospital's ruler markings rather than the tumor (a real documented failure), it will collapse in the real world. In lending, a model that denies loans needs an explanation both for the applicant and the law. "The network output 0.31" is a number, not a reason. Interpretability is the effort to turn those numbers back into reasons a human can check.
Concept: Interpretability serves trust, debugging, fairness, and safety. If you can see why a model decided something, you can catch mistakes, biases, and dangerous reasoning before they cause harm.
2. Two Kinds of "Why"
| Question | Approach |
|---|---|
| "Why this decision?" | Local explanations — which inputs drove one prediction |
| "How does the model work inside?" | Mechanistic — reverse-engineering the internal circuits |
Both are valuable: one helps a doctor trust a diagnosis; the other helps researchers understand the model itself.
The analogy from biology is useful. A local explanation is like observing behavior — "the patient reacted to this stimulus" — without knowing the underlying biology. Mechanistic interpretability is like neuroscience: mapping the actual circuits, finding that this group of neurons fires for that concept. Local explanations are quicker and often enough for a single decision; mechanistic work is slower but aims for the deep prize of genuinely understanding how the model computes.
3. Common Techniques
- Feature attribution — highlight which inputs mattered most (e.g., which pixels or words drove the output).
- Saliency maps — a heatmap over an image showing where the model "looked."
- Probing — test what information is stored in a model's internal layers.
- Mechanistic interpretability — trace specific neurons and circuits to the concepts they represent.
Explain like I'm new: It's like an MRI for a neural network. You can't ask it to explain itself in words, so you use tools to see which parts "light up" for which inputs — then piece together what's really going on.
4. Why It's Hard
Modern models are enormous and their concepts are tangled — a single neuron may respond to many unrelated things, and one concept may be spread across many neurons. This tangling has a name, superposition: because a network has fewer neurons than concepts it wants to represent, it packs multiple ideas into overlapping sets of neurons, like a cramped apartment where every room serves several purposes. Untangling that is a major reason mechanistic interpretability is so hard. Explanations can also be misleading: a tidy heatmap doesn't guarantee it reflects the model's true reasoning.
Common mistake: Trusting an explanation just because it looks convincing. An attribution map can be plausible yet unfaithful to what the model actually computed. Good interpretability research validates that its explanations are accurate, not just appealing.
5. Why It's a Frontier
Interpretability is one of AI's most important open areas: it underpins safety (Module 14), helps debug failures, supports fairness audits, and builds the trust needed to deploy AI in high-stakes settings like medicine and law.
Try this: Next time an AI tool gives you an answer, ask "what would convince me this reasoning is sound?" That instinct — demanding understandable justification — is exactly what interpretability research tries to provide at scale.
✅ Checkpoint
- Why does interpretability matter beyond curiosity?
- What's the difference between a local explanation and mechanistic interpretability?
- Why can an explanation be misleading?
Answers: 1) It supports trust, debugging, fairness, and safety by revealing why models act as they do. 2) Local explains a single decision's drivers; mechanistic reverse-engineers the model's internal workings. 3) A plausible-looking explanation may not faithfully reflect the model's actual computation.
Key Takeaway: Interpretability opens the black box so humans can understand model behavior — vital for trust, debugging, fairness, and safety. It ranges from local explanations (why one decision) to mechanistic analysis (how the model works inside), using tools like feature attribution, saliency maps, and probing. It's hard because concepts are tangled and explanations can look convincing yet be unfaithful.
Further Learning
Part of "Research & Advanced AI." Original content for this learning platform.