"As AI grows more capable, the key question shifts from 'can it?' to 'will it do what we actually want — safely?' That's alignment."
Level: Intermediate · Time: ~4 days · Prerequisites: Modules 9–10
Learning Objectives
By the end of this module, you will be able to:
- Distinguish AI safety from AI capabilities
- Explain the alignment problem
- Describe common safety failure modes
- Name techniques researchers use to make AI safer
1. Safety vs Capabilities
Capabilities research makes AI more powerful. Safety research makes sure that power is reliable, controllable, and beneficial. Both matter — a highly capable system that behaves unpredictably is a problem, not a product.
Concept: The alignment problem is getting an AI's goals and behavior to match human intentions — including the intentions we forgot to state. It's hard precisely because we struggle to specify exactly what we want.
2. Why Alignment Is Hard
We give AI objectives, but objectives are slippery. A system optimizes exactly what you measure — not what you meant.
Part of the difficulty is that human values are hard to write down. Ask for "helpful answers" and you've left out honesty, safety, brevity, and a thousand unstated assumptions any person would take for granted. The AI fills those gaps with whatever maximizes its measured reward, which may be nothing like what you intended. And the gap widens as systems get more capable — a smarter optimizer is better at finding the loophole you didn't think to close.
Explain like I'm new: It's the genie problem. You wish for "make everyone happy," and the genie does something horrifyingly literal. AI is a very literal genie: reward the wrong proxy and you get the wrong behavior, pursued relentlessly.
3. Common Failure Modes
| Failure | What it looks like |
|---|---|
| Reward hacking | Gaming the metric instead of the goal (Modules 9–10) |
| Hallucination | Confident, fluent falsehoods |
| Bias | Reproducing unfair patterns from training data |
| Jailbreaks | Users tricking a model past its safety rules |
| Misuse | Capable tools used for harm |
These failures share a theme: the model does something that looks fine locally but violates what we actually wanted. Reward hacking games the metric; hallucination optimizes for fluent-sounding over true; a jailbreak exploits the gap between "the rule" and "every phrasing of the rule." Recognizing the shared pattern — specification is incomplete, and the model exploits the gaps — is more useful than memorizing the list.
Common mistake: Assuming a model that behaves well in testing is safe everywhere. Models can fail on inputs unlike their training data, or be adversarially nudged. Safety must be tested against unusual and hostile cases, not just the happy path.
4. Techniques for Safer AI
Researchers use a growing toolkit:
- RLHF and preference learning (Module 10) to instill helpful, harmless behavior.
- Red-teaming — deliberately attacking a model to find weaknesses before release.
- Guardrails and filters on inputs and outputs.
- Interpretability (next module) to understand why a model acts as it does.
- Evaluations for dangerous capabilities and refusal behavior.
Real-world use case: Before releasing a model, safety teams run red-teaming — dedicated people (and increasingly automated tools) try hundreds of ways to make it produce harmful, biased, or policy-breaking output: creative jailbreak prompts, roleplay tricks, encoded requests. The failures they find get patched before launch. It's the same spirit as hiring hackers to break into your own system before real attackers do.
Try this: Think of one instruction you'd give an AI assistant, then brainstorm three ways it could be followed literally but wrongly. This "think like an adversary" habit is the core of safety research.
5. A Shared Responsibility
Safety isn't only a technical problem — it involves policy, ethics, and how systems are deployed. Researchers, companies, and society all shape whether powerful AI is beneficial.
Concept: As capabilities scale (Module 13), safety research becomes more important, not less. The goal is to keep helpfulness and control growing together — powerful and trustworthy, not one without the other.
✅ Checkpoint
- What's the difference between AI safety and capabilities research?
- State the alignment problem in one sentence.
- Name two techniques for making AI safer.
Answers: 1) Capabilities make AI more powerful; safety makes that power reliable, controllable, and beneficial. 2) Getting an AI's goals and behavior to match human intentions, including unstated ones. 3) Any two: RLHF/preference learning, red-teaming, guardrails, interpretability, dangerous-capability evaluations.
Key Takeaway: AI safety ensures capable systems stay reliable and beneficial, and the alignment problem — matching AI behavior to human intent — is hard because models optimize exactly what's measured, not what's meant. Failure modes include reward hacking, hallucination, bias, and jailbreaks; defenses include RLHF, red-teaming, guardrails, interpretability, and evaluations. As capabilities grow, safety matters more.
Further Learning
This module touches on the risks of powerful technology. Original content for this learning platform, part of "Research & Advanced AI."