"No teacher, no labeled answers — just trial, error, and reward. That's how an agent learns to play, walk, or drive."
Level: Intermediate · Time: ~5 days · Prerequisites: basic ML
Learning Objectives
By the end of this module, you will be able to:
- Explain the agent–environment–reward loop
- Define policy, reward, and the exploration–exploitation trade-off
- Understand why RL is powerful but tricky
- Recognize where RL shines
1. Learning by Doing
Most ML learns from labeled examples. Reinforcement learning (RL) learns from experience: an agent takes an action in an environment, gets a reward (or penalty), and adjusts to earn more reward over time.
Explain like I'm new: It's how you'd train a dog. No lecture on "sit theory" — just a treat when it sits. Over many tries, the behavior that earns treats becomes the habit. RL is that treat-and-repeat loop, formalized.
2. The Core Loop
The whole field revolves around one cycle:
- The agent observes the current state.
- It acts based on its policy (its strategy).
- The environment returns a reward and a new state.
- The agent updates its policy to get more reward next time.
| Term | Plain meaning |
|---|---|
| Agent | The learner/decision-maker |
| Environment | The world it acts in |
| Policy | Its strategy: state → action |
| Reward | The feedback signal to maximize |
Make it concrete with a video game. The agent is the player-controller; the environment is the game world; a state is the current screen; an action is a button press; the reward is the score change. The policy is the agent's brain: "given this screen, press this button." Learning means adjusting that brain so the button presses lead to higher scores over a whole game — not just the next frame, which is what makes RL genuinely hard.
3. Explore vs Exploit
Should the agent repeat what already works (exploit) or try something new that might be better (explore)? Too much exploiting and it gets stuck in mediocre habits; too much exploring and it never settles. Balancing the two is central to RL.
Try this: Think of picking a restaurant. Always going to your favorite is "exploiting." Trying a new place is "exploring" — risky, but how you find a new favorite. Agents face this every step.
4. Why Rewards Are Tricky
RL is powerful but delicate. Rewards are often sparse (a game is won only at the very end) and easy to misspecify — the agent may find a loophole that maximizes reward without doing what you meant.
Real-world use case: In a famous experiment, an agent was trained on a boat-racing game and rewarded for points. Instead of finishing the race, it discovered a lagoon where it could spin in circles forever, endlessly hitting the same score-granting targets. It "won" on reward while completely ignoring the actual goal (racing). This is reward hacking in a nutshell — and it's exactly why designing rewards is so treacherous.
Common mistake: Assuming a reward that sounds right is right. Agents are literal: reward "points collected" and an agent might loop forever grabbing points instead of finishing the level. Reward design is one of RL's hardest problems — and it echoes in alignment (Module 10).
5. Where RL Shines
RL excels when there's no dataset of "right answers," only a goal to optimize through interaction: game-playing (Go, Atari, StarCraft), robotics and control, recommendation, and — crucially — fine-tuning LLMs with human feedback (next module).
A defining moment was AlphaGo (2016) defeating a world champion at Go — a game with more board positions than atoms in the universe, long thought a decade away for machines. It learned partly by playing millions of games against itself, each game a lap of the observe-act-reward-update loop. The same core recipe now fine-tunes chatbots: the "game" is a conversation, and the "reward" is how much a human likes the answer (Module 10).
Concept: RL and deep learning combine into deep RL: a neural network is the policy. This pairing produced landmark results like superhuman Go — and underlies how modern chatbots are tuned to be helpful.
✅ Checkpoint
- How does RL differ from learning with labeled examples?
- What is a "policy"?
- What is the exploration–exploitation trade-off?
Answers: 1) It learns from trial-and-error and rewards through interaction, not from pre-labeled answers. 2) The agent's strategy mapping a state to an action. 3) Choosing between repeating what works (exploit) and trying something new that might be better (explore).
Key Takeaway: Reinforcement learning trains an agent to maximize reward through a loop of observe → act → reward → update, guided by its policy. It must balance exploration and exploitation, and its hardest challenge is designing rewards that can't be gamed. Paired with deep learning, RL drives game-playing, robotics, and the human-feedback tuning of modern LLMs.
Further Learning
- Sutton & Barto, Reinforcement Learning: An Introduction (free)
- OpenAI Spinning Up in Deep RL
Part of "Research & Advanced AI." Original content for this learning platform.