"A raw language model is smart but unruly. RLHF is how we teach it to be helpful, honest, and follow instructions — using human preferences as the reward."
Level: Intermediate · Time: ~4 days · Prerequisites: Modules 7, 9
Learning Objectives
By the end of this module, you will be able to:
- Explain why raw LLMs need alignment
- Describe the three steps of RLHF
- Understand what a "reward model" is
- Recognize the limits of learning from human feedback
1. Why Raw Models Need Tuning
A freshly pretrained LLM (Module 7) is a brilliant next-word predictor, but it doesn't naturally follow instructions or avoid harmful output — it just continues text. RLHF (Reinforcement Learning from Human Feedback) turns that raw ability into a helpful assistant.
Here's the gap in action. Ask a raw pretrained model "What is the capital of France?" and it might reply "What is the capital of Germany? What is the capital of Italy?" — because on the internet, questions are often followed by more questions, not answers. It's doing its job perfectly (continuing plausible text) yet uselessly. RLHF is what teaches it that when a human asks a question, the desired continuation is a helpful answer.
Concept: RLHF is the bridge between "a model that can produce any plausible text" and "a model that produces the text you actually want." It's a big part of why modern chatbots feel so usable.
2. The Three Steps of RLHF
RLHF combines everything from the last two modules:
- Supervised fine-tuning — show the model good example answers written by humans.
- Train a reward model — humans rank several model answers best-to-worst; a second model learns to predict those human preferences.
- Reinforcement learning — use that reward model as the reward signal (Module 9) to nudge the LLM toward answers humans prefer.
Explain like I'm new: Step 1 is showing examples of good work. Step 2 is training a "taste tester" that scores answers like a human would. Step 3 is letting the model practice, using the taste tester's scores as its reward — so it learns to give answers people like.
3. The Reward Model
Humans can't score millions of answers by hand, so we train a reward model to imitate their preferences. It becomes the automated stand-in for human judgment during RL training.
| Piece | Role |
|---|---|
| Human rankings | The ground-truth preferences |
| Reward model | Learns to predict those rankings |
| LLM | Optimized to score high on the reward model |
Why not skip the reward model and just use humans directly? Because RL needs millions of feedback signals during training, and no team can have people rank answers that many times. So we collect a manageable set of human rankings (say, tens of thousands), train the reward model to imitate that human taste, and then let it hand out unlimited automated "scores" during RL. The reward model is essentially a scalable clone of human preference — which is powerful, but also means any flaw in it gets amplified.
4. It's Powerful — But Imperfect
RLHF made assistants dramatically more useful, but it has real limits:
- Human raters disagree and carry biases, which the model absorbs.
- Models can learn to sound good rather than be correct ("please the rater").
- Over-optimizing the reward model can degrade true quality (reward hacking, echoing Module 9).
Common mistake: Treating an RLHF-tuned model as "aligned and safe, done." Alignment is ongoing — feedback is noisy, preferences are subjective, and a model optimized to please can still be confidently wrong. This is exactly why safety research (Module 14) matters.
5. Beyond RLHF
Newer methods aim to simplify or improve this pipeline — learning directly from preference data, or using AI feedback to assist human feedback. The goal is always the same: make capable models reliably helpful and safe.
Two directions worth knowing by name: DPO (Direct Preference Optimization) skips the separate reward model and RL loop, learning straight from the ranked pairs — simpler and often just as good. Constitutional AI / RLAIF uses an AI to generate much of the feedback against a written set of principles, so humans write the rules once instead of ranking endlessly. Both attack the same bottleneck: human feedback is slow and noisy, so the field keeps hunting for ways to get the same alignment with less of it.
Try this: Next time a chatbot gives a slick but subtly wrong answer, notice it. That's the "sounds good vs is right" gap RLHF struggles with — spotting it makes you a sharper user and researcher.
✅ Checkpoint
- Why doesn't a raw pretrained LLM follow instructions well?
- What are the three steps of RLHF?
- What is a reward model, and why is it needed?
Answers: 1) It's trained only to predict the next token, not to be helpful or safe. 2) Supervised fine-tuning, train a reward model from human rankings, then reinforcement learning against that reward. 3) A model that predicts human preferences, standing in for human judgment so training can scale.
Key Takeaway: RLHF aligns raw LLMs with what people actually want through three steps — supervised fine-tuning, training a reward model from human rankings, and reinforcement learning against that reward. It's the key to today's helpful assistants, but it's imperfect: human feedback is noisy and models can learn to sound right rather than be right — which keeps alignment an open research problem.
Further Learning
Part of "Research & Advanced AI." Original content for this learning platform.