"Train a transformer on a huge chunk of the internet and something remarkable happens: it learns to write, reason, and code — from one simple goal."
Level: Intermediate · Time: ~4 days · Prerequisites: Module 6
Learning Objectives
By the end of this module, you will be able to:
- Explain how large language models are trained
- Define "foundation model" and why the idea matters
- Describe pretraining, fine-tuning, and prompting
- Understand context windows and their limits
1. One Simple Goal: Predict the Next Word
An LLM is trained on a staggering amount of text with one objective: predict the next token (word-piece). To do that well across billions of examples, it must implicitly learn grammar, facts, reasoning patterns, and style. Intelligence emerges as a side effect of getting really good at "what comes next."
Why does one dull goal produce such rich behavior? Because predicting the next token well secretly requires understanding a lot. To finish "The capital of France is ___" you need a fact. To finish "2 + 2 = ___" you need arithmetic. To finish a mystery novel's last line you need to track the whole plot. There's no single "reasoning" switch being trained — but reasoning turns out to be the most efficient way to keep guessing right across trillions of varied sentences, so the model grows it on its own.
Explain like I'm new: It's like the world's most well-read autocomplete. By practicing "finish this sentence" trillions of times on the internet, it picks up how language — and the ideas behind it — actually work.
2. Foundation Models
A foundation model is one big model, pretrained once on broad data, that can then be adapted to countless tasks. Instead of training a fresh model per task, you build on the foundation.
Concept: The shift to foundation models is why AI accelerated. One expensive pretraining run creates a general base; everyone else adapts it cheaply for translation, summarizing, coding, and more.
3. Pretrain → Fine-Tune → Prompt
There are three ways knowledge gets into and out of an LLM:
| Stage | What happens |
|---|---|
| Pretraining | Learn general language from massive text (expensive, done once) |
| Fine-tuning | Nudge the model toward a specific task or style (cheaper) |
| Prompting | Steer the frozen model at use time with instructions/examples |
Most people only ever prompt — no training required.
Real-world use case: A startup wants a customer-support bot. Pretraining a model from scratch would cost millions, so they don't. Instead they take an existing open foundation model, optionally fine-tune it on a few thousand of their own support transcripts, and then prompt it with company policies at use time. Three stages, but they only paid for the cheap two — the expensive pretraining was done once, by someone else, and reused.
4. Context Windows
An LLM can only "see" a limited amount of text at once — its context window, measured in tokens. Everything it should consider (your question, documents, chat history) must fit inside. Bigger windows allow more context but cost more to run.
Common mistake: Assuming the model remembers past conversations forever. It only knows what's inside the current context window. Anything outside it is invisible unless you supply it again (a key reason RAG exists).
5. Strengths and Limits
LLMs are astonishing generalists, but they predict plausible text, not verified truth. They can state falsehoods confidently ("hallucinate"), reflect biases in their training data, and go stale as the world changes.
The root cause is worth remembering: the model was optimized to produce plausible-sounding text, and a confident fabrication often sounds more plausible than "I don't know." A made-up citation with a real-looking author and year fits the pattern of "what a citation looks like" perfectly — which is exactly why hallucinations are so convincing. The fix isn't to expect perfection but to design around it: retrieval, tools, and verification (later modules) all exist because raw next-token prediction has this built-in limitation.
Try this: Ask an LLM a niche factual question and then verify it independently. Noticing where it's confidently wrong builds the healthy skepticism every AI researcher and user needs.
✅ Checkpoint
- What single objective are LLMs trained on?
- What is a foundation model, and why is the idea powerful?
- What does the context window limit?
Answers: 1) Predicting the next token in text. 2) A broadly pretrained model adapted to many tasks — it saves everyone from training from scratch. 3) How much text the model can consider at once; anything outside it is invisible.
Key Takeaway: Large language models learn from one simple goal — predict the next token — over enormous text, and general abilities emerge as a result. As foundation models, they're pretrained once and then fine-tuned or prompted for many tasks. They're powerful generalists but bounded by their context window and prone to confident errors, so verify their claims.
Further Learning
Part of "Research & Advanced AI." Original content for this learning platform.