"An LLM is the world's most sophisticated autocomplete. It doesn't 'know' facts — it predicts, one small piece at a time, what text should come next."
Learning Objectives
By the end of this module, you will be able to:
- Explain how an LLM generates text, one token at a time
- Say what a token is
- Tell the difference between training and inference
- Understand why LLMs sometimes get things wrong
1. It's All About Predicting the Next Piece
At its heart, a large language model does one thing astonishingly well: it looks at the text so far and predicts the most likely next chunk of text. Then it adds that chunk and predicts again, and again — building a response piece by piece.
Given "The cat sat on the ___", it scores every possible continuation and finds "mat" most likely. That's it — repeated thousands of times, at incredible speed, across everything from poems to code.
It helps to picture the model reading the same sentence you would, but pausing at the blank to weigh options: "mat" (very likely), "floor" (possible), "roof" (unusual), "helicopter" (almost never). It picks a strong candidate, writes it, and then re-reads the whole thing — including the word it just added — before choosing the next one. Because each new word becomes part of what it reads next, the response stays coherent from start to finish.
Explain like I'm new: Think of texting with predictive suggestions above your keyboard. You tap a suggested word, and new suggestions appear based on what you now have. An LLM does the same thing, just far better and without you tapping — it keeps choosing the next piece until the thought is complete.
Key idea: An LLM is a next-token prediction machine. It has no beliefs or intentions — it produces the text that is statistically most likely to follow, based on patterns learned from enormous amounts of writing.
2. What's a "Token"?
Models don't read whole words the way we do. They break text into tokens — small pieces that are often a word, part of a word, or a punctuation mark. Roughly, 1 token ≈ ¾ of a word in English.
- "cat" → 1 token
- "unbelievable" → might split into "un", "believ", "able"
- "Hello, world!" → about 4 tokens
Tokens matter for practical reasons: AI tools often measure usage and cost in tokens, and every model has a limit on how many tokens it can consider at once (its "context window").
Did you know: Splitting text into pieces rather than whole words lets a model handle words it has never seen before — including names, typos, and made-up terms — by assembling them from familiar parts. It's a bit like sounding out an unfamiliar word syllable by syllable.
3. Training vs. Inference
There are two very different phases in an LLM's life:
| Training | Inference | |
|---|---|---|
| When | Once, before release (takes weeks/months) | Every time you use it |
| What happens | Learns patterns from huge text collections | Answers your prompt right now |
| Analogy | Years of school | Sitting an exam question |
| Changes the model? | Yes — this is where it learns | No — it just uses what it learned |
Key idea: Training is the model learning from data. Inference is the model using what it learned to answer you. When you chat with an AI, you're doing inference — the model isn't learning from your message in that moment.
4. Prompts In, Responses Out
When you send a prompt, the model reads your tokens, then generates response tokens one at a time until it's done. Everything it can "see" — your prompt plus the conversation so far — must fit inside its context window. Go past that limit and the earliest parts get forgotten.
This is why giving clear, complete context in your prompt matters so much — the model only knows what's in front of it. (That's the whole subject of Module 3.)
5. Why LLMs Get Things Wrong — Explained Simply
Because a model predicts likely text rather than looking up truth, it has real limitations:
- Hallucinations — it can state false things confidently, because a wrong answer can still "sound" likely.
- Knowledge cutoff — it only learned from data up to a certain date, so it may not know recent events.
- No live access by default — unless connected to tools, it can't browse the web or see your files.
- Sensitive to wording — small changes to your prompt can noticeably change the answer.
Common mistake: Trusting a confident tone as proof of correctness. Fluent, self-assured writing is exactly what an LLM is built to produce — even when the facts are wrong. Always verify anything important.
Key Takeaway: An LLM generates text by repeatedly predicting the next token from patterns it learned during training. Tokens are the small text pieces it reads and writes. Training is one-time learning; inference is answering you. Because it predicts likely text rather than retrieving facts, it can hallucinate, has a knowledge cutoff, and needs clear prompts — so always double-check important results.