"An LLM is the world's most sophisticated autocomplete — predicting text one small piece at a time, with uncanny skill."
Level: Intermediate · Time: ~14 min · Prerequisites: Modules 8, 11
Learning Objectives
By the end of this module, you will be able to:
- Explain how an LLM generates text
- Define tokens and the context window
- Tell training from inference
- Understand why LLMs hallucinate
1. Predicting the Next Token
At its core, a Large Language Model (LLM) does one thing: given the text so far, it predicts the most likely next token, adds it, and repeats — building a response piece by piece.
Explain like I'm new: It's autocomplete on steroids. Given "The capital of France is," it predicts "Paris." Do that repeatedly, at massive scale, and you get essays, code, and conversations — one token at a time.
Under the hood, the model doesn't pick one guaranteed word — it produces a probability for every possible next token ("Paris" 91%, "located" 3%, and so on) and samples from that list. This is why the same prompt can give slightly different answers each time, and why a setting called temperature controls how "adventurous" those picks are: low temperature stays safe and predictable, high temperature gets creative (and riskier).
2. Tokens & the Context Window
- Token: a chunk of text — roughly ¾ of a word. "unbelievable" might be 3 tokens.
- Context window: how much text the model can consider at once (its short-term memory). Go past it and the earliest text is forgotten.
Tokens matter practically: usage and cost are often measured in tokens, and the context window limits how much you can feed the model. As a rough rule of thumb, 1,000 tokens ≈ 750 words, so a context window of 128,000 tokens holds roughly a 300-page book. When you paste a long document and the model seems to "lose track" of the top, you've likely spilled past that window — the earliest tokens simply fall out of view.
Key idea: The LLM only "knows" what's in its context window right now, plus patterns baked in during training. It has no memory of past chats unless you include them — a crucial fact for building apps (and why frameworks like LangChain manage memory for you).
3. Training vs. Inference
| Training | Inference | |
|---|---|---|
| When | Once, before release (weeks/months) | Every time you use it |
| What | Learns patterns from huge text | Answers your prompt now |
| Analogy | Years of study | Taking an exam |
When you chat with an LLM, you're doing inference — it isn't learning from your message in that moment.
Real-world use case: An AI engineer rarely trains an LLM from scratch (it costs millions). Instead they use a pre-trained model via an API, fine-tune it on domain data, or ground it with retrieval (RAG, Module 21). Knowing training vs. inference tells you which lever to pull.
4. Why LLMs Hallucinate
Because an LLM predicts likely text rather than looking up truth, it can produce fluent, confident, and false output — a "hallucination." Asked for a source, it may even invent a real-sounding book title, author, and page number that don't exist — because a plausible-looking citation is exactly what "likely next text" produces. It also has a knowledge cutoff (it only learned up to some date) and no live data unless connected to tools, so it can't know today's news or your private files on its own.
Common mistake: Trusting a confident tone as proof of accuracy. Fluent, self-assured writing is exactly what an LLM is built to produce — even when wrong. Always verify facts, figures, and citations.
Hands-On: Try This
Try this: Ask an LLM a very specific factual question (a small-town population, an obscure date). Then verify it. Whether it's right or wrong, you'll feel the difference between "generating likely text" and "knowing a fact" — the heart of why grounding matters.
✅ Checkpoint
- What does an LLM predict, over and over?
- What is the context window?
- Why do LLMs hallucinate?
Answers: 1) The next token. 2) How much text the model can consider at once. 3) They predict likely text, not verified truth, so wrong answers can still "sound" likely.
Key Takeaway: An LLM generates text by repeatedly predicting the next token from patterns learned in training, limited by its context window (its working memory). Using it is inference; it doesn't learn from your chat. Because it predicts likely text rather than retrieving truth, it can hallucinate — so ground it and verify important output.
Further Learning
Part of "Zero to AI Engineer." Adapted from Microsoft's open curricula (MIT License).