Module 01: What Are LLMs? Architecture Intuition

Duration: 3 hours | Difficulty: Beginner | Prerequisites: Basic Python

Learning Objectives

By the end of this module, you will be able to:

• Trace the evolution of NLP from rule-based systems to transformers
• Explain the self-attention mechanism using intuitive analogies
• Describe how tokenization converts text into numbers LLMs can process
• Define key LLM terminology: tokens, context window, temperature, top-p, and more
• Compare the architecture approaches behind GPT, Claude, and Gemini

1. The Journey to Large Language Models

From Rules to Intelligence

Picture teaching someone language by immersion instead of handing them a giant rulebook. Classical NLP was mostly the rulebook; modern LLMs are closer to immersion—they pick up patterns from huge amounts of text.

The evolution in four acts:

Why RNNs Struggled

RNNs read like a human reading a novel one word at a time while trying to remember everything from page 1. By chapter 3, early details fade—that’s the vanishing gradient story in plain clothes. LSTMs added a scratchpad, but very long context still hurt.

The Transformer Idea (2017)

The “Attention Is All You Need” paper flipped the script: instead of a strict line, the model can relate each word to every other word in one go—like zooming out to see the whole paragraph when resolving what “it” refers to.

2. The Self-Attention Mechanism

The Cocktail Party Analogy

At a noisy party, when someone says “it,” your brain boosts the relevant voices and dims the rest. In “The cat sat on the mat because it was tired,” you lean toward cat, not mat. Self-attention is that focusing mechanism, learned from data.

How Attention Works (Simplified)

For each token, the model forms three roles (implemented as learned projections in real systems):

Scores come from comparing queries and keys; high scores mean “listen here more.” Outputs are weighted mixes of values.

Multi-Head Attention

One head might care about grammar links; another about meaning; another about position-like patterns. Multi-head just means many parallel attention patterns that later get combined—like several specialists glancing at the same sentence for different reasons.

Self-attention is how the model decides which words should strongly influence each other for the next layer’s representation.

3. Tokenization: Turning Text into Numbers

Why Tokenization Matters

The model never sees “hello”—it sees token IDs. Tokenization is the bridge: text → pieces → integers.

Think of subwords like Lego bricks: common words might be one brick; rare or long words split into a few bricks so the vocabulary stays manageable.

Types of Tokenization

Token Economics (No Code Needed)

• APIs usually bill by tokens, not characters.
• A context window is “how many tokens fit in one shot.”
• Shorter prompts (without losing intent) = lower cost and latency.

Key Example: See how one sentence becomes a short list of token IDs and how each ID maps back to a text piece.

[object Object], tiktoken

enc = tiktoken.encoding_for_model(,[object Object],)
text = ,[object Object],
tokens = enc.encode(text)
,[object Object],(,[object Object],)
,[object Object], token_id ,[object Object], tokens:
    ,[object Object],(,[object Object],)

4. How GPT, Claude, and Gemini Work

The Transformer Architecture Stack

Same rough story for many chat models: tokens in → embeddings → many transformer blocks → prediction head that scores next token (decoder-style stacks).

┌─────────────────────────────────┐
│         Output Layer            │  ← scores over vocabulary
├─────────────────────────────────┤
│    Transformer Block × N        │  ← attention + feed-forward
├─────────────────────────────────┤
│    Positional Encoding          │  ← order information
├─────────────────────────────────┤
│    Token Embedding Layer        │  ← IDs → vectors
├─────────────────────────────────┤
│    Tokenizer                    │  ← text → token IDs
└─────────────────────────────────┘

Model Comparison

Exact numbers change—always check current provider docs before you budget.

The Generation Process: Next-Token Prediction

Under the hood, the model repeatedly asks: given everything so far, what token comes next? Sampling settings (temperature, top-p) reshape which high-scoring tokens actually get picked—like choosing whether the “dice” are biased toward safe choices or exploratory ones.

Chat models are fancy next-token engines with a stack of attention layers; “creativity” is mostly controlled sampling, not a separate soul.

5. Key Terminology Reference

6. Seeing It in Action: Your First LLM API Call

Key Example: This shows the usual chat shape: roles (system / user), a model name, and usage metadata you’ll use for cost tracking.

[object Object], openai ,[object Object], OpenAI

client = OpenAI(api_key=,[object Object],)  ,[object Object],

response = client.chat.completions.create(
    model=,[object Object],,
    messages=[
        {,[object Object],: ,[object Object],, ,[object Object],: ,[object Object],},
        {,[object Object],: ,[object Object],, ,[object Object],: ,[object Object],}
    ],
    temperature=,[object Object],,
    max_tokens=,[object Object],,
)

,[object Object],(response.choices[,[object Object],].message.content)
,[object Object],(,[object Object],)
,[object Object],(,[object Object],)

Practice Exercises

Mini-Project: Build an LLM Knowledge Card Generator

Build a small script that: (1) accepts a concept name, (2) asks an LLM for a JSON card (definition, analogy, bullets, misconceptions), (3) prints token usage, (4) saves Markdown. No need for fancy UI—CLI is enough.

Key Takeaways

Resources for Further Learning

• Attention Is All You Need — Foundational transformer paper
• The Illustrated Transformer — Visual walkthrough
• OpenAI Tokenizer Tool — Interactive tokens
• 3Blue1Brown: But what is a GPT?
• Andrej Karpathy: Let's build GPT
• Hugging Face NLP Course

Next Module: Prompt Engineering Mastery →