Module 08: Memory & Context — Windows, RAG & Personal Knowledge Bases

"The model in the API does not remember you between calls. Persistence is a design choice you implement in databases, files, and retrieval—not magic in the weights."
— Common distillation of LLM systems engineering (paraphrased)

Duration: 5–7 hours · Difficulty: Intermediate · Prerequisites: Basic Python, comfort with JSON and environment variables, conceptual comfort with vectors at “arrow in space” level. Optional API keys for embeddings; local models possible for privacy-sensitive projects.

Learning Objectives

By the end of this module, you will be able to:

• Explain why a stateless chat API can still feel conversational (client-managed message history).
• Contrast short-term context (prompt + prior turns) with long-term memory (databases, notes, summaries, RAG indices).
• Reason about context window limits, lost-in-the-middle effects, and chunking tradeoffs for retrieval.
• Describe RAG (retrieval-augmented generation) as: embed → retrieve → augment prompt → generate, with citations.
• Build a toy or minimal RAG pipeline over public documents you have rights to use, with evaluation hooks.
• Evaluate when RAG helps versus when better prompting, structured tools, or fine-tuning (conceptually) is more appropriate.
• Identify privacy risks: confidential notes in third-party vector stores, embedding logs, and academic integrity (outsourcing reading).

1. There Is No Magical “Memory” in a Single Request

Most chat APIs process a concatenated transcript each call: system + user + assistant (+ optional tool) messages up to a token budget. Model weights do not update per user during ordinary inference—persistence is your database, filesystem, or vendor-specific memory products.

Implication: Anything not in the context window cannot influence the answer unless you retrieve it (RAG), fetch it (tool), or summarize prior turns into the window.

Real-world scenario: A student opens ChatGPT in two browser tabs with “different conversations” about the same thesis. Each tab sends only its thread history. There is no cross-tab memory—unless the product explicitly adds account-level memory features (read their privacy notice).

Pro Tips — memory and context design

• Pin “source of truth” metadata next to every chunk: doc_version, ingest_date, and license—staleness bugs are easier to fix when you can sort by version.
• Put the user’s hard constraint in the last user message when you observe position bias; repeat critical facts in both system and user slots sparingly but deliberately for must-not-forget rules.
• Summarize only after extracting structured facts (dates, names, thresholds) into a key-value store—compression is for narrative, not for legal or numeric precision.
• Budget tokens like money: If retrieval + history exceeds 60% of the window, you likely need smaller chunks, better routing, or a cheaper summarization model—not a bigger window.

Common mistakes — Module 08

• Embedding duplicates under different chunk ids—re-ranking cannot fix redundant near-copies; dedupe at ingest.
• Assuming citations imply faithfulness—models still paraphrase; enforce “answer must quote substring from chunk” for high-stakes drills.
• Storing raw chat in plaintext on shared lab machines—treat transcripts like any other sensitive note file.
• Mixing embedding models mid-corpus without re-indexing—similarity scores become incomparable across chunks.

Comparison table: Rolling summary vs. structured memory vs. RAG

2. Context Windows, Cost, and Position Effects

Windows grew from thousands to hundreds of thousands of tokens** in some systems (check current provider docs), but:

• Cost and latency generally scale with sequence length.
• Attention over extremely long prompts can be uneven—important facts may be repeated or placed in high-salience positions (start/end) per empirical studies on position bias.
• Raw PDF dump into context is often worse than curated retrieval—noise drowns signal.

2.1 Token counting (rough vs exact)

[object Object], ,[object Object],(,[object Object],) -> ,[object Object],:
    ,[object Object],
    ,[object Object], ,[object Object],(,[object Object],, ,[object Object],(text) // ,[object Object],)

long_doc = ,[object Object], * ,[object Object],
,[object Object],(,[object Object],, rough_token_estimate(long_doc))

[object Object],
,[object Object], tiktoken

enc = tiktoken.get_encoding(,[object Object],)
text = ,[object Object],
,[object Object],(,[object Object],, ,[object Object],(enc.encode(text)))

2.2 Budgeting a 128k window (exercise template)

Allocate tokens among:

• System instructions (behavior, safety)
• Tool JSON schemas
• Retrieved chunks (RAG)
• Rolling summary of old chat
• User’s latest message

Document your budget table in project README—TAs appreciate explicit engineering.

3. Memory Architectures for Assistants and Agents

3.0 Conversation JSON on disk (toy persistence)

[object Object], json
,[object Object], pathlib ,[object Object], Path
,[object Object], typing ,[object Object], ,[object Object],

,[object Object], ,[object Object],(,[object Object],) -> ,[object Object],:
    path.parent.mkdir(parents=,[object Object],, exist_ok=,[object Object],)
    history: ,[object Object],[,[object Object],[,[object Object],, ,[object Object],]] = []
    ,[object Object], path.exists():
        history = json.loads(path.read_text(encoding=,[object Object],))
    history.append({,[object Object],: role, ,[object Object],: content})
    path.write_text(json.dumps(history, indent=,[object Object],, ensure_ascii=,[object Object],), encoding=,[object Object],)

This is not scalable for large classes of users—it illustrates that you persist state, not the model weights.

• Scratchpad: short bullet notes kept in context (“user prefers APA”).
• Rolling summarization: compress older turns; lossy—verify critical facts are copied forward or stored structurally.
• Structured memory: key-value facts extracted with human review before storage.
• Tool memory: search calendar, email, or notes via governed APIs (Module 06).

When summarization hurts: Legal names, numeric thresholds, and citation metadata—store verbatim in a database instead of trusting compressed prose.

4. RAG: Pipelines, Chunking, and Failure Modes

4.0 Ingestion code sketch (PDF → text chunks)

[object Object], pathlib ,[object Object], Path
,[object Object], dataclasses ,[object Object], dataclass
,[object Object], json

,[object Object],
,[object Object], pypdf ,[object Object], PdfReader

,[object Object],
,[object Object], ,[object Object],:
    chunk_id: ,[object Object],
    text: ,[object Object],
    page: ,[object Object],
    source: ,[object Object],

,[object Object], ,[object Object],(,[object Object],) -> ,[object Object],[Chunk]:
    reader = PdfReader(,[object Object],(path))
    chunks: ,[object Object],[Chunk] = []
    cid = ,[object Object],
    ,[object Object], i, page ,[object Object], ,[object Object],(reader.pages, start=,[object Object],):
        text = page.extract_text() ,[object Object], ,[object Object],
        start = ,[object Object],
        ,[object Object], start < ,[object Object],(text):
            end = ,[object Object],(,[object Object],(text), start + max_chars)
            piece = text[start:end].strip()
            ,[object Object], piece:
                chunks.append(
                    Chunk(
                        chunk_id=,[object Object],,
                        text=piece,
                        page=i,
                        source=path.name,
                    )
                )
                cid += ,[object Object],
            start = end - overlap ,[object Object], end < ,[object Object],(text) ,[object Object], ,[object Object],(text)
    ,[object Object], chunks

,[object Object], ,[object Object],(,[object Object],) -> ,[object Object],:
    out.write_text(
        json.dumps([c.__dict__ ,[object Object], c ,[object Object], chunks], indent=,[object Object],, ensure_ascii=,[object Object],),
        encoding=,[object Object],,
    )

,[object Object],

OCR may be required for scanned PDFs—tesseract pipelines add noise; note limitations in your methods section.

4.1 Pipeline (systems view)

1. Ingest documents (rights verified).
2. Chunk into segments (size + overlap hyperparameters).
3. Embed chunks to vectors; store with metadata (source, page, heading).
4. Query: embed question; retrieve top-k similar chunks (optionally re-rank with a cross-encoder).
5. Generate answer conditioned on retrieved text; require citations (chunk id + page).

4.2 Chunking tradeoffs

Overlap (50–100 tokens) reduces boundary cuts through definitions.

4.3 Failure modes

• Wrong chunk retrieved (polysemy: “bank” river vs finance).
• Stale corpus (old student handbook).
• Leakage of confidential data to embedding API logs.
• Hallucination despite retrieval if model ignores context—citation enforcement in prompt helps but does not guarantee.

4.4 Minimal “manual RAG” without a vector DB (toy)

[object Object], math

,[object Object], ,[object Object],(,[object Object],) -> ,[object Object],:
    dot = ,[object Object],(x * y ,[object Object], x, y ,[object Object], ,[object Object],(a, b))
    na = math.sqrt(,[object Object],(x * x ,[object Object], x ,[object Object], a))
    nb = math.sqrt(,[object Object],(y * y ,[object Object], y ,[object Object], b))
    ,[object Object], na == ,[object Object], ,[object Object], nb == ,[object Object],:
        ,[object Object], ,[object Object],
    ,[object Object], dot / (na * nb)

,[object Object],
chunks: ,[object Object],[,[object Object],, ,[object Object],[,[object Object],]] = {
    ,[object Object],: [,[object Object],, ,[object Object],],
    ,[object Object],: [,[object Object],, ,[object Object],],
    ,[object Object],: [,[object Object],, ,[object Object],],
}
q = [,[object Object],, ,[object Object],]
best = ,[object Object],(chunks, key=,[object Object], k: cosine_sim(q, chunks[k]))
,[object Object],(,[object Object],, best)

Replace toy vectors with real embeddings; add metadata table keyed by chunk id.

4.5 Sketch: embed + retrieve with numpy stack

[object Object], numpy ,[object Object], np

,[object Object], ,[object Object],(,[object Object],) -> np.ndarray:
    norms = np.linalg.norm(m, axis=,[object Object],, keepdims=,[object Object],)
    norms = np.where(norms == ,[object Object],, ,[object Object],, norms)
    ,[object Object], m / norms

,[object Object],
X = l2_normalize(np.array([[,[object Object],, ,[object Object],], [,[object Object],, ,[object Object],], [,[object Object],, ,[object Object],]], dtype=np.float32))
q = l2_normalize(np.array([[,[object Object],, ,[object Object],]], dtype=np.float32))
scores = (q @ X.T)[,[object Object],]
topk = np.argsort(-scores)[:,[object Object],]
,[object Object],(,[object Object],, topk.tolist(), ,[object Object],, scores[topk])

For production scale, use FAISS, pgvector, Chroma, Qdrant, etc.—this math is the inner loop.

Hands-On: Python — Simple rolling window over messages (token-budget sketch)

Keeps only the last K user/assistant turns to stay under a rough character budget (swap in tiktoken for production).

[object Object], dataclasses ,[object Object], dataclass


,[object Object],
,[object Object], ,[object Object],:
    role: ,[object Object],
    content: ,[object Object],


,[object Object], ,[object Object],(,[object Object],) -> ,[object Object],[Msg]:
    ,[object Object],
    total = ,[object Object],(,[object Object],(m.content) ,[object Object], m ,[object Object], msgs)
    out = msgs[:]
    i = ,[object Object],
    ,[object Object], total > max_chars ,[object Object], i < ,[object Object],(out):
        total -= ,[object Object],(out[i].content)
        i += ,[object Object],
    ,[object Object], out[i:]


thread = [
    Msg(,[object Object],, ,[object Object], * ,[object Object],),
    Msg(,[object Object],, ,[object Object], * ,[object Object],),
    Msg(,[object Object],, ,[object Object], * ,[object Object],),
]
,[object Object],(,[object Object],(,[object Object],(m.content) ,[object Object], m ,[object Object], trim_history(thread, max_chars=,[object Object],)))
,[object Object],

Expected takeaway: Client-side trimming is blunt; pair with summaries or structured memory for long projects.

Hands-On: Python — Chunk hash for deduplication at ingest

[object Object], hashlib


,[object Object], ,[object Object],(,[object Object],) -> ,[object Object],:
    payload = ,[object Object],.encode(,[object Object],)
    ,[object Object], hashlib.sha256(payload).hexdigest()[:,[object Object],]


a = chunk_fingerprint(,[object Object],, ,[object Object],, ,[object Object],)
b = chunk_fingerprint(,[object Object],, ,[object Object],, ,[object Object],)
,[object Object],(a == b)
,[object Object],

5. RAG vs Alternatives (Conceptual Decision Table)

Fine-tuning updates behavior distribution; it is not a substitute for a dated knowledge base unless you retrain—expensive and data-hungry.

5.1 Academic integrity note

RAG over your lecture notes can help you study; RAG that replaces reading assigned papers does not. Use tools to locate passages you still must read and cite properly.

6. Building a Personal Knowledge Base Responsibly

6.1 Data you should not embed in third-party stores

• Graded exams you do not own the copyright to.
• Classmates’ work.
• Unredacted human subjects transcripts without IRB clearance.
• Proprietary employer code or unreleased datasets.

Prefer local embedding models (sentence-transformers) on your laptop for sensitive drafts.

6.2 Evaluation habit

For each test question, log:

• Retrieved chunk ids
• Hit/miss (did the chunk contain the answer?)
• Final answer supported by chunk text? (yes/no/unclear)

This mirrors industry RAG eval and impresses in capstone defense.

6.3 Prompt skeleton for grounded answers

You answer ONLY using the CONTEXT below. If CONTEXT is insufficient, say you do not know.
Cite sources as [chunk_id] after each sentence that uses CONTEXT.

CONTEXT:
{{retrieved_chunks}}

QUESTION:
{{user_question}}

Tight prompts reduce—but do not eliminate—ungrounded elaboration.

6.4 Optional: call OpenAI embeddings API (outline)

[object Object], os
,[object Object], requests

,[object Object], ,[object Object],(,[object Object],) -> ,[object Object],[,[object Object],[,[object Object],]]:
    url = ,[object Object],
    headers = {
        ,[object Object],: ,[object Object],,
        ,[object Object],: ,[object Object],,
    }
    payload = {,[object Object],: model, ,[object Object],: texts}
    r = requests.post(url, headers=headers, json=payload, timeout=,[object Object],)
    r.raise_for_status()
    data = r.json()[,[object Object],]
    ,[object Object], [row[,[object Object],] ,[object Object], row ,[object Object], ,[object Object],(data, key=,[object Object], x: x[,[object Object],])]

,[object Object],

Never commit API keys; use .env + python-dotenv locally. For EU privacy projects, confirm data processing terms.

6.5 Graph RAG (overview sentence)

Graph RAG links chunks via entities (people, courses, policies) to improve multi-hop questions (“Which office handles both X and Y?”). It adds ingestion complexity—sketch in design docs even if you do not implement for week-one homework.

Practice Exercises

Lab notebook hygiene

If using Jupyter, pin requirements.txt and record embedding model name + version in the first cell. Re-running notebooks months later with a different embedding model invalidates stored vectors—document migration steps.

1. Context budget math — For a 128k-token window, allocate tokens among system prompt, tool schemas, retrieved docs, rolling summary, and user input; justify each slice in 2–3 sentences. Expand: Add a contingency row: “If user pastes a 30k-token dump, which budget line gets cut first and why?”

2. Chunking experiment — Take a three-page public article; chunk at 200 vs 800 words with 0 vs 50 token overlap. Which chunks keep definitional sentences intact? Show one example split. Expand: For the worst split you find, propose a heading-aware or sentence-boundary rule and show the improved chunk boundaries (even if you do not code it).

3. Stale corpus — Identify a question where a 2023 training cutoff or outdated RAG index yields a wrong answer. How would you detect staleness in a product (metadata version, TTL, alerts)? Expand: Write one user-facing disclaimer string your UI would show when corpus version < current_policy_version.

4. Privacy red-team — List five information types that must not be embedded in a third-party vector store without contract review. Expand: For one item, describe a redacted surrogate you could embed safely (e.g., synthetic schedule instead of real roster).

5. Compare strategies — In 400 words, contrast RAG vs fine-tuning for a niche campus domain (student handbook QA). Include cost, update latency, and citation needs. Expand: Conclude with one hybrid architecture sentence (e.g., “RAG for facts + small prompt for tone”) tied to a concrete failure mode each side fixes.

6. Lost-in-the-middle drill — Craft a prompt where a critical fact sits in the middle of a long context block; test whether the model answers correctly. Expand: Try moving the fact to start/end and report qualitative difference in one short paragraph (no p-hacking across dozens of trials—this is conceptual).

Mini-Project

Tiny RAG over Public Campus PDFs

Build a minimal pipeline over 1–3 small public PDFs (handbook excerpts, public safety FAQs, registrar pages exported with permission):

1. Extract text (pypdf, pdfplumber, or approved library); record page numbers.
2. Chunk with documented size/overlap; store chunk_id → {text, page, source} in JSON or SQLite.
3. Embed with a documented API or local model; save vectors alongside metadata.
4. CLI or notebook that answers five instructor-provided test questions with citations (chunk_id + page).
5. Evaluation table: per question, mark retrieval hit/miss and explain misses (bad chunking, synonym gap, OCR error).

Stretch: Add re-ranking (cross-encoder) or hybrid search (BM25 + vectors) at high level.

AI disclosure: State if embeddings or answer drafting used cloud APIs; describe redaction steps.

Hardware note: Local models may need GPU; document latency on your machine vs cloud.

Rubric alignment: Full credit usually requires citations tied to chunks, not generic URLs. If the model paraphrases without chunk support, mark answer unsupported in your eval table—that honesty earns partial credit where blind confidence fails.

Mini-project extensions (depth)

• Negative test set: Write five questions that should receive “not in corpus” answers; log any false positives where the model answers anyway—tune prompts or retrieval top_k based on findings.
• Manifest file: Ship manifest.json listing every source file, SHA-256, ingest date, and chunk count; graders (and future you) can diff when the handbook updates.
• Latency note: Time embed → retrieve → generate separately on your hardware; one paragraph on which stage dominates and what you would cache first in a product.

Cross-major examples:

• Pre-med: RAG over public CDC FAQ PDFs for a study guide—not over patient notes.
• History: RAG over out-of-copyright primary texts you host; watch OCR errors on archaic fonts.
• Engineering: RAG over your lab’s SOP PDFs with version numbers in metadata; stale SOP = safety issue.

Key Takeaways

• Context is a scarce interface—design what enters each call deliberately.
• Client-side history creates the illusion of memory; long-term memory belongs in systems you control.
• RAG grounds answers in your corpus but does not guarantee truth—verify citations.
• Chunking is a first-class engineering decision, not an afterthought.
• Position and length effects matter; huge windows are not free intellectually or financially.
• Privacy and integrity constraints often favor local embeddings or strict redaction.
• Evaluate retrieval with a simple hit/miss log before scaling UI polish.
• Version your embedding model in the manifest—silent drift breaks comparability across semesters.
• OCR and PDF extractors introduce noise; cite extraction method in methods sections.

Resources

• Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (foundational RAG paper).
• OpenAI embeddings guide: https://platform.openai.com/docs/guides/embeddings
• tiktoken tokenizer library: https://github.com/openai/tiktoken
• Speech and Language Processing (Jurafsky & Martin) — NLP fundamentals: https://web.stanford.edu/~jurafsky/slp3/
• Survey literature on RAG (2024–2026) via your library for updated routing and graph-RAG overviews.