"Before writing code, you design the system: where data flows, where the model lives, and how predictions reach users. Good design prevents most problems."
Level: Intermediate · Time: ~13 min · Prerequisites: Module 38
Learning Objectives
By the end of this module, you will be able to:
- Describe the parts of an ML system
- Tell batch from real-time prediction
- Reason about trade-offs
- Sketch a simple architecture
1. A System, Not Just a Model
An ML system has many moving parts beyond the model: data sources, a pipeline to prepare data, training, storage, a serving layer, and monitoring. System design decides how these fit together to meet real requirements (speed, scale, cost).
Explain like I'm new: Designing an ML system is like planning a restaurant, not just hiring a chef (the model). You need suppliers (data), a kitchen (pipeline), a menu and waiters (serving), and quality checks (monitoring). The chef alone doesn't make a restaurant.
2. Batch vs. Real-Time Prediction
A core design choice:
| Batch | Real-time | |
|---|---|---|
| When | On a schedule (nightly) | On demand (per request) |
| Latency | Minutes/hours OK | Milliseconds |
| Example | Nightly churn scores | Fraud check at checkout |
| Complexity | Simpler | Harder (needs live serving) |
Key idea: Ask "how fresh must the prediction be?" If yesterday's answer is fine, batch is simpler and cheaper. If it must react to this request now, you need real-time serving. This one question shapes the whole architecture.
The reason this one question ripples so far is that real-time serving pulls a whole chain of complexity behind it. If a prediction must react to this request now, you need a live serving endpoint that's always up, features you can compute on the spot (not just yesterday's precomputed table), latency budgets measured in milliseconds, and infrastructure to absorb traffic spikes. Batch skips all of that: it runs on a quiet schedule, writes results to a table, and lets you read them instantly later. Choosing batch when freshness allows can cut your operational burden in half — which is why senior engineers ask about freshness before anything else.
3. Thinking in Trade-offs
Good design balances competing needs:
- Latency vs. cost: faster serving usually costs more
- Accuracy vs. speed: a bigger model may be slower
- Freshness vs. simplicity: real-time features add complexity
- Build vs. buy: a managed cloud service vs. your own
There's rarely one "right" answer — only the best fit for your requirements.
Real-world use case: A streaming service scores "what to recommend" in batch overnight for the home screen (cheap, fine to be a few hours old), but uses real-time signals to reorder as you browse. One product, two prediction modes, each chosen by its latency needs.
4. Sketching an Architecture
A simple, common shape:
Data sources → Pipeline (clean/features) → Training → Model store → Serving (API) → Users → Monitoring → (loop back)
Start by drawing boxes and arrows like this. It exposes where data flows, where things can break, and where to add monitoring — before you write a line of code.
Concept: Notice the arrow that loops back from monitoring to the pipeline. That feedback loop is what separates a real ML system from a one-time script. Models decay as the world shifts (Module 46 calls this drift), so a healthy system watches its own predictions, notices when quality slips, and triggers a retrain — feeding fresh data back through the same pipeline. When you sketch the boxes, always draw that return arrow; a design without it describes a model that's slowly dying and can't heal itself.
Common mistake: Designing for imaginary "web-scale" from day one. Over-engineering wastes time and money. Design for your real current requirements, and leave clear seams to scale later when you actually need to.
Hands-On: Try This
Try this: Pick an idea (e.g., "flag toxic comments"). Decide: batch or real-time? Then sketch the boxes: where's the data, the model, the serving, the monitoring? This 10-minute diagram is exactly how senior engineers start a project.
✅ Checkpoint
- Name three parts of an ML system beyond the model.
- When is batch prediction the better choice?
- Why avoid designing for "web-scale" too early?
Answers: 1) e.g., data sources, pipeline, training, model store, serving, monitoring. 2) When predictions don't need to be fresh to the second (yesterday's answer is fine). 3) Over-engineering wastes effort/money; design for real current needs with room to scale.
Key Takeaway: An ML system is data sources → pipeline → training → model store → serving → monitoring, not just a model. A key choice is batch vs. real-time prediction, driven by required freshness. Good design balances trade-offs (latency, cost, accuracy, build-vs-buy) for your real requirements — sketch the architecture first, and don't over-engineer.
Further Learning
Part of "Zero to AI Engineer." Simplified from the AI Engineer curriculum.