Module 42: Model Serving | Zero to AI Engineer

"A trained model in a file helps no one. Serving is how it becomes a live service that apps can call to get predictions."

Level: Advanced · Time: ~12 min · Prerequisites: Modules 39, 41

Learning Objectives

By the end of this module, you will be able to:

Explain what model serving is
Describe how a model becomes an API
Understand batching and scaling for serving
Recognize serving tools

Model serving wraps a trained model in a service — usually a web API — so applications can send inputs and get predictions back in real time. It's the "last mile" that turns a model into a product feature. A model sitting in a .pkl or .safetensors file is inert; serving is what lets a mobile app, a website, or another service actually use it. Note that serving comes in two flavors: real-time (answer one request now, as this module covers) and batch (score a big pile of inputs on a schedule, e.g., overnight). Both matter, but the real-time path is where most engineering effort — and most latency pressure — lives.

Explain like I'm new: Serving is putting your model behind a receptionist. Apps "call in" with a question (input), the model answers (prediction), and the receptionist hands it back — fast, reliably, to many callers at once.

2. Model → API

The common pattern:

Load the trained model in a server process
Expose an endpoint (e.g., POST /predict)
On each request: receive input → run the model → return the prediction

text

app → POST /predict {features} → [model] → {prediction} → app

For LLMs, "serving" also means streaming tokens back as they're generated, for a snappy feel.

Key idea: Serving is where latency becomes real. Users feel every millisecond. Design decisions — model size, hardware, batching, caching — all trade off speed, cost, and throughput here.

3. Batching & Scaling

Batching: group several incoming requests and run them together (GPUs love batches) — more throughput, slightly more latency.
Horizontal scaling: run many copies of the service behind a load balancer to handle more traffic.
Caching: if the same input recurs, return the stored answer instantly.

Two numbers describe serving performance, and it helps to keep them separate. Latency is how long one request takes — what a single user feels. Throughput is how many requests you handle per second overall. Batching improves throughput (the GPU does more useful work per pass) but can nudge latency up slightly, because a request waits a few milliseconds to be grouped with others. Tuning that trade-off — how long to wait, how big a batch to allow — is a core serving decision.

Real-world use case: A translation API serves millions of requests. It batches requests arriving within a few milliseconds to use the GPU efficiently, runs many replicas behind a load balancer, and caches common phrases. The result: fast responses at massive scale, at manageable cost.

4. Serving Tools

You rarely hand-roll all this. Tools handle serving concerns:

Tool	For
FastAPI / Flask	Simple custom serving
TorchServe / TF Serving	Framework-native serving
Triton, vLLM, TGI	High-performance, LLM-optimized serving
Cloud endpoints	Managed serving (SageMaker, Vertex, Azure ML)

Try this: Start with FastAPI for your first serving project — it's a few lines to expose /predict. Graduate to a specialized server (vLLM, TGI, Triton) only when you actually hit performance limits. The specialized tools add real speed for LLMs, but also real complexity.

Common mistake: Loading the model on every request. Loading is slow — do it once at startup and reuse it for all requests. Forgetting this is a classic cause of painfully slow serving.

Hands-On: Try This

Try this: Sketch the request/response for a "sentiment" API: what's the input JSON, what's the output JSON, and where does the model load (hint: once, at startup)? Designing this tiny contract is the core of model serving.

✅ Checkpoint

What does model serving do?
Why use batching?
Where should the model be loaded — per request or at startup?

Answers: 1) Wraps a model as a live service/API for real-time predictions. 2) To run several requests together for higher GPU throughput. 3) Once at startup, then reused.

Key Takeaway: Serving turns a trained model into a live API apps can call for predictions. It's where latency matters most, so you use batching (throughput), horizontal scaling (traffic), and caching (repeat inputs). Tools range from FastAPI to LLM-optimized servers (vLLM, TGI) to managed cloud endpoints. Load the model once at startup, not per request.

Further Learning

Part of "Zero to AI Engineer." Simplified from the AI Engineer curriculum.