"When one machine isn't enough, you spread training across many. Scaling up is how modern models get trained in days instead of years."
Level: Advanced · Time: ~12 min · Prerequisites: Module 5
Learning Objectives
By the end of this module, you will be able to:
- Explain why training needs to scale
- Understand data vs. model parallelism (simply)
- Recognize the role of GPUs and clusters
- Know practical scaling techniques
1. Why Scale Training?
Big models on big data can take an impractically long time on one machine. Scaling spreads the work across multiple GPUs or machines so training finishes in reasonable time. Consider the arithmetic: a model that takes 40 days to train on one GPU might finish in about five days on eight — turning an unshippable timeline into a workable one. Modern frontier models train on thousands of chips at once; without scaling they would take centuries on a single device.
Explain like I'm new: Training a huge model on one GPU is like painting a stadium with one brush. Add more painters (GPUs) working in parallel and the job finishes far faster — as long as they coordinate well.
Concept: Scaling rarely gives perfect speedup. Doubling the GPUs doesn't always halve the time, because the machines must pause to share their learning each step. This coordination cost is called communication overhead, and it's why 100 GPUs might give, say, an 80× speedup rather than a clean 100×.
2. Two Ways to Parallelize
| Approach | Idea | When |
|---|---|---|
| Data parallelism | Copy the model on each GPU; each processes different data; combine updates | Most common; model fits on one GPU |
| Model parallelism | Split the model itself across GPUs | Model too big for one GPU |
Data parallelism is the workhorse; model parallelism is for giant models that won't fit on a single device.
Key idea: Data parallelism = same model, different data batches, then average the learning. It's how most teams get a big speedup. Reach for model parallelism only when the model itself is too large to fit — it's more complex.
3. Hardware: GPUs & Clusters
Neural network math (lots of matrix multiplication) runs far faster on GPUs than CPUs. A GPU has thousands of small cores that all crunch numbers at once, which is exactly the shape of neural-network math — the same operation applied across huge grids of numbers. At scale, you use clusters of GPU machines, often rented from the cloud by the hour. Within a cluster, GPUs are wired together with fast interconnects (like NVLink or InfiniBand) so they can swap updates quickly — a slow network between machines can become the bottleneck. Specialized chips (TPUs, and edge NPUs) push this further.
Real-world use case: A team fine-tunes a vision model on 8 GPUs using data parallelism — each GPU trains on a slice of the images, updates are synced each step, and what would take a week on one GPU finishes overnight. They rent the cluster for the run, then shut it down to save cost.
4. Practical Techniques
- Mixed precision: use lower-precision math to train faster with less memory
- Gradient accumulation: simulate a big batch on limited memory
- Checkpointing: save progress so a crash doesn't lose everything
- Start small: prove the model works on a subset before scaling up
Checkpointing deserves special attention at scale. When a run spans days across dozens of machines, the odds that something fails — a GPU faults, a node reboots, the network hiccups — grow quickly. Saving a checkpoint every so often means a failure costs you an hour of recomputation, not the whole run. Most large training jobs simply resume from the last checkpoint and carry on as if nothing happened.
Real-world use case: During a multi-day language-model run, one GPU node crashes overnight. Because the team checkpointed every 30 minutes, training automatically restarts from the latest saved state and loses only the last half hour — instead of days of expensive compute.
Common mistake: Scaling to a big expensive cluster before the model even works. Debug on a small dataset and one GPU first. Scaling multiplies both your speed and your costs — and your bugs.
Hands-On: Try This
Try this: Imagine training on 100 GPUs. With data parallelism, describe how you'd split the work (each GPU gets a different batch, then they average their learning). Explaining it in your own words locks in the core idea of scaled training.
✅ Checkpoint
- Why scale training across machines?
- What's the difference between data and model parallelism?
- Why are GPUs used for training?
Answers: 1) To finish training big models/data in reasonable time. 2) Data parallelism copies the model and splits the data; model parallelism splits the model itself. 3) Their parallel math is far faster for neural-network computations.
Key Takeaway: Training at scale spreads work across GPUs/machines to finish in reasonable time. Data parallelism (same model, different data batches) is the common workhorse; model parallelism splits giant models across devices. GPUs/clusters power it, aided by mixed precision, gradient accumulation, and checkpointing. Prove the model small before scaling — scaling multiplies cost and bugs too.
Further Learning
Part of "Zero to AI Engineer." Simplified from the AI Engineer curriculum.