"A pipeline turns 'I ran some cells in order' into an automated, repeatable assembly line — the backbone of professional ML."
Level: Intermediate · Time: ~13 min · Prerequisites: Module 39
Learning Objectives
By the end of this module, you will be able to:
- Explain what an ML pipeline is
- Name the typical pipeline stages
- Understand orchestration and automation
- See why pipelines beat manual steps
1. From Manual Steps to an Assembly Line
Doing ML by hand — download data, clean it, train, save the model — works once. But real systems repeat this constantly, reliably, with new data. An ML pipeline automates these steps into a defined, repeatable sequence.
Explain like I'm new: A pipeline is a factory assembly line for your model. Raw data goes in one end; a trained, validated model comes out the other — the same way every time, no human clicking buttons. Consistency and automation are the whole point.
2. Typical Stages
| Stage | What happens |
|---|---|
| Ingest | Pull in fresh data |
| Validate | Check data quality (Module 38's data tests) |
| Transform | Clean, build features |
| Train | Fit the model |
| Evaluate | Score on test data; compare to current model |
| Register | Save the model if it's good enough |
| Deploy | Push it to serving |
Each stage's output feeds the next — and the whole thing can run on a schedule or a trigger.
Notice the two gate stages — validate and evaluate. They're what make a pipeline safe rather than just fast. Validate stops bad data from entering (if today's feed is half-empty, the run halts before wasting hours training on garbage). Evaluate stops a bad model from leaving (a freshly trained model only advances if it beats the one already in production). Without these gates, automation just repeats mistakes faster; with them, the pipeline actively defends quality every single run.
Key idea: A pipeline makes ML reproducible and automatable. Run it today or in six months, on this data or next quarter's, and it executes the exact same steps — so results are trustworthy and updates are painless.
3. Orchestration
Something has to run the stages in order, handle failures, and schedule them — that's an orchestrator. Common tools: Airflow, Kubeflow Pipelines, Prefect, Dagster. They define the pipeline as a graph of steps (a DAG) and manage execution.
Real-world use case: A retailer's pipeline runs every night: ingest yesterday's sales, validate them, retrain the demand-forecasting model, evaluate it, and — only if it beats the current model — deploy it. No human involved unless something fails. That reliability is why pipelines exist.
Concept: Orchestrators describe pipelines as a DAG — a directed acyclic graph. "Directed" means each arrow points one way (train happens after transform); "acyclic" means the arrows never loop back into a cycle, so the run always has a clear finish. Modeling the work as a DAG lets the orchestrator do smart things automatically: run independent steps in parallel, retry just the step that failed instead of the whole job, and skip stages whose inputs haven't changed. That's why "define your pipeline as a DAG" is the near-universal pattern across Airflow, Kubeflow, Prefect, and Dagster.
4. Why Pipelines Win
- Reproducible: same steps every time
- Automated: runs on schedule/trigger, not by hand
- Reliable: validation and evaluation gates catch problems
- Scalable: handle growing data without more manual work
Common mistake: Automating a bad manual process. A pipeline faithfully repeats whatever you build — including mistakes, at scale. Get the steps right by hand first, add validation gates, then automate.
Hands-On: Try This
Try this: Write out, as a numbered list, the steps you'd run to retrain a model weekly (ingest → validate → … → deploy). Circle the step where you'd add a "only deploy if better than current" gate. You've just designed a pipeline.
✅ Checkpoint
- What does an ML pipeline automate?
- Name three typical pipeline stages.
- What does an orchestrator do?
Answers: 1) The repeatable sequence of ML steps (ingest → … → deploy). 2) e.g., ingest, validate, transform, train, evaluate, register, deploy. 3) Runs the stages in order, on schedule, handling failures.
Key Takeaway: An ML pipeline automates the repeatable sequence — ingest, validate, transform, train, evaluate, register, deploy — turning manual steps into a reliable assembly line. An orchestrator (Airflow, Kubeflow, Prefect, Dagster) runs it on schedule with failure handling. Pipelines make ML reproducible, automated, and scalable — but perfect the steps (and add gates) before automating.
Further Learning
Part of "Zero to AI Engineer." Simplified from the AI Engineer curriculum.