"Before the fancy tools, master the ideas: where data comes from, what shape it's in, and how it flows. Everything else builds on this."
Level: Intermediate · Time: ~3 days · Prerequisites: Module 1
Learning Objectives
By the end of this module, you will be able to:
- Distinguish structured, semi-structured, and unstructured data
- Explain batch vs streaming data
- Describe what a data pipeline is
- Understand schemas and why they matter
1. The Shapes of Data
Data comes in three broad shapes, and how you store and process it depends on which:
| Type | Example | Lives well in… |
|---|---|---|
| Structured | Rows and columns (sales table) | Relational databases |
| Semi-structured | JSON, XML logs | Document stores, lakes |
| Unstructured | Images, video, free text | Object storage / lakes |
Concept: Most "big data" is semi-structured or unstructured. A big part of data engineering is imposing enough structure on messy data to make it useful — without losing what matters.
The line between the three shapes is blurrier than the table suggests. A JSON log is technically semi-structured, but if every record reliably has the same fields, you can treat it almost like a structured table. Free text inside a "notes" column sits in a structured database yet is unstructured content. Real datasets are usually a mix, and part of the engineer's judgment is deciding how much structure to impose and where.
2. Batch vs Streaming
There are two fundamental ways data moves:
- Batch — collect data and process it in chunks on a schedule (e.g., nightly sales rollup). Simple and efficient.
- Streaming — process each event as it arrives (e.g., fraud detection on every transaction). Fresh, but more complex.
Explain like I'm new: Batch is doing all your laundry once a week. Streaming is washing each shirt the moment you take it off. One is efficient; the other is always up to date. You pick based on how fresh the data must be.
3. What Is a Pipeline?
A data pipeline is an automated sequence that moves data from source to destination, transforming it along the way. Raw data enters, clean and useful data exits — reliably, on a schedule or continuously.
Picture a concrete one: every night at 2 a.m., a pipeline pulls yesterday's orders from an e-commerce database, removes cancelled orders, converts every price to one currency, joins each order to its customer's region, and writes a tidy summary table that the finance dashboard reads at 9 a.m. Nobody touches it — it just runs, and if the source is unreachable it retries and alerts rather than producing a blank report.
Concept: Pipelines should be repeatable and reliable. A one-off script you run by hand isn't a pipeline; a pipeline runs itself, handles errors, and produces the same result every time.
4. Schemas: The Contract
A schema defines the structure of your data — the fields and their types (this column is a date, that one a number). Schemas are a contract: everyone downstream can rely on the shape. When schemas change unexpectedly, pipelines break — "schema drift" is a classic source of 3 a.m. alerts.
Real-world use case: An upstream mobile team renames user_id to uid in an app update. It looks harmless to them, but three pipelines that join on user_id now find nothing to match, and every downstream dashboard silently shows zero active users. A schema check that fails the pipeline the moment the expected column is missing would have caught it in minutes instead of a frantic afternoon of debugging.
Common mistake: Ignoring schema management until something breaks. When an upstream team renames a field or changes a type without warning, dependent pipelines fail silently or corrupt data. Track and validate schemas deliberately.
5. The Engineer's Mindset
Good data engineering optimizes for reliability, scalability, and clarity: pipelines that don't silently fail, that handle growing volume, and that others can understand and maintain. It's software engineering applied to data flow.
Try this: Think of one data task you do manually (say, merging spreadsheets each month). Sketch how you'd make it an automatic, repeatable pipeline. That shift — from manual to automated and reliable — is the core of the discipline.
✅ Checkpoint
- Give an example of structured vs unstructured data.
- When would you choose streaming over batch?
- Why is a schema like a "contract"?
Answers: 1) Structured: a sales table with rows/columns; unstructured: images or free text. 2) When data must be acted on immediately, like fraud detection. 3) It defines the agreed shape of data so everyone downstream can depend on it; breaking it breaks pipelines.
Key Takeaway: Data comes structured, semi-structured, or unstructured, and moves in batch (scheduled chunks) or streaming (event-by-event). A pipeline automates moving and transforming it reliably and repeatably, while schemas act as contracts that keep everything downstream working. The engineer's job is reliability, scalability, and clarity — software engineering for data flow.
Further Learning
Part of "MLOps & Data Engineering." Original content for this learning platform.