"Some data can't wait for tonight's batch. Fraud, live dashboards, and alerts need answers now — that's streaming."
Level: Intermediate · Time: ~4 days · Prerequisites: Modules 2, 4
Learning Objectives
By the end of this module, you will be able to:
- Explain stream processing and how it differs from batch
- Describe what Kafka does as an event backbone
- Understand producers, topics, and consumers
- Recognize challenges unique to streaming
1. Data in Motion
Batch processing works on data at rest, in chunks. Stream processing works on data in motion — handling each event the instant it arrives. When freshness is critical (a fraudulent charge, a sensor spike, a live counter), you can't wait for a nightly job.
Explain like I'm new: Batch is reading the newspaper each morning — yesterday's news, all at once. Streaming is a live news ticker — events as they happen. Some questions ("what's trending right now?") only streaming can answer.
2. Kafka: The Event Backbone
Apache Kafka is the most common tool for moving streaming data. Think of it as a high-speed, durable conveyor belt for events: producers drop events on, consumers pick them up — decoupled, reliable, and able to handle enormous volume.
| Term | Meaning |
|---|---|
| Producer | Sends events (e.g., the checkout service) |
| Topic | A named stream of events (e.g., "orders") |
| Consumer | Reads events (e.g., the fraud checker) |
| Broker | A Kafka server that stores and serves events |
Concept: Kafka decouples systems. Producers don't know or care who consumes their events; many consumers can read the same stream independently. That flexibility is why it's the backbone of modern real-time platforms.
Decoupling pays off when teams change independently. The checkout service just publishes an "order placed" event and moves on. Months later, a new analytics team wants that data — they simply add a consumer to the existing topic, and the checkout team never has to know or change a line of code. Contrast that with the old approach of the checkout service directly calling every downstream system: every new consumer meant editing (and risking) the producer.
3. Producers, Topics, Consumers
An event (say, "order placed") is published by a producer to a topic. Any number of consumers — a dashboard, a fraud model, a warehouse loader — subscribe and react, each at its own pace. Kafka keeps the events durably so a slow or restarted consumer never misses one.
Try this: Picture a checkout event flowing to three consumers at once: one updates a live sales dashboard, one checks for fraud, one archives to the lake. One event, three reactions, all in real time — that's the streaming mindset.
4. Why Streaming Is Harder
Real-time brings unique challenges:
- Ordering & late data — events can arrive out of order or delayed.
- Exactly-once — avoiding double-processing when things retry.
- Always-on — streaming systems must run 24/7, not just at 2 a.m.
- State — computing "average over the last 5 minutes" requires tracking a moving window.
Common mistake: Choosing streaming because it sounds impressive. It's more complex to build and operate than batch. Use it when freshness genuinely matters; otherwise, batch is simpler and cheaper.
The "state" challenge is worth grasping because it trips up newcomers. A batch job that computes a daily total simply reads the whole day at once. But "average temperature over the last five minutes," computed continuously, means the system must remember recent events and keep updating as time slides forward — and survive a restart without losing that memory. This windowed, remembered state is a big part of why streaming systems are harder to build and reason about than batch.
5. Streaming in the Pipeline
Streaming and batch often coexist: Kafka feeds real-time dashboards and lands events in the lake/warehouse for later batch analysis. Modern engines (Spark Structured Streaming, Flink) process these streams with familiar, SQL-like logic.
Concept: The trend is "process once, use everywhere" — the same event stream powers instant reactions and historical analysis. Kafka is the hub that makes that possible.
This coexistence is worth internalizing: streaming rarely replaces batch, it complements it. The stream handles the "act now" jobs — alerts, live counters, fraud checks — while the same events, landed in a lake or warehouse, feed the slower, deeper batch analysis where correctness and completeness matter more than speed. Mature platforms run both, using each where it fits.
✅ Checkpoint
- How does stream processing differ from batch?
- In Kafka, what are producers, topics, and consumers?
- Name one challenge that's harder in streaming than batch.
Answers: 1) Streaming handles each event as it arrives (real-time); batch processes data in scheduled chunks. 2) Producers send events, topics are named event streams, consumers read them. 3) Any of: out-of-order/late data, exactly-once processing, 24/7 uptime, or managing windowed state.
Key Takeaway: Stream processing handles data in motion, event by event, for when freshness is critical. Kafka is the durable, decoupled backbone: producers publish to topics, and many consumers react independently. Streaming is powerful but harder than batch (ordering, exactly-once, always-on, state), so use it when real-time truly matters — often alongside batch.
Further Learning
Part of "MLOps & Data Engineering." Original content for this learning platform.