Module 10: Introduction to Big Data

"Big data is like moving from a minivan to a freight train — same cargo idea, totally different physics."

Learning Objectives

By the end of this module, you will be able to:

• Describe the core dimensions of “big data” and when scale actually matters
• Explain distributed computing in plain language (partitioning, shuffle, parallelism)
• Run basic read/filter/aggregate flows in PySpark
• Compare major cloud data warehouses at a high level
• Map components of the modern data stack to analyst workflows

Do You Actually Need Big Data?

If your data fits in RAM and queries finish in seconds, you might not. Big-data tooling pays off when volume, velocity, or variety breaks single-machine habits.

1. The V's of Big Data

The Classic 3 V's

When You Need Big Data (and When You Don't)

Need: petabyte logs, global clickstreams, cross-product event joins at raw grain. Don’t need: a 50MB CSV for a class project — Pandas is fine.

2. Distributed Computing Concepts

The Core Idea

Split the dataset across machines (partition), run the same operation locally, combine results. MapReduce was the original story; Spark generalizes it with a nicer programming model.

Key Concepts

Narrow transformations (filter, map) are cheap; wide (groupBy, join) shuffle data across the network — profile those.

3. Apache Spark (PySpark)

Setup

Local mode for learning; cluster managed by Databricks, EMR, Dataproc in production.

Creating DataFrames

Read Parquet (preferred), CSV, JSON. Schema inference costs a scan — specify schema in production.

Basic Operations

select, filter/where, withColumn — lazy until an action (show, count, write).

Aggregations

groupBy(...).agg(...) — same mental model as SQL/Pandas, different API.

Window Functions in Spark

Window.partitionBy(...).orderBy(...) with rank, sum over, lag — SQL window logic in DataFrames.

SQL in Spark

Register a temp view → spark.sql("SELECT ...") for teams who think in SQL.

Spark Performance Tips

• Prefer Parquet, partition keys aligned with filters
• Avoid collect on huge data
• Watch shuffle volume in Spark UI

Key Example: Minimal Spark session: read, filter, aggregate — the shape of almost every distributed pipeline.

[object Object], pyspark.sql ,[object Object], SparkSession
,[object Object], pyspark.sql ,[object Object], functions ,[object Object], F

spark = SparkSession.builder.appName(,[object Object],).getOrCreate()
df = spark.read.parquet(,[object Object],)  ,[object Object],
summary = (
    df.,[object Object],(F.col(,[object Object],) == ,[object Object],)
    .groupBy(,[object Object],)
    .agg(F.,[object Object],(,[object Object],).alias(,[object Object],))
    .orderBy(F.desc(,[object Object],))
)
summary.show(,[object Object],)

4. Cloud Data Warehouses

Google BigQuery

Serverless SQL; separates storage and compute; great for ad hoc analyst queries at scale. Watch bytes scanned = cost proxy.

Snowflake

Warehouses (compute) spin up/down; separation of storage/compute; strong multi-cloud story.

Amazon Redshift

MPP columnar warehouse; integrates with AWS data lake patterns; RA3 separates storage/compute in newer generations.

Comparison

5. The Modern Data Stack

Typical flow: ingest (Fivetran/Airbyte) → lake/warehouse (Snowflake/BigQuery) → transform (dbt) → BI (Looker/Mode/Metabase). Analysts live in SQL + BI; engineers own pipelines.

Practice Exercises

Exercise 1: PySpark Basics (Beginner)

Local Spark: load sample CSV, filter, groupBy, show.

Exercise 2: When to Scale (Intermediate)

For three scenarios, argue laptop vs warehouse vs Spark.

Exercise 3: SQL Across Platforms (Intermediate)

Same GROUP BY in SQLite and in BigQuery docs — note syntax differences.

Exercise 4: Data Format Comparison (Advanced)

Table comparing CSV vs Parquet on size, speed, schema.

Exercise 5: Cloud Warehouse Exploration (Advanced)

Free tier: run one public dataset query; note slots/bytes billed.

Mini-Project: Scalable Analytics Pipeline

Document: source → landing format → partition keys → aggregate table → BI dashboard; include one Spark or SQL snippet and a cost/perf note.

Key Takeaways

1. Scale tools to actual data size and query patterns.
2. Shuffles/joins dominate distributed cost — design partitions wisely.
3. Spark DataFrames ≈ lazy, distributed Pandas-like ops + SQL escape hatch.
4. Cloud warehouses often arrive before Spark in maturity curves.
5. Modern stack = EL + warehouse + transform + BI.

Resources for Further Learning

• Apache Spark Documentation
• Databricks Spark Guide
• BigQuery docs
• Snowflake guides

Next up: Module 11 — Data Storytelling and Communication — making your findings impossible to ignore.