Module 8: Advanced Data Processing | Data Analyst

"Fast enough isn't. When your data grows from thousands to millions of rows, the techniques that 'worked fine' will bring your machine to its knees."

Learning Objectives

By the end of this module, you will be able to:

Optimize Pandas operations for large datasets (vectorization, eval, query)
Manage memory efficiently with appropriate dtypes and chunked reading
Write clean, readable code using method chaining
Build analyses in Polars with lazy evaluation and expressions
Benchmark Pandas vs. Polars and choose the right tool for the job

Part 1: Pandas Performance Optimization

Why Your Pandas Code Is Slow

Three usual suspects: Python loops over rows (especially .iterrows()), accidental copies, and fat dtypes (object, float64 everywhere).

Smell	Fix direction
`.iterrows()`	Vectorize, `np.select`, `apply` only if needed
Giant intermediates	Chain, reuse views, drop unused columns
`object` columns	Categoricals, smaller numerics

Rule 1: Vectorize Everything

np.where / np.select for branching on whole columns beats row-wise Python. Think in arrays, not records.

Rule 2: Use eval() and query() for Complex Operations

DataFrame.eval and .query push work through numexpr-style paths for large frames — handy for arithmetic-heavy filters on many columns.

Rule 3: Optimize Data Types

Downcast integers, use category for low-cardinality strings, float32 if precision allows. Halving width ≈ halving memory.

Rule 4: Chunked Reading for Huge Files

read_csv(chunksize=...) or iterator=True lets you aggregate in passes without loading everything at once.

Method Chaining — Clean, Readable Pipelines

.pipe for custom steps; assign with lambdas; keep a linear story: load → clean → feature → aggregate. Easier to test and profile.

Try This! Print df.memory_usage(deep=True).sum() before and after dtype tuning on a sample file.

Part 2: Introduction to Polars

Why Polars?

Polars is a columnar DataFrame library built for multi-core and lazy query optimization. Often much faster than Pandas on big workloads; API is expression-based.

Pandas habit	Polars spirit
In-place `df["x"]=...`	Expressions + `with_columns`
Eager everything	`lazy()` then `collect()`
Mixed index tricks	Explicit columns

Polars Expressions

.select, .filter, .with_columns with pl.col("a").mean() etc. — composable and parallel-friendly.

Lazy Evaluation — Polars Superpower

Scan parquet/csv → build lazy plan → collect() once. The engine can push down predicates and projection before reading full data.

Scan — Read Files Lazily

pl.scan_parquet, pl.scan_csv for out-of-core style workflows.

Performance Comparison: Pandas vs. Polars

Benchmark on your data: groupbys, joins, filters. Polars often wins on wide tables and big aggregations; Pandas still wins on ecosystem familiarity and stack overflow depth.

When to Use Pandas vs. Polars

Choose Pandas	Choose Polars
Team knows it cold	Speed/memory critical
Library needs pandas object	ETL on huge parquet
Quick one-offs	Lazy pipelines on big files

Chunked reads (Pandas reminder): pd.read_csv(path, chunksize=100_000) returns an iterator — aggregate per chunk (running sums, filters) to stay under RAM. Pair with explicit dtype and usecols.

Polars mental model	One-liner
Add column	`with_columns`
Filter	`filter`
Lazy plan	`scan_parquet(...).filter(...).select(...).collect()`

Key Example: Same idea in Pandas (vectorized) vs Polars (expressions) — categorizing revenue tiers without Python row loops.

python

[object Object], numpy ,[object Object], np
,[object Object], pandas ,[object Object], pd
,[object Object], polars ,[object Object], pl

pdf = pd.DataFrame({,[object Object],: [,[object Object],, ,[object Object],, ,[object Object],, ,[object Object],]})
pdf[,[object Object],] = np.select(
    [pdf[,[object Object],] > ,[object Object],, pdf[,[object Object],] > ,[object Object],],
    [,[object Object],, ,[object Object],],
    default=,[object Object],,
)

ldf = pl.DataFrame({,[object Object],: [,[object Object],, ,[object Object],, ,[object Object],, ,[object Object],]})
ldf = ldf.with_columns(
    pl.when(pl.col(,[object Object],) > ,[object Object],)
    .then(pl.lit(,[object Object],))
    .when(pl.col(,[object Object],) > ,[object Object],)
    .then(pl.lit(,[object Object],))
    .otherwise(pl.lit(,[object Object],))
    .alias(,[object Object],),
)

At-a-glance: slow vs fast patterns

Slow pattern	Faster pattern
`for i, row in df.iterrows()`	Vectorized ops, `itertuples` only if needed
Repeated `df.copy()` in a loop	One copy before loop or chain without copies
`object` dtype for 50-level category	`category` dtype
Reading 10 GB CSV in one go	`chunksize` or Polars `scan_csv` lazy
Giant `.apply(lambda x: ...)`	NumPy ufuncs or Polars expressions
Multiple `.merge` on wide frames	Push joins to SQL/warehouse when possible

Profiling quick wins: %%time / time.perf_counter() around the suspect cell; memory_profiler if RAM spikes. Often one groupby or one bad join dominates — fix that before micro-optimizing syntax.

When Polars wins most: Parquet-heavy pipelines, parallel reads, complex expression chains with lazy optimization. When Pandas wins: Stack Overflow answers, sklearn input, quick notebooks, legacy team codebases.