Module 3: Pandas — Data Manipulation Mastery | Data Analyst

"Pandas is to data analysts what a scalpel is to surgeons — precise, versatile, and absolutely essential for the job."

Learning Objectives

By the end of this module, you will be able to:

Create and manipulate Series and DataFrame objects
Select data using loc, iloc, and boolean indexing
Aggregate data with groupby, pivot tables, and cross-tabulations
Combine datasets with merge, join, and concat
Work with dates, apply custom functions, and read/write multiple file formats

Why Pandas?

NumPy is a fast grid of numbers. Pandas adds names: column labels, row index, mixed types in one table, and first-class missing values (NaN).

Layer	Analogy
NumPy	Blank spreadsheet grid
Series	One named column
DataFrame	Full Excel sheet with headers

You’ll almost always start with import pandas as pd and import numpy as np when you vectorize.

1. Series — The Building Block

A Series is one column with an index — like a dictionary that cares about order. You slice it, do math on the whole thing, and filter with booleans; labels stay attached.

Try This! Build a Series of quarterly revenue labeled Q1–Q4 and compute quarter-over-quarter percent change.

Concept: Anything you can do to a NumPy 1D array, you can usually do to a Series — plus .loc by label.

2. DataFrame — The Star of the Show

Creating DataFrames

Most often: pd.DataFrame({"col": list, ...}). From APIs you’ll see list of dicts; from simulations you might wrap NumPy with columns=[...].

Check	Purpose
`.head()` / `.tail()`	Preview edges
`.info()`	Types, nulls, memory
`.describe()`	Numeric summaries
`.shape`	Rows × columns

From Other Sources

Same idea: records from JSON, arrays from NumPy — anything rectangular becomes a DataFrame when you need mixed types and names.

Fun Fact: read_csv is often the first function you run on day one of a new dataset.

3. Selecting Data: `loc` and `iloc`

Selector	Uses	Slice behavior
`loc`	Labels	Inclusive on both ends
`iloc`	Integer positions	Exclusive end (like Python lists)

That inclusive vs exclusive difference trips up almost everyone once.

Boolean Filtering

One condition: df[df["salary"] > 75000]
Several: (df["dept"] == "X") & (df["years"] > 3) — parentheses required
.isin([...]) for “any of these categories”
.query("salary > 70000") for SQL-ish readability

4. Adding and Modifying Columns

Assign with df["new"] = df["a"] * df["b"]. For tiered logic, np.where / np.select beats giant nested if loops. Use .apply when you truly need row-wise Python — it’s flexible but slower.

5. GroupBy — Split-Apply-Combine

Split by key, apply aggregations, combine into a summary. Named agg keeps outputs self-documenting. .transform broadcasts group stats back to every row — great for “% of region total.”

6. Merging and Joining DataFrames

merge — SQL-style Joins

`how`	Keeps
`inner`	Matching keys only
`left`	All left keys
`right`	All right keys
`outer`	Union of keys; gaps become NaN

Use on= when names match; left_on / right_on when they don’t.

concat — Stacking DataFrames

axis=0 stacks rows; axis=1 pastes columns. ignore_index=True resets row labels after a vertical stack.

7. Pivot Tables and Reshaping

Pivot tables: pick index, columns, values, and aggfunc — Excel pivot, but code. Melt: wide subject columns → long rows. Crosstab: counts or sums for two categoricals.

Task	Pandas verb	Shape change
Spread metrics across columns	`pivot_table`	Long → wide summary
Unpivot columns to rows	`melt`	Wide → tidy long
Frequency table	`crosstab`	Two dims → grid

Fun Fact: Many “tidy data” fights disappear once you standardize on one row per observation and melt/pivot intentionally.

Format	When
CSV	Sharing everywhere
Parquet	Speed and size at scale
Excel	Business stakeholders
JSON	APIs
SQL	`read_sql` / `to_sql`

On big CSVs, pass dtype or usecols to save memory and avoid bad type guesses.

Key Example: Selection, a derived column, aggregation, and a merge — the core loop of analyst work.

python

[object Object], pandas ,[object Object], pd

df = pd.DataFrame({
    ,[object Object],: [,[object Object],, ,[object Object],, ,[object Object],, ,[object Object],],
    ,[object Object],: [,[object Object],, ,[object Object],, ,[object Object],, ,[object Object],],
    ,[object Object],: [,[object Object],, ,[object Object],, ,[object Object],, ,[object Object],],
})
df[,[object Object],] = (df[,[object Object],] / ,[object Object],).,[object Object],(,[object Object],)
by_dept = df.groupby(,[object Object],, as_index=,[object Object],).agg(
    headcount=(,[object Object],, ,[object Object],),
    avg_salary=(,[object Object],, ,[object Object],),
)
budgets = pd.DataFrame({,[object Object],: [,[object Object],, ,[object Object],], ,[object Object],: [,[object Object],, ,[object Object],]})
report = by_dept.merge(budgets, on=,[object Object],, how=,[object Object],)

Practice Exercises

Exercise 1: Employee Analysis (Beginner)

Ten employees: average salary by department, longest tenure, salary range per department.

Exercise 2: Sales Dashboard Data (Intermediate)

From daily sales: monthly revenue, top 5 products by revenue, pivot of revenue by region and month.

Exercise 3: Data Merging Challenge (Intermediate)

Merge students, courses, enrollments: enrollments per course, average grade per course, top GPA.

Exercise 4: Multi-level GroupBy (Advanced)

Per region and product: revenue, % of regional total, rank within region.

Exercise 5: Method Chaining (Advanced)

Redo exercise 2 with chained .assign / groupby / agg only.

Mini-Project: Customer Segmentation Analysis

Synthetic e-commerce data: compute RFM (recency, frequency, monetary), assign segments, summarize segment sizes and top customers.

Key Takeaways

Master loc, iloc, and boolean filters first.
groupby + agg replaces most Excel pivot workflows.
merge is your SQL JOIN; know inner vs left cold.
Parse dates early; time series wants a proper DatetimeIndex.
Parquet for scale, CSV for portability.

Resources for Further Learning

Pandas Official Getting Started
10 Minutes to Pandas
Pandas Cookbook by Julia Evans
Modern Pandas blog series
Kaggle Pandas Course (free, interactive)

Key Takeaway

Model tables as DataFrames and one-dimensional extracts as Series.
Select with loc/iloc deliberately; remember inclusive vs exclusive slicing.
Summarize with groupby and named agg instead of manual loops.
Combine tables with merge/concat like SQL joins and unions.
Parse dates with to_datetime and use .dt plus resample for time patterns.

Next up: Module 4 — Data Cleaning and Preprocessing — because real-world data is never this neat.

Module 3: Pandas — Data Manipulation Mastery