Spark for analysts: when Pandas runs out of room

What Spark actually is

Apache Spark is a framework for distributed processing of large datasets. Where Pandas runs on a single machine and keeps the full dataframe in RAM, Spark splits work across a cluster of dozens or hundreds of nodes. The headline number to remember is the ~10 GB RAM ceiling for comfortable single-machine analytics — past that point, Pandas turns from a tool into a trap.

For an analyst, Spark shows up the moment data outgrows your laptop. At Netflix, Uber, DoorDash, Snowflake, Databricks, and similar shops, Spark is part of the standard analyst stack — sitting next to dbt, Airflow, and the warehouse. Interviewers rarely grill you on Spark internals; they care whether you can write a clean PySpark job, switch to Spark SQL when it reads better, and explain why a join exploded the cluster.

This is the minimum viable Spark knowledge for an analyst who already writes SQL and some Python: when to reach for Spark, the architecture you must be able to draw on a whiteboard, the PySpark patterns that cover most jobs, and the pitfalls that wreck the rest.

When Pandas runs out of room

Pandas loads the entire dataset into memory and then processes it on a single CPU core. That model is great until it isn't. A 10 GB CSV will eat roughly 30 GB of RAM after Pandas materializes intermediate copies during a groupby chain — on a 16 GB MacBook, the kernel dies before you see a result. A groupby over 100 million rows runs serially on one core, so you watch a progress bar for ten minutes and pray nothing crashes mid-way. If the process does crash, you start from scratch, because Pandas has no concept of fault tolerance.

Spark fixes all three problems at once. Data is partitioned across executor nodes, work runs in parallel across cores and machines, and when an executor dies, the cluster manager reschedules the failed tasks on another node. You pay for this in startup overhead and debugging friction — which is exactly why Spark is the wrong tool when your data fits comfortably in Pandas.

Spark architecture: driver, executors, cluster manager

Spark uses a master-worker topology. You write code; the driver builds a plan; executors do the work; the cluster manager hands out machines. If you can sketch this on a whiteboard from memory, you've cleared the "do they know what Spark is" bar for most analyst interviews.

When you write df.groupBy("city").count(), Spark does not execute anything yet. It builds a DAG (Directed Acyclic Graph) of transformations, hands that DAG to the Catalyst optimizer, and only runs the physical plan when you call an action — .show(), .collect(), .write, .count(). This pattern is called lazy evaluation, and it is the single most useful Spark concept to internalize. Lazy evaluation is why you can chain twenty filters and joins without paying any cost until the final .write.parquet(...) line.

PySpark basics

PySpark is the Python API for Spark. The syntax rhymes with Pandas but is deliberately not identical — F.col("revenue") instead of df["revenue"], groupBy() not groupby(), and column expressions rather than direct mutation.

Session and reading data

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName("analytics")
    .getOrCreate()
)

# CSV — fine for one-off exploration, slow at scale
df = spark.read.csv("s3://bucket/orders.csv", header=True, inferSchema=True)

# Parquet — the default for production data lakes
df = spark.read.parquet("s3://bucket/orders/")

Parquet is columnar, compressed, and self-describing. If your team is still writing pipeline outputs to CSV in 2026, that is the first thing to fix.

Filter, group, window

from pyspark.sql import functions as F
from pyspark.sql.window import Window

active = df.filter(F.col("status") == "active")

revenue_by_city = (
    df
    .groupBy("city")
    .agg(
        F.sum("revenue").alias("total_revenue"),
        F.countDistinct("user_id").alias("users"),
    )
    .orderBy(F.desc("total_revenue"))
)

# Top-3 by revenue inside each category
w = Window.partitionBy("category").orderBy(F.desc("revenue"))
top3 = (
    df
    .withColumn("rank", F.row_number().over(w))
    .filter(F.col("rank") <= 3)
)

Joins

orders = spark.read.parquet("s3://lake/orders/")
users = spark.read.parquet("s3://lake/users/")

joined = orders.join(users, orders.user_id == users.id, "left")

The default join strategy is a sort-merge join, which shuffles both sides across the network. For a small dimension table (under roughly 100 MB), broadcast it explicitly with F.broadcast(users) — this ships the table to every executor and skips the shuffle. Spark will sometimes do this automatically via Adaptive Query Execution, but doing it on purpose is the more reliable habit.

Spark SQL

If you already write SQL, Spark SQL is the lowest-friction way into Spark. Register a DataFrame as a temporary view and run ANSI-ish SQL against it.

-- After df.createOrReplaceTempView("orders") in Python
SELECT
    city,
    COUNT(DISTINCT user_id) AS users,
    SUM(revenue)            AS total_revenue,
    AVG(revenue)            AS avg_order
FROM orders
WHERE status = 'paid'
GROUP BY city
HAVING COUNT(DISTINCT user_id) >= 100
ORDER BY total_revenue DESC
LIMIT 10;

Spark SQL supports window functions, CTEs, subqueries, and lateral joins. Under the hood, the SQL parser and the DataFrame API both compile to the same Catalyst logical plan, so performance is identical. Pick whichever reads better for the task — multi-step joins and conditional logic are often clearer in PySpark; pure aggregation pipelines often read better in SQL.

PySpark vs Spark SQL vs Pandas

When an analyst actually meets Spark

In practice, Spark crosses an analyst's desk in four recurring shapes. First, ETL pipelines: a Spark job aggregates raw events into the daily fact tables your dashboards read from, and either you wrote it or you are debugging it after it broke at 3 a.m. Second, ad-hoc analysis on lake-scale data — your PM asks for a metric across all users for the last year, and the source is 500 million rows in S3. Third, feature engineering for ML, where features are computed on a cluster because a single machine would take a day. Fourth, lake-native analytics in Databricks, EMR, or Snowflake's Snowpark — when raw Parquet lives in S3 or GCS and there is no warehouse copy yet.

Common pitfalls

The first trap is forgetting that Spark is lazy. A new analyst writes ten transformations, runs the cell, sees it finish in 200 ms, and thinks Spark is magic. The real work only starts on the first action — and that is where the OOM error lands. Fix: when timing or debugging, end the chain with .count() or .write to force the plan to run, and read the Spark UI to see what actually executed.

The second trap is wide transformations on skewed data. A groupBy on a column where 90% of rows share one value (think country = "US" in a US-heavy product) sends nearly all data to a single executor. That executor either OOMs or runs for an hour while the rest of the cluster sits idle. The fix is to salt the skewed key (append a random integer to spread the load) or enable Adaptive Query Execution skew handling — spark.sql.adaptive.skewJoin.enabled = true.

The third trap is using collect() on anything other than a tiny result. df.collect() pulls every row to the driver. If the dataframe has 100 million rows, the driver dies. Use df.show(20) to peek, df.write.parquet(...) to materialize, and toPandas() only after you have aggregated to a few thousand rows or fewer.

The fourth trap is shuffling because of a missing broadcast hint. Joining a 10 TB fact table to a 50 MB dimension without F.broadcast(dim) produces a sort-merge join that shuffles the fact table across the network. The fix is one line of code and often cuts job runtime by 5-10x. This single change has rescued more interview take-home assignments than any other Spark tip.

The fifth trap is writing tiny Parquet files. A job that writes one file per partition with no repartition step can produce tens of thousands of files, each a few KB. Downstream readers crawl because object-storage list operations dominate. Before writing, call .repartition(N) or .coalesce(N) to target file sizes in the 128-512 MB range.

Related reading

• SQL window functions interview questions
• Pandas memory optimization guide
• Polars vs Pandas for analysts
• Spark broadcast joins — data engineering interview
• Spark Catalyst and AQE — data engineering interview

If you want to drill PySpark and Spark SQL questions the way real interviewers ask them, NAILDD is launching with worked problems that cover exactly this pattern.

FAQ

Does an analyst really need Spark, or is SQL plus Pandas enough?

It depends on where you work. At companies operating on tens or hundreds of terabytes — Netflix, Uber, DoorDash, Stripe, Databricks, Snowflake — basic PySpark and Spark SQL fluency is table stakes. At an early-stage startup with a few hundred GB in the warehouse, you can go your entire career on SQL plus Pandas. If you are targeting large-cap or growth-stage data teams, treat Spark as a baseline skill alongside Python and SQL.

Can I learn Spark locally without a cluster?

Yes. pip install pyspark gives you a working installation that runs in local[*] mode, using every core on your laptop as a fake executor. You won't see distributed-system behavior (shuffles across a network, executor failures) but you can practice the API, the DAG mental model, and the Spark SQL syntax. For free cluster experience, Databricks Community Edition gives you a small cluster with a notebook UI, and Google Colab can run PySpark in a notebook with one pip install.

What is the practical difference between Spark and Hadoop MapReduce?

Spark is the modern descendant. MapReduce writes intermediate results to disk after every stage, which makes iterative work — ML loops, multi-step aggregations — painfully slow. Spark keeps intermediate data in executor memory and only spills under pressure, making it 10-100x faster on those workloads. Almost no one writes new MapReduce jobs today.

When should I use Spark SQL vs the DataFrame API?

Use Spark SQL when the logic is a clean SELECT ... GROUP BY ... HAVING ... and a SQL reader will understand it instantly. Use the DataFrame API when you need conditional branching, UDFs, complex column expressions, or programmatic generation of the query (looping over columns, building joins from a config). They compile to the same Catalyst plan, so pick on readability, not performance.

What's the deal with repartition() vs coalesce()?

Both change the number of partitions in a DataFrame. repartition(N) does a full shuffle and produces evenly sized partitions — use it when you need to increase partition count or fix skew. coalesce(N) merges existing partitions without a shuffle and can leave uneven sizes — use it only when reducing partition count, typically right before a write. A common mistake is calling coalesce(1) to write a single output file from a job processing terabytes; this forces all data through one executor and usually OOMs. If you must have one file, write to many partitions, then run a separate compaction job.

How do I debug a Spark job that's slow but not failing?

Open the Spark UI — every Spark deployment exposes it on port 4040. In the stages tab, find the longest stage and check task duration distribution: if one task takes 10 minutes and the rest take 10 seconds, you have data skew. Check shuffle read/write columns; if a stage shuffles hundreds of GB, look for a missing broadcast hint or an over-eager repartition. Learning to read this UI turns Spark from a black box into something tractable.