Spark Catalog API on the DE interview

What the Catalog actually is

Picture the scene. You are interviewing for a Data Engineer role at Stripe or Databricks, the screen-share is open, and the senior interviewer asks: "What happens when you call spark.sql('SELECT * FROM events') — where does Spark look for that table?" Half of candidates start mumbling about HDFS paths. The other half mention Hive without knowing why. The correct answer starts with one phrase: the Spark Catalog.

The Catalog is Spark's logical namespace layer — a tree of databases, tables, views, and functions that the query optimizer consults before it ever touches a file. Without the Catalog, every query would need a fully-qualified path like parquet.\s3://bucket/year=2026/month=05/`. With it, the engine can resolve analytics.events` into a real location plus a schema plus partition metadata plus statistics — all in a single metastore call. That single layer of indirection is the difference between an ad-hoc script and a data warehouse.

In a Spark session you reach the Catalog via spark.catalog. It exposes the basics every interviewer expects you to recognize on sight.

spark.catalog.listDatabases()
spark.catalog.listTables('analytics')
spark.catalog.tableExists('analytics.events')
spark.catalog.currentDatabase()
spark.catalog.setCurrentDatabase('analytics')

If the candidate cannot name three methods on spark.catalog, the rest of the interview is mostly downhill. The metadata calls feel boring, but in production they prevent the most common runtime crash: querying a table that does not exist, then waiting four minutes for the executor to report it.

Temp views vs global temp views

A temp view is session-scoped, lives in the driver's memory only, and disappears the instant you call spark.stop(). It exists so you can register a DataFrame under a name and run SQL against it without touching any metastore.

df.createOrReplaceTempView('orders_today')
spark.sql('SELECT customer_id, SUM(amount) FROM orders_today GROUP BY 1')

Notice that the view is invisible from a second Spark session pointing at the same cluster. That confuses juniors who expect temp views to behave like Postgres tables. They do not. They are basically a name attached to a logical plan inside one driver JVM.

A global temp view widens the scope to the entire Spark application — multiple sessions in the same application can see it — but it still dies when the cluster restarts. The catch is that global temp views live under a special database called global_temp, so you must qualify the name.

df.createGlobalTempView('orders_global')
spark.sql('SELECT * FROM global_temp.orders_global')

Interview gotcha: if the question is "how do I share a DataFrame between two notebooks on the same cluster?", the answer is global temp view — not a temp view, not a persistent table.

Persistent tables and saveAsTable

The moment you call df.write.saveAsTable('analytics.events'), four things happen in order. Spark writes the data files to a managed location, registers a row in the metastore, attaches the schema, and stores partitioning plus statistics. The result is a persistent table that survives session and cluster restarts.

(df
  .write
  .mode('overwrite')
  .partitionBy('event_date')
  .saveAsTable('analytics.events'))

spark.sql('SELECT COUNT(*) FROM analytics.events WHERE event_date = DATE \'2026-05-22\'')

The distinction interviewers care about is managed vs external. A managed table means Spark owns the files — if you drop the table, the files are deleted. An external table (created with an explicit LOCATION) means Spark only owns the metadata — drop the table and the files survive. Use external tables for anything other teams might read. Managed tables are fine for staging layers no one else depends on.

Catalog implementations compared

Spark does not ship one Catalog — it ships an interface, and you choose the implementation via spark.sql.catalogImplementation. The interviewer will probe whether you understand the trade-offs across at least three of them.

The in-memory catalog is the default in a fresh SparkSession. It is fine for tests, useless for anything shared. The instant a second engine — Trino, Flink, Athena — needs to read your tables, you graduate to a persistent metastore.

Hive Metastore is still the workhorse in 2026 despite being older than most data engineers. It is battle-tested, JDBC-backed, and supported by every engine in the ecosystem. The downside is operational: you run a MySQL or Postgres behind it and pray it never hits connection limits at peak load.

AWS Glue Data Catalog is essentially a managed, Hive-compatible metastore. If your stack is on AWS, Glue is the path of least resistance — it integrates with Athena, Redshift Spectrum, EMR, and Lake Formation for permissions. The bill can surprise you at high partition counts, so cap the partition explosion early.

Unity Catalog is the Databricks flagship. Three-level namespace (catalog.schema.table), row-level and column-level ACLs, native Delta integration, and lineage built in. If the interview is at a Databricks shop, this is the only answer that earns full marks.

Polaris and Iceberg REST represent the open, engine-agnostic future. Polaris (open-sourced by Snowflake) speaks the Iceberg REST spec, which means Spark, Trino, Flink, and Snowflake can all read and write the same physical tables with consistent metadata. The selling point is no vendor lock-in at the metadata layer.

Iceberg and Delta integration

The modern interview rarely stops at Hive. Expect the follow-up: "How would you build a lakehouse table on this Catalog?" This is where Apache Iceberg and Delta Lake show up.

Iceberg integration is configured at session level and then the table is just SQL.

spark.sql("""
  CREATE TABLE analytics.events (
    id BIGINT,
    user_id BIGINT,
    event_type STRING,
    ts TIMESTAMP
  ) USING iceberg
  PARTITIONED BY (days(ts))
""")

What you get for that one keyword change: snapshot isolation, time travel, hidden partitioning, and schema evolution without rewrites. Hidden partitioning is the killer feature — queries that filter on ts automatically use the days(ts) partition, so users do not need to remember to add WHERE days(ts) = .... That alone removes a class of slow-query bug reports.

Delta Lake fills the same niche with a different bias. It is Databricks-first, but the open-source format works on any Spark cluster.

(df
  .write
  .format('delta')
  .partitionBy('event_date')
  .mode('overwrite')
  .save('s3://lake/analytics/events'))

spark.sql("""
  CREATE TABLE analytics.events
  USING delta
  LOCATION 's3://lake/analytics/events'
""")

Both formats give you ACID transactions, time travel via VERSION AS OF / TIMESTAMP AS OF, and MERGE INTO for upserts. The honest summary is: Iceberg wins on multi-engine reads, Delta wins inside the Databricks ecosystem. Both will appear in interviews; know one deeply and the other well enough to compare.

Common pitfalls

The first pitfall is confusing temp views with cached DataFrames. A temp view is just a name pointing at a logical plan — it does not persist results. If you create a temp view from a 10-minute aggregation and query it three times, you run that aggregation three times. The fix is to call .cache() or .persist() on the DataFrame before registering the view, or to materialize the result with saveAsTable for anything used in more than a single notebook cell.

A close cousin is assuming saveAsTable is idempotent. It is not — by default it appends or errors depending on the save mode. Production pipelines should pass mode('overwrite') or use MERGE INTO on an Iceberg or Delta table. Otherwise a rerun after a failure either doubles rows or hard-fails on AlreadyExistsException, and neither outcome is what your on-call wants at 3 AM.

The third trap is leaking managed tables across environments. Junior pipelines often run saveAsTable('events') against a Glue catalog shared by staging and production. Because the table name is unqualified, Spark resolves it via the current database, which is default, which is also where the production analyst is reading from. The fix is to always fully-qualify table names — staging_analytics.events, never events — and to enforce that in code review.

A fourth pitfall worth naming is Iceberg table corruption from concurrent writes with no commit retries. Iceberg uses optimistic concurrency: two writers that touch overlapping partitions race on the metadata commit, and the loser must retry. If your Spark job has zero retry logic and you run two backfills in parallel, one of them will fail with a CommitFailedException. Configure spark.sql.iceberg.commit.retry.num-retries and serialize backfills that touch the same partitions.

Finally, forgetting that catalogs are not interchangeable for ACLs. Unity Catalog enforces row-level and column-level filters that Hive simply ignores. If you migrate a pipeline from Unity to a generic Hive Metastore and forget to re-implement masking in SQL, you have created a data leak. Every catalog migration is also an access-control migration.

Related reading

• Apache Iceberg deep dive
• Lakehouse: Iceberg vs Delta
• Iceberg time travel
• Spark broadcast joins
• Spark Catalyst and AQE
• Spark + Iceberg integration

If you want to drill DE interview questions like Spark Catalog every day, NAILDD has 1,500+ data engineering problems across exactly these patterns.

FAQ

What is the difference between spark.catalog and spark.sessionState.catalog?

The first is the public API — stable, documented, safe to call from user code. The second is the internal API that the optimizer uses, and it changes between Spark releases without notice. Interview answer: always use spark.catalog in code, mention the internal one only if asked about the optimizer pipeline.

When should I use an external table instead of a managed one?

Use external whenever another team or another engine might read the files. Managed tables are convenient because Spark cleans up files when you drop them, but that "convenience" becomes a footgun the moment a downstream job depends on those paths. The rule of thumb is: anything beyond a personal staging table should be external with an explicit LOCATION.

How does Spark resolve an unqualified table name like events?

It walks a resolution chain. First it checks temp views in the current session, then global temp views in the global_temp database, then the current database in the active catalog. If none match, it raises AnalysisException: Table or view not found. The lesson is to always qualify table names in production code — never rely on the current database being what you think it is, because the next person to open the notebook will be set to default.

Can I use multiple catalogs in the same Spark session?

Yes, since Spark 3.0 the multi-catalog API lets you register several catalogs and reference them by prefix — glue.analytics.events, unity.production.events. You switch active catalog with spark.sql('USE CATALOG glue'). This is how organizations bridge Hive and Iceberg during migration: register the new Iceberg REST catalog alongside the old Hive one and migrate table-by-table.

Is Hive Metastore deprecated in favor of Unity or Polaris?

Not deprecated, but no longer the default choice for greenfield projects. New lakehouse stacks lean toward Iceberg REST (Polaris, Tabular, or self-hosted) for engine-agnostic access, or Unity Catalog for Databricks-native shops. Hive remains entrenched in legacy clusters and EMR, and will keep running for years — but if you are designing fresh, an Iceberg-native catalog is the modern answer.

What does REFRESH TABLE actually do?

It invalidates Spark's cached metadata for a table — the schema, file listing, and statistics. If an external process wrote new files to the table's location while Spark was running, the next query may not see them until you call REFRESH TABLE analytics.events. This is the single most common cause of "but the data is there, why does Spark not see it?" support tickets.