HDFS for Data Engineer interviews

Why HDFS still shows up in interviews

You will get an HDFS question even at shops that ripped out Hadoop years ago. The reason is not nostalgia. HDFS is the cleanest teaching example of distributed storage trade-offs — block sizing, metadata pressure, replication geometry — and every modern lakehouse (Iceberg on S3, Delta on ADLS, Hudi on GCS) inherits the same primitives. If you can reason about HDFS, you can reason about why your Snowflake bill spikes and why a tiny-files problem kills Iceberg compaction.

Most DE candidates lose points not on definitions but on the why. Why 128 MB blocks and not 4 KB like a Linux filesystem? Why does the NameNode collapse at 400 million files but happily serve 40 million files of 100 MB each? Why is replication factor 3 — not 2 or 5? Interviewers are testing whether you understand the cost surface, not whether you can recite the Hadoop docs.

HDFS architecture in one diagram

HDFS is a master-worker system. One process keeps the map of the filesystem in RAM, many processes hold the bytes on local disks, clients talk to both. The split between metadata and data is the load-bearing trick of the entire design.

        Client
          |
   1. open("/sales/y=2026/m=05/part.parquet")
          v
       NameNode  --- metadata ops (mkdir, rename, getBlockLocations)
       (heap-resident inode tree + block map)
          |
   2. returns [block_1: DN-a, DN-b, DN-c]
              [block_2: DN-d, DN-b, DN-e]
          |
   3. client reads bytes directly from DataNodes
          v
   DataNode-a   DataNode-b   DataNode-c   DataNode-d   DataNode-e
   (local disks, heartbeats + block reports every few seconds)

The NameNode never touches the bytes. That single decision is why HDFS scales linearly with DataNodes for throughput — and why it falls apart when you push the NameNode past its RAM budget. Keep this picture in your head when you answer any HDFS question; almost every trade-off in the system reduces to "what does this cost the NameNode?"

Blocks and why 128 MB is the default

Files are sliced into fixed-size blocks, default 128 MB in Hadoop 2+ and 3+ (the legacy default was 64 MB). A 1 GB file becomes 8 blocks; a 500 KB file becomes 1 block of 500 KB — blocks are a maximum size, not a minimum.

Why 128 MB and not 4 KB like ext4? Three reasons stacked.

A common follow-up: would you ever change the block size? Yes — for cold archive scans of terabyte-scale Parquet files teams raise to 256 MB or 512 MB to reduce task overhead in Spark. The answer interviewers want is "I tune block size against the read pattern, not against the data size."

NameNode and DataNode roles

The NameNode holds three things in memory: the inode tree (paths, permissions, owners), the block-to-DataNode map, and edits-log state. It does not persist the block locations — DataNodes report their blocks at startup and via periodic block reports. This is why a freshly started NameNode goes through safe mode: it is waiting for enough DataNodes to report so it knows where the blocks actually live.

The DataNode is the dumb-but-honest worker. It stores blocks as local files, sends heartbeats every 3 seconds, and ships a full block report every 6 hours by default. If the NameNode misses heartbeats for 10 minutes and 30 seconds, it declares the DataNode dead and schedules re-replication.

The NameNode is a database, the DataNode is a file server, and the client is the join engine. At roughly 1 GB of heap per million blocks, a NameNode with 64 GB of heap caps out around 64 million blocks — the practical reason federation and HA exist.

Replication and rack awareness

Default replication factor is 3. The placement policy is opinionated and worth memorizing because interviewers love asking "where does HDFS put the second replica?"

The asymmetry is deliberate. One copy survives a node failure, two copies survive a rack failure, and you only pay one cross-rack network hop on write — not two. If the policy were "spread across three racks," every write would saturate the cross-rack uplinks.

A senior follow-up: what happens if a write succeeds to only 2 of 3 replicas? HDFS uses pipeline writes — the client writes to DN-1, which forwards to DN-2, which forwards to DN-3. The write returns success once dfs.namenode.replication.min copies (default 1) are durable, and the NameNode lazily heals to the target factor.

Federation, HA, and erasure coding

HA runs an active NameNode and one (or more) standby. Both tail the same edit log via JournalNodes (quorum journal manager). ZooKeeper Failover Controller (ZKFC) fences the old active and promotes the standby on failure.

Federation lets you run multiple independent NameNodes, each serving a slice of the namespace (/finance, /ml, /raw). DataNodes register with all of them. Most modern shops skip federation and go straight to object storage instead.

Erasure Coding replaces 3x replication with Reed-Solomon (6,3) stripes — 6 data + 3 parity blocks, total 1.5x overhead vs 3x for replication, a 50% storage saving. The catch: partial reads require reconstruction (XORing parity blocks), fine for cold archive data and terrible for hot interactive queries.

HDFS vs S3 — the question that always comes

In 2026 almost no greenfield project deploys HDFS. Object stores won. But the comparison still matters because the failure modes carry over.

The decoupling of storage and compute is the real reason cloud storage won. With HDFS, adding compute means adding disks you do not need; adding storage means adding CPUs you do not need. Snowflake, Databricks, and lakehouses on S3 broke that coupling.

The rename problem is worth one extra beat in an interview. On HDFS, mv is a NameNode metadata edit — atomic and instant. On S3, mv is a server-side copy followed by delete of every object, turning 30-second Hive overwrites into 30-minute jobs. This is why Iceberg, Delta, and Hudi exist: they replace directory-rename atomicity with table-format metadata pointers.

Common pitfalls

The number-one HDFS failure in production is the small files problem. Each file consumes one inode plus at least one block on the NameNode, both costing roughly 150 bytes of heap each. A team that lands 50 million 10 KB JSON files per day will exhaust a 64 GB NameNode heap inside three months, and the failure mode is full cluster outage — not a graceful slowdown. The fix is upstream: compact at the ingestion layer, write Parquet or ORC in 64-256 MB chunks, and use HAR (Hadoop Archive) or SequenceFiles for unavoidable small-file workloads.

The second classic pitfall is misunderstanding replication as backup. Replication protects against hardware failure on a single cluster — it does not protect against accidental rm -r /critical/path, against bad ETL writing zeros to all replicas simultaneously, or against a misconfigured DistCp deleting both source and destination. Trash interval (fs.trash.interval) and off-cluster snapshots (or external backups to S3) are what protect data; replication is uptime engineering, not durability engineering.

A third trap is assuming block locality always helps. Spark and Hive use locality hints to schedule tasks on nodes holding the data, but on a busy cluster the scheduler falls back to rack-local. On modern fast networks (25 Gbps+ NICs), rack-local performance is within 5-10% of node-local, and chasing strict locality often slows down the cluster more than it helps.

The fourth pitfall is NameNode heap miscalibration during incident recovery. When you restart the NameNode after a crash, it loads the fsimage and replays edits, which on a billion-block cluster takes 15-45 minutes. During this period the cluster is in safe mode and writes fail. Teams that do not run rolling restarts on HA pairs turn a 5-minute config change into a 30-minute outage.

Related reading

• Spark RDD vs DataFrame vs Dataset
• Spark fundamentals
• Apache Iceberg deep dive
• DWH modeling
• Data engineer resume guide

If you want to drill DE questions like HDFS sizing, replication geometry, and the S3 trade-off every day, NAILDD is launching with curated interview prep for Data Engineering panels.

FAQ

Is HDFS dead in 2026?

For greenfield deployments, mostly yes — new analytics platforms are built on object storage with table formats (Iceberg, Delta, Hudi) on top. But HDFS is still running in tens of thousands of on-prem clusters, especially in regulated industries where data-residency requirements rule out public-cloud object stores. Knowing HDFS is still a hiring filter for senior DE roles at any company with a legacy Hadoop estate.

Why is the default block size 128 MB instead of bigger?

128 MB is a compromise. Smaller blocks waste NameNode heap and produce too many Spark/MR tasks; larger blocks reduce parallelism and waste space when files are smaller than the block size. 256 MB or 512 MB is common for cold archive workloads with very large Parquet files. Modern Iceberg-on-S3 setups often target 128-512 MB Parquet files for similar reasons.

What is the difference between replication factor and erasure coding?

Replication keeps N full copies of every block — simple, fast to read, expensive to store (3x for RF=3). Erasure coding splits a block into data + parity stripes using Reed-Solomon, so RS(6,3) keeps 1.5x total data instead of 3x. EC saves storage but adds CPU cost on reads and breaks data locality. EC is the right call for cold partitions; replication wins for hot data.

How does HA actually elect a new NameNode?

Both NameNodes share an edit log via a quorum of JournalNodes (usually 3 or 5). The active NN writes edits, the standby tails them. ZKFC processes monitor health via ZooKeeper. On failure, ZKFC fences the old active, promotes the standby, and clients automatically retry via the logical nameservice ID in core-site.xml. From the client perspective it is a brief stall, not a full outage.

What replaces HDFS in modern stacks?

The standard 2026 stack is object storage (S3, GCS, ADLS, or on-prem MinIO/Ceph) for bytes, plus a table format (Iceberg, Delta, Hudi) for schema and ACID semantics, plus a compute engine (Spark, Trino, DuckDB, Snowflake, BigQuery). The mental model of blocks + metadata + replication still applies — it just runs on someone else's hardware.