Data & Databases

The Hadoop Ecosystem: HDFS, YARN, MapReduce, Hive, HBase and Spark

Intermediate9 min readUpdated

Hadoop introduced the idea that shaped a decade of data engineering: move the computation to the data and tolerate failure by design. This guide explains HDFS blocks and the NameNode, YARN scheduling, MapReduce and the reasons Spark displaced it, the roles of Hive and HBase, the Spark DataFrame model, and how much of the stack still makes sense today.

What you will learn

  • Explain how HDFS splits, replicates and locates blocks and what the NameNode does
  • Describe how YARN allocates containers and how applications request resources
  • Trace a MapReduce job through map, shuffle and reduce, and explain Spark's advantages
  • Choose between Hive, HBase and Spark for a given access pattern
  • Assess which parts of the ecosystem are still worth adopting and what replaced the rest

Hadoop's contribution was not a piece of software but a stance: assume the machines are cheap and will fail, store the data across them, and send the computation to wherever the data already sits. That inverted the assumption of the previous generation, where data was pulled to a big expensive machine, and it made analysis over petabytes possible on hardware that ordinary companies could afford.

Two decades later most of the original implementation has been superseded, and yet the vocabulary — blocks, shuffles, partitions, jobs, executors — remains the vocabulary of data engineering. This guide covers what each component actually does, why Spark displaced MapReduce, and which parts still deserve a place in a 2026 platform. The surrounding architectural choices — lakes, table formats, streaming — are the subject of the big data architecture guide.

HDFS: a filesystem built for sequential reads

The Hadoop Distributed File System splits every file into fixed-size blocks, 128 MB by default, and stores each block on several machines — three by default, on at least two racks. That single design choice explains most of its behaviour.

Two kinds of process run the system:

  • The NameNode holds the entire directory tree and the block-to-DataNode mapping in memory, and records every change to an edit log. It never handles file data itself. Clients ask it where a block lives and then talk to the DataNodes directly.
  • DataNodes store blocks as ordinary files on local disks and report in with heartbeats and block reports. When a node stops reporting, the NameNode notices that its blocks are under-replicated and instructs other nodes to copy them.

The NameNode's memory is the system's real limit: every file, directory and block needs an in-memory record of roughly 150 bytes. A million small files therefore cost far more than one large file of the same total size — the small files problem that every Hadoop operator has met.

Three properties follow from the design and are worth stating plainly. HDFS is write-once, append-only: you cannot update in place, so pipelines rewrite partitions instead. It optimises throughput over latency, so it is a terrible database and an excellent scan target. And it is rack-aware, placing replicas so that losing an entire rack does not lose data, and so that readers can prefer a local copy.

Hadoop 3 added two significant changes. NameNode high availability runs an active and a standby NameNode sharing edits through a quorum of JournalNodes, with ZooKeeper handling failover — before that, the NameNode was a genuine single point of failure. And erasure coding stores parity instead of full replicas, cutting storage overhead from 200% to around 50% for cold data, at the cost of more CPU and slower recovery.

hdfs dfs -put sales-2026-09.parquet /data/raw/sales/dt=2026-09-07/
hdfs dfs -ls -h /data/raw/sales/dt=2026-09-07/
hdfs fsck /data/raw/sales -files -blocks -locations
hdfs dfsadmin -report

YARN: scheduling the cluster

Yet Another Resource Negotiator separates cluster resource management from the application logic running on it, which is what allowed anything other than MapReduce to run on a Hadoop cluster.

  • The ResourceManager is the cluster-wide scheduler. It knows total capacity and hands out containers, each a reservation of memory and CPU on a specific node.
  • A NodeManager runs on each machine, launches containers and reports their health and usage.
  • Every submitted job first gets an ApplicationMaster container of its own. The ApplicationMaster then negotiates further containers for the actual work, monitors them, and restarts what fails. Job-level failure handling therefore lives in the application, not in the scheduler.

Two schedulers matter in practice. The Capacity Scheduler divides the cluster into hierarchical queues with guaranteed shares and optional elasticity, which fits organisations where departments have contracted capacity. The Fair Scheduler aims to give every running application an equal share over time, which fits mixed ad-hoc workloads. Either way, the recurring operational lesson is the same: a single badly sized job requesting enormous containers can starve everything else, so queue limits and per-user caps are not optional.

MapReduce, and why Spark replaced it

MapReduce expresses computation as two functions and a sort in between.

  1. Map runs on each input split — usually one HDFS block, scheduled on a node holding it — and emits key-value pairs.
  2. Shuffle and sort moves every pair to the reducer responsible for its key, sorting by key on the way. This is the expensive part: it crosses the network and spills to disk.
  3. Reduce receives each key with all its values and emits the result.

Word count is the canonical illustration: map emits (word, 1) per token, reduce sums the values per word. The model is genuinely elegant, and the same map-then-reduce shape appears in ordinary application code — the Java Streams API is the single-machine version of the same idea.

Its weaknesses are structural, not cosmetic:

  • Every stage materialises to disk. A pipeline of five transformations writes and re-reads the full dataset five times, with replication on each write.
  • Iterative algorithms are pathological. Machine learning and graph algorithms loop over the same data dozens of times, paying the full I/O cost each pass.
  • Only two operators. Joins, filters, aggregations and sorts must be hand-built out of map and reduce, so real jobs became long chains of Java classes.
  • Batch only. No interactive querying, no streaming.

Apache Spark addressed all four with one change of model: keep the dataset in memory across stages and describe the whole computation as a directed acyclic graph of transformations before executing anything. Spark can then fuse narrow transformations into a single pass, insert a shuffle only where the data genuinely must be redistributed, and reuse cached data across iterations. Reported speedups of ten to a hundred times over MapReduce came mostly from removing intermediate writes, not from clever arithmetic.

Spark's programming model

Spark's original abstraction was the RDD, a resilient distributed dataset: an immutable, partitioned collection with a recorded lineage of the operations that produced it. Lineage is how it recovers — a lost partition is recomputed from its parents rather than restored from a replica. Transformations (map, filter, join) are lazy; nothing runs until an action (count, collect, write) forces evaluation.

Modern Spark code should use DataFrames and Datasets instead. They carry a schema, which lets the Catalyst optimizer rewrite the query — pushing filters down to the scan, pruning unread columns, choosing join strategies — and lets the Tungsten engine store rows in compact off-heap memory and generate specialised bytecode. The practical consequence is that a DataFrame pipeline usually beats hand-written RDD code, in the same way that declarative SQL beats a hand-written loop.

val orders = spark.read.parquet("s3a://lake/curated/orders")
val customers = spark.read.parquet("s3a://lake/curated/customers")

val topRegions = orders
  .filter($"placed_at" >= "2026-01-01" && $"status" === "SHIPPED")
  .join(customers.select($"id", $"region"), $"customer_id" === $"id")
  .groupBy($"region")
  .agg(sum($"total").as("revenue"), countDistinct($"customer_id").as("buyers"))
  .orderBy($"revenue".desc)

topRegions.explain(true)          // read the physical plan before trusting the job
topRegions.write.mode("overwrite").parquet("s3a://lake/serving/revenue_by_region")

Spark SQL runs the same engine over SQL text, so analysts and engineers share one execution path. Structured Streaming reuses the DataFrame API for unbounded input, with event-time windows and checkpointed state. Spark 4 continued the direction of travel: ANSI SQL semantics are now the default, so silent overflow and implicit truncation become errors; Spark Connect decouples the client from the driver, letting thin clients submit work to a remote cluster; and a native VARIANT type handles semi-structured data without exploding the schema.

Three performance concepts explain most Spark tuning:

  • Partitions are the unit of parallelism. Too few and cores sit idle; too many and scheduling overhead dominates. Adaptive query execution now coalesces them automatically at shuffle boundaries.
  • Shuffles are the expensive operation. Wide transformations (groupBy, join, distinct) cause them; narrow ones do not. A broadcast join avoids the shuffle entirely when one side is small enough to fit in each executor's memory.
  • Skew — one key holding a disproportionate share of rows — leaves a single task running long after the rest finish. Adaptive execution can split skewed partitions; salting the key is the manual remedy.

Hive and HBase

Apache Hive put SQL on top of files. A query in HiveQL is compiled into a job — historically MapReduce, later Tez, and often Spark — over data in HDFS or object storage. Its enduring contribution is not the query engine but the Metastore: a relational database recording which tables exist, their columns and types, their partitions and the location of their files. That catalogue outlived Hive itself; Spark, Trino, Flink and Impala all read it, and it remains a common interoperability point even in clusters with no Hadoop at all.

Hive's characteristics are those of a scan engine: high throughput, minute-scale latency for large jobs, no indexes worth the name, and ACID support that was retrofitted and is best avoided in favour of a modern table format. LLAP added long-lived daemons and caching to bring interactive queries within reach, though most teams now point Trino or Spark SQL at the same metastore instead.

Apache HBase is the opposite tool: a wide-column store modelled on Bigtable, running over HDFS, providing millisecond random reads and writes by row key. Data is organised into column families, split into regions by row-key range, and served by RegionServers, with writes going to a write-ahead log and an in-memory store that flushes to immutable files, compacted in the background.

Everything about using HBase well reduces to row-key design. Keys are stored in sorted order, so a monotonically increasing key (a timestamp, a sequence) sends every write to one region — the hotspotting problem, addressed by salting or hashing a prefix. Conversely, a key that begins with the value you scan by makes range scans efficient. There are no secondary indexes, no joins and no query planner: you design the key for the access pattern, or you scan everything. For a comparison with the document approach to the same trade-off, see the MongoDB guide.

Around these sit the rest of the familiar names: Sqoop for bulk transfer between relational databases and HDFS, largely displaced by change-data-capture into a broker as described in Kafka architecture; Oozie for workflow orchestration, displaced by Airflow and similar schedulers; Impala and Presto/Trino for interactive MPP SQL; ZooKeeper for coordination underneath several of the above; and Apache Ozone as a newer object store designed to fix HDFS's small-file and scale limits.

Where the ecosystem stands in 2026

The honest summary is that the ideas won and most of the implementation was replaced.

  • HDFS is in maintenance mode for most organisations. Cloud object storage offers effectively unlimited capacity, separates storage from compute so each scales independently, and costs less. Spark and Trino read it directly through the S3A connector.
  • MapReduce is legacy. Nothing new should be written against it; existing jobs are migrated to Spark.
  • YARN persists on-premises but new deployments overwhelmingly schedule Spark on Kubernetes, which unifies the operational model with the rest of the platform — containers, images and orchestration as covered in Docker fundamentals.
  • Hive the engine has largely given way to Trino, Spark SQL and cloud warehouses; the Metastore survives, though Iceberg REST catalogs are steadily replacing it.
  • HBase remains in place where it was deployed, with Cassandra, ScyllaDB and managed cloud key-value stores taking new workloads.
  • Spark is thriving, and it is the piece of the ecosystem most worth learning today.

If you are inheriting a Hadoop cluster, the pragmatic path is usually: move storage to object storage with an open table format, move MapReduce jobs to Spark, keep the metastore as a catalogue until an Iceberg catalog replaces it, and re-target the scheduler from YARN to Kubernetes. If you are starting fresh, you almost certainly need Spark and a table format, and almost certainly not HDFS.

Key takeaways

  • HDFS stores large, replicated, immutable blocks for sequential throughput; the NameNode's in-memory metadata is why small files hurt so much.
  • YARN separates resource management from application logic through the ResourceManager, NodeManagers and a per-job ApplicationMaster requesting containers.
  • MapReduce's map-shuffle-reduce model is sound but materialises every stage to disk, which is what made Spark's in-memory DAG execution ten times faster or more.
  • Prefer Spark DataFrames over RDDs so Catalyst and Tungsten can optimise; tune by watching partitions, shuffles and skew.
  • Hive is a scan-oriented SQL layer whose Metastore outlived it; HBase is a random-access wide-column store where row-key design determines everything.
  • In 2026, keep Spark and the catalogue idea, migrate storage to object storage with an open table format, and schedule on Kubernetes rather than YARN.

Sources and further reading

Frequently asked questions

Is Hadoop dead?
MapReduce as a programming model effectively is, and few new projects deploy HDFS. But the ecosystem's ideas and several of its components live on: the Hive Metastore still catalogues tables for engines that never touch Hadoop, YARN still schedules on-premises clusters, and Spark — which grew up inside the ecosystem — is more widely used than ever. Treat it as infrastructure to understand and inherit rather than to build new.
Why is the HDFS block size 128 MB?
Because a large block amortises the cost of seeking. With a small block, a job spends most of its time positioning disk heads and asking the NameNode where things are; with a large one, it streams. Large blocks also keep NameNode memory manageable, since every block needs an in-memory record. The trade-off is that many small files waste both block metadata and scheduling overhead.
What is the difference between Hive and HBase?
They solve opposite problems. Hive is a SQL layer over bulk files for full-scan analytical queries measured in seconds to minutes, with no notion of a point update. HBase is a wide-column store for millisecond random reads and writes by row key, with no query planner and no joins. Using either for the other's workload disappoints.
Should I run Spark on YARN or Kubernetes?
Kubernetes for anything new. It gives you the same container isolation your other workloads already use, one operational model for the whole platform, and elastic scaling against cloud instances. YARN remains a reasonable choice where an on-premises Hadoop cluster already exists and its capacity scheduler encodes real multi-tenant policies you would otherwise have to rebuild.