Data & Databases
Big Data Architecture: Lakes, Lakehouses, Formats and Processing
Intermediate9 min readUpdated
Big data architecture is a set of trade-offs about where data lands, how it is stored, who is allowed to change it and how fast answers must arrive. This guide examines the volume-velocity-variety framing critically, compares lake, warehouse and lakehouse designs, explains columnar formats and open table formats, and gives an honest account of when a single database is still the right answer.
What you will learn
- Judge whether a workload genuinely requires distributed data tooling
- Compare data lake, data warehouse and lakehouse architectures and their failure modes
- Choose between Parquet, ORC and Avro and partition data without creating small-file problems
- Explain what open table formats add over plain files in object storage
- Contrast lambda and kappa architectures and design ingestion for batch and streaming sources
Few areas of software have suffered more from vendor vocabulary than data platforms. "Big data" was coined to describe a genuine problem — datasets that no single machine could hold or process in acceptable time — and then applied to almost everything, with the result that many organisations run distributed clusters over data that would fit on a laptop.
This guide takes the architectural decisions seriously and the marketing framing sceptically. It covers how data arrives, where it is stored, in what format, who may change it, and how it is processed — plus the section most such guides omit, which is how to tell that you do not need any of this. The engine-level detail of HDFS, YARN and Spark lives in the Hadoop ecosystem guide; here the concern is the shape of the platform.
The three Vs, examined
Volume, velocity and variety are a useful checklist and a poor decision procedure. Each one deserves a harder question.
Volume. The relevant number is not how much data exists but how much a query touches. A hundred-terabyte event archive queried one day at a time is a hundred-gigabyte problem with good partitioning. Compression and columnar layout routinely cut scanned bytes by an order of magnitude before any distributed engine is involved.
Velocity. Ask what the decision latency is, not the arrival latency. Events arriving continuously do not require streaming computation if nobody acts on the result before tomorrow morning. Real streaming requirements — fraud scoring, dynamic pricing, operational alerting — are ones where a delayed answer is worth less than a fast approximate one.
Variety. This is the V that has held up best. Relational stores struggle when the schema differs per source and changes without notice, and that genuinely motivates a lake with schema-on-read.
A fourth V, veracity, is often added and matters more than the first three. A pipeline that delivers a hundred terabytes of subtly wrong numbers is worse than one that delivers nothing, because people act on it.
Lake, warehouse, lakehouse
A data warehouse stores curated, modelled, typed data optimised for analytical SQL. Data is transformed on the way in, the schema is enforced, and query performance is excellent because the layout was designed for the questions. The cost is rigidity: a new source means modelling work before anyone can use it, and data the model rejected is simply lost. Dimensional modelling for warehouses is covered in the business intelligence guide.
A data lake stores files in object storage — S3, Azure Blob, Google Cloud Storage — in whatever form they arrived, with schema applied at read time. It is cheap, accepts anything, and preserves raw data for questions nobody has asked yet. Its failure mode has a name: the data swamp, a bucket of undocumented files with no owner, no schema history and no way to know which copy is current. Lakes fail for governance reasons, not technical ones.
A lakehouse keeps files in open formats in object storage but adds a metadata layer that provides ACID transactions, schema evolution, time travel and efficient file pruning. That layer is an open table format: Apache Iceberg, Delta Lake or Apache Hudi. The result behaves like a table — you can UPDATE, DELETE, MERGE and roll back — while remaining readable by any engine that speaks the format.
A common and sensible layering, whatever you call it:
raw / bronze immutable landing zone, source format, partitioned by ingest date
curated / silver cleaned, deduplicated, typed, conformed to a common schema
serving / gold aggregated, modelled tables for dashboards, APIs and data science
Each layer is reproducible from the one before it, which is what makes reprocessing after a bug possible.
Storage formats and layout
Format choice has a larger effect on cost and speed than the processing engine does.
| Format | Layout | Best for |
|---|---|---|
| Apache Parquet | Columnar, per-column compression and encoding, row groups with min/max statistics | Analytical reads; the default for lake and lakehouse tables |
| Apache ORC | Columnar, stripes with lightweight indexes and bloom filters | Analytical reads, historically strongest in the Hive ecosystem |
| Apache Avro | Row-oriented, schema embedded, compact binary | Event streams and ingestion, where whole records are written and read |
| JSON / CSV | Text, row-oriented, no types | Interchange and landing only; never a query format |
Columnar formats win analytics for three compounding reasons. A query reading three of eighty columns reads only those three. Values of one column are similar, so compression is dramatically better than row-wise. And per-row-group statistics let the reader skip whole blocks whose min/max range cannot satisfy the predicate. Together these turn a full scan into a fraction of the I/O.
Partitioning is directory-level pruning: writing dt=2026-09-07/region=eu lets a query filtering on those columns skip everything else. Two rules keep it useful. Partition on columns that appear in almost every filter, usually a date. And keep cardinality low — partitioning by customer id produces millions of directories and a metadata problem far worse than the scan you avoided.
File size is the other half. Aim for files of a few hundred megabytes. Streaming writers naturally produce many tiny files, so schedule compaction; every table format provides a maintenance operation for it. Within a file, sorting on a commonly filtered column makes row-group statistics far more selective — the same reasoning as clustering in a relational engine, discussed in the index and execution plan guide.
Open table formats add what plain directories cannot: atomic commits, so a reader never sees a half-written update; snapshot isolation and time travel, so you can query the table as of yesterday or roll back a bad load; hidden partitioning, so queries need not mention the partition column; and schema evolution by column id rather than position, so renaming a column does not corrupt old files.
Ingestion
Sources fall into a few recurring shapes.
Batch extracts — nightly dumps, vendor files, API pulls. Simple, well understood, and the right answer far more often than people admit. Prefer incremental extraction on a watermark column over re-reading everything.
Change data capture reads a database's transaction log and emits a stream of inserts, updates and deletes. It is the least intrusive way to mirror an operational database into the platform: no queries against the source, no batch window, and the ordering the log provides. It is also where most correctness problems appear, particularly around schema changes and deletes.
Event streams. A log-structured broker gives you a durable, replayable, ordered buffer between producers and consumers, and decouples their availability and speed. That replay property is what makes reprocessing possible at all — see Kafka architecture for the mechanics of partitions, offsets and retention.
Whatever the source, land the raw payload unchanged before transforming it. Storage is cheap; an unrecoverable transformation bug is not.
The ETL versus ELT distinction belongs here too. Traditional ETL transforms before loading, so the warehouse only ever contains modelled data. ELT loads raw and transforms with the target engine's own compute, which is cheaper, keeps the raw copy, and turns transformations into version-controlled SQL. ELT is the modern default wherever the storage layer is cheap and the engine is elastic.
Processing
Batch processing re-computes over a bounded dataset. Apache Spark dominates here, with Trino and Presto covering interactive federated SQL and cloud warehouse engines covering the rest. Batch is simpler to reason about, easy to re-run, and correct by construction because the input does not move under you.
Stream processing computes continuously over an unbounded input, and introduces three concepts batch never needs:
- Event time versus processing time. Events arrive late and out of order. Windowed results must be based on when something happened, not when it was received.
- Watermarks. A watermark is the engine's assertion that no event older than a given timestamp is still expected, which is what allows a window to close and emit.
- State and exactly-once. Aggregations hold state that must survive failures, and sinks must be idempotent or transactional for results not to be duplicated on retry.
Apache Flink is the reference implementation for low-latency, stateful streaming; Spark Structured Streaming offers the same programming model as its batch API with micro-batch or continuous execution, which is often decisive for teams already using Spark. Kafka Streams suits transformations that live inside a service rather than a cluster.
Lambda architecture ran a batch layer for correctness alongside a speed layer for freshness, and merged them at query time. It works, and its cost is duplicated business logic in two codebases with two sets of bugs. Kappa architecture keeps one streaming path and reprocesses history by replaying the retained log, which is feasible precisely because modern brokers and table formats support replay and idempotent writes. Start with kappa if you need streaming at all; start with plain batch if you do not.
Governance, quality and cost
The platform work that determines whether anyone trusts the data is unglamorous and mandatory.
- A catalogue — Apache Hive Metastore, Iceberg REST catalogs, or a cloud equivalent — records what tables exist, their schemas and their locations. Without it, discovery happens by asking colleagues.
- Lineage records which job produced which table from which inputs. When a number looks wrong, lineage is the difference between an hour and a week.
- Data contracts make the schema a producer publishes an explicit, versioned agreement with tests, so upstream changes break a build rather than a dashboard.
- Quality checks run as part of the pipeline: row counts within expected bounds, null rates, referential checks, freshness thresholds. Fail the job loudly rather than publishing bad data quietly.
- Privacy and retention. Personal data in a lake is still personal data. Know where it is, encrypt it, restrict it, and be able to delete it — a requirement that plain files cannot satisfy and table formats can.
- Cost. In cloud object storage the bill is driven by bytes scanned and requests made, so partition pruning, file sizing and column projection are cost controls as much as performance ones. Set lifecycle rules that move cold partitions to cheaper tiers.
When you do not need any of this
Be willing to reach the conclusion that a distributed platform is the wrong answer.
- A single PostgreSQL instance handles hundreds of gigabytes of analytical data with partitioning and good indexing, as the SQL best practices guide describes.
- A single-node columnar engine reads Parquet files directly and outperforms a small Spark cluster on datasets up to hundreds of gigabytes, without a cluster to operate.
- Log and observability data belongs in a purpose-built platform, not a general lake — see centralised log management.
- If the requirement is "the business wants dashboards", the answer may be a warehouse and a BI tool, with no lake at all.
The strongest reasons to build a real data platform are that data comes from many systems with incompatible models, that consumers are numerous and varied, or that volumes genuinely exceed one machine. Absent those, the complexity buys nothing and costs a team.
Key takeaways
- Judge volume by bytes scanned per query, velocity by decision latency, and add veracity to the list — wrong data at scale is worse than no data.
- Warehouses give schema and speed over curated data, lakes give cheap flexible storage, and lakehouses use open table formats to combine transactions and schema control with object-storage economics.
- Columnar formats with partition pruning, sensible file sizes and sorted row groups deliver more performance than any change of processing engine.
- Land raw data unchanged, layer transformations so every stage is reproducible, and prefer ELT with version-controlled transformations.
- Choose streaming only when a decision depends on it, and prefer a single replayable path over maintaining batch and streaming copies of the same logic.
- Catalogue, lineage, data contracts, quality checks and retention rules are what make a platform trustworthy; without them a lake becomes a swamp.
Sources and further reading
Frequently asked questions
- How much data is 'big data'?
- There is no threshold, and the honest answer is behavioural rather than numeric: your data is big when a single well-tuned database server can no longer meet your latency, throughput or cost targets. In 2026 a commodity machine holds terabytes of RAM and tens of terabytes of fast storage, so that point arrives much later than most architecture diagrams assume.
- Data lake or data warehouse?
- A warehouse gives you schema, constraints, transactions and fast SQL over curated data. A lake gives you cheap storage for anything, including raw and semi-structured data you have not modelled yet. Most organisations end up with both, and the lakehouse pattern is the attempt to keep the lake's cost and openness while adding the warehouse's transactions and schema enforcement.
- Why do small files matter so much?
- Every file carries metadata, a listing cost and, on object storage, a separate request with its own latency. A job reading a million ten-kilobyte files spends nearly all its time on listings and opens rather than on data. Aim for files in the hundreds of megabytes, and run compaction as a scheduled maintenance job on any table written by a streaming pipeline.
- Is the lambda architecture still relevant?
- Its problem statement is, its solution mostly is not. Maintaining the same business logic twice, once for batch and once for streaming, was always the weak point. Modern stream processors provide event-time windows, exactly-once sinks and reprocessing from a retained log, so a single streaming path with replay — the kappa idea — covers most cases that lambda was invented for.