The Evolution of Data Lakes: Moving Towards Lakehouse Architectures

The data lake promise, and where it broke down

The original pitch of the data lake was simple: store everything, in its raw form, cheaply, and decide what to do with it later. That worked well for storage cost. It worked less well in practice, because "store everything and decide later" tends to become "store everything and never decide" — a few years in, most data lakes accumulated inconsistent schemas, duplicate files, and no reliable way to know which version of a table was current. The industry nickname for this stage is the "data swamp."

What a lakehouse actually adds

A lakehouse isn't a different storage location — it's the same cloud object storage (S3, ADLS, GCS), with a transactional table format layered on top: Delta Lake, Apache Iceberg, or Apache Hudi are the three that matter today. That table format is what turns a folder of files back into something that behaves like a database table:

ACID transactions: a write either fully succeeds or fully fails — no more half-written files from a crashed job leaving a table in a broken state.

Schema enforcement: a column type change or a dropped field is caught at write time, not discovered three dashboards downstream.

Time travel: every table version is queryable, which makes both debugging and point-in-time reporting straightforward.

A concrete example: time travel with Delta Lake


# Query the table exactly as it looked yesterday at 9am
df = spark.read.format("delta") \
    .option("timestampAsOf", "2026-09-19 09:00:00") \
    .load("s3://lakehouse/curated/orders/")

This single option replaces what used to require a separate snapshot-export process, or worse, no ability to answer "what did this table say before the bad job ran."

When a plain lake is still the right call

Not every dataset needs a lakehouse. A landing zone that gets read exactly once by an ingestion job, and never queried directly by analysts, doesn't benefit much from transactional guarantees — plain partitioned files are simpler and just as effective there. The lakehouse pattern earns its complexity for tables that multiple teams query concurrently, that get updated (not just appended to), or where "what did this look like last week" is a question someone will eventually ask.

Data Architecture