Data Lakehouse Architecture: Iceberg, Delta & Medallion

Q: What is a data lakehouse and how does it differ from a lake and a warehouse?

A data lakehouse is a data platform that puts a transactional table layer — an open format such as Apache Iceberg or Delta Lake — directly on top of cheap object storage, giving you database-grade guarantees (ACID transactions, safe schema evolution, time travel) on storage that costs a fraction of a warehouse, so a single copy of your data can serve both BI/SQL analytics and ML/AI workloads. A raw data lake stores files (Parquet, JSON) on object storage with no transactions and ad-hoc schema, which is cheap and ML-friendly but painful and unreliable for analytics, and it tends to degrade into a swamp. A data warehouse offers ACID, managed schema, and fast SQL but in a proprietary, tightly coupled system that is expensive and awkward for ML, and it locks you in. The lakehouse takes the warehouse's table semantics and puts them on the lake's cheap open storage, removing the long-standing split that forced teams to keep a lake for ML and a warehouse for BI with brittle pipelines copying between them. The trade is that you now own maintenance — compaction, metadata, governance — that the warehouse hid from you.

Q: What is the medallion architecture (bronze, silver, gold)?

The medallion architecture is the standard way to organise data inside a lakehouse as three progressively refined layers, each an open table derived from the one before. Bronze is the raw landing zone: data arrives from sources (apps, events, CDC, files, APIs) append-only and immutable, preserved exactly as received so it is a replayable source of truth. Silver is where quality lives: bronze data is cleaned, deduplicated, type-cast, conformed to consistent schemas, and validated against contracts, typically using a transformation framework. Gold is purpose-built: silver data is aggregated into business-ready tables — BI metrics and marts, and ML feature tables — shaped for specific consumers. The discipline that keeps this from decaying into a swamp is that bronze is never edited (you fix problems in silver, not by mutating raw data), silver is the single place cleaning happens (so consumers do not each re-implement it and disagree), and gold is rebuilt for purpose. Because every layer is a derived open table, you can drop and rebuild silver or gold from bronze at any time, which gives the platform the same replay and recovery property as an event-sourced log and makes the pipeline debuggable and trustworthy.

Q: Why do open table formats like Iceberg and Delta matter?

Open table formats are what turn a pile of files on object storage into a real, reliable table, and they are the defining component of a lakehouse. Apache Iceberg, Delta Lake, and Apache Hudi all add a metadata layer that tracks which data files belong to which version of a table, and from that they provide ACID transactions (so concurrent writes do not corrupt the table), snapshot isolation, safe schema evolution (you can add, rename, or reorder columns without breaking readers), and time travel (query the table as of a past snapshot). The metadata also carries column statistics and partition information, so a query engine can prune to just the files it needs and skip the rest, which is what makes scans fast at scale. Crucially these formats are open and decoupled from compute, so many engines — Spark, Trino, DuckDB, Flink, and various warehouse engines — can all read and write the same tables, which avoids the lock-in of a proprietary warehouse and lets you choose the best engine per workload. Without a table format you have a lake: no ACID, no safe schema changes, and readers that break whenever the data shape shifts, which is precisely the swamp the lakehouse exists to prevent.

Q: What is the small-files problem and how do I fix it?

The small-files problem is the most common cause of a slow lakehouse. Streaming ingestion and frequent writes each produce small Parquet files, and over time a table accumulates millions of tiny files; query latency then collapses because the engine spends most of its time opening files and reading metadata rather than scanning data, and object-storage request costs climb. The fix is compaction: a scheduled maintenance job that rewrites many small files into fewer large ones, typically targeting file sizes in the 128 MB to 1 GB range, which dramatically reduces the file count the engine must open and restores fast scans. Compaction is not optional for any table with frequent or streaming writes — it is a routine operational requirement, like vacuuming. It pairs with two other maintenance tasks: expiring old snapshots and vacuuming the files they referenced, so that time-travel history does not grow storage and metadata without bound, and tuning partitioning so you prune effectively without partitioning so finely that you recreate the small-file problem. Teams that schedule compaction, snapshot expiry, and sensible partitioning keep petabyte-scale tables answering in seconds; teams that ignore them watch the lakehouse silently degrade into a slow, expensive swamp despite having chosen the right format.

Q: How should I partition tables in a lakehouse?

Partition by the columns you actually filter on, coarsely enough to avoid creating too many small files. Partitioning physically groups data files by a column value so the engine can skip entire partitions that cannot match a query, which is one of the biggest levers for scan performance — but only if your queries filter on the partition column. The most common and usually correct choice is time (by day or month), because analytical and ML queries overwhelmingly filter by date ranges. The classic mistake is over-partitioning: partitioning by a high-cardinality column like user ID, or by too fine a time grain like minute, scatters data into millions of tiny files and reintroduces the small-files problem you were trying to avoid, hurting performance instead of helping. Newer table-format features ease this: Iceberg supports hidden partitioning (you query on a column and the format manages the partition transform) and clustering/sort-ordering as an alternative to rigid partition columns, which lets you get pruning benefits without committing to a partition scheme that is hard to change later. The rule of thumb is to partition on your dominant filter dimension, keep partitions large enough that files stay in the hundreds-of-megabytes range after compaction, and lean on column statistics and clustering for finer-grained pruning rather than ever-finer partitions.

Q: How does governance work when BI and AI share one copy of the data?

Governance in a lakehouse must live at the table and catalog layer, not in each consuming tool, precisely because the lakehouse's biggest win — one shared copy that BI users, data scientists, and AI pipelines all read — is also its governance challenge. The control point is a catalog (such as an Iceberg REST catalog, Unity Catalog, AWS Glue, or Polaris) that maps table names to their current snapshots and enforces access policy: who can read which tables, columns, and rows, with lineage tracking and audit. Enforcing access centrally means every consumer inherits the same controls and you get a single auditable record, which is far stronger than configuring permissions separately in Spark, in the BI tool, and in the ML pipeline where they inevitably drift. This matters acutely for regulated data — under EU GDPR and DORA, Singapore PDPA and MAS guidance, or UAE rules — where column-level controls, masking of personal data, and demonstrable audit over one governed copy beat data scattered and re-permissioned across a lake and a warehouse. Because the open table format decouples storage from compute, governance applied at the table layer holds regardless of which engine reads the data, so a policy set once is enforced for Spark, Trino, DuckDB, and any AI training job alike. The anti-pattern is enforcing governance per consumer, which is inconsistent, unauditable, and eventually leaks.

Q: When should a team adopt a lakehouse instead of a warehouse or lake?

Adopt a lakehouse when you need both reliable SQL analytics and ML/AI on the same data and do not want to maintain two systems with pipelines copying between them — which is most data-driven organisations in 2026. If your needs are purely BI on modest, well-structured data and you value zero operational overhead over cost and flexibility, a managed warehouse can still be the simpler choice. If you only stage raw files for occasional batch ML and never need transactions or fast SQL, a plain lake may suffice, though it tends to decay. The lakehouse is the right default when AI training and RAG pipelines need the same governed data the dashboards read, when data volume makes warehouse storage costs painful, when you want to avoid proprietary lock-in by keeping data in an open format, or when you need ACID and safe schema evolution on lake-scale data. The trade-off to accept consciously is that a lakehouse shifts maintenance responsibility to you: compaction, snapshot expiry, partitioning, and catalog governance are now your jobs, and an unmaintained lakehouse can be slower and cost more than the warehouse it replaced. So adopt it when the one-copy-for-BI-and-AI benefit and the open-format flexibility outweigh that maintenance burden, and commit to the maintenance from the start rather than discovering it after petabytes have landed.

Q: What is a sensible maturity path for building a lakehouse?

Stage 0 is a raw data lake: Parquet and JSON files on object storage with no transactions and chaotic schema — useful cheap storage but painful, unreliable analytics. Stage 1 adds an open table format (Iceberg or Delta) over the lake, which brings ACID transactions and safe schema evolution and makes the data genuinely queryable and reliable. Stage 2 introduces the medallion pattern — immutable bronze, conformed silver, purpose-built gold — so one governed copy serves both BI and ML and any layer can be rebuilt from bronze, which is the point at which the platform stops being a swamp. Stage 3 treats maintenance as first-class: scheduled compaction, snapshot expiry, and partition or clustering tuning keep queries fast at scale, and a catalog enforces access control and lineage, so the lakehouse stays performant and governed as data grows. Stage 4 runs multiple engines per workload over the same tables, enforces column-level governance and audit for compliance, models and controls cost per layer, and treats the lakehouse as the single source feeding dashboards, model training, and RAG alike. Most teams stall between a Stage-0 swamp and a Stage-1 "we added Delta" and then wonder why queries are slow and storage costs climb; the platforms that successfully serve both analytics and AI from one copy are at Stage 2 and above, where layering, maintenance, and governance are designed in from the start rather than bolted on after problems appear.