The Streaming Lakehouse: Delete Your Lambda Pipeline

Q: What is a streaming lakehouse and how is it different from a normal lakehouse?

A streaming lakehouse is a data architecture where a continuous stream-processing engine — most commonly Apache Flink — writes incrementally into an open table format such as Apache Iceberg, Apache Paimon or Delta Lake, and that single table serves both real-time consumers and batch or BI queries. A normal lakehouse is fundamentally batch: data lands in the table on a schedule, usually via periodic Spark or ELT jobs, and freshness is measured in hours. The streaming variant changes two things. First, the write path is continuous and stateful, so the table reflects new events within seconds to low minutes rather than at the next batch window. Second, and more importantly, it targets the elimination of the Lambda architecture — the long-standing pattern of running a fast approximate speed layer alongside a slow authoritative batch layer. In a streaming lakehouse the table itself is the source of truth for both freshness and history, so you maintain one pipeline and one definition of each metric instead of two that must be reconciled. The cost of this is real operational work: a long-running stateful job and a compaction plane that a batch lakehouse does not need.

Q: How does exactly-once actually work in a streaming lakehouse?

Exactly-once is enforced by a two-phase commit between the stream processor and the table catalog, coordinated with the processor checkpoint. In a Flink pipeline the job periodically takes a checkpoint of its operator state — say every 30 to 60 seconds — which captures both the in-flight computation and the source offsets consumed. The table sink writes its data files to object storage but keeps them staged and invisible; only when the checkpoint completes does the sink commit a new table snapshot that makes those files visible and records the offset in the same transaction. If the job crashes mid-write, the uncommitted files are orphaned and later cleaned up, and the job restarts from the last successful checkpoint, reprocessing exactly the events after that offset with no duplicates and no gaps. The critical failure mode is configuration: if you disable checkpointing or let the sink autocommit on every write, you fall back to at-least-once semantics, where a restart replays events that were already written and every count, sum and average is inflated. Because the inflation is small and continuous, it typically goes unnoticed until a downstream number is audited against the source system.

Q: Why do streaming lakehouses generate a small-file problem, and how do you fix it?

The small-file problem is the direct physical consequence of frequent commits, not a bug. Every time the stream sink commits a snapshot it writes at least one data file per partition touched in that commit. A pipeline committing every 30 seconds across a couple of hundred partitions can therefore produce hundreds of thousands of small files a day. The lake tolerates this on the write side, but query performance collapses because a planner must list and open every file before it can read data, so a query can spend several seconds in file enumeration before touching a single byte — often making the streaming table slower and more expensive than the nightly batch job it replaced. The fix is a maintenance plane that runs independently of the write path: a compaction service that bin-packs small files into targets of roughly 128MB to 512MB, applies accumulated deletion vectors, optionally sorts rows within files to match the common query predicate, and expires old snapshots past the time-travel window you actually promise. Compaction is not an afterthought to add after the first incident; it ships with the pipeline. Tracking file count per partition as an explicit SLO is the practical way to catch drift before it becomes a performance outage.

Q: Should I use Apache Iceberg, Apache Paimon or Delta Lake for a streaming lakehouse?

The three formats overlap heavily but were designed with different centres of gravity, and the streaming case sharpens the differences. Apache Iceberg is the broad interoperability winner: its v3 spec added deletion vectors and row lineage that make merge-on-read updates efficient, and its engine-neutral design plus the surrounding catalog ecosystem make it the safe default when many engines must read the same table. Apache Paimon was built LSM-first specifically for the streaming case; it treats a changelog as a first-class output, so a downstream job can consume inserts, updates and deletes from the lake the way it would from a Kafka topic, which is a genuine advantage when you are chaining streaming transformations. Delta Lake is strongest inside the Databricks and Spark ecosystem and has mature streaming reads and writes there. The honest decision rule is to choose by your read fanout and your engine commitments rather than by feature checklists: Iceberg when the table must be neutral across Trino, Spark, Flink and warehouses; Paimon when streaming updates and changelog consumption are the core workload; Delta when you are already committed to Spark and Databricks. The catalog you put in front of the format matters as much as the format itself.

Q: Do I even need streaming, or is micro-batch good enough?

For most tables micro-batch is good enough and is the correct default, because it delivers the large majority of the freshness value at a small fraction of the operational cost. Landing change-data-capture events into an Iceberg table on a 5 to 15 minute schedule gives you data that is fresh enough for nearly every dashboard, report and analytical workload, and it does so with a simple scheduled job that has no long-running state, no checkpoint tuning and a far lighter compaction burden. True continuous streaming is justified only when a specific consumer has a real sub-minute requirement and someone acts on the data in that window — fraud and abuse signals, live inventory or pricing, an operational dashboard that drives an immediate human decision. The engineering mistake is to make streaming the default because it sounds modern, which leaves you operating a 24/7 stateful job and paying its compaction bill to serve queries that a 15-minute micro-batch would have answered for a tenth of the cost. The discipline is to be able to name the reader who needs sub-minute freshness before you commit to a streaming pipeline for a given table; if you cannot name them, micro-batch.

Q: What is the Lambda architecture and why does the streaming lakehouse replace it?

The Lambda architecture is a data-processing pattern that runs two parallel pipelines over the same input: a speed layer that processes events in real time to produce a low-latency but approximate view, and a batch layer that periodically recomputes the same results from scratch over the full dataset to produce the authoritative, corrected view, with a serving layer merging the two. It was a rational answer to a decade in which no single system could deliver both freshness and correct, replayable history, and it worked. Its cost is structural: every metric is implemented twice, once in the streaming code and once in the batch code, in different frameworks with different semantics, so a change or a bug must be fixed in two places and the two implementations inevitably drift. The reconciliation job that papers over the drift becomes a chronic source of incidents that no team wants to own. The streaming lakehouse replaces Lambda by making one incremental pipeline write into a table that is simultaneously the fresh view and the historical system of record, so there is a single implementation of each metric and a single place to fix it. It does not replace Lambda for free — it substitutes the two-pipeline maintenance burden for the burden of operating one stateful job and its compaction plane.

Q: How is a streaming lakehouse different from Kafka plus a stream processor (Kappa)?

The Kappa architecture keeps the log — typically Kafka — as the single source of truth and reprocesses history by replaying from an offset, deliberately avoiding a batch layer. It solves the two-codebase problem of Lambda, but it creates a different one: a log is not a query surface. When an analyst needs to answer an ad-hoc historical question, Kappa forces a full replay from the log through a processing job, which can mean reprocessing months of events to answer one question, because the log stores a sequence of events rather than a queryable, indexed, columnar table. The streaming lakehouse keeps the streaming ingestion and processing that Kappa is good at, but commits the results into an open lake table that is directly queryable by Trino, Spark, DuckDB or a warehouse, with partitioning and columnar layout that make historical and ad-hoc queries cheap. In effect it adds a proper analytical serving surface to the streaming world. The log still has a role — it is the durable, ordered capture buffer in front of the processor — but it stops being the thing analysts query. The newest storage designs blur this line further by exposing the streaming buffer itself as a lake table, but the principle holds: keep the log for capture, keep the table for query.

Q: What are the operational costs I am signing up for with a streaming lakehouse?

You are trading the maintenance of two batch-and-speed pipelines for the operation of one continuously running stateful system plus a maintenance plane, and it is important to size that honestly before committing. The first cost is the stateful job itself: a Flink pipeline with large keyed state needs memory, checkpoint storage and tuning, and it must run and be monitored 24/7 rather than as a job that starts and finishes. The second cost is the compaction and maintenance plane — scheduled, monitored jobs for file compaction, deletion-vector application, snapshot expiry and orphan-file cleanup — which consumes compute continuously and which, if neglected, silently degrades query performance until an incident forces attention. The third is the catalog and concurrency surface: a streaming writer and batch readers hitting the same table need a catalog that handles concurrent commits safely, or they corrupt snapshots. The fourth is observability: you now need to track lag, checkpoint duration, file counts per partition and commit failures as first-class SLOs. The benefit — one metric definition, one place to fix a bug, fresh data on a queryable table — is real and often worth it for the tables that need it, but it is not free, and applying it to tables that did not need streaming makes your platform more expensive and more fragile, not less.

Satyam Kumar

Retour au blog

ai-architecture

The Streaming Lakehouse: One Pipeline Instead of Two

By Satyam KumarAugust 19, 202611 min read

streaming lakehouse apache flink apache iceberg apache paimon lambda architecture kappa architecture incremental processing exactly-once change data capture compaction open table formats data engineering real-time analytics confluent tableflow delta lake 2026

The Streaming Lakehouse: One Pipeline Instead of Two

Frequently Asked Questions

Partager l'article

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

Architecte AI & Cloud. J'aide les équipes à construire des systèmes qui scalent pour des millions.

LinkedIn GitHub

The Streaming Lakehouse: One Pipeline Instead of Two

Frequently Asked Questions

Partager l'article

Comments

Leave a comment