Zero-ETL: Read the Consistency Guarantees First

Q: What is zero-ETL, and is it really "no ETL"?

Zero-ETL is a set of managed integrations, most prominently AWS Aurora and RDS into Amazon Redshift or a SageMaker lakehouse, that replicate transactional data from a source database into an analytics target without you building or operating a pipeline. AWS describes it as integrations that "minimize the need to build ETL data pipelines" and "facilitate point-to-point data movement." The honest reading is that it removes the pipeline you would otherwise run, not the concept of moving and landing data. Under the hood it is log-based change data capture: for Aurora MySQL it relies on binary logging, and for Aurora PostgreSQL it reads the logical replication stream, which is the same mechanism a hand-built Debezium and Kafka Connect stack uses. So "no ETL" is accurate about operational toil and misleading about semantics. There is no transform stage because the movement is point-to-point, source schema to identical target schema; if you need reshaping, enrichment or redaction, that work still has to happen somewhere, and doing it downstream of the target reintroduces a pipeline you own. Zero-ETL is best understood as managed CDC with an opinionated envelope, not as the elimination of data engineering.

Q: Does zero-ETL give strong consistency between the source and the warehouse?

No. Zero-ETL is asynchronous and eventually consistent, applied table by table, and AWS deliberately describes it as "near real-time" rather than "real-time" or "consistent." There is no read-your-writes guarantee across the boundary: a row committed to the source and then immediately queried in the target may not be there yet, because the change still has to flow through the log capture and apply path. In the vocabulary of consistency models, what you get is well short of the strong consistency people often assume when they see data appearing quickly. This matters most for any workflow that writes to the operational database and then reads the warehouse expecting currency, such as reconciliation checks, "did my update land" verifications, or cross-system joins that assume both sides are current. Those are building on a guarantee that was never offered. The correct mental model is the same one that applies to asynchronous read replicas: the target trails the source by a lag that is usually small but is neither bounded nor transactional, and grows under write bursts, large transactions and resynchronizations. Design for staleness, annotate dashboards with freshness, and never treat the warehouse as an authoritative, up-to-the-moment mirror of the primary.

Q: How fresh is zero-ETL data — what is the real latency?

In steady state the replication lag is typically seconds, which is what "near real-time" refers to, but two documented realities break the assumption that it is always seconds. First, the cold start is measured in tens of minutes, not seconds: AWS states that when you initially create an integration, or when a table is being resynchronized, data seeding from source to target "can take 20-25 minutes or more depending on the size of the source database," and that this can itself increase replica lag. A large source is simply not queryable in the target the moment you create the integration, so first analytics runs and cutovers must be planned around the seed. Second, a stopped source strands the tail: AWS notes that if you stop the source DB cluster, "the last few transactions might not be replicated to the target until you resume the cluster," so maintenance windows and cost-saving stop/start cycles create gaps that surface no error. Beyond those, ongoing lag is a variable, not a constant, and grows under bursty writes, very large transactions and any table resync. The operational takeaway is to treat replication lag as a first-class metric with alerting, and to remember that an "Active" integration status is a liveness signal about the pipeline, not a freshness guarantee about the data.

Q: What happens to zero-ETL when I run a schema migration (DDL) on the source?

Schema changes are the sharpest edge of zero-ETL because they can take an analytics table offline. AWS documents that "performing DDL commands (for example ALTER TABLE) on a source table can trigger a table resynchronization, making the table unavailable for querying while it is resynchronizing." In practice that means a routine change like adding a column to a large, busy table can reseed that table in the target, and for the duration of the reseed — which shares the 20-25-minutes-or-more characteristic of the initial seed on large tables — queries against it in the warehouse fail or see stale data. The consequence is architectural: your source-side schema-migration cadence is now coupled to warehouse availability, which is the opposite of the independence teams usually assume between an operational database and its analytics copy. The mitigation is process, not configuration. Treat source DDL on replicated tables as coordinated events with a maintenance window and an owner who is paged, rather than as local edits a developer makes independently, and lean harder on schema-change discipline and data contracts so that changes are reviewed, batched and scheduled. This is a case where a managed pipeline removes the plumbing toil but increases, rather than decreases, the need for disciplined schema governance.

Q: What are the main limitations of AWS zero-ETL integrations?

The documented limitations define the envelope you are buying into, and stepping outside it means either a workaround or a different tool. Topology: the source DB cluster must be in the same Region as the target, so cross-region analytics is not supported directly, and if an Aurora global database fails over to a secondary cluster the integration becomes inactive and must be deleted and recreated — coupling your disaster-recovery posture to your analytics feed. Data modelling: for Aurora PostgreSQL, primary keys are required on all replicated tables and any table without one is put into a failed state, and you must specify at least one data filter. Transactional shapes: PostgreSQL two-phase transactions are unsupported, MySQL XA transactions push the integration into a Syncing state, and MySQL foreign-key CASCADE, SET NULL and SET DEFAULT actions as well as non-InnoDB storage engines are rejected. Coverage: system tables, temporary tables and views are not replicated. Quotas: an account gets, by default, 100 integrations total, 50 per target and 5 per source cluster, which bounds fan-in and fan-out designs. None of these are defects; they are the specification. The mistake is discovering them after committing, rather than checking your workload against them first and pricing in the workarounds where they bite.

Q: When should I use zero-ETL versus hand-rolled CDC like Debezium?

Use zero-ETL when your requirement fits its envelope and you value not operating a pipeline: same-region source and target, no need for in-flight transformation, tolerance for asynchronous eventual consistency with seconds-scale lag, primary keys on your tables, and a schema-change cadence you can coordinate with warehouse availability. For operational analytics, dashboards over live data and feature freshness, that describes a large share of real workloads, and building a bespoke CDC stack instead would be wasted effort and a new on-call burden. Use hand-rolled CDC with Debezium and Kafka Connect when you need what zero-ETL structurally cannot give: transformation, enrichment, filtering or PII redaction while the data is in the stream rather than after it lands; explicit control over ordering, dead-letter handling and replay; cross-region topologies; multiple heterogeneous targets from one change stream; or integration with an event-driven architecture where the change events themselves are the product. The two are not rivals so much as different points on a control-versus-convenience curve: zero-ETL trades control for the elimination of operational toil, and hand-rolled CDC trades operational toil for control. Many organisations run both, using zero-ETL for the straightforward replication cases and reserving a maintained CDC platform for the flows that need shaping or fan-out.

Q: Is zero-ETL actually free, or where does the cost go?

The integration layer itself is inexpensive to effectively free, which is part of why zero-ETL feels like a pure win, but the cost does not disappear — it moves to the target. Zero-ETL lands full-fidelity replicated data into an analytics warehouse or lakehouse that is priced on storage and compute, so every table you replicate consumes storage in Redshift or the lakehouse and every query against it consumes compute. Replicating tables you do not actually analyse, or replicating at full history when you only need recent data, quietly grows the warehouse bill even though the pipeline was "free." There is also an indirect cost in the source: log-based capture depends on binary logging or logical decoding being enabled and retained, which has its own storage and I/O footprint on the primary. The right way to reason about zero-ETL economics is to ignore the headline that the integration is cheap and instead model the target-side storage and compute of the data you are about to replicate, apply data filters so you only move what you will use, and treat the warehouse cost as the real price of the convenience. Convenience that makes replication one click also makes over-replication one click, so the discipline of choosing what to replicate is where the savings actually live.

Q: Can zero-ETL replace my data warehouse ETL entirely?

Only for the portion of your workload that is genuinely point-to-point replication with no transformation, which is rarely the whole thing. Zero-ETL excellently handles the "get this transactional table into the warehouse fresh" problem, and if that is all your analytics needs, it can indeed remove a large ETL surface. But most warehouse programmes also do modelling, conforming, deduplication, slowly-changing-dimension handling, joining across sources, and business-logic transformation — none of which zero-ETL performs, because it is deliberately point-to-point. Those responsibilities do not vanish; they relocate to inside the target, typically as SQL transformations, materialized views or a tool like dbt running on the landed raw tables. That is a legitimate and often good architecture — land raw with zero-ETL, transform in the warehouse — but it is important to see that you have not eliminated ETL, you have moved the T to after the L, turning it into ELT. So the accurate statement is that zero-ETL can replace the extract-and-load plumbing of your pipeline while your transformation logic continues to exist downstream. Teams that plan for this build clean raw and modelled layers from the start; teams that assume zero-ETL means no transformation discover the gap when the first business question needs a join the point-to-point copy cannot provide.

Satyam Kumar

返回博客

ai-architecture

Zero-ETL Deletes the Pipeline You Operate, Not the Semantics You Inherit

By Satyam KumarAugust 16, 202613 min read

zero-etl change data capture cdc aurora redshift integration data replication eventual consistency replication lag debezium data pipeline operational analytics schema migration data warehouse lakehouse aws data integration data engineering 2026

Zero-ETL Deletes the Pipeline You Operate, Not the Semantics You Inherit

Frequently Asked Questions

分享这篇文章

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

人工智能和云架构师。帮助团队构建可扩展到数百万的系统。

LinkedIn GitHub

Zero-ETL Deletes the Pipeline You Operate, Not the Semantics You Inherit

Frequently Asked Questions

分享这篇文章

Comments

Leave a comment