CDC at Scale: When Debezium Is the Critical Path (2026)

Q: What does it mean that CDC becomes the critical path at scale?

It means that once change data capture is streaming from a production database that other systems depend on, a failure anywhere in the CDC pipeline can threaten the source database itself, so CDC is no longer an optional side-channel but a load-bearing production dependency. In a proof of concept, CDC feels like a harmless read-only tap: it streams a few changes from an idle database and nothing bad happens if it stops. In production the coupling is much tighter and much more dangerous. Log-based CDC works by holding a position in the database's transaction log, and the database will retain the log from that position onward until the consumer confirms it has processed it. That means a downstream outage — a crashed connector, a stalled sink, a full Kafka cluster — stops the source database from reclaiming its own log space, and left unmonitored it can fill the source disk and take down the exact OLTP system CDC was supposed to observe without disturbing. The reframing that follows is that at scale CDC must be engineered, monitored, and contracted like any other piece of load-bearing infrastructure: with alerting on the source-coupling metrics, non-disruptive backfill, enforced schema compatibility, and idempotent sinks. Teams that treat CDC as a connector they configured once tend to turn it off after their second source-disk incident; teams that treat it as infrastructure run it calmly across hundreds of tables.

Q: How can a CDC consumer failure fill up the source database disk?

Through the replication slot, which is the mechanism that guarantees no change is ever lost and, for exactly that reason, is also the mechanism that can fill your source disk. In PostgreSQL, logical decoding exposes changes through a replication slot, and a slot guarantees the database will retain every write-ahead log (WAL) segment since the consumer's last confirmed position — even if the consumer has been dead for hours. That retention guarantee is the feature: it is what lets a consumer restart after a crash and resume exactly where it left off with no data loss. It is also the footgun: because the database cannot discard WAL the slot still needs, an unhealthy or stalled consumer causes WAL to accumulate without bound. If your Kafka Connect worker crashes at midnight and nobody is paged, the slot keeps pinning WAL, the WAL directory grows through the night, and by morning the primary database — often a payments or orders database, the worst possible blast radius — is out of disk. This is why CDC lag must be treated as a source-availability metric and not merely a data-freshness metric, and why it deserves the same alerting seriousness as replication lag on a hot standby. The control is to monitor slot lag and alert well before it becomes disk pressure, to bound retention where the platform allows, and to have a runbook for a stalled consumer that includes safely advancing or dropping a slot that has been abandoned.

Q: What is an incremental snapshot and why does it matter for CDC?

An incremental snapshot is a technique for capturing the existing contents of a large table without stopping the stream of live changes and without holding long locks or long transactions on the source. It matters because the initial snapshot — reading the data that already exists before CDC started — is where change data capture most often hurts the source database. Streaming the ongoing changes is cheap once the connector is caught up, because reading the transaction log has modest overhead. Getting to caught up on a table that already holds a billion rows is the expensive part, and it is the step teams skip in testing because their test table has ten thousand rows. The old approach was a blocking snapshot: pause, lock the table, read everything, then start streaming — which on a production OLTP table is indistinguishable from an outage. The modern approach, exposed through Debezium's signalling mechanism as the watermark-based DBLog algorithm, reads the table in bounded chunks while streaming continues, interleaving snapshot chunks with live changes and reconciling them with watermarks, so there is no long lock and no stop-the-world read. It can be paused, resumed, and triggered on demand to re-snapshot a single table without restarting the connector, which is what makes backfills survivable: when a new downstream consumer needs history, you signal an incremental snapshot of just the tables it needs and the rest of the pipeline never notices. Keeping each chunk small enough to stay a sub-second query against the source turns the snapshot into background noise instead of a load event.

Q: Why is schema evolution such a dangerous failure mode in CDC?

Because it is the quietest failure in the whole pipeline: the producer's schema change succeeds instantly and locally with no error, while the break lands downstream, asynchronously, and often minutes or hours later. A source team ships a routine ALTER TABLE — dropping a column, changing a type — and at the source everything is fine; the migration committed, the application works. But CDC faithfully propagates that schema change into the change stream, and a downstream consumer that was written against the old schema now deserializes the new event shape into garbage or crashes outright. Nothing at the source ever indicated a problem, so the source team has no idea they just paged the data platform. Without a contract sitting between producer and consumers, CDC becomes exactly the path that carries an unagreed schema change straight into every system that reads the stream. The control is a schema registry with an enforced compatibility mode: every change event carries a schema, the registry versions it, and a producer change that would break existing consumers is rejected or flagged at registration rather than discovered in production. Backward compatibility — where a new schema can still read old data — is the usual default and lets consumers upgrade on their own timeline, while a dropped or retyped column that violates it is caught before it ships. This is the CDC-specific edge of the broader data-contracts discipline: the contract has to live on the change stream, not only on the batch tables, because the change stream is the fast path from an upstream migration to a downstream outage.

Q: Is CDC exactly-once or at-least-once, and what does that mean for sinks?

CDC delivery is at-least-once in practice, which means your sinks must be idempotent or you will get duplicate rows every time the pipeline hiccups. The reasons are structural: distributed connector runtimes like Kafka Connect rebalance work across workers when the cluster membership changes, connectors restart after failures, and offset commits are not perfectly synchronised with the emission of every event. Any of these can cause the same change event to be delivered more than once. If a sink responds to a change event by inserting a row, then every redelivery inserts another copy, and after a few rebalances your downstream table is full of duplicates. The fix is to make the sink idempotent: apply each change as an upsert keyed on the row's stable primary key, so that re-applying the same event converges to the same state rather than accumulating copies. This is the same at-least-once reality that governs any Kafka-based architecture, and it means replay is safe: after fixing a bug you can re-emit a range of events and the idempotent sink absorbs the repeats. Pair idempotent sinks with a topic-per-table layout so a replay of one table does not disturb the others, and treat exactly-once as a property you engineer at the sink through idempotency, not a guarantee you assume from the transport. Assuming exactly-once delivery and writing non-idempotent inserts is one of the most common ways a CDC pipeline silently corrupts downstream data.

Q: When should you NOT use CDC?

When the requirement is a synchronous, transactional, or sub-second guarantee, because CDC is asynchronous and eventually consistent by construction. Change data capture observes committed changes after the fact and propagates them downstream on a best-effort, at-least-once basis with latency that is usually low but never zero and never guaranteed. If a business rule needs "commit both of these or neither" — a distributed transaction semantic — CDC cannot provide it, because by the time CDC sees a change it has already committed at the source and cannot be rolled back atomically with anything else. Likewise, if a rule needs "take this action within a specified number of milliseconds of the event," CDC is the wrong tool, because it offers no latency SLA and a backlog or rebalance can delay an event arbitrarily; the right tool is an event emitted directly on the transactional path. A second case to avoid is capturing everything by default: streaming every column of every table because it is easy to enable costs you in WAL volume, topic storage, and blast radius, and it makes every schema change a bigger risk surface — capture only the tables and columns a consumer actually needs, and treat adding a new capture as a deliberate change. The general principle is that CDC is superb for propagating committed state changes to systems that can tolerate eventual consistency, and wrong for anything that needs synchronous coordination, a hard latency bound, or a transactional guarantee spanning the source and the destination.

Q: What are the most common CDC-at-scale anti-patterns?

Six recur across pipelines that turn into incident generators. First, no slot-lag alert: treating CDC lag as a freshness metric instead of a source-availability metric, so an unmonitored stalled consumer pins WAL and fills the production database's disk — the incident that turns a data pipeline into a database outage. Second, a blocking initial snapshot on a hot table: locking or long-transaction-reading a billion-row OLTP table, when incremental chunked snapshots let streaming continue with no stop-the-world read. Third, schema changes with no registry contract: letting producers ALTER freely with no compatibility enforcement on the stream, so the break is silent and downstream. Fourth, non-idempotent sinks: assuming exactly-once when delivery is at-least-once, so rebalances and replays duplicate rows because the sink inserts rather than upserts on a stable key. Fifth, using CDC for a synchronous need: reaching for change capture when the requirement is a transactional guarantee or a sub-second command, both of which need an event on the transactional path instead. Sixth, capturing everything: streaming every column of every table by default and paying in WAL volume, storage, and blast radius. Every one of these traces to the same root: CDC at scale is a production dependency of the source database, and the disciplines that make it safe — source-coupling alerts, non-disruptive backfill, enforced schema compatibility, idempotent sinks — are exactly the ones a proof of concept never forces you to build.

Q: How should CDC maturity be sequenced?

Sequence the work so the source-protection controls come before breadth, because one table captured safely is worth more than fifty tables streamed on hope. Level 0 is the proof of concept: one connector, one low-traffic table, a blocking snapshot, no schema registry, no lag alerting — it proves the mechanism and nothing about operations. Level 1 is source-safe capture: slot-lag monitoring and alerting in place, bounded WAL retention, and a runbook for a stalled consumer, so CDC can no longer silently threaten the source disk; this is the first level that is safe to point at a production database. Level 2 is non-disruptive backfill: incremental, signal-driven snapshots so onboarding a new table or consumer never locks the source, turning backfills from change-managed events into routine operations. Level 3 is contracted schema evolution: a schema registry with an enforced compatibility mode on every stream, so an upstream migration cannot silently break consumers and the source team can change against the stream safely. Level 4 is CDC as a platform: distributed Kafka Connect with offset hygiene, idempotent sinks, per-table topics feeding warehouse, lakehouse and search, and end-to-end observability, at which point CDC is boring infrastructure — the goal — and composes cleanly with downstream storage and processing decisions. The reframing that carries the whole path is that CDC at scale is not a feature you switch on but a production dependency you engineer, monitor, and contract like any other load-bearing system.

Satyam Kumar

عودة إلى المدونة

ai-architecture

CDC at Scale: When Debezium Becomes the Critical Path

By Satyam KumarAugust 27, 202611 min read

change data capture cdc debezium kafka connect replication slot write-ahead log incremental snapshot schema registry schema evolution logical decoding postgres cdc data engineering streaming data backfill data contracts 2026

CDC at Scale: When Debezium Becomes the Critical Path

Frequently Asked Questions

شارك هذه المقالة

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

مهندس الذكاء الاصطناعي والسحابة. مساعدة الفرق على بناء أنظمة تتسع للملايين.

LinkedIn GitHub

CDC at Scale: When Debezium Becomes the Critical Path

Frequently Asked Questions

شارك هذه المقالة

Comments

Leave a comment