Database-per-Service + CDC with Debezium (2026)

Q: Why is database-per-service worth the cost when shared databases are simpler?

Because schema coupling is the most expensive coupling there is, and shared databases trade short-term simplicity for permanent schema-evolution paralysis. When Orders, Finance, Search, and Analytics all read the same orders table directly, the moment one team renames a column or adds a NOT NULL constraint the other three have a production incident, and within six months the architecture has frozen because no migration is small enough to coordinate across four teams. Database-per-service makes the schema invisible to everyone but its owner — the Orders team can split a column, add a JSONB extension, or migrate to a partitioned table over a weekend because no one else can see the change. The cost is real (cross-service joins move to a CDC pipeline) but the cost of cross-service schema coupling is multiples larger. The architectural rule is one writer per schema, all readers consume events, joins happen in materialised views — not in live queries that reach across service boundaries. Teams that "do microservices" without enforcing this rule end up with a distributed monolith that has all of microservices’ deployment overhead and none of the schema-evolution velocity that justifies it.

Q: When should I use log-based CDC versus query-based CDC ("WHERE updated_at > last_run")?

Always log-based; never query-based for cross-service consistency. Query-based CDC has three fatal problems: it misses deletes (a deleted row is no longer in the table to query), it misses intermediate states of rows updated multiple times between polls (an order that goes pending → confirmed → shipped within one polling window shows up as only "shipped"), and it has a thundering-herd problem at scale because every poll scans an indexed range against the source database’s OLTP load. Log-based CDC reads the database transaction log directly (Postgres logical replication via pgoutput, MySQL binlog with row-based logging, SQL Server CT/CDC, MongoDB oplog) and captures every committed change including deletes, in commit order, without adding read load to the source. The two failure modes of log-based CDC are replication-slot bloat (Debezium dies or falls behind and the source can’t vacuum WAL) and schema-change incompatibility (a new NOT NULL column without a registry-compatibility check breaks the connector), but both have well-documented runbooks. Query-based CDC has no runbook because its failure modes are silent data loss.

Q: Do I need the outbox pattern if Debezium and Kafka Connect already support exactly-once?

You need one or the other; the outbox pattern is usually preferable for cross-aggregate consistency. The dual-write problem is real even with log-based CDC: if the application writes to the business table and emits a domain event to a separate topic in separate operations, one of them can fail and the system has an inconsistent state. The outbox pattern solves this by writing the event to an outbox table in the same transaction as the business write, then having Debezium stream only the outbox — since both writes commit together or not at all, the event stream is atomic with the database state. Native Kafka Connect EOS (enabled with exactly.once.support=enabled on the connector) works at the transport level, ensuring each Debezium record reaches Kafka exactly once, but it does not solve the dual-write problem because it only governs Debezium-to-Kafka, not application-to-database-plus-application-to-Debezium. Outbox is preferred because it also lets the application control the event schema (a clean domain event rather than a raw CDC envelope), the event ID (deterministic, hence trivially idempotent at consumers), and the timing (the application chooses when the event is published, not Debezium’s reader cycle). Native EOS is simpler if you are content streaming raw CDC envelopes and your application has no cross-aggregate consistency requirement.

Q: How do I choose between ksqlDB and Flink for stream processing?

ksqlDB is the right tool when the join is naturally expressible as SQL, when join state fits in a few hundred million keys, and when the team is small and would prefer not to operate a Flink cluster. ksqlDB runs on top of Kafka Streams, materialises compacted Kafka topics as queryable tables, and gives you a low operational floor with first-class Kafka integration. Flink is the right tool when join state runs into billions of keys, when you need event-time semantics with watermarks for late-arriving data, when the join is one-to-many fan-out with stateful complexity, when you need exactly-once end-to-end with two-phase commit into a downstream sink, or when the team already runs Flink for other streaming workloads and adding more jobs is cheap. Flink’s checkpointing, savepoints, state TTL, and async I/O give you operational levers ksqlDB does not. The 2026 reality is that most teams start with ksqlDB for the first three to five materialised views, hit the operational ceiling, and migrate the heaviest joins to Flink while keeping ksqlDB for the lighter ones. There is no shame in running both; the architectural rule is to pick per join, not per team. Avoid Apache Spark Structured Streaming for sub-second freshness use cases — it is mini-batch and the latency floor is in the seconds.

Q: What does the operational runbook for Debezium actually need to cover?

Six procedures and four dashboards, none optional. The dashboards are (1) replication-slot lag in bytes for every source database, alerting at 10 GB and paging at 50 GB because Postgres cannot vacuum WAL until the slot consumes it; (2) connector status across all Debezium tasks with FAILED state pager-level alerting; (3) schema-registry compatibility-check failures across all topics; (4) consumer-group lag for every downstream projection. The procedures are (a) re-snapshot a single table when the binlog position is unrecoverable, using Debezium’s ad-hoc signal-table mechanism; (b) rewind a consumer to a known LSN or timestamp, including the implications for downstream projections (most need to be rebuilt); (c) the schema-evolution workflow that ships every column change as nullable-then-backfill-then-NOT-NULL across at least two releases, with registry compatibility verified in CI; (d) drop-and-recreate a Postgres replication slot when slot bloat is unrecoverable, including the dual-write window during which application traffic must be paused or buffered; (e) cut over a connector from one Debezium version to the next without losing offsets; (f) replay a single hour of events from S3 archive when a downstream had a bug and produced corrupt projections. Without all six, the first 03:00 incident becomes a multi-day outage.

Q: How do I handle a referential-integrity gap in a cross-service join (an order whose customer was hard-deleted)?

By making every cross-service join a LEFT JOIN with explicit sentinel handling and an alert on null-rate. The architectural truth is that referential integrity across service boundaries is not enforceable by the database — the Orders service knows nothing about the Customers service’s deletes — so the materialised view must handle the gap gracefully rather than dropping rows or emitting null-laden records that break downstream. The pattern is to LEFT JOIN with the right-side service’s CDC topic, project null-right-side cases to a sentinel value ("customer_deleted" or a typed object with the original customer_id preserved), and emit a metric on null-rate so operations can see when it climbs. A small steady-state null-rate is normal (race between order placement and customer deletion); a spike indicates either a real bug (customer service emitted delete events without coordinating with order service’s data-retention policy) or a CDC pipeline failure (customer topic is lagging by hours and the join is reading stale state). The deeper architectural fix is to never hard-delete in services with downstream readers — use soft-delete with a deleted_at timestamp — but in practice many services do hard-delete and the projection layer must absorb the consequence.

Q: Should the CDC pipeline’s Kafka topics use log compaction or infinite retention?

It depends on whether the topic’s purpose is live state or audit — and the answer for most production setups is both, using two topics with different retention policies. The compacted topic keeps only the latest value per key, which means the topic functions as a queryable changelog of "current state of every entity" and is what downstream materialised views read. Compaction is essential for these because without it, every consumer starting fresh would need to read the entire history of every change ever made — untenable at production volumes. The non-compacted (or long-retention) topic keeps every event for the regulatory retention period (7 years for HIPAA-bound workloads, 6 for GDPR-aligned, indefinite for DPDPA breach-notification provenance) and serves audit, replay, and forensic-investigation use cases. Some teams stream to a WORM store (S3 Object Lock in Compliance mode) for audit and let the live Kafka topic compact — cheaper at scale because Kafka storage is expensive for long-retention data. The architectural rule is to make the choice consciously per topic, name it clearly in the topic taxonomy (orders.cdc.compacted versus orders.cdc.audit), and never assume a default — the wrong default has destroyed audit trails in real production incidents.

Q: How does CDC compose with the saga orchestration pattern for distributed transactions?

CDC topics are the natural input to a saga state machine, and the composition is one of the cleanest examples of why event-driven microservices work. A saga that spans order-placement → inventory-reservation → payment-authorisation → shipment-creation is fed by the CDC topics from each of those services: when the order CDC topic emits "order placed", the saga orchestrator (typically a Flink job, a state-machine engine like Temporal or Camunda, or a custom Kafka Streams app) receives it, calls the inventory service’s reservation API, listens for the inventory CDC topic’s "reserved" or "reservation_failed" event, and progresses or compensates accordingly. The benefit over direct API polling is that the saga never has to ask "what is the state of this order" — the CDC pipeline tells it as soon as a state change is committed, with at-least-once delivery and idempotent handling. The deeper treatment is in [the saga orchestration pattern](/en/blog/microservices-pattern-saga-orchestration-distributed-transactions-2026); the architectural rule worth surfacing here is that a saga without CDC inputs is fragile because it must poll, and a saga with CDC inputs is robust because it reacts. Most production saga implementations in 2026 are Flink jobs or Temporal workflows fed by CDC.

Q: When is CDC overkill and a simpler ELT pipeline the better architecture?

When the workload meets three conditions simultaneously: only one or two downstream consumers, latency tolerance measured in minutes or hours rather than seconds, and no compliance requirement for a replayable audit log. A nightly ELT job that reads from a Postgres read-replica with Airbyte or Fivetran into Snowflake or BigQuery costs $300–1,000/month, runs without Kafka or Debezium operations, and is fully adequate for BI reporting with daily latency. CDC pays for itself when (a) the same source data is consumed by three or more downstream projections — search index, warehouse, audit log, real-time dashboard, saga input, ML feature store — because the per-additional-consumer cost is near-zero once the pipeline exists; or (b) any single downstream needs sub-second freshness — a user-facing search index, a real-time fraud-detection model, a customer-facing dashboard; or (c) compliance requires an immutable replayable record of every change. If none of those apply, Kafka and Debezium are operational debt without payoff. The architectural rule is to start with the cheapest pipeline that meets the actual requirements, not the most sophisticated pipeline that meets imagined future requirements — and to migrate to CDC when the third downstream appears, not before.

Q: What’s the five-stage maturity ladder for database-per-service with CDC, and where do most teams stall?

Stage 0 is shared database — one cluster, one schema, everyone reads and writes whatever they like; microservices in name only, with monolithic deployment coupling and a multi-team standup for every schema change. Stage 1 is schema-per-service with polite contracts — same cluster, schema-scoped grants on paper, but operators still log in as superuser and the ORM still has cross-schema models; compliant only in theory. Stage 2 is strict ownership with API-only reads — each service owns its schema, no cross-schema reads, all cross-service data via synchronous APIs; compliant, but cross-service joins are expensive (the dashboard makes 10 API calls per row, latency is awful, the team longs for the shared database). Stage 3 is outbox + Debezium CDC + materialised views — outbox in the write transaction, Debezium streaming to Kafka, schema registry enforced, ksqlDB or Flink building per-team materialised views, idempotent consumers everywhere; the read-side problem is solved and OLTP stores are free to evolve. Stage 4 is event sourcing for aggregates that need audit and time-travel + CDC for everything else + warehouse as analytical truth; every projection rebuildable from the log, every schema evolves independently, adding a new read-side use case is a one-team task. Most teams stall at Stage 1 or Stage 2 because Stage 3 looks like a Kafka project and they have no Kafka operator — but the operational burden of managed Kafka (MSK, Confluent Cloud, Aiven) plus managed Debezium is well within reach of a four-person platform team, and the alternative is a distributed monolith forever.

Satyam Kumar

عودة إلى المدونة

microservices-patterns

Database-per-Service and Cross-Service Joins with CDC — The 2026 Architecture for Reporting Without Distributed Transactions

May 29, 202620 min read

Frequently Asked Questions

شارك هذه المقالة

Twitter LinkedIn WhatsApp

Satyam

مهندس الذكاء الاصطناعي والسحابة. مساعدة الفرق على بناء أنظمة تتسع للملايين.

Database-per-Service and Cross-Service Joins with CDC — The 2026 Architecture for Reporting Without Distributed Transactions

Frequently Asked Questions

شارك هذه المقالة

Comments

Leave a comment