The Outbox Pattern, CDC, and Exactly-Once DB-to-Broker Writes
Writing to a database and then publishing to a broker in two separate operations is the most common silent data corruption pattern in distributed systems. This article implements the Transactional Outbox + Debezium CDC pipeline that eliminates the dual-write hazard atomically, and closes the loop with consumer Inbox deduplication for end-to-end effectively-once semantics.
Messaging at Cloud Scale
The Outbox Pattern, CDC, and Exactly-Once DB-to-Broker Writes
At 9:47 a.m., the payment service writes a successful charge to the database: INSERT INTO payments (id, status) VALUES ('P-42', 'charged'). At 9:47:00.003, it publishes payment.charged to Kafka. At 9:47:00.004, the process receives SIGTERM from a rolling deploy. The Kafka send() is in-flight and is dropped. The database row exists. The Kafka event does not. The fulfillment service never receives it. Order P-42 is charged but never fulfilled. The customer calls support three hours later.
The reverse: the Kafka send() succeeds at 9:47:00.003. At 9:47:00.004, the database INSERT fails due to a unique constraint violation (the payment was already processed by a concurrent request). The event is published. The database has no record. The fulfillment service processes a payment that was rejected.
Both failure modes have the same root cause: two operations, two systems, no atomicity. The dual-write hazard is not a race condition to be fixed with retry logic. It is a structural guarantee gap. The only correct fix is to make the database write and the event emission the same atomic operation.
Series positioning: This is Part 3 of Messaging at Cloud Scale. The prerequisite articles are Distributed Transactions: Two-Phase Commit and the Transactional Outbox and Database Internals: B-Tree, LSM, WAL, and Query Plans (for WAL mechanics). This article implements the complete pipeline end-to-end.
1. The Dual-Write Hazard
1.1 The Four Failure Windows
2. The Transactional Outbox Pattern
2.1 Core Mechanism
The outbox pattern solves dual-write by writing the event as a row in the same database transaction as the business entity. The event is committed atomically with the business data. A separate relay process reads from the outbox and publishes to the broker:
2.2 Outbox Relay with SKIP LOCKED
FOR UPDATE SKIP LOCKED is essential for running multiple relay instances without deadlocks. Standard FOR UPDATE blocks — two relay instances processing the same batch will deadlock. SKIP LOCKED means each relay instance skips rows locked by others and processes only what is available, enabling safe horizontal scaling of the relay.
3. Debezium CDC: WAL Tailing (Zero Polling Load)
3.1 How CDC Works
Change Data Capture with Debezium tails the PostgreSQL Write-Ahead Log (WAL) directly — the same mechanism that drives physical replication. Debezium acts as a logical replication slot consumer and converts WAL change events into Kafka messages:
Advantages over polling relay:
- Zero DB polling load — WAL tailing uses the replication protocol, not SELECT queries
- Sub-100ms latency from DB commit to Kafka publish
- Ordered delivery: WAL events reflect the exact commit order
- Works even if the application is down — Debezium catches up from its WAL position
3.2 Debezium Configuration
Logical replication slots hold WAL segments on disk until the slot consumer (Debezium) acknowledges them. If Debezium is down for an extended period, WAL files accumulate on disk and can fill the PostgreSQL data volume. Monitor pg_replication_slots for confirmed_flush_lsn lag and alert when disk-retained WAL exceeds 5 GB. Drop the slot immediately if Debezium will be offline for > 24 hours.
4. Consumer Inbox Deduplication
Debezium delivers at-least-once — WAL events can be replayed on connector restart. The consumer must be idempotent:
Summary
| Concept | Rule |
|---|---|
| Dual-write is structural | The dual-write hazard is not a race condition — it is a structural guarantee gap; the only correct fix is to make the DB write and the event emission the same atomic operation. |
| Debezium vs polling | Debezium WAL tailing adds zero polling load to the primary database and delivers events with sub-100ms latency; outbox polling with SKIP LOCKED is a workable fallback with measurable DB CPU overhead. |
| Inbox is non-negotiable | Consumer Inbox deduplication is non-negotiable even with Outbox+CDC: Debezium delivers at-least-once; the consumer must handle redelivery idempotently. |
What's Next
Part 4: Kafka Streams — Stateful Processing, Windows, and KTable moves from data transport to data transformation: using the Kafka Streams DSL to build stateful aggregations, sliding windows for time-bucketed metrics, and KTable changelog topics for maintaining materialized views — entirely within the Kafka cluster, no external state store required.
This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.