Siddhant Deval
Siddhant Deval
backend20 min read

The Outbox Pattern, CDC, and Exactly-Once DB-to-Broker Writes

Writing to a database and then publishing to a broker in two separate operations is the most common silent data corruption pattern in distributed systems. This article implements the Transactional Outbox + Debezium CDC pipeline that eliminates the dual-write hazard atomically, and closes the loop with consumer Inbox deduplication for end-to-end effectively-once semantics.

The Outbox Pattern, CDC, and Exactly-Once DB-to-Broker Writes

At 9:47 a.m., the payment service writes a successful charge to the database: INSERT INTO payments (id, status) VALUES ('P-42', 'charged'). At 9:47:00.003, it publishes payment.charged to Kafka. At 9:47:00.004, the process receives SIGTERM from a rolling deploy. The Kafka send() is in-flight and is dropped. The database row exists. The Kafka event does not. The fulfillment service never receives it. Order P-42 is charged but never fulfilled. The customer calls support three hours later.

The reverse: the Kafka send() succeeds at 9:47:00.003. At 9:47:00.004, the database INSERT fails due to a unique constraint violation (the payment was already processed by a concurrent request). The event is published. The database has no record. The fulfillment service processes a payment that was rejected.

Both failure modes have the same root cause: two operations, two systems, no atomicity. The dual-write hazard is not a race condition to be fixed with retry logic. It is a structural guarantee gap. The only correct fix is to make the database write and the event emission the same atomic operation.

Architectural Note

Series positioning: This is Part 3 of Messaging at Cloud Scale. The prerequisite articles are Distributed Transactions: Two-Phase Commit and the Transactional Outbox and Database Internals: B-Tree, LSM, WAL, and Query Plans (for WAL mechanics). This article implements the complete pipeline end-to-end.


1. The Dual-Write Hazard

1.1 The Four Failure Windows

TYPESCRIPT
// ❌ Naive dual-write — four distinct failure windows
async function processPayment(payment: Payment): Promise<void> {
  // Window 1: DB write succeeds, process crashes before produce
  await db.query('INSERT INTO payments (id, status) VALUES ($1, $2)', [payment.id, 'charged'])

  // Window 2: Produce times out / network error after DB write
  await producer.send({
    topic:    'payments.charged',
    messages: [{ key: payment.id, value: JSON.stringify(payment) }]
  })
  // If send() throws: DB row exists, event missing → unfulfilled order

  // Window 3: Produce succeeds, process crashes before returning
  // (rare but possible with at-most-once acks=0 or acks=1)

  // Window 4: Produce succeeds, DB write was actually rolled back by concurrent transaction
  // (possible if DB write and produce are not in the same transaction)
}

2. The Transactional Outbox Pattern

2.1 Core Mechanism

The outbox pattern solves dual-write by writing the event as a row in the same database transaction as the business entity. The event is committed atomically with the business data. A separate relay process reads from the outbox and publishes to the broker:

SQL
-- Outbox table: lives in the same PostgreSQL schema as business tables
CREATE TABLE outbox (
  id              UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
  aggregate_type  VARCHAR(100) NOT NULL,   -- 'payment', 'order', etc.
  aggregate_id    VARCHAR(255) NOT NULL,   -- the business entity ID
  event_type      VARCHAR(100) NOT NULL,   -- 'payment.charged'
  payload         JSONB        NOT NULL,
  topic           VARCHAR(255) NOT NULL,   -- target Kafka topic
  partition_key   VARCHAR(255),            -- Kafka message key
  created_at      TIMESTAMPTZ  NOT NULL DEFAULT NOW(),
  published_at    TIMESTAMPTZ,             -- NULL = unpublished
  retry_count     INT          NOT NULL DEFAULT 0
);

CREATE INDEX idx_outbox_unpublished ON outbox (created_at)
  WHERE published_at IS NULL;
TYPESCRIPT
// ✅ Outbox write: DB + event in a single transaction — atomically committed or rolled back
async function processPaymentWithOutbox(payment: Payment): Promise<void> {
  await db.transaction(async (trx) => {
    // 1. Business write
    await trx.query(
      'INSERT INTO payments (id, status, amount_cents) VALUES ($1, $2, $3)',
      [payment.id, 'charged', payment.amountCents]
    )

    // 2. Outbox write — same transaction, same atomicity boundary
    await trx.query(
      `INSERT INTO outbox (aggregate_type, aggregate_id, event_type, payload, topic, partition_key)
       VALUES ($1, $2, $3, $4, $5, $6)`,
      [
        'payment',
        payment.id,
        'payment.charged',
        JSON.stringify({ paymentId: payment.id, amountCents: payment.amountCents }),
        'payments.charged',
        payment.id,  // partition key: same payment → same partition → ordered
      ]
    )
    // If either write fails: entire transaction rolls back
    // If process crashes after commit: outbox row exists, relay will publish it
  })
}

2.2 Outbox Relay with SKIP LOCKED

TYPESCRIPT
// ✅ Outbox relay: polls for unpublished rows with SKIP LOCKED (no blocking)
async function runOutboxRelay(): Promise<void> {
  while (true) {
    await db.transaction(async (trx) => {
      // Lock up to 100 unpublished outbox rows — skip any currently locked by other relay instances
      const { rows } = await trx.query<OutboxRow>(`
        SELECT id, topic, partition_key, payload, aggregate_id, event_type
        FROM outbox
        WHERE published_at IS NULL
        ORDER BY created_at
        LIMIT 100
        FOR UPDATE SKIP LOCKED
      `)

      if (rows.length === 0) return

      // Publish all rows to Kafka
      await producer.sendBatch({
        topicMessages: rows.map(row => ({
          topic:    row.topic,
          messages: [{
            key:     row.partition_key ?? row.aggregate_id,
            value:   JSON.stringify(row.payload),
            headers: {
              'event-type':   row.event_type,
              'outbox-id':    row.id,
              'x-correlation-id': row.aggregate_id,
            }
          }]
        }))
      })

      // Mark as published only after successful Kafka write
      const ids = rows.map(r => r.id)
      await trx.query(
        'UPDATE outbox SET published_at = NOW() WHERE id = ANY($1)',
        [ids]
      )
      // If Kafka publish fails: transaction rolls back, published_at stays NULL → retry next cycle
    })

    await new Promise(resolve => setTimeout(resolve, 100))  // 100ms polling interval
  }
}
Pro Tip & Optimization

FOR UPDATE SKIP LOCKED is essential for running multiple relay instances without deadlocks. Standard FOR UPDATE blocks — two relay instances processing the same batch will deadlock. SKIP LOCKED means each relay instance skips rows locked by others and processes only what is available, enabling safe horizontal scaling of the relay.


3. Debezium CDC: WAL Tailing (Zero Polling Load)

3.1 How CDC Works

Change Data Capture with Debezium tails the PostgreSQL Write-Ahead Log (WAL) directly — the same mechanism that drives physical replication. Debezium acts as a logical replication slot consumer and converts WAL change events into Kafka messages:

Advantages over polling relay:

  • Zero DB polling load — WAL tailing uses the replication protocol, not SELECT queries
  • Sub-100ms latency from DB commit to Kafka publish
  • Ordered delivery: WAL events reflect the exact commit order
  • Works even if the application is down — Debezium catches up from its WAL position

3.2 Debezium Configuration

JSON
// Debezium PostgreSQL connector configuration
{
  "name": "outbox-connector",
  "config": {
    "connector.class":           "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname":         "postgres",
    "database.port":             "5432",
    "database.user":             "debezium",
    "database.password":         "${DB_PASSWORD}",
    "database.dbname":           "payments",
    "plugin.name":               "pgoutput",
    "slot.name":                 "debezium_outbox_slot",
    "publication.name":          "debezium_publication",
    "table.include.list":        "public.outbox",

    "transforms":                "outbox",
    "transforms.outbox.type":    "io.debezium.transforms.outbox.EventRouter",
    "transforms.outbox.table.field.event.id":     "id",
    "transforms.outbox.table.field.event.key":    "aggregate_id",
    "transforms.outbox.table.field.event.payload":"payload",
    "transforms.outbox.route.by.field":           "aggregate_type",
    "transforms.outbox.route.topic.replacement":  "${routedByValue}.events",

    "key.converter":   "org.apache.kafka.connect.storage.StringConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter"
  }
}
SQL
-- PostgreSQL: configure logical replication for Debezium
-- Run once as superuser:
ALTER SYSTEM SET wal_level = logical;
ALTER SYSTEM SET max_replication_slots = 5;
ALTER SYSTEM SET max_wal_senders = 5;

CREATE ROLE debezium REPLICATION LOGIN PASSWORD 'xxx';
GRANT SELECT ON ALL TABLES IN SCHEMA public TO debezium;
CREATE PUBLICATION debezium_publication FOR TABLE outbox;
Performance / Safety Warning

Logical replication slots hold WAL segments on disk until the slot consumer (Debezium) acknowledges them. If Debezium is down for an extended period, WAL files accumulate on disk and can fill the PostgreSQL data volume. Monitor pg_replication_slots for confirmed_flush_lsn lag and alert when disk-retained WAL exceeds 5 GB. Drop the slot immediately if Debezium will be offline for > 24 hours.


4. Consumer Inbox Deduplication

Debezium delivers at-least-once — WAL events can be replayed on connector restart. The consumer must be idempotent:

TYPESCRIPT
// ✅ Consumer Inbox table — deduplication gate for Debezium-delivered events
// (identical to Series 1, Part 5 idempotency gate — the transport changed, not the pattern)

async function processPaymentEvent(event: PaymentEvent): Promise<void> {
  // outbox-id header = the UUID from the outbox table row
  // Stable across Debezium retries — same outbox row → same outbox-id
  const idempotencyKey = event.outboxId  // from Kafka message header

  await db.query(`
    INSERT INTO payment_inbox (idempotency_key, payment_id, amount_cents, processed_at)
    VALUES ($1, $2, $3, NOW())
    ON CONFLICT (idempotency_key) DO NOTHING
  `, [idempotencyKey, event.paymentId, event.amountCents])
  // Debezium redelivery → ON CONFLICT DO NOTHING → no duplicate side effect
}

Summary

Concept Rule
Dual-write is structural The dual-write hazard is not a race condition — it is a structural guarantee gap; the only correct fix is to make the DB write and the event emission the same atomic operation.
Debezium vs polling Debezium WAL tailing adds zero polling load to the primary database and delivers events with sub-100ms latency; outbox polling with SKIP LOCKED is a workable fallback with measurable DB CPU overhead.
Inbox is non-negotiable Consumer Inbox deduplication is non-negotiable even with Outbox+CDC: Debezium delivers at-least-once; the consumer must handle redelivery idempotently.

What's Next

Part 4: Kafka Streams — Stateful Processing, Windows, and KTable moves from data transport to data transformation: using the Kafka Streams DSL to build stateful aggregations, sliding windows for time-bucketed metrics, and KTable changelog topics for maintaining materialized views — entirely within the Kafka cluster, no external state store required.

Research & Synthesis Note

This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.

#Outbox Pattern#CDC#Debezium#Kafka#Exactly Once#PostgreSQL#Backend#Distributed Systems
Siddhant Deval

Written by Siddhant Deval

Senior Full-Stack Engineer building high-scale architectures, browser performance engineering systems, and SaaS platforms.