Siddhant Deval
Siddhant Deval
backend17 min read

Delivery Guarantees: At-Most-Once, At-Least-Once, and Effectively-Once

Delivery guarantees are not broker features you toggle on — they are contracts that require coordinated design on the producer, broker, and consumer sides simultaneously. This article derives all three delivery modes from first principles, implements idempotency key deduplication, and shows why at-least-once transport plus idempotent consumer equals effectively-once business outcome.

Delivery Guarantees: At-Most-Once, At-Least-Once, and Effectively-Once

The payment service publishes a payment.charged event. The Kafka broker receives it, writes it to the leader's log, and then the broker crashes before replicating to followers. On broker recovery with acks=1, the message is gone — the consumer never sees it. The payment was charged, but the downstream fulfillment service has no record of it. The order is in limbo.

The inverse: the broker receives the event, replicates it, but crashes before sending the producer acknowledgement. The producer's send() call times out, triggers a retry, and the consumer processes payment.charged twice for the same transaction. The customer is charged twice.

Both scenarios are not edge cases. They are the inherent ambiguity of distributed systems — the two-generals problem applied to message delivery. Delivery guarantees are not broker settings you enable. They are contracts that require coordinated design across all three participants: producer, broker, and consumer.

Architectural Note

Series positioning: This is Part 5 of Distributed Messaging Systems — the convergence point where the Kafka (Part 3) and RabbitMQ (Part 4) paths meet. The three delivery modes apply identically to both brokers; the mechanism differs. The prerequisite mental model is consistency trade-offs from Consistency Models and CAP Theorem. Follow-up: Retry Engineering: Exponential Backoff, Jitter, and Idempotency.


1. The Three Delivery Modes

1.1 At-Most-Once: Fire and Forget

At-most-once means a message is delivered zero or one time — never more. Duplicates are impossible. Data loss is possible.

TYPESCRIPT
// ❌ At-most-once on Kafka — auto-commit before processing
const consumer = kafka.consumer({ groupId: 'analytics' })
await consumer.run({
  // enable.auto.commit = true (kafkajs default): offset committed on interval
  // regardless of whether eachMessage handler has finished
  eachMessage: async ({ message }) => {
    // If process crashes here — after auto-commit, before insert — message is lost
    await analyticsDb.insert(JSON.parse(message.value!.toString()))
  }
})

// ❌ At-most-once on RabbitMQ — autoAck: true
await channel.consume('analytics', (msg) => {
  if (!msg) return
  // Message acked by broker the moment it is delivered — before this line runs
  analyticsDb.insert(JSON.parse(msg.content.toString()))
  // If process crashes mid-insert: message already acked, permanently gone
}, { noAck: true })
Performance / Safety Warning

enable.auto.commit=true (the Kafka default) is at-most-once, not at-least-once. The auto-commit interval fires on a timer — it commits the offset of the last fetched message, not the last processed message. A crash between auto-commit and processing completes = silent data loss. The naming is counterintuitive but the behavior is unambiguous.

Use at-most-once when: loss is acceptable and recoverability is not required — analytics sampling, metrics aggregation, telemetry where approximate counts are sufficient.

1.2 At-Least-Once: Commit After Processing

At-least-once means a message is delivered one or more times — loss is impossible, duplicates are possible.

TYPESCRIPT
// ✅ At-least-once on Kafka — manual offset commit AFTER processing
const consumer = kafka.consumer({ groupId: 'fulfillment' })
await consumer.run({
  autoCommit: false,   // disable auto-commit
  eachMessage: async ({ topic, partition, message }) => {
    // 1. Process first
    await fulfillmentDb.insert(JSON.parse(message.value!.toString()))
    // 2. Commit offset AFTER successful processing
    await consumer.commitOffsets([{
      topic,
      partition,
      offset: (BigInt(message.offset) + 1n).toString(),
    }])
    // Crash window: if process dies after insert but before commitOffsets,
    // consumer restarts at previous offset → re-processes the same message
    // → duplicate insert → idempotency required
  }
})

// ✅ At-least-once on RabbitMQ — manual ack AFTER processing
await channel.prefetch(10)
await channel.consume('fulfillment', async (msg) => {
  if (!msg) return
  await fulfillmentDb.insert(JSON.parse(msg.content.toString()))
  channel.ack(msg)   // ack AFTER insert — crash before ack = redelivery
})

The at-least-once crash window:

This is the exactly-once illusion most systems need to handle: at-least-once delivery with idempotent processing.

1.3 Effectively-Once: At-Least-Once + Idempotent Consumer

"Effectively-once" is not a delivery guarantee — it is a business outcome achieved by combining at-least-once transport with idempotent consumer logic. The message may arrive twice; the effect on the system is identical to arriving once.

TYPESCRIPT
// ✅ Idempotency gate — PostgreSQL ON CONFLICT DO NOTHING
async function processPaymentEvent(event: PaymentEvent): Promise<void> {
  await db.query(`
    INSERT INTO payment_processed (idempotency_key, payment_id, amount, processed_at)
    VALUES ($1, $2, $3, NOW())
    ON CONFLICT (idempotency_key) DO NOTHING
  `, [event.idempotencyKey, event.paymentId, event.amount])
  // If this row already exists: no error, no duplicate charge, no side effect
  // Second delivery of the same event is silently discarded
}

// ✅ Redis-based idempotency gate — for consumers without DB write
async function processAnalyticsEvent(event: AnalyticsEvent): Promise<void> {
  const key = `idem:analytics:${event.idempotencyKey}`
  const isNew = await redis.set(key, '1', 'NX', 'EX', 86400) // 24h TTL
  if (!isNew) return   // duplicate — skip processing
  await analyticsQueue.add(event)
}
Crucial Requirement

Idempotency keys must be stored durably — in a database row or Redis with persistence enabled. An in-memory Set<string> of processed IDs resets on every crash or restart, making it useless for exactly the scenario it is meant to protect against: post-crash redelivery.


2. True Exactly-Once: Kafka Transactions

Kafka's transactional producer provides genuine exactly-once semantics within the Kafka ecosystem — a message is written to the topic and the consumer offset is committed atomically, with no duplicates possible even across producer retries.

TYPESCRIPT
// ✅ Kafka transactional producer — exactly-once write + offset commit
const producer = kafka.producer({
  transactionalId: 'payment-processor-1',   // stable across restarts — must be unique per instance
  idempotent: true,                          // required for transactions
  maxInFlightRequests: 1,                    // transactions require sequential inflight
})
await producer.connect()
await producer.transaction(async (tx) => {
  // 1. Produce the output event
  await tx.send({
    topic: 'fulfillment.requests',
    messages: [{ key: event.paymentId, value: JSON.stringify(event) }]
  })
  // 2. Commit the source consumer's offset atomically with the produce
  await tx.sendOffsets({
    consumerGroupId: 'payment-processor',
    topics: [{ topic: 'payments.charged', partitions: [{ partition, offset }] }]
  })
  // Both succeed or both roll back — never a state where offset is committed
  // without the output event being written, or vice versa
})

When exactly-once is and is not justified:

Scenario Recommendation
Financial transactions (charges, refunds) At-least-once + DB idempotency key (more portable than Kafka-specific transactions)
Read model projections (CQRS) At-least-once + ON CONFLICT DO NOTHING
Kafka Streams aggregations Exactly-once via processing.guarantee=exactly_once_v2
Cross-broker pipelines (Kafka → RabbitMQ) At-least-once + idempotent consumer — Kafka transactions do not span brokers
Analytics / metrics At-most-once — approximate is acceptable, simplicity wins
Architectural Note

Kafka transactional exactly-once adds latency — the transaction coordinator must write the transaction markers and wait for quorum before the consumer can read committed messages (transactions use isolation.level=read_committed). For most business applications, at-least-once + application-level idempotency is simpler, more portable, and fast enough.


3. The Idempotency Key Contract

3.1 Who Generates It

The idempotency key must be generated by the event source (the system that first observed the business fact), not by the producer or consumer:

TYPESCRIPT
// ❌ Consumer generates idempotency key — not idempotent across retries
// A new UUID is generated on every message delivery (including redeliveries)
async function processOrder(msg: Message): Promise<void> {
  const idempotencyKey = uuid()   // different value on every invocation
  await db.insert({ idempotencyKey, orderId: msg.orderId })
  // Re-delivery → new UUID → INSERT succeeds → duplicate row
}

// ✅ Idempotency key from the event itself — stable across redeliveries
interface OrderEvent {
  eventId:         string   // UUID generated at the point of order creation, never changes
  orderId:         string
  customerId:      string
  totalCents:      number
}

async function processOrder(event: OrderEvent): Promise<void> {
  await db.query(`
    INSERT INTO orders_processed (idempotency_key, order_id, customer_id, total_cents)
    VALUES ($1, $2, $3, $4)
    ON CONFLICT (idempotency_key) DO NOTHING
  `, [event.eventId, event.orderId, event.customerId, event.totalCents])
}

3.2 Idempotency Key TTL and Storage

TYPESCRIPT
// ✅ Database idempotency table — permanent, no TTL needed for financial events
CREATE TABLE payment_idempotency (
  idempotency_key  VARCHAR(128) PRIMARY KEY,
  payment_id       UUID NOT NULL,
  processed_at     TIMESTAMPTZ DEFAULT NOW()
);
-- Keep indefinitely — financial audit trail requires it

// ✅ Redis idempotency — appropriate for events with a natural time window
const TTL_SECONDS = 7 * 24 * 3600   // 7 days — longer than max broker retention
await redis.set(
  `idem:order:${event.eventId}`,
  event.orderId,
  'NX',    // set only if not exists
  'EX',    // with expiry
  TTL_SECONDS
)
// TTL must exceed max possible redelivery window (broker retention + consumer lag)
Performance / Safety Warning

Set Redis idempotency TTL longer than your broker's message retention period. If messages can be retained for 7 days, your idempotency TTL must be at least 7 days + maximum expected consumer lag. An expired idempotency key before the message expires means a late-replayed message will bypass the deduplication gate.


4. Delivery Mode Decision Framework


Summary

Concept Rule
Effectively-once outcome At-least-once + idempotent consumer = effectively-once business outcome; true exactly-once requires transactional producers and is rarely justified.
Idempotency key storage Idempotency keys must be stored durably (DB row or Redis) — in-memory deduplication resets on every crash.
Auto-commit is at-most-once Auto-commit (enable.auto.commit=true) is at-most-once, not at-least-once — it commits offsets before processing is confirmed complete.

What's Next

Part 6: Consumer Patterns — Groups, Lag, Backpressure, and Rebalancing moves from delivery guarantees to the consumer lifecycle: how partition assignment works under load, measuring and responding to consumer lag, and the cooperative rebalancing strategies that prevent the stop-the-world pauses that plagued Kafka consumer groups before 2.4.

Research & Synthesis Note

This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.

#Delivery Guarantees#Idempotency#Exactly Once#Kafka#Distributed Systems#Backend#Reliability
Siddhant Deval

Written by Siddhant Deval

Senior Full-Stack Engineer building high-scale architectures, browser performance engineering systems, and SaaS platforms.