Queues vs Logs: The Architecture Decision That Changes Everything

The engineering team that builds the notification system using RabbitMQ ships quickly, the product works, and eighteen months later they need to add a real-time analytics pipeline that replays the last 90 days of events to train a recommendation model. They discover that RabbitMQ deleted every message the moment the original consumer acknowledged it. The events are gone. The only record of what happened is the application database, which was not designed to be an event log. The rewrite takes six weeks. This is not a RabbitMQ failure — it is a model mismatch. The team needed a log. They built with a queue.

A message is a fact about the world — and the single most consequential decision in any messaging architecture is what happens to that fact after the first consumer reads it.

1. Two Primitives, Two Mental Models

1.1 The Queue: Task Dispatch

A message queue operates on a simple contract: a message exists until exactly one consumer acknowledges it, then it is deleted. The queue is a work distributor — tasks flow in, workers compete to claim them, and the queue tracks only which tasks are outstanding.

Key properties of the queue model:

1.2 The Log: Event Streaming

A commit log operates on the opposite contract: messages are appended to a sequential, durable log and never deleted on consumption. Each consumer group maintains its own offset cursor — a pointer to the position it has read up to. Two consumer groups reading the same log are fully independent.

Key properties of the log model:

2. The Offset Model: Why Consumer Independence Matters

2.1 What an Offset Is

An offset is a 64-bit integer — the position of a message within a partition. The log is immutable; offsets are monotonically increasing. A consumer group's progress is entirely represented by a single number per partition: the committed offset, which is the next message it will read.

// ✅ Kafka consumer: offset cursor is per consumer group, per partition
const consumer = kafka.consumer({ groupId: 'analytics-service' })
await consumer.subscribe({ topic: 'orders', fromBeginning: false })

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    const offset = message.offset  // "127" — position in partition
    const value  = message.value?.toString()

    await analyticsDb.insert(JSON.parse(value!))

    // Offset committed after processing — not before
    // Committing before processing = at-most-once (data loss on crash)
    await consumer.commitOffsets([{
      topic,
      partition,
      offset: (BigInt(offset) + 1n).toString(),
    }])
  }
})

2.2 Independent Consumer Groups

The same log, consumed twice, independently:

// ✅ Two completely independent consumer groups reading the same topic
// Adding a new consumer group requires zero changes to the producer or existing consumers

const analyticsConsumer      = kafka.consumer({ groupId: 'analytics-service' })
const notificationsConsumer  = kafka.consumer({ groupId: 'notifications-service' })
const auditConsumer          = kafka.consumer({ groupId: 'audit-service' }) // added 6 months later

// All three subscribe to the same topic
// All three maintain independent offsets
// Analytics at offset 12,400 — Notifications at offset 12,399 — Audit at offset 0 (replaying from start)

This is impossible with a queue. Once Worker 1 acknowledges a message, Worker 2 cannot read it. Consumer group independence is a log property.

3. The Decision Matrix

Choosing the wrong primitive is not a performance problem you can tune away. It is an architectural mismatch that forces a rewrite. Use this matrix to make the decision before you write any code:

3.1 The Replay Use Cases That Force Logs

These requirements all mandate a log. If you discover any of them after building on a queue, you face a rewrite:

// These are impossible on a queue — only possible on a log

// 1. Replay to train a new ML model
await analyticsConsumer.seek({ topic: 'orders', partition: 0, offset: '0' })
// Reprocess all 6 months of order events from the beginning

// 2. Add a new microservice and backfill its read model
const inventoryConsumer = kafka.consumer({ groupId: 'inventory-read-model' })
// Starts at beginning — catches up independently while other consumers run at head

// 3. Debug a production incident by replaying a 2-hour window
await consumer.seek({ topic: 'payments', partition: 0, offset: incidentStartOffset })
// Replay 2 hours of events through the debug consumer to reproduce the issue

// 4. Time-travel: consumer crashed for 4 days — no data loss
// Queue: messages expired or stack depth unmanageable
// Log: consumer resumes from its committed offset — all events retained

4. Hybrid Architectures: When to Use Both

Real systems use both primitives — each for the role it is designed for:

• Kafka (log) for the primary event stream — multiple consumers, replay, audit, analytics
• RabbitMQ (queue) for task dispatch within a single bounded context — fraud alert workers compete to process each alert exactly once

5. Managed Primitives: Cloud Equivalents

Understanding the abstract model lets you map it to managed cloud services:

6. The Retention Model: How Long Does the Log Remember?

Kafka and Kinesis retain messages by time or size, not by consumption:

# Kafka topic retention configuration
kafka-configs.sh --alter --topic orders \
  --add-config 'retention.ms=604800000'  # 7 days (not "until consumed")
  # OR
  --add-config 'retention.bytes=107374182400'  # 100 GB per partition

# Kinesis shard retention
aws kinesis increase-stream-retention-period \
  --stream-name orders \
  --retention-period-hours 168  # 7 days (max without extended retention: 8760 = 1 year)

This has a critical implication: a consumer that falls behind and does not catch up within the retention window permanently loses access to the unread messages. The log is not unlimited storage — it is a sliding window.

// ✅ Handle OffsetOutOfRangeError — consumer was offline longer than retention
const consumer = kafka.consumer({
  groupId: 'analytics-service',
  // On out-of-range: jump to earliest available offset (data gap accepted)
  // Alternative: 'latest' — skip the gap and start from now (data loss accepted)
})

consumer.on(consumer.events.GROUP_JOIN, async () => {
  // Emit metric: consumer group caught up / offset delta to head
})

Summary

What's Next

Part 3: Kafka Internals — Partitions, Leaders, and the Commit Log dives below the offset model into how Kafka actually stores and replicates data: the write path from producer.send() through leader election, ISR replication, and acks semantics. Part 4 covers the RabbitMQ and AMQP model in equivalent depth — both are independent reads grounded in the queue vs log distinction established here.