Siddhant Deval
Siddhant Deval
backend22 min read

Redis Pub/Sub vs. Streams: Choosing the Right Messaging Primitive

Redis Pub/Sub is a stateless, fire-and-forget broadcast bus — any subscriber that disconnects loses every message published during its absence, permanently. Redis Streams is a persistent, ordered log with consumer groups, delivery acknowledgment, and a pending entry list that tracks every unacknowledged message. This article maps both primitives to their correct use cases and shows when Streams replace Kafka.

Redis Pub/Sub vs. Streams: Choosing the Right Messaging Primitive

Redis is not a cache you bolt onto a slow database — it is a data structure server with a precisely bounded contract: sub-millisecond latency, in-memory semantics, and optional persistence. Nowhere is this contract more consequential than in Redis's messaging primitives. Pub/Sub and Streams both allow producers to send messages and consumers to receive them. But their durability contracts are as different as TCP and UDP. Pub/Sub is a stateless broadcast bus with zero durability — the server makes no attempt to persist messages, and any subscriber that disconnects for any reason loses every message published during its absence, permanently, with no error and no log entry. Streams are a persistent, ordered log with delivery acknowledgment. Choosing the wrong primitive does not produce a deployment error; it produces a production incident weeks after launch when users report missing notifications.

Architectural Note

This is Part 5 of the Redis Mastery series. It can be read independently after Part 4.

On-ramp: This article uses two concepts from earlier parts. From Part 1: Streams store entries in listpack-packed radix tree nodes — the same compact encoding used by Hashes and Sorted Sets at small sizes. From Part 2: Streams are persisted via AOF/RDB like any other Redis data structure — enabling AOF gives you durable Streams; without persistence, Stream entries are lost on restart like all other keys.


1. Pub/Sub: Fire and Forget

1.1 The Protocol

Pub/Sub is a channel-based publish-subscribe system. Subscribers register interest in named channels; publishers broadcast to channels; Redis delivers the message to all currently-connected subscribers.

BASH
# Terminal 1: Subscribe to a channel
redis-cli SUBSCRIBE notifications:user:42
# Reading messages... (press Ctrl-C to quit)
# 1) "subscribe"
# 2) "notifications:user:42"
# 3) (integer) 1    ← number of channels subscribed

# Terminal 2: Publish a message
redis-cli PUBLISH notifications:user:42 '{"type":"order_shipped","orderId":"ORD-9871"}'
# → (integer) 1   ← number of subscribers that received the message

# Terminal 1 receives:
# 1) "message"
# 2) "notifications:user:42"
# 3) "{\"type\":\"order_shipped\",\"orderId\":\"ORD-9871\"}"

1.2 The Durability Guarantee: Zero

Redis Pub/Sub has no message queue, no persistence, no acknowledgment, and no backpressure. The PUBLISH return value (the integer count of subscribers that received the message) is the only signal that a message was delivered — and it is 0 for messages published when all subscribers are disconnected.

1.3 Pattern Subscribe

BASH
# Subscribe to all channels matching a glob pattern
redis-cli PSUBSCRIBE "notifications:user:*"
# Matches: notifications:user:42, notifications:user:99, notifications:user:any-value

redis-cli PSUBSCRIBE "cache:*:invalidate"
# Matches: cache:product:invalidate, cache:category:invalidate, etc.

Pattern subscriptions consume more CPU on the server because every published message is matched against all registered patterns. Use channel subscriptions (SUBSCRIBE) for known channel names; reserve PSUBSCRIBE for genuinely dynamic channel spaces.

1.4 Pub/Sub Legitimate Use Cases

Pub/Sub is the right tool when message loss is explicitly acceptable:

Use case Why Pub/Sub is correct
Live dashboard metric updates Stale metrics are refreshed on next publish; missing one update is cosmetically acceptable
RESP3 cache invalidation signals As used in Caching Topologies §3.1 — a missed invalidation results in a stale cache entry, not lost data
Chat presence heartbeats A missed heartbeat means one "online" indicator is stale for one TTL window — acceptable
Real-time game state sync Players with poor connections expect degraded experience; missing frames are normal
Mental Model Check

Pub/Sub is UDP, not TCP. If a subscriber is not connected at the exact moment of publish, the message is gone. If this is unacceptable for your use case, you need Streams.


2. Redis Streams: The Persistent Message Log

Redis Streams (introduced in Redis 5.0) is an append-only log data structure. Unlike Pub/Sub, Streams:

  • Persist entries in the server's memory (and to disk via AOF/RDB)
  • Retain entries even after all consumers have read them (until trimmed)
  • Support consumer groups with delivery acknowledgment and pending entry tracking
  • Guarantee at-least-once delivery through re-delivery of unacknowledged messages

2.1 Stream Internals: Radix Tree of listpack Nodes

Internally, a Stream is stored as a radix tree (rax) where each node contains a listpack of entries sharing the same millisecond timestamp prefix. This is the same listpack encoding from Part 1 — compact, cache-friendly, and extremely space-efficient for time-series data where entries arrive in bursts at the same millisecond.

2.2 Producing to a Stream

TYPESCRIPT
// ✅ XADD — Append an entry to a stream
const entryId = await redis.xadd(
  'orders:events',              // Stream key
  '*',                          // Auto-generate entry ID: {milliseconds}-{sequence}
  'orderId',  'ORD-9871',
  'userId',   '42',
  'type',     'order_placed',
  'amount',   '149.99'
)
// entryId: "1725350400000-0"  ← {millisecond_timestamp}-{sequence_within_ms}

// Bounded stream: auto-trim to 100,000 entries (MAXLEN with ~ for approximate)
await redis.xadd('orders:events', { MAXLEN: ['~', 100000] }, '*',
  'orderId', 'ORD-9872', 'type', 'order_shipped'
)

The entry ID format {milliseconds}-{sequence} is significant: entries are always ordered by time, and within the same millisecond, by sequence number. This guarantees strict ordering within a single stream — a guarantee Kafka provides only within a partition.

2.3 Reading Without Consumer Groups

BASH
# Read all entries from the beginning
redis-cli XRANGE orders:events - +
# 1) 1) "1725350400000-0"
#    2) 1) "orderId" 2) "ORD-9871" 3) "userId" 4) "42" ...

# Read the last 10 entries
redis-cli XREVRANGE orders:events + - COUNT 10

# Blocking read: wait for new entries (like BLPOP for Streams)
redis-cli XREAD BLOCK 5000 STREAMS orders:events $
# BLOCK 5000 = block for up to 5000ms waiting for new entries
# $           = only return entries added AFTER this command started

3. Consumer Groups: At-Least-Once Delivery

Consumer groups are the mechanism that transforms Streams from a broadcast log into a work queue with delivery guarantees.

3.1 The Pending Entry List (PEL)

When a consumer reads an entry via XREADGROUP, Redis adds that entry to the Pending Entry List (PEL) for that consumer. The entry remains in the PEL until the consumer explicitly acknowledges it with XACK. If the consumer crashes before XACK, the entry stays in the PEL and is re-delivered to another consumer.

3.2 Creating and Reading from a Consumer Group

TYPESCRIPT
// Step 1: Create a consumer group (run once at startup)
// '$' = start from new entries only (use '0' to process existing entries from the beginning)
await redis.xgroup('CREATE', 'orders:events', 'processors', '$', 'MKSTREAM')
// MKSTREAM: create the stream if it doesn't exist

// Step 2: Read up to 10 pending entries as consumer-1
const entries = await redis.xreadgroup(
  'GROUP', 'processors', 'consumer-1',
  'COUNT', 10,
  'BLOCK', 5000,        // Block up to 5s waiting for new entries
  'STREAMS', 'orders:events', '>'
  // '>' means: give me entries not yet delivered to any consumer in this group
)
// entries: [['orders:events', [['1725350400000-0', ['orderId', 'ORD-9871', ...]]]]]

// Step 3: Process and acknowledge
for (const [streamKey, messages] of entries) {
  for (const [entryId, fields] of messages) {
    await processOrder(fields)      // Your business logic
    await redis.xack('orders:events', 'processors', entryId)
    // Remove from PEL — entry will not be re-delivered
  }
}

3.3 XPENDING: The First Diagnostic

BASH
# Summary: how many entries are pending in the consumer group?
redis-cli XPENDING orders:events processors - + 10
# 1) 1) "1725350400001-0"    ← entry ID
#    2) "consumer-2"          ← which consumer it was delivered to
#    3) (integer) 45231       ← idle time in ms (45 seconds — consumer likely crashed)
#    4) (integer) 1           ← delivery count (delivered once)

# Full summary
redis-cli XPENDING orders:events processors
# 1) (integer) 1        ← 1 entry pending in this group
# 2) "1725350400001-0"  ← min pending entry ID
# 3) "1725350400001-0"  ← max pending entry ID
# 4) 1) 1) "consumer-2" 2) "1"   ← consumer-2 has 1 pending entry
Pro Tip & Optimization

XPENDING is the first command to run when debugging a stalled consumer group. A growing PEL idle time indicates a crashed or slow consumer. A delivery count > 1 indicates an entry that has been delivered and failed processing multiple times — a candidate for a Dead Letter Queue.

3.4 XAUTOCLAIM: Automatic Orphan Recovery (Redis 6.2+)

XCLAIM (manual) and XAUTOCLAIM (automatic, Redis 6.2+) allow a healthy consumer to claim ownership of orphaned pending entries.

TYPESCRIPT
// XAUTOCLAIM: claim entries from consumer-2 that have been idle > 30 seconds
const [nextCursor, claimedEntries] = await redis.xautoclaim(
  'orders:events',    // Stream key
  'processors',       // Consumer group
  'consumer-3',       // New owner
  30000,              // Minimum idle time in ms
  '0-0',             // Start from the beginning of the PEL
  'COUNT', 10         // Claim at most 10 entries
)
// Process and ack claimed entries — same as normal XREADGROUP processing

3.5 Dead Letter Queue Pattern

An entry that fails processing repeatedly should not loop forever. Track delivery count and route poison messages to a DLQ stream:

TYPESCRIPT
const MAX_DELIVERY_ATTEMPTS = 3

async function processWithDLQ(entryId: string, fields: string[], deliveryCount: number) {
  if (deliveryCount > MAX_DELIVERY_ATTEMPTS) {
    // Route to DLQ stream for manual inspection
    await redis.xadd('orders:events:dlq', '*',
      'originalId', entryId,
      'failureReason', 'max_retries_exceeded',
      'deliveryCount', String(deliveryCount),
      ...fields
    )
    // Acknowledge the original to remove from PEL — it's in the DLQ now
    await redis.xack('orders:events', 'processors', entryId)
    return
  }
  await processOrder(fields)
  await redis.xack('orders:events', 'processors', entryId)
}

4. Stream Trimming and Bounded Streams

Without trimming, a Stream grows indefinitely and consumes unbounded memory. Redis provides two trimming strategies:

BASH
# MAXLEN — trim to a maximum number of entries
redis-cli XADD orders:events MAXLEN 100000 * orderId ORD-9873 type order_placed
# '~' prefix = approximate trim (Redis may keep slightly more for efficiency)
redis-cli XADD orders:events 'MAXLEN' '~' 100000 '*' orderId ORD-9874 type order_shipped

# MINID — trim entries with IDs older than a given ID (time-based retention)
redis-cli XTRIM orders:events MINID 1725264000000  # Remove entries older than 24 hours ago

# Explicit trim (separate from XADD)
redis-cli XTRIM orders:events MAXLEN ~ 50000
Strategy Use case Trade-off
MAXLEN N Bounded-size queues (work queues, event buses) Entry count bounded; time range unbounded
MINID timestamp Time-series data with retention policies (24h, 7d) Time range bounded; entry count unbounded
No trimming Audit logs requiring complete history Unbounded memory growth
Crucial Requirement

Always pair Streams with a trimming strategy. An untrimmed Stream that receives 10,000 events/sec will consume approximately 1GB of RAM per hour (assuming ~30 bytes per entry). At scale, an untrimmed Stream becomes the primary Redis OOM cause.


5. Streams vs. Pub/Sub vs. Kafka

Concern Pub/Sub Redis Streams Apache Kafka
Message persistence ❌ None ✅ In-memory + AOF/RDB ✅ Disk (log segments)
Message retention ❌ Zero — delivered or lost ✅ Until trimmed ✅ Configurable (days/weeks)
Delivery guarantee ❌ At-most-once ✅ At-least-once (with groups + XACK) ✅ At-least-once / exactly-once
Consumer groups ❌ No ✅ Yes (competitive consumption) ✅ Yes (partition-based)
Ordering N/A (broadcast) ✅ Strict per-stream ✅ Strict per-partition
Throughput ceiling ~1M msgs/s (bounded by Redis CPU) ~100K–500K msgs/s Millions/s (distributed)
Multi-stream fan-out ✅ Native (PSUBSCRIBE) Manual (read multiple streams) ✅ Native (topic partitions)
Operational complexity Minimal Low High
When to use Ephemeral broadcasts Durable work queues < ~500K/s High-throughput event streaming
Mental Model Check

Redis Streams is a Kafka replacement for workloads below ~100K–500K events per second on a single stream. Above that: Kafka's partition model is required for horizontal throughput. The operational cost difference is enormous — Redis Streams requires no ZooKeeper/KRaft, no partition assignment, no consumer lag tooling. For a 10-service startup processing 50K order events per second, Streams is the right choice. For a 500-service platform processing 5M events per second, it is not.

5.1 LMPOP / BLMPOP: Modern List-Based Queuing (Redis 7.0+)

For simple FIFO work queues that do not require consumer groups or delivery guarantees, LMPOP and BLMPOP (Redis 7.0+) provide a cleaner API than the older BLPOP:

BASH
# BLMPOP: blocking pop from the left of one or more lists
redis-cli BLMPOP 5 2 jobs:high jobs:normal LEFT COUNT 5
# Waits up to 5 seconds
# Checks 2 lists: jobs:high first (priority), then jobs:normal
# Pops from LEFT, returns up to 5 elements
# Returns from the first non-empty list

# vs. old BLPOP (single list, single element only)
redis-cli BLPOP jobs:high 5

BLMPOP is preferred over BLPOP for new code — it supports multiple lists in priority order and returns multiple elements in one call.


Summary

Concept Rule
Pub/Sub durability Zero — a disconnected subscriber loses every message published during its absence, permanently. Use only when message loss is explicitly acceptable.
Streams durability Persisted in memory (and to disk with AOF) — entries survive reconnection, consumer crashes, and Redis restarts (with persistence enabled).
Consumer groups Provide at-least-once delivery — an entry stays in the PEL until XACKed, and is re-delivered to another consumer on timeout.
XPENDING First diagnostic for stuck consumers — reveals idle entries in the PEL with their consumer assignment and delivery count.
XAUTOCLAIM Automatic orphan recovery (Redis 6.2+) — claims entries idle longer than a threshold from crashed consumers.
DLQ pattern Route entries with delivery count > N to a {stream}:dlq stream and XACK the original — prevents infinite retry loops on poison messages.
Stream trimming Always trim with MAXLEN ~N or MINID timestamp. Untrimmed Streams become the primary Redis OOM cause under sustained write load.
Streams vs Kafka Streams is correct below ~100K–500K events/s on a single stream. Above that, Kafka's partition model is required for throughput.

What's Next

In Part 6: Lua Scripting, MULTI/EXEC & True Atomicity in Redis, we deconstruct the most dangerous Redis misconception: that MULTI/EXEC provides rollback. It does not. Runtime errors inside a transaction partially apply the batch — leaving the dataset in an indeterminate state. Lua scripts provide true all-or-nothing atomicity but block the event loop for their entire duration, making script duration the critical production constraint.

Research & Synthesis Note

This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.

#Redis#Redis Streams#Pub/Sub#Messaging#Event-Driven Architecture
Siddhant Deval

Written by Siddhant Deval

Senior Full-Stack Engineer building high-scale architectures, browser performance engineering systems, and SaaS platforms.