Redis Pub/Sub vs. Streams: Choosing the Right Messaging Primitive
Redis Pub/Sub is a stateless, fire-and-forget broadcast bus — any subscriber that disconnects loses every message published during its absence, permanently. Redis Streams is a persistent, ordered log with consumer groups, delivery acknowledgment, and a pending entry list that tracks every unacknowledged message. This article maps both primitives to their correct use cases and shows when Streams replace Kafka.
Redis Pub/Sub vs. Streams: Choosing the Right Messaging Primitive
Redis is not a cache you bolt onto a slow database — it is a data structure server with a precisely bounded contract: sub-millisecond latency, in-memory semantics, and optional persistence. Nowhere is this contract more consequential than in Redis's messaging primitives. Pub/Sub and Streams both allow producers to send messages and consumers to receive them. But their durability contracts are as different as TCP and UDP. Pub/Sub is a stateless broadcast bus with zero durability — the server makes no attempt to persist messages, and any subscriber that disconnects for any reason loses every message published during its absence, permanently, with no error and no log entry. Streams are a persistent, ordered log with delivery acknowledgment. Choosing the wrong primitive does not produce a deployment error; it produces a production incident weeks after launch when users report missing notifications.
This is Part 5 of the Redis Mastery series. It can be read independently after Part 4.
On-ramp: This article uses two concepts from earlier parts. From Part 1: Streams store entries in listpack-packed radix tree nodes — the same compact encoding used by Hashes and Sorted Sets at small sizes. From Part 2: Streams are persisted via AOF/RDB like any other Redis data structure — enabling AOF gives you durable Streams; without persistence, Stream entries are lost on restart like all other keys.
1. Pub/Sub: Fire and Forget
1.1 The Protocol
Pub/Sub is a channel-based publish-subscribe system. Subscribers register interest in named channels; publishers broadcast to channels; Redis delivers the message to all currently-connected subscribers.
1.2 The Durability Guarantee: Zero
Redis Pub/Sub has no message queue, no persistence, no acknowledgment, and no backpressure. The PUBLISH return value (the integer count of subscribers that received the message) is the only signal that a message was delivered — and it is 0 for messages published when all subscribers are disconnected.
1.3 Pattern Subscribe
Pattern subscriptions consume more CPU on the server because every published message is matched against all registered patterns. Use channel subscriptions (SUBSCRIBE) for known channel names; reserve PSUBSCRIBE for genuinely dynamic channel spaces.
1.4 Pub/Sub Legitimate Use Cases
Pub/Sub is the right tool when message loss is explicitly acceptable:
| Use case | Why Pub/Sub is correct |
|---|---|
| Live dashboard metric updates | Stale metrics are refreshed on next publish; missing one update is cosmetically acceptable |
| RESP3 cache invalidation signals | As used in Caching Topologies §3.1 — a missed invalidation results in a stale cache entry, not lost data |
| Chat presence heartbeats | A missed heartbeat means one "online" indicator is stale for one TTL window — acceptable |
| Real-time game state sync | Players with poor connections expect degraded experience; missing frames are normal |
Pub/Sub is UDP, not TCP. If a subscriber is not connected at the exact moment of publish, the message is gone. If this is unacceptable for your use case, you need Streams.
2. Redis Streams: The Persistent Message Log
Redis Streams (introduced in Redis 5.0) is an append-only log data structure. Unlike Pub/Sub, Streams:
- Persist entries in the server's memory (and to disk via AOF/RDB)
- Retain entries even after all consumers have read them (until trimmed)
- Support consumer groups with delivery acknowledgment and pending entry tracking
- Guarantee at-least-once delivery through re-delivery of unacknowledged messages
2.1 Stream Internals: Radix Tree of listpack Nodes
Internally, a Stream is stored as a radix tree (rax) where each node contains a listpack of entries sharing the same millisecond timestamp prefix. This is the same listpack encoding from Part 1 — compact, cache-friendly, and extremely space-efficient for time-series data where entries arrive in bursts at the same millisecond.
2.2 Producing to a Stream
The entry ID format {milliseconds}-{sequence} is significant: entries are always ordered by time, and within the same millisecond, by sequence number. This guarantees strict ordering within a single stream — a guarantee Kafka provides only within a partition.
2.3 Reading Without Consumer Groups
3. Consumer Groups: At-Least-Once Delivery
Consumer groups are the mechanism that transforms Streams from a broadcast log into a work queue with delivery guarantees.
3.1 The Pending Entry List (PEL)
When a consumer reads an entry via XREADGROUP, Redis adds that entry to the Pending Entry List (PEL) for that consumer. The entry remains in the PEL until the consumer explicitly acknowledges it with XACK. If the consumer crashes before XACK, the entry stays in the PEL and is re-delivered to another consumer.
3.2 Creating and Reading from a Consumer Group
3.3 XPENDING: The First Diagnostic
XPENDING is the first command to run when debugging a stalled consumer group. A growing PEL idle time indicates a crashed or slow consumer. A delivery count > 1 indicates an entry that has been delivered and failed processing multiple times — a candidate for a Dead Letter Queue.
3.4 XAUTOCLAIM: Automatic Orphan Recovery (Redis 6.2+)
XCLAIM (manual) and XAUTOCLAIM (automatic, Redis 6.2+) allow a healthy consumer to claim ownership of orphaned pending entries.
3.5 Dead Letter Queue Pattern
An entry that fails processing repeatedly should not loop forever. Track delivery count and route poison messages to a DLQ stream:
4. Stream Trimming and Bounded Streams
Without trimming, a Stream grows indefinitely and consumes unbounded memory. Redis provides two trimming strategies:
| Strategy | Use case | Trade-off |
|---|---|---|
MAXLEN N |
Bounded-size queues (work queues, event buses) | Entry count bounded; time range unbounded |
MINID timestamp |
Time-series data with retention policies (24h, 7d) | Time range bounded; entry count unbounded |
| No trimming | Audit logs requiring complete history | Unbounded memory growth |
Always pair Streams with a trimming strategy. An untrimmed Stream that receives 10,000 events/sec will consume approximately 1GB of RAM per hour (assuming ~30 bytes per entry). At scale, an untrimmed Stream becomes the primary Redis OOM cause.
5. Streams vs. Pub/Sub vs. Kafka
| Concern | Pub/Sub | Redis Streams | Apache Kafka |
|---|---|---|---|
| Message persistence | ❌ None | ✅ In-memory + AOF/RDB | ✅ Disk (log segments) |
| Message retention | ❌ Zero — delivered or lost | ✅ Until trimmed | ✅ Configurable (days/weeks) |
| Delivery guarantee | ❌ At-most-once | ✅ At-least-once (with groups + XACK) | ✅ At-least-once / exactly-once |
| Consumer groups | ❌ No | ✅ Yes (competitive consumption) | ✅ Yes (partition-based) |
| Ordering | N/A (broadcast) | ✅ Strict per-stream | ✅ Strict per-partition |
| Throughput ceiling | ~1M msgs/s (bounded by Redis CPU) | ~100K–500K msgs/s | Millions/s (distributed) |
| Multi-stream fan-out | ✅ Native (PSUBSCRIBE) | Manual (read multiple streams) | ✅ Native (topic partitions) |
| Operational complexity | Minimal | Low | High |
| When to use | Ephemeral broadcasts | Durable work queues < ~500K/s | High-throughput event streaming |
Redis Streams is a Kafka replacement for workloads below ~100K–500K events per second on a single stream. Above that: Kafka's partition model is required for horizontal throughput. The operational cost difference is enormous — Redis Streams requires no ZooKeeper/KRaft, no partition assignment, no consumer lag tooling. For a 10-service startup processing 50K order events per second, Streams is the right choice. For a 500-service platform processing 5M events per second, it is not.
5.1 LMPOP / BLMPOP: Modern List-Based Queuing (Redis 7.0+)
For simple FIFO work queues that do not require consumer groups or delivery guarantees, LMPOP and BLMPOP (Redis 7.0+) provide a cleaner API than the older BLPOP:
BLMPOP is preferred over BLPOP for new code — it supports multiple lists in priority order and returns multiple elements in one call.
Summary
| Concept | Rule |
|---|---|
| Pub/Sub durability | Zero — a disconnected subscriber loses every message published during its absence, permanently. Use only when message loss is explicitly acceptable. |
| Streams durability | Persisted in memory (and to disk with AOF) — entries survive reconnection, consumer crashes, and Redis restarts (with persistence enabled). |
| Consumer groups | Provide at-least-once delivery — an entry stays in the PEL until XACKed, and is re-delivered to another consumer on timeout. |
| XPENDING | First diagnostic for stuck consumers — reveals idle entries in the PEL with their consumer assignment and delivery count. |
| XAUTOCLAIM | Automatic orphan recovery (Redis 6.2+) — claims entries idle longer than a threshold from crashed consumers. |
| DLQ pattern | Route entries with delivery count > N to a {stream}:dlq stream and XACK the original — prevents infinite retry loops on poison messages. |
| Stream trimming | Always trim with MAXLEN ~N or MINID timestamp. Untrimmed Streams become the primary Redis OOM cause under sustained write load. |
| Streams vs Kafka | Streams is correct below ~100K–500K events/s on a single stream. Above that, Kafka's partition model is required for throughput. |
What's Next
In Part 6: Lua Scripting, MULTI/EXEC & True Atomicity in Redis, we deconstruct the most dangerous Redis misconception: that
MULTI/EXECprovides rollback. It does not. Runtime errors inside a transaction partially apply the batch — leaving the dataset in an indeterminate state. Lua scripts provide true all-or-nothing atomicity but block the event loop for their entire duration, making script duration the critical production constraint.
This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.