Priority Queues, Delayed Messages, and Scheduled Delivery
Priority queues and delayed delivery are common application requirements that each broker solves differently — RabbitMQ has native priority support, SQS has built-in message delay, and Kafka has no native delay mechanism. This article maps each broker's approach, implements RabbitMQ x-max-priority and delayed exchange, SQS DelaySeconds, and the Kafka timestamp-hold workaround.
Distributed Messaging Systems
Priority Queues, Delayed Messages, and Scheduled Delivery
The email notification service processes three types of messages from a single queue: password reset requests (must deliver in seconds), weekly digest emails (can wait hours), and marketing campaigns (can wait days). With a simple FIFO queue, a burst of 50,000 campaign emails blocks password reset messages for hours. The product team files a P0 incident. The fix is not a faster queue — it is a priority mechanism that lets the broker skip low-priority messages when high-priority ones are waiting.
Separately, the subscription renewal service needs to send a reminder email exactly 7 days before a subscription expires. The instinct is setTimeout(sendEmail, 7 * 24 * 3600 * 1000) — which crashes on every process restart, silently losing all pending timers.
Both problems — prioritisation and time-based delivery — require the messaging layer, not application code, to hold the scheduling state.
Series positioning: This is Part 8 of Distributed Messaging Systems. It covers the time and priority dimensions of messaging that the previous parts left implicit. The Redis Streams pattern for durable scheduled jobs builds on Redis Pub/Sub, Streams, and Consumer Groups.
1. Priority Queues
1.1 RabbitMQ: x-max-priority
RabbitMQ's native priority queue maintains up to N separate internal sub-queues — one per priority level. The broker delivers the highest-priority available message to the consumer, regardless of arrival order:
RabbitMQ priority queues consume memory proportional to x-max-priority × queue depth. Each priority level maintains a separate internal heap. Setting x-max-priority: 255 on a queue with 100,000 messages creates 255 internal heaps — the broker's memory usage can spike unexpectedly. Cap at 5 in production. If you need fine-grained prioritisation, use 3–5 separate physical queues instead.
1.2 Kafka: Priority via Dedicated Topics
Kafka has no native priority mechanism — the commit log is strictly ordered within a partition. The correct pattern is dedicated topics per priority tier, with consumers polling the high-priority topic more frequently:
2. Delayed Delivery
2.1 RabbitMQ: Dead-Letter TTL Pattern
The standard approach without the rabbitmq-delayed-message-exchange plugin uses the dead-letter exchange mechanism: messages are published to a holding queue with a TTL; on expiry, they route via DLX to the actual work queue:
For precise, arbitrary delays (e.g., "in exactly 7 days"), use the rabbitmq-delayed-message-exchange plugin rather than per-delay holding queues. The plugin supports arbitrary delays per message without creating a new queue per delay bucket. For short delays (< 60 seconds) the TTL pattern is simpler and requires no plugin.
2.2 SQS: DelaySeconds
SQS has built-in message delay — the message is invisible to consumers for DelaySeconds after publishing, then becomes available:
| Broker | Max delay | Mechanism | Requires plugin/workaround |
|---|---|---|---|
| RabbitMQ (TTL+DLX) | Unlimited | Holding queue per delay bucket | No |
| RabbitMQ (plugin) | Unlimited | rabbitmq-delayed-message-exchange |
Yes |
| SQS | 15 minutes | DelaySeconds parameter |
No |
| Kafka | None native | Timestamp-hold (starves partition) | Workaround only |
2.3 Kafka: The Timestamp-Hold Workaround (and Why to Avoid It)
The timestamp-hold pattern starves the entire partition. Kafka assigns one consumer thread per partition — a sleeping consumer holding a partition blocks all other messages behind it in that partition for the duration of the delay. For delays over 30 seconds, use an external scheduler (BullMQ, Temporal, or a cron-triggered Lambda) that publishes to Kafka at the correct time. Never implement delays > 30 seconds inline in a Kafka consumer.
3. BullMQ: Durable Scheduled Jobs over Redis Streams
For applications that already run Redis, BullMQ provides durable scheduled jobs with arbitrary delays, priority, retries, and concurrency control — without running a separate broker:
BullMQ delays are stored in Redis sorted sets (ZSET with scheduled_at as score). Delayed jobs survive Redis restarts if appendonly yes is configured. They do not survive if Redis is used in volatile-only mode — always use Redis with AOF persistence for any BullMQ deployment handling business-critical scheduled jobs.
Summary
| Concept | Rule |
|---|---|
| Priority memory cost | RabbitMQ priority queues consume memory proportional to x-max-priority × queue depth; cap priority levels at 5 in production to avoid uncontrolled heap growth. |
| SQS delay limit | SQS per-message delay max is 15 minutes; for longer scheduled delivery, use an external scheduler (BullMQ, Temporal) that publishes to SQS at the correct time. |
| Kafka delay anti-pattern | Kafka has no native delay mechanism — the timestamp-based consumer hold pattern works but starves the partition thread; use a dedicated scheduler for any delay > 30 seconds. |
What's Next
Part 9: Schema Evolution — Avro, Schema Registry, and Backward/Forward Compatibility tackles the problem that every messaging system eventually faces: how to change the shape of a message without breaking existing producers or consumers. Schema registries, Avro's resolution rules, and the three compatibility modes (backward, forward, full) make schema evolution safe at scale.
This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.