Siddhant Deval
Siddhant Deval
backend13 min read

Priority Queues, Delayed Messages, and Scheduled Delivery

Priority queues and delayed delivery are common application requirements that each broker solves differently — RabbitMQ has native priority support, SQS has built-in message delay, and Kafka has no native delay mechanism. This article maps each broker's approach, implements RabbitMQ x-max-priority and delayed exchange, SQS DelaySeconds, and the Kafka timestamp-hold workaround.

Priority Queues, Delayed Messages, and Scheduled Delivery

The email notification service processes three types of messages from a single queue: password reset requests (must deliver in seconds), weekly digest emails (can wait hours), and marketing campaigns (can wait days). With a simple FIFO queue, a burst of 50,000 campaign emails blocks password reset messages for hours. The product team files a P0 incident. The fix is not a faster queue — it is a priority mechanism that lets the broker skip low-priority messages when high-priority ones are waiting.

Separately, the subscription renewal service needs to send a reminder email exactly 7 days before a subscription expires. The instinct is setTimeout(sendEmail, 7 * 24 * 3600 * 1000) — which crashes on every process restart, silently losing all pending timers.

Both problems — prioritisation and time-based delivery — require the messaging layer, not application code, to hold the scheduling state.

Architectural Note

Series positioning: This is Part 8 of Distributed Messaging Systems. It covers the time and priority dimensions of messaging that the previous parts left implicit. The Redis Streams pattern for durable scheduled jobs builds on Redis Pub/Sub, Streams, and Consumer Groups.


1. Priority Queues

1.1 RabbitMQ: x-max-priority

RabbitMQ's native priority queue maintains up to N separate internal sub-queues — one per priority level. The broker delivers the highest-priority available message to the consumer, regardless of arrival order:

TYPESCRIPT
// ✅ Declare a priority queue — x-max-priority must be set at declaration time
await channel.assertQueue('notifications', {
  durable: true,
  arguments: {
    'x-max-priority': 5,            // 5 priority levels: 0 (lowest) to 5 (highest)
    'x-queue-type':   'classic',    // Priority queues are classic only — NOT quorum
  }
})

// Publish with explicit priority
channel.sendToQueue('notifications', Buffer.from(JSON.stringify({
  type:   'password-reset',
  userId: 'u-42',
  token:  'abc123',
})), {
  priority:   5,      // highest — delivered before any lower-priority messages
  persistent: true,
})

channel.sendToQueue('notifications', Buffer.from(JSON.stringify({
  type:   'weekly-digest',
  userId: 'u-42',
})), {
  priority:   2,      // delivered after priority-5 messages are drained
  persistent: true,
})

channel.sendToQueue('notifications', Buffer.from(JSON.stringify({
  type:   'marketing-campaign',
  campaignId: 'CAMP-2026-Q4',
})), {
  priority:   0,      // lowest — processed only when higher-priority queue is empty
  persistent: true,
})
Performance / Safety Warning

RabbitMQ priority queues consume memory proportional to x-max-priority × queue depth. Each priority level maintains a separate internal heap. Setting x-max-priority: 255 on a queue with 100,000 messages creates 255 internal heaps — the broker's memory usage can spike unexpectedly. Cap at 5 in production. If you need fine-grained prioritisation, use 3–5 separate physical queues instead.

TYPESCRIPT
// ✅ Better pattern for most use cases: separate physical queues per priority tier
// Worker processes high-priority queue first, falls back to low when empty
async function smartConsumer(): Promise<void> {
  // Check high-priority first
  const highMsg = await channel.get('notifications.high', { noAck: false })
  if (highMsg) {
    await processNotification(JSON.parse(highMsg.content.toString()))
    channel.ack(highMsg)
    return
  }
  // Fall back to low-priority
  const lowMsg = await channel.get('notifications.low', { noAck: false })
  if (lowMsg) {
    await processNotification(JSON.parse(lowMsg.content.toString()))
    channel.ack(lowMsg)
  }
}

1.2 Kafka: Priority via Dedicated Topics

Kafka has no native priority mechanism — the commit log is strictly ordered within a partition. The correct pattern is dedicated topics per priority tier, with consumers polling the high-priority topic more frequently:

TYPESCRIPT
// ✅ Kafka: separate topics + weighted polling
const highConsumer = kafka.consumer({ groupId: 'notifications-high' })
const lowConsumer  = kafka.consumer({ groupId: 'notifications-low'  })

await highConsumer.subscribe({ topic: 'notifications.high' })
await lowConsumer.subscribe({  topic: 'notifications.low'  })

// Poll high-priority 4× for every 1 low-priority poll
let pollCount = 0
setInterval(async () => {
  pollCount++
  if (pollCount % 5 !== 0) {
    // 4 out of 5 cycles: drain high-priority
    await highConsumer.run({ eachMessage: processNotification })
  } else {
    // 1 out of 5 cycles: process low-priority
    await lowConsumer.run({ eachMessage: processNotification })
  }
}, 100)

2. Delayed Delivery

2.1 RabbitMQ: Dead-Letter TTL Pattern

The standard approach without the rabbitmq-delayed-message-exchange plugin uses the dead-letter exchange mechanism: messages are published to a holding queue with a TTL; on expiry, they route via DLX to the actual work queue:

TYPESCRIPT
// ✅ Delayed delivery via TTL + DLX — no plugin required
async function scheduleNotification(
  payload:   object,
  delayMs:   number
): Promise<void> {
  const holdingQueue = `notifications.delay.${delayMs}`

  // Declare a holding queue with this exact delay duration
  await channel.assertQueue(holdingQueue, {
    durable: true,
    arguments: {
      'x-message-ttl':             delayMs,
      'x-dead-letter-exchange':    '',                     // default exchange
      'x-dead-letter-routing-key': 'notifications.ready', // routes here after TTL
      'x-expires':                 delayMs + 60_000,       // auto-delete queue after delay + buffer
    }
  })

  channel.sendToQueue(holdingQueue, Buffer.from(JSON.stringify(payload)), {
    persistent: true,
  })
  // Message sits in holdingQueue for delayMs, then routes to notifications.ready
}

await scheduleNotification({ type: 'reminder', userId: 'u-42' }, 60_000)   // 1 minute
await scheduleNotification({ type: 'reminder', userId: 'u-43' }, 300_000)  // 5 minutes
Pro Tip & Optimization

For precise, arbitrary delays (e.g., "in exactly 7 days"), use the rabbitmq-delayed-message-exchange plugin rather than per-delay holding queues. The plugin supports arbitrary delays per message without creating a new queue per delay bucket. For short delays (< 60 seconds) the TTL pattern is simpler and requires no plugin.

2.2 SQS: DelaySeconds

SQS has built-in message delay — the message is invisible to consumers for DelaySeconds after publishing, then becomes available:

TYPESCRIPT
// ✅ SQS per-message delay — up to 15 minutes (900 seconds)
import { SQSClient, SendMessageCommand } from '@aws-sdk/client-sqs'

const sqs = new SQSClient({ region: 'us-east-1' })

async function scheduleRenewalReminder(
  userId:    string,
  delayDays: number
): Promise<void> {
  const delaySeconds = Math.min(delayDays * 86400, 900) // SQS max: 900s = 15 min

  if (delayDays * 86400 > 900) {
    // For delays > 15 minutes: store in scheduler DB, publish to SQS at correct time
    await schedulerDb.insert({
      userId,
      scheduledAt: new Date(Date.now() + delayDays * 86400 * 1000),
      payload:     { type: 'renewal-reminder', userId },
    })
    return
  }

  await sqs.send(new SendMessageCommand({
    QueueUrl:     process.env.NOTIFICATIONS_QUEUE_URL!,
    MessageBody:  JSON.stringify({ type: 'renewal-reminder', userId }),
    DelaySeconds: delaySeconds,
  }))
}
Broker Max delay Mechanism Requires plugin/workaround
RabbitMQ (TTL+DLX) Unlimited Holding queue per delay bucket No
RabbitMQ (plugin) Unlimited rabbitmq-delayed-message-exchange Yes
SQS 15 minutes DelaySeconds parameter No
Kafka None native Timestamp-hold (starves partition) Workaround only

2.3 Kafka: The Timestamp-Hold Workaround (and Why to Avoid It)

TYPESCRIPT
// ❌ Kafka timestamp-hold — blocks the partition thread, starves other messages
await consumer.run({
  eachMessage: async ({ message, pause }) => {
    const scheduledAt = Number(message.headers?.['scheduled-at'])
    const now = Date.now()

    if (scheduledAt > now) {
      const waitMs = scheduledAt - now
      if (waitMs > 30_000) {
        // More than 30 seconds: pause partition and reschedule check
        const resume = pause()
        setTimeout(resume, Math.min(waitMs, 30_000))
        return
      }
      // Wait inline — blocks entire partition thread
      await new Promise(resolve => setTimeout(resolve, waitMs))
    }
    await processScheduledJob(JSON.parse(message.value!.toString()))
  }
})
Performance / Safety Warning

The timestamp-hold pattern starves the entire partition. Kafka assigns one consumer thread per partition — a sleeping consumer holding a partition blocks all other messages behind it in that partition for the duration of the delay. For delays over 30 seconds, use an external scheduler (BullMQ, Temporal, or a cron-triggered Lambda) that publishes to Kafka at the correct time. Never implement delays > 30 seconds inline in a Kafka consumer.


3. BullMQ: Durable Scheduled Jobs over Redis Streams

For applications that already run Redis, BullMQ provides durable scheduled jobs with arbitrary delays, priority, retries, and concurrency control — without running a separate broker:

TYPESCRIPT
// ✅ BullMQ: durable job scheduling with priority and delay
import { Queue, Worker } from 'bullmq'
import Redis from 'ioredis'

const connection = new Redis({ host: 'redis', port: 6379, maxRetriesPerRequest: null })

const notificationQueue = new Queue('notifications', { connection })

// Priority job (lower number = higher priority in BullMQ)
await notificationQueue.add('password-reset', {
  userId: 'u-42',
  token:  'abc123',
}, {
  priority: 1,     // highest priority
  attempts: 3,
  backoff:  { type: 'exponential', delay: 1000 },
})

// Delayed job: send in 7 days
await notificationQueue.add('renewal-reminder', {
  userId:         'u-43',
  subscriptionId: 'sub-789',
}, {
  delay:    7 * 24 * 3600 * 1000,   // 7 days in ms — stored durably in Redis
  priority: 10,                      // low priority
  attempts: 5,
})

// Repeatable (cron) job
await notificationQueue.add('weekly-digest', {}, {
  repeat: { pattern: '0 8 * * 1' },  // every Monday at 08:00
})

// Worker: processes jobs from the queue
const worker = new Worker('notifications', async (job) => {
  switch (job.name) {
    case 'password-reset':
      await emailService.sendPasswordReset(job.data)
      break
    case 'renewal-reminder':
      await emailService.sendRenewalReminder(job.data)
      break
    case 'weekly-digest':
      await emailService.sendWeeklyDigest()
      break
  }
}, {
  connection,
  concurrency: 10,
})

worker.on('failed', (job, err) => {
  console.error(`Job ${job?.id} failed: ${err.message}`)
})
Crucial Requirement

BullMQ delays are stored in Redis sorted sets (ZSET with scheduled_at as score). Delayed jobs survive Redis restarts if appendonly yes is configured. They do not survive if Redis is used in volatile-only mode — always use Redis with AOF persistence for any BullMQ deployment handling business-critical scheduled jobs.


Summary

Concept Rule
Priority memory cost RabbitMQ priority queues consume memory proportional to x-max-priority × queue depth; cap priority levels at 5 in production to avoid uncontrolled heap growth.
SQS delay limit SQS per-message delay max is 15 minutes; for longer scheduled delivery, use an external scheduler (BullMQ, Temporal) that publishes to SQS at the correct time.
Kafka delay anti-pattern Kafka has no native delay mechanism — the timestamp-based consumer hold pattern works but starves the partition thread; use a dedicated scheduler for any delay > 30 seconds.

What's Next

Part 9: Schema Evolution — Avro, Schema Registry, and Backward/Forward Compatibility tackles the problem that every messaging system eventually faces: how to change the shape of a message without breaking existing producers or consumers. Schema registries, Avro's resolution rules, and the three compatibility modes (backward, forward, full) make schema evolution safe at scale.

Research & Synthesis Note

This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.

#Priority Queue#Delayed Messages#Scheduling#RabbitMQ#SQS#Kafka#Backend
Siddhant Deval

Written by Siddhant Deval

Senior Full-Stack Engineer building high-scale architectures, browser performance engineering systems, and SaaS platforms.