Siddhant Deval
Siddhant Deval
backend12 min read

Request-Reply over Messaging: Async RPC, Correlation IDs, and Inbox Topics

Engineers migrating from HTTP to async messaging always need request-reply semantics — but the naive shared response queue pattern introduces race conditions and message bleed between callers. This article implements the correct per-caller inbox pattern with correlation IDs and timeout handling for both RabbitMQ reply-to and Kafka reply topics.

Request-Reply over Messaging: Async RPC, Correlation IDs, and Inbox Topics

The fraud scoring service is deployed as a Kafka consumer — it processes high-volume transaction events and has its own autoscaling, independent deployment cycle, and its own DLQ. The checkout service needs fraud scores synchronously before confirming an order. The instinct is to call the fraud service over HTTP, but the fraud service team does not want to run an HTTP server alongside their consumer — it doubles their operational surface and bypasses their existing queue-based backpressure. The checkout team proposes a shared fraud.responses Kafka topic. Three weeks later, under load, two checkout instances start consuming each other's fraud responses — instance A's request gets consumed by instance B, which times out waiting for a response that already arrived.

The shared response topic is the wrong pattern. The correct one is a per-caller inbox — an isolated consumption channel that guarantees only the caller who issued the request will receive its response.

Architectural Note

Series positioning: This is Part 7 of Distributed Messaging Systems. The prerequisite mental model for request-reply semantics is RESTful API Design: Resources, HTTP Methods, and Status Codes — this article assumes familiarity with synchronous RPC and focuses on when and how to implement RPC semantics over async messaging. The decision of whether to use this pattern is as important as how.


1. When Request-Reply over Messaging Is Justified

1.1 The Default Answer: Use HTTP/gRPC

TYPESCRIPT
// ✅ Default: direct HTTP call — lowest latency, simplest failure model
const score = await fraudClient.score({ transactionId, amount, userId })
// Round-trip: ~5–20ms over HTTP/2 gRPC
// Failure: immediate 4xx/5xx with structured error
// No broker dependency, no correlation ID, no timeout management

// ❌ Request-reply over messaging adds at least 2× broker latency
// publish → broker → consumer → publish response → broker → caller
// Round-trip: ~20–100ms minimum (2 broker round-trips)
// Adds correlation ID management, timeout handling, orphaned response cleanup
Crucial Requirement

Request-reply over messaging always adds complexity and latency relative to direct HTTP. Choose it only when at least one of these is true: (1) the responder must queue incoming requests for backpressure, (2) the caller is already async and does not hold a synchronous HTTP connection open, (3) the responder team refuses to run an HTTP server, (4) the request fan-out is massive and queue-based load levelling is required.

1.2 The Decision Matrix

Criterion Use HTTP/gRPC Use Request-Reply over Messaging
Latency requirement < 50ms p99 > 50ms acceptable
Responder deployment model HTTP server Pure consumer (no HTTP surface)
Caller model Synchronous web request Async event processor
Request volume Predictable Bursty — needs queue buffering
Backpressure Not needed Responder must control intake rate
Response fan-out One caller, one response One request, multiple partial responses

2. The Broken Pattern: Shared Response Topic

2.1 Why a Shared Topic Bleeds Responses

TYPESCRIPT
// ❌ Shared response topic — multiple callers consume each other's responses
const correlationId = uuid()

// Caller A publishes request
await producer.send({
  topic: 'fraud.requests',
  messages: [{ value: JSON.stringify({ correlationId, transactionId: 'TXN-1' }) }]
})

// Caller A and Caller B both subscribe to the same fraud.responses topic
// They are in the SAME consumer group — Kafka assigns each partition to ONE consumer
// Caller A's response may be delivered to Caller B

const consumer = kafka.consumer({ groupId: 'fraud-response-consumers' }) // shared!
await consumer.subscribe({ topic: 'fraud.responses' })
await consumer.run({
  eachMessage: async ({ message }) => {
    const { correlationId: id, score } = JSON.parse(message.value!.toString())
    // Is this OUR response? Maybe. Maybe it belongs to another caller instance.
    const pending = pendingRequests.get(id)
    if (pending) pending.resolve(score)
    // If not found: silently dropped — other caller is stuck waiting
  }
})

The race condition: under concurrent load, Kafka's partition assignment determines which consumer instance receives which message. A shared consumer group cannot guarantee that the caller who published a request is the same instance that consumes the response.


3. The Correct Pattern: Per-Caller Inbox

3.1 RabbitMQ: reply-to Pseudo-Queue

RabbitMQ has built-in support for per-caller inbox via the replyTo property and amq.rabbitmq.reply-to pseudo-queue — an exclusive, auto-delete, server-named queue allocated per connection:

TYPESCRIPT
// ✅ RabbitMQ Direct Reply-To — broker manages per-caller queue lifecycle
import amqplib from 'amqplib'

const connection = await amqplib.connect('amqp://rabbitmq:5672')
const channel    = await connection.createChannel()

class FraudRpcClient {
  private pending = new Map<string, {
    resolve: (score: number) => void
    reject:  (err: Error)   => void
    timer:   NodeJS.Timeout
  }>()

  async init(): Promise<void> {
    // Subscribe to the pseudo-queue — only THIS connection receives its own replies
    await channel.consume(
      'amq.rabbitmq.reply-to',
      (msg) => {
        if (!msg) return
        const { correlationId, score } = JSON.parse(msg.content.toString())
        const pending = this.pending.get(correlationId)
        if (!pending) return   // timed out or already resolved
        clearTimeout(pending.timer)
        this.pending.delete(correlationId)
        pending.resolve(score)
      },
      { noAck: true }   // reply-to pseudo-queue requires noAck
    )
  }

  async score(transactionId: string, timeoutMs = 5000): Promise<number> {
    const correlationId = crypto.randomUUID()

    return new Promise((resolve, reject) => {
      const timer = setTimeout(() => {
        this.pending.delete(correlationId)
        reject(new Error(`Fraud score timeout after ${timeoutMs}ms [corrId=${correlationId}]`))
      }, timeoutMs)

      this.pending.set(correlationId, { resolve, reject, timer })

      // correlationId set by caller — echoed verbatim by responder
      channel.sendToQueue('fraud.requests', Buffer.from(JSON.stringify({ transactionId })), {
        correlationId,
        replyTo:    'amq.rabbitmq.reply-to',   // broker routes response back to this queue
        persistent: true,
      })
    })
  }
}
TYPESCRIPT
// ✅ Fraud service responder — echoes correlationId and replyTo
await channel.prefetch(20)
await channel.consume('fraud.requests', async (msg) => {
  if (!msg) return
  const { transactionId } = JSON.parse(msg.content.toString())

  const score = await fraudModel.evaluate(transactionId)

  // CRITICAL: echo correlationId from the request, never generate a new one
  channel.sendToQueue(
    msg.properties.replyTo,   // routes directly to caller's pseudo-queue
    Buffer.from(JSON.stringify({
      correlationId: msg.properties.correlationId,  // verbatim echo
      score,
    })),
    { correlationId: msg.properties.correlationId }
  )
  channel.ack(msg)
})

3.2 Kafka: Per-Caller Reply Topic

Kafka has no built-in reply-to mechanism. The correct pattern is a per-caller inbox topic — each caller service has its own topic, partitioned for parallelism, and subscribes to it in its own consumer group:

TYPESCRIPT
// ✅ Kafka per-caller inbox — caller has its own topic, no shared consumer group
const CALLER_REPLY_TOPIC = `fraud.responses.checkout-service`   // unique per service

// Caller: subscribes to its own reply topic in its own group
const replyConsumer = kafka.consumer({ groupId: 'checkout-fraud-reply' })
await replyConsumer.subscribe({ topic: CALLER_REPLY_TOPIC })

const pending = new Map<string, {
  resolve: (score: number) => void
  reject:  (err: Error)   => void
  timer:   NodeJS.Timeout
}>()

await replyConsumer.run({
  eachMessage: async ({ message }) => {
    const { correlationId, score } = JSON.parse(message.value!.toString())
    const req = pending.get(correlationId)
    if (!req) return
    clearTimeout(req.timer)
    pending.delete(correlationId)
    req.resolve(score)
  }
})

// Caller: publishes request with replyTopic header
async function requestFraudScore(transactionId: string, timeoutMs = 5000): Promise<number> {
  const correlationId = crypto.randomUUID()

  return new Promise((resolve, reject) => {
    const timer = setTimeout(() => {
      pending.delete(correlationId)
      reject(new Error(`Timeout [corrId=${correlationId}]`))
    }, timeoutMs)

    pending.set(correlationId, { resolve, reject, timer })

    producer.send({
      topic: 'fraud.requests',
      messages: [{
        key:   correlationId,
        value: JSON.stringify({ correlationId, transactionId }),
        headers: {
          'reply-topic': CALLER_REPLY_TOPIC,   // responder reads this header
          'correlation-id': correlationId,
        },
      }]
    })
  })
}
TYPESCRIPT
// ✅ Fraud responder: routes to reply-topic from request headers
await consumer.run({
  eachMessage: async ({ message }) => {
    const { correlationId, transactionId } = JSON.parse(message.value!.toString())
    const replyTopic    = message.headers?.['reply-topic']?.toString()
    const correlationId = message.headers?.['correlation-id']?.toString()

    if (!replyTopic || !correlationId) {
      // Fire-and-forget request — no reply expected
      await processEvent(transactionId)
      return
    }

    const score = await fraudModel.evaluate(transactionId)

    await producer.send({
      topic: replyTopic,
      messages: [{
        key:     correlationId,   // same key → same partition → ordered replies per caller
        value:   JSON.stringify({ correlationId, score }),
        headers: { 'correlation-id': correlationId },
      }]
    })
  }
})
Performance / Safety Warning

Never let the responder generate the correlation-id. The correlation-id must be set by the caller and echoed verbatim. A responder-generated ID breaks the lookup — the caller's pending map keyed on the original UUID will never match the responder's new UUID.


4. Timeout Handling and Orphaned Responses

Every request-reply implementation over messaging must handle the timeout case explicitly:

TYPESCRIPT
// ✅ Timeout contract: caller cleans up pending state on expiry
const timer = setTimeout(() => {
  pending.delete(correlationId)
  reject(new Error(`Timeout after ${timeoutMs}ms`))
  // The response may still arrive after the timeout:
  // - It will be consumed from the reply topic/queue
  // - The pending.get(correlationId) lookup will return undefined
  // - The response is silently discarded — no memory leak, no crash
}, timeoutMs)

// ✅ Cleanup on graceful shutdown
process.on('SIGTERM', () => {
  for (const [id, req] of pending.entries()) {
    clearTimeout(req.timer)
    req.reject(new Error('Service shutting down'))
    pending.delete(id)
  }
})
Pro Tip & Optimization

Set the caller's timeout to request TTL + expected responder processing time + one round-trip broker latency. If the fraud model takes 200ms and broker round-trip is 10ms, set timeoutMs = 500ms for a 2× safety margin. Publish the request with x-message-ttl: 500 (RabbitMQ) or set a short message retention on the request topic — an unanswered request past its timeout is a zombie that will be processed by the responder for no benefit.


Summary

Concept Rule
Shared queue race condition A shared response queue is a race condition — two callers will occasionally consume each other's responses; always use per-caller inbox topics or RabbitMQ's reply-to pseudo-queue.
Correlation ID ownership correlation-id must be set by the caller and echoed verbatim by the responder — never let the responder generate a new ID.
Latency trade-off Request-reply over messaging adds at least 2× broker round-trip latency vs direct HTTP; only choose it when the caller is already async or the responder needs queue-based backpressure.

What's Next

Part 8: Priority Queues and Delayed Messaging — Time-Based Routing with BullMQ and RabbitMQ covers the patterns that handle time as a first-class messaging concern: priority-based message ordering, job scheduling with delayed delivery, TTL-based routing via dead-letter exchanges, and the Redis Streams alternative for durable scheduled jobs.

Research & Synthesis Note

This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.

#Request-Reply#RPC#Correlation ID#RabbitMQ#Kafka#Async Patterns#Backend
Siddhant Deval

Written by Siddhant Deval

Senior Full-Stack Engineer building high-scale architectures, browser performance engineering systems, and SaaS platforms.