Request-Reply over Messaging: Async RPC, Correlation IDs, and Inbox Topics
Engineers migrating from HTTP to async messaging always need request-reply semantics — but the naive shared response queue pattern introduces race conditions and message bleed between callers. This article implements the correct per-caller inbox pattern with correlation IDs and timeout handling for both RabbitMQ reply-to and Kafka reply topics.
Distributed Messaging Systems
Request-Reply over Messaging: Async RPC, Correlation IDs, and Inbox Topics
The fraud scoring service is deployed as a Kafka consumer — it processes high-volume transaction events and has its own autoscaling, independent deployment cycle, and its own DLQ. The checkout service needs fraud scores synchronously before confirming an order. The instinct is to call the fraud service over HTTP, but the fraud service team does not want to run an HTTP server alongside their consumer — it doubles their operational surface and bypasses their existing queue-based backpressure. The checkout team proposes a shared fraud.responses Kafka topic. Three weeks later, under load, two checkout instances start consuming each other's fraud responses — instance A's request gets consumed by instance B, which times out waiting for a response that already arrived.
The shared response topic is the wrong pattern. The correct one is a per-caller inbox — an isolated consumption channel that guarantees only the caller who issued the request will receive its response.
Series positioning: This is Part 7 of Distributed Messaging Systems. The prerequisite mental model for request-reply semantics is RESTful API Design: Resources, HTTP Methods, and Status Codes — this article assumes familiarity with synchronous RPC and focuses on when and how to implement RPC semantics over async messaging. The decision of whether to use this pattern is as important as how.
1. When Request-Reply over Messaging Is Justified
1.1 The Default Answer: Use HTTP/gRPC
Request-reply over messaging always adds complexity and latency relative to direct HTTP. Choose it only when at least one of these is true: (1) the responder must queue incoming requests for backpressure, (2) the caller is already async and does not hold a synchronous HTTP connection open, (3) the responder team refuses to run an HTTP server, (4) the request fan-out is massive and queue-based load levelling is required.
1.2 The Decision Matrix
| Criterion | Use HTTP/gRPC | Use Request-Reply over Messaging |
|---|---|---|
| Latency requirement | < 50ms p99 | > 50ms acceptable |
| Responder deployment model | HTTP server | Pure consumer (no HTTP surface) |
| Caller model | Synchronous web request | Async event processor |
| Request volume | Predictable | Bursty — needs queue buffering |
| Backpressure | Not needed | Responder must control intake rate |
| Response fan-out | One caller, one response | One request, multiple partial responses |
2. The Broken Pattern: Shared Response Topic
2.1 Why a Shared Topic Bleeds Responses
The race condition: under concurrent load, Kafka's partition assignment determines which consumer instance receives which message. A shared consumer group cannot guarantee that the caller who published a request is the same instance that consumes the response.
3. The Correct Pattern: Per-Caller Inbox
3.1 RabbitMQ: reply-to Pseudo-Queue
RabbitMQ has built-in support for per-caller inbox via the replyTo property and amq.rabbitmq.reply-to pseudo-queue — an exclusive, auto-delete, server-named queue allocated per connection:
3.2 Kafka: Per-Caller Reply Topic
Kafka has no built-in reply-to mechanism. The correct pattern is a per-caller inbox topic — each caller service has its own topic, partitioned for parallelism, and subscribes to it in its own consumer group:
Never let the responder generate the correlation-id. The correlation-id must be set by the caller and echoed verbatim. A responder-generated ID breaks the lookup — the caller's pending map keyed on the original UUID will never match the responder's new UUID.
4. Timeout Handling and Orphaned Responses
Every request-reply implementation over messaging must handle the timeout case explicitly:
Set the caller's timeout to request TTL + expected responder processing time + one round-trip broker latency. If the fraud model takes 200ms and broker round-trip is 10ms, set timeoutMs = 500ms for a 2× safety margin. Publish the request with x-message-ttl: 500 (RabbitMQ) or set a short message retention on the request topic — an unanswered request past its timeout is a zombie that will be processed by the responder for no benefit.
Summary
| Concept | Rule |
|---|---|
| Shared queue race condition | A shared response queue is a race condition — two callers will occasionally consume each other's responses; always use per-caller inbox topics or RabbitMQ's reply-to pseudo-queue. |
| Correlation ID ownership | correlation-id must be set by the caller and echoed verbatim by the responder — never let the responder generate a new ID. |
| Latency trade-off | Request-reply over messaging adds at least 2× broker round-trip latency vs direct HTTP; only choose it when the caller is already async or the responder needs queue-based backpressure. |
What's Next
Part 8: Priority Queues and Delayed Messaging — Time-Based Routing with BullMQ and RabbitMQ covers the patterns that handle time as a first-class messaging concern: priority-based message ordering, job scheduling with delayed delivery, TTL-based routing via dead-letter exchanges, and the Redis Streams alternative for durable scheduled jobs.
This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.