Siddhant Deval
Siddhant Deval
backend18 min read

AWS Managed Messaging Primitives: SQS, SNS, and EventBridge

SQS, SNS, and EventBridge are AWS's production-ready implementations of the queue and pub/sub patterns — but their managed constraints (visibility timeout, message group IDs, filter policies, content-based routing rules) require deliberate design, not default configuration. This article maps the broker-agnostic concepts from Parts 1–5 to concrete AWS primitives using AWS SDK v3 and LocalStack.

Series·Part 1 of 6

Messaging at Cloud Scale

AWS Managed Messaging Primitives: SQS, SNS, and EventBridge

The payment service deploys to production. The first load test reveals that at 3,000 requests per second, the downstream fulfillment Lambda starts timing out — not from a code bug, but because the SQS visibility timeout is set to its default 30 seconds, and each message takes up to 45 seconds to process. After 30 seconds, SQS assumes the consumer failed and makes the message visible again. A second Lambda invocation picks it up. Now two invocations are processing the same payment simultaneously. The fulfillment database records a double-charge.

The visibility timeout is not a tuning parameter. It is the at-least-once delivery contract — the mechanism by which SQS implements the retry semantics that Kafka implements through committed offsets. Getting it wrong has the same consequences: duplicate processing.

Architectural Note

Series positioning: This is Part 1 of Messaging at Cloud Scale (Series 2). It assumes full familiarity with delivery guarantees (Series 1, Part 5), consumer patterns (Part 6), and dead-letter queues (Part 10). This article maps every broker-agnostic concept from Series 1 to concrete AWS managed primitives — SQS, SNS, and EventBridge — using @aws-sdk/client-sqs@3.x, @aws-sdk/client-sns@3.x, and LocalStack for local development.


1. SQS: The Managed Queue

1.1 Visibility Timeout — The At-Least-Once Contract

SQS does not use committed offsets. Instead, when a consumer receives a message, SQS marks it invisible for VisibilityTimeout seconds. The consumer must explicitly delete the message before the timeout expires. If it does not, the message reappears:

TYPESCRIPT
// ❌ Default VisibilityTimeout=30s — too short for slow processing
import {
  SQSClient,
  ReceiveMessageCommand,
  DeleteMessageCommand,
  ChangeMessageVisibilityCommand,
} from '@aws-sdk/client-sqs'

const sqs = new SQSClient({ region: 'us-east-1' })
const QUEUE_URL = process.env.FULFILLMENT_QUEUE_URL!

// ✅ Size VisibilityTimeout to p99 processing latency × 1.5
// If p99 processing = 30s → VisibilityTimeout = 45s
// Set at queue creation or per-receive via ReceiveMessage parameter
const { Messages } = await sqs.send(new ReceiveMessageCommand({
  QueueUrl:            QUEUE_URL,
  MaxNumberOfMessages: 10,
  VisibilityTimeout:   45,   // seconds — sized to p99 × 1.5
  WaitTimeSeconds:     20,   // long polling: wait up to 20s for messages
}))

// ✅ Extend visibility during long processing (heartbeat pattern)
async function processWithHeartbeat(
  receiptHandle: string,
  processFn:     () => Promise<void>,
  visibilityTimeout = 45,
): Promise<void> {
  const extensionInterval = Math.floor(visibilityTimeout * 0.6) * 1000  // 60% of timeout

  const heartbeat = setInterval(async () => {
    await sqs.send(new ChangeMessageVisibilityCommand({
      QueueUrl:          QUEUE_URL,
      ReceiptHandle:     receiptHandle,
      VisibilityTimeout: visibilityTimeout,  // reset the clock
    }))
  }, extensionInterval)

  try {
    await processFn()
    await sqs.send(new DeleteMessageCommand({
      QueueUrl:      QUEUE_URL,
      ReceiptHandle: receiptHandle,
    }))
  } finally {
    clearInterval(heartbeat)
  }
}
Performance / Safety Warning

Size VisibilityTimeout to p99 processing latency × 1.5, not the average. The average will result in timeouts for the slowest 50% of messages. If processing time varies widely (e.g., 5ms–120s), use the heartbeat extension pattern rather than a fixed timeout — it keeps the message invisible throughout the actual processing duration.

1.2 SQS Standard vs FIFO

Feature SQS Standard SQS FIFO
Ordering Best-effort (not guaranteed) Strict within MessageGroupId
Throughput Unlimited 300 msg/s per queue (3,000 with batching)
Deduplication None (application must be idempotent) Content-based or MessageDeduplicationId (5-min window)
Price Lower Higher
Use case Decoupled async work, idempotent processing Ordered entity events, payment sequences
TYPESCRIPT
// ✅ SQS FIFO — MessageGroupId is the partition key
import { SendMessageCommand } from '@aws-sdk/client-sqs'

// All events for order-42 use the same MessageGroupId → strict ordering
await sqs.send(new SendMessageCommand({
  QueueUrl:               `${process.env.ORDERS_QUEUE_URL}`,
  MessageBody:            JSON.stringify({ orderId: 'ORD-42', event: 'created' }),
  MessageGroupId:         'ORD-42',     // same group = same consumer = strict order
  MessageDeduplicationId: `ORD-42-created-${Date.now()}`,  // 5-min dedup window
}))

// SQS FIFO deduplication: if two messages with the same MessageDeduplicationId
// are published within 5 minutes, SQS discards the duplicate silently
// — equivalent to Kafka's idempotent producer sequence number

// ❌ Don't use one MessageGroupId for all messages
// MessageGroupId='default' → single consumer → zero parallelism
// MessageGroupId=orderId   → one consumer per entity → ordered + parallel ✅

1.3 SQS Dead-Letter Queue Configuration

TYPESCRIPT
// ✅ Configure DLQ via RedrivePolicy — after maxReceiveCount failures, route to DLQ
import { CreateQueueCommand, SetQueueAttributesCommand } from '@aws-sdk/client-sqs'

// 1. Create the DLQ first
const dlqResult = await sqs.send(new CreateQueueCommand({
  QueueName: 'fulfillment-dlq',
  Attributes: { MessageRetentionPeriod: '1209600' }  // 14 days
}))
const dlqArn = (await sqs.send(new GetQueueAttributesCommand({
  QueueUrl:       dlqResult.QueueUrl!,
  AttributeNames: ['QueueArn'],
}))).Attributes!['QueueArn']

// 2. Attach DLQ to main queue
await sqs.send(new SetQueueAttributesCommand({
  QueueUrl: QUEUE_URL,
  Attributes: {
    RedrivePolicy: JSON.stringify({
      deadLetterTargetArn: dlqArn,
      maxReceiveCount:     '5',  // 5 delivery attempts before DLQ routing
    })
  }
}))
// VisibilityTimeout expiry counts as one receive attempt
// After 5 failed attempts: message routes to DLQ, main queue moves on

2. SNS: Fan-Out and Filter Policies

2.1 SNS + SQS Fan-Out Pattern

SNS delivers a single published event to multiple SQS subscribers simultaneously — the managed equivalent of a Kafka topic with multiple consumer groups:

TYPESCRIPT
// ✅ SNS → multiple SQS subscribers (fan-out)
import { SNSClient, PublishCommand } from '@aws-sdk/client-sns'

const sns = new SNSClient({ region: 'us-east-1' })

// Publish once to SNS — all subscribed SQS queues receive a copy
await sns.send(new PublishCommand({
  TopicArn: process.env.ORDER_EVENTS_TOPIC_ARN!,
  Message:  JSON.stringify({
    orderId:    'ORD-42',
    event:      'order.created',
    totalCents: 9900,
    customerId: 'C-1',
  }),
  MessageAttributes: {
    'event.type': {
      DataType:    'String',
      StringValue: 'order.created',
    },
    'customer.tier': {
      DataType:    'String',
      StringValue: 'premium',
    }
  }
}))

// Three SQS queues subscribed to the same SNS topic:
// - fulfillment-queue  → processes all order.created events
// - analytics-queue    → receives all events (no filter)
// - premium-queue      → only receives order.created for premium customers (filter policy)

2.2 SNS Filter Policies

TYPESCRIPT
// SNS filter policy — applied at the subscription level (not the publisher)
// Only messages where event.type='order.created' AND customer.tier='premium'
// are delivered to the premium-sqs subscription

// Filter policy JSON (set via AWS console or CDK/Terraform):
const filterPolicy = {
  'event.type':    ['order.created'],
  'customer.tier': ['premium', 'vip'],  // OR condition within the array
}

// SDK: CreateSubscription with FilterPolicy attribute
import { SubscribeCommand } from '@aws-sdk/client-sns'

await sns.send(new SubscribeCommand({
  TopicArn: process.env.ORDER_EVENTS_TOPIC_ARN!,
  Protocol: 'sqs',
  Endpoint: process.env.PREMIUM_QUEUE_ARN!,
  Attributes: {
    FilterPolicy:           JSON.stringify(filterPolicy),
    FilterPolicyScope:      'MessageAttributes',  // or 'MessageBody' for payload-based filtering
    RawMessageDelivery:     'true',               // skip SNS envelope — SQS receives raw JSON
  }
}))
Pro Tip & Optimization

Enable RawMessageDelivery: true on SNS→SQS subscriptions. Without it, SQS receives the SNS envelope (a JSON wrapper containing the original message, topic ARN, signature, etc.) rather than your raw message body. Consumers must then unwrap the envelope before parsing. Raw delivery simplifies consumer code and removes the SNS-specific parsing dependency.


3. EventBridge: Content-Based Routing

3.1 EventBridge vs SNS Filter Policies

EventBridge is the managed equivalent of a RabbitMQ topic exchange — it routes events based on content-based rules applied to the event payload, not just message attributes:

Feature SNS Filter Policies EventBridge Rules
Routing basis Message attributes only Full event payload (any JSON field)
Rule expressiveness Simple match / prefix prefix, suffix, equals, numeric range, exists, anything-but
Schema registry No Yes (Schema Registry built-in)
Cross-account routing No Yes (cross-account event buses)
Max targets per rule 1 subscription = 1 target 5 targets per rule
Archive and replay No Yes (EventBridge Archive)
Pricing Per message delivery Per event published ($1/million)

3.2 EventBridge Rule Implementation

TYPESCRIPT
// ✅ EventBridge: publish a structured event to the default bus
import { EventBridgeClient, PutEventsCommand } from '@aws-sdk/client-eventbridge'

const eb = new EventBridgeClient({ region: 'us-east-1' })

await eb.send(new PutEventsCommand({
  Entries: [{
    EventBusName: 'default',
    Source:       'com.company.orders',
    DetailType:   'OrderCreated',
    Detail:       JSON.stringify({
      orderId:    'ORD-42',
      totalCents: 9900,
      customerId: 'C-1',
      tier:       'premium',
      region:     'eu-west-1',
    }),
    Time: new Date(),
  }]
}))
JSON
// EventBridge rule — routes premium EU orders to the priority fulfillment Lambda
// Rule pattern (content-based — inspects Detail fields, not just metadata)
{
  "source": ["com.company.orders"],
  "detail-type": ["OrderCreated"],
  "detail": {
    "tier": ["premium", "vip"],
    "totalCents": [{ "numeric": [">", 10000] }],
    "region": [{ "prefix": "eu-" }]
  }
}
// This is the managed equivalent of a RabbitMQ topic exchange binding:
//   routing_key: "order.created.premium.eu.*"
// Without requiring the producer to construct a routing key
TYPESCRIPT
// ✅ EventBridge input transformer — reshape the event before delivery to Lambda
// Strips unnecessary fields, reduces Lambda payload size
const inputTransformer = {
  inputPathsMap: {
    orderId:    '$.detail.orderId',
    totalCents: '$.detail.totalCents',
    customerId: '$.detail.customerId',
  },
  inputTemplate: JSON.stringify({
    orderId:    '<orderId>',
    totalCents: '<totalCents>',
    customerId: '<customerId>',
    source:     'eventbridge',
  })
}

3.3 EventBridge Archive and Replay

TYPESCRIPT
// ✅ EventBridge archive — replay events after consumer bug fix
import { CreateArchiveCommand, StartReplayCommand } from '@aws-sdk/client-eventbridge'

// Create archive: retain all OrderCreated events for 90 days
await eb.send(new CreateArchiveCommand({
  ArchiveName:     'order-events-archive',
  EventSourceArn:  process.env.ORDER_BUS_ARN!,
  RetentionDays:   90,
  EventPattern:    JSON.stringify({
    source: ['com.company.orders'],
  }),
}))

// After fixing a consumer bug: replay events from a specific window
await eb.send(new StartReplayCommand({
  ReplayName:          'fulfillment-replay-2026-12-08',
  EventSourceArn:      process.env.ORDER_ARCHIVE_ARN!,
  EventStartTime:      new Date('2026-12-08T00:00:00Z'),
  EventEndTime:        new Date('2026-12-08T06:00:00Z'),
  Destination: {
    Arn: process.env.ORDER_BUS_ARN!,
    FilterArns: [process.env.FULFILLMENT_RULE_ARN!],
  }
}))
// EventBridge replays all archived events in the window to the specified rule target
// — the equivalent of Kafka consumer offset reset + replay, as a managed API call

4. LocalStack: Local AWS Development

YAML
# docker-compose.yml — LocalStack for local SQS/SNS/EventBridge development
version: '3.8'
services:
  localstack:
    image: localstack/localstack:3.4
    ports:
      - '4566:4566'
    environment:
      SERVICES:       sqs,sns,events,iam
      DEFAULT_REGION: us-east-1
      DEBUG:          '0'
    volumes:
      - ./scripts/localstack-init.sh:/etc/localstack/init/ready.d/init.sh
BASH
#!/bin/bash
# scripts/localstack-init.sh — create all queues and topics on LocalStack startup

awslocal sqs create-queue --queue-name fulfillment-queue \
  --attributes VisibilityTimeout=45,MessageRetentionPeriod=86400

awslocal sqs create-queue --queue-name fulfillment-dlq

awslocal sqs set-queue-attributes \
  --queue-url http://localhost:4566/000000000000/fulfillment-queue \
  --attributes '{"RedrivePolicy":"{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:000000000000:fulfillment-dlq\",\"maxReceiveCount\":\"5\"}"}'

awslocal sns create-topic --name order-events

awslocal sns subscribe \
  --topic-arn arn:aws:sns:us-east-1:000000000000:order-events \
  --protocol sqs \
  --notification-endpoint arn:aws:sqs:us-east-1:000000000000:fulfillment-queue \
  --attributes '{"RawMessageDelivery":"true"}'

echo "✅ LocalStack queues and topics initialized"
TYPESCRIPT
// ✅ SDK client configured for LocalStack in development
const isLocal = process.env.AWS_ENV === 'local'

const sqs = new SQSClient({
  region:   'us-east-1',
  endpoint: isLocal ? 'http://localhost:4566' : undefined,
  credentials: isLocal
    ? { accessKeyId: 'test', secretAccessKey: 'test' }
    : undefined,   // production: uses IAM role via ECS task role
})

Summary

Concept Rule
Visibility timeout contract SQS visibility timeout IS the at-least-once contract — if your consumer takes longer than the timeout to process, the message reappears and is processed twice; size the timeout to p99 processing latency × 1.5.
FIFO MessageGroupId SQS FIFO's MessageGroupId is the partition key equivalent: all messages in the same group are strictly ordered and processed sequentially by a single consumer.
EventBridge routing EventBridge content-based routing is the managed equivalent of RabbitMQ topic exchange bindings — prefer it over SNS filter policies when routing rules depend on event payload fields, not just attributes.

What's Next

Part 2: AWS Streaming Primitives — Kinesis Data Streams and Amazon MSK covers the managed streaming alternatives to self-hosted Kafka: Kinesis Data Streams for serverless, per-shard streaming at cloud scale, and Amazon MSK for teams that need full Kafka compatibility without operating the brokers. The shard-vs-partition model, enhanced fan-out consumers, and the MSK cost model are the primary decision points.

Research & Synthesis Note

This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.

#AWS#SQS#SNS#EventBridge#Managed Messaging#Cloud Architecture#Backend#Distributed Systems
Siddhant Deval

Written by Siddhant Deval

Senior Full-Stack Engineer building high-scale architectures, browser performance engineering systems, and SaaS platforms.