Siddhant Deval
Siddhant Deval
backend20 min read

Lambda at Scale: Concurrency Architecture, ESM & Advanced Runtimes

Lambda's concurrency model is a shared regional resource — not an infinite scaling guarantee. This article covers the regional concurrency pool, Reserved Concurrency as a dual-purpose tool, burst quotas, ESM error handling for Kinesis and SQS, BisectBatchOnFunctionError, parallelization factor, OCI container images, and Lambda Extensions for telemetry and secret caching.

Lambda at Scale: Concurrency Architecture, ESM & Advanced Runtimes

The series mindset holds: every AWS primitive is a tradeoff surface, not a feature toggle. Lambda's default mental model — "it auto-scales to handle any load" — is the most dangerous oversimplification in serverless architecture. Lambda does scale automatically, but it scales against a shared regional concurrency pool with hard account-level limits. A background ETL job that spawns 3,000 concurrent invocations at 3am can exhaust that pool and cause your payment-processing API to return throttle errors — not because the payment API is overloaded, but because a noisy neighbor consumed the shared resource it depends on. This article covers the concurrency model that determines whether your serverless system degrades gracefully under load or cascades catastrophically, and the ESM error-handling primitives that determine whether a malformed event record is handled safely or blocks a shard for days.


1. The Regional Concurrency Pool

Lambda's scaling model is a regional shared resource. Every Lambda function in every account in a region draws from the same pool of concurrent execution capacity. Understanding this pool is the prerequisite for every concurrency-related architectural decision.

1.1 Account-Level Concurrency Limits

The default regional concurrency limit is 1,000 concurrent executions per account. This limit is a hard ceiling — not a soft guideline.

Account Concurrency Limit: 1,000 (default, adjustable via quota increase)
  ├── Function A (Reserved Concurrency: 200) ← isolated slice
  ├── Function B (Reserved Concurrency: 50)  ← isolated slice
  └── Unreserved Pool: 750                   ← shared by all other functions

When a function has no Reserved Concurrency configured, it draws from the unreserved pool. If any function in the account — regardless of which team owns it — exhausts the unreserved pool, every other function drawing from it starts receiving TooManyRequestsException (HTTP 429) errors.

TYPESCRIPT
// ❌ Mid-level: no Reserved Concurrency on any function
// What happens during a batch job at 3am:
// BatchETL function spins up 900 concurrent executions (large SQS batch)
// → Unreserved pool: 1000 - 0 reserved = 1000 available
// → BatchETL consumes 900 → only 100 remaining in the pool
// → PaymentProcessor API gets traffic spike → needs 150 concurrent executions
// → 50 invocations receive 429 ThrottleException
// → API Gateway returns 502 to clients
// → Payment failures — not because PaymentProcessor is broken

// ✅ Senior: Reserved Concurrency isolates critical paths
// After setting Reserved Concurrency:
// PaymentProcessor:    Reserved = 200 (guaranteed, isolated from ALL other functions)
// BatchETL:            Reserved = 100 (caps its own blast radius at 100)
// Unreserved pool:     700 (for everything else)
// Now: BatchETL can never consume more than 100, PaymentProcessor always has 200

1.2 Reserved Concurrency — The Dual-Purpose Tool

Reserved Concurrency does two things simultaneously, and most engineers know only one:

  1. Reservation: The function is guaranteed this many concurrent executions, even if the rest of the account's unreserved pool is depleted.
  2. Cap: The function can never exceed this many concurrent executions, even if the rest of the account has unused capacity.

This dual nature is the source of the most common Reserved Concurrency misconfiguration:

TYPESCRIPT
// ❌ Misconfiguration: Reserved Concurrency set too low on a scaling function
// A function processing real-time video transcoding:
// Reserved: 10 (engineer thought "10 concurrent jobs is plenty")
// Traffic spike: 200 simultaneous transcoding requests arrive
// → Only 10 can run concurrently — the other 190 get throttled
// → SQS queue backs up, visibility timeouts expire, messages redeliver
// → Cascade: DLQ fills up, operational alarm fires at 2am

// Reserve = guarantee + cap. Set it to the MAXIMUM you want, not the expected baseline.
Crucial Requirement

Set Reserved Concurrency on critical-path functions (payment processing, user auth, real-time APIs) to guarantee their availability. Set Reserved Concurrency on background functions (batch jobs, ETL pipelines, report generators) to cap their blast radius. Never set Reserved Concurrency below the function's peak concurrency requirement — that transforms a safety tool into a throttle.

1.3 Burst Quotas and Scaling Rate

When a Lambda function scales up, it cannot add unlimited concurrent instances instantaneously. AWS imposes a burst quota — the rate at which new instances can be initialized per minute in a region:

Region Burst limit (concurrent executions per minute)
us-east-1, us-west-2, eu-west-1 3,000
ap-northeast-1, eu-central-1 1,000
Other regions 500

After the initial burst, Lambda scales at +500 concurrent executions per minute until reaching the account limit. This means a sudden traffic spike from 0 to 5,000 concurrent requests in us-east-1 takes:

Minute 0: 3,000 burst quota initialized
Minute 1: +500 → 3,500
Minute 2: +500 → 4,000
Minute 3: +500 → 4,500
Minute 4: +500 → 5,000 (reached target)

Traffic arriving during the ramp-up period that exceeds current capacity receives 429 throttle errors. Your callers must implement retry with exponential backoff and jitter — API Gateway, SQS, EventBridge, and Kinesis each have different retry semantics for throttled Lambda invocations.

Performance / Safety Warning

API Gateway (synchronous invocation) does NOT retry throttled Lambda invocations — a 429 from Lambda propagates as a 502 to the client immediately. SQS and Kinesis ESM do retry throttled invocations using their own backoff policy. Design API-facing Lambda functions to be throttle-resistant by scaling Provisioned Concurrency to cover burst traffic, or by accepting async (queue-backed) patterns for burst-tolerant workloads.


2. Event Source Mapping (ESM) Deep Dive

Event Source Mapping is Lambda's poll-based integration model for streams and queues. Lambda manages a fleet of pollers that read from the source and invoke your function with batches of records. The concurrency semantics differ fundamentally between SQS ESM and Kinesis/DynamoDB Streams ESM.

2.1 SQS ESM: Message-Batch Concurrency

SQS ESM scales by adding Lambda instances, one per in-flight batch. Lambda's internal poller reads up to BatchSize messages from the queue per poll, invokes one Lambda instance with that batch, and increases concurrency as the queue depth grows — up to the function's Reserved Concurrency limit.

TYPESCRIPT
// SQS ESM configuration (CDK)
import { SqsEventSource } from 'aws-cdk-lib/aws-lambda-event-sources'
import { Queue } from 'aws-cdk-lib/aws-sqs'

const dlq = new Queue(this, 'ProcessorDLQ', {
  retentionPeriod: Duration.days(14)
})

const queue = new Queue(this, 'OrderQueue', {
  visibilityTimeout: Duration.seconds(300), // Must be >= 6× Lambda timeout
  deadLetterQueue: { queue: dlq, maxReceiveCount: 3 }
})

orderProcessorFn.addEventSource(new SqsEventSource(queue, {
  batchSize: 10,                 // Records per invocation (1–10,000)
  maxBatchingWindow: Duration.seconds(5), // Wait up to 5s to fill batch
  reportBatchItemFailures: true  // Enable partial batch success response
}))
Crucial Requirement

The SQS queue visibilityTimeout must be at least 6× your Lambda function timeout. If Lambda takes 30 seconds per batch and the visibility timeout is 60 seconds, messages become visible again after 60 seconds and a second Lambda invocation picks them up before the first has finished — creating duplicate processing.

For partial batch failures, return SQSBatchResponse with only the failed records:

TYPESCRIPT
export const handler = async (
  event: AWSLambda.SQSEvent
): Promise<AWSLambda.SQSBatchResponse> => {
  const failures: AWSLambda.SQSBatchItemFailure[] = []

  await Promise.allSettled(
    event.Records.map(async (record) => {
      try {
        const order = JSON.parse(record.body)
        await processOrder(order)
      } catch (err) {
        console.error({ messageId: record.messageId, error: (err as Error).message })
        failures.push({ itemIdentifier: record.messageId })
      }
    })
  )

  // SQS will redeliver only the failed records; successful records are deleted
  return { batchItemFailures: failures }
}

2.2 Kinesis ESM: Shard-Bounded Concurrency

Kinesis ESM operates on a fundamentally different concurrency model. Lambda assigns one poller per shard. Each shard is processed by at most one Lambda instance at a time (by default). Concurrency is therefore bounded by shards × parallelization_factor, not by message volume.

Kinesis Stream with 4 shards (no parallelization):
  Shard 0 → Lambda Instance A  (max 1 concurrent)
  Shard 1 → Lambda Instance B  (max 1 concurrent)
  Shard 2 → Lambda Instance C  (max 1 concurrent)
  Shard 3 → Lambda Instance D  (max 1 concurrent)
  Max concurrent Lambda executions: 4 (bounded by shard count)

At 10,000 records/second across those 4 shards:
  Each Lambda instance processes records sequentially within its shard.
  Adding more records does NOT add more concurrency.
  The only way to increase concurrency: add shards OR increase parallelization factor.

The parallelization factor (1–10) allows multiple Lambda instances per shard:

TYPESCRIPT
import { KinesisEventSource } from 'aws-cdk-lib/aws-lambda-event-sources'
import { Stream } from 'aws-cdk-lib/aws-kinesis'

processorFn.addEventSource(new KinesisEventSource(stream, {
  batchSize: 100,
  startingPosition: StartingPosition.TRIM_HORIZON,
  parallelizationFactor: 5,     // 5 Lambda instances per shard
  bisectBatchOnFunctionError: true,
  maxRecordAge: Duration.hours(1),
  retryAttempts: 3,
  onFailure: new SqsDestination(dlq)
}))
// With 4 shards × 5 parallelization = 20 max concurrent Lambda instances

2.3 Error Handling — The Configuration That Prevents Shard Blocking

This is the most critical Kinesis ESM configuration. Without it, a single malformed record can halt an entire shard for hours.

The failure chain without BisectBatchOnFunctionError:

Kinesis shard contains: [record 1] [record 2] [MALFORMED record 3] [record 4] [record 5]

Batch attempt 1: Lambda processes [records 1–5] → fails on record 3 → entire batch retried
Batch attempt 2: Lambda processes [records 1–5] → fails on record 3 → entire batch retried
Batch attempt 3: Lambda processes [records 1–5] → fails on record 3 → entire batch retried
...until maxRetryAttempts or maxRecordAge reached

Result without explicit config:
  - maxRetryAttempts default: unlimited retries
  - maxRecordAge default: Kinesis retention (up to 7 days)
  - Shard iterator: BLOCKED — no records beyond record 3 are processed
  - Impact: all events after the malformed record age out unprocessed

The safe configuration:

TYPESCRIPT
// ✅ Complete Kinesis ESM error handling configuration
processorFn.addEventSource(new KinesisEventSource(stream, {
  batchSize: 100,
  startingPosition: StartingPosition.LATEST,

  // CRITICAL: Bisect narrows down the poison pill to a batch of 1
  bisectBatchOnFunctionError: true,
  // How it works: batch of 100 fails → splits into [50] and [50]
  //               failing half retried → splits again → [25] [25]
  //               continues until the single failing record is isolated
  //               isolated record → sent to DLQ → shard continues

  maximumRetryAttempts: 3,               // Don't retry forever
  maximumRecordAgeInSeconds: 3600,       // Drop records older than 1 hour
  onFailure: new SqsDestination(dlq),    // Capture exhausted records for inspection

  reportBatchItemFailures: true          // Only retry individual failed records when supported
}))
Performance / Safety Warning

bisectBatchOnFunctionError and maximumRetryAttempts have NO sensible defaults — bisectBatchOnFunctionError defaults to false and maximumRetryAttempts defaults to unlimited. Every Kinesis and DynamoDB Streams ESM configuration must set both explicitly or risk shard blocking on the first malformed record.


3. Packaging: Zip Archives vs Container Images

Lambda supports two deployment package formats with distinct size limits, cold-start characteristics, and use-case profiles.

Attribute Zip archive OCI Container image
Maximum unzipped size 250 MB 10 GB
Maximum compressed size 50 MB direct upload / 250 MB via S3 10 GB
Cold-start overhead Lower (package unzip) Higher (image layer cache pull)
Build toolchain esbuild, webpack, SAM Docker
Custom runtime support Runtime API bootstrap binary Dockerfile with any base
Layer support Yes (up to 5 layers, 250MB total) N/A (layers built into image)
Local testing SAM CLI local invoke Docker run
DOCKERFILE
# ✅ Container image: use only when zip's 250MB limit is genuinely exhausted
# (e.g., ML inference with model weights embedded, native binaries)
FROM public.ecr.aws/lambda/nodejs20.x:latest

# Copy pre-built artifacts only — don't run npm install in Docker for Lambda
COPY dist/handler.js ${LAMBDA_TASK_ROOT}/
COPY node_modules ${LAMBDA_TASK_ROOT}/node_modules/

CMD ["handler.handler"]
Pro Tip & Optimization

Before reaching for container images, verify whether Lambda Layers can solve the size problem. A shared layer with large native binaries (FFmpeg, ImageMagick, Sharp) can be attached to multiple functions without duplicating the payload per function, and layers maintain the zip archive's lower cold-start profile.


4. Lambda Extensions

Lambda Extensions are processes that run alongside the handler inside the MicroVM. They integrate with Lambda's lifecycle via the Extensions API and are used for telemetry collection, secret caching, and compliance enforcement without modifying handler code.

4.1 Internal vs External Extensions

Type Process model Lifecycle Use case
Internal Same process as handler Shares handler PID Thin wrappers, middleware patterns
External Separate process Independent lifecycle (receives Init, Invoke, Shutdown events) Telemetry agents, secret caching daemons
TYPESCRIPT
// Example: AWS Lambda Powertools as an internal extension pattern (Node.js)
// Powertools wraps the handler — no external process needed
import { Logger } from '@aws-lambda-powertools/logger'
import { Tracer } from '@aws-lambda-powertools/tracer'

const logger = new Logger({ serviceName: 'order-service' })
const tracer = new Tracer({ serviceName: 'order-service' })

// captureLambdaHandler wraps the handler internally — no separate extension process
export const handler = tracer.captureLambdaHandler(
  async (event: AWSLambda.APIGatewayProxyEvent, context: AWSLambda.Context) => {
    logger.addContext(context)
    logger.info('Processing request', { path: event.path })
    // ...
  }
)

4.2 Secret Caching via Extension

The AWS Parameters and Secrets Lambda Extension (a managed external extension) caches Secrets Manager and SSM Parameter Store values locally via a local HTTP server at http://localhost:2773. This eliminates direct API calls from the handler and reduces secret rotation latency impact:

TYPESCRIPT
// ✅ Secret caching via Lambda Extension — no direct Secrets Manager API calls
// The extension handles caching with configurable TTL (default: 300s)
async function getSecret(secretId: string): Promise<string> {
  const port = process.env.PARAMETERS_SECRETS_EXTENSION_HTTP_PORT ?? '2773'
  const token = process.env.AWS_SESSION_TOKEN!

  const response = await fetch(
    `http://localhost:${port}/secretsmanager/get?secretId=${encodeURIComponent(secretId)}`,
    { headers: { 'X-Aws-Parameters-Secrets-Token': token } }
  )
  const data = await response.json() as { SecretString: string }
  return data.SecretString
}
// Cache TTL controlled by PARAMETERS_SECRETS_EXTENSION_CACHE_SIZE and
// PARAMETERS_SECRETS_EXTENSION_MAX_CONNECTIONS environment variables
Crucial Requirement

External extensions receive CPU time from the same vCPU allocation as the handler. An extension that performs heavy processing (log aggregation, metrics flushing) during the Invoke phase will degrade handler latency. Monitor extension overhead with InitDuration (Init phase) and Duration (Invoke phase) CloudWatch metrics separately.


5. Concurrency Design Patterns

5.1 Reserved Concurrency Allocation Strategy

A production account should categorize all Lambda functions into three tiers and assign Reserved Concurrency accordingly:

Tier Functions Reserved Concurrency Rationale
Critical path Payment processing, user auth, real-time APIs 100–500 (explicit guarantee) Isolated from account noise; SLA-backed
Background workers ETL jobs, report generators, batch processors 10–50 (blast-radius cap) Prevent these from starving critical paths
Unreserved Low-traffic utilities, dev functions None Share the remaining pool; no SLA requirement
TYPESCRIPT
// CDK: per-function Reserved Concurrency with Auto Scaling
import { Function } from 'aws-cdk-lib/aws-lambda'

const paymentFn = new Function(this, 'PaymentProcessor', {
  // ... function config
  reservedConcurrentExecutions: 200, // Guaranteed + capped at 200
})

// For functions with unpredictable traffic, use Application Auto Scaling
// to adjust Provisioned Concurrency dynamically
const target = paymentFn.addAutoScaling({ minCapacity: 5, maxCapacity: 50 })
target.scaleOnUtilization({ utilizationTarget: 0.75 })
// This scales Provisioned Concurrency between 5 and 50 based on utilization
// Reserved Concurrency remains at 200 (the hard ceiling)

5.2 Throttle Response Strategy by Invocation Type

Invocation type Source Throttle behavior Correct mitigation
Synchronous API GW, ALB 429 propagates to caller immediately Provisioned Concurrency; client-side retry
Asynchronous EventBridge, SNS Lambda retries with exponential backoff (up to 6 hours) Destination DLQ; idempotency in handler
Stream/Queue SQS, Kinesis ESM retries with backoff per batch ESM retry config; DLQ destination

Summary

Concept Rule
Regional concurrency pool Shared account-wide limit (default 1,000); noisy neighbors deplete unreserved capacity
Reserved Concurrency Simultaneously a guarantee and a ceiling — protects critical paths AND caps blast radius
Burst quota Lambda scales at 3,000/minute (major regions) then +500/minute — synchronous callers see 429 during ramp
SQS ESM concurrency One Lambda instance per in-flight batch; scales with queue depth up to Reserved Concurrency
Kinesis ESM concurrency Bounded by shards × parallelization_factor; add shards or increase parallelization to scale
BisectBatchOnFunctionError Mandatory for Kinesis/DynamoDB Streams ESM; defaults to false; without it, one bad record blocks a shard
Lambda Extensions Share handler vCPU; external extensions that consume excessive CPU degrade handler latency

What's Next

In Part 3: API Gateway Essentials — Routing, Integrations & Authorization, we move to the ingress layer: why the default REST API choice costs 70% more than HTTP API for identical Lambda Proxy use cases, what the Proxy Integration event object actually contains, and why Lambda Authorizer caching with TTL=0 invokes your auth function on every single API call.

Research & Synthesis Note

This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.

#AWS Lambda#Concurrency#Event Source Mapping#Kinesis#SQS#Reserved Concurrency#Serverless
Siddhant Deval

Written by Siddhant Deval

Senior Full-Stack Engineer building high-scale architectures, browser performance engineering systems, and SaaS platforms.