Siddhant Deval
Siddhant Deval
backend18 min read

From Mid-Level to Senior: Production Readiness Checklist for Serverless Systems

Production readiness is not a property of code — it is a property of systems. A Lambda function that passes all unit tests can still silently discard 5% of events, destroy exactly-once guarantees under retry, or incur 10× expected DynamoDB billing because of a schema design that was never profiled under load. This capstone article synthesizes the series into a production readiness checklist, organized by failure domain: compute boundaries, data contracts, failure envelopes, observability baselines, and cost models.

Series·Part 13 of 13

AWS Serverless Engineering: Lambda to Production

From Mid-Level to Senior: Production Readiness Checklist for Serverless Systems

At the senior level, you don't write serverless functions — you design execution boundaries, capacity contracts, and failure envelopes. Every AWS primitive is a tradeoff surface, not a feature toggle.

This series began with Lambda's MicroVM lifecycle and ends with a harder question: when is a serverless system actually ready for production? Not "does it work in staging" — that test passes on almost anything. The correct question is: does it participate correctly in a distributed system under partial failure, at peak load, after a deployment, and on retry? These are distinct requirements. Only the last three require senior-level design thinking.

This capstone article synthesizes the series into a production readiness framework organized by failure domain. Each domain has a checklist. Each checklist item has a failure mode it prevents. Where the failure mode is covered in an earlier article, this one links to it rather than repeating the explanation.


The Five Production Readiness Domains

1. Compute Boundaries     — concurrency limits, scaling behavior, throttle response
2. Data Contracts         — schema validation, idempotency, ordering guarantees
3. Failure Envelopes      — DLQ coverage, retry policies, blast radius isolation
4. Observability Baseline — three-signal coverage, correlation IDs, alarm set
5. Cost Models            — unit economics per service, anti-pattern identification

Domain 1: Compute Boundaries

Checklist

Item Failure Prevented
Reserved Concurrency set on critical-path functions Noisy neighbor functions depleting the unreserved pool → payment/auth functions throttled
Reserved Concurrency set (as cap) on background jobs Batch job consuming 90% of account concurrency and starving API functions
Burst quota modeled for traffic patterns Synchronous API clients receiving 429/502 during traffic ramp-up that exceeds burst quota
Lambda timeout ≤ SQS visibility timeout / 6 Messages reappearing during processing → duplicate Lambda invocations
Provisioned Concurrency or SnapStart on latency-SLA functions P99 cold-start exceeding SLA on first invocations
TYPESCRIPT
// ✅ Compute boundary configuration template
const criticalFn = new Function(this, 'PaymentProcessor', {
  timeout: Duration.seconds(30),          // Timeout set explicitly — never default 3s
  memorySize: 1024,                        // Set from Power Tuning output, not guessed
  reservedConcurrentExecutions: 200,       // Guarantee + cap; never 'undefined'
  environment: { POWERTOOLS_SERVICE_NAME: 'payment-service' },
})

// Provisioned Concurrency via alias for P99 SLA
const prodAlias = criticalFn.addAlias('prod')
prodAlias.addAutoScaling({
  minCapacity: 5,
  maxCapacity: 50,
}).scaleOnUtilization({ utilizationTarget: 0.75 })

Coverage in series: Part 1 · Part 2


Domain 2: Data Contracts

Checklist

Item Failure Prevented
All Lambda handlers are idempotent Retried invocations double-applying side effects (charges, inventory decrements)
Input validation before any write operation Malformed events causing partial writes and inconsistent state
Schema registered in EventBridge Schema Registry Schema drift between producers and consumers silently breaking consumers
DynamoDB conditional writes on all PutItem for new items Race conditions creating duplicate records under concurrent invocations
FIFO queue used where ordering is a business constraint Out-of-order event processing corrupting state machines
TYPESCRIPT
// ✅ Idempotency pattern: Powertools Idempotency on handler
import { makeHandlerIdempotent } from '@aws-lambda-powertools/idempotency'
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb'

const persistenceStore = new DynamoDBPersistenceLayer({
  tableName: 'IdempotencyStore',
  keyAttr: 'id',
})

export const handler = makeHandlerIdempotent(
  async (event: AWSLambda.SQSEvent) => {
    // Idempotency is per-SQS message ID by default
    // Re-invocation with same event → returns cached result without re-executing
    return processOrder(event.Records[0])
  },
  { persistenceStore, config: { expiresAfterSeconds: 3600 } }
)

Coverage in series: Part 8 (DynamoDB conditional writes) · Part 10 (Schema Registry)


Domain 3: Failure Envelopes

Checklist

Item Failure Prevented
DLQ configured on every SQS queue Failed messages silently discarded after maxReceiveCount
DLQ alarm (threshold: 0) on every DLQ DLQ filling with failed work with no operational signal
Lambda async invocations have destination DLQ Async event loss with no capture path
Kinesis ESM: bisectBatchOnFunctionError: true + maximumRetryAttempts: N Single malformed record blocking shard permanently
SNS → SQS fan-out instead of SNS → Lambda SNS delivery exhaustion silently dropping messages
Reserved Concurrency ≠ 0 on all production functions ConcurrentExecutionLimitExceeded errors if concurrency limit accidentally set to 0
TYPESCRIPT
// ✅ Complete failure envelope template
const fn = new Function(this, 'Fn', { /* ... */ })
const asyncDlq = new Queue(this, 'AsyncDLQ')
const sqsDlq = new Queue(this, 'SQSDLQ')

// Async invocation failure capture
fn.configureAsyncInvoke({
  onFailure: new SqsDestination(asyncDlq),
  maxEventAge: Duration.hours(1),
  retryAttempts: 2,
})

// DLQ alarm — zero tolerance
new Alarm(this, 'AsyncDLQAlarm', {
  metric: asyncDlq.metricApproximateNumberOfMessagesVisible({ period: Duration.minutes(1) }),
  threshold: 0,
  evaluationPeriods: 1,
  alarmDescription: `${fn.functionName} async failures captured — requires investigation and redrive`,
})

Coverage in series: Part 2 (ESM error handling) · Part 9 (SQS/SNS DLQs) · Part 10 (EventBridge DLQ semantics)


Domain 4: Observability Baseline

Checklist

Item Failure Prevented
Structured logging with correlationId propagation Incident debugging requires manual log correlation by timestamp under high concurrency
X-Ray active tracing on all functions Bottleneck identification requires code instrumentation changes during an incident
EMF custom metrics (OrderProcessed, PaymentFailed, etc.) Business-level failures invisible in AWS service metrics
Mandatory alarm set per function (error rate, P99, throttles, concurrency) Silent degradation reaching users before any operational alert fires
DLQ alarm per DLQ Data loss occurring with no operational signal
CloudWatch dashboard per service (not per function) Incident response requires navigating 40 individual function dashboards
TYPESCRIPT
// ✅ Service-level CloudWatch Dashboard — one per service, all signals
import { Dashboard, GraphWidget, SingleValueWidget } from 'aws-cdk-lib/aws-cloudwatch'

new Dashboard(this, 'OrderServiceDashboard', {
  dashboardName: 'order-service-production',
  widgets: [
    [
      new GraphWidget({ title: 'Error Rate (%)', left: [/* errorRate metric */], width: 12 }),
      new GraphWidget({ title: 'P99 Duration (ms)', left: [/* p99 metric */], width: 12 }),
    ],
    [
      new GraphWidget({ title: 'Concurrent Executions', left: [/* concurrency metric */], width: 8 }),
      new GraphWidget({ title: 'Throttles', left: [/* throttle metric */], width: 8 }),
      new SingleValueWidget({ title: 'DLQ Depth', metrics: [dlqDepth], width: 8 }),
    ],
  ]
})

Coverage in series: Part 12 (full observability setup)


Domain 5: Cost Models

Checklist

Item Cost Anti-Pattern Prevented
Lambda memory set from Power Tuning output Memory set to 512MB by default losing 40% performance headroom or paying 2× for unused memory
DynamoDB access patterns use Query, not Scan Scan+FilterExpression costs 99.99% more than Query for same result at scale
REST API audited for migration to HTTP API Paying $3.50/M instead of $1.00/M for identical Lambda Proxy use cases
NAT Gateway replaced with VPC Endpoints for DynamoDB/S3 Paying $0.045/GB data processing charge on AWS-internal traffic
Authorizer TTL set (not 0) Paying for Authorizer Lambda invocation on every API request
DynamoDB On-Demand vs Provisioned modeled at current traffic level On-Demand costs 3× provisioned at steady-state traffic; provisioned costs 4× minimum at zero traffic

The Senior Mental Model — Internalized as Design Principles

The checklist above codifies behaviors. The mental model behind them is a different thing — it is the reasoning that generates the right behavior for situations not covered by any checklist.

At the senior level:

"My function works" ←  mid-level completion criterion

"My function participates correctly in a distributed system
  under partial failure:    Does every failure path have a capture mechanism?
  at peak load:            Does every resource have a capacity boundary?
  after a deployment:      Does every write path have idempotency?
  on retry:                Does every retry produce the same observable state?
  under noisy neighbors:   Is every critical-path function isolated from the shared pool?"

These questions do not have a universal answer. They have answers that depend on the specific workload, traffic pattern, and failure tolerance of the system being built. The senior skill is not knowing the answers in advance — it is knowing which questions to ask before shipping, and having the primitives to answer them.


Series Navigation

This article is Part 13 of AWS Serverless Engineering: Lambda to Production.

Part Article Domain
1 Lambda Internals: Execution Model, Context Reuse & Cold Starts Compute
2 Lambda at Scale: Concurrency Architecture, ESM & Advanced Runtimes Compute
3 API Gateway Essentials: Routing, Integrations & Authorization Ingress
4 API Gateway Advanced: VTL Templates, Direct Integrations & WAF Ingress
5 AppSync Foundations: Schema, Unit Resolvers & Data Sources GraphQL
6 AppSync Advanced: Pipeline Resolvers, Subscriptions & Multi-Auth GraphQL
7 DynamoDB Data Modeling: Access-Pattern-First Design Data
8 DynamoDB at Scale: Partition Internals, GSI Backpressure & ACID Transactions Data
9 Messaging Foundations: SQS, SNS & Fan-Out Patterns Messaging
10 EventBridge Architecture: Event Mesh, Pipes & Failure Routing Messaging
11 Serverless Security: IAM Least Privilege, VPC Networking & Secrets Governance Security
12 Serverless Observability & FinOps: CloudWatch, X-Ray & Cost Modeling Observability
13 From Mid-Level to Senior: Production Readiness Checklist Synthesis
Research & Synthesis Note

This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.

#Production Readiness#System Design#Serverless#AWS#Senior Engineering#Checklist
Siddhant Deval

Written by Siddhant Deval

Senior Full-Stack Engineer building high-scale architectures, browser performance engineering systems, and SaaS platforms.