Siddhant Deval
Siddhant Deval
backend20 min read

Serverless Observability & FinOps: CloudWatch, X-Ray & Cost Modeling

Serverless observability requires structured logs, distributed traces, and metrics correlated by correlation ID — without this three-signal discipline, a performance regression in one Lambda surfaces as a P99 latency spike in a completely different service's dashboard with no causal link. This article covers EMF (Embedded Metrics Format) for zero-overhead custom metrics, X-Ray sampling and segment annotation, PowerTools Logger correlation IDs, and the FinOps cost models for Lambda, DynamoDB, API Gateway, and EventBridge.

Serverless Observability & FinOps: CloudWatch, X-Ray & Cost Modeling

Every AWS primitive is a tradeoff surface, not a feature toggle. Serverless systems fail at the seams — between Lambda functions, between message queues, between service boundaries. A Lambda function can succeed (return 200) while silently discarding 5% of events that failed a downstream SQS write that was never retried. The function's own CloudWatch metrics look healthy. Only a distributed trace that follows the event across the Lambda invocation, the SQS SendMessage call, the downstream Lambda processing, and the DLQ depth shows the full picture. This article covers the three-signal observability model (structured logs + distributed traces + metrics) and the FinOps cost models that surface where Lambda memory misconfiguration and DynamoDB Scan operations are creating silent billing surprises.


1. Structured Logging with Correlation IDs (Lambda Powertools)

1.1 The Correlation ID Contract

Every log line in a distributed system must carry a correlation ID that traces a logical request across all the services it touched. Without it, debugging a production incident requires manually correlating log lines across Lambda, API Gateway, SQS, and EventBridge by timestamp — a technique that is unreliable under high concurrency.

TYPESCRIPT
import { Logger } from '@aws-lambda-powertools/logger'
import { injectLambdaContext } from '@aws-lambda-powertools/logger/middleware'
import middy from '@middy/core'

const logger = new Logger({
  serviceName: 'order-service',
  logLevel: 'INFO',
})

// ✅ Middy middleware: injects Lambda context (request ID, cold start flag, function name)
// into every log line automatically — no manual logger.addContext() calls required
export const handler = middy(
  async (event: AWSLambda.APIGatewayProxyEvent): Promise<AWSLambda.APIGatewayProxyResult> => {
    // Set correlation ID from upstream: API GW requestId or custom header
    const correlationId = event.headers['x-correlation-id'] ?? event.requestContext.requestId
    logger.appendKeys({ correlationId, orderId: event.pathParameters?.orderId })

    logger.info('Processing order request', {
      path: event.path,
      method: event.httpMethod,
    })

    try {
      const order = await processOrder(event)
      logger.info('Order processed successfully', { orderId: order.id, status: order.status })
      return { statusCode: 200, body: JSON.stringify(order) }
    } catch (err) {
      logger.error('Order processing failed', { error: (err as Error).message })
      throw err
    }
  }
).use(injectLambdaContext(logger))

// Every log line now contains:
// { "level": "INFO", "service": "order-service", "correlationId": "abc123",
//   "orderId": "o99", "function_name": "OrderProcessor",
//   "cold_start": false, "xray_trace_id": "..." }
Pro Tip & Optimization

Pass the correlation ID in a custom header (x-correlation-id) from every caller — API Gateway, Lambda function calling another Lambda, SQS message attributes, EventBridge detail fields. When every hop propagates the same correlation ID, CloudWatch Logs Insights can find all log lines for a single business operation across all services with: filter correlationId = "abc123".

1.2 EMF — Custom Metrics Without PutMetricData

EMF (Embedded Metrics Format) publishes custom CloudWatch metrics as structured JSON log lines. CloudWatch agents extract them asynchronously — no synchronous PutMetricData API calls, no added invocation latency, no per-API-call billing.

TYPESCRIPT
import { Metrics, MetricUnit } from '@aws-lambda-powertools/metrics'
import { logMetrics } from '@aws-lambda-powertools/metrics/middleware'

const metrics = new Metrics({ namespace: 'OrderService', serviceName: 'order-processor' })

export const handler = middy(
  async (event: AWSLambda.SQSEvent) => {
    for (const record of event.Records) {
      const order = JSON.parse(record.body)

      metrics.addDimension('orderRegion', order.region) // Dimension per invocation
      metrics.addMetric('OrderProcessed', MetricUnit.Count, 1)
      metrics.addMetric('OrderTotal', MetricUnit.NoUnit, order.total)

      if (order.status === 'FRAUD_FLAGGED') {
        metrics.addMetric('FraudFlaggedOrder', MetricUnit.Count, 1)
      }
    }
    // Metrics are flushed at end of handler as structured JSON — no synchronous API call
  }
).use(logMetrics(metrics))

// EMF output (auto-flushed to CloudWatch Logs):
// { "_aws": { "Timestamp": ..., "CloudWatchMetrics": [{ "Namespace": "OrderService",
//   "Dimensions": [["service", "orderRegion"]], "Metrics": [{ "Name": "OrderProcessed", "Unit": "Count" }] }] },
//   "service": "order-processor", "orderRegion": "us-east", "OrderProcessed": 1, "OrderTotal": 149.99 }

2. X-Ray Distributed Tracing

X-Ray traces the path of a request across Lambda, DynamoDB, SQS, HTTP, and other AWS services. Each service in the path emits a segment (the service's own processing) and creates subsegments (nested operations within that service).

2.1 Auto-Instrumentation with Powertools Tracer

TYPESCRIPT
import { Tracer } from '@aws-lambda-powertools/tracer'
import { captureLambdaHandler } from '@aws-lambda-powertools/tracer/middleware'
import { DynamoDBDocumentClient } from '@aws-sdk/lib-dynamodb'

const tracer = new Tracer({ serviceName: 'order-service' })

// Auto-capture: DynamoDB SDK calls appear as X-Ray subsegments automatically
const docClient = tracer.captureAWSv3Client(
  DynamoDBDocumentClient.from(new DynamoDBClient({ region: process.env.AWS_REGION }))
)

export const handler = middy(
  async (event: AWSLambda.SQSEvent) => {
    for (const record of event.Records) {
      const order = JSON.parse(record.body)

      // Custom annotation — searchable in X-Ray console
      tracer.putAnnotation('orderId', order.orderId)
      tracer.putAnnotation('orderRegion', order.region)

      // Custom metadata — visible in trace detail, not searchable
      tracer.putMetadata('orderPayload', { total: order.total, itemCount: order.items.length })

      const segment = tracer.getSegment()!
      const subsegment = segment.addNewSubsegment('## processPayment')
      try {
        await processPayment(order)
      } catch (err) {
        subsegment.addError(err as Error)
        throw err
      } finally {
        subsegment.close()
      }
    }
  }
).use(captureLambdaHandler(tracer))

2.2 X-Ray Sampling Rules

X-Ray does not trace every request by default. The default rule samples 5% of requests. For production diagnosis, configure custom sampling rules:

TYPESCRIPT
// CDK: X-Ray sampling rule — trace all order failures, 10% of successes
import { CfnSamplingRule } from 'aws-cdk-lib/aws-xray'

new CfnSamplingRule(this, 'OrderSamplingRule', {
  samplingRule: {
    ruleName: 'OrderErrorRule',
    resourceArn: '*',
    host: '*',
    httpMethod: '*',
    urlPath: '*',
    serviceName: 'order-service',
    serviceType: 'AWS::Lambda::Function',
    fixedRate: 0.10,   // 10% of normal requests
    reservoirSize: 5,  // Always trace up to 5 req/s regardless of rate
    priority: 1,
    version: 1,
  }
})

3. CloudWatch Dashboards and Alarms — The Observability Baseline

3.1 The Mandatory Alarm Set (Per Lambda Function)

TYPESCRIPT
// The minimum alarm set — without these, a production incident is invisible until users report it
const fn: Function = ...

// 1. Error rate alarm (percentage, not absolute count)
new Alarm(this, `${fn.functionName}-ErrorRate`, {
  metric: fn.metricErrors().createAlarm // percentage requires MathExpression
  // Use MathExpression: errors / (errors + invocations) × 100
  metric: new MathExpression({
    expression: '100*(errors/invocations)',
    usingMetrics: {
      errors: fn.metricErrors({ period: Duration.minutes(5) }),
      invocations: fn.metricInvocations({ period: Duration.minutes(5) }),
    }
  }),
  threshold: 5, // Alert if error rate > 5%
  evaluationPeriods: 2,
})

// 2. P99 duration alarm
new Alarm(this, `${fn.functionName}-P99Duration`, {
  metric: fn.metricDuration({ statistic: 'p99', period: Duration.minutes(5) }),
  threshold: fn.timeout!.toMilliseconds() * 0.8, // Alert at 80% of timeout
  evaluationPeriods: 2,
})

// 3. Throttle alarm
new Alarm(this, `${fn.functionName}-Throttles`, {
  metric: fn.metricThrottles({ period: Duration.minutes(1) }),
  threshold: 0,
  evaluationPeriods: 1,
})

// 4. Concurrent executions (approaching Reserved Concurrency limit)
new Alarm(this, `${fn.functionName}-ConcurrencyUtilization`, {
  metric: fn.metricConcurrentExecutions({ period: Duration.minutes(1), statistic: 'Maximum' }),
  threshold: (fn.reservedConcurrentExecutions ?? 1000) * 0.8,
  evaluationPeriods: 3,
})

4. FinOps — Cost Modeling for Each Service

4.1 Lambda Cost Model

Lambda cost = (Invocation cost) + (Compute cost) + (Provisioned Concurrency cost, if any)

Invocation cost: $0.20 per million invocations (same regardless of memory or duration)
Compute cost:    GB-seconds × $0.0000166667 per GB-second

GB-seconds = (memory_MB / 1024) × duration_seconds × invocation_count

Example: 512MB, 200ms avg, 10M invocations/month:
  GB-seconds = (512/1024) × 0.2 × 10,000,000 = 1,000,000 GB-seconds
  Compute cost: 1,000,000 × $0.0000166667 = $16.67
  Invocation cost: (10,000,000 / 1,000,000) × $0.20 = $2.00
  Total: $18.67/month

Power Tuning insight: increasing memory to 1024MB reduces duration to 100ms:
  GB-seconds = (1024/1024) × 0.1 × 10,000,000 = 1,000,000 GB-seconds (SAME)
  But: execution completes in half the time → same cost, lower latency
  Further: 1769MB (1 vCPU) at 80ms:
  GB-seconds = (1769/1024) × 0.08 × 10,000,000 = 1,382,000 GB-seconds (MORE EXPENSIVE)
  → 1024MB is the cost-optimal point for this workload
Pro Tip & Optimization

Run AWS Lambda Power Tuning for every production function before finalizing memory allocation. The tool invokes your function at multiple memory sizes and plots cost vs performance — identifying the exact GB-second minimum for the specific workload. The "balanced" mode finds the setting where performance gains no longer offset cost increases.

4.2 DynamoDB Cost Model

On-Demand:
  Write Request Units (WRU): $1.25 per million
  Read Request Units (RRU):  $0.25 per million

Capacity consumption by operation:
  GetItem:              1 RRU (eventually consistent) per 4KB
  PutItem/UpdateItem:   1 WRU per 1KB
  Query:                1 RRU per 4KB of data returned
  Scan:                 1 RRU per 4KB of data SCANNED (not returned — filter does not reduce cost)
  TransactWriteItems:   2 WRU per item
  BatchWriteItem:       1 WRU per item

FinOps anti-pattern: Scan + FilterExpression
  10M-item table × 1KB avg size = 10,000,000 RRU per Scan
  Even if FilterExpression keeps 1% of items (100,000 items) → STILL 10M RRU charged
  Cost: 10,000,000 / 1,000,000 × $0.25 = $2.50 per Scan
  1 Scan/day × 30 days = $75/month — for one query, mostly discarded data

FinOps fix: Query with KeyConditionExpression
  Query by PK = "STATUS#PENDING" with limit 1,000 → reads only PENDING items
  Cost: 1,000 items × 1KB = 1,000 RRU = $0.00025 per query
  → 99.99% cost reduction vs Scan for same result

4.3 API Gateway Cost Model

REST API:  $3.50 per million requests + $0.09/GB data transfer
HTTP API:  $1.00 per million requests + $0.09/GB data transfer

At 100M requests/month:
  REST API:  $350 + data transfer
  HTTP API:  $100 + data transfer
  Savings if HTTP API is functionally equivalent: $250/month (71% reduction)

Summary

Signal Tool Key config
Structured logs Powertools Logger injectLambdaContext middleware + correlationId propagation
Custom metrics Powertools Metrics + EMF logMetrics middleware — zero API call overhead
Distributed traces Powertools Tracer + X-Ray captureAWSv3Client for SDK auto-instrumentation; annotations for searchability
Lambda cost Power Tuning Find GB-second minimum for each function's workload
DynamoDB cost Replace Scan with Query KeyConditionExpression instead of FilterExpression
API Gateway cost HTTP API over REST 71% cheaper for identical Lambda Proxy use cases

What's Next

In Part 13: From Mid-Level to Senior — Production Readiness Checklist for Serverless Systems, we synthesize the entire series into a production readiness framework: the five domains every senior engineer must verify before a serverless system serves real traffic, and the mental model shift that separates "my function works" from "my system participates correctly in a distributed environment under partial failure."

Research & Synthesis Note

This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.

#CloudWatch#X-Ray#Observability#FinOps#Lambda Powertools#EMF#AWS
Siddhant Deval

Written by Siddhant Deval

Senior Full-Stack Engineer building high-scale architectures, browser performance engineering systems, and SaaS platforms.