Siddhant Deval
Siddhant Deval
backend20 min read

API Gateway Advanced: VTL Templates, Direct Service Integrations & WAF

The most impactful API Gateway optimization is removing Lambda from routes that don't require it. This article covers VTL mapping templates, direct service integrations (API GW → DynamoDB, SQS, EventBridge without Lambda), WebSocket connection tracking, request-parameter Lambda Authorizers, IAM SigV4 for machine-to-machine APIs, and WAF wiring for rate-based and managed rule groups.

API Gateway Advanced: VTL Templates, Direct Service Integrations & WAF

Every AWS primitive is a tradeoff surface, not a feature toggle. The most expensive Lambda function in a serverless architecture is sometimes one that does nothing except call another AWS service — reading an HTTP body, constructing an SQS message, and returning a 202. Every one of those invocations pays a cold-start risk, a billing dimension, and a failure surface that does not need to exist. API Gateway's VTL mapping templates enable direct integrations with DynamoDB, SQS, EventBridge, and other AWS services, removing Lambda from the hot path entirely. This article closes the gap between "I use Lambda Proxy for everything" and "I know which routes genuinely require Lambda and which can be served faster and cheaper by a direct integration."


1. VTL Mapping Templates — The Three Namespaces

VTL (Velocity Template Language) is the transformation engine built into REST API. A mapping template receives the incoming HTTP request and transforms it into the payload the integration target expects. It has three namespaces:

Namespace Purpose Key methods
$input Access the raw request body and path/query parameters $input.json('$.field'), $input.body, $input.params('header')
$context Access request metadata (requestId, accountId, stage, auth context) $context.requestId, $context.authorizer.userId
$util Encoding and escaping utilities $util.urlEncode(), $util.escapeJavaScript(), $util.parseJson()
Crucial Requirement

VTL templates have no network access, no state, no loops (beyond #foreach over arrays), and no conditionals beyond #if/#else. They are pure transformation functions. Any operation requiring an external API call, database lookup, or complex business logic requires Lambda — VTL cannot replace that.

1.1 Reading the Request Body

VTL
## Request template: map HTTP body fields into a DynamoDB PutItem payload
## API route: POST /orders

{
  "TableName": "Orders",
  "Item": {
    "orderId": { "S": "$context.requestId" },
    "userId":  { "S": "$input.json('$.userId')" },
    "total":   { "N": "$input.json('$.total')" },
    "status":  { "S": "pending" },
    "createdAt": { "S": "$context.requestTimeEpoch" }
  },
  "ConditionExpression": "attribute_not_exists(orderId)"
}
VTL
## Response template: transform DynamoDB response into clean HTTP JSON
## $input.path('$') accesses the full DynamoDB response body

#if($input.path('$.ConditionalCheckFailedException') != "")
  #set($context.responseOverride.status = 409)
  { "error": "Order already exists" }
#else
  {
    "orderId": "$context.requestId",
    "status": "created"
  }
#end

1.2 Accessing Headers and Query Parameters

VTL
## Access Authorization header (useful in custom integration auth flows)
#set($token = $input.params('Authorization'))

## Access query string parameter
#set($page = $input.params('page'))
#set($limit = $input.params('limit'))

## URL-encode a value for use in a downstream URL (e.g., EventBridge detail)
$util.urlEncode($input.json('$.callbackUrl'))

## Escape a value for safe inclusion in a JSON string
"message": "$util.escapeJavaScript($input.json('$.userInput'))"

2. Direct Service Integrations — Removing Lambda from the Hot Path

Direct service integrations wire API Gateway directly to an AWS service using VTL to transform the request. The integration target is the AWS service API endpoint — not a Lambda function.

2.1 API Gateway → SQS (The Most Common Direct Integration)

TYPESCRIPT
// CDK: REST API → SQS direct integration
import { AwsIntegration, PassthroughBehavior } from 'aws-cdk-lib/aws-apigateway'
import { Queue } from 'aws-cdk-lib/aws-sqs'
import { Role, ServicePrincipal, PolicyStatement } from 'aws-cdk-lib/aws-iam'

const queue = new Queue(this, 'OrderQueue')

// IAM role for API Gateway to call SQS
const integrationRole = new Role(this, 'ApiGwSqsRole', {
  assumedBy: new ServicePrincipal('apigateway.amazonaws.com'),
})
integrationRole.addToPolicy(new PolicyStatement({
  actions: ['sqs:SendMessage'],
  resources: [queue.queueArn],
}))

const sqsIntegration = new AwsIntegration({
  service: 'sqs',
  path: `${this.account}/${queue.queueName}`,
  integrationHttpMethod: 'POST',
  options: {
    credentialsRole: integrationRole,
    passthroughBehavior: PassthroughBehavior.NEVER,
    requestParameters: {
      'integration.request.header.Content-Type': "'application/x-www-form-urlencoded'",
    },
    requestTemplates: {
      'application/json': `Action=SendMessage&MessageBody=$util.urlEncode($input.body)&MessageGroupId=$input.json('$.orderId')`,
    },
    integrationResponses: [{
      statusCode: '202',
      responseTemplates: {
        'application/json': '{ "queued": true, "requestId": "$context.requestId" }'
      }
    }]
  }
})

api.root.addResource('orders').addMethod('POST', sqsIntegration, {
  methodResponses: [{ statusCode: '202' }]
})

Before (Lambda pass-through) vs After (direct integration):

Dimension Lambda pass-through Direct SQS integration
Failure surfaces API GW + Lambda cold start + Lambda execution + SQS API GW + SQS
Cold start risk Present None
Cost REST API + Lambda invocation + GB-seconds REST API only
Latency P50 40–200ms (cold), 10–30ms (warm) ~5ms
Code to maintain Handler function VTL template (20 lines)

2.2 API Gateway → DynamoDB

VTL
## Request template: API GW → DynamoDB GetItem
## Route: GET /users/{userId}

{
  "TableName": "Users",
  "Key": {
    "PK": { "S": "USER#$input.params('userId')" },
    "SK": { "S": "PROFILE" }
  }
}
VTL
## Response template: DynamoDB → HTTP JSON
## DynamoDB returns { "Item": { "name": { "S": "Alice" }, "email": { "S": "..." } } }

#set($item = $input.path('$.Item'))
{
  "userId": "$input.params('userId')",
  "name":   "$item.name.S",
  "email":  "$item.email.S"
}

2.3 API Gateway → EventBridge

VTL
## Request template: POST /events → EventBridge PutEvents
## Transforms HTTP body into EventBridge event format

{
  "Entries": [{
    "Source": "com.myapp.api",
    "DetailType": "$input.json('$.eventType')",
    "Detail": "$util.escapeJavaScript($input.body)",
    "EventBusName": "myapp-events"
  }]
}
Before (Client→REST API→Lambda→SQS, 4 failure surfaces) vs After (Client→REST API→SQS direct via VTL, 2 failure surfaces) with cost delta and latency improvement annotated
Before (Client→REST API→Lambda→SQS, 4 failure surfaces) vs After (Client→REST API→SQS direct via VTL, 2 failure surfaces) with cost delta and latency improve…

3. WebSocket APIs

WebSocket API provides persistent bidirectional connections. Unlike REST/HTTP APIs, the connection lifecycle is stateful — each client gets a unique connectionId that persists for the connection lifetime.

3.1 Connection Lifecycle

Client connects   → $connect route → Lambda validates auth → stores connectionId in DynamoDB
Client sends msg  → $default route → Lambda processes, can push back via Management API
Server pushes msg → Lambda uses connectionId to call Management API → client receives
Client disconnects → $disconnect route → Lambda removes connectionId from DynamoDB
TYPESCRIPT
// DynamoDB: store connection IDs for server-initiated pushes
export const connectHandler = async (event: AWSLambda.APIGatewayProxyWebsocketEventV2) => {
  const connectionId = event.requestContext.connectionId

  await docClient.send(new PutCommand({
    TableName: process.env.CONNECTIONS_TABLE!,
    Item: {
      PK: `CONN#${connectionId}`,
      SK: 'METADATA',
      userId: event.requestContext.authorizer?.userId,
      connectedAt: new Date().toISOString(),
      ttl: Math.floor(Date.now() / 1000) + (2 * 60 * 60), // 2-hour TTL for auto-cleanup
    }
  }))
  return { statusCode: 200, body: 'Connected' }
}

// Server-initiated push: Lambda uses the Management API to push to a specific connection
export const broadcastHandler = async (event: { userId: string, message: string }) => {
  const { Items } = await docClient.send(new QueryCommand({
    TableName: process.env.CONNECTIONS_TABLE!,
    KeyConditionExpression: 'begins_with(PK, :prefix)',
    ExpressionAttributeValues: { ':prefix': 'CONN#' }
  }))

  const endpoint = `https://${process.env.API_ID}.execute-api.${process.env.REGION}.amazonaws.com/${process.env.STAGE}`
  const client = new ApiGatewayManagementApiClient({ endpoint })

  await Promise.allSettled(Items!.map(async (item) => {
    try {
      await client.send(new PostToConnectionCommand({
        ConnectionId: item.PK.replace('CONN#', ''),
        Data: Buffer.from(JSON.stringify({ message: event.message }))
      }))
    } catch (err: any) {
      if (err.statusCode === 410) {
        // GoneException: connection closed without triggering $disconnect
        await docClient.send(new DeleteCommand({
          TableName: process.env.CONNECTIONS_TABLE!,
          Key: { PK: item.PK, SK: 'METADATA' }
        }))
      }
    }
  }))
}
Performance / Safety Warning

WebSocket connections that close without triggering $disconnect (network drop, tab close) leave stale connectionId records in DynamoDB. Attempting to push to a stale connection returns GoneException (HTTP 410). Handle 410 responses by deleting the stale record. Set a DynamoDB TTL on connection records (2–4 hours) as a safety net for orphaned connections.


4. Advanced Authorization Patterns

4.1 Request-Parameter Lambda Authorizer

Unlike token-based authorizers (which receive only the bearer token), request-parameter authorizers receive headers, query parameters, path parameters, and stage variables — enabling multi-factor authorization decisions:

TYPESCRIPT
// REQUEST-type Lambda Authorizer event
export const requestAuthorizer = async (
  event: AWSLambda.APIGatewayRequestAuthorizerEvent
): Promise<AWSLambda.APIGatewayAuthorizerResult> => {
  const apiKey = event.headers?.['x-api-key']
  const tenantId = event.headers?.['x-tenant-id']
  const jwtToken = event.headers?.Authorization?.replace('Bearer ', '')

  if (!apiKey || !tenantId || !jwtToken) {
    throw new Error('Unauthorized')
  }

  // Multi-factor: validate JWT AND verify API key belongs to tenant
  const [jwtPayload, tenantConfig] = await Promise.all([
    verifyJwt(jwtToken),
    getTenantConfig(tenantId, apiKey) // DynamoDB lookup
  ])

  if (jwtPayload.tenantId !== tenantId) throw new Error('Unauthorized')

  return {
    principalId: jwtPayload.sub,
    policyDocument: allowPolicy(event.methodArn),
    context: { tenantId, tier: tenantConfig.tier }
  }
}

4.2 IAM Authorization with SigV4 (Machine-to-Machine)

For internal service-to-service APIs where the caller is an AWS resource (another Lambda, an EC2 instance, an ECS task):

TYPESCRIPT
// Client Lambda calling an IAM-authorized API
import { SignatureV4 } from '@smithy/signature-v4'
import { Sha256 } from '@aws-crypto/sha256-js'

const signer = new SignatureV4({
  credentials: fromEnv(),  // Uses Lambda's execution role credentials automatically
  region: process.env.AWS_REGION!,
  service: 'execute-api',
  sha256: Sha256,
})

const url = new URL('https://abc123.execute-api.us-east-1.amazonaws.com/prod/orders')
const signed = await signer.sign({
  method: 'POST',
  hostname: url.hostname,
  path: url.pathname,
  headers: { 'Content-Type': 'application/json', host: url.hostname },
  body: JSON.stringify({ orderId: '42' }),
})

const response = await fetch(url.toString(), {
  method: signed.method,
  headers: signed.headers,
  body: signed.body,
})

5. WAF Integration

AWS WAF integrates with REST API (and HTTP API via CloudFront) to inspect and filter requests before they reach the API Gateway.

5.1 Rate-Based Rules

TYPESCRIPT
// CDK: WAF WebACL with rate-based rule for API Gateway
import { CfnWebACL } from 'aws-cdk-lib/aws-wafv2'

const webAcl = new CfnWebACL(this, 'ApiWaf', {
  scope: 'REGIONAL',
  defaultAction: { allow: {} },
  rules: [
    {
      name: 'RateLimitByIP',
      priority: 1,
      action: { block: {} },
      visibilityConfig: { sampledRequestsEnabled: true, cloudWatchMetricsEnabled: true, metricName: 'RateLimit' },
      statement: {
        rateBasedStatement: {
          limit: 2000,            // 2000 requests per 5 minutes per IP
          aggregateKeyType: 'IP',
        }
      }
    },
    {
      name: 'AWSManagedRulesSQLi',
      priority: 2,
      overrideAction: { none: {} },
      visibilityConfig: { sampledRequestsEnabled: true, cloudWatchMetricsEnabled: true, metricName: 'SQLi' },
      statement: {
        managedRuleGroupStatement: {
          vendorName: 'AWS',
          name: 'AWSManagedRulesSQLiRuleSet',
        }
      }
    }
  ],
  // ... visibilityConfig
})

// Associate WAF with REST API stage
new CfnWebACLAssociation(this, 'WafAssociation', {
  resourceArn: `arn:aws:apigateway:${this.region}::/restapis/${api.restApiId}/stages/${api.deploymentStage.stageName}`,
  webAclArn: webAcl.attrArn,
})
Pro Tip & Optimization

WAF rate-based rules count by IP by default. Attackers using rotating residential proxies or VPNs bypass per-IP rules trivially. For APIs that require abuse-resistant rate limiting, add a JA3 fingerprint-based rule (available via AWS WAF Fraud Control) or rate-limit on a custom header that is harder to rotate than an IP address (e.g., a stable device fingerprint token).

VTL template execution: HTTP request body → $input.json() → DynamoDB PutItem request mapping → DynamoDB response → $util.escapeJavaScript() → HTTP response; each VTL variable namespace labeled
VTL template execution: HTTP request body → $input.json() → DynamoDB PutItem request mapping → DynamoDB response → $util.escapeJavaScript() → HTTP response;…

Summary

Concept Rule
Direct SQS integration Removes Lambda from the enqueue path — eliminates cold start, Lambda billing, and two failure surfaces
VTL namespaces $input (request data), $context (request metadata), $util (encoding) — no network, no loops, pure transformation
WebSocket connection IDs Store in DynamoDB on $connect; delete on $disconnect and on 410 GoneException from push
Request-parameter authorizer Accesses headers, query params, stage variables — enables multi-factor auth decisions beyond the bearer token
WAF per-IP rate limiting Bypassed by rotating-IP attacks; add JA3 fingerprinting or header-based identification

What's Next

In Part 5: AppSync Foundations — Schema, Unit Resolvers & Direct Data Sources, we shift from REST to managed GraphQL: why wiring every AppSync operation to a Lambda resolver is the same anti-pattern as using Lambda as a SQS pass-through, and how unit resolvers connect AppSync directly to DynamoDB with zero Lambda code and sub-10ms data access latency.

Research & Synthesis Note

This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.

#API Gateway#VTL#Direct Integrations#WAF#WebSocket#SigV4#Serverless
Siddhant Deval

Written by Siddhant Deval

Senior Full-Stack Engineer building high-scale architectures, browser performance engineering systems, and SaaS platforms.