Siddhant Deval
Siddhant Deval
backend18 min read

API Gateways: Routing, Auth Offloading, Rate Limiting & Trace Initiation

An API Gateway is not a reverse proxy — it is the system's single policy enforcement point for routing, auth offloading, rate limiting, circuit breaking, and trace initiation. Teams that skip it re-implement all of these incorrectly in every downstream service. This article builds a complete gateway understanding from routing mechanics through distributed rate limiting to OpenTelemetry root span injection.

API Gateways: Routing, Auth Offloading, Rate Limiting & Trace Initiation

Senior engineers don't just wire services together — they design the boundary: the contract, the trust model, the failure envelope, and the signal pipeline that proves it's working. The API Gateway is that boundary made concrete. Most teams discover they need one after they have already implemented JWT validation in seven services (each slightly differently), added rate limiting middleware to five of them (with incompatible counters), and tried to correlate an incident across logs from twelve services with no shared request identifier. By then, the gateway is a refactor, not a design. This article builds it correctly — from first principles.

Architectural Note

Series positioning: This is Part 2 of the API Architecture & System Resilience series. It builds on the RESTful design vocabulary from Part 1 and establishes the enforcement layer that all subsequent articles depend on — Part 3 (Service Mesh), Part 5 (Auth Architecture), and Part 6 (Observability) all assume a gateway as the entry point.


1. Gateway vs BFF vs Reverse Proxy vs Service Mesh

These four patterns are frequently conflated. They solve different problems at different layers of the stack.

Pattern Owns What Protocol-Aware? Auth Enforcement Observability Role
Reverse Proxy (NGINX, HAProxy) TLS termination, load balancing, static routing HTTP method only None Access logs
API Gateway (Kong, AWS APIGW, Apigee) Request policy enforcement (auth, rate-limit, routing, transform, trace) Full HTTP + gRPC JWT/OAuth2 validation Root span initiation
BFF (Backend-for-Frontend) Client-shaped data aggregation Full HTTP + gRPC → JSON Delegates to gateway Downstream span
Service Mesh (Istio, Linkerd) East-west (service-to-service) traffic, mTLS, telemetry L4/L7 (sidecar) mTLS workload identity Automatic span per hop
Crucial Requirement

A BFF is not a gateway. A BFF owns client contracts and aggregates data for a specific client type (desktop, mobile). A gateway owns cross-cutting policy: who is allowed in, how fast, and where the request goes. A BFF sits behind the gateway. Merging them creates a service that is too large to own by a single team and too coupled to evolve independently.


2. Dynamic Routing & Service Registry Integration

TYPESCRIPT
// ❌ Static route configuration — breaks every time a service moves
// In nginx.conf:
// upstream order-service { server 10.0.1.45:3000; }
// When the container restarts, its IP changes. The gateway returns 502.

// ✅ Dynamic routing via Kubernetes service discovery
// Kong declarative config (kong.yaml):
services:
  - name: order-service
    url: http://order-service.default.svc.cluster.local:3000
    # Kubernetes CoreDNS resolves this to the current pod IPs automatically
    # When pods restart, the DNS record updates — gateway never touches a static IP

routes:
  - name: order-routes
    service: order-service
    paths:
      - /api/v1/orders
    strip_path: true
    # Strips /api/v1 prefix before forwarding to the service

3. JWT Authentication Offloading

3.1 The Distributed Validation Anti-Pattern

TYPESCRIPT
// ❌ JWT validation in every service — 7 services, 7 slightly different implementations
// order-service/middleware/auth.ts
import jwt from 'jsonwebtoken'

export function authMiddleware(req: Request, res: Response, next: NextFunction) {
  const token = req.headers.authorization?.replace('Bearer ', '')
  if (!token) return res.status(401).json({ error: 'Missing token' })
  try {
    const decoded = jwt.verify(token, process.env.JWT_SECRET!) // HS256 — shared secret
    req.user = decoded
    next()
  } catch {
    return res.status(401).json({ error: 'Invalid token' })
  }
}
// Problems:
// - JWKS rotation requires restarting all 7 services simultaneously
// - Some services check 'exp', others don't check 'aud' — inconsistent security posture
// - Every service is an availability dependency on the auth server if using token introspection

3.2 Gateway-Level JWT Validation

YAML
# Kong JWT plugin — validates at the gateway perimeter, never reaches services
plugins:
  - name: jwt
    config:
      secret_is_base64: false
      claims_to_verify:
        - exp
        - nbf
      key_claim_name: kid
      # Gateway fetches public keys from JWKS endpoint automatically
      # Invalid tokens are rejected with 401 before hitting any service
TYPESCRIPT
// ✅ Service receives pre-validated identity via gateway-injected headers
// order-service/middleware/auth.ts — no JWT validation code
export function authMiddleware(req: Request, res: Response, next: NextFunction) {
  const userId = req.headers['x-user-id'] as string
  const roles  = (req.headers['x-roles'] as string)?.split(',') ?? []

  if (!userId) {
    // Gateway should have caught this — but defense in depth: reject if missing
    return res.status(401).json({ error: 'Missing identity headers' })
  }

  req.user = { id: userId, roles }
  next()
}
// Services trust the gateway. The gateway is the single auth enforcement point.
// Rotating the JWT signing key requires updating only the gateway's JWKS cache.
API Gateway request lifecycle flow trace showing sequential enforcement pipeline: TLS termination → JWT validation (401 on failure) → rate limit check (429 on exceeded) → circuit breaker check (503 if open) → trace ID injection → protocol translation → upstream service routing.
API Gateway request lifecycle flow trace showing sequential enforcement pipeline: TLS termination → JWT validation (401 on failure) → rate limit check (429 o…

4. Distributed Rate Limiting

4.1 The Per-Instance Counter Failure

TYPESCRIPT
// ❌ In-memory rate limiter — 3 gateway instances = 3× the configured limit
import rateLimit from 'express-rate-limit'

app.use(rateLimit({
  windowMs: 60_000,    // 1 minute
  max: 100,            // 100 requests per minute — per instance
  // With 3 gateway instances behind a load balancer:
  // Client can make 100 requests to instance 1, 100 to instance 2, 100 to instance 3
  // Actual effective limit: 300 requests/minute — 3× the intended limit
}))

4.2 Redis-Backed Distributed Rate Limiting

TYPESCRIPT
// ✅ Distributed rate limiter — accurate across all gateway replicas
import { RateLimiterRedis } from 'rate-limiter-flexible'
import { createClient } from 'redis'

const redis = createClient({ url: process.env.REDIS_URL })
await redis.connect()

const rateLimiter = new RateLimiterRedis({
  storeClient: redis,
  keyPrefix: 'rl:gateway',
  points: 100,          // 100 requests
  duration: 60,         // per 60 seconds
  blockDuration: 60,    // block for 60s after limit exceeded
})

export async function rateLimitMiddleware(req: Request, res: Response, next: NextFunction) {
  const key = req.headers['x-user-id'] ?? req.ip   // Per-user or per-IP
  try {
    const result = await rateLimiter.consume(key)
    res.setHeader('X-RateLimit-Remaining', result.remainingPoints)
    res.setHeader('X-RateLimit-Reset', new Date(Date.now() + result.msBeforeNext).toISOString())
    next()
  } catch (rejRes) {
    res.setHeader('Retry-After', Math.round(rejRes.msBeforeNext / 1000))
    res.setHeader('X-RateLimit-Remaining', 0)
    res.status(429).json({
      type: 'https://api.example.com/problems/rate-limit-exceeded',
      title: 'Rate Limit Exceeded',
      status: 429,
      detail: `Rate limit of 100 requests/minute exceeded. Retry after ${Math.round(rejRes.msBeforeNext / 1000)}s.`,
    })
  }
}
// All 3 gateway instances share the same Redis counter.
// The limit is accurate regardless of which instance the load balancer routes to.
Crucial Requirement

Always set the Retry-After header on 429 responses. Clients that do not receive a Retry-After value will retry immediately, converting a rate-limit event into a thundering herd. See Part 7: Retry Engineering for how clients should consume this header correctly.


5. Global Circuit Breaking at the Gateway

TYPESCRIPT
// ✅ Gateway-level circuit breaker — protects all downstream services uniformly
import CircuitBreaker from 'opossum'

// Per-service circuit breaker at the gateway level
const circuitBreakers = new Map<string, CircuitBreaker>()

function getBreaker(serviceName: string) {
  if (!circuitBreakers.has(serviceName)) {
    const breaker = new CircuitBreaker(
      (req: ProxyRequest) => forwardToService(serviceName, req),
      {
        timeout: 5000,
        errorThresholdPercentage: 50,
        resetTimeout: 30_000,
        volumeThreshold: 10,
      }
    )
    breaker.fallback(() => ({ status: 503, body: {
      type: 'https://api.example.com/problems/service-unavailable',
      title: 'Service Temporarily Unavailable',
      status: 503,
      detail: `${serviceName} is temporarily degraded. Please retry in 30 seconds.`,
    }}))
    circuitBreakers.set(serviceName, breaker)
  }
  return circuitBreakers.get(serviceName)!
}
Mental Model Check

The gateway circuit breaker is a system-level backstop. Individual services may have their own circuit breakers for inter-service calls. The gateway breaker protects the entire system from a client-visible cascade when a service starts failing — even if the service has no internal circuit breaker at all. Defense in depth: both layers should exist; the gateway layer is the last resort.


6. Protocol Translation: REST → gRPC

TYPESCRIPT
// ✅ External REST clients → gateway translates → internal gRPC services
// The public API remains REST/JSON; the internal service is gRPC-native

import * as grpc from '@grpc/grpc-js'
import * as protoLoader from '@grpc/proto-loader'

const packageDef = protoLoader.loadSync('order.proto')
const orderProto = grpc.loadPackageDefinition(packageDef).order as any
const orderClient = new orderProto.OrderService(
  'order-service:50051',
  grpc.credentials.createInsecure()
)

// Express REST endpoint on the gateway — translates to gRPC upstream
app.get('/api/v1/orders/:id', async (req, res) => {
  orderClient.getOrder({ orderId: req.params.id }, (err: any, response: any) => {
    if (err) {
      if (err.code === grpc.status.NOT_FOUND) return res.status(404).json({
        type: 'https://api.example.com/problems/not-found',
        title: 'Order Not Found',
        status: 404,
        instance: `/orders/${req.params.id}`,
      })
      return res.status(502).json({ type: 'https://api.example.com/problems/upstream-error', status: 502 })
    }
    res.json(response) // gRPC Protobuf response serialized to JSON for the REST client
  })
})

7. OpenTelemetry Trace ID Initiation

TYPESCRIPT
// ✅ Gateway generates the root trace span — all downstream services inherit it
import { NodeSDK } from '@opentelemetry/sdk-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
import { ExpressInstrumentation } from '@opentelemetry/instrumentation-express'
import { HttpInstrumentation } from '@opentelemetry/instrumentation-http'
import { context, trace, propagation } from '@opentelemetry/api'
import { W3CTraceContextPropagator } from '@opentelemetry/core'

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({ url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT }),
  instrumentations: [new HttpInstrumentation(), new ExpressInstrumentation()],
  serviceName: 'api-gateway',
})
sdk.start()

// Gateway middleware: inject traceparent header before forwarding
app.use((req, res, next) => {
  const span = trace.getActiveSpan()
  if (span) {
    const carrier: Record<string, string> = {}
    propagation.inject(context.active(), carrier)
    // carrier now contains: { 'traceparent': '00-{traceId}-{spanId}-01' }
    // Forward all trace context headers to upstream services
    Object.entries(carrier).forEach(([key, val]) => req.headers[key] = val)
  }
  next()
})
Pro Tip & Optimization

Return the X-Trace-Id header in all gateway responses. Client-facing error messages can say "Contact support with trace ID: abc123" — support engineers can then pull the full distributed trace in Jaeger or Grafana Tempo in seconds without needing to reproduce the issue.


8. API Composition & Fan-Out

TYPESCRIPT
// ✅ Gateway aggregates: one client request → multiple upstream calls → one response
// Reduces network round trips from 4 to 1 for a product detail page

app.get('/api/v1/product-details/:id', async (req, res) => {
  const { id } = req.params
  const userId = req.headers['x-user-id'] as string

  // Fan out in parallel — all 4 calls run concurrently
  const [product, inventory, pricing, wishlist] = await Promise.allSettled([
    productClient.getProduct({ id }),
    inventoryClient.getStock({ sku: id }),
    pricingClient.getPrice({ sku: id, userId }),
    wishlistClient.checkWishlist({ sku: id, userId }),
  ])

  res.json({
    product: product.status === 'fulfilled' ? product.value : null,
    inventory: inventory.status === 'fulfilled' ? inventory.value : { available: null },
    pricing: pricing.status === 'fulfilled' ? pricing.value : null,
    inWishlist: wishlist.status === 'fulfilled' ? wishlist.value.inList : false,
    // Partial failures are surfaced as null — client degrades gracefully
  })
})
// Total latency = max(4 parallel calls), not sum(4 sequential calls)
Comparison matrix: API Gateway vs BFF vs Service Mesh vs Reverse Proxy across role, traffic direction (north-south vs east-west), protocol awareness, auth enforcement, primary use case, and examples.
Comparison matrix: API Gateway vs BFF vs Service Mesh vs Reverse Proxy across role, traffic direction (north-south vs east-west), protocol awareness, auth en…

Summary

Concern Gateway Rule
Routing DNS-based dynamic routing via k8s service registry — never static IPs
Auth JWT validated at gateway perimeter; services receive X-User-Id + X-Roles headers
Rate Limiting Redis-backed distributed counter — per-user or per-IP key; always set Retry-After
Circuit Breaking Gateway-level breaker per service; returns 503 fallback when OPEN
Protocol Translation External REST → Internal gRPC; public contract is decoupled from internal protocol
Trace Initiation Gateway generates root span; injects traceparent into all upstream requests
Composition Fan-out to multiple services in parallel; surface partial failures as null fields

What's Next

In Part 3, we go deeper into the east-west traffic layer — Part 3: Service Mesh & mTLS examines how Istio's Envoy sidecar provides automatic mTLS, workload identity, and declarative traffic management between every service pair, without any application code changes.

Research & Synthesis Note

This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.

#API Gateway#Kong#Rate Limiting#JWT#OpenTelemetry#Microservices
Siddhant Deval

Written by Siddhant Deval

Senior Full-Stack Engineer building high-scale architectures, browser performance engineering systems, and SaaS platforms.