Siddhant Deval
Siddhant Deval
backend17 min read

The SAGA Pattern: Orchestrating Distributed Workflows Across Service Boundaries

The SAGA pattern coordinates long-running distributed transactions by decomposing them into a sequence of local transactions, each with a compensating rollback. The critical choice between choreography (event-driven) and orchestration (centralized command) determines failure observability, deadlock risk, and operational complexity across the entire workflow.

The SAGA Pattern: Orchestrating Distributed Workflows Across Service Boundaries

Every boundary is a failure isolation decision — and when a business transaction must span multiple service boundaries, you can no longer rely on a single database transaction to roll everything back on failure. Consider the order placement flow: to complete an order, you must reserve inventory, charge payment, and notify fulfillment — three local transactions across three independent services with three independent databases. If the payment charge succeeds but the fulfillment notification fails, you now have a charged customer with no fulfillment record. ACID guarantees cannot reach across service boundaries. The SAGA pattern is the response: decompose the transaction into a sequence of local transactions, each with an explicit compensating rollback, designed to run as if failure in any step is the default case.

Architectural Note

Series positioning: This is Part 8 of the Distributed Architecture & System Design series. Building on CQRS and messaging topologies, this article explores multi-service workflow coordination via the SAGA pattern, contrasting choreography and orchestration before we examine the Transactional Outbox and CDC in Part 9: Distributed Transactions & CDC.


1. The Two-Phase Commit Problem Across Services

TYPESCRIPT
// ❌ Naive multi-service transaction — no rollback if any step fails
async function placeOrder(cmd: PlaceOrderCommand): Promise<void> {
  // Step 1: Reserve inventory (Inventory Service)
  await inventoryClient.reserve(cmd.productId, cmd.quantity)
  // If crash here: inventory reserved, payment not charged, no order record

  // Step 2: Charge payment (Payment Service)
  const charge = await paymentClient.charge(cmd.userId, cmd.totalCents)
  // If crash here: inventory reserved, payment charged, no order record

  // Step 3: Create order record (Order Service)
  await orderRepository.save({ ...cmd, chargeId: charge.id, status: 'confirmed' })
  // If crash here: inventory reserved, payment charged, no order record in DB
}
// There is no global rollback mechanism across these three services
// Each failure scenario leaves the system in a different inconsistent state
Mental Model Check

A SAGA replaces "atomicity across services" (which doesn't exist) with "eventual consistency via compensating transactions" (which does). The key insight is that each step in the SAGA must have a corresponding compensating transaction that reverses its effect — and each compensating transaction must itself be idempotent and guaranteed to succeed.


2. Choreography SAGAs

In a choreography SAGA, services react to events published by other services. There is no central coordinator — each service knows only its own step and which events to emit next.

2.1 Choreography Flow

2.2 Choreography Compensation

TYPESCRIPT
// ✅ Choreography compensation — each service handles its own rollback
// When PaymentFailed is published, Inventory Service must release the reserved stock

class InventoryService {
  async onPaymentFailed(event: PaymentFailedEvent): Promise<void> {
    // Find the reservation created for this order
    const reservation = await this.reservationRepository.findByOrderId(event.orderId)
    if (!reservation) return // idempotent — already released or never reserved

    await this.reservationRepository.release(reservation.id)
    await eventBus.publish('StockReservationReleased', {
      orderId: event.orderId,
      productId: reservation.productId,
      quantity: reservation.quantity,
      releasedAt: new Date().toISOString(),
    })
  }
}

2.3 Choreography Limitations

Problem Description
No saga status query "Is this order's SAGA complete?" requires querying every participant service
Distributed event graph The workflow logic is spread across 4+ services — understanding it requires reading all of them
Livelock risk Circular compensations (A compensates B's failure, B compensates C's failure, C compensates A) can loop indefinitely
Observability gap A saga stalled mid-flight is invisible without cross-service event correlation

3. Orchestration SAGAs

In an orchestration SAGA, a central saga orchestrator (a durable state machine) issues commands to each participant service and handles compensations centrally. Each service is a pure executor — it performs work when commanded and emits results.

3.1 Orchestration Flow

SAGA workflow execution trace showing forward progress through Order, Payment, and Inventory reservation failure triggering reverse compensating transactions to release hold and refund payment.
SAGA workflow execution trace showing forward progress through Order, Payment, and Inventory reservation failure triggering reverse compensating transactions…

3.2 State Machine Implementation

TYPESCRIPT
// ✅ Orchestration SAGA — explicit state machine with compensating transactions
type OrderSagaState =
  | 'RESERVING_INVENTORY'
  | 'CHARGING_PAYMENT'
  | 'CREATING_FULFILLMENT'
  | 'COMPENSATING_PAYMENT'
  | 'COMPENSATING_INVENTORY'
  | 'SAGA_COMPLETE'
  | 'SAGA_FAILED'

interface OrderSagaInstance {
  sagaId: string
  orderId: string
  userId: string
  state: OrderSagaState
  reservationId?: string
  chargeId?: string
  retryCount: number
  updatedAt: Date
}

class OrderSagaOrchestrator {
  async handleEvent(saga: OrderSagaInstance, event: SagaEvent): Promise<OrderSagaInstance> {
    switch (saga.state) {
      case 'RESERVING_INVENTORY':
        if (event.type === 'StockReserved') {
          await inventoryClient.reserve(saga.orderId, event.reservationId)
          await commandBus.send('ChargeCard', { orderId: saga.orderId, userId: saga.userId })
          return { ...saga, state: 'CHARGING_PAYMENT', reservationId: event.reservationId }
        }
        if (event.type === 'StockReservationFailed') {
          return { ...saga, state: 'SAGA_FAILED' } // nothing to compensate
        }
        break

      case 'CHARGING_PAYMENT':
        if (event.type === 'PaymentCharged') {
          await commandBus.send('CreateFulfillment', { orderId: saga.orderId })
          return { ...saga, state: 'CREATING_FULFILLMENT', chargeId: event.chargeId }
        }
        if (event.type === 'PaymentFailed') {
          // Begin compensation — release the inventory reservation
          await commandBus.send('ReleaseStock', { orderId: saga.orderId, reservationId: saga.reservationId! })
          return { ...saga, state: 'COMPENSATING_INVENTORY' }
        }
        break

      case 'COMPENSATING_INVENTORY':
        if (event.type === 'StockReleased') {
          return { ...saga, state: 'SAGA_FAILED' }
        }
        // StockRelease failed — must retry until success (cannot leave inventory reserved)
        if (event.type === 'StockReleaseFailed') {
          await commandBus.send('ReleaseStock', { orderId: saga.orderId, reservationId: saga.reservationId! })
          return { ...saga, state: 'COMPENSATING_INVENTORY', retryCount: saga.retryCount + 1 }
        }
        break

      // ... additional states
    }
    return saga
  }
}
Crucial Requirement

Every compensating transaction must be idempotent and must never fail permanently. A compensation that throws causes the saga to enter a livelock — it cannot reach either SAGA_COMPLETE or SAGA_FAILED. If a compensation fails transiently, retry with exponential backoff. If it fails permanently, alert a human operator — the saga is stuck.


4. Temporal.io for Durable Saga Orchestration

Temporal.io provides a durable execution engine that persists saga workflow event history, allowing saga orchestrators to crash and recover without losing state:

TYPESCRIPT
// ✅ Temporal.io SAGA workflow — durable execution, automatic retry, crash-safe
import { proxyActivities, sleep } from '@temporalio/workflow'

const activities = proxyActivities<typeof orderActivities>({
  scheduleToCloseTimeout: '5 minutes',
  retry: { maximumAttempts: 5, initialInterval: '1s', backoffCoefficient: 2 }
})

export async function orderSagaWorkflow(input: OrderSagaInput): Promise<OrderSagaResult> {
  let inventoryReserved = false
  let paymentCharged = false

  try {
    // Step 1: Reserve inventory
    const reservation = await activities.reserveInventory(input.orderId, input.items)
    inventoryReserved = true

    // Step 2: Charge payment
    const charge = await activities.chargePayment(input.userId, input.totalCents, input.orderId)
    paymentCharged = true

    // Step 3: Create fulfillment
    await activities.createFulfillment(input.orderId, reservation.id, charge.id)

    return { status: 'success', orderId: input.orderId }
  } catch (err) {
    // Compensation — run in reverse order of successful steps
    if (paymentCharged) await activities.refundPayment(input.orderId)
    if (inventoryReserved) await activities.releaseInventory(input.orderId)

    return { status: 'failed', orderId: input.orderId, reason: err.message }
  }
  // If this process crashes mid-workflow: Temporal replays from the last durable checkpoint
  // Activities already completed are not re-executed — their results are replayed from history
}

5. Choreography vs Orchestration Decision Matrix

Choreography versus orchestration SAGA topologies comparing decentralized pinball event coupling with cyclic dependencies (left) against centralized durable orchestrator state machine directing commands and compensations (right).
Choreography versus orchestration SAGA topologies comparing decentralized pinball event coupling with cyclic dependencies (left) against centralized durable…
Criterion Choreography Orchestration
Single point of failure None — no central process Orchestrator process (mitigated with Temporal/k8s HA)
Saga status visibility Requires cross-service event correlation Single orchestrator state — queryable directly
Workflow logic location Distributed across all participants Centralized in the orchestrator
Compensation handling Each service handles its own Orchestrator issues compensating commands
Debugging failed saga Hard — requires log correlation across services Easy — orchestrator state machine has full history
Service coupling Participants know event names (loose) Participants know command names (slightly tighter)
Best for Simple 2–3 step workflows with stable participants Complex workflows (5+ steps), frequent failures, human approval steps
Pro Tip & Optimization

Start with choreography for simple flows. Migrate to orchestration when: (a) you cannot answer "is this saga complete?" without querying multiple services, (b) compensation logic involves more than 3 steps, or (c) workflow changes require coordinated releases across multiple teams.


6. SAGA Isolation and the ACD Property

SAGAs are not ACID — they are ACD: Atomic (via compensation), Consistent (eventually), Durable (each local transaction commits durably), but not Isolated. Concurrent SAGAs may observe each other's intermediate state:

TYPESCRIPT
// ❌ SAGA isolation hazard — two concurrent order sagas for the same product
// Saga A: reserves last 3 units (inventory = 0)
// Saga B: also reads inventory = 3 and attempts reservation — succeeds momentarily
// Saga A payment fails → compensates → releases 3 units
// Saga B payment succeeds → fulfillment tries to ship inventory that was double-reserved

// ✅ Mitigation: pessimistic lock on the inventory reservation step for the same SKU
// Or: idempotent reservation with version-checked inventory decrement
await db.transaction(async (trx) => {
  const result = await trx.raw(`
    UPDATE inventory SET reserved = reserved + $1
    WHERE sku = $2 AND (quantity_on_hand - reserved) >= $1
    RETURNING sku, quantity_on_hand, reserved
  `, [quantity, sku])

  if (result.rowCount === 0) throw new InsufficientStockError(sku)
})
// Atomic check-and-reserve prevents double-reservation at the inventory level

Summary

Architectural Concern Production Rule
SAGA Workflows Replaces distributed ACID with a sequence of local transactions coordinated with compensating rollbacks.
Choreography Tradeoffs No central coordinator; services react to events; harder to trace and debug when a workflow stalls mid-flight.
Orchestration Durability Central state machine issues commands and handles compensation; directly queryable; use Temporal.io for durable execution.
Compensating Idempotency Compensations must be idempotent and must never fail permanently; permanent compensation failure creates livelocks.
Isolation Anomalies Concurrent SAGAs may see each other's intermediate uncommitted state — mitigate with semantic locks at the resource level.

What's Next

In the series capstone, Part 9: Distributed Transactions, Transactional Outbox & CDC solves the fatal dual-write hazard between database transactions and message brokers, implementing Debezium WAL tailing and Inbox deduplication for end-to-end exactly-once business outcomes.

Research & Synthesis Note

This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.

#SAGA#Distributed Transactions#Microservices#Event-Driven#Backend
Siddhant Deval

Written by Siddhant Deval

Senior Full-Stack Engineer building high-scale architectures, browser performance engineering systems, and SaaS platforms.