Siddhant Deval
Siddhant Deval
backend20 min read

Saga Choreography and Event-Driven Workflow Patterns

Saga choreography distributes long-running business workflows across services via events, eliminating the central orchestrator as a bottleneck. This article designs a 4-step order saga with typed compensating transactions, causal ordering via correlation-id and causation-id headers, semantic locking, and the pivot transaction boundary — and shows when to reach for Temporal.io orchestration instead.

Saga Choreography and Event-Driven Workflow Patterns

An order spans four services: Payment, Inventory, Fulfillment, and Notification. The payment charges the card. Inventory reserves the items. Fulfillment picks and ships. Notification emails the customer. Each service has its own database, its own deployment, and its own failure modes. There is no shared transaction. If Inventory fails to reserve after Payment has charged, the customer's card is debited but the order cannot be fulfilled. The business must refund — and the system must do it automatically, without a human intervention.

The naive fix is a central Order Orchestrator that calls each service over HTTP and rolls back on failure. The orchestrator becomes the single point of failure for every order in the system, adds synchronous coupling between services, and recreates the distributed monolith it was meant to avoid.

Saga choreography distributes the workflow responsibility: each service listens for events from the previous step, performs its local work, and publishes an event for the next step. Compensation — the event-driven equivalent of ROLLBACK — flows backward through the same chain.

Architectural Note

Series positioning: This is Part 5 of Messaging at Cloud Scale. The prerequisite mental model is the Saga pattern from Saga Pattern: Choreography vs Orchestration and Compensating Transactions. This article provides a complete, production-hardened implementation with typed events, compensation chains, and the pivot transaction concept.


1. The Four-Step Order Saga

1.1 Forward Path: All Steps Succeed

1.2 The Event Contract

TYPESCRIPT
// ✅ Typed saga event interface — every event carries causal chain headers
interface SagaEvent {
  eventId:       string   // unique event ID (idempotency key for the consumer)
  correlationId: string   // original orderId — stable for the entire saga lifetime
  causationId:   string   // eventId of the event that caused THIS event
  orderId:       string
  timestamp:     number
}

interface OrderCreated extends SagaEvent {
  type:        'order.created'
  customerId:  string
  amountCents: number
  items:       Array<{ sku: string; quantity: number }>
}

interface InventoryReserved extends SagaEvent {
  type:              'inventory.reserved'
  reservationId:     string
  items:             Array<{ sku: string; quantity: number }>
}

interface InventoryReservationFailed extends SagaEvent {
  type:   'inventory.reservation.failed'
  reason: 'out_of_stock' | 'sku_not_found'
  skus:   string[]
}

// Compensation events — must be typed, not generic
interface PaymentRefunded extends SagaEvent {
  type:        'payment.refunded'
  refundId:    string
  amountCents: number
  reason:      string
}
Crucial Requirement

causation-id is the event ID of the event that caused the current event. correlation-id is the original business entity ID (orderId) that ties the entire saga together. Both must be present on every event from day one. With just correlation-id, you can find all events for an order. With causation-id, you can reconstruct the exact causal chain — critical for debugging compensations where a refund triggered a re-reservation that triggered another failure.


2. The Pivot Transaction and Compensation

2.1 Identifying the Pivot

The pivot transaction is the point of no return in a saga — the step whose effects cannot be compensated because they are externally visible or irreversible:

Before pivot — compensatable:
  PaymentAuthorized    → compensate: PaymentVoided (card not yet captured)
  InventoryReserved    → compensate: InventoryReleased (un-reserve SKUs)

Pivot:
  PaymentCaptured      → card is charged; compensation = refund (different operation)

After pivot — must retry until success:
  OrderShipped         → package is in transit; cannot un-ship
  TrackingNotified     → email sent; cannot un-send
TYPESCRIPT
// ✅ Saga compensation: Inventory service listens for InventoryReservationFailed
// to release reservations it holds when the saga is compensating

const consumer = kafka.consumer({ groupId: 'inventory-service-compensation' })
await consumer.subscribe({ topics: ['payment.voided', 'order.cancelled'] })

await consumer.run({
  eachMessage: async ({ message }) => {
    const event = JSON.parse(message.value!.toString()) as SagaEvent

    // Idempotency gate: compensation events processed twice must be safe
    const result = await db.query(`
      INSERT INTO compensation_log (idempotency_key, order_id, action, processed_at)
      VALUES ($1, $2, 'inventory.released', NOW())
      ON CONFLICT (idempotency_key) DO NOTHING
      RETURNING id
    `, [`${event.correlationId}:inventory.released`, event.orderId])

    if (result.rows.length === 0) return   // already compensated — idempotent

    // Perform compensation — append a release record, never delete the reservation
    await db.query(`
      INSERT INTO inventory_releases (reservation_id, order_id, released_at, reason)
      SELECT id, $1, NOW(), $2
      FROM inventory_reservations
      WHERE order_id = $1 AND status = 'reserved'
    `, [event.orderId, 'saga_compensation'])

    await db.query(`
      UPDATE inventory_reservations SET status = 'released' WHERE order_id = $1
    `, [event.orderId])

    await producer.send({
      topic:    'inventory.released',
      messages: [{
        key:   event.orderId,
        value: JSON.stringify({
          eventId:       crypto.randomUUID(),
          correlationId: event.correlationId,
          causationId:   event.eventId,
          orderId:       event.orderId,
          type:          'inventory.released',
          timestamp:     Date.now(),
        } satisfies InventoryReleased)
      }]
    })
  }
})
Performance / Safety Warning

Never model compensation as a DELETE. If InventoryReleased is processed twice (at-least-once delivery), a delete-based compensation either deletes an already-deleted row (silent success, wrong semantics) or fails (unhandled error). Model compensation as an append — insert a released status row. The second processing finds the row already exists via the idempotency gate and is silently skipped. The database represents the release history accurately, not just the final state.

2.2 Semantic Locking

Semantic locking prevents a concurrent saga step from acting on a resource that is in mid-saga state:

TYPESCRIPT
// ✅ Semantic lock: mark the reservation as 'pending' during the saga
// Prevents concurrent orders from reserving the same SKU while this saga is in-flight
await db.query(`
  INSERT INTO inventory_reservations (id, order_id, sku, quantity, status)
  VALUES (uuid_generate_v4(), $1, $2, $3, 'pending')
  -- status = 'pending' = semantic lock: this inventory is in-flight
  -- Other saga steps that check availability must exclude 'pending' rows
`, [orderId, sku, quantity])

// Only after InventoryReserved event is published: update to 'reserved'
// On compensation (InventoryReleased): update to 'released'
// Stock availability query: WHERE status NOT IN ('pending', 'reserved')

3. Choreography vs Orchestration: When to Switch

Criterion Choreography Orchestration (Temporal.io)
Coupling Services know event contracts only Services know the orchestrator only
Observability Requires correlation-id tracing across topics Single workflow history in Temporal
Compensation Distributed — each service listens for failure events Centralized — workflow code calls compensate()
Debugging Hard — reconstruct saga state from event stream Easy — Temporal UI shows full workflow history
Steps Works well up to 5–7 steps Preferred for 8+ steps or complex branching
Long-running Works if all consumers stay healthy Better — workflow state survives process crashes
TYPESCRIPT
// When choreography becomes too complex: Temporal.io workflow
// Provides orchestration with durable execution, explicit compensation, and full audit trail
import { proxyActivities, sleep } from '@temporalio/workflow'

export async function orderWorkflow(orderId: string): Promise<void> {
  const { chargePayment, reserveInventory, createShipment, sendNotification,
          refundPayment, releaseInventory } = proxyActivities({ startToCloseTimeout: '30s' })

  let paymentCharged = false
  let inventoryReserved = false

  try {
    await chargePayment(orderId)
    paymentCharged = true

    await reserveInventory(orderId)
    inventoryReserved = true

    await createShipment(orderId)  // pivot — past this point, retry not compensate
    await sendNotification(orderId)
  } catch (err) {
    // Compensate in reverse order — only undo what was done
    if (inventoryReserved) await releaseInventory(orderId)
    if (paymentCharged)    await refundPayment(orderId)
    throw err
  }
}
// Temporal: workflow state is durable — process crash = resume from last checkpoint
// Compensation is explicit and sequential — no event routing needed

Summary

Concept Rule
Compensation is append Compensating events must be idempotent: InventoryReleased processed twice must produce the same final state as processed once — never model compensation as a delete, always as an append.
Pivot transaction The pivot transaction is the point of no return; every step before it must have a compensating event, every step after it must be retried until success.
Causal headers from day one Choreography observability requires correlation-id + causation-id on every event from day one; retrofitting distributed saga tracing after an incident across 6 services is weeks of forensic work.

What's Next

Part 6: Security, Multi-Region Replication, and Disaster Recovery closes the series with the operational concerns that apply to every messaging system at cloud scale: mTLS and SASL authentication for Kafka clusters, IAM-based access control for AWS managed services, MirrorMaker 2 for cross-region replication, and the RPO/RTO model for messaging disaster recovery.

Research & Synthesis Note

This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.

#Saga#Choreography#Event-Driven Architecture#Compensating Transactions#Distributed Systems#Backend
Siddhant Deval

Written by Siddhant Deval

Senior Full-Stack Engineer building high-scale architectures, browser performance engineering systems, and SaaS platforms.