Saga Choreography and Event-Driven Workflow Patterns
Saga choreography distributes long-running business workflows across services via events, eliminating the central orchestrator as a bottleneck. This article designs a 4-step order saga with typed compensating transactions, causal ordering via correlation-id and causation-id headers, semantic locking, and the pivot transaction boundary — and shows when to reach for Temporal.io orchestration instead.
Messaging at Cloud Scale
Saga Choreography and Event-Driven Workflow Patterns
An order spans four services: Payment, Inventory, Fulfillment, and Notification. The payment charges the card. Inventory reserves the items. Fulfillment picks and ships. Notification emails the customer. Each service has its own database, its own deployment, and its own failure modes. There is no shared transaction. If Inventory fails to reserve after Payment has charged, the customer's card is debited but the order cannot be fulfilled. The business must refund — and the system must do it automatically, without a human intervention.
The naive fix is a central Order Orchestrator that calls each service over HTTP and rolls back on failure. The orchestrator becomes the single point of failure for every order in the system, adds synchronous coupling between services, and recreates the distributed monolith it was meant to avoid.
Saga choreography distributes the workflow responsibility: each service listens for events from the previous step, performs its local work, and publishes an event for the next step. Compensation — the event-driven equivalent of ROLLBACK — flows backward through the same chain.
Series positioning: This is Part 5 of Messaging at Cloud Scale. The prerequisite mental model is the Saga pattern from Saga Pattern: Choreography vs Orchestration and Compensating Transactions. This article provides a complete, production-hardened implementation with typed events, compensation chains, and the pivot transaction concept.
1. The Four-Step Order Saga
1.1 Forward Path: All Steps Succeed
1.2 The Event Contract
causation-id is the event ID of the event that caused the current event. correlation-id is the original business entity ID (orderId) that ties the entire saga together. Both must be present on every event from day one. With just correlation-id, you can find all events for an order. With causation-id, you can reconstruct the exact causal chain — critical for debugging compensations where a refund triggered a re-reservation that triggered another failure.
2. The Pivot Transaction and Compensation
2.1 Identifying the Pivot
The pivot transaction is the point of no return in a saga — the step whose effects cannot be compensated because they are externally visible or irreversible:
Never model compensation as a DELETE. If InventoryReleased is processed twice (at-least-once delivery), a delete-based compensation either deletes an already-deleted row (silent success, wrong semantics) or fails (unhandled error). Model compensation as an append — insert a released status row. The second processing finds the row already exists via the idempotency gate and is silently skipped. The database represents the release history accurately, not just the final state.
2.2 Semantic Locking
Semantic locking prevents a concurrent saga step from acting on a resource that is in mid-saga state:
3. Choreography vs Orchestration: When to Switch
| Criterion | Choreography | Orchestration (Temporal.io) |
|---|---|---|
| Coupling | Services know event contracts only | Services know the orchestrator only |
| Observability | Requires correlation-id tracing across topics | Single workflow history in Temporal |
| Compensation | Distributed — each service listens for failure events | Centralized — workflow code calls compensate() |
| Debugging | Hard — reconstruct saga state from event stream | Easy — Temporal UI shows full workflow history |
| Steps | Works well up to 5–7 steps | Preferred for 8+ steps or complex branching |
| Long-running | Works if all consumers stay healthy | Better — workflow state survives process crashes |
Summary
| Concept | Rule |
|---|---|
| Compensation is append | Compensating events must be idempotent: InventoryReleased processed twice must produce the same final state as processed once — never model compensation as a delete, always as an append. |
| Pivot transaction | The pivot transaction is the point of no return; every step before it must have a compensating event, every step after it must be retried until success. |
| Causal headers from day one | Choreography observability requires correlation-id + causation-id on every event from day one; retrofitting distributed saga tracing after an incident across 6 services is weeks of forensic work. |
What's Next
Part 6: Security, Multi-Region Replication, and Disaster Recovery closes the series with the operational concerns that apply to every messaging system at cloud scale: mTLS and SASL authentication for Kafka clusters, IAM-based access control for AWS managed services, MirrorMaker 2 for cross-region replication, and the RPO/RTO model for messaging disaster recovery.
This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.