System Design Patterns
Saga Pattern
You tap Place order on checkout, or request an Uber ride. One action should create the order, authorize payment, reserve inventory or a driver, and notify you, but each step often lives in a different service and database.
A breaks that story into local commits. If a later step fails, undo earlier work: refund the hold, cancel the order, without freezing every database in a distributed lock (two-phase commit, or 2PC).
Checkout saga
Create order
Commit the order locally and record durable saga progress for recovery.
Authorize payment
Hold or charge money so later failure can release the authorization.
Reserve inventory
If stock fails, stop the forward path and start compensations.
Release the payment hold and cancel the order so state stays consistent.
What actually happens
The checkout story above is the default whenever one business action spans services. Traditional 2PC across microservices blocks on coordinators and couples teams. Instead, each service commits locally. An (or of events) records and drives the next step.
When inventory fails after payment authorized, compensations run in reverse: release the hold, cancel the order. Every handler must be . Irreversible work is a : put it last or keep it pending. Publish reliably with an ; park poison steps in a .
Between steps the world is briefly inconsistent. Show pending UI, never treat a half-finished saga as success. This pattern sits next to distributed transactions.
How to choose a saga shape
Once you accept local commits and business undo, three questions decide whether to centralize the flow or let events drive it.
1. How many steps and how critical? Simple 2–3 step flows between decoupled teams can use . Complex 5+ step money paths usually want for visibility.
2. Can the product show pending? If users must never see intermediate states, reconsider the boundary: sagas trade atomicity for autonomy.
3. Are compensations real? Every forward step needs an undo or a clear manual-review path. Missing one leaves permanent inconsistency.
Simple rule of thumb
Prefer when money, inventory, or support must answer "where is this order?" Prefer when teams own thin event contracts and the flow stays short. Persist , use an , and never call a true rollback. In simple terms: local commit forward, business undo backward, pending UI in between.
From problem to mechanism
The choice above names two shapes. Here is the same checkout slowed down: each service commits locally, then run if a later step fails.
Forward path (happy path)
Each service commits in its own database. There is no global rollback, only local commits.
Saga orchestrator
Runs the workflow
Orders
Orders DB
committed
order saved
Payments
Payments DB
committed
card charged
Inventory
Stock DB
committed
stock held
When inventory fails
Stock is unavailable. Undo earlier steps with refund and cancel order.
Item out of stock: payment and order were already committed in their own databases.
Saga orchestrator
Runs the workflow
Orders
Orders DB
cancel order
undo step 1
Payments
Payments DB
refund
undo step 2
Inventory
Stock DB
out of stock
nothing reserved
End state: money returned, order cancelled. Customer is not charged for an unbooked product.
Dotted arrows show message direction. Animation suggests messages in flight (respects reduced-motion settings).
Failure path (sequence view)
A accepts that different services commit at different times. Correctness comes from durable , , retryable compensations, and clear user-facing pending states.
Read the diagram definition
sequenceDiagram
participant O as Orchestrator
participant Order
participant Payment
participant Inventory
O->>Order: create order
Order-->>O: ok
O->>Payment: authorize payment
Payment-->>O: ok
O->>Inventory: reserve item
Inventory-->>O: failed
O->>Payment: release authorization
Payment-->>O: compensated
O->>Order: cancel order
Order-->>O: compensatedIn choreography, each service reacts to a durable event and publishes the result of its own local transaction. OrderCreated might trigger payment authorization; PaymentAuthorized might trigger inventory reservation. The dependency is encoded in event contracts rather than one central workflow definition. Carry a saga identifier and explicit state events so operators can reconstruct the whole story, and avoid circular reactions that repeatedly trigger each other.
- Loosely coupled services stay independent, so teams can deploy without a shared boss.
- No single orchestrator SPOF (single point of failure) to take the whole flow down.
- Harder to see the full path, so debugging end-to-end failure takes more detective work.
- No central place to query , so support answers "where is it?" slowly.
Best for: simple flows (3–5 steps), decoupled teams
The orchestrator drives each forward step and records ; failures trigger compensations in reverse order.
Forward
- OrderService.create(orderId): success
- PaymentService.charge(orderId, amount): success
- InventoryService.reserve(orderId): failed (stop forward path)
Compensate (reverse)
- PaymentService.refund(orderId)
- OrderService.cancel(orderId)
Saga ends failed but consistent: inventory was never committed; money returned; order cancelled.
- Clear flow: one orchestrator defines steps, so the path is easy to read.
- Central saga state makes query and debug simpler when support must answer fast.
- Orchestrator must stay highly available, or forward and compensate paths stall together.
Best for: complex flows (5+ steps), critical paths
Compensating transaction design
Mechanism without safe undo fails the moment a step retries. Design every against these four properties, then map forward actions to business undo.
Safe to retry: refund(orderId, amount) twice still yields one refund via keys.
Commutative
Order of compensation should not change the final business state when possible.
Retryable
Must eventually succeed with backoff; permanent failure goes to DLQ for humans.
Semantic undo
Not a technical rollback: often a new compensating business action with audit trail.
| Forward | Compensate |
|---|---|
| ChargePayment($100) | RefundPayment($100) |
| ShipOrder() | CreateReturnLabel() |
| SendEmail() | SendCancellationEmail() |
Irreversible actions ()
Some effects cannot be undone in the original sense. A shipment that has left the warehouse may require a return, while a sent email cannot be unsent. Delay such effects until required earlier decisions are settled, or represent them as pending requests that can still be cancelled. After an irreversible pivot, later failures may need forward recovery and manual handling rather than pretending the saga can return to its initial state.
Durable state, numbers, and a ride playbook
Compensations only help if you know which step ran after a crash. Persist , recover from it, then apply the same rules to a multi-service ride booking.
saga_instances table
Persist the saga identifier, workflow version, current state, step attempt identifiers, results, and compensation progress. Record an intent before dispatching an external action and its confirmed result afterward. A crash between those records is recovered by looking up or retrying the same idempotent attempt, not by assuming the action never ran. Update saga state and any outgoing command in one local transaction or the workflow engine’s durable equivalent.
| saga_id | type | current_step | status |
|---|---|---|---|
| saga-abc123 | OrderSaga | 3 | COMPENSATING |
| saga-def456 | OrderSaga | 5 | COMPLETED |
| saga-ghi789 | OrderSaga | 2 | RUNNING |
Recovery after crash
- On startup, query: SELECT * FROM saga_instances WHERE status IN ('RUNNING','COMPENSATING')
- Resume from current_step
- Re-execute or re-compensate (, so safe)
Durable workflow engines persist saga state in workflow history. Cloud state-machine services persist in managed storage.
State machine
| State | Entered when | Next transitions | Invariant |
|---|---|---|---|
| PENDING | Saga row created before first side effect | RUNNING or FAILED | User sees pending, not success. |
| RUNNING | Forward steps are executing | CONFIRMED, COMPENSATING, or MANUAL_REVIEW | Each completed step is recorded before the next starts. |
| CONFIRMED | All forward steps and pivot transaction succeeded | Terminal | No compensation should run automatically. |
| COMPENSATING | A retryable forward step failed before final confirmation | COMPENSATED or MANUAL_REVIEW | Compensations run in reverse order and are idempotent. |
| MANUAL_REVIEW | Irreversible step or compensation failed repeatedly | Operator resolves to CONFIRMED or COMPENSATED | No silent data rewrite without audit trail. |
Numbers to know
- Five steps at 99.5% each: about 0.995^5 = 97.5% end-to-end before retries.
- That 2.5% failure tail is why durable state, retries, and manual review exist.
- Ride-scale reference: 100M+ rides/year, with 10+ microservices touched per ride.
Testing panel
Crash after each persisted step, retry every handler twice, force failure, and verify users never see CONFIRMED until the is durable.
Timeouts & dead sagas
A step timeout means its result is unknown, not necessarily that it failed. If payment authorization times out, the provider may already have created the hold. Query or retry using the same operation identifier before choosing compensation. Starting a new payment attempt after every timeout can create several holds that the saga no longer knows how to release.
Give the saga an overall deadline and explicit states for retrying, compensating, and manual review. When the deadline expires, stop scheduling new forward work and reconcile ambiguous completed steps before running the applicable compensations. A stuck-state alert should include the last confirmed step, pending external reference, and retry history so an operator can act without guessing.
Real-world saga: ride booking
If step 4 fails (no drivers available)
Compensate 3: PaymentService.releaseHold → Compensate 1: RiderService.cancelRequest → Notify rider: "No drivers available, try again later"
Saga visibility
- Metrics: saga_started, saga_completed, saga_failed, saga_compensated for health at a glance.
- Distributed tracing: each step is a span; the whole saga is one trace.
- Dashboard: active sagas, stuck sagas, and compensation rate for operators.
- Alert when compensation rate exceeds 5%, because something is wrong upstream.
Reliable events and orchestration frameworks
State on disk is useless if the "next step" event never leaves the service. Fix dual-write first, then pick a workflow runtime that already persists .
Problem: dual-write (DB update + publish is not atomic)
1. Update DB (success) 2. Publish event (fails) → DB updated, event lost → saga stuck
-- Solution: Transactional Outbox Pattern
BEGIN TRANSACTION;
INSERT INTO orders (id, status) VALUES ('ord-123', 'CREATED');
INSERT INTO outbox (id, topic, payload) VALUES (
'evt-456', 'order-events',
'{"type": "OrderCreated", "orderId": "ord-123"}'
);
COMMIT;Polling publisher
A polling publisher reads a bounded batch of unsent outbox rows and publishes each with its durable event identifier. If it crashes after publication but before marking the row sent, it will publish the same event again, so consumers must deduplicate. Index the unsent workload, coordinate concurrent pollers safely, and retain enough published history for diagnosis without letting a table scan grow with every historical order.
CDC with transaction-log relay
A change-data-capture relay reads committed database changes from the transaction log and turns outbox inserts into stream events. Its checkpoint records how far it has read; consumer checkpoints separately record how far each downstream service has applied events. Monitor both positions because a healthy relay can feed a stuck consumer. CDC removes polling queries but still consumes network, log retention, and operational attention.
An (or CDC relay) is the reliability layer; durable workflow engines and cloud state machines are how many teams store the coordinator itself.
Durable workflow engine
@WorkflowMethod
void orderSaga(OrderRequest req) {
order = activities.createOrder(req);
try {
payment = activities.charge(order);
inventory = activities.reserve(order);
activities.ship(order);
} catch (Exception e) {
activities.compensate(order);
}
}- Durable execution survives crashes so you resume without reinventing recovery.
- Built-in retries and timeouts cover the per-step and overall limits above.
- Visibility UI for debugging helps answer "where is this order?" quickly.
Cloud state-machine workflow
{
"StartAt": "CreateOrder",
"States": {
"CreateOrder": {
"Type": "Task",
"Next": "ChargePayment",
"Catch": [{"ErrorEquals": ["States.ALL"], "Next": "CancelOrder"}]
}
}
}- Serverless: no workflow infra to manage, so ops cost stays lower.
- Visual workflow editor makes the forward and catch paths easier to review.
- Cloud lock-in and throughput limits need budgeting before critical money paths.
Failure modes, and what still breaks after the fix
Putting the pieces together: here is how a stuck checkout plays out, the anti-patterns that recreate it, and the trade-offs you still own after , durable state, and undos.
Timeline: inventory fails after payment
t0 Create order + outbox OrderCreated (local commit) t1 Authorize payment succeeds; saga state = step 2 RUNNING t2 Reserve inventory fails; orchestrator starts COMPENSATING t3 Release payment hold (idempotent); cancel order t4 If release keeps failing → DLQ / MANUAL_REVIEW with audit +1h Background job flags any still-RUNNING saga (see Numbers to know)
Saga anti-patterns
- Non-idempotent compensations: a retried refund doubles money moved; store results and keys first.
- Missing compensation steps: every forward action needs an undo, or inconsistency becomes permanent.
- Saga nesting: nested sagas explode failure handling; keep flat, treat inner flow as one compensated step.
- Ignoring eventual consistency: between steps data disagrees; show pending, never expose partial success.
Still fails when…
- A runs too early, so automatic undo becomes impossible afterward.
- Compensation fails repeatedly while nobody reviews the .
- DB commits but publish is not atomic, so the next step event never arrives.
Trade-offs
- Better service autonomy than 2PC, but no instant global atomicity across databases.
- centralizes control but adds a critical workflow service to keep alive.
- lowers coupling but hides end-to-end saga state from operators.
Choosing and explaining saga pattern
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
Use sagas when a business transaction spans services and 2PC would be too blocking or too coupled.
Each local transaction commits independently and has an idempotent compensating action.
Orchestration gives visibility for complex flows; choreography keeps simple flows loosely coupled.
When it belongs
Order, ride, booking, or payment flows touch multiple bounded contexts.
The product can tolerate pending and compensating states.
You need durability across worker crashes and retries.
How it fails
Compensation itself fails or is not idempotent.
An irreversible step happens too early.
An event is lost because DB update and publish were not atomic.
Trade-offs to weigh
Better service autonomy than 2PC, but no instant global atomicity.
Orchestration centralizes control but adds a critical workflow service.
Choreography lowers coupling but hides end-to-end state.
Common misunderstandings
Calling compensation a true rollback.
Forgetting outbox/CDC for event reliability.
Not exposing pending state to users while the saga runs.
Try it: saga forward steps and compensation
Execute a checkout saga step by step. When payment fails, unwind earlier steps in reverse order instead of holding a distributed lock across services.
Reserve inventory
Forward: Hold 1 unit — Compensate: Release hold
Status: pending
Charge payment
Forward: Capture $49 — Compensate: Refund $49
Status: pending
Create shipment
Forward: Book label — Compensate: Cancel label
Status: pending
- Phase
- idle
- Done steps
- 0
- Compensated
- 0
Run the saga forward. If a step fails, compensation runs in reverse order on completed steps.
Going deeper
Saga pattern
The mechanism and its boundary
A saga breaks one business transaction into local commits plus compensating actions. It trades immediate atomicity for availability and clear recovery states, which is often better for long-running workflows.
Local commits with compensation
Step succeeds
Persist state
Next step
Compensate on failure
Work through the sizing assumptions
When it belongs
A workflow spans multiple services and cannot hold locks for its full duration. Each service owns its database and can commit locally. The business can define compensating actions.
Assumptions that matter
Each step has a timeout, retry policy, and idempotency key. Operators can query in-flight, failed, and compensated workflows. Some business effects are irreversible and need explicit handling.
Easy mistakes
Compensation is a new business action, not a database rollback: a refund cannot make an already sent email or observed shipment disappear. Persist the workflow’s progress and pending actions durably at each transition, or an orchestrator restart can lose the information needed to continue or compensate. Both forward and compensating handlers must recognize retries, because repeated messages must not reserve stock or issue a refund twice.
How it fails
Saga state is lost and the workflow becomes stuck. Compensation fails and requires manual repair. Out-of-order events advance the wrong step. Step timeout with no reply triggers compensation while a late ghost reply still mutates a service (needs reconciliation). Orchestrator crash after outbox commit but before reply consumption: recovered via startup sweep plus reply backlog.
Trade-offs to name
Local commits release locks without waiting for a whole workflow, but expose intermediate business states and require compensating actions when later steps fail. Choreography lets services react to events without one central workflow controller, while making it harder to find the current overall state or understand a chain of reactions. Orchestration gives one durable place to inspect progress and schedule recovery, but the coordinator itself needs reliable storage, replay, and carefully versioned workflow rules.
What actually happens in orchestration
Five end-to-end paths for a coordinator-driven checkout saga. Each step pairs a local database commit with an outbox row; CDC turns those rows into Kafka commands and replies. The orchestrator only advances when it consumes a reply: except when timeouts or terminal saga status force compensation or cleanup.
| Path | Trigger | Orchestrator status path |
|---|---|---|
| Happy path | All services healthy | RUNNING → COMPLETED |
| Business rejection | Inventory out of stock | RUNNING → COMPENSATING → FAILED |
| Dead saga | Inventory offline / timeout | RUNNING → TIMEOUT → COMPENSATING → FAILED |
| Stale saga | Late ghost reply after FAILED | FAILED + RELEASE_STOCK cleanup |
| Orchestrator crash | Power loss after outbox commit | RUNNING (resume) → COMPLETED |
Kafka topics (reference)
All services succeed
Order → payment → inventory all succeed. Every hop uses a local DB transaction plus outbox, with CDC pushing commands and replies through Kafka.
1. Saga initiation and order processing (step 1)
- Request arrives: the customer clicks Buy Now. The saga orchestrator receives the request.
- Atomic write (orchestrator DB): in one transaction, write saga_instances {"saga_id": "saga-100", "current_step": 1, "status": "RUNNING"} and orchestrator outbox {"target": "order_service", "action": "CREATE_ORDER", "saga_id": "saga-100"}.
- Orchestrator CDC: captures the outbox row and publishes to the order-commands Kafka topic.
- Order service: consumes the message, creates a PENDING order, and inserts {"saga_id": "saga-100", "status": "ORDER_CREATED"} into its local outbox in the same transaction.
- Order CDC: streams the reply to the saga-orchestrator-replies Kafka topic.
2. Payment processing (step 2)
- Orchestrator evaluates: consumes ORDER_CREATED from the replies topic and advances state.
- Atomic state update (orchestrator DB): update saga_instances to current_step: 2; write outbox {"target": "payment_service", "action": "PROCESS_PAYMENT", "saga_id": "saga-100"}.
- Orchestrator CDC: pushes the row to the payment-commands Kafka topic.
- Payment service: charges the card, records success locally, and outboxes {"saga_id": "saga-100", "status": "PAYMENT_SUCCESSFUL"} in one transaction.
- Payment CDC: streams PAYMENT_SUCCESSFUL to saga-orchestrator-replies.
3. Inventory processing and closing (step 3)
- Orchestrator evaluates: consumes PAYMENT_SUCCESSFUL and advances to step 3.
- Atomic state update (orchestrator DB): current_step: 3; outbox {"target": "inventory_service", "action": "RESERVE_STOCK", "saga_id": "saga-100"}.
- Orchestrator CDC: publishes to inventory-commands.
- Inventory service: decrements stock and outboxes {"saga_id": "saga-100", "status": "STOCK_RESERVED"} atomically.
- Inventory CDC: forwards STOCK_RESERVED to saga-orchestrator-replies.
- Saga completion: orchestrator reads STOCK_RESERVED and updates saga_instances status to COMPLETED.
Outcome: Saga ends COMPLETED after STOCK_RESERVED.
Service is up, but the write fails
Order and payment succeeded. Inventory is healthy but stock is zero: a local business rule rejects the step without corrupting prior commits.
1. The local rejection
- Inventory receives RESERVE_STOCK for saga-200 from inventory-commands.
- Local business failure: stock is 0; further inventory mutations are skipped.
- Atomic failure write (inventory DB): write outbox {"saga_id": "saga-200", "status": "INVENTORY_FAILED", "reason": "OUT_OF_STOCK"}.
- Inventory CDC: pushes INVENTORY_FAILED to saga-orchestrator-replies.
2. Triggering compensating steps
- Orchestrator switches mode: reads INVENTORY_FAILED; payment must be undone.
- Atomic state update (orchestrator DB): saga_instances status COMPENSATING; outbox {"target": "payment_service", "action": "REFUND_PAYMENT", "saga_id": "saga-200"}.
- Orchestrator CDC: routes REFUND_PAYMENT to payment-commands.
- Payment service: processes refund, updates ledger, outboxes {"saga_id": "saga-200", "status": "PAYMENT_REFUNDED"}.
- Payment CDC: pushes PAYMENT_REFUNDED to saga-orchestrator-replies.
3. Finalizing the cancellation
- Orchestrator reads PAYMENT_REFUNDED and outboxes {"target": "order_service", "action": "CANCEL_ORDER", "saga_id": "saga-200"}.
- Order service: marks the order CANCELLED and replies ORDER_CANCELLED via outbox and CDC.
- Saga closed (failed): orchestrator reads ORDER_CANCELLED and sets saga_instances for saga-200 to FAILED.
Outcome: Compensation runs payment refund then order cancel; saga ends FAILED.
Target service crashes or times out
Order and payment succeeded. The orchestrator emits RESERVE_STOCK, but inventory infrastructure is entirely offline: no reply ever arrives.
1. The silence
- Orchestrator sends command: updates saga_instances to current_step: 3 and outboxes RESERVE_STOCK for saga-300.
- Orchestrator CDC and Kafka delivery: CDC copies the command to inventory-commands; the message sits on the broker unread.
- The unconsumed message: inventory is dead, so nothing hits saga-orchestrator-replies.
- Timeout breaks: the orchestrator timer for saga-300 hits its 30-second limit and marks the step TIMEOUT_EXPIRED.
2. Active compensation of online services
- Forced rollback injection: orchestrator assumes inventory failed; writes saga_instances status COMPENSATING and outbox REFUND_PAYMENT for saga-300.
- Orchestrator CDC: routes the refund to payment-commands.
- Execution: online payment service refunds (PAYMENT_REFUNDED); online order service cancels (ORDER_CANCELLED): same loop as the business-rejection path.
- Premature close: orchestrator writes saga_instances status FAILED; the transaction is closed from its perspective even though inventory never answered.
Outcome: Timeout forces compensation on online services; saga ends FAILED from the orchestrator perspective.
A late ghost message arrives
Continuation of the timeout path: two hours later inventory restarts and consumes the backlogged RESERVE_STOCK for an already-failed saga.
1. Awakening and processing the stale request
- Inventory boot-up: reconnects to Kafka and reads backlogged inventory-commands.
- Consuming the ghost message: stale RESERVE_STOCK for saga-300 executes without knowing two hours elapsed.
- Blind execution: inventory decrements stock and outboxes {"saga_id": "saga-300", "status": "STOCK_RESERVED"}.
- Inventory CDC: forwards STOCK_RESERVED to saga-orchestrator-replies.
2. Reconciliation and cleanup
- Orchestrator rejection: consumes STOCK_RESERVED for saga-300; saga_instances already shows FAILED and cancelled.
- Detecting inconsistency: inventory reserved stock for a dead, refunded order: a ghost allocation.
- Atomic remediation write: orchestrator outboxes {"target": "inventory_service", "action": "RELEASE_STOCK", "saga_id": "saga-300"}.
- Orchestrator CDC: pushes RELEASE_STOCK to inventory-commands.
- Final reconciliation: inventory increments stock back and confirms via outbox; absolute consistency is restored.
Outcome: Orchestrator detects inconsistency and issues RELEASE_STOCK; stock is restored and the system converges.
Orchestrator crashes mid-flight
Order succeeded. Orchestrator commits step 2 and the PROCESS_PAYMENT outbox row, then loses power before consuming the reply.
1. The sudden crash
- State is saved: saga_instances shows current_step: 2, status RUNNING; outbox holds PROCESS_PAYMENT for saga-400.
- Orchestrator goes dark: the server dies; durable DB state survives but in-memory progress is lost.
2. Automatic CDC continuity
- CDC is independent: Debezium (or similar) reads the new outbox row and publishes to payment-commands even though the orchestrator process is dead.
- Payment continues: payment service succeeds and outboxes PAYMENT_SUCCESSFUL; payment CDC pushes the reply to saga-orchestrator-replies where it waits.
3. Orchestrator reboot and state reconstruction
- Orchestrator boot-up: server restarts and runs a sweep against durable saga state.
- Log reconciliation query: load every saga still RUNNING or COMPENSATING.
- Reconstructing state: saga-400 is hanging on step 2.
- Consuming backlog: reconnect to saga-orchestrator-replies and read PAYMENT_SUCCESSFUL for saga-400.
- Seamless resume: advance to step 3, write the inventory command to outbox, and continue forward.
Outcome: CDC delivers the command while the orchestrator is down; on reboot, startup sweep plus reply backlog resume the saga with no lost data.
Crash recovery query
On startup the orchestrator reloads in-flight sagas before consuming the reply backlog.
SELECT * FROM saga_instances
WHERE status IN ('RUNNING', 'COMPENSATING');Simple rule of thumb
- Atomic orchestrator writes (saga state + outbox) prevent dual-write gaps between workflow progress and command publication.
- CDC decouples command delivery from orchestrator process liveness: payment can finish while the coordinator is down.
- Timeouts and terminal saga status are first-class: compensate without a reply when needed, then reconcile late replies against persisted truth.
Wrap-up
You have seen the failure path, the two shapes, and the operational signals. Follow the references for the primary sources, then keep the takeaway close when you design the next multi-service checkout.
Topic-specific references
- Garcia-Molina & Salem: Sagas (1987) : original long-lived transaction paper behind the pattern name.
- microservices.io: Saga pattern : orchestration versus choreography with compensation examples.
- Microsoft Azure: Saga guidance : compensation and failure handling in cloud architectures.
Key takeaway
Sagas are the default distributed transaction style for microservices: local commits forward, compensations backward, pending UI in between. Prefer orchestration for complex money paths; choreography for short team-owned flows. Persist state, use outbox, make every undo idempotent, and put irreversible steps last.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Model orchestration or choreography with semantic compensation, pivot points, locks/reservations, workflow versions, human repair, and reconciliation.
Explain the decision
What is a pivot transaction in a saga?
Read the reasoning
The point after which the workflow is committed to completion and remaining steps should be retryable/compensatable under a different policy.
Apply it to a production scenario
A saga definition changes while ten thousand instances are in flight. What happens?
Read one approach
Pin each instance to a workflow version or migrate explicitly; never reinterpret persisted state under incompatible step semantics.
Work through the numbers
Eight steps each fail independently 0.5%. Approximate saga success without retries.
Read the worked calculation
0.995^8 ≈ 96.07%; retry/compensation and operator paths are routine.
Reason through a failure
What must compensation preserve?
Read the recovery reasoning
It is idempotent, auditable, ordered by domain dependency, and acknowledges that some external actions require semantic rather than literal undo.
Sources and further study
Continue with Heartbeat Mechanism.