System Design Patterns
Distributed transactions
You tap Place order on Amazon-style checkout. One click should create the order, charge the card, reserve stock, and queue a confirmation email. Each of those steps may live in a different service and database.
A is the plan for that story: either everything that must succeed does, or you have a clear cleanup path when something stops halfway, not a silent half-charged, half-shipped mess.
Checkout journey
Create order
Local commit plus an event for the next step.
Charge payment
Next forward step in the ; payment may still fail later.
Reserve inventory
Out of stock? Run on earlier steps.
Ship / notify
Succeed and notify, or refund and cancel the order cleanly.
What actually happens
One database can commit everything in a single ACID transaction (all-or-nothing on one engine). With microservices you usually cannot freeze every database at once. Teams run a : forward steps, then if a later step fails, plus an so the business row and the event leave together.
still appears inside one product or tight trust boundary, driven by a . Across a large mesh it blocks and couples teams, so production systems prefer sagas, , and .
At peak events (think Prime Day scale), tens or hundreds of thousands of orders per minute each touch many services. The hard parts are retries, mid-flight crashes, compensation that must be safe twice, and logs that show what each step did.
How to choose coordination
Before picking a pattern, ask how wide the write really is. Distributed coordination is a last resort, not a badge of sophistication.
1. Can one database own it? Keep the write local. A normal ACID transaction beats any cross-service protocol when one engine holds every row you need.
2. Reserve or commit immediately? Soft-holds fit . Independent business commits fit a with .
3. Truly need atomic commit? Use only inside a trusted boundary, and budget for blocking recovery when the dies.
Narrowest primitive first
- One database owns all rows: use a normal ACID transaction.
- One shard owns the invariant: keep the write shard-local or move ownership there.
- Multiple services must reserve but not finalize: consider .
- Multiple services commit steps independently: use a plus idempotent compensations.
- Atomic commit inside one trust boundary: use , then budget blocking recovery.
Simple rule of thumb
Skip cross-service 2PC unless the boundary is tiny and trusted. Default to sagas, outbox, idempotent consumers, and reconciliation. State the business rule: what must never happen, and what you can fix later with compensation.
One server vs many servers
Once you know the write spans more than one database, the comparison below shows why local ACID stops being enough.
Single server (monolith)
One app and one database. Checkout can use a single transaction: all tables commit or all roll back together.
Browser / app
Place order
Monolith API
Orders + payments + stock
PostgreSQL
orders · payments · inventory
BEGIN … COMMIT
One atomic unit
Multiple servers (microservices)
Each service owns its database. There is no shared COMMIT. If step 3 fails, steps 1 and 2 may already be committed.
Browser / app
Place order
Checkout steps (each commits locally)
Order
Service
Orders DB
Payment
Service
Payments DB
Inventory
Service
Stock DB
Example failure: payment goes through (money deducted), but inventory never reserves the item. The customer is charged, yet the product is not booked. There is no single ROLLBACK across databases. You must refund, cancel the order, or run compensating steps (saga).
()
When every participant truly must share one commit decision, is the classic answer. Everyone votes, then everyone applies the same decision. It works inside one database; it is rare as a blanket pattern across many microservices.
Phase 1: Prepare
Coordinator asks each database: can you commit? Participants lock rows and vote.
Coordinator
Runs 2PC
Orders
Orders DB
vote: YES
Payments
Payments DB
vote: YES
Inventory
Stock DB
vote: YES
Phase 2: Commit
If every vote was yes, coordinator tells everyone to commit. If any vote was no, all abort.
Coordinator
Runs 2PC
Orders
Orders DB
COMMIT
Payments
Payments DB
COMMIT
Inventory
Stock DB
COMMIT
Dotted arrows show message direction. Animation suggests messages in flight (respects reduced-motion settings).
How 2PC runs
Phase 1 (prepare)
The asks each participant: can you commit? Each one locks rows, writes to its log, and answers yes or no.
Phase 2 (commit or abort)
If every participant durably promises it can commit, the coordinator records a commit decision before sending commit messages. If any participant rejects preparation, the coordinator decides abort. The messages can be delayed or duplicated, so participants remember transaction identifiers and apply the same final decision safely more than once. Participants may learn the decision at different times; atomic commit means they ultimately obey one decision, not that every disk changes at the same instant.
If the coordinator crashes
A coordinator must durably record the transaction and its final decision according to the protocol before notifying participants. After recovery it retransmits that decision to participants that may have missed it. Merely asking which participants voted yes is insufficient to invent a new outcome: some participants may already have committed based on a decision whose message another participant never received.
A participant that has voted yes is prepared: it has promised to commit if instructed and may be holding locks or other resources. If it cannot discover the decision, it generally cannot unilaterally abort just because a timeout elapsed; another participant may already know the transaction committed. Replicating coordinator state improves recovery availability, but the transaction still depends on reaching the required decision authority.
Why teams avoid 2PC across services
- Locks can sit held if the coordinator disappears mid-decision.
- At least two round trips per participant, which slows wide networks.
- The coordinator must be replicated or it becomes a single point of failure.
- Hard to run across separate teams that do not share one ops model.
Common inside one database (Postgres prepared transactions). Uncommon as the default glue between dozens of microservices.
Numbers to know
If each of 5 participants has a 0.2% chance of crossing the prepare timeout, the transaction timeout probability is roughly 1 - 0.998^5, or about 1%. Tail latency also grows because commit waits for the slowest required participant and may hold locks while the coordinator recovers.
Coordinator failure timeline
| Time | Coordinator | Participants | Runbook |
|---|---|---|---|
| T0 | Writes prepare intent | Rows locked, local prepare log durable | Alert on lock age before user SLO is exhausted. |
| T1 crash | Unavailable before final decision | Cannot safely commit or abort alone | Elect/restart coordinator from durable log; stop new work on affected aggregate. |
| T2 recover | Reads decision log or asks participants | Replay commit/abort idempotently | Verify no participant remains prepared; reconcile outbox and audit rows. |
and
Because cross-service 2PC blocks so easily, most checkout flows use a instead: a chain of local commits, one per service. If a later step fails, you run that undo earlier work (refund payment, cancel order, release stock).
Forward path (happy path)
Each service commits in its own database. There is no global rollback, only local commits.
Saga orchestrator
Runs the workflow
Orders
Orders DB
committed
order saved
Payments
Payments DB
committed
card charged
Inventory
Stock DB
committed
stock held
When inventory fails
Stock is unavailable. Undo earlier steps with refund and cancel order.
Item out of stock: payment and order were already committed in their own databases.
Saga orchestrator
Runs the workflow
Orders
Orders DB
cancel order
undo step 1
Payments
Payments DB
refund
undo step 2
Inventory
Stock DB
out of stock
nothing reserved
End state: money returned, order cancelled. Customer is not charged for an unbooked product.
Dotted arrows show message direction. Animation suggests messages in flight (respects reduced-motion settings).
Forward steps do the happy path. Compensating steps walk backward when something breaks. Choreography vs orchestration, compensations, and workflow engines are covered on the dedicated saga page.
Read more about the Saga patternForward
Create order, charge payment, reserve inventory, create shipment.
Compensate (reverse order)
Cancel shipment, release inventory, refund payment, cancel order.
1.
Running the same compensation twice should be safe. Example: refund order 123 checks if a refund already exists and skips if so.
2. Order of compensations
Independent compensations can run in parallel when their business effects commute, but arbitrary compensation order is not guaranteed safe. For example, stop shipment before refunding a transaction whose goods could otherwise still leave the warehouse. Express dependencies in the workflow and make each compensation idempotent. A compensation is a new business action with its own failure and audit history; it does not erase the fact that the original action occurred.
3. Retries
Compensation should eventually succeed. If the refund API fails, retry with backoff. If it never succeeds, send to a , page someone, and fix by hand.
4. Undo the business meaning
Cancel the order in a way finance can audit. Prefer updating status to cancelled over deleting the row.
Orchestration vs choreography
Orchestration (complex flows)
One workflow engine (Temporal, AWS Step Functions, Camunda) drives the steps. Easier to debug and add new steps.
Choreography (simple flows)
Each service listens and publishes on a message bus. Very decoupled, but harder to see the full story in logs.
Reliable events: and
Sagas only work if the next service hears about each local commit. The pattern saves the business row and the outgoing event in the same database transaction, then publishes the event separately. That avoids the classic bug where the order saved but the message never fired.
Why two separate writes are risky
BEGIN; INSERT INTO orders (...); COMMIT;
kafka.publish("OrderCreated"); // ← What if this fails?
// Order saved but event never published: inconsistent stateOutbox (one transaction for row + event)
BEGIN;
INSERT INTO orders (id, ...) VALUES (...);
INSERT INTO outbox (id, aggregate_type, aggregate_id, event_type, payload)
VALUES (uuid, 'Order', order_id, 'OrderCreated', '{...}');
COMMIT;
// Row and outbox row commit together: atomicOption 1: Polling publisher
SELECT * FROM outbox WHERE published = false ORDER BY created_at LIMIT 100;
-- Publish each to durable log
-- UPDATE outbox SET published = true WHERE id IN (...)Simple to build. Downside: a short delay (often 100ms to 1s) and extra read load on the database.
Option 2: CDC from the transaction log
A relay (for example Debezium) reads the database WAL, sees new , and publishes to Kafka quickly without polling the outbox table. This path uses from the transaction log.
connector: "io.debezium.connector.postgresql.PostgresConnector"
table.include.list: "public.outbox"
transforms: "outbox"
transforms.outbox.type: "io.debezium.transforms.outbox.EventRouter"Near real-time and light on the database. You operate another moving part (connectors, lag monitoring).
At high order volume
- Prefer from the transaction log for fast publish under load.
- Delete outbox rows after about seven days so the table stays small.
- Use Kafka or similar to fan events out to many downstream services.
- Require so at-least-once retries never double-charge.
Kafka and most queues deliver at least once. Your handler must tolerate the same event arriving twice without creating two orders. Processing OrderCreated twice can insert two orders unless you dedupe on event id.
- Message arrives with a unique event_id that names this delivery.
- Look up whether that event_id was already handled before writing.
- If yes, acknowledge the message and skip business work.
- If no, insert event_id and run business logic in one database transaction.
- Acknowledge the message only after that local transaction commits.
BEGIN;
INSERT INTO processed_events (event_id) VALUES ($1);
INSERT INTO orders (...) VALUES (...); -- business logic
COMMIT;
-- Atomic: dedup check + business logic in same transaction
-- Crash before ACK? Redelivery is caught by the dedupe insertChoose deduplication retention from the longest supported replay path, including delayed queues, dead-letter reprocessing, restored backups, and manual reconciliation. Seven days of queue retention is not enough evidence if an operator can replay a month-old event. After the retention boundary, use a durable business uniqueness key or explicitly reject an old operation rather than treating it as a brand-new effect. Delete records in indexed batches to avoid cleanup overwhelming the database.
and when you can skip strict coordination
Not every multi-service write needs a full saga with refunds. (try, confirm, cancel) is the reserve-first option: each service exposes three operations, and it feels like but lives in application code instead of a global lock manager.
Try
The try phase creates a reservation with an identifier and explicit expiry, such as three stock units held for order 123. Other requests must see the reduced available amount even though ownership has not finally transferred. Confirm and cancel refer to the same reservation identifier and must tolerate duplicates, late messages, and a cancel arriving before a delayed try. Reservation expiry needs a clear authority so a late confirm cannot consume stock already released and sold elsewhere.
Confirm
Turn reservations into real commits: deduct stock, capture payment.
Cancel
Drop reservations: stock goes back to available, payment hold is voided.
How this differs from a
In a saga each forward step usually commits right away and undoes real work (like a refund). In the try phase only reserves; confirm or cancel finishes the story. You need less undo logic, but every service must implement all three handlers.
Common in large Chinese e-commerce stacks (Seata). Heavy traffic events like Singles' Day are a classic use case.
Best-effort side effects
- Order service saves the order and publishes an event with best effort.
- Notification service sends email or push without joining the money path.
- If send fails, retry a few times, then give up without rolling money back.
Missing an email is annoying but not a money problem. A failed payment still needs a refund path.
Strongest to weakest (rough guide)
- Fire-and-forget notification for logging or email that can be late.
- with choreography (events plus ).
- with orchestration (central workflow plus ).
- (reserve, then confirm or cancel) when holds are natural.
- across services (usually avoid outside a tiny trust boundary).
Order flow in production, and stronger guarantees
Putting the patterns together, a simplified checkout path shows how and show up in real systems. Each service commits locally and emits events for the next step.
Checkout steps (simplified)
- Order service saves the order and writes an for OrderCreated.
- Event bus delivers OrderCreated to payment and other interested services.
- Payment service charges the card and outboxes PaymentCharged for the next step.
- Inventory service reserves stock and outboxes InventoryReserved when the hold succeeds.
- Fulfillment service creates the shipment once stock and payment are confirmed.
If inventory is out of stock
Inventory publishes failure, payment runs refund, order service marks the order cancelled.
Peak events like Prime Day can exceed 100K orders per minute, each touching many services.
Workflow engines (Temporal and similar)
- Workflow state survives process crashes so long flows can resume safely.
- Built-in retries you can tune per step without custom sleep loops.
- Full history per run for debugging which step failed and why.
- Popular when the flow has many branches and compensation paths.
AWS Step Functions style orchestration
States for charge, reserve, ship. On inventory failure, branch to refund and cancel order.
Global consistency (-style)
Most teams use . A few global databases ( is the famous example) use tight clocks plus replicated commit to offer stronger guarantees in one product. They combine Paxos-style replication with clocks that stay within a small uncertainty window.
Bounded Time API
- TrueTime returns an interval [earliest, latest], not one perfect instant.
- GPS plus atomic clocks in each datacenter bound clock drift tightly.
- Uncertainty is often a few milliseconds, far tighter than casual NTP.
How tight clocks help writes
- Assign the transaction a timestamp from the latest bound.
- Wait until the earliest bound passes that timestamp (commit wait, often a few ms).
- If transaction A finishes before B starts, A's timestamp sorts before B's.
Cross-region writes
- Lock participants across regions before taking a commit decision.
- Run two-phase commit across Paxos groups that own the data.
- Pick a commit timestamp from the bounded-time API.
- Commit wait until uncertainty no longer overlaps that timestamp.
- Release locks so other writers can proceed.
Often tens of milliseconds end to end for a cross-region write.
A timestamped read-only transaction reads a consistent snapshot across the relevant shards without taking part in a new write commit. Each replica must be able to serve the chosen timestamp safely; an arbitrary lagging replica cannot immediately answer a timestamp it has not applied. Systems such as Spanner combine replication, timestamp selection, and uncertainty handling to provide their documented guarantees. Clock infrastructure alone does not make independently stored rows globally atomic.
Failure modes, and what still breaks after the fix
Even with sagas, outbox, and idempotent consumers in place, partial failure still shows up. Here is what to watch, what still goes wrong, and the trade-offs you are accepting.
What to watch
- Saga steps stuck pending while compensation retries keep failing quietly.
- Outbox lag climbing, so downstream services see commits minutes late.
- Duplicate event_id inserts failing or succeeding in ways that hint at double work.
- 2PC lock age and prepare timeouts when a coordinator path is still in use.
Still fails when…
- Compensation is not idempotent, so retries refund the customer twice.
- Outbox publishes, but consumers lack dedupe and create duplicate orders.
- Users see intermediate states while a long compensation chain is still running.
Trade-offs
- 2PC is atomic but hurts availability and team autonomy across services.
- Sagas scale well, but users briefly see intermediate states during compensation.
- Outbox is reliable, but you must operate CDC or polling and lag alerts.
Choosing and explaining distributed transactions
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
Skip cross-service 2PC unless the boundary is tiny, trusted, and you can live with blocking.
Default to sagas, outbox, idempotent consumers, and reconciliation jobs for real workflows.
State the business rule: what must never happen, and what you can fix later with compensation.
When it belongs
One user action updates several services or databases.
Payment, inventory, shipping, or notifications can fail halfway.
They ask how you avoid saving a row but losing the event.
How it fails
Coordinator crash leaves databases holding locks.
Order row commits but the publish to Kafka never runs.
User sees a partial state while compensation is still failing.
Trade-offs to weigh
2PC is atomic but hurts availability and team autonomy.
Sagas scale but users see intermediate states.
Outbox is reliable but you operate CDC or polling.
Common misunderstandings
Treating a distributed lock as a full transaction design.
Retrying handlers that are not idempotent.
No playbook for workflows stuck in pending forever.
Try it: two-phase commit prepare and decision
Walk one distributed transaction through prepare votes and a coordinator decision. Toggle a forced no vote on the ledger to see why blocking and rollback appear.
Inventory DB
Phase: idle — vote: unknown
Ledger DB
Phase: idle — vote: unknown
Coordinator
State: idle
- Coordinator
- idle
- Prepared count
- 0
- Committed
- No
Two-phase commit starts idle. Run prepare to collect votes, then commit if every participant voted yes.
Going deeper
Distributed transactions
The mechanism and its boundary
A distributed transaction tries to make several databases or services behave like one commit: all succeed or all roll back. That is powerful for hard rules, but every participant sits on the critical path, so latency, blocking, and recovery get harder fast.
Coordinated commit path
How two-phase commit (2PC) moves from prepare to a durable decision, including recovery after a crash.
Coordinator
Orchestrates votes
Prepare
Coordinator asks each database: can you commit?
Persist vote
Each participant locks rows and logs YES or NO
Commit decision
All voted YES → commit. Any NO → abort everywhere
Recovery log
Crash? Replay the log and finish the open decision
Dotted arrows show protocol order. Animation suggests messages in flight (respects reduced-motion settings).
Work through the sizing assumptions
When it belongs
The business rule cannot tolerate even a short inconsistent window. Every participant is owned by the same team and ops boundary. Traffic is low enough that coordinator overhead is acceptable.
Assumptions that matter
Cross-region RTT drives tail latency, not median LAN times. Each participant must log prepare before saying yes. Retries can deliver duplicate coordinator messages.
Easy mistakes
A long checkout workflow with external calls or human actions can hold resources far too long inside a global transaction; explicit saga states are appropriate when the business accepts intermediate states and compensation. Coordinator recovery must recover the recorded decision and resolve prepared participants, because a participant cannot safely infer an abort merely from silence after voting to commit. Coordinator messages can be repeated during recovery even when the final transaction outcome is atomic, so handlers must recognize an already processed prepare, commit, or abort.
How it fails
Coordinator dies after prepare; databases keep holding locks. Participant committed but the coordinator never got the ack. Hot rows stay locked and slow everything behind them.
Trade-offs to name
Atomic commit keeps the participating updates together, but a prepared participant may have to wait for a durable decision during an outage, reducing availability. One commit decision simplifies the business outcome while adding recovery work for coordinators, durable logs, and locks held by transactions whose outcome is unknown. Fewer participants and shorter transactions reduce coordination and lock contention, but moving an update outside that boundary requires an explicit weaker guarantee or a separate recovery workflow.
Wrap-up
You have seen why one-database ACID stops at service boundaries, which coordination tools fit which writes, and how outbox plus idempotent consumers keep events honest. Use the sources below for the primary papers and docs.
Topic-specific references
- Gray: Notes on Database Operating Systems (2PC roots) : classic distributed commit thinking behind prepare and decide.
- Garcia-Molina & Salem: Sagas : alternative to 2PC for long-running multi-step transactions.
- PostgreSQL Docs: PREPARE TRANSACTION : two-phase commit SQL semantics inside one database product.
Key takeaway
Do not run across a big microservice mesh. Use with clear , plus for reliable events, and so retries are safe. fits when every service can implement try, confirm, and cancel. Workflow engines help when the flow gets long. Every undo step should be safe to retry and should make business sense (cancelled order, not deleted row).
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Compare 2PC, sagas, reservations, and reconciliation by atomicity need, blocking risk, irreversible effects, operator recovery, and isolation.
Explain the decision
What happens if a 2PC coordinator crashes after participants prepare?
Read the reasoning
Participants can remain in-doubt holding resources until coordinator state recovers or an operator resolves the transaction.
Apply it to a production scenario
Shipping begins before payment later fails. Can a saga simply “undo” it?
Read one approach
Physical shipping may be irreversible; use semantic compensation such as intercept/return/refund and model manual intervention explicitly.
Work through the numbers
A saga has five steps each succeeding 99.8% independently. Approximate all-step success.
Read the worked calculation
0.998^5 ≈ 99.00%, so compensation/retry paths are normal operations, not edge cases.
Reason through a failure
What must a workflow engine preserve?
Read the recovery reasoning
Each command/compensation is idempotent and state transitions are monotonic/versioned so replay cannot skip or duplicate irreversible effects.