System Design Case Study
Payment system
Phone retries; the ledger may already have charged. The spinner hung for two seconds, signal dropped, and checkout fired again. Did the card charge once, twice, or not at all? That uncertainty is the real product problem, not drawing boxes labeled Stripe.
A payment platform is a careful state machine for money: every retry, timeout, refund, ledger row, and must be traceable and safe from double charging.
Checkout → money journey
Same key returns the first attempt result.
Fraud + validate
Rules and scores before money moves.
/
Hold funds, then settle when ready.
Balanced books, then merchant webhook.
Try it: payment authorization and capture
Idempotency keys protect retries. Authorization reserves funds; capture finalizes or voids on failure.
- Merchant
- Payment API
- Idempotency store
- Processor
- Ledger
- Auths
- 0
- Captures
- 0
- Mode
- auth
- Path hop
- 0
Idempotency
keyRetries must not create a second capture.
Money movement
ledger0 auths, 0 captures.
Failure
voidFailed captures void authorizations so holds do not linger.
Authorize, capture, or replay the same idempotency key safely.
What actually happens
Merchants call your API with a tokenized method (, never a raw PAN). You validate, run fraud checks, then talk to a card processor to and . An makes client retries safe.
When the outcome is known, you append a and fire a signed . If the processor times out, you have an , not a failure, and must finish the story before you post money twice.
Subscriptions, refunds, multi-currency, and add more states. Correctness beats microservice count.
How to choose the design moves
1. Can a retry hurt money? Require an and atomically create an operation record containing its scope, request hash, and in-progress status before calling the processor. Save the response when it becomes known. Concurrent callers with the same key find that record; a different payload is rejected instead of being treated as the same charge.
2. Do you know the processor result? Timeout after send is . Poll or read settlements; do not invent a second attempt key.
3. Can you prove the books? Immutable rows plus daily beat fancy dashboards that cannot explain a $0.03 mismatch.
Simple rule of thumb
Design for retries and unknowns first. Tokenize, authorize/capture carefully, ledger once, webhook with backoff. In simple terms: never guess about money: look it up, then write history that cannot lie.
Case-study upgrade
Payment State Math
A payment system values traceability over clever decomposition. Every state change must be idempotent, auditable, and reconcilable.
Assumptions and formulas
- Successful transactions: 10M/day. Peak: 1K TPS. Attempts can exceed successes because of retries and failures.
- Ledger entries are append-only. A single transaction can create several entries for fees, reserves, refunds, and settlement.
- Idempotency records are retained long enough for client retries.
Concrete math
- Transactions: 10M/day = 116/sec average; peak can be 1K TPS.
- Attempts: retries and follow-up actions can make attempts 1.1x to 1.3x successful transactions.
- Ledger: 10M transactions/day x 4 entries x 500 bytes = 20 GB/day raw; 5 years is about 36.5 TB before indexes and replication.
- Idempotency store: 10M requests/day x 1 KB response x 24-hour TTL = 10 GB active raw plus indexes.
- Webhooks: 10M payments/day x 2 events = 20M/day; 1% retrying 5 times adds about 1M retry deliveries/day.
Design decisions forced by numbers
- Use idempotency keys on create/confirm/refund APIs.
- Keep a double-entry ledger and reconcile external settlement files.
- Model unknown outcomes after timeouts as explicit states, not guesses.
Scaling to millions
- At 1M payments/day: payment state table, ledger, idempotency store, and reconciliation job are enough.
- At 10M payments/day: partition ledger, isolate webhooks, and add retry queues.
- At 100M+ payments/day: regional authorization, strict audit pipelines, and settlement backpressure become central.
Readable flow
Interview follow-ups to expect
- What happens when the processor outcome is unknown?
- How do we avoid double charge on retry?
- How are refunds, disputes, fees, and settlement represented in the ledger?
Operational readiness checklist
After the math, explain how this capacity plan is measured, protected, launched, and evolved.
Observability
- Attach each capacity estimate to a dashboard metric: p99 latency, queue age, cache hit rate, shard skew, error rate, and freshness lag.
- Alert on rate-of-change and saturation, not only hard failures; backlogs and hot keys usually warn before an outage.
Security and abuse
- Budget for auth checks, rate limits, quota isolation, audit logging, privacy filters, and abuse-review side paths.
- Separate trusted internal events from user-controlled input so retries and backfills cannot bypass policy checks.
Degraded modes
- Define the fallback when a dependency is slow: stale cache, partial results, read-only mode, delayed async work, or admission control.
- Protect the invariant first; reduce freshness, personalization, or analytics quality before risking duplicate or corrupt state.
Testing and rollout
- Test average, peak, and hot-object traffic; include retry storms, cold caches, lagging workers, and regional dependency loss.
- Use canaries, shadow reads, dual-write validation, feature flags, and explicit rollback criteria for risky path changes.
Rollout, backfill, and runbooks
- Make migrations and backfills idempotent, resumable, rate-limited, and visible through progress and error metrics.
- Write the runbook before launch: owner, dashboards, paging thresholds, mitigation switches, replay steps, and repair commands.
10x evolution
- For payment state math, state which number forces the next move: cache memory, shard count, write amplification, queue lag, egress, or failover reserve.
- Add the next layer only when the metric forces it: sharding, regional reads, async replay, hot-key isolation, or stricter backpressure.
Scope you should nail down
Product surface
- Cards, bank transfer, wallets via API/SDKs
- One-time payments, full/partial refunds
- Subscriptions / recurring billing
- Multi-currency; fraud rules + ML
- PCI DSS posture; merchant status webhooks
Scale targets
- On the order of $1B/day processed
- ~99.99% availability for create/status APIs
- Idempotent, auditable operations
- Sub-500ms p99 for payment API (excluding 3DS)
Capacity picture
Transaction records: ~1KB → ~10GB/day
Average TPS: 10M/day / 86,400 ≈ 116 TPS; plan ~1K TPS peak
Attempts: ~1.2 attempts per success → ~12M attempts/day with retries and 3DS
Ledger storage: 10M × 4 entries × 500B ≈ 20GB/day raw
Idempotency state: At twelve million distinct operation records/day, one-day retention and 1 KB/record imply about 12 GB active raw state before indexes and replicas. Retention must cover the supported retry window, and the durable payment identity must prevent unsafe recreation after cache expiry. A cache hit is an optimization; it cannot be the only record that a payment already exists.
Webhooks: 10M × 2 events = 20M deliveries/day; 1% fail × 5 retries adds ~1M
APIs worth sketching
createPayment(merchant_id, amount, currency, payment_method_id, idempotency_key, metadata?)Payment pending / succeeded / failed
getPayment(payment_id)Payment details
refundPayment(payment_id, amount?, idempotency_key?)Full or partial refund
createSubscription(merchant_id, customer_id, plan_id, payment_method_id)Subscription
listPayments(merchant_id, cursor?, limit?)Payment[]
Core data model
merchants
payment_methods
transactions
ledger
subscriptions
idempotency
High-level architecture
Payment flow (critical path)
Six-step pipeline
- Validate: amount, currency, payment method
- Fraud check: rules + ML; block or flag
- : processor holds funds
- : move hold toward settlement
- Ledger: double-entry debit/credit
- Notify: merchant
Each step needs a repeat-safe identity. An authorization, capture, and refund are distinct operations under one payment, not interchangeable calls using arbitrary fresh keys. Persist the request identity before sending, then store the known result. If a worker crashes in between, recovery queries or safely retries the same external operation. The local record cannot atomically commit with a remote processor call.
Payment flow sketch
// Payment flow (simplified)
1. Check idempotency_key → return cached if exists
2. Validate(amount, currency, payment_method)
3. FraudService.check(transaction) → approve/reject
4. processor.authorize(amount, token)
5. processor.capture(auth_id)
6. LedgerService.record(debit, credit)
7. WebhookService.notify(merchant, "payment.succeeded")Idempotency
- Client sends an (UUID) with each create/refund
- Unique constraint on (merchant_id, idempotency_key)
- First request processes and stores the response
- Duplicate key returns the stored response without re-charging
- TTL ~24h for idempotency records
Idempotency SQL pattern
INSERT INTO idempotency (key, request_hash, response)
VALUES ($1, $2, $3)
ON CONFLICT (key) DO UPDATE SET updated_at = now()
RETURNING response;
-- If conflict: return existing response
-- Else: process and storeDouble-entry ledger
Every transaction creates balanced : debit one account, credit another. Sum of entries is zero. That enables with the processor. For a single-currency posting set, total debits equal total credits under a consistent account convention. That arithmetic catches an unbalanced posting but does not prove the amount, merchant, or payment reference is correct. Link each posting set to a unique business operation and record corrections as new entries. For multiple currencies, balance the relevant currency legs and record exchange rates and conversion accounts explicitly.
| Entry | Account | Amount | Type |
|---|---|---|---|
| Payment $100 | Customer | -100 | Debit |
| Payment $100 | Merchant | +100 | Credit |
| Refund $50 | Merchant | -50 | Debit |
| Refund $50 | Customer | +50 | Credit |
Reconciliation compares individual provider transactions and settlement records with internal payment states and ledger entries, then checks totals. Match by stable processor reference, amount, currency, and operation type so equal aggregate totals cannot hide one missing payment and one duplicate. Distinguish timing differences from real discrepancies. Repair known missing transitions idempotently and put ambiguous cases into an owned review queue with evidence.
Fraud detection
Rule engine
- Velocity: too many payments from one card
- Geo: IP country vs card country
- Amount unusual for merchant/customer
- AVS / CVV mismatch
ML model
A fraud score estimates risk from features such as amount, history, device, and behavior; it is not a proof that a payment is legitimate or fraudulent. Choose approve, step-up, decline, or review thresholds using the cost of fraud and false declines. Track decisions and feature versions for audit and model evaluation. Define what happens if the risk service is unavailable instead of silently bypassing it for every transaction.
Strong customer authentication with the issuer. Adds a requires_action path; reduces some chargebacks.
PCI DSS posture
- : never store raw PAN
- Encrypt at rest: AES-256 for sensitive fields
- Encrypt in transit: TLS 1.3
- Access: least privilege, MFA for admin
- Audit: log every access to payment data
- Segmentation: payment services in an isolated network
Retry and error handling
- Processor timeout: retry with the same key; exponential backoff
- Stuck auth: reconciliation captures or voids
- Webhook down: retry 1m → 5m → 30m → 2h
- Client retries: always safe when keys are enforced
Hard part: exactly-once money movement
drill
- Client sends
idempotency_key=K; attempt A starts as processing. - Processor call times out after the request left: local outcome is unknown, not failed.
- A new key can double charge. Retry only with K or query by processor reference.
- Mark requires_reconciliation; worker polls until succeeded, failed, voided, or refunded.
- Ledger writes once after truth is known; corrections are new rows, never edits.
Invariant
One merchant-scoped request identity should map to one logical payment workflow, with explicit authorized capture and refund operations. Provider challenges and safe retries may require several technical requests, so a single HTTP attempt is not the invariant. Each confirmed business effect gets one ledger posting set, protected by durable uniqueness. Keep references across all attempts so a timeout or late webhook resolves the existing workflow.
Reconciliation math
If 0.1% of twelve million daily attempts have unknown outcomes, 12,000 cases/day require reconciliation. At four checks per case and two seconds waiting per check, that is 96,000 request-seconds/day, or about 1.11 requests concurrently waiting on average. This is concurrency, not 1.11 CPU cores. Size CPU separately and reserve capacity for incident bursts, provider throttling, and cases that require human review.
Operator view
Group unresolved payments by processor, age, merchant, and amount so investigation follows both impact and likely cause. Keep a next-check time and bounded retry policy, and stop automatic transitions once a case needs evidence the system cannot infer safely. A queue that merely retries forever hides risk. Expose pending status to support and the customer without describing uncertainty as a definitive failure.
Processor timeout branch
Read the diagram definition
sequenceDiagram
autonumber
participant C as Client
participant P as Payment service
participant X as Payment processor
participant L as Ledger
participant R as Reconciliation worker
C->>P: createPayment(idempotency_key=K)
P->>P: insert attempt processing
P->>X: authorize
X-->>P: timeout (unknown)
P->>P: mark requires_reconciliation
P-->>C: 202 accepted / pending
C->>P: retry with same K
P-->>C: return cached pending response
R->>X: poll status / read settlement file
alt succeeded
R->>L: append ledger entries once
R->>P: mark succeeded
else failed
R->>P: mark failed
endBottlenecks to call out
Processor rate limits
Queue, batch, and multiple processor routes.
Ledger write contention
Append-only ledger; partition by time/merchant.
Fraud latency
Cache scores; keep hot path bounded.
Double charge
Enforce keys; document retry rules clearly.
Production readiness
SLOs and metrics
- API p99 <500ms excluding customer action; 99.99% create/status
- Unknown queue p95 age <15 min; mismatch <0.01%
- Webhook p95 <5 min; settlement import lag <30 min
Alerting
- Page on processor success drop, hash conflicts, ledger imbalance
- Warn on webhook age, fraud timeouts, settlement delay, backlog growth
- Burn-rate on payment success SLO, not only uptime
Degraded mode
- Processor outage → stop new captures on that route; honest retries
- Fraud slow → cached low-risk allow; stricter review for high risk
- Ledger unhealthy → fail closed; do not accept unpostable money
Threat model
- Assets: tokens, ledger, keys, webhooks, settlements
- Actors: fraudsters, compromised merchants, insiders, replay, spoofed callbacks
- Controls: signing, hash binding, least privilege, audit, webhook signatures
Test plan
- Property-test ledger balances to zero per currency
- Replay duplicates, timeout after send, delayed webhooks, settlement mismatch
- Load 1K TPS createPayment with retry storms and latency injection
Rollout and 10x
- Shadow then canary processor paths by merchant/currency
- DR: near-zero RPO for ledger; reconcile before reopening writes
- 10x: partition ledger, isolate processor queues, sharded reconciliation
Takeaway
- Flow: validate → fraud → auth → capture → ledger → webhook
- Idempotency: client key + request hash + cached response
- Ledger: immutable double-entry; reconcile unknowns
- Safety: tokenize, encrypt, audit; never guess after a timeout
Beginner to advanced path(open guidance)
Beginner answer
Payment intents, tokenized methods, processor calls, statuses, refunds, and webhooks.
Advanced answer
Idempotency hashes, auth vs capture, immutable ledger, reconciliation, webhook retries, fraud, chargebacks.
Deeper walkthrough
Payment system deeper walkthrough
Payments are state machines with money movement, retries, audit logs, and external processors. Correctness matters more than microservice count.
Requirement refinements
Clarify one-time payments, auth/capture, refunds, subscriptions, chargebacks, fraud checks, webhooks, and multi-currency.
State the invariant: retries must not double charge, ledger entries must balance, and processor state must be reconcilable.
API design details
Require idempotency_key and store request_hash plus response so duplicate keys with different payloads are rejected.
Expose payment state transitions such as requires_action, authorized, captured, failed, refunded, and disputed.
Data model and storage
Separate payment intent, payment attempt, processor reference, ledger entry, refund, dispute, and webhook delivery records.
Use immutable ledger rows and correction entries rather than editing historical money movement.
Scaling strategy
Keep payment creation synchronous enough for user feedback, but move webhooks, reconciliation, and reporting to async workers.
Partition merchant-facing queries by merchant_id and time while keeping processor reconciliation indexed by processor_ref.
Reliability, recovery, and operations
Run reconciliation jobs that compare internal state with processor settlements and repair stuck transitions.
Monitor success rate by processor, idempotency conflicts, webhook retry age, reconciliation mismatches, and fraud false positives.
Deepening reference
Payment System: realistic production notes
Beginner mental model: Payments are state machines plus idempotency plus ledger plus reconciliation.
Production scope: A payment-gateway answer must separate authorization, capture, refund, chargeback, webhook, fraud, ledger, and settlement. Never store raw card numbers.
Choose what the first version must guarantee
Start by deciding which payment system capabilities are essential. Describe the main read and write operations before choosing infrastructure. For each operation, identify the stored records it uses and the condition that makes its result correct. A fast response is useful only if it satisfies that condition.
Next, separate work the user must wait for from work that can finish later. Estimate requests per second, storage growth, and the busiest object or customer. Use those estimates to choose one bottleneck to investigate. The calculations below are example workloads with stated assumptions; they are not measured limits of a particular product.
Core flow to walk through
Create payment intent with idempotency key and amount.
Run fraud/risk checks, authorize with processor, capture funds when business action succeeds.
Write immutable ledger entries and notify merchant with retryable webhook.
Production readiness
Payment System: ship, operate, and evolve
Now examine the same request flow during failures and deployments. Tie each operational control to the user behavior it protects.
Observability
Track payment system p95/p99 latency, error rate, saturation, and user-visible freshness separately for read, write, and async paths.
Alert on leading indicators such as queue age, hot-key skew, cache miss storms, reconciliation mismatches, and error-budget burn.
Security and abuse
Enforce authentication, authorization, rate limits, input validation, audit trails, and privacy checks at the service boundary.
Treat abuse controls as a first-class path: throttling, quota isolation, moderation hooks, revocation, and safe handling of sensitive data.
Degraded modes
Name what still works if cache, search, ranking, stream processing, object storage, or a regional dependency is slow.
Prefer stale-but-safe reads, partial results, async retry, admission control, and feature disablement over corrupting core state.
Testing and rollout
Load test average, peak, and hot-object traffic; add failure-injection for retries, timeouts, backlog growth, and dependency outages.
Ship with canaries, shadow reads or dual writes when useful, rollback switches, schema compatibility checks, and replay validation.
Runbooks
Document dashboards, paging thresholds, owner actions, rollback steps, data repair commands, and customer-impact language.
Keep backfills and replays idempotent, resumable, rate-limited, and observable so repair work does not create a second incident.
10x evolution
Say which metric forces the next architecture change: shard count, cache footprint, write amplification, queue lag, p99 latency, or regional failover reserve.
Evolve one bottleneck at a time: partition by access pattern, isolate hot tenants, regionalize reads, add replayable async pipelines, then harden multi-region behavior.
Hard parts worth deepening
Idempotency must compare request hash, not only key existence.
Ledger entries should be immutable and balanced.
Processor state and internal state can diverge; reconciliation is mandatory.
Webhooks are at-least-once and merchants must dedup them.
Failure modes and recovery
Client retries charge after timeout; return stored result for same idempotency key.
Processor captured but internal DB failed; reconciliation finds processor reference and repairs state.
Webhook delivery fails; retry with exponential backoff and signed payloads.
Refund races with capture; enforce state machine transitions.
Signals to monitor
Payment success must use a clear final-state definition and exclude operations still awaiting reconciliation. Processor error rate identifies upstream failures, while idempotency conflicts reveal attempts to reuse one key with inconsistent request data.
Reconciliation mismatches identify disagreements between your durable records and the processor. Webhook latency measures how long external status updates take to arrive and apply; signed, duplicate callbacks must still be handled safely.
| Choice | Best when | Cost |
|---|---|---|
| Authorize then capture | Fulfillment may fail after checkout. | More states and expiry handling. |
| Immediate capture | Digital goods are fulfilled instantly. | Refunds needed for downstream failure. |
| Double-entry ledger | Money movement must be auditable. | More modeling and reconciliation work. |
Try this scenario
Explain what happens if the payment processor times out but later confirms the charge succeeded.
Failure timeline: timeout + new idempotency key
T+0 Client createPayment with idempotency_key=K; processor accepts charge T+8s Client times out; local state = unknown (not failed) T+9s Naive retry with new key K2 → second processor capture T+1h Webhooks disagree; ledger shows two posts for one checkout T+1d Reconciliation finds mismatch; refunds one: customer already charged twice briefly Outcome Invariant broken: one merchant intent must map to one attempt chain
Counterexample to "retry on timeout": timeout is unknown outcome: retry only with the same key or query by processor reference.
Exercises
- 0.1% of 12M attempts/day are unknown. At 2s waiting time × 4 reconciliation polls each, estimate average concurrent outstanding polls before headroom. What additional measurement would you need to estimate CPU demand, and what alert detects growing unresolved age?
- Same idempotency key arrives with a different amount hash. Should you return the stored response or reject? Defend with the ledger invariant.
- Processor webhook arrives before your local capture row. Design ordering so you never post money twice and never mark succeeded without a reconcilable processor_ref.
Primary references
Ledger, idempotency, and payment-ops primary sources.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Design idempotent payment commands, immutable double-entry ledger, processor/webhook verification, reconciliation, refunds, disputes, failover, and compliance boundaries.
Explain the decision
Why separate payment state from a ledger?
Read the reasoning
Workflow state describes authorization/capture/refund progress; immutable balanced entries provide the financial source of truth and audit trail.
Apply it to a production scenario
Processor capture succeeds but the response is lost. What does retry do?
Read one approach
Reuse the idempotency key, query/reconcile processor state, and post ledger effects once; never blindly issue a new capture.
Work through the numbers
At 8,000 payments/s with six 300-byte ledger entries per payment and three replicas, estimate write payload.
Read the worked calculation
About 43.2 MB/s before indexes, WAL, protocol, and backups.
Reason through a failure
What is the ledger invariant?
Read the recovery reasoning
For every currency and transaction, posted debits equal posted credits, entries are immutable, and retries cannot create a second economic effect.
Sources and further study
Continue with Design News Feed.