Case study
Design a flash sale
Imagine Prime Day or a sneaker drop: at 12:00:00, a million shoppers open the same deal page and mash Buy. To them it feels like one checkout. Under the hood you must stay fast, never oversell a scarce SKU, and clean up when payment succeeds but inventory says no.
A flash sale is inventory locking under a stampede. Protect browse with a CDN, protect purchase with , reject sold-out in milliseconds with a , then run the real commit as a .
What happens on Buy?
Get a short-lived pass to checkout.
Atomic decrement; reject sold-out fast.
Order → pay → reserve, async pending.
Refund, cancel, restock on failure.
Try it: flash-sale inventory under burst traffic
A sale needs a reserved counter or queue ahead of the database so oversell stops at the hot path.
- Client
- Edge
- Rate limit
- Inventory counter
- Order DB
- Attempts
- 0
- Orders
- 0
- Mode
- open
- Path hop
- 0
Hot key
counter0/5 units sold.
Protection
rate limit0 attempts; bots shed at the edge.
Consistency
oversellNever trust a cached “in stock” flag alone for the final decrement.
Hammer buy attempts while inventory remains, then watch sold-out reject before the DB.
Go deeper on the building blocks
These notes pair with the diagram above.
What actually happens
Most people are only browsing. The CDN serves images and sale HTML. When someone joins the waiting room, issues a signed token so bots and overload stay at the edge. A signed waiting-room token proves that the holder was admitted under a particular policy; it does not by itself establish fair ordering or prevent replay. Bind it to the account or session, event, expiry, and allowed attempt budget. Track issued admissions against inventory and payment capacity, and serve queue status from a cheap path so waiting users do not overload the service protecting checkout.
On Buy, the gateway runs a . If stock is gone, return immediately: no payment attempt. If the counter says yes, start a and return pending. The orchestrator drives order, payment, and inventory over the broker. Each service uses an so events are not lost after a crash.
If inventory fails after payment, refunds and restocks. An on checkout keeps double-clicks from starting two sagas. Compensation is a new operation with its own durable state and retries. Refunding payment does not erase the original charge, and restocking must occur only if that checkout actually owned an unreleased reservation. If compensation fails, retain a visible pending-recovery state and alert an owner. Otherwise an apparently cancelled order can leave both money and inventory inconsistent.
How to choose the flash-sale design
1. Separate looking from buying? Yes. Cache browse aggressively. Put a waiting room and rate limits on checkout so 10K tries/sec do not melt the orchestrator.
2. Sync stock or async commit? Use Redis only for a fast no. The real commit is an async across order, payment, and inventory databases.
3. Pending or block until done? Prefer pending. Blocking the HTTP thread until payment and inventory finish does not survive a stampede.
Simple rule of thumb
Tell the story in order: edge protects origin, queue protects purchase, Redis rejects sold-out in one millisecond, runs the real checkout with in every service, and fixes the messy middle when step three fails after step two succeeded.
In simple terms: do not invent a giant distributed transaction. Invent a fair line, a fast no, and an undo plan.
The map for the rest of this note
The reference architecture uses a , an in each microservice, a message broker for async commands, and Redis for the millisecond stock check. The diagram below is the map; architecture layers, capacity math, and saga scenarios deepen every box.
High-level design diagram
Follow the arrows top to bottom, then open each architecture layer in the next section (same order as the drawing).
Waits for an answer before moving on
Fire-and-forget over the broker
Service answers the orchestrator
Rows saved in one local transaction
Event leaves the database reliably
Architecture layer by layer
Each block matches the HLD diagram. Expand any layer for interview talking points.
01Clients and edgeKeep the crowd from crushing the origin.
Keep the crowd from crushing the origin.
Think of it like: Like opening extra doors at a stadium: most people never reach the main hall.
- CDN caches images, JS, and sale HTML close to users.
- WAF and DDoS filters block bad traffic before it hits your servers.
- L7 load balancer spreads healthy traffic across gateway pods.
02API gatewayThe bouncer plus the quick stock check.
The bouncer plus the quick stock check.
Think of it like: Checks your queue ticket, then peeks at a sticky note on the wall (Redis) before starting paperwork.
- Auth, rate limits, and signed admission tokens from the waiting room.
- Sync call to Redis: Lua script atomically checks and decrements stock.
- If sold out, return immediately. No saga, no payment attempt.
Redis provides an early admission estimate, while authoritative inventory decides whether a sale can be confirmed. A stale cache may reject an available item or allow a request that later loses the reservation race; it must never create stock by itself. Track each cache reservation and release by a stable operation ID so duplicate compensation cannot increment the counter twice. Reconcile the estimate against durable inventory after failures.
-- stock:sku:{id} (Lua runs atomically on Redis)
local left = tonumber(redis.call('GET', KEYS[1]) or '0')
if left < tonumber(ARGV[1]) then return 0 end
redis.call('DECRBY', KEYS[1], ARGV[1])
return 103Saga orchestratorThe stage manager who remembers every step.
The stage manager who remembers every step.
Think of it like: One person with a clipboard: who paid, who reserved stock, what to undo if step 3 fails.
- Intake queue: new checkouts from the gateway.
- Reply queue: OrderCreated, PaymentCharged, InventoryReserved, and failures.
- Replay queue: retries and compensation commands with backoff.
- Orchestrator DB: saga id, current step, audit trail for support.
04Message brokerThe post office between teams.
The post office between teams.
Think of it like: Orchestrator drops letters in labeled boxes; each service picks up only its mail.
- Topics: order, payment, inventory (Kafka or RabbitMQ).
- Partition by order_id so one order’s messages stay in order.
- Replies on a dedicated topic the orchestrator consumes.
05Order, payment, inventoryThree small databases, one big story.
Three small databases, one big story.
Think of it like: Each team keeps its own filing cabinet and slips a copy of every change into an outbox tray for the mail room.
- Every change: business row + outbox row in one DB commit.
- Outbox processor or CDC publishes events so crashes cannot lose messages.
- Payment talks to an external card gateway; inventory holds rows authoritatively.
06ObservabilityWhen 10K things happen per second, you need one id to follow a single order.
When 10K things happen per second, you need one id to follow a single order.
Think of it like: Like a package tracking number that works across warehouse, truck, and front desk.
- Logs: same saga_id on gateway, orchestrator, and all three services.
- Metrics: queue depth, Redis reject rate, saga success %, payment timeouts.
- Tracing: Jaeger or Zipkin across the full checkout path.
- Alerts: replay queue or DLQ spikes during the event window.
Why shows up three times in the diagram
- Order service saves PENDING order and outbox OrderCreated together.
- Payment service saves CHARGED and outbox PaymentCharged together.
- Inventory service saves RESERVED and outbox InventoryReserved together.
What the shopper experiences
Maps the architecture above to a story interviewers remember.
Browse the drop
You: You open the sale page. Product photos and prices may be a few seconds old. That is fine.
Behind the scenes: CDN serves static pages. Catalog reads hit cache. No heavy writes yet.
Join the virtual line
You: The site says “You are #4,521 in line.” You wait, then get a short-lived pass to checkout.
Behind the scenes: Admission control protects the purchase path. Bots and overload stay at the edge.
Tap Buy
You: You see “Processing…” while the store checks stock and runs payment behind the scenes.
Behind the scenes: Gateway does a fast Redis check, then starts a saga. You get pending status, not instant success.
Confirmed or clearly failed
You: Either “Order confirmed” or “Sold out / payment failed” with money safe.
Behind the scenes: Orchestrator finishes all steps or runs compensation (refund, cancel, restock).
Requirements
Clarify these before capacity math and APIs.
Functional
- Flash-sale landing page and SKU list (cache-friendly)
- Virtual waiting room with fair admission
- Short checkout path for admitted users only
- Multi-step purchase: order → pay → reserve stock → confirm
- Automatic undo on failure (refund, cancel, restock)
- Order status: pending, confirmed, or failed
Non-functional
- No oversell on flash SKUs (harder rule than fresh catalog text)
- Purchase path highly available; browse may slow down
- Full audit trail per checkout for support and finance
- Idempotent retries so double-clicks do not double-charge
Numbers that drive the design
You do not need perfect math in an interview. You need numbers that force a decision.
Most are browsing, not buying
Peak on a few hot SKUs
CDN absorbs most reads
After user leaves the queue
Why a waiting room? Without admission control, 10K checkout attempts per second would crush the orchestrator and inventory DB. The queue turns a stampede into a steady trickle of real buyers.
Case-study upgrade
Flash Sale HLD Math
A flash sale is a burst of checkout attempts on a tiny hot SKU set. Browse and catalog can lag; inventory reservation, payment, and saga state must stay correct under 10K+ attempts/sec.
Assumptions and formulas
- 1M+ clients during the event window; only a fraction completes checkout.
- Peak checkout attempts: 10K/sec on a handful of SKUs (extreme hot keys).
- Fast path: Redis atomic counters for pre-check; durable inventory in the inventory service DB.
- Saga steps: create order → charge payment → reserve inventory → confirm; compensate on failure.
Concrete math
- Browse/API: 50K peak QPS at edge; only admitted users reach checkout (virtual waiting room cuts effective load).
- Redis DECR: 10K atomic ops/sec per hot SKU shard; Lua script keeps check-and-decrement one round trip.
- Orchestrator intake: 10K saga starts/sec → partitioned intake queue; DB stores saga state (~2 KB/row).
- Outbox: 10K writes/sec x 3 services x ~500 bytes/event ≈ 15 MB/sec async publish load (CDC lowers poll cost).
- Inventory holds: 10K/sec x 10 min hold ≈ 6M live holds if every attempt reserves (admission control lowers this).
Design decisions forced by numbers
- Never expose Redis inventory counters to clients; gateway or inventory service owns the fast check.
- Orchestrate with a central saga engine when flows branch and need compensation visibility.
- Outbox every command side-effect so crash between DB write and publish cannot lose events.
- Idempotent consumers and compensations so retries and duplicate events are safe.
Scaling to millions
- At 100K users: single-region Redis + one orchestrator + three services is enough for practice.
- At 10M users: shard orchestrator DB, partition Kafka by order_id, isolate hot SKU keys in Redis.
- At 100M+ users: waiting room, per-SKU admission tokens, regional edge, and replay queues with DLQ alerts.
Readable flow
Interview follow-ups to expect
- What happens if payment succeeds but inventory is gone?
- How do you restock Redis when compensation runs?
- How do you debug one stuck saga across ten services?
Operational readiness checklist
After the math, explain how this capacity plan is measured, protected, launched, and evolved.
Observability
- Attach each capacity estimate to a dashboard metric: p99 latency, queue age, cache hit rate, shard skew, error rate, and freshness lag.
- Alert on rate-of-change and saturation, not only hard failures; backlogs and hot keys usually warn before an outage.
Security and abuse
- Budget for auth checks, rate limits, quota isolation, audit logging, privacy filters, and abuse-review side paths.
- Separate trusted internal events from user-controlled input so retries and backfills cannot bypass policy checks.
Degraded modes
- Define the fallback when a dependency is slow: stale cache, partial results, read-only mode, delayed async work, or admission control.
- Protect the invariant first; reduce freshness, personalization, or analytics quality before risking duplicate or corrupt state.
Testing and rollout
- Test average, peak, and hot-object traffic; include retry storms, cold caches, lagging workers, and regional dependency loss.
- Use canaries, shadow reads, dual-write validation, feature flags, and explicit rollback criteria for risky path changes.
Rollout, backfill, and runbooks
- Make migrations and backfills idempotent, resumable, rate-limited, and visible through progress and error metrics.
- Write the runbook before launch: owner, dashboards, paging thresholds, mitigation switches, replay steps, and repair commands.
10x evolution
- For flash sale hld math, state which number forces the next move: cache memory, shard count, write amplification, queue lag, egress, or failover reserve.
- Add the next layer only when the metric forces it: sharding, regional reads, async replay, hot-key isolation, or stricter backpressure.
APIs worth naming in the interview
/flash-sale/{eventId}SKU list (cached at CDN)/flash-sale/{eventId}/join-queueQueue position or admission token/checkoutStart purchase; requires Idempotency-Key + admission token/orders/{orderId}pending | confirmed | failedInternal events (orchestrator ↔ broker)
Not HTTP. JSON messages on Kafka (or RabbitMQ) topics, keyed by order_id.
Commands
Orchestrator → broker → service
The orchestrator publishes the next step. One service consumes and acts.
CreateOrderorderOrder service·Insert PENDING order + outbox
ChargePaymentpaymentPayment service·Call payment gateway
ReserveInventoryinventoryInventory service·Authoritative stock hold
Replies
Service → broker → orchestrator
Success or failure replies unblock the saga state machine.
OrderCreatedreplyOrchestrator·Move to payment step
PaymentChargedreplyOrchestrator·Move to inventory step
InventoryReservedreplyOrchestrator·Mark saga COMPLETED
PaymentFailedreplyOrchestrator·Start compensation
InventoryFailedreplyOrchestrator·Refund + cancel
Compensation commands
Orchestrator → replay queue → broker
Published when a forward step fails after earlier steps succeeded.
RefundPaymentpaymentPayment service·Idempotent refund
CancelOrderorderOrder service·Status → CANCELLED
ReleaseInventoryinventoryInventory service·Release hold + Redis INCRBY
Rule of thumb: commands and replies use different topics so the orchestrator never confuses “charge card” with “payment succeeded.” Partition by order_id so one checkout stays ordered. A partition key orders records inside that topic partition, not across several command and reply topics. The orchestrator must still validate the expected saga state, step ID, and version when replies arrive. A delayed payment response cannot advance an order that has already been cancelled without following the reconciliation policy. Persist state transitions before scheduling the next command.
Use an idempotency key on the public checkout API so retries do not start two sagas. On the consumer side, use so duplicate broker messages are harmless.
The checkout saga (forward and undo)
After the architecture makes sense, study how steps succeed, fail, and compensate.
Forward path (happy path)
Each service commits in its own database. There is no global rollback, only local commits.
Saga orchestrator
Runs the workflow
Orders
Orders DB
committed
order saved
Payments
Payments DB
committed
card charged
Inventory
Stock DB
committed
stock held
When inventory fails
Stock is unavailable. Undo earlier steps with refund and cancel order.
Item out of stock: payment and order were already committed in their own databases.
Saga orchestrator
Runs the workflow
Orders
Orders DB
cancel order
undo step 1
Payments
Payments DB
refund
undo step 2
Inventory
Stock DB
out of stock
nothing reserved
End state: money returned, order cancelled. Customer is not charged for an unbooked product.
Dotted arrows show message direction. Animation suggests messages in flight (respects reduced-motion settings).
Happy path (sequence)
Return pending only after the checkout command and stable operation ID are durably stored. The shopper polls or subscribes to that operation for confirmed, failed, or recovery-required status. A lost initial response should lead a retry to the existing operation, not another saga. Also expire queued intent that is no longer useful so old checkout requests do not unexpectedly purchase items long after the sale.
Read the diagram definition
sequenceDiagram
autonumber
participant U as Shopper
participant G as API gateway
participant R as Redis stock
participant O as Saga orchestrator
participant K as Message broker
participant Ord as Order service
participant Pay as Payment service
participant Inv as Inventory service
U->>G: Buy now (admission token)
G->>R: Lua check and decrement
R-->>G: ok
G->>O: Start saga (async)
G-->>U: 202 accepted, order pending
O->>K: CreateOrder
K->>Ord: command
Ord-->>O: OrderCreated reply
O->>K: ChargePayment
K->>Pay: command
Pay-->>O: PaymentCharged reply
O->>K: ReserveInventory
K->>Inv: command
Inv-->>O: InventoryReserved reply
O-->>U: Order confirmed (push or poll)When inventory fails after payment
This is the money question. runs on the replay queue; Redis stock goes back up.
Read the diagram definition
sequenceDiagram
autonumber
participant O as Saga orchestrator
participant Pay as Payment service
participant Inv as Inventory service
participant R as Redis stock
participant Q as Replay queue
Note over O, Inv: Payment already succeeded
O->>Inv: ReserveInventory
Inv-->>O: InventoryFailed
O->>Q: Schedule compensation
O->>Pay: RefundPayment
Pay-->>O: Refunded
O->>R: INCRBY restock counter
O-->>O: Saga FAILED, user notifiedFrom the pattern note
Saga scenarios for this flash sale
The flash-sale design uses orchestration (a central coordinator). Below are the same teaching scenarios as the Saga pattern page, rewritten with checkout examples so you can stay on this note while the ideas land.
Choreography vs orchestration
Flash sales almost always pick orchestration. Here is why.
Choreography (event chain)
Order service emits OrderCreated → payment listens → payment emits PaymentCharged → inventory listens. No central brain.
- ✓ Teams stay decoupled
- ✗ Hard to see one checkout end-to-end during an incident
- ✗ No single place to answer “what step failed for order 8821?”
OK for 2–3 step flows between mature teams.
Orchestration (this design)
One orchestrator owns the script, queues, and saga row. Matches the diagram's intake / reply / replay queues.
Forward
- OrderService.create(orderId): ok
- PaymentService.charge(orderId, amount): ok
- InventoryService.reserve(orderId):failed (stop forward path)
(reverse)
- PaymentService.refund(orderId): ok
- OrderService.cancel(orderId): ok
- Redis INCRBY restock flash SKU counter
Saga ends failed but consistent: no stock reserved, money returned, order cancelled, shopper sees a clear failure (not a silent hang).
- ✓ One saga_id in logs and traces
- ✓ Replay queue for retries and compensation
- ✗ Orchestrator must be highly available (replicated)
Flash-sale checkout saga, step by step
Same idea as the ride-booking walkthrough on the Saga page, but for a timed product drop.
If step 5 fails (inventory gone at the DB)
Even if Redis pre-check passed milliseconds ago, the authoritative DB can still reject (race or reconciliation lag). Compensate: refund payment → cancel order → bump Redis stock → notify shopper “Sold out, refund issued.”
checkout attempts at peak
services + orchestrator per purchase
Designing compensations for checkout
Every forward step in the saga needs a safe undo story.
Idempotent
Refund(orderId) twice must still produce one refund. Store idempotency keys on payment and compensation handlers. See .
Commutative
Refund before release stock vs release before refund should end in the same business state.
Retryable
PSP timeouts get exponential backoff. Permanent failures go to a for humans.
Semantic undo
Mark order CANCELLED; do not delete rows finance must audit. Email cancellation is not a DB rollback.
| Forward step | Compensate |
|---|---|
| CreateOrder → status PENDING | CancelOrder → status CANCELLED (keep row for audit) |
| ChargePayment($amount) | RefundPayment($amount) with idempotency key |
| ReserveInventory(qty) | ReleaseInventory(qty) + INCRBY Redis counter |
| SendConfirmationEmail() | SendCancellationEmail() (best effort) |
Irreversible steps (pivot transactions)
Once the warehouse ships, you cannot “compensate” with a DB rollback. In flash sales, keep ship / fulfill last, or use a pending state until payment and inventory both succeed. Email can stay best-effort.
Saga state and crash recovery
| saga_id | type | step | status |
|---|---|---|---|
| saga-abc123 | FlashCheckout | 3 | COMPENSATING |
| saga-def456 | FlashCheckout | 5 | COMPLETED |
| saga-ghi789 | FlashCheckout | 2 | RUNNING |
Recovery after orchestrator crash
- On startup: load sagas where status is RUNNING or COMPENSATING.
- Resume from current_step (idempotent handlers make re-run safe).
- Re-publish stuck commands or continue compensation from replay queue.
Managed options (Temporal, Step Functions) persist this history for you. Custom orchestrators need the sagas table plus disciplined replay workers.
Outbox when a service crashes mid-publish
The dual-write trap
Order saved in Postgres, then crash before Kafka publish → orchestrator never hears OrderCreated → shopper stuck on “Processing…” forever.
BEGIN;
INSERT INTO orders (id, status) VALUES ('ord-99', 'PENDING');
INSERT INTO outbox (id, topic, payload)
VALUES ('evt-1', 'order-events', '{"type":"OrderCreated","orderId":"ord-99"}');
COMMIT;
-- Relay or CDC publishes evt-1; crash after COMMIT is OKFlash sale runs this pattern in all three services. That is why the design shows dashed outbox arrows on the diagram. See on the distributed transactions note for polling vs CDC.
Timeouts and stuck sagas during the event
What to watch during go-live
- Metrics: saga_started, saga_completed, saga_failed, saga_compensated
- One trace id = saga_id across gateway, orchestrator, and services
- Alert if compensation rate spikes above ~5% (upstream inventory or PSP issue)
Traps to avoid
These mistakes show up fast under flash-sale load.
Non-idempotent refund
Retry compensation → double refund. Always key refunds by orderId + saga step.
Missing compensation mapping
Charge without Refund defined → money stuck. Enforce 1:1 forward/compensate in code review.
Showing “success” too early
UI green check before saga COMPLETED → angry users. Show pending until orchestrator confirms.
Skipping outbox
Publish then DB, or DB without outbox → lost events under load. Flash sale traffic exposes this fast.
Data model (what to store)
sagas
Support and ops read this first.
id, order_id, user_id
state, current_step
compensation_status JSON
created_at, updated_atorders
Business record for the shopper.
id, user_id, status
total_amount, idempotency_key
created_atoutbox (×3 services)
Reliable bridge to the broker.
id, aggregate_id
event_type, payload
published, created_atinventory_reservations
Authoritative stock hold.
sku_id, order_id, qty
status, expires_at, versionpayments
Match PSP receipts.
order_id, gateway_ref
amount, status, idempotency_keyprocessed_events
Dedupe at-least-once delivery.
event_id PK
handler, processed_atAdmission control (do not skip this)
The difference between a busy site and a melted site.
Virtual waiting room
Users see queue position. Gateway only accepts checkout with a signed token that expires in minutes.
Rate limits
Per user and per IP on checkout. Stops bots from burning Redis and saga capacity.
Graceful degradation
Turn off recommendations and heavy cart features. Keep the buy button path warm.
Load shedding
Return 503 with Retry-After on non-critical APIs before the orchestrator intake queue explodes.
Hard part: no stock oversell
The fast counter protects latency, but the inventory database protects truth.
Invariant
Confirmed quantity for a SKU must never exceed sellable quantity: sum(confirmed) <= stock - safety_buffer.
Counter drift
If an illustrative 10,000 admitted checkouts/s creates one provisional counter reservation each, and 1% needs release, the compensation stream sees 100 releases/s. Five minutes without processing produces 30,000 pending corrections. That estimate depends on actual successful reservations, not raw clicks. Release records need deduplication, and restored workers need spare capacity beyond live arrivals to clear the backlog.
Fencing check
Inventory reserve uses WHERE available >= qty AND version = expected; stale workers cannot confirm old holds. Version checks prevent a worker from applying a decision based on stale inventory, but the version and quantity checks must execute in the same authoritative write. A hold should have its own identity, owner, status, and expiry. Confirmation consumes that exact active hold; expiry and cancellation transition it once. This prevents both overselling and duplicate release under retried saga messages.
Deep-dive drill
- Gateway decrements the fast stock counter and starts a saga, returning pending to the user.
- Payment succeeds, but inventory reserve fails because the authoritative row is already sold out.
- Orchestrator marks the saga failed, emits an idempotent refund command, and sends a restock correction only if the fast counter was decremented.
- Reconciliation compares
initial_stock - confirmed - holdswith the fast counter and writes bounded corrections. - If drift exceeds a threshold, pause admission for the SKU and serve sold-out until inventory is reconciled.
Bottlenecks and how to talk about them
Hot SKU in Redis
Risk: One key sees 10K decrements per second.
Mitigation: Shard counters per SKU, coalesce on gateway, or split stock into N logical buckets.
Orchestrator DB
Risk: Every saga step writes state.
Mitigation: Partition by event_id, batch audit logs async, scale writers horizontally.
Payment gateway limits
Risk: PSP throttles you mid-sale.
Mitigation: Bounded parallel payment workers; honest “try again” UX.
Second-zero thundering herd
Risk: Everyone arrives at once.
Mitigation: Warm CDN and Redis, stagger admission waves, load-test replay queue before go-live.
When parts break (degraded modes)
Production readiness
How to operate the sale without relying on luck.
SLOs and metrics
- Admitted checkout p99 <500ms until pending response; order finalization p95 <2 min.
- No confirmed oversell; saga success rate, compensation age, queue lag, counter drift, and payment timeout rate are primary dashboards.
- Error budget is scoped to checkout, not cached browse pages.
Alerting
- Page on oversell invariant breach, replay/DLQ growth, payment capture mismatch, or inventory reserve error spike.
- Warn on queue wait estimate drift, CDN origin miss surge, hot-key saturation, and compensation retries.
- Alert per sale event and SKU so one hero SKU does not hide systemic failure.
Runbook and degraded mode
- Fast counter down: lower admission and use authoritative inventory only for a smaller buyer pool.
- Broker lag high: pause new sagas, keep queue positions, and show "processing delayed" instead of accepting blind orders.
- Payment outage: stop checkout intake for affected methods and keep browse/queue alive.
Security and threat model
- Assets: admission tokens, payment intents, stock counters, inventory rows, and order status.
- Actors: bots, token replay, compromised accounts, cart hoarding, and fake callbacks.
- Controls: signed short-lived tokens, per-account/device limits, idempotency keys, callback signatures, and immutable audit logs.
Test plan
- Load test second-zero arrival, hot SKU key pressure, worker scale-up, and queue wait accuracy.
- Fault inject payment success plus inventory failure, lost outbox publish, duplicate broker delivery, and restock replay delay.
- Run reconciliation tests that prove confirmed units never exceed initial stock.
Rollout, DR, and 10x scale
- Dry run sale events with shadow traffic, prewarm cache/counters, and freeze risky deploys during the sale window.
- DR uses event-level write ownership; do not multi-write one SKU across regions without quota partitioning.
- At 10x: split stock into regional or logical buckets, isolate hero SKUs, preallocate dedicated worker pools, and move analytics off the checkout path.
Beginner vs advanced answers
Beginner (5 minutes)
CDN for reads, queue for overload, Redis to reject sold-out fast, then order → pay → stock with rollback if stock fails.
Advanced (deep dive)
The deeper design traces one checkout through admission, durable acceptance, authoritative reservation, payment outcome, confirmation, and compensation. Outbox publishing preserves recoverable step requests, while consumer deduplication and state versions reject repeated or stale replies. Reconciliation compares the order, inventory hold, payment provider reference, and provisional cache reservation. Monitor oldest pending saga age and repair progress so uncertain outcomes do not disappear behind a healthy request rate.
Choosing and explaining flash sale checkout
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
Separate “looking” from “buying”: cache browse, protect purchase with admission + saga.
Sync Redis only for a fast no; async saga for the real commit across three databases.
Outbox + orchestrator = every step auditable; compensation when payment and inventory disagree.
When it belongs
Traffic spikes 100× on a few SKUs for minutes, not hours.
They ask how you avoid overselling without one giant database transaction.
They want saga, queue, cache, and failure handling in one coherent story.
How it fails
Charged card, inventory says no, refund job stuck in replay queue.
Double checkout click starts two sagas (missing idempotency key).
Redis counter and DB quantity diverge after partial compensation.
Trade-offs to weigh
Orchestration vs choreography: orchestration is easier to explain and debug in interviews.
Redis pre-check vs saga-only: faster rejects, but you must restock Redis on undo.
Showing “pending” vs blocking until done: pending scales; blocking melts thread pools.
Common misunderstandings
Letting the mobile app call Redis stock keys directly.
Publishing to Kafka before the local database commit.
No waiting room on second zero of the sale.
Deeper walkthrough
Flash sale deeper walkthrough
A flash sale is inventory locking under a stampede: admit users fairly, reject sold-out in milliseconds, run a saga with outbox-backed services, and compensate when any step fails.
Requirement refinements
How many SKUs, max queue wait, refund SLA, single-SKU only or cart?
Invariants: no oversell on flash lines; one charge per idempotency key; saga state machine is auditable.
API design details
Checkout requires Idempotency-Key and admission token; all internal events carry saga_id.
Expose order status matching orchestrator states, not raw service logs.
Data model and storage
Orchestrator sagas table is ops source of truth; links order_id and idempotency_key.
Per-service outbox + processed_events for dedupe; Redis mirrors hot SKU availability.
Scaling strategy
Partition broker and saga shards by order_id; dedicated Redis shard per hero SKU.
Autoscale orchestrator on intake queue depth, not CPU alone.
Reliability, recovery, and operations
Replay queue with backoff and DLQ; nightly reconciliation: Redis vs inventory DB vs payment captures.
Dashboards: admission rate, redis reject %, saga success, compensation latency, broker lag.
Key takeaway
Draw the HLD, then tell a story: edge protects origin, queue protects purchase, Redis says no in one millisecond, runs the real checkout over Kafka with in every service, and on the replay queue fixes the messy middle when step three fails after step two succeeded. That is a complete flash-sale answer.
Primary sources
- Google SRE: Handling Overload: admission control under spikes.
- Redis Docs: INCR / counters: inventory counters under contention.
- Transactional Outbox pattern: order placement without dual-write gaps.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Protect a scarce inventory invariant with admission, fair queueing, reservations, idempotency, bot controls, and reconciliation under extreme skew.
Explain the decision
Why is a distributed lock per purchase not the complete inventory design?
Read the reasoning
The authoritative conditional decrement/reservation and durable order linkage prove stock; locks add contention and can expire around stalled clients.
Apply it to a production scenario
One SKU has 10,000 units and 5 million buyers arrive in a minute. What is the path?
Read one approach
Edge admission and bot checks, fair virtual queue, bounded reservation service, conditional inventory updates, hold expiry, and honest sold-out state.
Work through the numbers
Five million arrivals/minute equals what average ingress QPS?
Read the worked calculation
About 83,333 QPS; only a capacity-sized fraction should reach scarce inventory or payment dependencies.
Reason through a failure
What is the stock invariant?
Read the recovery reasoning
Confirmed plus unexpired reserved units never exceeds available units, across retries, timeout ambiguity, expiry, cancellation, and recovery.
Sources and further study
Continue with Design Distributed Rate Limiter.