System Design Case Study
E-commerce marketplace
A shopper finds a deal during a Prime Day–style , taps Buy Now, and expects the item, not a surprise "out of stock" email after being charged. Browsing can be slightly stale. The cannot.
Think of e-commerce as search plus a money-safe checkout workflow: recheck price, inventory, coupons, payment, and order state before promising the item.
Browse → pay journey
Cached catalog is OK if slightly stale.
Atomic hold before you charge.
Authorize/capture without double charge.
Order events; reverse on failure.
Try it: browse vs checkout on a marketplace
Browse is cache-friendly and eventually consistent. Checkout needs inventory reservation and strong payment outcomes.
- Client
- Catalog
- Cart
- Inventory
- Payments
- Catalog reads
- 0
- Reservations
- 0
- Mode
- browse
- Path hop
- 0
Browse
eventual0 catalog reads.
Checkout
reservation0 inventory holds.
Failure
paymentFailed capture must release reservation to avoid phantom stock locks.
Browse catalog pages, then switch to checkout to reserve inventory before payment.
What actually happens
Product pages and recommendations may show an old "in stock" label. Checkout revalidates against authoritative inventory, creates an , then authorizes payment with an . Only then does the order commit. Create a durable pending checkout record before beginning external payment work, so every attempt has an owner and recovery state. The final confirmed order appears only after the required inventory and payment conditions are satisfied. Search may display stale availability, but the reservation operation checks the authoritative stock condition atomically and returns a clear conflict when another shopper wins.
Across Order, Inventory, and Payment you use a : each step has a compensation (release hold, refund, cancel). on inventory versions prevents when thousands of workers race the same SKU.
Carts merge on login; payments use ; flash sales admit traffic through queues and rate limits so inventory writers stay protected.
How to choose consistency zones
1. Browse vs money? Search and recs can lag. Inventory, payment, and order ledger need strong controls.
2. Reserve before charge? For this scarce-stock design, reserve inventory before payment capture so a successful charge is likely to have an item behind it. Other products may authorize first under a different workflow, so state the chosen order and compensation. If a later commit fails, reconcile the existing payment reference and hold ID rather than starting another charge.
3. Flash sale? Queue admission, per-SKU throttling, pre-warm caches, circuit-break flaky payment providers: do not let browse capacity steal checkout capacity.
Simple rule of thumb
Search can be eventually consistent; a paid order must be traceable, inventory must not be silently oversold, and payment retries must be idempotent.
In simple terms: let people browse loosely, but settle money and stock like a bank teller.
Case-study upgrade
E-Commerce Marketplace Math
An e-commerce marketplace can show stale catalog data, but checkout must revalidate price, inventory, and payment state.
Assumptions and formulas
- Orders: 1M/day normally. Flash-sale peak: 10K checkout attempts/sec.
- Product reads and search are far larger than successful purchases.
- Inventory holds expire automatically and must be visible to checkout.
Concrete math
- Orders: 1M/day = 11.6/sec average; flash sale can reach 10K/sec.
- Product reads: 100M users x 20 views/day = 2B/day = 23K QPS average; peak can reach 70K to 200K QPS.
- Cart writes: 100M users x 3 mutations/day = 300M/day = 3.5K/sec average.
- Inventory holds: 10K checkout/sec x 10-minute hold = 6M concurrent holds during a flash sale.
- Order events: 1M orders/day x 8 events x 500 bytes = about 4 GB/day raw.
Design decisions forced by numbers
- Cache browse and search, but always revalidate authoritative inventory at checkout.
- Use admission control for flash-sale checkout so inventory service stays within budget.
- Make payment and order transitions idempotent and auditable.
Scaling to millions
- At 1M users: catalog cache, cart store, inventory table, and order workflow are enough.
- At 10M users: add search index, product cache warming, inventory holds, and async order events.
- At 100M+ users: add waiting rooms, per-item hot partition isolation, and checkout degradation.
Readable flow
Interview follow-ups to expect
- Which reads can be stale, and which checkout checks must be authoritative?
- How do we prevent overselling a hot item?
- What happens when payment succeeds but order confirmation times out?
Operational readiness checklist
After the math, explain how this capacity plan is measured, protected, launched, and evolved.
Observability
- Attach each capacity estimate to a dashboard metric: p99 latency, queue age, cache hit rate, shard skew, error rate, and freshness lag.
- Alert on rate-of-change and saturation, not only hard failures; backlogs and hot keys usually warn before an outage.
Security and abuse
- Budget for auth checks, rate limits, quota isolation, audit logging, privacy filters, and abuse-review side paths.
- Separate trusted internal events from user-controlled input so retries and backfills cannot bypass policy checks.
Degraded modes
- Define the fallback when a dependency is slow: stale cache, partial results, read-only mode, delayed async work, or admission control.
- Protect the invariant first; reduce freshness, personalization, or analytics quality before risking duplicate or corrupt state.
Testing and rollout
- Test average, peak, and hot-object traffic; include retry storms, cold caches, lagging workers, and regional dependency loss.
- Use canaries, shadow reads, dual-write validation, feature flags, and explicit rollback criteria for risky path changes.
Rollout, backfill, and runbooks
- Make migrations and backfills idempotent, resumable, rate-limited, and visible through progress and error metrics.
- Write the runbook before launch: owner, dashboards, paging thresholds, mitigation switches, replay steps, and repair commands.
10x evolution
- For e-commerce marketplace math, state which number forces the next move: cache memory, shard count, write amplification, queue lag, egress, or failover reserve.
- Add the next layer only when the metric forces it: sharding, regional reads, async replay, hot-key isolation, or stricter backpressure.
Scope you should nail down
Product surface
- Catalog, variants, search with facets and sort
- Cart persist +
- Checkout, orders, tracking, reviews, recommendations
- Payments (PCI), inventory, flash-sale behavior
Scale targets
- ~300M users, ~100M DAU
- ~1M orders/day; peak ~10K orders/sec in sales
- ~100M searches/day; search p99 <200ms; checkout p99 <500ms
- ~99.99% availability
Capacity picture
Searches: 100M/day → ~1,200/s (peak ~5K/s)
CDN: product images ~10TB/day
Orders: 1M/day ≈ 11.6/sec average; sale peak ~860× average
Product reads: 100M DAU × 20 views ≈ 23K QPS avg; peaks 70K–200K
Cart writes: ~3.5K/sec average
Inventory holds: If 10,000 successful holds/s each lasts the full ten minutes, Little’s Law gives six million concurrent holds. Checkout attempts do not all become holds, and purchases or cancellations shorten residence time. Size the actual hold store from successful arrival rate and measured duration, while separately limiting attempts that contend for a small hot SKU.
Browse vs checkout paths
Read the diagram definition
sequenceDiagram
autonumber
participant U as User
participant S as Search / catalog
participant C as Cart service
participant I as Inventory service
participant P as Payment service
participant O as Order service
Note over U,S: Browse path (can be stale)
U->>S: searchProducts / getProduct
S-->>U: cached catalog + search results
Note over U,O: Checkout path (authoritative)
U->>C: checkout preview
C->>I: reserve inventory (sync)
I-->>C: hold token
C->>P: authorize payment
P-->>C: auth id
C->>O: create order (idempotent)
O->>I: confirm reservation
O->>P: capture
O-->>U: order confirmedSearch and product detail can be eventually consistent; checkout must revalidate price, inventory, and promotions against authoritative services.
APIs worth sketching
searchProducts(query, filters?, sort?, page, limit) → Product[] with facets
getProduct(product_id) → Product details, variants, reviews summary
addToCart(user_id, product_id, quantity, variant?) → Cart updated
getCart(user_id) → Cart items
checkout(user_id, cart_id, shipping_address, payment_method) → Order preview, validation
placeOrder(user_id, cart_id, shipping_id, payment_id) → Order ID (idempotent)
getOrderStatus(order_id) → status, tracking
updateInventory(product_id, warehouse_id, delta) → New stock level (internal API)Core data model
Products
PK id
sku, name, description
price, currency
category_id, brand_id
image_urls JSON
created_at, updated_atUsers
PK id
email, name, created_at
default_shipping_address_idOrders
PK id
user_id, status, total_amount
shipping_address_id, payment_id
created_atOrderItems
order_id, product_id
quantity, price_at_purchase
variant JSONCart
user_id (or session_id)
product_id, quantity
variant, updated_atInventory
product_id, warehouse_id
quantity, reserved
version (optimistic lock)Reviews
id, product_id, user_id
rating (1-5), content
created_atCategories
id, name, parent_id
path (materialized)Architecture
Catalog, search, and cart
Search index
Full-text on name/description/brand; fuzzy tolerance; analyzers for facets.
Aggregations for price, brand, category, rating; return facet counts with results.
Ranking
BM25 + popularity + recency; boost by rating/reviews; A/B test models.
Session cart
Anonymous: in-memory cache by session_id, TTL ~7 days. Fast hash/JSON of product, qty, variant.
Keep the signed-in cart durable and treat its cache as a fast view. On login, merge the anonymous and account carts using a declared rule for matching variants, quantities, and removed items. Make the merge idempotent so retries do not keep adding quantities. Cart prices and availability are estimates until checkout revalidates them; saving an item in a cart does not reserve stock.
Order pipeline and saga
Flow
- Validate cart and addresses
- Check stock; reserve with optimistic lock
- Process payment (idempotent)
- Confirm order + items; notify
- Fulfillment: warehouse allocate, ship
// Saga steps with compensations
1. ReserveInventory() → Compensate: ReleaseInventory()
2. ChargePayment() → Compensate: RefundPayment()
3. CreateOrder() → Compensate: CancelOrder()
// On any failure: run compensations in reverse orderHard part: checkout inventory invariant
Search can be stale; checkout cannot
Authoritative hold, not a cache update: before creating a confirmed order.
-- Authoritative hold, not a cache update
UPDATE inventory
SET reserved = reserved + :qty,
version = version + 1
WHERE sku_id = :sku
AND warehouse_id = :warehouse
AND quantity - reserved >= :qty
AND version = :expected_version;
-- Success means one hold was created.
-- 0 rows means sold out or stale version; do not charge.Invariant
For each SKU and warehouse, define the stock fields before writing the invariant. If allocatable supply is the total budget before outstanding holds and confirmed allocations, require held units + confirmed allocations ≤ allocatable supply. An atomic conditional transition moves units between those categories. Cancellations and expiry release only their own active hold once, while fulfillment adjusts the stock model under its separately defined transition.
A cached count is a display hint because many callers can read the same value before any reservation changes it. If five units remain and 100 workers each read five, a read-then-write implementation can accept all 100. An atomic conditional decrement or version-checked reservation serializes that decision, admitting only the remaining units. Retry conflicts within a budget; aggressive retries on one sold-out SKU only add contention.
Payment ordering
Reserve first, then authorize. Reconcile payment reference + hold id if commit fails after charge.
Inventory, payments, recommendations
Inventory
Store on-hand units, reserved units, and a version with precisely defined transitions. One possible model derives available = on_hand − reserved: a hold increases reserved, cancellation decreases reserved, and shipping decreases both on_hand and the shipped reservation. If confirmed-but-unshipped orders use another field, include it in availability. Allocate among warehouses under those local invariants, with an explicit partial-failure policy for multi-item orders.
- Tokenize cards; never store PANs
- Authorize/capture + webhooks
- Idempotency keys; idempotent refunds
- Circuit breakers across providers
Recommendations
Collaborative filtering (ALS/SVD), item-item co-purchase similarity, hybrid with content and trending. Batch precompute + real-time recent views; hide modules if the service is down.
and bottlenecks
Queue-based ordering
Accept into a stream; workers process at controlled rate; notify on confirm.
Rate limiting
Per-user checkout limits; token bucket at the gateway.
Pre-warm cache
Hot product details and inventory counts before the sale opens.
Circuit breakers
Fail fast when a payment provider is known unavailable before a useful attempt begins. After a request has been sent, a timeout is an unknown outcome, so do not assume that no payment exists. Keep the checkout pending while querying or retrying with the same payment identity. Hold expiry can still proceed under policy, but a later successful charge must reconcile to a valid order or a refund.
Inventory oversell
Optimistic locking, reserved qty, sale queues.
Payment gateway
Multi-provider, backoff, circuit breakers.
Search latency
Index tuning, cache popular queries, pre-aggregate facets.
Order write spike
Queue orders; shard by user_id or time.
Production readiness
SLOs and metrics
- Product page p99 <200ms from cache; checkout confirm p99 <500ms for admitted users
- Oversell incidents → zero; hold expiry p95 lag <1 min
- Search freshness, cart merge failures, payment reconciliation lag, order event lag
Alerting
- Page on oversell breach, payment/order mismatch, checkout success drop
- Warn on index lag, cart errors, abandoned checkout surge, high hold age
- Separate browse vs checkout alerts
Degraded modes
- Search down: cached categories / popular / SKU lookup; checkout still revalidates
- Recs outage: hide modules; preserve cart/checkout
- Payment outage: pause confirm for that method; release holds on timeout
Security
- Assets: tokens, addresses, inventory, coupons, order history
- Threats: bots, coupon abuse, seller fraud, payment replay
- Controls: authz, signed webhooks, idempotency binding, fraud checks, audit logs
Takeaway
Microservices for product, search, cart, order, payment, inventory, and recommendations. Browse loosely; checkout with holds, sagas, and idempotent payments, especially under flash-sale load.
Beginner answer
Catalog, search, cart, inventory, checkout, payment, order creation, fulfillment, notifications.
Advanced answer
Inventory reservation, checkout idempotency, payment/order reconciliation, stale search validation, flash-sale admission, saga compensation.
Deeper walkthrough
E-commerce deeper walkthrough
E-commerce is a collection of different consistency zones. Search and recommendations can be eventually consistent; inventory, order state, payment, and ledger records need much stronger controls.
Requirement refinements
Clarify catalog browsing, search, cart, checkout, inventory, payment, fulfillment, returns, coupons, and flash-sale behavior.
State the core invariant: a paid order must be traceable, inventory must not be silently oversold, and payment retries must be idempotent.
API design details
Use checkout idempotency keys and return the same order or payment result for duplicate client submissions.
Validate final price, tax, coupon, and inventory at checkout even if search or cart data is stale.
Data model and storage
Separate catalog, inventory, cart, order, payment, shipment, and return records by ownership and access pattern.
Use order_events for state transitions so support, fulfillment, and reconciliation can audit the order lifecycle.
Scaling strategy
Use search indexes and caches for browse traffic, but use authoritative inventory services during checkout.
For flash sales, add admission control, per-SKU throttling, and short inventory reservations.
Reliability, recovery, and operations
If payment succeeds but order creation fails, reconcile by payment processor reference and checkout idempotency key.
Monitor checkout success rate, oversell incidents, inventory hold age, payment reconciliation lag, and search freshness.
Deepening reference
E-Commerce Platform: realistic production notes
Beginner mental model: E-commerce is catalog search plus cart state plus inventory correctness plus payment/order workflows.
Production scope: The hardest parts are inventory reservation, checkout idempotency, order state transitions, search freshness, recommendations, and flash-sale overload.
Choose what the first version must guarantee
Start by deciding which e-commerce platform capabilities are essential. Describe the main read and write operations before choosing infrastructure. For each operation, identify the stored records it uses and the condition that makes its result correct. A fast response is useful only if it satisfies that condition.
Next, separate work the user must wait for from work that can finish later. Estimate requests per second, storage growth, and the busiest object or customer. Use those estimates to choose one bottleneck to investigate. The calculations below are example workloads with stated assumptions; they are not measured limits of a particular product.
Core flow to walk through
Browse: search index returns products, pricing, availability, and ranking signals.
Checkout: validate cart, reserve inventory, authorize payment, create order, publish fulfillment event.
Fulfillment: warehouse allocation, shipment tracking, notification, return/refund flow.
Production readiness
E-Commerce Platform: ship, operate, and evolve
Now examine the same request flow during failures and deployments. Tie each operational control to the user behavior it protects.
Observability
Track e-commerce platform p95/p99 latency, error rate, saturation, and user-visible freshness separately for read, write, and async paths.
Alert on leading indicators such as queue age, hot-key skew, cache miss storms, reconciliation mismatches, and error-budget burn.
Security and abuse
Enforce authentication, authorization, rate limits, input validation, audit trails, and privacy checks at the service boundary.
Treat abuse controls as a first-class path: throttling, quota isolation, moderation hooks, revocation, and safe handling of sensitive data.
Degraded modes
Name what still works if cache, search, ranking, stream processing, object storage, or a regional dependency is slow.
Prefer stale-but-safe reads, partial results, async retry, admission control, and feature disablement over corrupting core state.
Testing and rollout
Load test average, peak, and hot-object traffic; add failure-injection for retries, timeouts, backlog growth, and dependency outages.
Ship with canaries, shadow reads or dual writes when useful, rollback switches, schema compatibility checks, and replay validation.
Runbooks
Document dashboards, paging thresholds, owner actions, rollback steps, data repair commands, and customer-impact language.
Keep backfills and replays idempotent, resumable, rate-limited, and observable so repair work does not create a second incident.
10x evolution
Say which metric forces the next architecture change: shard count, cache footprint, write amplification, queue lag, p99 latency, or regional failover reserve.
Evolve one bottleneck at a time: partition by access pattern, isolate hot tenants, regionalize reads, add replayable async pipelines, then harden multi-region behavior.
Hard parts worth deepening
Inventory reservation needs short holds and clear release rules.
Checkout must be idempotent because users retry and networks fail.
Catalog search can be eventually consistent; payment and inventory cannot be sloppy.
Flash sales need queueing, admission control, and per-SKU hot-key protection.
Failure modes and recovery
Payment succeeds but order creation fails; reconcile by idempotency key and processor reference.
Inventory oversold; use atomic reservation per SKU and compensate quickly.
Search index stale; checkout validates true inventory and price before purchase.
Promo service outage; checkout degrades by disabling new promo validation, not by losing orders.
Signals to monitor
Checkout success measures completed purchases, while oversell incidents identify violations of the inventory rule. Track the age of inventory holds to detect stock trapped by abandoned or ambiguous checkouts.
Payment reconciliation lag measures how long local order state disagrees with the processor. Search-to-cart conversion describes the earlier product journey and helps distinguish discovery problems from checkout failures.
| Choice | Best when | Cost |
|---|---|---|
| Reserve inventory at cart time | Limited stock items need strong UX. | Users can hoard stock. |
| Reserve inventory at checkout | Most inventory is plentiful. | Cart may fail late. |
| Microservices by domain | Catalog, order, payment, fulfillment evolve independently. | Distributed transactions become workflows. |
Try this scenario
Explain the exact state transitions when payment authorization succeeds but inventory reservation fails.
Primary sources
- Amazon Dynamo paper: cart-style availability trade-offs.
- AWS Well-Architected Framework: reliability/security pillars for retail.
- Saga pattern: checkout spanning inventory/payment.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Coordinate catalog, price/promotion, inventory, cart, order, payment, seller settlement, returns, and reconciliation without one giant transaction.
Explain the decision
Where should inventory become authoritative during checkout?
Read the reasoning
A conditional reservation/hold in the inventory owner, with expiry and idempotent order linkage; the cart remains advisory.
Apply it to a production scenario
Payment succeeds but order confirmation times out. What does the client do?
Read one approach
Retry with the same idempotency key, query operation state, and let reconciliation repair any processor/order mismatch without charging again.
Work through the numbers
Peak 20,000 checkouts/s reserve an average 3 items. What inventory mutation rate?
Read the worked calculation
About 60,000 item reservations/s before retries, releases, and hot-SKU contention.
Reason through a failure
What must price presentation preserve?
Read the recovery reasoning
The checkout records the exact price/promotion version accepted by the user; later catalog changes cannot silently alter a committed order.
Sources and further study
Continue with Design Flash Sale (HLD).