System Design Drills
Realistic Case Studies
Imagine you open a Google Doc with a teammate, tap Buy on a flash checkout, or hail a ride while the city spikes after a concert. Each of those products has one sharp bottleneck, not generic CRUD.
These drills are timed practice, not textbooks. Each one isolates a realistic hard part: convergence, , ingestion, , matching, or device storms. So you rehearse product story → numbers → architecture → failure handling in 45–75 minutes.
How to run a drill
Pick one case
Set a timer for the suggested minutes.
Write the promise before drawing boxes.
Derive two formulas
Force at least two capacity numbers from the story.
Self-check the math
Compare your estimates to the worked answers after.
Try it: realistic case study with retries and idempotency
Simulate a marketplace order transition. The first delivery commits state; a retry with the same idempotency key must return the prior outcome without duplicating side effects.
- Product event
- Durable log
- Worker
- Read model
- User view
- State transitions applied
- 0
- Duplicate attempts blocked
- 0
- Worker queue depth
- 0
- Idempotency key
- evt-7
Durability
logThe log is the source of truth; workers and read models are derivations.
Idempotency
exactly-once effectApplied: 0. Blocked duplicates: 0.
Backlog
capacityQueue depth 0. If workers fall behind, age-based shedding or scale-out is required.
Deliver the same state-changing event twice. With an idempotency key the second attempt should not double-apply.
What actually happens in a strong drill
You start from a user story: typing in a shared doc, completing checkout, matching a rider. Then name the . For collaboration that often means durable entries and convergence via sequencing or a . For checkout it means a that can compensate. For marketplace spikes it means a fair under load.
Next you estimate , storage, and amplification until a number forces a move: shard, cache, queue, or . Only then do you draw architecture. The worked math below is a self-check after you finish, not a script to memorize.
How to choose which drill
1. Need concurrent edits? Start with the collaborative document drill: ordering, presence, and / OT trade-offs.
2. Need multi-step money flows? Use the flash-sale or booking-style drills:, compensation, and .
3. Need markets or device storms? Practice matching and ingestion under spike load with explicit .
Simple rule of thumb
One drill, one bottleneck. Do not try to redesign the entire company. Protect the , show the math, and explain failure handling for that hard part.
Case-study upgrade
Realistic Case Study Math
Realistic problems are product stories with one hidden bottleneck. First find what must be exact, then calculate the stream, storage, and worker load.
Assumptions and formulas
- For collaboration, count edit operations, hot documents, snapshots, and presence fan-out.
- For marketplace and matching systems, count state transitions, candidate scans, and timeout windows.
- For telemetry and metrics, count messages/sec, series cardinality, compressed blocks, and alert evaluator work.
Concrete math
- Collaborative editor: 1B edit operations/day = 11.6K ops/sec average; a hot document with 1K editors can create thousands of presence messages/sec.
- Delivery marketplace: 100K orders/minute in a region = 1.67K orders/sec; scanning 50 candidates/order gives 83K candidate scores/sec.
- Device telemetry: 10M devices x 1 message/minute = 167K messages/sec; a 5x reconnect burst becomes 835K messages/sec.
Design decisions forced by numbers
- State-machine cases need idempotency keys, audit logs, and clear timeout transitions.
- Ingestion cases need partitioned streams, backpressure, replay, and late-event handling.
- Hot collaborative objects need a single sequencer or explicit conflict-merge policy.
Scaling to millions
- At 1M users: one region and simple queues are enough for most practice cases.
- At 10M users: isolate hot tenants, shard streams, and separate online serving from analytics.
- At 100M+ users: design replay, regional failover, quota protection, and partial-result serving.
Readable flow
Interview follow-ups to expect
- What must be exactly correct, and what can lag?
- How do retries avoid duplicate state changes?
- How large can the backlog become during a regional or worker outage?
Operational readiness checklist
After the math, explain how this capacity plan is measured, protected, launched, and evolved.
Observability
- Attach each capacity estimate to a dashboard metric: p99 latency, queue age, cache hit rate, shard skew, error rate, and freshness lag.
- Alert on rate-of-change and saturation, not only hard failures; backlogs and hot keys usually warn before an outage.
Security and abuse
- Budget for auth checks, rate limits, quota isolation, audit logging, privacy filters, and abuse-review side paths.
- Separate trusted internal events from user-controlled input so retries and backfills cannot bypass policy checks.
Degraded modes
- Define the fallback when a dependency is slow: stale cache, partial results, read-only mode, delayed async work, or admission control.
- Protect the invariant first; reduce freshness, personalization, or analytics quality before risking duplicate or corrupt state.
Testing and rollout
- Test average, peak, and hot-object traffic; include retry storms, cold caches, lagging workers, and regional dependency loss.
- Use canaries, shadow reads, dual-write validation, feature flags, and explicit rollback criteria for risky path changes.
Rollout, backfill, and runbooks
- Make migrations and backfills idempotent, resumable, rate-limited, and visible through progress and error metrics.
- Write the runbook before launch: owner, dashboards, paging thresholds, mitigation switches, replay steps, and repair commands.
10x evolution
- For realistic case study math, state which number forces the next move: cache memory, shard count, write amplification, queue lag, egress, or failover reserve.
- Add the next layer only when the metric forces it: sharding, regional reads, async replay, hot-key isolation, or stricter backpressure.
Fast comparison
| Case | Main hard part | Core patterns |
|---|---|---|
| Collaborative Document Editor | All acknowledged edits are durable, ordered, permission-checked, and eventually converge on every authorized client. | WebSocket session service, Collaboration service, Operation log, Snapshot worker |
| Food Delivery Marketplace | An order has one clear state at every moment, payment changes are auditable, and a delivery task is assigned to at most one active courier. | Catalog service, Order service, Payment service, Dispatch service |
| Metrics and Alerting Platform | Accepted samples are queryable within a bounded delay, tenant limits are enforced, and alerts are evaluated on the intended data window. | Ingest router, stream log / partitioned log, TSDB writers, Query frontend |
| Ad Click and Conversion Attribution | A billable event is deduplicated, attributable, auditable, and computed from immutable raw events. | stream event stream, Stream dedup processor, Attribution processor, OLAP warehouse |
| Stock Trading Platform / Matching Exchange | Orders for a symbol are matched in deterministic price-time priority, every execution is durable, and client retries cannot create duplicate live orders. | Order sequencer, Matching engine per symbol, Durable journal, Execution report service |
| IoT Telemetry and Device Command Platform | Device data is authenticated, tenant-isolated, durable after acceptance, and commands are delivered at most once or safely retried according to command semantics. | Auth policy service, stream log telemetry stream, Validation + enrichment, Hot TSDB + object storage |
Collaborative Document Editor
Collaborative document editing with real-time sync
Why this drill belongs here: Practice convergence and ordering, not generic CRUD. The unique bottleneck is concurrent edits, not document storage size.
Design a multi-user document editor where people can type together, see presence, comment, work offline briefly, and recover version history. The hard part is making every client converge to the same document even when edits arrive concurrently or out of order.
Invariant: All acknowledged edits are durable, ordered, permission-checked, and eventually converge on every authorized client.
Scale assumptions
Worked capacity math
Formulas tied to design decisions
- 1B edit ops/day / 86,400 ≈ 11.6K ops/sec average; peak 3× ≈ 35K ops/sec.
- Hot doc: 500 editors × 30 ops/min ≈ 250 ops/sec on one document: needs single sequencer per doc.
- Operation log bandwidth: 11.6K average ops/sec × 500 bytes/op ≈ 5.8 MB/sec, or about 500 GB/day. At the assumed 3× peak, bandwidth is about 17.5 MB/sec before replication and protocol overhead.
- Snapshot every 1,000 operations at 35K peak operations/sec implies about 35 snapshots/sec across documents in steady state. Active document count alone cannot determine snapshot rate; snapshot bytes and idle-document policies also affect cost.
- Presence fan-out: 500 viewers × 2 heartbeats/min ≈ 17 presence msgs/sec per hot doc: keep out of durable log.
Requirements
Functional
- Create, edit, rename, share, and delete documents
- Real-time cursor, selection, and presence updates
- Comments, suggestions, mentions, and notifications
- Version history and restore to an older snapshot
- Offline edits that sync when the client reconnects
Non-functional
- Document convergence across all clients
- Low perceived latency while typing
- Durable operation history for recovery
- Permission isolation between workspaces
- Graceful degradation when real-time sync is unavailable
API and data model
Core APIs
- POST /docs -> create document
- GET /docs/{doc_id}?version=latest -> load snapshot + recent ops
- WS /docs/{doc_id}/sync -> send edits, receive edits, presence, acks
- POST /docs/{doc_id}/comments -> create comment or suggestion
- GET /docs/{doc_id}/history?cursor=... -> paginated version history
Important entities
High-level architecture
Document edit state machine (correctness-heavy)
Read the diagram definition
stateDiagram-v2
[*] --> Received: client sends op
Received --> Deduped: duplicate client op_id
Received --> Rejected: permission fail
Received --> Sequenced: auth ok
Sequenced --> Acked: assign seq_no + durable log
Acked --> Applied: broadcast to room
Applied --> Snapshotted: every N ops
Snapshotted --> Applied: continue editing
Deduped --> Acked: return prior ackDeep dives
OT vs CRDT
Beginner: Concurrent editing needs a rule for operations made from different document versions. If Alice inserts at the start while Bob deletes a later character, using their original numeric positions blindly can delete the wrong character. Operational Transformation adjusts operations against intervening edits; a CRDT uses identifiers and merge rules designed to converge under its delivery assumptions.
Advanced: OT is simpler with a central sequencer and strong server authority. CRDTs are stronger for offline-first editing but add metadata overhead, tombstones, and compaction complexity.
Ordering and deduplication
Beginner: Give each client operation a stable ID before sending it so a retry represents the same edit. In this sequenced design, the server durably assigns a document sequence and returns that assignment on duplicate submission. A client receiving sequence 105 after 103 detects a gap and retrieves 104 instead of silently applying a incomplete history.
Advanced: Use (doc_id, client_id, op_id) for idempotency. The server assigns monotonic seq_no per doc. Clients apply acked operations in seq_no order and buffer gaps.
Presence
Beginner: Cursors and online status do not need durable storage.
Advanced: Store presence in in-memory cache store or memory with TTL. Broadcast via the same WebSocket room, but keep it out of the operation log to avoid write amplification.
Hot documents
Beginner: A popular document can overload one collaboration worker.
Advanced: Pin each doc to one active sequencer for ordering, then shard fan-out with pub/sub side channels. Apply admission control for giant all-hands documents.
Failure modes
Client retries the same edit after timeout
Idempotency by (doc_id, client_id, op_id).
WebSocket disconnect loses presence
Presence TTL expires automatically; durable edits are replayed from last acked seq_no.
Snapshot worker writes a bad snapshot
Keep operation log as source of truth and validate snapshot version before promotion.
Search index lags
Document editing continues; search result freshness is eventually consistent.
Trade-offs
| Choice | Pick when | Watch out |
|---|---|---|
| Central sequencer per document | You want simpler ordering and easier debugging. | Hot documents need careful scaling. |
| CRDT offline-first model | Offline editing is a primary product feature. | Metadata growth and compaction are hard. |
| Snapshot every N operations | Opening old large docs from the full log is too slow. | Too frequent snapshots waste storage and CPU. |
Beginner to advanced summary
Beginner answer
Start with WebSockets, an append-only operation log, snapshots, and presence TTLs.
Advanced answer
The real discussion is convergence, ordering, idempotent retries, permission checks, hot documents, and log compaction.
Food Delivery Marketplace
food delivery ordering and dispatch
Why this drill belongs here: Practice sagas and geo-local partitioning. The hard part is order state plus dispatch, not restaurant catalog search.
Design a three-sided marketplace for customers, restaurants, and delivery partners. Users browse menus, place orders, pay, track preparation and delivery, and receive support updates. The hard part is coordinating payment, restaurant acceptance, courier dispatch, ETA, and cancellation without losing money or orders.
Invariant: An order has one clear state at every moment, payment changes are auditable, and a delivery task is assigned to at most one active courier.
Scale assumptions
Worked capacity math
Formulas tied to design decisions
- If 100K orders/hour is the global peak, it is about 27.8 orders/sec globally and 0.278/sec per city under an unrealistic even split across 100 cities. Size hot cities separately using their actual share; do not multiply global traffic by the city count.
- 50K couriers × location every 3 sec ≈ 17K location updates/sec globally.
- Dispatch scoring at the stated global peak: 27.8 orders/sec × 50 candidate couriers ≈ 1,390 candidate scores/sec globally. A city receiving 25% of those orders needs about 347 scores/sec before retries or batching.
- Offer timeout throughput: if 30 sec TTL and 10K active offers, expiry worker ≈ 333 expirations/sec at steady state.
- Active trip state: 200K concurrent trips × 300 bytes ≈ 60 MB hot state before indexes.
Requirements
Functional
- Browse restaurants by location, cuisine, rating, and ETA
- Cart, coupon, taxes, delivery fee, and checkout
- Restaurant accept/reject with prep time
- Courier dispatch, live tracking, proof of delivery
- Customer support, refunds, cancellation policy
Non-functional
- No lost orders after payment authorization
- Low-latency nearby restaurant and courier lookup
- Event-driven updates to customer, restaurant, and courier apps
- Regional isolation because traffic is strongly local
- Graceful degradation for maps, promos, or recommendations
API and data model
Core APIs
- GET /restaurants?lat=...&lng=...&cursor=... -> nearby restaurants
- POST /carts/{cart_id}/checkout with idempotency_key -> order_id
- POST /restaurants/{id}/orders/{order_id}/accept -> prep estimate
- POST /dispatch/tasks/{task_id}/offer -> courier offer with TTL
- WS /orders/{order_id}/track -> order state and location updates
Important entities
High-level architecture
Order lifecycle state machine
Read the diagram definition
stateDiagram-v2
[*] --> Created: checkout
Created --> Paid: authorize ok
Created --> Cancelled: payment fail
Paid --> Accepted: restaurant accept
Paid --> Refunded: restaurant timeout
Accepted --> Preparing: kitchen start
Preparing --> PickedUp: courier pickup
PickedUp --> Delivered: proof of delivery
Delivered --> [*]
Cancelled --> [*]
Refunded --> [*]Deep dives
Order state machine
Beginner: Define legal transitions and the owner that commits each one. A restaurant rejection after payment authorization should release the authorization; rejection after capture requires a refund. Store the transition and an outgoing event together so dispatch and notification can recover from a missed publish. A timeout creates an uncertain pending state until the external result is reconciled.
Advanced: Use allowed transitions, optimistic concurrency, and an order_events log. Compensate with refund or cancellation events if restaurant acceptance or courier pickup fails.
Dispatch matching
Beginner: Nearby lookup creates candidates; it does not assign the job. Score candidates using travel time, existing load, and eligibility, then make a time-limited offer. Acceptance must conditionally claim the still-open task and eligible courier so two dispatchers cannot both win. Expired offers may be retried, but a late acceptance must not overwrite a newer assignment.
Advanced: Score candidates by distance, ETA, active load, vehicle type, acceptance rate, and fairness. Use short offer TTLs and requeue if rejected or timed out.
Location streaming
Beginner: Courier app sends location every few seconds.
Advanced: Ingest to a location service, store latest point with TTL, publish order-scoped tracking events, and downsample updates for customers to control cost.
Checkout saga
Beginner: Payment and order creation are separate systems, so handle partial success.
Advanced: Authorize payment, create order, request restaurant acceptance, capture payment after acceptance, release authorization on rejection or timeout.
Failure modes
Payment authorized but order creation times out
Use checkout idempotency key and reconciliation job for orphaned authorizations.
Restaurant rejects after payment
Release authorization or refund; notify user with clear terminal state.
Courier accepts then disconnects
Heartbeat courier app; reassign after missed heartbeats before pickup.
Dinner rush overloads one city
Partition by city/geo_cell and shed non-critical recommendations.
Trade-offs
| Choice | Pick when | Watch out |
|---|---|---|
| Greedy nearest-courier matching | You need a simple low-latency dispatch baseline. | Can be globally inefficient during rush hour. |
| Batch optimization every few seconds | You want better fleet utilization. | Adds dispatch latency and implementation complexity. |
| Strong menu inventory checks | Sold-out items cause high support cost. | Restaurants may update menus slowly, so stale data still exists. |
Beginner to advanced summary
Beginner answer
Model the order lifecycle first. Then add dispatch, location, payment, and notifications around that state machine.
Advanced answer
The hard parts are sagas, geo partitioning, courier matching, ETA feedback loops, and partial failure recovery.
Metrics and Alerting Platform
Datadog / Prometheus style time-series ingestion
Why this drill belongs here: Practice write-heavy ingestion and cardinality control, not a generic dashboard CRUD app.
Design a platform that receives high-volume metrics from many customer services, stores time-series data, supports dashboard queries, and triggers alerts. The hard part is handling ingestion spikes and label cardinality while still answering queries fast.
Invariant: Accepted samples are queryable within a bounded delay, tenant limits are enforced, and alerts are evaluated on the intended data window.
Scale assumptions
Worked capacity math
Formulas tied to design decisions
- 2M samples/sec × 16 bytes/sample ≈ 32 MB/sec ingest before labels and replication.
- Series cardinality: 50K tenants × 10K series/tenant cap = 500M series upper bound: must reject earlier.
- Compressed blocks: 2M samples/sec × 86400 sec × 4 bytes compressed ≈ 690 GB/day hot tier before downsampling.
- Query fan-out: 10K concurrent dashboards × 20 series each × 24 h window → query frontend shards prune blocks by time and label.
- Alert evaluators: 500K rules/min ≈ 8.3K eval/sec; if each eval reads 50 ms, need ~415 concurrent eval workers before headroom.
Requirements
Functional
- Ingest metrics with labels and timestamps
- Query recent and historical metrics for dashboards
- Create alert rules, silences, and notification routes
- Downsample old data for long retention
- Tenant-level quotas and usage reporting
Non-functional
- High write throughput with backpressure
- Fast range queries on recent data
- Protection against label cardinality explosions
- At-least-once ingestion with dedup where possible
- Alert delivery with deduplication and escalation
API and data model
Core APIs
- POST /api/v1/metrics/write -> batch samples
- GET /api/v1/query_range?expr=...&start=...&end=...&step=...
- POST /api/v1/alerts/rules -> create alert rule
- POST /api/v1/silences -> mute alert labels for a time window
- GET /api/v1/usage -> tenant sample count and series cardinality
Important entities
High-level architecture
Metrics ingestion pipeline (stream-heavy)
Read the diagram definition
flowchart LR
A[Agent push/scrape] --> B[Ingest router]
B --> C[Cardinality guard]
C --> D[Partitioned log]
D --> E[TSDB writers]
E --> F[Hot blocks]
F --> G[Query frontend]
F --> H[Compactor / downsample]
H --> I[Cold object storage]
D --> J[Alert evaluator]
J --> K[Notification router]Deep dives
Cardinality control
Beginner: A time series is identified by its metric name and full label combination. Adding a request ID can create a new series for every request, increasing index, memory, and query work even when sample rate stays modest. Enforce tenant and metric budgets at ingestion and report which labels caused rejection so the customer can repair instrumentation.
Advanced: Apply tenant quotas, label allow/drop rules, top-N diagnostics, and ingestion rejection before data reaches the TSDB writer path.
TSDB storage
Beginner: Store samples in time-based compressed chunks.
Advanced: Partition by tenant and time. Keep recent blocks on fast storage, compact immutable blocks, downsample older data, and move cold blocks to object storage.
Query fan-out
Beginner: A dashboard query first identifies relevant series and time blocks, then reads and merges their samples. Time and label pruning reduce work before fan-out. Limit scanned bytes, series count, and execution time so one broad query does not starve live ingestion. If partial results are allowed, label the missing coverage instead of presenting an incomplete graph as a complete measurement.
Advanced: Use query frontends, result cache, partial responses, shard pruning, and downsampled data for long windows to keep p99 bounded.
Alert evaluation
Beginner: Run alert rules every interval and notify when the expression is true.
Advanced: Track rule state per label fingerprint, handle missing data explicitly, dedup notifications, and separate evaluation delay from notification delay.
Failure modes
Bad deployment emits millions of unique labels
Cardinality quotas and label drop rules at ingestion.
stream log lag delays alerts
Expose ingest lag, alert on evaluator lag, and prioritize alert-critical tenants.
Long dashboard query overloads TSDB
Query timeout, max samples limit, result cache, and downsampling.
Duplicate samples from retries
Dedup by series_id + timestamp on write or query merge.
Trade-offs
| Choice | Pick when | Watch out |
|---|---|---|
| Push-based ingestion | Agents sit behind firewalls or NAT. | Requires stronger rate limiting and auth at the edge. |
| Pull-based scraping | You control the infrastructure and want service discovery. | Harder across customer networks. |
| Exact high-resolution retention | Short hot windows need detail. | Gets expensive for long retention; downsampling is usually needed. |
Beginner to advanced summary
Beginner answer
Think of metrics as write-heavy time-series data: ingest, partition, compress, query, alert.
Advanced answer
The advanced answer centers on cardinality, backpressure, query fan-out, downsampling, and alert correctness under lag.
Ad Click and Conversion Attribution
Event pipeline for ads, billing, and reporting
Why this drill belongs here: Practice fast sync path vs heavy async pipeline. Correctness is dedup and attribution windows, not ad serving latency.
Design a system that records ad impressions, clicks, conversions, and campaign reports. Click redirects must be fast, duplicate events must not overbill advertisers, and conversions may arrive hours or days after the click.
Invariant: A billable event is deduplicated, attributable, auditable, and computed from immutable raw events.
Scale assumptions
Worked capacity math
Formulas tied to design decisions
- 5B raw events/day ≈ 57.9K events/sec average; clicks are a small fraction but need p99 redirect <50 ms.
- Dedup store: 57.9K/sec × 100 bytes/key × 86400 sec TTL window ≈ 500 GB/day active dedup if naïve: bucket fingerprints instead.
- For an assumed 100M simultaneously active click keys retained within a 24-hour attribution window, 64 bytes/key gives about 6.4 GB raw index state. The active-key count already includes the retention window; do not multiply by 24 hours again.
- Late corrections depend on conversion volume, not all raw events. If the five-billion-event daily stream includes 100M conversions and 2% arrive late, allow about 2M correction events/day, plus replay and retry overhead.
- Rollups: hourly campaign counters at 1M campaigns × 24 buckets × 32 bytes ≈ 768 MB/day: serve dashboards from rollups, not raw.
Requirements
Functional
- Record impressions, clicks, and conversions
- Redirect click to advertiser landing page
- Deduplicate retries and repeated clicks
- Attribute conversions to campaign, ad, click, and user context
- Produce near-real-time dashboards and billing exports
Non-functional
- Click path must be extremely low latency
- Raw events are immutable and replayable
- Late events are handled with watermarks
- Privacy deletion and consent constraints are enforceable
- Reporting can be eventually consistent; billing needs stronger auditability
API and data model
Core APIs
- GET /click/{ad_click_token} -> record click and 302 redirect
- POST /events/impression -> batch impression events
- POST /events/conversion -> conversion with click_id or attribution identifiers
- GET /reports/campaigns/{id}?start=...&end=... -> aggregated metrics
- POST /privacy/deletions -> remove or anonymize user-linked identifiers
Important entities
High-level architecture
Click redirect vs analytics pipeline
Read the diagram definition
flowchart LR
U[User click] --> R[Redirect service]
R --> L[Durable event log]
R --> U2[Landing page 302]
L --> D[Dedup processor]
D --> A[Attribution processor]
A --> W[OLAP warehouse]
W --> B[Billing export]
W --> P[Dashboard rollups]Deep dives
Fast click redirect
Beginner: Record the click quickly and redirect the user.
Advanced: Do not block on the full analytics pipeline. Validate token, emit event to a durable local/remote log, then return 302. Use fallback buffering during broker issues.
Deduplication
Beginner: A client retry or broker redelivery can carry the same logical click more than once. Prefer a stable event ID and atomic deduplication at the counting or billing boundary. A heuristic fingerprint can accidentally merge distinct clicks, so document its error behavior. Probabilistic membership alone is unsuitable when a false positive would silently discard a billable event.
Advanced: Dedup with event_id when provided, else fingerprint by token, user_key, campaign_id, and time bucket. TTL should cover attribution and billing windows.
Late conversions
Beginner: Attribution joins a conversion to eligible earlier interactions under a defined lookback and credit rule. Separate when a conversion happened from when it arrived. Late conversions may revise an earlier dashboard or create a correction entry, while closed billing periods need an explicit adjustment policy. Watermarks control processing progress; they cannot determine the business attribution rule.
Advanced: Use event-time processing, watermarks, correction events, and backfill jobs. Dashboards can update retroactively while billing closes on a defined schedule.
Privacy constraints
Beginner: Some identifiers cannot be retained forever.
Advanced: Separate raw PII from pseudonymous event ids, enforce consent at ingest, and support deletion/anonymization jobs that preserve aggregate billing integrity.
Failure modes
Broker unavailable on click path
Use local durable buffer or regional fallback queue; protect redirect latency.
Duplicate conversions overcount revenue
Dedup by conversion_id and advertiser transaction id.
Campaign report differs from billing export
Both derive from immutable raw events plus auditable correction records.
Hot campaign creates partition skew
Partition by campaign_id plus salt, then aggregate in a second stage.
Trade-offs
| Choice | Pick when | Watch out |
|---|---|---|
| Near-real-time reporting | Advertisers need live campaign tuning. | Numbers may shift as late events arrive. |
| Batch billing close | Accuracy and auditability matter more than instant finality. | Less immediate feedback for finance workflows. |
| Exactly-once stream processing | The platform supports it well end to end. | Often simpler to use at-least-once plus idempotent sinks. |
Beginner to advanced summary
Beginner answer
Separate the fast click redirect path from the heavier analytics and billing pipeline.
Advanced answer
The hard parts are deduplication, late events, attribution windows, privacy deletion, and reconciling reports with billing.
Stock Trading Platform / Matching Exchange
Order entry, risk checks, matching engine, and market data
Why this drill belongs here: Practice deterministic sequencing and durability, not microservice decomposition. One symbol = one writer.
Design an educational trading exchange where clients submit buy/sell orders, cancel open orders, receive executions, and subscribe to market data. This is a correctness-heavy system: ordering, durability, and auditability are more important than adding generic microservices.
Invariant: Orders for a symbol are matched in deterministic price-time priority, every execution is durable, and client retries cannot create duplicate live orders.
Scale assumptions
Worked capacity math
Formulas tied to design decisions
- 100K orders/sec hot symbol × 200 bytes/journal record ≈ 20 MB/sec durable write bandwidth per symbol.
- Market-data fan-out: 100K executions/sec × 500 subscribers ≈ 50M messages/sec before coalesced deltas.
- Replay lower bound: 1B journal events × 200 bytes ≈ 200 GB. Reading at 500 MB/sec takes 400 seconds, but actual rebuild also requires decoding and applying every event, restoring indexes, and validation. Measure replay throughput before assigning an RTO.
- Risk pre-check: 100K orders/sec × 1 ms CPU ≈ 100 cores for risk alone on one hot symbol path.
- Idempotency index: active day × 50M client_order_ids × 64 bytes ≈ 3.2 GB per trading session shard.
Requirements
Functional
- Submit limit and market orders
- Cancel or replace open orders
- Maintain order book per symbol
- Publish execution reports to clients
- Publish market data snapshots and deltas
- Run pre-trade risk checks
Non-functional
- Deterministic ordering for each symbol
- Durable journal before externally visible execution
- Strict idempotency for order entry retries
- Low latency in the matching path
- Replayable audit log for recovery and compliance
API and data model
Core APIs
- POST /orders with client_order_id -> accepted/rejected
- DELETE /orders/{order_id} with client_order_id -> cancel request
- GET /orders/{order_id} -> current order state
- WS /executions -> private execution reports
- WS /market-data/{symbol} -> public book deltas and trades
Important entities
High-level architecture
Order and execution state machine
Read the diagram definition
stateDiagram-v2
[*] --> PendingRisk: client submit
PendingRisk --> Rejected: risk fail
PendingRisk --> Open: accepted to book
Open --> PartialFill: match partial qty
PartialFill --> Open: remaining qty
PartialFill --> Filled: qty zero
Open --> Cancelled: cancel ack
Open --> Filled: full match
Filled --> [*]
Cancelled --> [*]
Rejected --> [*]Deep dives
Single writer per symbol
Beginner: For this simplified exchange model, one active sequencer orders accepted commands for a symbol and applies them deterministically to its order book. This removes concurrent mutation races inside the book, while other symbols can run on other owners. A failover must fence the old owner and recover the committed sequence before new matching begins; two live writers cannot safely choose independent fills.
Advanced: Partition symbols across matching workers. Inside one symbol, process a totally ordered input stream with no locks and emit deterministic journaled events.
Price-time priority
Beginner: Best price wins; same price uses oldest order first.
Advanced: Use buy and sell price levels with FIFO queues per price. Market orders consume opposite book levels until filled or rejected by protection limits.
Durability and replay
Beginner: Record enough ordered input and decisions to reconstruct the book deterministically after a crash. The acknowledgement contract states which journal boundary is durable before a client is told an order was accepted. Restore a validated snapshot and replay the later sequence, then compare invariants before reopening. Disk read speed alone does not determine recovery speed because every replayed command must also be applied.
Advanced: Persist journal records before publishing executions. On restart, replay journal to rebuild in-memory books and compare checksums at sequence boundaries.
Idempotent order entry
Beginner: A retry should not create two orders.
Advanced: Use (account_id, client_order_id) uniqueness before sequencing. Return the original accepted/rejected response for duplicate client submissions.
Failure modes
Client retries after timeout
Idempotency by account_id + client_order_id.
Matching engine crashes after match
Journal before publish; replay on recovery and redeliver execution reports idempotently.
Market data lags private executions
Use sequence numbers so subscribers detect gaps and request snapshot.
Clock skew changes fairness
Do not order by wall clock; order by sequencer-assigned sequence number.
Trade-offs
| Choice | Pick when | Watch out |
|---|---|---|
| In-memory book + durable journal | Low latency and deterministic replay are required. | Requires careful recovery tooling. |
| Generic database transactions | Volume is low and simplicity matters. | Usually too slow for hot matching paths. |
| Active-passive failover | Determinism matters more than multi-writer availability. | Failover time must be tightly controlled. |
Beginner to advanced summary
Beginner answer
Do not design this like a normal CRUD app. The order book and journal are the center.
Advanced answer
The discussion should focus on deterministic sequencing, single-writer matching, idempotent entry, replay, and market data gap recovery.
IoT Telemetry and Device Command Platform
Millions of devices sending telemetry and receiving commands
Why this drill belongs here: Practice bursty ingestion, tenant isolation, and offline command delivery, not a simple REST CRUD API.
Design a platform for connected devices such as sensors, vehicles, or industrial machines. Devices authenticate, send telemetry, report health, receive remote commands, and may disconnect for long periods. The hard part is absorbing bursty writes from unreliable networks while keeping per-tenant isolation.
Invariant: Device data is authenticated, tenant-isolated, durable after acceptance, and commands are delivered at most once or safely retried according to command semantics.
Scale assumptions
Worked capacity math
Formulas tied to design decisions
- 10M devices × 1 msg/min average = 167K msg/sec; firmware bug burst 6× → 1M msg/sec requires broker quotas.
- Payload 200 bytes × 1M msg/sec ≈ 200 MB/sec ingress before protocol overhead.
- Hot TSDB: 167K samples/sec × 24 bytes compressed ≈ 4 MB/sec hot tier; 7-year cold in object storage dominates cost.
- Command queue: 10M devices × 0.1 commands/day ≈ 12 commands/sec average; reconnect storms can spike 100× briefly.
- Offline backlog: 1M devices offline 1 hr × 60 msg/hr = 60M queued messages ≈ 12 GB raw if not downsampled on replay.
Requirements
Functional
- Device registration and certificate rotation
- Telemetry ingest over MQTT or HTTP
- Latest device shadow state
- Dashboards, historical queries, and anomaly alerts
- Remote command delivery and acknowledgement
- Firmware rollout orchestration
Non-functional
- Backpressure during firmware bugs or network storms
- Tenant isolation and per-device authentication
- Durable hot and cold telemetry storage
- Support offline devices and delayed delivery
- Schema validation without blocking all ingestion
API and data model
Core APIs
- MQTT topic tenants/{tenant_id}/devices/{device_id}/telemetry
- MQTT topic tenants/{tenant_id}/devices/{device_id}/command
- POST /devices -> register device and issue credentials
- GET /devices/{id}/shadow -> latest reported and desired state
- POST /devices/{id}/commands with command_id -> enqueue command
Important entities
High-level architecture
Telemetry ingestion and command pipeline
Read the diagram definition
flowchart LR
D[Device MQTT] --> B[Broker cluster]
B --> Q[Auth policy]
Q --> S[Telemetry stream]
S --> V[Validate + enrich]
V --> H[Hot TSDB]
V --> O[Object storage raw]
C[Command API] --> G[Command gateway]
G --> SH[Device shadow]
SH --> B
B --> DDeep dives
MQTT QoS
Beginner: MQTT QoS describes a protocol delivery agreement between communicating peers. A QoS 1 message can be delivered again, so a database counter still needs an application event ID. QoS 2 adds protocol exchanges to avoid duplicate delivery at that boundary, but does not make a downstream database write and an external device action one transaction. Select the level from the cost of loss and repeat handling.
Advanced: QoS 0 is fastest but lossy. QoS 1 can duplicate messages, so consumers need message ids and idempotency. QoS 2 is rarely worth the overhead at huge scale.
Device shadow
Beginner: A device shadow separates the state the device last reported from the state the control service wants. An offline thermostat can reconnect, read a newer desired temperature version, apply it, and report the result. Use versions to reject stale updates and keep commands distinct from observations. A desired-state document works well for convergent settings, while a one-time action needs an explicit command ID and outcome.
Advanced: Shadow state lets commands survive disconnects. Use version numbers so stale devices do not overwrite newer desired state after reconnect.
Partitioning
Beginner: Partition data by tenant, device, and time.
Advanced: Partition by tenant and device when per-device order matters, and isolate hot fleets with explicit quotas and capacity. Salting one device across partitions spreads load but loses a single ordered stream, so introduce it only with a sequence-and-merge design or independent substreams. Use event time and bounded retention when older telemetry moves into time-partitioned storage.
Firmware rollout
Beginner: Roll out updates gradually.
Advanced: Use staged cohorts, health checks, pause criteria, command idempotency, and rollback channels. Firmware bugs can multiply telemetry volume instantly.
Failure modes
Device firmware bug floods telemetry
Per-tenant and per-device quotas, broker throttling, and dead-letter topics.
Offline device misses command
Persist desired shadow state and command TTL; deliver on reconnect if still valid.
Duplicate telemetry from QoS 1
Dedup by device_id + message_id + event_time window.
Regional broker outage
Device bootstrap list with multiple broker endpoints and regional failover policy.
Trade-offs
| Choice | Pick when | Watch out |
|---|---|---|
| MQTT | Devices need persistent sessions, topics, and low overhead. | Broker operations and auth policy are more specialized. |
| HTTP ingest | Devices are simple or send infrequent batches. | Less natural for commands and persistent sessions. |
| Raw forever plus rollups | Compliance or ML training needs raw history. | Storage cost and privacy retention must be managed. |
Beginner to advanced summary
Beginner answer
Think of IoT as authenticated high-volume ingestion plus latest device state plus command delivery.
Advanced answer
The hard parts are QoS duplicates, offline devices, tenant quotas, schema evolution, firmware storm protection, and hot/cold storage split.
Reusable event envelope
A shared event envelope gives every consumer the basic information needed to interpret and operate a message: a stable event ID, event type and schema version, occurrence time, producer, and correlation context. Keep the business payload specific to the domain. An envelope does not guarantee ordering or deduplication by itself; the partition key, consumer transaction, retention policy, and authorization rules supply those behaviors.
{
"event_id": "uuid",
"event_type": "order.accepted",
"tenant_id": "tenant_123",
"entity_id": "order_456",
"idempotency_key": "client_or_server_key",
"occurred_at": "2026-05-04T12:00:00Z",
"schema_version": 3,
"trace_id": "trace_abc",
"payload": {}
}Primary sources
- AWS Architecture Center: production reference architectures.
- Google SRE Workbook: Evolving SRE Engagement: what realistic ops looks like.
- High Scalability archive: public postmortems and scale write-ups.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Practice contrasting workloads instead of copying one architecture template across feeds, reservations, messaging, storage, search, and payments.
Explain the decision
Why should a reservation system and a news feed not share the same consistency answer?
Read the reasoning
Reservations protect scarce exclusive inventory; feeds can often trade freshness for availability. Their correctness invariants drive different ownership and storage.
Apply it to a production scenario
Choose one design for a 20-minute drill. What artifact should you produce?
Read one approach
A one-page requirements/estimate sheet, architecture, one write or read sequence, a failure table, and a trade-off/next-bottleneck summary.
Work through the numbers
Six mini-cases receive a 90-minute practice block with 5-minute review after each. How much design time per case?
Read the worked calculation
60 minutes remain for design, or 10 minutes each; use the constraint to practice prioritization.
Reason through a failure
What practice invariant avoids memorized answers?
Read the recovery reasoning
Every component must be justified by this case’s quantified requirement or invariant, and removed when that justification disappears.
Sources and further study
Continue with Design URL Shortener.