System Design Case Study
Ticket booking
A stadium drop opens at noon. You join a line that says "You are #48,291," wait for a pass, tap two seats on the map, and race the checkout clock. To you it is one purchase. Underneath, a million people are fighting for the same inventory.
Ticket booking is inventory under a stampede. Queue fairly with a , hold seats briefly, charge once, and never sell the same seat twice. That last clause is the , not a property of the lock store alone.
Sale → seat journey
Admit buyers in waves, not all at once.
Only admitted users may attempt holds.
Atomic claim with a short TTL before pay.
Capture once; convert hold to sold.
Try it: seat hold then purchase
Holds reserve seats with a TTL. Payment success converts the hold; expiry returns seats to inventory.
- Client
- Event service
- Seat map
- Hold store
- Payment
- Holds
- 0
- Sold
- 0
- Mode
- hold
- Path hop
- 0
Inventory
holds0 holds, 0 sold.
TTL
expiryExpired holds must free seats exactly once.
Consistency
overbookNever sell the same seat twice across concurrent checkouts.
Hold seats, pay to convert, or expire a hold before payment lands.
What actually happens
Browsing events and seat maps can be cached at the edge. The hard path starts when the sale opens: users enter a , receive an , then call reserve. The inventory service claims seats with a and records a versioned .
Checkout charges the processor with an . On success, the moves held → sold. On timeout or abandon, the hold expires and seats return. A sold-out show can still feed a when someone cancels.
Scale the boring reads everywhere, but keep write authority per . One viral concert must not melt every other venue in the database.
How to choose the design moves
1. Can inventory survive the stampede? If peak concurrent viewers dwarf safe hold throughput, put a in front and admit only as many users as inventory can hold per second.
2. Who owns seat truth? Cache maps may look green while inventory already held the seat. Enforce transitions in one authoritative service with versions, not only in the UI.
3. What if payment is slow? Extend the hold when payment starts. If money succeeds after expiry, reconcile: confirm if still free, otherwise refund. Never invent a second sold row.
Simple rule of thumb
Protect the hold API with admission, claim seats atomically with TTL, confirm with idempotent payment, shard by event. In simple terms: fairness at the door, authoritative seat state in inventory, locks as the fast gate, one charge, one owner.
Case-study upgrade
Ticket Booking Capacity Math
Ticket booking is a hot inventory problem. Browsing can be cached, but seat hold and sold state must be authoritative.
Assumptions and formulas
- Hot event: 1M concurrent users and 80K seats. Queue status polling every 5 seconds if not optimized.
- Seat holds last 10 minutes and each admitted user may hold up to 4 seats.
- The waiting room should admit only what the inventory service can safely process.
Concrete math
- Queue polling: 1M users / 5 seconds = 200K queue status QPS; use long polling or backoff.
- Admission budget: if inventory can handle 10K hold attempts/sec, waiting room should not send 1M/sec.
- 80K seats / 4 seats per user = 20K admitted users can temporarily hold the full venue inventory.
- Lock memory: 80K seat locks x 200 bytes = 16 MB raw per event.
- Holds: 10K successful holds/sec x 10 minutes = 6M live holds globally.
Design decisions forced by numbers
- Use admission tokens so queue traffic cannot overload inventory.
- Seat hold, payment, confirmation, and expiry must be one clear state machine.
- Seat maps may be stale, but hold attempts must check authoritative state.
Scaling to millions
- At 1M users: queue, seat inventory store, and hold expiry are enough.
- At 10M users: isolate hot events, add admission control, and reduce polling.
- At 100M+ users: add regional queues, bot controls, per-event capacity budgets, and reconciliation.
Readable flow
Interview follow-ups to expect
- How do we prevent double-booking?
- How many users should enter the seat-hold flow at once?
- What happens if payment times out after a seat is held?
Operational readiness checklist
After the math, explain how this capacity plan is measured, protected, launched, and evolved.
Observability
- Attach each capacity estimate to a dashboard metric: p99 latency, queue age, cache hit rate, shard skew, error rate, and freshness lag.
- Alert on rate-of-change and saturation, not only hard failures; backlogs and hot keys usually warn before an outage.
Security and abuse
- Budget for auth checks, rate limits, quota isolation, audit logging, privacy filters, and abuse-review side paths.
- Separate trusted internal events from user-controlled input so retries and backfills cannot bypass policy checks.
Degraded modes
- Define the fallback when a dependency is slow: stale cache, partial results, read-only mode, delayed async work, or admission control.
- Protect the invariant first; reduce freshness, personalization, or analytics quality before risking duplicate or corrupt state.
Testing and rollout
- Test average, peak, and hot-object traffic; include retry storms, cold caches, lagging workers, and regional dependency loss.
- Use canaries, shadow reads, dual-write validation, feature flags, and explicit rollback criteria for risky path changes.
Rollout, backfill, and runbooks
- Make migrations and backfills idempotent, resumable, rate-limited, and visible through progress and error metrics.
- Write the runbook before launch: owner, dashboards, paging thresholds, mitigation switches, replay steps, and repair commands.
10x evolution
- For ticket booking capacity math, state which number forces the next move: cache memory, shard count, write amplification, queue lag, egress, or failover reserve.
- Add the next layer only when the metric forces it: sharding, regional reads, async replay, hot-key isolation, or stricter backpressure.
Scope you should nail down
Product surface
- Browse events by category, date, location
- Interactive seat map with availability
- Temporary seat hold, then checkout and payment
- Confirmation email; cancel within policy
- Waitlist when sold out; notify on cancellation
Scale targets
- ~100M users; peak ~1M concurrent on popular drops
- Prevent double booking (critical invariant)
- Fair FIFO queue; bot defenses
- Hold TTL ~5–10 min; ~10M bookings/day; high availability
Capacity picture
Peak: 100K+ bookings in the first minute of a popular sale
Events storage: 500K × 5 × 2KB ≈ 5GB
Queue polling: 1M waiting / 5 sec poll ≈ 200K queue-status QPS; prefer long poll, backoff, or push
Admission budget: Convert safe inventory throughput into admitted sessions using their expected hold attempts. If the store safely handles 10,000 hold attempts/s and each admitted user generates four attempts, a starting admission ceiling is roughly 2,500 users/s before headroom. Retries and multi-seat operations also consume work, so adjust from measured saturation and successful booking rate.
Venue hold math: 80K seats / 4 seats per user = 20K users can temporarily hold a whole venue
Lock memory: 80K locks × 200 bytes ≈ 16MB raw per hot event
Live holds: 10K holds/sec × 10 min ≈ 6M live hold records in a global wave
Booking storage: 10M/day × 1KB = 10GB/day; ~18TB over 5 years before indexes
APIs worth sketching
searchEvents(query?, category?, date?, location?)Returns Event[]
getEventDetails(event_id)Event, venue, pricing tiers
getAvailableSeats(event_id, section_id?)Seat[] with available / reserved / sold
reserveSeat(event_id, seat_ids[], user_id)ReservationToken (TTL 5–10 min)
confirmBooking(reservation_token, payment_method, idempotency_key)Booking confirmation
cancelBooking(booking_id)Refund + seats released
addToWaitlist(event_id, user_id, seat_preferences?)Waitlist position
Core data model
events
venues
seats
reservations
users
payments
waitlist
High-level architecture
Seat reservation (critical path)
Optimistic locking
Optimistic concurrency reads a seat version and attempts a conditional transition only if the seat is still available and the version still matches. Two buyers can read the same version, but only one atomic update wins. This preserves the seat invariant under the store’s transaction guarantees. Under extreme contention, bound retries and tell losers the seat changed rather than repeatedly hammering the same row.
Pessimistic locking
A database row lock serializes competing transactions while one inspects and changes the seat. Keep that transaction short: create the durable hold and commit, then perform payment outside the database lock. Holding a transaction open while a customer enters card details wastes connections and blocks other work. For several seats, lock in a stable order and commit the hold as one unit where the transaction boundary permits it.
A cache SETNX with expiry can cheaply reduce competing hold attempts, but it is not the authoritative sale record. A process can pause past expiry or a cache failover can lose a lock. The inventory store must still conditionally create and confirm the seat hold using an owner and version. Release a cache lock only when its stored owner token matches, so an old request cannot delete a newer buyer’s lock.
Flow: Admission permits an attempt; an optional cache lock reduces contention; the inventory transaction creates a versioned hold; payment runs against that reservation; and confirmation conditionally converts the still-valid hold to sold. Expiry or cancellation releases only the matching active hold. A timeout does not prove payment failed, so late outcomes enter reconciliation instead of bypassing seat ownership.
Distributed lock sketch
// in-memory cache store distributed lock for seat
SET seat:12345:456 "user_789" NX EX 600
→ "OK" if seat was free, else nil
// Reserve multiple seats atomically (pseudo)
for seat in seats:
if cacheStore.set(seat_key, user_id, "NX", "EX", 600) == nil:
return "seat_taken"
else:
add to reservation_list
return reservation_tokenHandling the thundering herd
Admit users in batches (for example 10K at a time) so 1M concurrent viewers never all hit seat service together.
FIFO position token while waiting. When admitted, a session token unlocks reserveSeat; others get 403.
Admission flow
- User joins queue (list or stream)
- Return position + estimated wait
- Worker admits N users/sec and issues session tokens
- Token holders may reserve; everyone else is blocked
Seat booking state machine
Read the diagram definition
stateDiagram-v2
[*] --> Queued: join waiting room
Queued --> Admitted: token issued
Admitted --> Held: reserveSeat atomic ok
Admitted --> Queued: hold failed / retry
Held --> Paid: confirmBooking success
Held --> Released: hold TTL expired
Paid --> Confirmed: payment captured
Confirmed --> [*]
Released --> Available: seat returned
Available --> [*]Distributed locking details
SETNX with TTL
SET key value NX EX ttl claims seat:{event_id}:{seat_id} for one reservation or user id.
Redlock (when needed)
A multi-node cache-lock algorithm introduces assumptions about timing, independence, and how long clients believe a lease is valid. Obtaining a majority of cache locks is not equivalent to a consensus-backed seat transaction and cannot replace the authoritative inventory check. Keep the no-double-sale guarantee in a store that atomically enforces state and version; use the distributed lock only as a carefully bounded coordination aid.
Lua script for atomic multi-seat reserve
-- Lua script for atomic multi-seat reserve
local keys = KEYS -- seat keys
local user_id = ARGV[1]
local ttl = ARGV[2]
for i, key in ipairs(keys) do
if cacheStore.call("EXISTS", key) == 1 then
return false -- seat taken
end
end
for i, key in ipairs(keys) do
cacheStore.call("SET", key, user_id, "EX", ttl)
end
return trueHard part: no seat sells twice
Correctness invariant
Authoritative
The seat-state machine is authoritative. Cached maps and distributed locks are mechanisms: they speed claims and reduce races, but inventory must still enforce one legal transition per seat with a version check. A lock without a successful state transition does not sell a seat; a successful sold transition without a lock store is still a sold seat. COUNT(sold bookings for event_id, seat_id) must never exceed 1.
available --hold(reservation_id, ttl, version+1)--> held
held --confirm(payment_id, version+1)-----------------> sold
held --expire/release(version+1)----------------------> available
sold --refund_policy_allows(version+1)----------------> released_or_blocked
Invariant:
COUNT(sold bookings for event_id, seat_id) <= 1Multi-seat atomicity
For seats that share a database transaction boundary, acquire them in a stable order and either commit the complete requested group or release the transaction. If holds span independent owners, coordinate a reservation workflow with explicit partial-hold cleanup and expiry. Do not show the whole group as secured while only some seats are held. Every release must identify the original reservation so duplicate cleanup cannot free another buyer’s seat.
Payment timeout
When payment begins, conditionally extend only the matching active reservation under a bounded policy. A delayed payment success must check the latest hold and sold state before confirmation. If the original hold expired and another buyer now owns the seat, refund or escalate the payment; never create a second sold owner. If reacquisition is allowed, it is a fresh atomic inventory decision with a recorded reconciliation link.
Audit query
Continuously check duplicate sold rows by event and seat. One duplicate is a page, refund, and incident.
Payment integration
Two-phase flow
- Reserve: transition available → held in inventory (cache lock is the fast gate; TTL 5–10 min)
- Charge: call processor with
- Confirm: success → held → sold in inventory; failure → release hold (and drop any lock)
Same idempotency_key + reservation_token returns the cached result so retries do not double charge.
Waitlist
- Sold out → add user to a priority queue (sorted set or DB)
- On cancel → release seats, notify first N waitlist users
- Time-limited offer (for example 24h); no response → next person
- Store section/price preferences for better matching
Scalability moves
Seats, reservations, and bookings live per event. Isolate hot drops without slowing quiet shows.
Read replicas
Event details and seat maps read from replicas. Holds and confirms write to the primary.
CDN
Cache event pages, static assets, and seat-map images at the edge. Revalidate availability on hold.
Bottlenecks to call out
Race conditions
Two users claim one seat → SETNX / Lua atomic scripts.
Hot events
1M users on one event → waiting room, rate limits, shards.
Payment timeouts
Pay succeeds after hold expiry → extend lease + reconcile.
Bot / scalper abuse
CAPTCHA, per-IP and account limits, verification, signed tokens.
Production readiness
SLOs and metrics
- Hold API p99 <250ms for admitted users; queue-status p99 <1s
- Zero double-sold seats; hold expiry p95 lag <30s
- Track admission rate, hold success, stale map age, payment timeouts, bot rejects
Alerting
- Page on duplicate-sold invariant, inventory primary errors, confirm backlog
- Warn on wait estimate drift, lock latency, 409 spikes, cleanup lag
- Alert by event_id so one hot show does not hide others
Degraded mode
- Inventory slow → shrink admission waves; keep browsing cached
- Payment degraded → pause confirms; keep holds briefly; reconcile
- Stale maps → mark approximate; revalidate only on hold
Threat model
- Assets: tokens, holds, payments, identity, inventory
- Actors: bots, scalpers, token replay, compromised accounts
- Controls: signed tokens, rate limits, bot scoring, idempotency, audit logs
Test plan
- Thousands of contenders for the same seat and multi-seat groups
- Lock-store failover, payment timeout after hold extension, token replay
- 1M waiting users with admission at 5K–20K hold attempts/sec
Rollout and 10x
- Canary reservation code on low-demand events; shadow duplicate-sold checks
- DR: single-writer per event; fence old writers; reconcile holds
- 10x: dedicated hot shards, pre-created seat state, regional waiting rooms, seat-map deltas
Failure timeline: hold expiry vs late payment
T0 User A holds seats S1,S2 (TTL 10 min); inventory: available→held T1 Payment starts at T+9m; network slow; hold not extended / fence missing T2 TTL fires; cleanup returns S1,S2 → available; User B holds S1 T3 User A payment succeeds late; naive confirm writes sold without version check T4 Two sold owners for S1: invariant broken T5 Correct path: confirm is conditional on hold_owner+version; late pay → refund/reconcile
Counterexample to "SETNX is enough": the cache lock is a mechanism; sold truth lives in the seat-state machine.
Takeaway
- Invariant: seat state machine is authoritative, never two sold rows for one seat
- Holds: atomic cache locks with TTL are the mechanism; inventory owns sold truth
- Stampede: waiting room + signed admission tokens
- Payment: reserve → charge → confirm with idempotency
- Scale: shard by event; CDN for browse; waitlist on cancel
Exercises
1. Late payment after hold expiry
Payment succeeds 30s after hold TTL. Write the confirm decision table (version match vs conflict) and whether you refund, re-hold, or page. Keep the seat-state invariant as the source of truth.
2. Admission vs inventory budget
Inventory safely handles 10K holds/sec; 1M users wait. Set admission rate and queue-status QPS strategy (long poll vs push) so the waiting room does not DDoS itself.
3. Lock vs state machine
A design doc says "Redis SETNX guarantees no double sell." Write the review comment that separates mechanism from the authoritative seat-state invariant, and name the audit query you page on.
Topic-specific references
- Stripe: Idempotent requests : confirm/charge retries without double-selling or double-charging.
- Kleppmann on distributed locking . That is why a cache lock alone is not the correctness story (inventory must still fence/check).
- AWS Builders' Library: Challenges with distributed systems : retries, timeouts, and why invariants beat hope.
Beginner to advanced path(open guidance)
Beginner answer
Model events, venues, seats, reservations, payments, and bookings. Use a temporary hold before payment.
Advanced answer
Waiting rooms, signed tokens, atomic holds, bot defense, stale map caches, payment timeout reconciliation, hot-event isolation.
Deeper walkthrough
Ticket booking deeper walkthrough
Ticket booking is an inventory problem under extreme burst traffic. Browsing can be cached, but the seat-state invariant (never two sold owners) is authoritative; locks are a mechanism, not the truth.
Requirement refinements
Clarify assigned seats versus general admission, hold duration, cancellation rules, queue fairness, bot defense, and payment timeout behavior.
State the invariant first: a seat can be available, held, sold, or blocked, but never sold to two users. Then name locks/TTLs as enforcement mechanisms.
API design details
Use signed queue admission tokens so only admitted users can attempt seat holds.
Use reservation tokens with expiry and idempotent confirmBooking calls tied to payment attempts.
Data model and storage
Store seat inventory by event_id and seat_id with status, version, hold_owner, hold_expires_at, and booking_id.
Keep seat map layout cacheable, but treat hold and sold status as authoritative from the inventory service.
Scaling strategy
Shard by event_id because hot events should not affect unrelated events.
Admit users in batches from a waiting room and rate limit hold attempts per token, account, and device.
Reliability, recovery, and operations
If payment completes after hold expiry, reconcile with booking state and refund or manually resolve.
Monitor hold success rate, queue length, payment timeout rate, double-book prevention errors, and bot rejection rate.
Deepening reference
Ticket Booking: realistic production notes
Beginner mental model: Ticketing is an inventory locking problem under extreme burst traffic.
Production scope: A realistic design needs a waiting room, seat map caching, atomic holds, payment timeout handling, bot defense, and fairness guarantees.
Choose what the first version must guarantee
Start by deciding which ticket booking capabilities are essential. Describe the main read and write operations before choosing infrastructure. For each operation, identify the stored records it uses and the condition that makes its result correct. A fast response is useful only if it satisfies that condition.
Next, separate work the user must wait for from work that can finish later. Estimate requests per second, storage growth, and the busiest object or customer. Use those estimates to choose one bottleneck to investigate. The calculations below are example workloads with stated assumptions; they are not measured limits of a particular product.
Core flow to walk through
Admission: waiting room admits users in controlled batches with signed session tokens.
Hold: user selects seats, system atomically creates short-lived hold.
Checkout: payment succeeds before hold expires, booking is confirmed, seats become sold.
Production readiness
Ticket Booking: ship, operate, and evolve
Now examine the same request flow during failures and deployments. Tie each operational control to the user behavior it protects.
Observability
Track ticket booking p95/p99 latency, error rate, saturation, and user-visible freshness separately for read, write, and async paths.
Alert on leading indicators such as queue age, hot-key skew, cache miss storms, reconciliation mismatches, and error-budget burn.
Security and abuse
Enforce authentication, authorization, rate limits, input validation, audit trails, and privacy checks at the service boundary.
Treat abuse controls as a first-class path: throttling, quota isolation, moderation hooks, revocation, and safe handling of sensitive data.
Degraded modes
Name what still works if cache, search, ranking, stream processing, object storage, or a regional dependency is slow.
Prefer stale-but-safe reads, partial results, async retry, admission control, and feature disablement over corrupting core state.
Testing and rollout
Load test average, peak, and hot-object traffic; add failure-injection for retries, timeouts, backlog growth, and dependency outages.
Ship with canaries, shadow reads or dual writes when useful, rollback switches, schema compatibility checks, and replay validation.
Runbooks
Document dashboards, paging thresholds, owner actions, rollback steps, data repair commands, and customer-impact language.
Keep backfills and replays idempotent, resumable, rate-limited, and observable so repair work does not create a second incident.
10x evolution
Say which metric forces the next architecture change: shard count, cache footprint, write amplification, queue lag, p99 latency, or regional failover reserve.
Evolve one bottleneck at a time: partition by access pattern, isolate hot tenants, regionalize reads, add replayable async pipelines, then harden multi-region behavior.
Hard parts worth deepening
Prevent double booking with atomic seat state transitions.
Separate browsing seat maps from the write-heavy hold path.
Virtual waiting rooms protect the backend and improve fairness.
Payment timeouts must either extend holds safely or release them deterministically.
Failure modes and recovery
Payment succeeds after hold expires; refund or manual reconciliation using payment idempotency.
Seat map cache stale; final hold operation is authoritative.
Bot traffic consumes queue; use device fingerprint, CAPTCHA, velocity limits, and token binding.
Popular event overloads one shard; shard by event and isolate hot event infrastructure.
Signals to monitor
Hold success measures whether an eligible request reserves a seat. Distinguish expected contention rejections from actual double-booking invariant violations; these demand very different responses.
Queue admission rate should stay within the reservation path capacity. Compare payment timeouts with hold expiry to find ambiguous bookings, and review bot rejections so protection does not unnecessarily exclude legitimate buyers.
| Choice | Best when | Cost |
|---|---|---|
| In-memory seat locks | Low-latency temporary holds are needed. | Must reconcile with durable booking store. |
| Database row locks | Scale is moderate and correctness simplicity matters. | Poor under extreme contention. |
| Waiting room | Traffic spikes are predictable. | Adds user waiting and queue operations. |
Try this scenario
Walk through the exact race between two users clicking the same seat at the same millisecond.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Protect seat ownership with authoritative state transitions, expiring holds, virtual queues, payment ambiguity handling, audit, and reconciliation.
Explain the decision
What prevents two users from holding the same seat?
Read the reasoning
A conditional transition from AVAILABLE to HELD with hold ID/version and expiry in the authoritative database, not an advisory cache lock alone.
Apply it to a production scenario
Payment capture succeeds after the seat hold appears expired. What happens?
Read one approach
Quarantine the outcome, reconcile by hold/payment idempotency IDs, extend/confirm only through policy, or refund without double-selling.
Work through the numbers
A venue has 60,000 seats and 3 million arrivals over 10 minutes. What average arrival QPS?
Read the worked calculation
5,000 QPS, but synchronized opening creates a much higher peak; admission should match hold/database capacity.
Reason through a failure
What is the seat invariant?
Read the recovery reasoning
A seat has at most one valid hold or sale for an event/version, across expiration, retry, payment timeout, and recovery.
Sources and further study
Continue with Design Payment System.