System Design Scalability
Scalability Principles
Black Friday morning: checkout QPS jumps from 2k to 20k. If every request still needs a single database primary, a sticky in-memory session, and a synchronous email send, the product does not “need more servers”; it needs a design that can absorb load by adding capacity where it actually helps.
is the discipline of finding the next bottleneck, choosing vs , and applying a short list of levers: stateless tiers, caching, sharding, and async offload, without pretending any one lever solves every layer.
Scale journey
Measure the bottleneck
CPU, DB connections, lock wait, queue lag, or network.
Any instance can serve; sessions live in shared storage.
Shard writes, cache hot reads, offload slow side effects.
Cap admission so overload becomes controlled shed, not cascade.
What actually happens
Growth shows up as more concurrent requests, more stored bytes, or both. The first instinct is a bigger box: . That works until you hit a machine size limit, a single-disk IOPS ceiling, or a blast radius where one failure takes the whole product offline. After that, is how most internet systems keep growing: add app instances, cache nodes, shards, and consumers.
Horizontal growth only works if work can be split. A app tier behind a load balancer is the easy win. Reads often move to and replicas first. Writes eventually need because one primary’s commit log and lock manager cannot ingest infinite QPS. Slow or optional work should leave the user path via so spikes become queue depth instead of thread exhaustion.
Two failure modes dominate interviews and outages. A concentrates traffic on one key or shard no matter how many machines you bought. Unbounded turns one user action into hundreds of downstream calls whose p99 multiplies. is the reminder: shared serial pieces (global locks, single sequences, one primary) cap how much parallel capacity helps.
How to choose a scale lever
1. Is the app tier stateful? Sticky sessions and in-process caches prevent . Externalize session and durable state first.
2. Are reads or writes the fire? Read storms → and replicas. Write storms → batching, queues, then .
3. Must the user wait? If not, . If yes, keep the critical path short and put at every dependency.
Simple rule of thumb
Scale the edge horizontally, cache before you shard, shard before you invent a custom database, and never let a single hot key or unbounded fan-out pretend the cluster is healthy. should name peak QPS, storage growth, and the first serial bottleneck you expect at 10×.
Scale-out decision picture
Traffic does not scale “the system”; it scales one bottleneck at a time. Follow the growth signal to the lever that actually moves it.
Read the diagram definition
flowchart TD
A[Traffic grows 10x] --> B{Where is the bottleneck?}
B -->|CPU / RAM on one box| C[Vertical scale once]
B -->|Request concurrency| D[Stateless app tier + LB]
B -->|Read QPS| E[Cache + read replicas]
B -->|Write / storage| F[Shard by key]
B -->|Slow side effects| G[Async offload to queue]
D --> H[Measure p99 and error budget]
E --> H
F --> H
G --> H
C --> H
H --> I{Still overloaded?}
I -->|Yes| B
I -->|No| J[Keep headroom + alerts]
Hot path after the levers
A checkout that survives 10× keeps inventory reservation synchronous (correctness) while receipt email and analytics leave through a queue (elasticity).
Read the diagram definition
sequenceDiagram
participant U as User
participant LB as Load balancer
participant App as App instances
participant Cache as Cache
participant DB as Primary DB
participant Q as Queue
U->>LB: Checkout request
LB->>App: Route to any instance
App->>Cache: GET session / cart
alt cache miss
App->>DB: Read cart + inventory
DB-->>App: Rows
App->>Cache: SET with TTL
end
App->>DB: Reserve inventory (sync)
App->>Q: Enqueue receipt email (async)
App-->>U: 200 + order id
Note over Q: Side effects absorb spike without holding the user thread
Vertical vs horizontal in practice
| Dimension | Vertical | Horizontal |
|---|---|---|
| Speed to try | Minutes: resize instance | Needs LB, shared state, deploy plumbing |
| Ceiling | Largest SKU / single-disk IOPS | Coordination and shard ops become the limit |
| Failure blast radius | One big box = large outage | Smaller units; still need redundancy |
| Best first use | Buy time while you fix architecture | Steady growth and multi-AZ capacity |
Interview framing: vertical is a bridge; horizontal is the destination for request-serving tiers. Databases often use both: bigger primaries early, then read replicas, then shards.
The five levers (and what each buys)
1. Stateless compute
Put cookies, JWTs, or a shared session store in front of identical app pods. Autoscaling then adds capacity without sticky affinity. If you must keep WebSocket state local, shard connections by user id and plan reconnect storms.
2. Caching
Cache keys with high reuse and acceptable staleness. Size for hit rate, not vanity RAM. Guard miss storms with single-flight / request coalescing, and define TTL vs explicit invalidation per data class.
3. Sharding
Pick a shard key that matches access patterns (user_id for profiles, order_id for orders). Avoid keys that create one hot celebrity shard. Plan resharding: consistent hashing, fixed slots, or dual-write migrations.
4. Async offload
Anything the user does not need before the next screen can leave through a durable queue. Cap consumer lag and DLQ age as first- class SLOs; otherwise you traded request latency for silent backlog debt.
5. Backpressure and shed
Load sheds optional work (recommendations, previews) before it kills checkout. Rate limits, queue max depth, and dependency timeouts are scalability tools, not only security tools.
Numbers to know: capacity sketch
starts with rough arithmetic so you know which lever to pull first. Precision comes later from load tests; interviews reward honest envelopes.
| Assumption | Math | Implication |
|---|---|---|
| 2k → 20k checkout QPS | 10× peak; if each holds a DB conn 50 ms → 1,000 concurrent DB ops | Pool size and primary CPU matter before “more app pods” |
| 90% cache hit | 20k reads → 2k origin hits | Caching buys 10× origin relief if keys repeat |
| Celebrity key 5% of traffic | 1k QPS on one shard of 20 | dominates; more shards alone fail |
| Feed write fan-out 1→500 | 100 posts/s → 50k timeline writes/s | Push async or hybrid; never sync on request path |
Failure timeline: scale without the principles
T0 Sale opens; QPS climbs; sticky sessions pin users to few pods T1 Those pods CPU-saturate while idle pods sit empty T2 Synced email SMTP holds request threads; pools exhaust T3 Cache stampedes on cold start after deploy → primary melts T4 Ops “adds 50 pods” but primary and hot keys still choke T5 Fix path: shared sessions, cache single-flight, async email, shard inventory Outcome Machines only help when work is split and serial choke points are named
The timeline is why interview answers that only say “add load balancers and Redis” are incomplete. Name the serial piece and the hot-key plan.
Break the improved design
Stateless apps + cache + shards still fail: a on a sale SKU, cache TTL stacked on replica lag that surprises product, or a queue without that grows until retention drops work.
Cross-shard joins reintroduce a hidden primary. Global autoincrement sequences serialize writers. Multi-layer retries without budgets amplify a recovering dependency into a second outage. Scalability without admission control is just a larger failure surface.
Trade-offs
Vertical simplicity vs horizontal headroom
Vertical scaling keeps one machine and one mental model; it buys time. Horizontal scaling survives growth and reduces blast radius but demands shared state, rebalancing, and operational maturity. Prefer vertical as a bridge, horizontal as the steady plan for request tiers.
Cache hit rate vs freshness and stampede risk
Longer TTLs raise hit rate and lower origin load; they also widen staleness and make invalidation bugs more painful. Short TTLs or write-through keep data fresher but invite miss storms unless you coalesce. Pick per data class, not one global TTL.
More shards vs operational and query complexity
More shards raise write throughput until hot keys and cross-shard queries dominate. Resharding, dual writes, and fan-in analytics become permanent tax. Shard when a single primary’s metrics force it, not because “distributed” sounds strong.
Sync finality vs async elasticity
Keeping work on the request path simplifies UX when it fits the latency budget. Async offload absorbs spikes and survives crashes, but needs job ids, idempotency, and lag SLOs. Money and inventory usually stay sync; notifications and indexes usually leave.
Write fan-out vs read fan-out
Write fan-out makes reads cheap and timelines fast for active users; it wastes work on inactive followers and amplifies write storms. Read fan-out saves write bandwidth and pays at query time. Hybrid (push for active, pull for cold) is the usual mature answer.
Choosing and explaining scalability principles
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
Scalability means adding capacity where the next bottleneck is, not sprinkling servers everywhere.
I make the edge stateless, cache repeated reads, shard when one primary cannot absorb writes, and offload non-critical work asynchronously.
I call out hot keys, fan-out amplification, and serial choke points explicitly—Amdahl still applies.
When it belongs
Traffic, data, or both are expected to grow 10× or more.
A single primary, sticky sessions, or sync side effects sit on the critical path.
The interviewer asks how the design evolves from MVP to millions of users.
How it fails
Hot partitions and celebrity keys concentrate load on one shard.
Cache stampedes and multi-layer retries amplify recovering backends.
Unbounded queues or fan-out turn latency into backlog debt.
Trade-offs to weigh
Vertical simplicity versus horizontal headroom and ops cost.
Cache freshness versus origin load.
Sync correctness on the hot path versus async elasticity for side effects.
Common misunderstandings
Saying “add a load balancer and Redis” without naming the bottleneck.
Assuming more shards fix a single hot key.
Ignoring backpressure and treating infinite queues as capacity.
Wrap-up
You have a lever list, a decision flow, and the failure modes that make “just scale out” false. Use the exercises to rehearse naming bottlenecks out loud.
Exercises
- Bottleneck hunt. Given p99 checkout 1.8s, DB CPU 95%, cache hit 40%, queue lag 2s: which lever do you pull first, and what metric proves it worked?
- Hot key plan. One product id is 8% of inventory checks during a flash sale. Compare caching, key salting, and a dedicated shard; pick one and name the new failure mode.
- 10× sketch. Estimate storage and QPS at 10× for a URL shortener. Where does vertical stop being enough, and what is the first shard key?
Primary sources
- Google SRE: Handling Overload : load shedding and capacity as product behavior.
- AWS Builders’ Library: Scaling : practical scale-out patterns.
- Dynamo framing (partitioning at scale) : why single boxes stop working.
Key takeaway
Scalability is bottleneck-driven engineering: make compute replaceable, cache what repeats, shard when one writer cannot keep up, offload what the user need not wait for, and apply backpressure before overload becomes cascade. Machines multiply good design; they do not invent it.