System Design Foundations
Horizontal vs Vertical Scaling
Imagine an Instagram story goes viral during a sale evening. Uploads, likes, and feed refreshes all spike at once. The product should not freeze just because yesterday's quiet traffic became today's rush.
Scaling is how you keep the app usable when more people show up. You either make one machine stronger () or add more machines that share the work (), and most real systems mix both. The panel on the right is the order experienced teams follow when a spike arrives; the rest of this page explains why that order works.
When traffic spikes
Measure the bottleneck
Find which of CPU, RAM, disk, network, or locks is saturated before spending money.
Buy a bigger box while one stateful node (usually the database) is the limit.
Add interchangeable replicas behind a load balancer once the app tier is stateless.
Use a CDN for static bytes and regional copies for latency-sensitive dynamic work.
What actually happens
The sale-evening spike from the story is a compressed version of what every growing product lives through over months. Early products often run the app and database on one server. When a viral post or exam-results night arrives, that box runs out of CPU, memory, or disk I/O (how fast it can read and write storage). The first instinct is usually : rent a bigger machine and buy months of runway without redesigning the app.
When the web tier is busy but interchangeable, teams switch to . They make the app , put sessions and uploads in shared stores, and place a in front of many replicas (identical copies of the service). Reads often get help from a , an app cache, and before anyone splits the primary database.
Writes and storage eventually hit a wall that bigger hardware cannot cheaply fix. That is when , queues, and multi-region designs show up, not because they sound advanced, but because metrics prove one primary (the single database that accepts writes) cannot keep up. Everything below is a way of deciding which of these moves you need next, and of proving it with numbers rather than instinct.
Try it: compare scale-up cost with scale-out headroom
Move the demand slider, then grow capacity vertically or horizontally. The numbers are illustrative, but the shape matches real trade-offs: bigger boxes cost more per marginal QPS, while more small nodes spread risk and track load linearly when the app tier is stateless.
Single machine
Medium tier · about 900 QPS ceiling · one failure domain.
Overflow
healthyDemand fits; latency stays dominated by per-request work rather than backlog.
- Capacity
- 900 QPS
- Served
- 600 QPS
- Est. monthly cost
- $280
- Over capacity
- 0
600 of 600 QPS served · 0 over capacity · ~$280/mo · 67% of capacity used. You are serving 600 QPS on a medium box. Compare scaling up one machine versus adding stateless replicas with the same monthly budget in mind.
Match the bottleneck. Scale up while one database primary is the limit; scale out when web replicas are interchangeable behind a load balancer.
Compare at equal pain. At 600 QPS demand, a Medium vertical box (900 QPS, $280/mo) versus 2 horizontal nodes (1000 QPS, $190/mo) shows how linear spend can buy the same relief with different ops work.
How to choose a scaling move
The story above gives the order most teams stumble into; three questions let you make the same call deliberately when your own metrics arrive.
1. What is the measured bottleneck? Do not start with orchestration or . Name whether CPU, memory, disk I/O, network, or database locks are hurting first, because each points at a different fix.
2. Is the work parallel? A API tier usually scales out, because a can spread requests without breaking anyone's session. A single-writer database usually scales up first, because one stateful node is the limit, consistency is valuable, and bigger hardware still buys economical runway. The tier ladder further down shows what comes after that.
3. Is the pain distance, not capacity? When physics dominates, static bytes belong on a , and latency-sensitive dynamic flows need regional app and data copies, not only a bigger central box.
Simple rule of thumb
Design early, externalise sessions and uploads, and keep the database on one strong primary until metrics prove you need the next rung. Automate replica counts with sane cooldowns (waiting periods between scaling moves), and treat giant company anecdotes as intuition, not as day-one requirements for your product.
Scaling decision path
The three questions above collapse into this flow: start with traffic growth, measure the bottleneck, then pick the smallest move that buys headroom without inventing a distributed system you do not need yet.
Traffic grows
Measure the bottleneck
CPU or memory bound?
Yes, one server near limit
Scale up first
Stateless tier
Scale out replicas
Database write limit?
Reads dominate
Add cache and read replicas
Writes dominate
Shard or partition carefully
Users far away?
Static content
Use CDN
Dynamic content
Regional app and data
Mechanics: what scaling up and scaling out really change
First identify the constrained resource. High CPU may improve with more cores or application replicas; a full disk needs storage capacity; slow queries may need a different index; lock contention may need shorter transactions. These conditions can all appear as “the app is slow,” but adding the same resource does not fix all of them. Measure utilization, queueing, and per-request work at the limiting tier.
The decision path names the moves; this card explains what each one changes inside the system and what it costs you.
You buy a bigger VM or bare-metal server: more CPU, RAM, or faster disks. The codebase and deployment shape stay the same until physics or budget says stop. It is especially good for databases early on, because keeping one primary avoids distributed transactions (one write that must succeed on several machines at once). Many mature systems buy years of runway by upgrading the primary database before splitting it.
Why teams like it
- Operationally simple, because there is no cluster membership to manage on day one.
- A single-writer database stays strongly consistent without any shard routing logic.
- No extra network hops, since every call stays inside the same server.
Where it stops
- One giant machine is still one failure domain unless you also replicate it.
- Price often climbs faster than throughput, so each upgrade buys less (diminishing returns).
- Hardware migrations mean planned downtime or a carefully rehearsed failover to the new box.
- You cannot stretch one machine across continents, so distant users still feel latency.
To make the ceiling concrete, these are the largest machine shapes public clouds and bare-metal vendors offer; a single box can go a very long way before you truly run out of room.
| Example shape | vCPUs | RAM |
|---|---|---|
| memory-heavy bare-metal | 448 | 24 TB |
| large memory VM | 128 | 2 TB |
| enterprise memory VM | 416 | 12 TB |
| mega-memory VM | 128 | 1.9 TB |
Once one box is no longer the answer, you run many copies of the same service behind a . That works smoothly only if each copy is : it can answer any request without secret local memory, so sessions and uploads live in a shared cache, database, or object storage instead. Huge fleets are the extreme case, but the pattern is the same at a dozen instances: identical pods (container copies), centralised configuration, and external state stores.
Design habits that make it work
- Keep sessions in a shared cache or signed cookies, never RAM pinned to one host.
- Store user files in object storage (a shared blob service), not one server's disk.
- Ship logs and metrics to centralised telemetry, so a dead instance loses nothing.
- Deploy immutably: replace instances instead of SSH hotfixes, so every copy stays identical.
What gets harder
- CAP trade-offs: during a network partition you must deliberately choose consistency or availability.
- Chatty services: every remote call (RPC) adds latency, so fan-out needs budgets and timeouts.
- Distributed workflows: sagas (multi-step compensating workflows) and idempotent retries replace one big SQL transaction.
- Operations: automation and observability become mandatory, because nobody can babysit dozens of instances by hand.
Autoscaling: match capacity to demand
Scaling out by hand does not survive a viral evening, so turns the replica count into a dial driven by signals like CPU, queue backlog, or latency. Reactive scaling reacts after load rises; predictive scaling tries to warm capacity before a known spike such as a release or a sports event.
Common reactive triggers
- CPU high for several minutes: add replicas, if your bottleneck really is CPU-bound work.
- Memory pressure: scale or fix leaks, because scaling only hides a leak briefly.
- Queue depth growing: add consumers or workers, since the backlog means demand outran processing.
- Latency percentiles rising: capacity is short, or a downstream dependency is slowing everyone.
In Kubernetes the built-in autoscaler is the HorizontalPodAutoscaler (HPA); its stabilization windows are the cooldowns that stop the fleet flapping.
Example: horizontal autoscaler snippet
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
minReplicas: 10
maxReplicas: 1000
scaleUp stabilization: 60s
scaleDown stabilization: 300sPredictive & pre-warming
Scheduled traffic (evenings, launches, shopping holidays) benefits from scaling curves learned from history or calendars. Pre-warming also means filling caches and JIT warm-up (letting the just-in-time compiler optimise hot code) so new instances are not cold when users arrive.
- Managed platforms expose predictive scaling that learns from historical metrics and calendars.
- Cooldown windows reduce flapping (rapid add and remove) when metrics oscillate around a threshold.
- Event-driven autoscalers scale on stream lag, queue depth, or custom metrics instead of CPU.
Scaling each tier
Adding application servers increases the number of requests that can reach shared dependencies. If each of 20 instances opens a 50-connection database pool, the database may face 1,000 connections even though its capacity has not changed. Limit total downstream concurrency and consider connection pooling outside individual processes. A scaling plan should state which resource grows and which resource remains fixed.
Autoscaling is easy to picture for web servers, but the same request touches three tiers that scale very differently. HTTP servers are the easiest layer to duplicate. Caches sit in the middle: fast, but they need warming and an eviction strategy. Relational writes are the usual long pole (the slowest part that sets the pace).
Application tier (often easiest). Add identical instances behind a . Autoscalers can grow replica counts quickly when metrics justify it, and because each instance is interchangeable, deploys and failures stay boring.
Cache tier (medium). Distributed cache clusters add shards and failover. Watch for cold caches after deploys: every miss falls through at once (a thundering herd) and can flood the database. Shadow traffic or gradual rollouts warm the cache before real users depend on it.
Database tier (usually hardest). The database ladder below is the sequence most teams climb, one rung at a time, only when a metric proves the current rung is short.
- Buy a bigger machine while the economics still allow it (vertical scaling).
- Add so read-heavy workloads stop competing with writes on the primary.
- Put an application cache in front to shield hot reads from the database entirely.
- Use connection pooling so many app instances reuse a few database connections efficiently.
- or federate (split by key or by domain) when write throughput demands multiple primaries.
Large social products often combine relational shards with proxy layers so applications see one simpler API instead of many databases.
Numbers to know: the million-user scenario
Every section so far says "when metrics prove it"; this is what those metrics look like for one shared scenario that the rest of the page reuses. Convert users into rates before choosing infrastructure: one million daily users can be modest or huge depending on actions per user, payload size, , and cacheability. The figures are illustrative orders of magnitude, not benchmarks for your stack.
1M DAU
20 actions per user per day
20M requests per day
231 average QPS
5× peak = 1.2K peak QPS
960 read QPS and 240 write QPS
Edge offload
CDN byte offload
Cache relief
85% app cache hits
About 144 DB read QPS
Write growth
About 20.7 GB writes per day
Numbers to know
The flow above shows the shape of the maths; these four inputs are the assumptions behind it. Later sections refer back here instead of repeating them.
DAU means daily active users. Peak matters more than average, so the workbook below multiplies by a peak factor.
An SLO (service level objective) is the latency you promise. p95: 95% of requests finish faster; p99 protects the slowest 1%.
Of every 100 reads, 85 come from cache and never reach the database; the workbook shows what is left over.
The CDN serves most bytes at the edge; the origin carries only the remaining 8% plus miss bursts.
Scaling math workbook
Assume a load test finds that one instance safely handles 200 requests per second while meeting the latency target. A 2,000-request-per-second peak needs ten instances for nominal capacity. To survive one instance failure at that same peak, at least eleven are needed; to survive losing one of three equally sized zones, total capacity must be at least 3,000 requests per second. These are capacity bounds under the stated assumptions, before uneven traffic, warmup, and operational reserve.
With the inputs fixed, each row turns them into a request rate, a replica count, or a storage growth figure. Once you have these, choosing load balancers, cache nodes, databases, and targets becomes far clearer.
| Question | Calculation | Answer for shared scenario |
|---|---|---|
| Average QPS | 20M requests/day ÷ 86,400 | ~231 QPS |
| Peak QPS | 231 × 5 peak factor | ~1.2K QPS |
| Read/write split | 80% reads, 20% writes | ~960 read QPS, ~240 write QPS |
| DB read load | 960 × (1 - 0.85 cache hit) | ~144 DB read QPS |
| API replicas | 1,200 ÷ 120 safe QPS × 1.3 headroom | ~13 replicas |
| Write storage | 240 writes/s × 86,400 × 1KB | ~20.7GB/day before indexes |
| Provisioned storage | 20.7GB × 3 replication × 2 index/metadata | ~124GB/day |
Formula panel
average_qps = daily_requests / 86,400peak_qps = average_qps x peak_factorread_qps = peak_qps x read_ratiowrite_qps = peak_qps x write_ratiodb_read_qps = read_qps x (1 - cache_hit_ratio)origin_bandwidth = edge_bandwidth x (1 - cdn_byte_hit_ratio)replicas = ceil(peak_qps / safe_qps_per_replica x headroom_factor)cache_memory = objects x average_value_size x overhead x replication_factorstorage_growth = writes_per_day x record_size x retention x replication x index_factorconcurrency = qps x latency_seconds
How to say the math aloud
In a design discussion, walking the workbook in order (average , peak QPS, read/write split, , byte offload, then the bottleneck that remains) shows you are sizing the system instead of naming technologies at random.
Capacity estimate steps
- Convert daily users into peak QPS using actions per day and a peak factor.
- Split reads from writes; reads can use cache and replicas, writes hit the source of truth.
- Estimate payload size, because it exposes network, CDN, and storage bottlenecks early.
- Add headroom for p99 spikes, deployments, and losing one zone or one region.
- State the bottleneck that remains after cache and CDN offload, then scale that tier first.
Good follow-up questions
- Is the workload read-heavy, write-heavy, or bursty? Each points to a different tier.
- Which data must be strongly consistent right after a write, ruling out lagging replicas?
- Can requests be retried safely, or do we need idempotency keys (dedupe tokens) first?
- What happens when one region, cache cluster, or database replica fails under peak load?
Million-user baseline architecture
Put the numbers above into a picture and you get the usual first production-shaped layout for the shared scenario, before sharding or multi-region complexity. Every figure it depends on comes from Numbers to know above.
- A for static and cacheable bytes, sized to the byte offload target in the workbook.
- An L7 (HTTP-aware) in front of , at the replica count the workbook computed.
- A distributed cache for hot reads and sessions, running at the hit ratio assumed above.
- A primary database plus , so reads scale without touching write capacity.
- A queue for slow or retryable work, keeping it off the critical request path.
- Metrics-driven on QPS, /p99, or queue age, not CPU alone.
How teams usually grow
The baseline above is one snapshot; over years a product passes through several. Think of traffic as water and each tier as a pipe: the narrowest pipe, often database writes or a single server, floods first. Teams rarely jump to the last row without living through the earlier pains, and the costs here are illustrative orders of magnitude rather than benchmarks for your stack.
~1K users: Single-server phase
$50-$200/mo (order of magnitude)- One machine usually runs the app and the database together, because traffic is tiny
- You scale by upgrading CPU or RAM on that server, which is vertical scaling
- Typical for prototypes and early products, where simplicity matters more than redundancy
~100K users: Split responsibilities
$2K-$8K/mo (order of magnitude)- Separate the app tier from the database so each can be sized on its own
- Several app servers sit behind a load balancer, which spreads requests across them
- Add a cache for sessions or hot reads, plus read replicas for read-heavy traffic
~10M users: Horizontal app tier
$50K-$250K/mo (order of magnitude)- Many stateless app instances run behind load balancers, so any copy can serve anyone
- Sharding or specialised databases appear once writes no longer fit one primary
- Message queues and larger cache clusters absorb bursts before they reach the database
~100M users: Multi-region & specialization
$1M-$10M/year (order of magnitude)- Containers run across regions, with each service owned by one team per domain
- Many database shards and replicas, plus global traffic routing to the nearest region
- Strong observability and on-call rotas, because failures are routine at this size
~1B users: Planet-scale
$50M-$500M+/year (public reports vary)- Huge fleets of stateless compute, custom CDNs, and chaos testing (deliberately breaking things)
- Massive sharded storage and multi-home databases that serve from several regions at once
- Costs and complexity become as much about organisation and process as about technology
What forces the next stage
Scaling becomes harder when work cannot be divided independently. A single global counter, a hot inventory row, or one customer producing most traffic can remain a serial bottleneck across hundreds of servers. Partitioning helps only if the application can distribute ownership across multiple keys. For the remaining hot key, reduce work, batch updates, isolate it, or change the product’s coordination requirements deliberately.
Do not upgrade architecture only because the user count looks big. Move when one of these metrics proves the current stage is hurting latency, availability, or operational safety.
CPU, memory, database I/O, and deploy risk all fight for one server.
App CPU saturates, or you need more replicas to deploy without downtime.
Database read QPS or p95 read latency becomes the measured bottleneck.
Primary write QPS, lock contention, or data size exceeds what one database handles.
Distance-driven p95 latency, data residency rules, or an availability target forces local copies.
Cost shape
The cost column in the growth stages hides a pattern worth seeing on its own. Giant single instances get expensive fast, while many cheaper instances often track load more linearly, especially when paired with interruptible (spot) or committed-use discounts. Still pay attention to data egress (bytes leaving the cloud provider), managed database pricing, and operations headcount: dollars are not only compute hours.
Vertical pricing illustration (illustrative)
| Shape | $/mo (example) | Note |
|---|---|---|
| 4 vCPU / 16 GB | $150 | Baseline toy number |
| 16 vCPU / 64 GB | $600 | Roughly linear step |
| 64 vCPU / 256 GB | $3,200 | Cost acceleration begins |
| 448 vCPU / 24 TB | $120,000 | Ceiling niche |
Read the table bottom-up and the lesson is clear. Many small nodes usually win on cost per request for parallelisable work: as one example narrative, 100 modest VMs with discounts can outperform one huge server. The goal is roughly linear spend: when traffic doubles, you hope infrastructure roughly doubles rather than growing tenfold, assuming the architecture fits. Public write-ups cite pennies per user-year at extreme scale; treat those as maturity milestones, not guarantees for your product mix.
Case study pattern: incremental growth
The growth stages and cost curve above are abstract; here is how one social product actually walked them. The useful pattern for beginners is that they did not rewrite everything monthly. They doubled down when a tier broke (single server, database reads, shard boundaries), which is how most teams survive hypergrowth without freezing features forever.
Started very small: often one server running app and database together. Traffic spikes taught the team to upgrade hardware first, then split tiers.
Multiple app servers behind a load balancer, relational replicas, and fast session or hot-data caches. Kept the same core framework instead of rewriting from scratch.
Relational storage sharded by user, specialised stores for feeds. Shows the pattern: scale the thing that hurts first (writes, feed storage), not everything at once.
Large container fleets, hundreds of database shards, heavy caching. Still evolved incrementally rather than one giant rewrite.
Multi-region serving, video pipelines, ML ranking (machine-learned feed ordering): each layer scaled when demand forced it.
Lesson: add complexity when metrics force it, backed by load tests, not ego or hype.
Common mistakes
The case study shows the healthy path; these are the ways teams leave it. Most anti-patterns boil down to hidden state, using the wrong tool as a queue, or scaling distributed pieces before measuring the bottleneck.
Where scaling plans break, and what must never become false
Even a well-sized system from the sections above fails in production when a dependency dies and traffic has nowhere safe to go. Name the degraded mode (what the product still does when a piece is missing) before the outage, then drill it.
Degraded modes to name in advance
Temporary overload needs a policy for the period before new capacity becomes ready. Bound queues, reject excess work quickly, and preserve capacity for essential actions. A checkout service may stop generating recommendations while continuing order creation. A report service may accept fewer new jobs while finishing already accepted work. Define what “accepted” guarantees so shedding does not silently discard obligations already acknowledged.
Cache unavailable
Serve critical paths from the database with strict rate limits, disable non-critical reads, and prefer stale-but-safe local fallbacks over open-ended database fan-out.
Primary write saturation
Queue non-urgent writes, reject or shed (deliberately drop) low-priority traffic, keep payments and auth on a protected pool, and stop autoscaling that multiplies write pressure.
Region unhealthy
Fail over DNS or traffic steering only if the surviving regions have headroom. Accept higher latency or read-only mode rather than overload the healthy region.
Queue backlog
Protect user-facing APIs first. Scale consumers, drop or delay optional jobs, and publish a backlog ETA so product can choose which features pause.
Test and drill notes
A degraded mode that has never been exercised is a hope, not a plan. An AZ below is an availability zone, one isolated data centre inside a cloud region.
- Load test to 2 to 5 times the expected peak QPS with production-like cache hit ratios, not empty caches.
- Run a game-day (planned failure rehearsal): kill one AZ or cache shard and verify SLOs.
- Drill autoscaling cooldown and warm-up time; the degraded mode must cover that gap.
- Say which metric forces the next scaling stage, and what still works without it.
Failure timeline: single-server sale spike
Here is the sale evening from the hero, replayed minute by minute against the shared scenario, so you can see where each scaling move helps and where it does not.
t=0 One box runs app+DB; steady ~200 QPS; p95 fine t=5m Sale spike → ~1.2K peak QPS; CPU and disk I/O peg t=6m App threads wait on DB locks; p99 explodes; clients retry t=7m Vertical upgrade buys hours, but one failure domain remains t=8m Stateless scale-out + LB works for reads; primary write QPS still ~240 t=9m Cache 85% hits cut DB reads to ~144 QPS; writes still single-primary Outcome Horizontal app tier helps only if state is external and writes are sized
Counterexample to "just add app servers": more replicas without cache or write planning can amplify database connections and make the primary fail faster.
Correctness invariant: what must never become false
Scale the measured bottleneck, not the fashionable tier. App replicas stay so any healthy instance can serve the next request. Capacity after scale-out must absorb one-zone loss without silently multiplying write pressure onto a saturated primary: stop that worsens the write wall.
Wrap-up
You have seen the decision path, the mechanics, the numbers, and the failure modes. Use these exercises to check the ideas stuck, then follow the references for the primary sources.
Exercises
- 5M DAU × 10 actions/day, 4× peak factor, 70% reads. Compute peak QPS and database read QPS at 80% vs 95% cache hit. Which tier do you scale first?
- Your primary saturates at 300 write QPS. Would you scale up, add replicas, shard, or queue? Name the metric that forces the next stage.
- One AZ of a 3-AZ app pool dies at the shared peak QPS from Numbers to know. How many replicas and what headroom keep survivors under 100 QPS each?
References
- Google SRE Book: Addressing Cascading Failures / Managing Load (capacity, overload, and load shedding).
- Martin L. Abbott & Michael T. Fisher: The Art of Scalability (scale cube, AKF axes).
- Kubernetes Horizontal Pod Autoscaler docs: stabilization windows and custom metrics.
What to remember
Measure the bottleneck, then choose the smallest scaling move. Prefer app tiers and until metrics force the next rung of the ladder. Automate capacity with cooldowns, load test before launches, and keep giant-scale stories as intuition, not a checklist for your MVP.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Choose scale-up or scale-out from bottleneck shape, state, partitionability, failure domain, benchmark evidence, and total cost.
Explain the decision
When should a database scale vertically before sharding?
Read the reasoning
When the working set, IOPS, locks, and connection load still fit a larger node and sharding complexity costs more than the headroom it buys.
Apply it to a production scenario
CPU is 35% but p99 latency rises with traffic. Why may adding replicas fail?
Read one approach
The bottleneck may be a shared database, lock, queue, network, or dependency; profile saturation and queueing before scaling the stateless tier.
Work through the numbers
A node handles 4,000 QPS at 65% safe utilization. How many nodes support 40,000 peak QPS with N+2 fault headroom?
Read the worked calculation
ceil(40,000 / 4,000) + 2 = 12 nodes, subject to partition and downstream limits.
Reason through a failure
What must online resharding preserve?
Read the recovery reasoning
Every key has one authoritative owner at each migration state, and reads/writes cannot be lost or applied twice during routing-map changes.
Sources and further study
Continue with Regionalisation & Edge.