System Design Foundations

System Design for Large-Scale Applications

A side project can run on one laptop. A product like Instagram, ChatGPT, or a payments API cannot: millions of clients, petabytes of state, and machines that fail every day. A is any system whose load, data, or geography forces you off a single machine and into coordinated fleets.

This note is the map of that world: how traffic enters, how work fans out, where state lives, how capacity is estimated, and which trade-offs you must name in an interview before you draw a pretty diagram.

Scale-up journey

Users → QPS, storage, bandwidth, and latency budgets.

Put load balancers in front of interchangeable app replicas.

Protect the data plane

Cache, replicate, and shard so state survives growth and failure.

Decide which features drop before the whole product does.

What actually happens

Start with a single server: app, database, and files on one host. It is easy to reason about and easy to lose. When traffic grows, the first instinct is —a bigger box. That buys time. It does not buy redundancy, and hardware ceilings arrive faster than product ambition.

The durable pattern is of a tier behind a , with durable state pushed into stores that support and, when one primary cannot hold the working set, . A sits in front of slow reads. Each of those moves removes one bottleneck and introduces a new failure mode: cache stampedes, hot shards, replica lag, balancer misconfiguration.

At product scale, a user click rarely hits one service. It across auth, profile, ranking, ads, and media. Your job is to keep that graph inside an , remove every , and keep a story for the day a dependency burns.

How to choose a scale strategy

1. Is the bottleneck CPU/RAM on one box, or traffic volume? Vertical scaling helps a hot primary briefly. Sustained multi-region traffic almost always needs horizontal scale of the request path.

2. Is the data partitionable? If a natural shard key exists (user id, tenant id, object id) and cross-shard queries are rare, sharding unlocks linear growth. If every query needs a global join, redesign the read model before you split writes.

3. What must stay correct under failure? Payments and inventory need stronger write paths. Feeds and recommendations can often degrade. Name the invariant before you pick the topology.

Simple rule of thumb

Estimate first. Make the app tier disposable. Put truth in replicated stores. Cache aggressively for reads you can afford to be slightly stale. Never ship a design with an unnamed SPOF or an unnamed degradation plan.

Request path at scale

The sketch below is the default skeleton for most internet products. CDN and DNS sit closest to the user. The load balancer is the stable front door. App replicas should be interchangeable. State lives behind them.

mermaid

flowchart LR
    U[User / mobile / browser] --> DNS[DNS]
    DNS --> CDN[CDN / edge]
    CDN --> LB[Load balancer]
    LB --> A1[App replica A]
    LB --> A2[App replica B]
    LB --> A3[App replica C]
    A1 --> Cache[(Distributed cache)]
    A2 --> Cache
    A3 --> Cache
    Cache --> DB[(Primary + replicas)]
    A1 --> Q[[Async queue]]
    Q --> W[Workers]

Edge

Static assets, TLS termination, geo routing, and sometimes cached API responses. Cuts RTT for bytes that do not need origin.

App tier

Stateless workers: auth, validation, orchestration. Scale by replica count; drain unhealthy nodes via the balancer.

Data + async

Cache and DB for the critical path. Queues for email, feeds, media, analytics—work the user should not wait on.

Back-of-envelope: turn product goals into machines

Interviewers expect a pass before boxes. The numbers do not need to be exact; they need to force the right shape.

plaintext

Example: 50M DAU social read-heavy product

Assumptions:
  Average 20 requests / user / day
  Peak = 3× average (lunch / evening spike)
  Average payload response ≈ 5 KB
  Read:write ≈ 100:1
  Retain 2 years of metadata

Daily requests = 50e6 × 20 = 1e9
Average QPS   = 1e9 / 86400 ≈ 11,600
Peak QPS      ≈ 35,000

If one app replica sustains 500 QPS comfortably:
  Need ≈ 70 replicas + headroom → plan ~100

Storage (toy):
  50M users × 2 KB profile ≈ 100 GB
  200 posts / user / year × 1 KB × 2 years ≈ 20 TB metadata
  Media lives in object storage, not the OLTP DB

Bandwidth at peak:
  35k QPS × 5 KB ≈ 175 MB/s egress from origin (CDN absorbs most static)

Quantity	Rough formula	Design implication
Peak QPS	DAU × actions/day × peak factor / 86400	Sizes app fleets and DB connections
Storage growth	writes/day × bytes × retention	Forces sharding / tiered storage
Latency budget	SLO − edge − TLS − known DB floor	Caps fan-out depth and remote calls
Cache hit ratio	1 − (DB QPS / read QPS)	Tells you if caching is mandatory

State: where truth lives

Horizontal scale is easy for CPU-bound request handlers and hard for shared mutable state. Most large systems separate disposable compute from durable truth, then add caches as an acceleration layer—not as a second source of truth.

mermaid

flowchart TB
    subgraph compute [Disposable compute]
      LB[Load balancer]
      APP[Stateless app replicas]
      LB --> APP
    end
    subgraph accelerate [Acceleration]
      L1[In-process cache]
      L2[(Redis / Memcached)]
      APP --> L1 --> L2
    end
    subgraph truth [Source of truth]
      P[(Primary)]
      R1[(Replica)]
      R2[(Replica)]
      P --> R1
      P --> R2
      L2 --> P
    end
    subgraph blob [Bulk / media]
      S3[(Object store)]
      APP --> S3
    end

Replication answers “what if a disk dies?” and “can reads scale?” It does not answer “which write wins?”—that is consistency policy.

Sharding answers “what if one primary cannot hold the working set?” It creates cross-shard joins, rebalancing, and hot-key risk. Prefer a key that matches access patterns (user id for social profiles; order id for commerce) and avoid keys that concentrate traffic (a single celebrity id without secondary mitigation).

Caches answer “can we avoid repeating expensive work?” They create staleness and stampede risk. Treat cache as a performance optimization with an explicit invalidation or TTL story, never as the ledger for money.

Scale dimensions that pull the design apart

Dimension	What breaks first	Typical response
Concurrent users	App CPU, connection pools, balancer	Horizontal app scale, pooling, async I/O
Data volume	Primary disk, backup windows, index size	Sharding, tiered storage, archival
Write intensity	Primary WAL, lock contention	Partition writes, queues, CQRS-style reads
Geography	Cross-region RTT, partition risk	Regional active stacks, eventual sync
Team / services	Deploy coupling, ownership chaos	Service boundaries, contracts, platforms

Fan-out and the critical path

A home-page request that touches twenty services does not have twenty independent latencies—it has one user-visible wait equal to the slowest required dependency (plus serialization and queueing). Optional dependencies should time out into degraded UI, not block the whole response.

plaintext

Critical path (must succeed):
  auth → profile → timeline core → render shell

Optional (degrade if slow):
  recommendations, ads, “people you may know”, unread badges

Budget example for p95 ≤ 400 ms page:
  edge + TLS          40 ms
  auth                30 ms
  profile (cached)    15 ms
  timeline core       120 ms
  render / serialize  40 ms
  slack / jitter      155 ms
  ─────────────────────────
  Total budget        400 ms

If recommendations need 200 ms p95, they cannot sit on the critical path.

Interview signal

Saying “we’ll call N services in parallel” without a budget and a degradation plan is incomplete. Parallelism helps only when calls are independent and when the user is allowed to proceed without the slowest optional result.

Failure is the steady state

Common SPOFs to name explicitly

Single primary database without failover rehearsal
One region owning global writes with no read path elsewhere
Shared cache cluster with no eviction / memory policy
Config or secret store that blocks every boot
A “temporary” manual step in deploy or failover

Resilience moves that usually pay off

Health-checked load balancing and fast drain
Deadlines and bounded retries (not unbounded hope)
Circuit breakers and bulkheads around fragile deps
Cached or empty fallbacks for non-critical cards
Chaos drills and runbooks before the real outage

Trade-offs you must say out loud

Choice	You gain	You pay
Horizontal vs vertical	Elasticity, redundancy	Ops complexity, distributed failure modes
Strong vs eventual consistency	Clearer user mental model	Latency, availability under partitions
Cache aggressively	Lower DB load and latency	Staleness, invalidation bugs, stampedes
Microservices early	Team autonomy (sometimes)	Network hops, ops tax, distributed debugging
Multi-region active-active	Lower RTT, regional survival	Conflict resolution, higher cost
Sync user path vs async	Immediate confirmation	Longer critical path, tighter coupling

There is no free axis. A good large-scale design names which product requirement owns each trade-off: money paths favor correctness; media delivery favors throughput and locality; social feeds often favor availability and freshness windows.

A worked mini design: URL shortener at scale

Apply the skeleton. Requirements: 100M new links/month, 10:1 read:write, 99.9% reads under 50 ms from cache or edge, durable mapping, custom aliases rare.

plaintext

Write path:
  Client → LB → API → generate ID → write primary → ack
  Async: replicate to replicas, warm cache, analytics

Read path (hot):
  Client → CDN/edge cache → hit? return
  miss → LB → API → Redis → hit? return
  miss → read replica → populate cache → return

Scale levers:
  App: stateless, autoscale on QPS
  Cache: sharded Redis, TTL + write-through
  DB: hash-partition by short code; primary per shard
  IDs: preallocated ranges or snowflake-style to avoid central lock

Degradation:
  If cache fleet is sick: serve from replicas with higher latency
  If a shard primary is down: block writes to that shard; keep reads on replicas
  Never invent redirects from a Bloom filter alone

Choosing and explaining large-scale application design

A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.

Explain the core idea

Start with requirements, SLOs, and a back-of-envelope estimate before drawing boxes.

Keep the request path horizontally scalable and push durable truth into replicated, possibly sharded stores.

Name SPOFs, consistency needs per API, and which features degrade under partial failure.

When it belongs

Traffic, data, or geography clearly exceeds one reliable machine.

The interviewer asks how the system grows from MVP to millions of users.

You must justify caches, queues, shards, or multi-region placement.

How it fails

Hot shards and celebrity keys overwhelm an otherwise balanced design.

Cache stampedes and thundering herds after expiry or restart.

Unbounded fan-out blows the latency SLO even when each service looks fine.

Unrehearsed failover turns a redundant primary into an effective SPOF.

Trade-offs to weigh

Horizontal scale buys capacity and redundancy at the cost of distributed complexity.

Strong consistency simplifies product rules but hurts latency and partition behavior.

Aggressive caching improves read SLOs while introducing staleness and invalidation risk.

Common misunderstandings

Jumping to microservices or Kafka before estimating load.

Treating the cache as the source of truth.

Ignoring peak QPS and designing only for daily averages.

Drawing a multi-region diagram without a conflict or lag story.

Wrap-up

Large-scale design is disciplined decomposition: estimate, separate compute from truth, scale each axis deliberately, and plan for the day a dependency fails.

Exercises

Estimate a chat product. 20M DAU, 50 messages/user/day average, peak 4×. Size peak write QPS and argue whether one relational primary is plausible for metadata alone.
Mark the critical path. For a newsfeed request with eight dependencies, label each must-have vs optional and invent a degradation for two of the optional ones.
Kill a component. On the request-path diagram, remove the cache tier, then the primary DB, then one region. For each, state what users still can and cannot do.

Key takeaway

A large-scale application is a fleet with a product promise. Estimate load, keep apps disposable, put truth in replicated (and if needed sharded) stores, budget fan-out against SLOs, and design graceful degradation before the outage teaches you which features were actually optional.