System Design Foundations
Computer Systems Overview
Every distributed diagram eventually lands on physics: a CPU executes instructions, holds the hot set, disks persist bytes, and networks move them. When Instagram serves a photo or ChatGPT streams a token, the design is constrained by how far those bytes sit in the and how many the path requires.
This note treats CPU, memory, disk, and network as design constraints—not OS trivia. Know the orders of magnitude, classify work as or , and place data for .
Constraint checklist
CPU
Is the hot path compute-heavy or mostly waiting?
Does the working set fit in RAM?
Disk
Random vs sequential? fsync on the critical path?
Network
How many RTTs and how many bytes per request?
What actually happens
A request enters a machine, burns some CPU on parsing and business logic, touches memory, may read or write disk, and often waits on another machine across the network. Latency is dominated by the slowest necessary wait—not by the average instruction cost. That is why adding CPU to an API often does nothing useful, while moving a hot index into can change the product feel overnight.
The is the organizing idea. L1 is tiny and nanoseconds; RAM is large and still fast; is persistent but ~1000× slower than RAM; is cheaper still and seek-bound; the public internet adds tens of milliseconds of . Good designs keep the local and high, and they leave so bursts do not queue forever.
Two more hidden taxes show up at scale: (CPU and bandwidth to turn objects into bytes) and poor (pointer-chasing structures, cross-AZ chatter, or cold caches). Numbers below make these costs concrete.
How to use these constraints in design
1. Classify the bottleneck. Profile or estimate: CPU cycles, memory footprint, disk IOPS/bandwidth, or network RTT/bytes. Optimize the dominant resource first.
2. Fit the working set. If hot keys fit in RAM or a distributed cache, design for that. If they do not, accept disk latency or change the access pattern (batch, sequential, approximate).
3. Count RTTs on the critical path. A “simple” microservice call that needs five sequential remote dependencies spends five RTTs before business logic finishes—often worse than one slightly fatter service.
Simple rule of thumb
Memorize orders of magnitude, not vendor SKUs. Keep hot data in memory, sequentialize disk, minimize cross-region chatter on the user path, and leave headroom so p99 does not explode at 90% utilization.
Latency hierarchy (orders of magnitude)
flowchart TB
CPU[CPU register / L1 ~ 1 ns] --> L2[L2/L3 cache ~ few ns]
L2 --> MEM[DRAM ~ 100 ns]
MEM --> SSD[SSD random read ~ 50-200 μs]
SSD --> HDD[HDD seek ~ 2-10 ms]
HDD --> DC[Same-datacenter RTT ~ 0.5-2 ms]
DC --> REG[Cross-region RTT ~ 30-150 ms]
REG --> WAN[Congested / mobile path ~ 100-500+ ms]| Operation | Rough time | In 100 ms you can… |
|---|---|---|
| L1 cache reference | ~1 ns | ~100 million |
| Main memory reference | ~100 ns | ~1 million |
| SSD random read | ~100 μs | ~1,000 |
| HDD seek | ~5 ms | ~20 |
| Same-region RPC | ~1 ms | ~100 sequential (impractical) |
| Cross-region RPC | ~50–100 ms | ~1–2 on the critical path |
These are teaching approximations (Jeff Dean–style numbers and modern flash blur the edges). The design lesson is relative: memory beats disk; disk sequential beats random; local network beats WAN; and one cross-region hop can consume an entire mobile SLO.
CPU vs I/O: pick the right lever
Misclassifying the bottleneck wastes money. A fleet waiting on Postgres will not get faster from more application cores. A video transcoder will not get faster from a bigger Redis.
Usually CPU-bound
- Compression, encryption, image/video encode
- Heavy JSON/XML parsing at huge QPS
- ML inference without a dedicated accelerator story
- Tight loops over in-memory structures
Levers: better algorithms, SIMD/vectorization, binary protocols, more cores, batching to amortize fixed CPU cost.
Usually I/O-bound
- CRUD APIs waiting on DB or remote services
- Cache misses falling through to disk
- Cross-service orchestration on the user path
- Large object downloads limited by bandwidth
Levers: cache, colocate, async off critical path, connection pooling, fewer RTTs, sequential/batched disk access.
Memory: working set and cache lines
Application heaps, OS page cache, and distributed caches all try to keep the above disk. When memory pressure forces paging or cold misses, latency distributions grow heavy tails even if average CPU looks fine.
Working-set check for a session store:
Active sessions: 5M
Session blob: 2 KB
Raw working set ≈ 10 GB
With indexes + fragmentation overhead ≈ 15-20 GB
Design choices:
- Fit in one large RAM machine (vertical, SPOF risk)
- Shard across N cache nodes (horizontal, hot-key risk)
- Spill cold sessions to DB (cheaper, higher miss latency)
Cache-line intuition:
Reading 1 byte of a struct often pulls ~64 bytes.
Arrays of structs scan faster than pointer-linked nodes
for sequential analytics over hot data.awareness rarely appears as a box on a whiteboard, but it explains why compact columnar layouts and contiguous buffers beat pointer-rich object graphs for high-throughput in-process work.
Disk: persistence has a price
Disks (and cloud block volumes) give durability. They also give latency cliffs. Databases that call fsync on every commit trade throughput for crash safety. LSM trees turn random writes into sequential ones; B-trees optimize point lookups differently. As a designer, ask whether the user path truly needs durable acknowledgment before reply.
sequenceDiagram
participant App
participant DB
participant Disk
App->>DB: BEGIN write
DB->>Disk: Write WAL pages
Disk-->>DB: Data in page cache (fast)
DB->>Disk: fsync WAL
Note over Disk: Milliseconds possible under load
Disk-->>DB: Durable
DB-->>App: COMMIT ack
Note over App,Disk: Skipping fsync boosts QPS and risks acknowledged-but-lost writes after power loss| Pattern | Good for | Watch out |
|---|---|---|
| Random small reads on HDD | Almost nothing online | Seek storm; move to SSD or cache |
| Sequential scan | Analytics, log shipping | Competes with latency-sensitive traffic |
| fsync per commit | Money, inventory | Caps write QPS; needs batching carefully |
| Async durability | Metrics, logs, feed fan-out | Acknowledge only what you can lose |
Network: bandwidth vs latency as constraints
Bandwidth answers “how many bytes per second can this pipe move?” Latency answers “how long until the first useful byte returns?” Large downloads are bandwidth-bound; chatty RPC graphs are latency-bound. Compression trades CPU for bandwidth; batching trades wait time for fewer RTTs.
Bandwidth-delay example:
100 Mbps link, 80 ms RTT
Bandwidth-delay product ≈ 100e6/8 × 0.08 ≈ 1 MB in flight
Tiny windows underutilize the path; need enough concurrency or window
Design implications:
- Many small sequential RPCs waste the path (latency bound)
- One bulk transfer needs windowing / parallelism (bandwidth bound)
- Cross-AZ chat at high QPS becomes a real bill and a real latency tax
- Place caches and compute near data; place edges near usersHidden CPU taxes on “simple” APIs
TLS handshakes, JWT validation, JSON encode/decode, protobuf reflection, and object allocation all sit on the path before the database call. Connection reuse and binary protocols reduce repeated work. Oversized heap churn creates GC pauses that show up as p99 spikes people misattribute to “the network.”
Interview move: when proposing JSON-over-HTTP between internal services at huge QPS, mention serialization cost and consider gRPC/protobuf or at least pooled buffers—not as fashion, but as CPU and bandwidth budget.
Trade-offs
| Choice | Gain | Cost |
|---|---|---|
| Keep working set in RAM | Stable low latency | $$ memory; cold-start / failover pain |
| Durable fsync on commit | Crash-safe acks | Lower write throughput |
| More microservice hops | Team isolation | RTT + serialization tax |
| Compress on the wire | Less bandwidth | CPU; can hurt loaded p95 |
| Run at 95% utilization | Lower idle cost | Queueing → terrible tails |
| Binary vs JSON | CPU and bytes | Debuggability, tooling |
Worked example: why “just call the other service” fails
Suppose a page needs profile, counters, and recommendations. Naive design: three sequential HTTP calls to co-located services, each 3 ms compute + 1 ms RTT + 2 ms JSON.
Sequential:
3 × (1 ms RTT + 2 ms JSON + 3 ms work) = 18 ms best case
Plus queues under load → p95 much worse
Parallel (same machine room):
max(6 ms, 6 ms, 6 ms) ≈ 6 ms compute/RTT path
Still pays three JSON codecs and three connection/pool slots
Better:
One BFF/aggregator with a join query or batched RPC
Cache profile + counters in RAM (working set fit)
Recommendations async or deferred below the fold
Hardware reminder:
If counters live on HDD-backed rows without cache,
one miss can be 5 ms+ and dominate the parallel max.Choosing and explaining computer systems constraints
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
State orders of magnitude for memory, disk, and network before proposing topology.
Classify the bottleneck as CPU-bound or I/O-bound and optimize the dominant resource.
Keep the working set local and high in the memory hierarchy; count RTTs on the critical path.
When it belongs
Capacity or latency targets force a choice between cache, disk, and remote calls.
Someone proposes many fine-grained services without a latency budget.
Durability requirements (fsync, replication) must be weighed against write QPS.
How it fails
Working set spills to disk and p99 collapses under load.
Cross-region chatter on the user path burns the entire SLO.
Running near 100% utilization creates nonlinear queueing delays.
Serialization and GC dominate after the database is no longer the bottleneck.
Trade-offs to weigh
Memory cost versus disk latency.
Durability (fsync/replication) versus write throughput.
Compression and binary protocols save bandwidth at CPU and complexity cost.
Common misunderstandings
Assuming bigger CPUs fix database waits.
Ignoring serialization cost at high QPS.
Treating “same cloud region” as free when cross-AZ traffic is heavy.
Designing to average latency while utilization leaves no headroom for bursts.
Wrap-up
Distributed systems are computer systems glued by networks. The whiteboard boxes only work if the bytes fit in memory, the disk pattern matches the medium, and the RTT budget matches the geography.
Exercises
- Budget a 150 ms mobile API. Allocate edge, TLS, app, cache, DB, and at most one remote dependency. What must be cached?
- Size a working set. 20M active keys × 1.5 KB. How many cache nodes at 40 GB usable RAM each, with 30% overhead?
- Classify three endpoints in a product you know as CPU-bound or I/O-bound and name one lever for each.
Key takeaway
CPU, memory, disk, and network are the real capacity plan. Keep hot data high in the hierarchy, count RTTs, leave headroom, and only then draw the microservices.