System Design Foundations

Networking Fundamentals

Protocol notes teach TCP, HTTP, and QUIC mechanics. This page asks a different question: how do networks constrain system design? A ChatGPT response, an Uber location ping, or a Netflix segment fetch succeeds or fails based on , , loss, geography, and how many round trips your architecture invented.

Treat the network as an unreliable, delayed, capacity-limited messenger. Design for , , , and —then leave deep handshake diagrams to the Network Protocols note.

Design questions

How many RTTs?

Count sequential remote waits on the user path.

How many bytes?

Bandwidth and serialization cost for hot payloads.

How does it fail?

Loss, timeout, partition, and retry amplification.

Where do bytes live?

Edge vs origin vs cross-region replication.

What actually happens

From a designer's seat, the network has four properties that matter more than header layouts: delay (RTT), capacity (bandwidth), unreliability (, reordering), and variability (). Reliable transports hide loss by waiting and retransmitting, which turns loss into . Realtime paths often prefer dropping stale data over waiting forever.

Geography multiplies delay. Same-rack calls are cheap; cross-region calls can consume an entire mobile budget. That is why and regional data copies exist. A is the extreme case: some nodes cannot talk, and your consistency story must already know what to do when fire.

Finally, connections are not free. Handshake and TLS cost show up under connection churn. and pools amortize setup; stops producers from flooding a sick peer. Protocol selection (TCP vs UDP vs QUIC, HTTP/1.1 vs HTTP/2 vs HTTP/3) is covered deeply elsewhere—here we care about the design consequences: multiplexing, head-of-line risk, and migration on mobile.

How to choose network-aware topology

1. Is the path latency-bound or bandwidth-bound? Many small RPCs → cut RTTs (batch, colocate, cache). Large media → place near users and provision pipes; windowing and concurrency matter ().

2. Must the user wait on remote truth? If yes, put that data in-region and budget the RTT. If no, async replicate and serve local/cached approximations.

3. What is the failure policy? Timeout values, retry ownership, and degradation UI belong in the design doc next to the sequence diagram—not as afterthoughts.

Simple rule of thumb

Budget milliseconds and bytes first. Eliminate chatty sequential calls. Keep warm connections. Put cacheable bytes at the edge. Assume partitions and write the timeout story before you claim “multi-region.”

RTT map: where delay comes from

mermaid

flowchart LR
    U[User] -->|mobile RTT 30-80ms| E[Edge / CDN]
    E -->|same region 1-5ms| LB[Load balancer]
    LB --> APP[App]
    APP -->|0.5-2ms| CACHE[(Cache)]
    APP -->|1-5ms| DB[(Regional DB)]
    APP -->|50-120ms| REMOTE[Cross-region dependency]
    REMOTE -.->|often too expensive| X[Keep off critical path]

Hop	Typical delay	Design move
User ↔ edge	Tens of ms (mobile/wifi)	CDN, TLS session reuse, HTTP/2+
App ↔ cache/DB same AZ	Sub-ms to few ms	Pool connections; avoid N+1
Cross-AZ	Low ms + bandwidth bill	Keep chatty pairs co-located
Cross-region	~30–150+ ms RTT	Async, local reads, careful writes

Bandwidth vs latency: two different bottlenecks

A 10 Gbps link does not make a 20-RPC waterfall fast. Conversely, a low-RTT path does not stream a 4K movie if the pipe is tiny. Name which constraint you are fighting.

Latency-bound symptoms

Many sequential dependencies
Small payloads, high QPS
p99 dominated by waits, not transfer time
Fixes: batch, parallelize safe calls, cache, colocate

Bandwidth-bound symptoms

Large objects, video, backups, bulk sync
Transfer time ≈ size / throughput
Underfilled long fat networks (BDP / windowing)
Fixes: compress, chunk, parallel streams, edge caches

plaintext

BDP quick check:
  1 Gbps × 100 ms RTT = 12.5 MB in flight to fill the pipe
  One timid flow with a 64 KB window saturates only a fraction

Interview phrasing:
  "This path is latency-bound; adding bandwidth will not help.
   We should remove RTTs, not buy a bigger NIC."

Loss, retries, and tail latency

mermaid

sequenceDiagram
    participant C as Client fleet
    participant E as Edge
    participant S as Sick service

    C->>E: Requests
    E->>S: Forward
    S-->>E: Slow / drop
    E-->>C: Timeout
    C->>E: Retry wave 1
    C->>E: User refresh wave
    E->>S: 3-5x load
    Note over S: Recovery stalls; p99 explodes
    Note over C,S: Fix: one retry owner, jitter, deadline, shed load

on a multiplexed connection can stall many streams together (classic TCP + HTTP/2 HOL risk). That is a design reason teams care about HTTP/3/QUIC on lossy mobile—not a trivia flex. For APIs, the practical controls are deadlines, bounded retries with jitter, retry budgets, and circuit breakers.

Fan-out makes contagious: if each of twenty parallel calls has 1% chance of a slow network tail, the page often sees at least one. Hedging can help if carefully bounded; unbounded hedging is just a polite retry storm.

Timeouts, partitions, and what you promise users

A is how software notices a (or overload that looks like one). Too aggressive and you flap; too generous and you hold threads, connections, and user attention hostage.

plaintext

Deadline pattern (preferred over fixed blind timeouts):

  User budget: 800 ms
  Edge reserve: 100 ms for fallback UI
  Remaining: 700 ms propagated to service A
  A calls B with 200 ms sub-deadline
  A calls C with 150 ms sub-deadline
  If remaining < 80 ms: skip non-critical calls; serve partial

Partition policy examples:
  Reads: serve regional stale copy (availability)
  Payments: refuse write if primary quorum unreachable (correctness)
  Chat typing: drop updates; next message reconciles

Connection cost and reuse

Cold starts pay DNS + TCP + TLS. Under load, that dwarfs a 1 KB JSON body. via pools, keep-alive, and multiplexed protocols turns setup into an amortized cost. Oversized pools create thundering herds of connections to databases; undersized pools queue requests in the client.

Pattern	Helps	Risk
HTTP keep-alive / H2 multiplex	Fewer handshakes	HOL on lossy TCP for H2
DB/API connection pool	Stable latency under QPS	Pool stampede after restart
gRPC channel reuse	Cheap internal fan-out	Hidden blast radius if misconfigured
Short DNS TTLs everywhere	Faster failover	Extra lookups; cache wisely

Edge, origin, and geographic placement

CDNs shine for immutable or long-TTL assets and can accelerate some API GETs. Personalized feeds, carts, and balances usually need origin logic. Multi-region active-active reduces read RTT but demands conflict rules. Active-passive is simpler operationally and worse for distant readers unless you add read replicas with explicit staleness.

Replication itself is a network product: cross-region pipes, backlog lag, and bandwidth cost during catch-up after an outage. Size those links in the same pass as user QPS.

Chatty APIs: the self-inflicted network tax

plaintext

Anti-pattern (N+1 over the network):
  GET /users/1
  GET /users/1/posts
  GET /users/1/posts/10/comments
  GET /users/1/posts/11/comments
  ...

Better:
  GET /users/1/timeline?cursor=...   (one RTT, server-side join)
  or batched GetPosts(ids=[...]) over a warm channel

Mobile twist:
  Each RTT on cellular is expensive and lossy.
  Prefer fewer larger responses with clear pagination
  over dozens of 200-byte calls.

Trade-offs

Choice	Gain	Cost
More edge caching	Lower user RTT, origin shield	Invalidation complexity, stale risk
Short timeouts	Fast fail, free resources	False errors under normal jitter
Aggressive retries	Mask blips	Amplify outages
Cross-region sync reads	Stronger freshness	RTT burns SLO
Fat batched responses	Fewer RTTs	Wasted bytes, slower TTFB if oversized
Multiplexed long connections	Amortized setup	Shared failure domains / HOL effects

Choosing and explaining networking for system design

A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.

Explain the core idea

Separate latency-bound paths from bandwidth-bound paths and optimize accordingly.

Budget RTTs and bytes on the critical path; push cross-region work off it when possible.

Pair every remote dependency with timeouts, retry ownership, and a partition/degradation story.

When it belongs

The design spans regions, mobile clients, or many service hops.

SLOs are tight relative to geographic RTT.

The interviewer asks what happens when a dependency is slow or unreachable.

How it fails

Chatty sequential RPCs exhaust the latency budget.

Retry storms amplify a partial outage.

Connection pool stampedes after restarts overwhelm backends.

Cross-region synchronous calls make p99 unusable.

Trade-offs to weigh

Edge freshness versus origin load and invalidation complexity.

Timeout aggressiveness versus false failure rate.

Batching reduces RTTs but can inflate payloads and TTFB.

Common misunderstandings

Confusing this topic with memorizing TCP header fields.

Assuming a fat pipe fixes a waterfall of RPCs.

Claiming multi-region without timeout and conflict policies.

Retrying at every layer without a budget.

Wrap-up

Networking fundamentals for system design are about budgets, geography, failure detection, and amplification—not about winning a protocol trivia contest. Use Network Protocols when you need handshake and stream mechanics.

Exercises

Rewrite a chatty mobile flow of eight GETs into at most two round trips. State what moved server-side.
Pick timeout numbers for an 600 ms SLO with two parallel dependencies and one optional recommendation call.
Classify video segment fetch vs checkout price quote as bandwidth-bound or latency-bound and list two levers for each.

Key takeaway

Design to the network you have: finite RTT, finite bandwidth, loss that becomes delay, and partitions that force a consistency choice. Cut chattiness, reuse connections, place bytes wisely, and never leave timeouts and retries unnamed.