System Design Networking
Client-Server Model
You type "cats" into Google Search and expect results in a couple hundred milliseconds. To you it is one box and one Enter key. Behind that click, a browser asks, and a fleet of machines answers.
The starts the conversation. The listens, runs logic, reads data, and replies. Load balancers, caches, and regions grow around that idea, but the shape stays the same.
One search click
Sends HTTPS for the query page.
Serve cached bytes when possible.
Pick a healthy app server.
Compute, cache, or query, then respond.
What actually happens
Client and server are roles within a particular conversation. A browser is a client of the notes API, while that API becomes a client when it queries a database. This is why a system can contain many client-server relationships without changing the pattern. Each boundary needs a request format, authentication context, timeout, and response or error contract.
Consider saving a note: the browser sends the note identifier, new content, and credentials. The server verifies the caller’s permission and commits the update before returning success. The browser may optimistically display the new content while waiting, but must keep track of whether it is pending or confirmed. A locally updated screen is not evidence that the server persisted the data.
The browser is the . It cannot safely hold every user's search index or enforce every product rule. So it asks a that owns business logic and data. There may be many machines behind one hostname. That is still client-server from the browser's point of view.
At huge scale the plumbing grows: a near the user, a in front of app fleets, caches, and databases. Search, streaming, and social workloads at billion-user scale show why a "one server" mental model breaks quickly. Exact traffic shapes live in Numbers to know.
Clients initiate work; servers centralize rules and data so you can patch, secure, and scale one place instead of every laptop. That is why this pattern dominates the public internet.
Before the deeper walkthroughs, lock three design choices: who owns truth, whether any server can take the next request, and what the client can speak.
How to choose the shape
1. Who owns truth and policy? Prefer client-server when you need central security, consistency, and easy deploys. Reach for peer-to-peer only when bandwidth between users is the main cost ( video is the common exception).
2. Can any server handle the next request? Default to with state in Redis or a database. Avoid unless you truly need in-memory affinity.
3. What can the client actually speak? Browser JavaScript cannot open raw UDP sockets. Plan HTTPS or WebSockets for web clients; use WebRTC only when you need peer media or data channels.
Simple rule of thumb
Start with HTTPS over TCP (or QUIC/HTTP/3 when the edge already enables it), request/response, and app servers behind a reverse proxy or load balancer. Deviate only when the product forces it: chat may need long-lived sockets, live media may need WebRTC, internal fleets may prefer .
Client owns vs server owns
| Concern | Client responsibility | Server responsibility |
|---|---|---|
| Retries | Retry safe reads with jitter and timeout. | Make writes idempotent when clients may retry. |
| Auth tokens | Store tokens safely and refresh before expiry. | Validate tokens, scopes, and revocation rules. |
| Caching | Use browser or app cache for repeated reads. | Send cache headers and invalidate changed data. |
| Rendering | Render UI and handle slow networks gracefully. | Return stable, small response shapes. |
| Business state | Keep temporary UI state only. | Own source-of-truth state in services and databases. |
What actually happens across the layers
L3: IP
This layer decides where packets should go. A public IP can be reached from the internet. A private IP usually works only inside your home network, office network, or cloud VPC.
Ask yourself: what address does the user connect to, and does a load balancer forward it somewhere else?
L4: TCP / UDP / QUIC
This layer decides how data is delivered. TCP provides an ordered byte stream by recovering missing data while the connection remains viable. UDP supplies datagrams without built-in delivery or ordering guarantees. QUIC builds reliable streams and congestion control over UDP, so using UDP underneath does not mean an application accepts lost data.
Ask yourself: is it more important to be reliable, or to stay realtime even if a few packets are lost?
L7: App protocols
This is the layer your product usually talks about. DNS finds the server. HTTP loads pages and APIs. WebSockets keep a chat-style connection open. WebRTC helps with calls and realtime media.
Ask yourself: is this a normal request, a live chat stream, or a realtime call?
A simple web request, end to end
One browser click feels like a single GET. Underneath, it is a short story: find the address, say hello, exchange the HTTP request and response, then say goodbye. Many packets hide behind one conceptual request.
Read the diagram definition
sequenceDiagram
autonumber
participant Browser
participant DNS
participant Server
Browser->>DNS: Resolve example.com
DNS-->>Browser: Destination IP
Browser->>Server: SYN
Server-->>Browser: SYN-ACK
Browser->>Server: ACK (TCP ready)
Browser->>Server: HTTP request (GET / path, headers)
Server-->>Browser: HTTP response (status, headers, body)
Browser->>Server: FIN
Server-->>Browser: ACK
Server->>Browser: FIN
Browser-->>Server: ACK (connection closed)What your app sees
An ordered stream of bytes. TCP handles missing pieces and reordering so you can focus on request and response logic.
One click, many packets
DNS, the TCP hello, HTTP pieces, acknowledgements, and goodbye steps can easily become dozens of packets.
Why connections cost
Each live session uses memory and CPU. A fresh hello adds wait time before useful data. Idle connections still hold resources.
Reuse when you can
Keep-alive reuses the same pipe for later requests. HTTP/2 can share many requests on one connection instead of opening several.
After that walkthrough, the useful questions are where latency is spent and what the browser cannot do for you.
Where time goes on one click
Latency budget
Separate one-time connection setup from work repeated on every request. A warm connection can reuse DNS results, transport state, and encryption keys; the request still pays network travel, server queuing, application work, and response transfer. If a page issues dependent requests sequentially, each network round trip adds to the critical path. Concurrent independent requests can overlap, but they still compete for bandwidth and server capacity.
Split time across DNS, connection setup, TLS, edge wait, origin queueing, database, payload download, and browser render.
Client constraints
Browsers enforce CORS, cookie rules, connection limits, and mixed-content blocking; JavaScript cannot open raw TCP/UDP sockets.
Debug first questions
Is it slow for one region or all users? Are cache hits dropping? Did DNS or TLS change? Is the origin saturated?
The same click, drawn at search scale, shows how many hops sit between typing and a rendered page.
Request journey at search scale
Single request for a search page
Most products stay client-server. Peer-to-peer shows up when moving media between users is cheaper than shipping every byte through your fleet.
Client-server vs peer-to-peer
Client-server
Centralized servers control data and logic. Clients are consumers. This is the dominant model for web applications.
- Deploy server-side only so every client picks up fixes without an app store wait.
- Central authority for security and consistency keeps product rules in one place you control.
- Single point of failure if unreplicated, so you add copies before traffic peaks.
- Server costs scale with users, which is the trade for centralized control and easy deploys.
Peer-to-peer (P2P)
Every node is both client and server. No central authority. Used when high-bandwidth paths between users matter.
- More peers add capacity naturally, so bandwidth grows with the participant count.
- No single point of failure for media paths when peers can reach each other directly.
- Hard to enforce consistency or security without some client-server signaling and trust layer.
- Common in file sharing, calls, blockchain, and content-addressed peer networks.
for calls
Signaling (client-server): exchange connection metadata so peers can find each other.
Media/data (P2P): video, audio, and data channels can bypass the media server.
Why teams use it: can cut server bandwidth costs dramatically for video.
Limitation: traversal still needs STUN/TURN helpers when direct paths fail.
Why reusing a connection feels so much faster
Saying hello once and keeping the pipe warm beats paying the hello tax on every request, especially when latency is high or you make many calls.
Without reuse
Twenty separate cold calls can burn seconds just on setup if they cannot overlap well.
With keep-alive
The first call pays setup. The next ones mostly pay for server work and moving the payload.
With multiplexing
Many requests can share one encrypted connection at once, which is how modern pages stay efficient.
Your app vs the plumbing underneath
Your code talks in HTTP or gRPC meaning. Underneath, sockets, packets, and congestion control move the bytes. Keep those two stories separate and systems get easier to reason about.
Your app
- Methods, paths, bodies, status codes, auth
- Retries, timeouts, and business rules
- Choosing HTTP, WebSockets, gRPC, or custom framing
The transport
- IP routing, TCP or UDP behavior, congestion windows
- Buffers, resends, and whether loss pauses one stream or everything
- QUIC may live in userspace, but the idea is the same: meaning vs packet plumbing
Keep-alive, multiplexing, and pooling live in the reuse block above. Below is the classic connection-count story, then how fleets grow past one machine.
The / problem
OS limits and memory per socket become the bottleneck once you try to park millions of live TCP sessions on one machine. Capacity math for billion-user fan-out is in Numbers to know.
How does a single server handle millions of simultaneous connections? The evolution of connection management is a key story in scaling client-server systems.
(1999)
Can one server handle 10,000 connections? Thread-per-connection: 10K threads × 1MB stack = 10GB RAM, not practical. Event-driven I/O (select → poll → epoll/kqueue) became the workable answer.
(2010s)
Can one server handle 10 million connections? epoll scales, but kernel overhead becomes the bottleneck. Teams then try kernel bypass through tools such as DPDK, more efficient kernel I/O through io_uring, userspace networking, and lightweight processes (actor runtimes / BEAM).
| Technology | Concurrent connections | Notes |
|---|---|---|
| Event-driven reverse proxy | ~500K | Event-driven, C |
| Event-loop runtime | ~600K–1M | Event loop, JS |
| Go-like runtime | ~1M | goroutines, ~2KB each |
| Actor-based runtime / BEAM | ~2M | Large chat services: ~900K per server reported |
| Actor-based language | ~2M | Realtime frameworks: ~2M on a single server |
and ephemeral ports
Home and office NAT boxes map many internal clients to one public IP. Outbound TCP connections use source ports, while NAT devices also retain translation state. Port reuse and TIME_WAIT behavior depend on the endpoint tuple, connection lifecycle, and network implementation. A busy client behind NAT can exhaust ports long before the server does: design connection reuse and avoid chatty APIs.
ceilings
Each socket is a file descriptor. Default ulimit -n values (often 1024–65535) cap concurrent connections per process. Tune fs.file-max, somaxconn, and per-process limits; otherwise accept() fails even with free RAM.
Once one machine is no longer enough, you split presentation, logic, and storage, then optionally add mesh policy between many internal services.
How fleets grow past one box
3-tier, N-tier, and microservices
3-tier (classic)
Most web apps start here with a clean split of UI, logic, and storage.
N-tier (scale)
Each tier scales independently. This is how very large internet companies operate.
Microservices
Large streaming services can run 1000+ microservices, each owned by a team.
: control plane and
At large platform scale, individual microservices should not re-implement every networking concern. A mesh handles retries, , metrics, and traffic rules via sidecar proxies so polyglot services share policy.
Pod A
Pod B
Control plane manages config, certs, and policies. Sidecars typically handle mTLS, L7 load balancing, circuit breaking, retries, tracing, and canary traffic splits.
Protocols and compute placement
Prefer for browser and public APIs (JSON, CDN caching, curl-friendly). Prefer for typed high-throughput chatter between services you control (protobuf, HTTP/2 streams). fits bursty short work; runtimes run light auth, routing, or personalization near users. Heavy state stays regional. Use a from app to DB so you do not open a fresh TCP+TLS path per query (see reuse block above; headcount math in Numbers to know).
The figures below are the capacity and traffic shapes referenced earlier; use them when sizing connection tiers and fan-out.
Numbers to know
Connection and memory math
An open connection consumes resources even while idle. As an illustrative assumption, 100,000 connections at 20 KiB of combined application and buffer memory need about 1.9 GiB before shared process memory, queues, and kernel overhead. Measure the real per-connection cost rather than treating this assumption as a platform constant. Event-driven I/O avoids one blocked thread per socket, but it cannot remove socket state or the CPU required for active messages.
- Thread-per-connection: 10K clients × ~1 MB stack ≈ 10 GB RAM before application heaps (classic C10K cliff).
- At 1B users online you need ~1000 servers at about 1M connections per server.
- 1B users × 5% online = 50M concurrent connections; 50M / 1M per server ≈ 50 connection servers.
- Each connection ~10–15KB memory ≈ 500GB total RAM across that connection tier.
- 1000 app servers × 50 pooled DB connections = 50K DB sockets: use a database proxy/pooler.
- HTTP/1.1 keep-alive reuses TCP and saves ~1.5 RTT per later request; typical idle timeout 60–120s.
Edge and serverless latency
- Serverless cold starts often land in the 100ms–1s range and constrain sticky long sessions.
- Edge light work can drop latency from ~100ms toward single-digit milliseconds near users.
Real workloads at billion-user scale
| Workload | Daily requests | Servers | Architecture |
|---|---|---|---|
| Search | ~8.5B searches/day | Millions | Custom infra / cluster schedulers |
| Streaming | ~1B hours/week | 100K+ | Microservices on cloud platforms |
| Social | 10B+ API calls/day | Millions | Custom graph storage and RPC |
| Commerce | ~2.5B items viewed | 100K+ | Microservices |
| Chat | ~100B msgs/day | ~1000 | Actor runtimes (~900K conn/server) |
Close with the invariant, a sticky-session failure path, practice prompts, and the short takeaway.
Correctness invariant
What must never become false
Stateless request handling means that an instance does not own the only copy of information required by the next request. It does not mean the whole product has no state. Credentials, sessions, notes, and job status still exist; they live in stores with appropriate durability and consistency. A WebSocket server also holds connection state, so a reconnect must rebuild that state from an authenticated identity and a durable progress cursor.
Servers remain with respect to session truth: any healthy instance can answer a request given credentials and externalized state. Clients may cache and retry, but they must not invent authoritative business state the server never committed.
Failure timeline: sticky session server dies
T0 Login stores session only in App-3 memory; LB sticky → App-3 T1 App-3 crashes; stickiness still routes cookies to App-3 T2 Clients get 502/timeout; "logged in" UI lies until cookie cleared T3 New instances cannot read the in-memory session map T4 Externalize sessions (Redis/DB); any instance validates the token
hide statefulness until the sticky target disappears.
What to remember
At billion-user scale, client-server becomes a multi-layered, globally distributed system. Prefer stateless servers behind load balancers, with state externalized to caches and databases. Know the C10K / C10M story, use P2P selectively (WebRTC for video), and remember that edge and serverless push light compute closer to users, without changing the basic idea that clients ask and servers answer. Capacity details sit in Numbers to know.
Exercises
Exercise 1
Mobile clients retry POST /orders on timeout. Where does idempotency live: client UUID, gateway, or DB unique constraint, and what must never double-charge?
Exercise 2
Choose REST vs gRPC for browser storefront vs inventory microservice. One sentence each on cacheability and streaming.
Exercise 3
Push image resize to the edge. What stays on the origin, and what failure mode appears if the edge cache is wrong for signed URLs?
Try it: stateless rotation versus sticky sessions
Send traffic from two browser tabs represented as client-1 and client-2. Toggle routing, mark a server down, and watch whether the next request still succeeds or loses session-local state.
Server A
healthyAccepts new assignments.
Server B
healthyAccepts new assignments.
Server C
healthyAccepts new assignments.
- Requests sent
- 0
- Session failures
- 0
- client-1 pinned to
- none yet
- client-2 pinned to
- none yet
Stateless mode sends each request to the next healthy server. Sticky mode pins a client id to one server until that server is marked down.
Stateless servers push session data into Redis or signed cookies so any instance can continue the conversation. Sticky sessions are a shortcut that breaks the moment that instance disappears unless you also replicate session state.
Topic-specific references
- Dan Kegel: The C10K problem explains why thread-per-connection fails once sockets multiply.
- RFC 9110: HTTP Semantics defines the request/response contract clients and servers share.
- W3C WebRTC covers peer media when you skip a classic request/response middle.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Trace a request through name resolution, connection establishment, proxying, application work, state access, and response while accounting for resource limits.
Explain the decision
Why can a server with idle CPU still refuse clients?
Read the reasoning
File descriptors, listen backlog, ephemeral ports, connection memory, thread pools, or downstream pools can saturate before CPU.
Apply it to a production scenario
A deployment doubles connection attempts but request QPS is flat. What do you inspect?
Read one approach
Check keep-alive reuse, proxy resets, TLS handshakes, NAT/port exhaustion, accept queues, and whether clients reconnect in synchrony.
Work through the numbers
Estimate memory for 400,000 live connections at 24 KiB application-plus-kernel state each.
Read the worked calculation
400,000 × 24 KiB ≈ 9.2 GiB; add TLS buffers, allocator overhead, and safety headroom.
Reason through a failure
What must remain bounded during a reconnect storm?
Read the recovery reasoning
Concurrent handshakes, accepted sockets, per-client retry rate, and downstream work queues must stay within admitted capacity.
Sources and further study
Continue with Networking Fundamentals.