System Design Networking

Proxies

What machine answers first when checkout hits api.shop.example? The phone thinks that hostname is the app. In reality the first machine it reaches is often a : a middle hop that terminates TLS, checks rate limits, and only then forwards the request to a private app server.

Proxies absorb spikes, hide backends, and enforce policy so product code does not reinvent networking for every service.

Checkout path

DNS → VIP

Client dials one public address so backends stay private.

Cache and DDoS filters cut load before origin.

TLS, route, compress, and rate-limit at the edge.

Private app servers

Business logic and payment APIs stay off the public internet.

What actually happens

Think of a proxy as a middle hop: it accepts a connection, then opens or reuses another toward the real backend (or the internet). A usually sits near users: corporate egress or VPN. A sits in front of your app tier. Interview diagrams almost always mean reverse proxies.

Large sites terminate TLS and absorb traffic spikes at proxy and layers before hits reach application code. Service meshes add next to each pod for observability and policy. You do not need exact vendor stats: know which concerns belong at the proxy versus in business logic. Capacity figures for pooling, cache hits, and hop cost live in Numbers to know.

Before drawing boxes, decide whether you need to read HTTP, whether traffic is public or internal, and whether you need API product features. The interview block later expands ownership and bottlenecks; this card is the short decision tree.

How to choose a proxy

1. Do you need to read HTTP? Path routing, header canaries, compression, , and rate limits need an . Raw TCP databases or opaque streams often prefer an .

2. Is this the public edge or east-west traffic? Put a reverse proxy or managed L7 LB in front of HTTP services. Use mesh for internal , retries, and tracing across many languages.

3. Do you need API product features? Auth schemes, versioning, and developer portals push you toward an . A plain reverse proxy may still sit in front of it.

Simple rule of thumb

Default: a (or managed L7 LB) in front of HTTP services: TCP + HTTPS to the edge, private backends behind it. Reach for raw L4 or UDP only when the workload is not HTTP or when you need minimal inspection latency.

Proxy timeouts, retries, and streaming

Timeout budget

Suppose a client will wait two seconds. An illustrative budget gives the edge 1.8 seconds, gateway 1.5 seconds, service 1.2 seconds, and database 800 milliseconds. These are nested limits, not durations to add together. Each caller reserves time to receive and handle a downstream failure. Propagate the remaining deadline so time spent waiting at an earlier hop is not granted again.

Streaming compatibility

A proxy that waits for a complete response can hide a stream until its buffer fills. For SSE and gRPC streaming, configure the route to forward chunks promptly and set suitable idle timeouts. WebSockets need supported upgrade or extended-CONNECT handling. Long polling needs a timeout longer than the intended wait; large request uploads have separate request-buffering controls.

Retry rule

Retry only idempotent requests or writes with idempotency keys. Retrying POST blindly can create duplicate orders, messages, or payments.

Where proxies sit in the stack

A terminating reverse proxy creates two separate conversations: client to proxy and proxy to origin. It can reuse a pool of origin connections even when many clients connect separately. This makes TLS policy, header handling, buffering, and timeout behavior explicit design decisions at that boundary. If the proxy only passes encrypted traffic through, it cannot inspect HTTP paths or bodies without terminating that encryption.

Proxies are not a separate OSI layer. They terminate or forward traffic at L4 (TCP/UDP) or L7 (HTTP/gRPC), often after DNS pointed the client at a public . Clients dial that VIP while backends keep private IPs, so you can swap machines without teaching every client a new address.

An routes by IP and port: fast and opaque to HTTP, useful for raw TCP, databases, or when TLS is terminated elsewhere. An speaks HTTP(S): path routing, header-based canaries, compression, , rate limits (including 429), and .

Client

DNS → public VIP

Reverse proxy / LB

L4: TCP/UDP pipe

Private app servers

L7: HTTP route / TLS / cache

Private app servers

A simple web request, end to end

One browser click feels like a single GET. Underneath, it is a short story: find the address, say hello, exchange the HTTP request and response, then say goodbye. Many packets hide behind one conceptual request.

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

sequenceDiagram
    autonumber
    participant Browser
    participant DNS
    participant Server

    Browser->>DNS: Resolve example.com
    DNS-->>Browser: Destination IP
    Browser->>Server: SYN
    Server-->>Browser: SYN-ACK
    Browser->>Server: ACK (TCP ready)
    Browser->>Server: HTTP request (GET / path, headers)
    Server-->>Browser: HTTP response (status, headers, body)
    Browser->>Server: FIN
    Server-->>Browser: ACK
    Server->>Browser: FIN
    Browser-->>Server: ACK (connection closed)

What your app sees

An ordered stream of bytes. TCP handles missing pieces and reordering so you can focus on request and response logic.

One click, many packets

DNS, the TCP hello, HTTP pieces, acknowledgements, and goodbye steps can easily become dozens of packets.

Why connections cost

Each live session uses memory and CPU. A fresh hello adds wait time before useful data. Idle connections still hold resources.

Reuse when you can

Keep-alive reuses the same pipe for later requests. HTTP/2 can share many requests on one connection instead of opening several.

With the hop story in mind, the next distinction is whose side the proxy sits on: the user leaving a network, or your services facing the internet.

Forward vs reverse proxy

Client──→Forward proxy──→Server

Server sees proxy IP, not client

Corporate firewalls filter egress so policy lives outside every laptop.
VPN and anonymization hide client IPs when destination must not see users.
Caching repeated requests cuts WAN traffic for shared office or campus clients.
Geo bypass is a product of path choice, not something backends should invent.

Client──→Reverse proxy──→Server 1 / 2 / 3

Client sees proxy, not backends

Load balancing spreads traffic so one backend failure does not take the site down.
centralizes certificates so apps skip crypto setup on every box.
Static file serving and caching keep hot assets off application CPU paths.
DDoS protection and rate limiting absorb abuse before it reaches product code.

Reverse proxies shine at cross-cutting network policy; product authorization and complex transforms belong in services. Timeouts, retries, streaming buffers, and header trust are the usual footguns (also covered in the timeout block above and the interview block next).

How to choose proxies and gateways

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

sequenceDiagram
    participant Client
    participant Proxy as Reverse proxy
    participant Gateway as API gateway
    participant Service

    Client->>Proxy: HTTPS request
    Proxy->>Proxy: TLS, compression, request ID
    Proxy->>Gateway: Forward sanitized headers
    Gateway->>Gateway: Auth, quota, routing
    Gateway->>Service: Call upstream with timeout
    Service-->>Gateway: Response or error
    Gateway-->>Proxy: Status and trace context
    Proxy-->>Client: Response

Owns

TLS termination, routing, health checks, compression, request IDs, coarse auth checks, quotas, and safe retries.

Does not own

Deep product authorization, complex business transformations, or unbounded buffering for streaming and large uploads.

Finding bottlenecks

Compare edge time, gateway time, upstream time, payload download, retry count, timeout class, and connection reuse per hop.

Header hygiene

Trust forwarded client IP and trace headers only after the first trusted proxy has sanitized them.

Simple rule of thumb

Put shared network policy at the edge, keep product authorization in services, and measure each hop so a proxy never becomes an invisible timeout black box.

Once you know what a proxy owns, pick among mature families by static-file edge work, raw TCP performance, or programmable mesh features.

Comparing common proxy families

Feature	Event-driven reverse proxy	High-performance LB	Service proxy
Origin	Igor Sysoev (2004)	Willy Tarreau (2000)	Lyft / Envoy era (2016)
Primary use	Web server + reverse proxy	L4/L7 load balancer	Service mesh sidecar
gRPC support	Module (since 1.13)	Basic	Native, first-class
Config reload	File-based, graceful reload	File-based, seamless	xDS API (dynamic, no restart)
Observability	Access logs, basic metrics	Detailed stats dashboard	Built-in metrics, tracing, logging
Best for	Edge proxy, static files	High-performance TCP/HTTP LB	Kubernetes / service mesh

Declarative config shows the same ideas in practice: upstream pools, timeouts, TLS at listen, and path fan-out.

Reverse proxy config at scale

Terminal

worker_processes auto;  // 1 per CPU core
worker_connections 65535;
 
http {
  upstream backend {
    least_conn;
    server app1:8080 weight=3;
    server app2:8080 weight=3;
    server app3:8080 weight=1;
    keepalive 256;  // persistent backend connections
  }
  server {
    listen 443 ssl http2;
    server_name api.example.com;
    // SSL termination
    ssl_certificate /etc/ssl/cert.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    // Compression (60-80% size reduction)
    gzip on;
    // Rate limiting (100 req/sec per IP)
    limit_req_zone $binary_remote_addr zone=api:10m rate=100r/s;
    location /api/ {
      proxy_pass http://backend;
      proxy_connect_timeout 5s;
      proxy_read_timeout 30s;
    }
    location /static/ {
      root /var/www;
      expires 1y;
      add_header Cache-Control "public, immutable";
    }
  }
}

At fleet scale, many teams push shared east-west policy into a data-plane proxy beside each app instead of rewriting every language's HTTP client.

proxy and

Pod with sidecar

App container──→Service proxy (sidecar)←── all traffic through sidecar

App talks to localhost; sidecar talks to the world for policy and observability.

Sidecar handles transparently:

encrypts service calls so identity is not left to app code.
Circuit breaking sheds failing dependencies before queues and retries explode.
Retry with exponential backoff recovers brief blips without flooding upstreams.
Load balancing (round-robin, least-conn, ring-hash) spreads east-west traffic more evenly.
Distributed tracing injects headers so one request is visible across hops.
Metrics export latency and errors so SREs see mesh health, not only app logs.
Traffic splitting (canary 5% to v2) rolls out safely without client changes.
Fault injection supports chaos tests without rewriting networking in every service.

Your app code knows little about networking. Common in large polyglot service fleets.

Edge crypto and API product features often share a reverse-proxy foundation; decide whether you only need TLS and LB, or also auth portals and quotas.

and API gateway vs reverse proxy

SSL/TLS termination

Client──[TLS]──→Proxy──[plain HTTP]──→Server 1/2

TLS handled once so N backends skip repeating expensive handshake CPU.
Centralized certificate management keeps rotation off every application host.
Backend freed from crypto CPU so product work gets the cores.
Internal plain HTTP still needs or a private VPC boundary.

API gateway vs reverse proxy

SSL termination, LB, compression, basic rate limiting, static files. Operates at network/transport and HTTP edge levels.

All of the above plus authentication (JWT, OAuth, keys), API versioning and transformation, advanced rate limiting per user, analytics, developer portal.

Use both when needed: proxy at the edge, gateway for API management.

Caching and keep origin load down and route around bad instances; hit-rate and check cadence sit in Numbers to know.

Proxy caching and health checks

Readiness asks whether an instance should receive new traffic; liveness asks whether restarting it may help. If readiness depends on every shared downstream service, a database incident can make all application instances disappear from the load balancer at once. Choose checks that reflect useful serving ability and combine them with observed failures. During shutdown, stop accepting new work, let bounded in-flight work finish, and provide a separate policy for streams that may otherwise live indefinitely.

Terminal

// Proxy Caching (event-driven reverse proxy example):
proxy_cache_path /var/cache levels=1:2 keys_zone=api_cache:100m max_size=10g;
 
location /api/products {
  proxy_cache api_cache;
  proxy_cache_valid 200 5m;  // cache 200s for 5 min
  proxy_cache_valid 404 1m;  // cache 404s for 1 min
  add_header X-Cache-Status $upstream_cache_status;
}

Active checks hit /health on a schedule; repeated failures mark a server unhealthy and successes bring it back. A backup server line (server app3:8080 backup;) only takes traffic when primaries fail. Draining stops new requests while in-flight work finishes so deploys drop zero connections.

Closer to users, CDNs act as the first reverse-proxy hop; preserve client IP across the chain for rate limits, geo, and audit logs.

Edge proxy, CDN as proxy, and PROXY protocol

Edge proxy

First proxy layer users hit. Runs at edge locations. Handles TLS termination, DDoS mitigation, bot detection, rules, and basic auth. Edge caches often serve most video traffic without hitting origin.

CDN as reverse proxy

Modern CDNs act as programmable reverse proxies, not just static file caches. They run custom logic at the edge, cache API responses, and route dynamically based on headers, geography, or cookies.

When requests pass through multiple proxies, the original client IP can be lost. PROXY protocol prepends client IP info to the TCP stream. does the same at L7. Trust details and spoof failure modes are in Wrap-up below; do not restate them at every hop.

Production drawings usually compose managed edge LB, gateway filters, and path-based target groups; pooling savings are in Numbers to know, not restated here.

Real proxy architectures

Edge gateway path

Internet→Managed edge LB→Edge gateway→Microservices

Pre-filtersauth, rate limiting, CORS so abuse dies early

Routingservice discovery so paths map to live targets

Post-filtersmetrics and response headers for ops visibility

Error filtersfallback responses when upstreams fail cleanly

Handles external API traffic for large streaming services (billions of API requests/day). Async gateways can hold tens of thousands of concurrent connections per instance.

Managed application load balancer

/api/*→ API target group

/static/*→ object storage

/ws/*→ WebSocket target group (sticky)

Features: path-based routing, gRPC support, integration, identity-provider auth, weighted canaries. Auto-scales.

at the proxy layer

Reuse upstream TCP+TLS instead of handshaking per request. Exact ~50ms vs ~1ms figures and the 20K QPS handshake math live in Numbers to know.

Multiple hops (CDN → gateway → mesh sidecar) compose features but require tracing and header hygiene; each hop's latency budget is summarized below.

Proxy chains at scale

Timeouts need to shrink along the request path. If a browser waits one second while an inner proxy retries a two-second origin timeout several times, the system continues expensive work after the user has already left. Propagate a request budget and coordinate retries across layers. Only retry writes when their application semantics tolerate duplication; a proxy cannot infer that a timeout means the origin performed no side effect.

Client

↓CDN edge proxyCaching, DDoS protection, edge compute near users

↓Global load balancer (DNS-based)Geo-routing and failover between regions

↓Regional L4 load balancerTCP load balancing plus health checks

↓Reverse proxy (L7)SSL termination, compression, rate limiting

↓API gatewayAuth, versioning, request transformation

↓Service mesh sidecarmTLS, circuit breaking, observability

↓Application serverBusiness logic

↓DB proxy / poolerConnection pooling and read/write splitting

↓DatabaseSource of truth

Numbers to know

These capacity figures appear once here. Earlier sections should cross-reference this block instead of restating rates.

: ~50ms new TCP+TLS vs ~1ms reuse; at 20K QPS saves ~1,000 CPU-sec/sec of handshakes.
Proxy cache at ~90% hit rate and 1B req/day leaves only ~100M hitting backends.
Active every ~5s: 3 failures mark unhealthy; 2 successes restore.
Each hop may add ~0.1-1ms; compression at high QPS can save terabytes daily.
Gzip often cuts JSON ~60-80%; sample edge rate limit is 100 req/sec per IP.

Close with the trust boundary, a spoof timeline, drills, references, and one takeaway so the lesson sticks without another stack of cards.

Wrap-up

Proxies see headers and bodies, but they must not invent authorization from client-supplied fields alone. Use this card for the invariant, a failure story, exercises, references, and the interview takeaway.

Correctness invariant

Forwarded client-address headers are ordinary text unless a trusted hop establishes them. The public edge should remove or overwrite untrusted values and append verified connection information according to a documented convention. The origin should accept that convention only from known proxies. Otherwise a caller can impersonate another address to influence rate limits or audit records. The same trust-boundary reasoning applies to user and tenant headers.

Identity and client IP used for authz or rate limits come only from the first trusted hop after sanitizing / , never from a spoofable client header alone. Proxies may terminate TLS and route, but they must not invent authorization the origin would reject. Safe edge work: terminate TLS, rate-limit with 429, sanitize forwarded headers, and pass verified auth context (or require services to re-validate tokens). Never authorize from a userId in the body or a spoofable header; derive the caller from verified session/JWT/mTLS identity after the trusted proxy boundary. L4 proxies cannot rewrite HTTP auth; L7 proxies can: treat that power as a security boundary.

Failure timeline: trusted header spoof

T0  App trusts X-User-Id from any incoming request
T1  Attacker bypasses CDN path (direct origin IP) with X-User-Id: admin
T2  Rate limiter keys on spoofed X-Forwarded-For → victim IP blocked
T3  Authz grants admin actions; audit log blames the wrong client
T4  Only edge strips/sets forwarded headers; origin requires mesh mTLS

A reverse proxy is a trust boundary. Misconfigured trust turns helper headers into an auth bypass.

Exercises

Draw the hop list for browser → CDN → L7 LB → API gateway → sidecar → app. At which hop do you strip client XFF and set the real client IP?
WebSocket upgrades die after 60s idle behind your proxy. Which timeouts and heartbeat settings must align, and who owns them?
Choose nginx vs Envoy sidecar for mTLS between 40 services. What do you gain and what operational cost do you accept?

Try it: trace a request through forward versus reverse proxies

Step one HTTP request through each topology. Toggle a broken trust configuration on the reverse path to see why origins must treat forwarding headers as edge-controlled data.

Browser

pending

Calls https://api.example.com

Reverse proxy / LB

pending

TLS termination, routing, rate limits

App server pool

pending

Business logic on private network

Origin response

pending

Returns through the proxy to the client

Advance the request hop by hop. Forward proxies sit with clients; reverse proxies sit in front of servers.

Topic-specific references

HAProxy: The PROXY protocol : preserving client addresses across L4 hops.
RFC 7239: Forwarded HTTP Extension : standardized forwarding metadata (vs ad-hoc XFF).
Envoy: What is Envoy : L7 proxy / mesh data plane used in modern sidecars.

Key takeaway

In every system design, include a for SSL termination, compression, and load balancing. For microservices, mention an for auth and rate limiting when you need API product features. For internal service communication, mention a with sidecars for and observability. At billion-user scale, requests often pass through several proxy layers, each handling a specific concern; cite capacity from Numbers to know.

Practice and further reading

Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.

Focus for this lesson: Place forward and reverse proxies deliberately, define trust boundaries for client identity, and account for buffering, retries, streaming, and failure ownership.

Explain the decision

Why is X-Forwarded-For unsafe by default?

Read the reasoning

A client can forge it unless each trusted proxy strips untrusted values and appends its own; services must trust only a configured proxy chain.

Apply it to a production scenario

A proxy retry causes duplicate order creation. Where is the bug?

Read one approach

Retry ownership and method safety were undefined; disable retries for unsafe operations or combine explicit retry policy with end-to-end idempotency.

Work through the numbers

A proxy buffers 2 MiB uploads for 5,000 concurrent requests. Estimate buffer demand.

Read the worked calculation

About 10 GiB before metadata and headroom; streaming or disk spill may be required, with backpressure limits.

Reason through a failure

What must stay correct when bypassing a failed proxy tier?

Read the recovery reasoning

Authentication context, identity headers, rate limits, routing, TLS policy, and observability cannot silently disappear on the fallback path.

Sources and further study

Continue with GraphQL.

Proxies

What actually happens

How to choose a proxy

Simple rule of thumb

Proxy timeouts, retries, and streaming

Timeout budget

Streaming compatibility

Retry rule

Where proxies sit in the stack

A simple web request, end to end

What your app sees

One click, many packets

Why connections cost

Reuse when you can

Forward vs reverse proxy

Forward proxy

Reverse proxy

How to choose proxies and gateways

Owns

Does not own

Finding bottlenecks

Header hygiene

Simple rule of thumb

Comparing common proxy families

Reverse proxy config at scale

Sidecar proxy and service mesh

Pod with sidecar

TLS termination and API gateway vs reverse proxy

SSL/TLS termination

API gateway vs reverse proxy

Reverse proxy

API gateway

Proxy caching and health checks

Edge proxy, CDN as proxy, and PROXY protocol

Edge proxy

CDN as reverse proxy

PROXY protocol

Real proxy architectures

Edge gateway path

Managed application load balancer

Connection pooling at the proxy layer

Proxy chains at scale

Numbers to know

Wrap-up

Correctness invariant

Failure timeline: trusted header spoof

Exercises

Try it: trace a request through forward versus reverse proxies

Browser

Reverse proxy / LB

App server pool

Origin response

Topic-specific references

Key takeaway

Practice and further reading

Explain the decision

Apply it to a production scenario

Work through the numbers

Reason through a failure

Sources and further study

proxy and

and API gateway vs reverse proxy

at the proxy layer