Load Balancing

What actually happens

How to choose a load balancer

Vertical vs horizontal scaling

Before going deeper into balancer types, it helps to see why a balancer appears at all: it is the price of scaling out instead of up.

Load balancing at every tier

Once you scale out, a single request rarely meets just one balancer. It crosses several hops, and each hop can balance in a different way.

The table maps the same idea onto the hops a typical request crosses inside one region, from the edge down to the database.

Client-side vs dedicated load balancing

The last two tiers above (mesh sidecars and app-level clients) are where the caller picks a backend itself. That is the client-side option from the first choosing question; here is how it compares with a dedicated balancer.

Decision tree: when is client-side balancing reasonable?

The same trade-offs in flowchart form. Each "No" sends you back to a dedicated balancer; only a full run of "Yes" answers earns the client-side path.

Layer 4 vs layer 7 load balancers

Whichever side does the picking, the balancer works at one of two layers of the network stack, and that choice (question two above) decides what it can see.

For orientation, L3 (the IP layer below both) is where ECMP (equal-cost multipath, spreading packets across equivalent routes) and anycast live; they give packet-path diversity with almost no application context. Typical L4 examples are managed L4 services, TCP-mode proxies, bare-metal L4 services, and many game and VPN front doors. Typical L7 examples are managed L7 services, HTTP ingress proxies, reverse proxies with virtual hosts, and edge HTTP routing.

An L4 TCP proxy uses two connections; packet forwarding is another design

The diagram shows a TCP proxy: it accepts the client connection and opens a separate upstream connection, copying opaque payloads between them. TLS can still terminate at the backend. A packet-based L4 design instead uses forwarding, address translation, or tunneling; direct server return can keep the response path away from the balancer. Do not infer two TCP handshakes from the label L4 alone.

sequenceDiagram
    autonumber
    participant C as Client
    participant LB as L4 TCP proxy
    participant O as Origin

    rect rgb(30, 41, 59)
    Note over C, LB: TCP session A client to LB
    C->>LB: SYN
    LB-->>C: SYN-ACK
    C->>LB: ACK
    end

    rect rgb(30, 58, 50)
    Note over LB, O: TCP session B LB to origin
    LB->>O: SYN
    O-->>LB: SYN-ACK
    LB->>O: ACK
    end

    Note over C, O: LB splices bytes and does not parse HTTP
    C->>LB: TCP payload TLS or HTTP bytes
    LB->>O: Forward same payload
    O-->>LB: Response bytes
    LB-->>C: Response bytes

    rect rgb(55, 48, 40)
    Note over C, O: Teardown FIN and ACK on both legs
    C->>LB: FIN
    LB-->>C: ACK
    LB->>O: FIN
    O-->>LB: ACK
    end

L7 flow: terminate client HTTP, proxy to backends

The client speaks to the LB as the HTTP server. The LB chooses a backend using path, host, or weights, then acts as an HTTP client on the second hop.

sequenceDiagram
    autonumber
    participant C as Client
    participant LB as L7 ALB style LB
    participant O as Origin

    rect rgb(30, 41, 59)
    Note over C, LB: Client terminates HTTP and TLS at LB
    C->>LB: SYN
    LB-->>C: SYN-ACK
    C->>LB: ACK
    C->>LB: HTTP GET path and headers visible to LB
    LB-->>C: HTTP response to client may differ from origin
    end

    rect rgb(30, 58, 50)
    Note over LB, O: Separate TCP to chosen origin
    LB->>O: SYN
    O-->>LB: SYN-ACK
    LB->>O: ACK
    LB->>O: New HTTP request possibly rewritten
    O-->>LB: HTTP response from origin
    end

    Note over LB: Routing uses URL Host cookies and weights

Three questions come up as soon as both layers are on the table.

For a concrete L7 example, Envoy models HTTP requests as streams independently of the downstream wire protocol and manages routing and upstream proxying at that level. See its HTTP connection management documentation.

Numbers to know: million-user load math

Layers and algorithms only matter once the pool is sized. Every capacity figure used on this page lives in this one table; later sections refer back here instead of repeating them.

The retry row is the one that surprises people, so here it is as a flow: the same incoming traffic, tripled by two well-meaning retries.

Retry storms and gray failures

Removing a slow backend shifts its traffic onto the survivors. If they are already near saturation, aggressive ejection can produce a cascade in which each newly overloaded instance also fails health checks. Combine ejection limits with admission control and spare capacity. Retry only within a total request budget, and measure backend attempts separately from incoming user requests so retry amplification remains visible.

Managed load balancer offerings

Cloud vendors package the L4 and L7 shapes above under different product names. The names change; the layer each one works at does not.

GSLB and DNS-based balancing

The last row of that table, DNS and edge routing, is the first hop from the tier list: deciding which region a user reaches at all.

GSLB picks a datacenter per query using latency, geography, weighted splits, or failover rules. Anycast advertises the same IP from many POPs (points of presence, the edge sites) so BGP (the routing protocol between networks) sends users to the closest surviving edge automatically.

Algorithms (how picks are made)

The scheduling unit matters as much as the algorithm. Round robin per request can distribute small independent HTTP calls evenly, while round robin per connection may put thousands of HTTP/2 streams on one backend. Least connections helps when connection lifetimes differ, but one quiet socket and one busy video stream are not equal work. Choose a load signal that tracks the scarce resource and validate it under the actual request mix.

Every tier so far answers "which balancer". Inside any of them, an algorithm answers "which backend gets the next unit of work".

The same families in table form, with the interview fit for each.

How to choose an LB algorithm

With the families in view, the pick usually comes down to a short mapping and four questions about your traffic.

WebSockets, SSE, and hour-long connections

New capacity only helps new placements unless the system can reconnect existing clients. If one gateway holds most active sockets, adding five empty gateways does not migrate those sockets automatically. During deployment, stop assigning new connections to the old gateway, let short work finish, then ask long-lived clients to reconnect with randomized delay and a resume token. The durable event history must survive that gateway’s removal.

Two of the questions above (long-lived connections and affinity) dominate one workload in particular, so it gets its own section.

• Least connections or bounded loads when opening new upgrades, so busy nodes receive fewer long-lived sockets.
• Layer choice follows the L4 vs L7 section above: L7 only when you need path-based Upgrade routing.
• Consistent hashing on user or room id when you need memory-local fan-out, staying aware of hot keys.
• Drain plus connection age: stop sending new sockets to an old version while existing sockets age out.
• Use a publish-only bus or stream for cross-node events when clients on different machines must chat.
• Align idle timeouts across LB, proxy, and app keepalives so intermediaries do not silently half-close you.
• SSE heartbeats matter for the same reason: a quiet stream looks idle to every timeout in the path.

Health checks (deep dive)

A health check is a delayed observation, not an immediate guarantee. With a five-second interval and three failed checks required, a failed backend can receive traffic for several seconds before removal. Requests already assigned to it need their own timeout and recovery behavior. Passive error measurements can react to failures that a lightweight health endpoint misses, while hysteresis prevents a briefly recovered node from repeatedly entering and leaving service.

Every algorithm above assumes the candidate list is already trustworthy. Health checks build that list, and the details decide how long a sick node keeps receiving traffic.

Health signal hierarchy

The checks above sit on a ladder: each rung proves more about the real traffic path and carries its own risk.

Stickiness and draining

Health checks remove nodes that fail. Deploys remove nodes on purpose, and how gracefully that happens depends on two settings.

Blue-green and canary via the LB

Draining makes it safe to take one node out. Blue-green and canary deploys use the same traffic-shifting control to take a whole version out.

LB high availability

Everything so far assumes the balancer itself stays up. It is also a machine, so it needs the same redundancy it gives your backends.

Reference throughput (orders of magnitude)

With the shapes of a highly available tier covered, here is roughly how much each shape can move. Treat these as talk-track comparisons, not benchmarks.

Failure timeline: gray node plus retry storm

t=0      Pool healthy: 1.2K QPS across 13 replicas (~92 QPS each)
t=10s    One replica: TCP probes green, p99 → 3s, 30% 5xx
t=15s    Round-robin still feeds it; clients timeout and retry ×2
t=20s    Attempted QPS → ~3.6K; healthy replicas saturate
t=25s    Without outlier ejection: cascade to pool-wide timeouts
t=30s    With passive ejection + retry budget: sick node drains; load sheds
Outcome  Health checks alone are not enough: bound retries and eject gray failures

Wrap-up