System Design Patterns
Service Discovery
You request an Uber ride. Pricing, matching, payments, and maps all talk to each other while pods scale up, restart, and move. Nobody can keep a static IP list for "the payment service right now."
is the live address book: register healthy instances, forget dead ones, and let callers find who can take traffic. Stale answers still happen, so timeouts and retries remain mandatory.
Find payment service
Publish address, zone, weight, and health so callers know who can serve.
TCP, HTTP, or gRPC probes keep dead instances out of the serving set.
Resolve via DNS, client watch, or mesh push as pods come and go.
Call + retry
Use timeouts and try the next instance when a cached answer is stale.
What actually happens when instances churn
Payment, matching, and maps pods scale, restart, and move hosts constantly. Callers need a live address book of who can take traffic right now. Hardcoding three payment IPs in config, or treating DNS as always-fresh truth with a 5-minute TTL and no client retry, fails the first rolling deploy: half the fleet still dials ghosts for minutes while checkout error rates climb.
A new instance should advertise readiness only after it can handle its intended requests, including required configuration and initialization. In Kubernetes, a normal Service usually resolves to a stable virtual IP while endpoint routing changes behind it; a headless Service can return individual pod addresses, making client DNS caching more directly relevant. Identify which path your application uses before blaming pod replacement on DNS TTL alone.
A graceful shutdown marks the instance unready or removes it from the serving set, waits for routing changes to propagate, and drains existing requests within a deadline. Existing keep-alive connections may continue after discovery changes, so the server or proxy must also stop accepting new work on those connections appropriately. A crashed instance cannot perform this sequence; clients need connection deadlines and bounded retries that respect operation idempotency.
Static lists rot. Long TTLs leave dead endpoints discoverable. A that refuses answers during a partition freezes every caller even when local last-known-good endpoints would still work. Discovery without timeouts and retries is not resilience.
How to choose discovery
Once you accept that addresses churn and caches lag, three questions decide how much machinery you need.
1. Is cluster DNS enough? For most systems, orchestrator Service + CoreDNS works. Add a when you need traffic splitting, mTLS, and rich observability without putting an SDK in every language.
2. Stale vs unavailable? Everyday discovery usually prefers an : always answer, tolerate brief staleness. Use a for leader election and correctness-critical config.
3. How fast must failover be? Lower DNS TTL, watch-based updates, and health-aware load balancing beat hoping clients refresh instantly.
Simple rule of thumb
Start with platform DNS and . Treat discovery as eventually consistent. Pair it with timeouts, retries to another instance, and graceful drain on deploy. Reach for mesh when policy, not just names, must be centralized.
From problem to mechanism
Here is the ride-hailing story again, slowed down so you can see where the naive address book fails and what a working discovery path actually guarantees.
Problem. Payment, matching, and maps pods scale, restart, and move hosts. Callers need a live address book of who can take traffic right now.
Naive system. Hardcode three payment IPs, or treat DNS as always-fresh with a 5-minute TTL and no client retry. A rolling deploy replaces all three IPs; half the fleet still dials ghosts for minutes.
Break it. Static lists rot. Long TTLs leave dead endpoints discoverable. A that refuses answers during a partition freezes every caller even when last-known-good lists would still serve.
Mechanism. Gate new traffic on a , publish healthy instances in a or cluster DNS, propagate via poll, watch, or , and drain with a . Prefer an for everyday lookups when stale-and-retry beats freeze-the-world. Callers still fail over with timeouts and retries.
Correctness invariant
No instance receives new serving traffic before its readiness probe passes. After drain or hard death, the instance must leave the serving set within a declared removal window, knowing DNS and client caches may lag that window. Everyday discovery may return briefly stale answers; callers must still fail over. CAP trade-offs for the registry matter during a partition: prefer AP for lookups when stale-and-retry beats freeze-the-world.
Discovery mechanisms in practice
The Kubernetes DNS documentation distinguishes normal Services, which return a service address, from headless Services, which return endpoint addresses. That distinction determines how much endpoint churn is exposed to DNS clients.
The mechanism names DNS, health probes, registries, and mesh proxies; this section shows what each looks like when you open the tools.
Container orchestrator DNS (CoreDNS)
- ClusterIP service
- payment-service.prod.svc.cluster.local → virtual IP balanced to endpoints.
- Headless service
- Same DNS name resolves to all pod IPs for StatefulSets or direct endpoint picks.
- Pod A records
- <pod-ip>.payment-service.prod.svc.cluster.local
CoreDNS commonly serves tens of thousands of queries/sec per cluster; client caches and TTL still matter for failover speed.
Service-discovery system DNS interface
Many registries expose DNS such as payment.service.consul resolving to healthy instance addresses and ports.
- Language-agnostic: any client that can resolve names works without a special SDK.
- Legacy apps keep calling the same hostname; no code change for basic discovery.
- DNS caching leaves stale entries after pods die, until TTL or watch catches up.
- Limited metadata; some clients ignore low TTLs and refresh slower than you expect.
TCP health check
Connect to port; success means the process accepted the TCP handshake.
Pros: Fast and low overhead for shallow liveness.
Cons: Process may accept TCP but not handle real requests.
Use: L4 load balancers, basic liveness
HTTP health check
GET /health → 200 OK = healthyReadiness should answer whether this instance can serve the traffic it will receive. It may include a necessary database connection or completed initialization, but checking every optional dependency can remove the entire fleet during a shared dependency outage. Keep liveness focused on failures a restart can fix and readiness focused on serving ability; otherwise a slow external database can trigger pointless restart loops.
Use: L7 LBs, orchestrator readiness
gRPC health check
- RPC
grpc.health.v1.Health/Checkreturns SERVING or NOT_SERVING. - Example:
Check("payment")→ SERVING; reporting subsystem → NOT_SERVING while draining.
Use: gRPC services, proxy/orchestrator gRPC probes
: CP vs AP
- Consensus-backed stores; during partition may refuse reads or writes.
- Guarantee: if you get a response, it is correct rather than stale.
- Risk: discovery unavailable during partition freezes callers entirely.
- Best for leader election, config, and correctness-critical coordination.
- Availability-first registries, common in large streaming platforms.
- During partition: returns potentially stale data rather than silence.
- Risk: routing to dead instances; handle with client retries.
- Best for discovery where a stale answer beats total silence.
Separate the registry’s authoritative state from the lookup path. A consensus-backed registry can preserve safe membership changes while clients continue using a cached endpoint list during a control-plane outage. That is different from making every registry write available in every partition. Stale addresses are often acceptable for ordinary routing with bounded retries, while lock ownership and exclusive leadership require stronger checks than a cached discovery answer.
discovery (proxy )
xDS API types
Flow: App → proxy → Service
App → localhost:port (proxy sidecar) → proxy routes to the correct service. The app does not know about discovery at all. Used widely for traffic policy, mTLS, and canary without per-language SDKs.
Zero-downtime deploys and numbers to know
Readiness and drain only help if you wire them into the deploy sequence. Here is the pod lifecycle discovery must follow, plus the timings that decide whether users notice.
Numbers to know
- Discovery propagation is often 1–5s after readiness or deregister updates land.
- Typical DNS cache TTL is 30–60s, so clients can keep dialing removed IPs.
- A drain cuts dropped requests during rolling deploys.
- Registry may learn a death in ~10s while a 30s TTL still serves the dead IP.
Closing the stale-cache gap
Discovery removal, client selection, and connection lifetime are three separate clocks. A registry may remove an endpoint at time t, a client may refresh at t+5 seconds, and an already-open HTTP/2 connection may survive longer still. Measure requests sent to draining endpoints and design the shutdown grace period around the observed propagation and request duration. A smaller DNS TTL only addresses the paths that actually resolve endpoint addresses through DNS.
When an instance dies, the registry can be right within about 10s while DNS still answers the dead IP for up to the TTL (see Numbers to know). Pair freshness with these mitigations.
Quantify load, then pick a tool
The timings above matter at fleet scale. Walk an endpoint lifecycle, estimate registry write and stale-route cost, then compare common implementations.
Endpoint lifecycle walkthrough
- Start instance and keep it out of discovery while caches, connections, and warm data load.
- Pass readiness, register endpoint with version, zone, weight, and health metadata.
- Receive traffic only after client caches or watch streams observe the new endpoint.
- On shutdown, mark draining, stop new traffic, finish in-flight requests, then deregister.
Worked example
80_000 instances × heartbeat every 10s
→ 8_000 registry writes/sec steady state
Deploy 5_000 pods; each update watched by 20 sidecars
→ 100_000 notifications if unbatched
→ batch + jitter or the control plane stampedes itself
Stale-route tax (DNS TTL 30s, 2_000 QPS to payment):
Worst window ≈ 30s × 2_000 = 60_000 attempts that may hit a dead IP
With retry-to-next (1 extra hop): ~60_000 extra connection attempts
Without retry: those become user-visible 5xx / timeoutsTTL tradeoff test
Lower TTL from 60s to 5s and confirm authoritative query load, client CPU, and stale-route errors stay inside budget.
Registry outage drill
During a registry outage, callers can retain a last-known-good endpoint list and continue contacting healthy instances. Track when that list was last refreshed, use passive connection failures to avoid repeatedly selecting bad addresses, and cap retry traffic. A stale list is useful only while some endpoints remain reachable; if a whole region or deployment has changed, the system needs an explicit failure response rather than indefinite retries against vanished instances.
Watch / runbook
Page on endpoints empty while replicas > 0, discovery cache age p99, and connect-timeout rate to recently removed IPs.
Implementation comparison
| Tool | Consensus | Health checks | Multi-DC | Used by |
|---|---|---|---|---|
| Container orchestrator | Coordination store (Raft) | Probes | Federation | Most cloud platforms |
| Service-discovery system | Raft | TCP/HTTP/gRPC | WAN gossip | Infrastructure vendor users |
| AP registry | AP (none) | Client heartbeat | Per-region | Large streaming platforms |
| Coordination store | Raft | Lease TTL | No (single) | Orchestrator control plane |
| Coordination service | ZAB | Ephemeral nodes | No (single) | Older distributed systems |
Orchestrator DNS is the most common path today: Service + CoreDNS needs no external registry for most apps. Multi-cluster meshes exist when you outgrow one cluster. For the vast majority of systems, Service + CoreDNS is enough. Add mesh or proxy when you need traffic splitting, mTLS, and deep observability.
Failure timeline, and what still breaks after the fix
Putting the pieces together, here is how a dead payment pod plays out second by second, what still fails after you add readiness and retries, and what you pay for each discovery style.
Timeline: pod dies, DNS still answers
t=0s payment-7 healthy; CoreDNS / client cache holds A records (TTL 30s) t=2s Node OOM-kills payment-7; TCP accepts stop t=12s Readiness/liveness fail; Endpoints controller removes pod from Service t=12-30s Clients with cached answers still dial payment-7 → connect timeout / 502 t=18s Without retry-to-next: checkout threads pile up waiting on dead IP t=31s TTL expires; fresh lookup omits payment-7; traffic shifts to live pods Outcome Registry was right at t=12s; user pain lasted until caches + retries caught up
The counterexample to "discovery is always fresh": control-plane removal can be seconds ahead of every client's cache. Pair low TTLs or watch/ push with health-aware load balancing and bounded retries.
Still fails when…
- Half-open clients ignore outlier ejection and keep retrying one bad host.
- A shorter than LB drain leaves streams on a dying pod.
- Mesh control-plane lag pushes stale EDS sets after a zone failure.
- Strictly CP registry lookups freeze discovery when last-known-good would serve.
- Deep readiness checks create circular outages (A waits on B; B waits on A).
Trade-offs
- Cluster DNS + readiness is simplest; client-side registry watch is fresher but SDK-heavy.
- xDS centralizes policy and EDS at sidecar and control-plane cost.
- Hard-coded VIP or external LB alone is weak for east-west instance churn.
- Heartbeat and watch load plus stale-cache windows are the ongoing operational tax.
- AP lookups trade brief wrong answers for availability during registry partitions.
Choosing and explaining service discovery
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
Service discovery is a dynamic address book for healthy service instances.
Choose client-side, server-side, DNS, or service mesh discovery based on latency, freshness, and operational control.
Stale entries are normal, so callers still need timeouts, retries, and load balancing.
When it belongs
Instances scale up and down frequently.
Hardcoded hostnames or IPs would break deployments.
Traffic must move away from unhealthy nodes quickly.
How it fails
Registry partition returns stale healthy instances.
Heartbeat delay keeps dead instances discoverable.
Client cache TTL is too long for failover.
Trade-offs to weigh
DNS is simple but coarse-grained.
Client-side discovery is fast but pushes logic to every client.
Service meshes centralize policy but add sidecar and control-plane cost.
Common misunderstandings
Assuming discovery health checks replace application health checks.
Ignoring startup thundering herds against the registry.
Forgetting regional isolation for service registries.
Try it: registry freshness versus client cache TTL
Service discovery keeps a live registry, but clients often cache lookups. Mark an instance unhealthy, advance time past the TTL, and compare a cached pick with a fresh registry read.
Registry (control plane)
- api-1 — 10.0.0.11:8080 — healthy
- api-2 — 10.0.0.12:8080 — healthy
Client cache snapshot
No cached endpoints yet.
Cache age: 0s / TTL 30s
- Healthy in registry
- 2
- Healthy in client cache
- —
- Routing to failed node
- No
The registry lists two healthy API instances. Refresh the client cache to snapshot what callers would use.
Production callers combine short TTLs, periodic refresh, and health-aware load balancing so a stale cache does not keep sending traffic to a node the registry already marked down.
Going deeper
Production deep dive: keep discovery off the request path
Discovery maps a stable service identity to a changing endpoint set. Clients should serve from a local snapshot and refresh asynchronously, so a registry incident degrades freshness rather than stopping every request.
What must stay true
No single registry lookup is required for an ordinary data-plane request, and every endpoint is authenticated as the logical service it claims to be.
How the pieces fit
Bootstrap seeds
DNS / static config
Registry quorum
authoritative membership
Watch / DNS answer
versioned snapshot
Client or proxy
local LB + health
mTLS endpoint
service identity
Watchers
20,000
Clients and sidecars watching one popular service.
Deploy churn
500 changes
Rolling replacement of a large fleet.
Naive fan-out
10M updates
500 × 20,000; coalesce snapshots and add jitter.
DNS caching and stale endpoints
TTL is a hint interpreted by recursive resolvers, OS caches, runtimes, and connection pools. Even after an address expires, an existing HTTP/2 connection may remain. Use readiness drain, connection max-age, retry to a different endpoint, and measured stale-endpoint rate.
Watch-storm control
Send versioned endpoint deltas or coalesced snapshots, resume from revisions, jitter reconnects, cap per-client streams, and fall back to the last known good set. A deploy should not cause every sidecar to reconnect simultaneously.
Registry failover
Use a quorum-backed control plane across failure domains. During loss of quorum, preserve cached reads and reject ambiguous registration writes. Test snapshot restore and define the maximum endpoint staleness the data plane will tolerate.
Client versus proxy load balancing
Client-side balancing avoids a hop and sees per-request semantics but multiplies library logic. Sidecar/proxy balancing centralizes policy and telemetry but adds resource cost and another failure mode. Both must eject unhealthy endpoints without synchronized thrashing.
Identity and bootstrap
Registry name is routing metadata, not proof of identity. Authenticate workload certificates against the expected service identity. New nodes start from signed static seeds or DNS, establish trust, fetch a full snapshot, then open a resumable watch.
Worked example: Staleness budget
- 1DNS TTL is 30 seconds, endpoint drain is 20 seconds, and clients keep pooled connections for 120 seconds.
- 2Changing TTL alone cannot move all traffic in 30 seconds: an already-open connection may survive four TTL windows.
- 3Set connection max-age near the failover target, stop readiness first, wait for drain, then remove the endpoint and keep cross-endpoint retry bounded.
Failover time is controlled by the slowest cache or live connection layer, not by the authoritative DNS TTL printed in the zone.
Discovery failure matrix
| Scenario | Consequence | How to contain it |
|---|---|---|
| Registry quorum lost | Membership cannot change safely. | Use cached endpoints for reads; freeze joins/removals and page the control plane. |
| Mass client reconnect | Registry CPU and egress collapse. | Exponential backoff with jitter, resumable revisions, and admission limits. |
| Stolen endpoint address | Traffic reaches the wrong workload. | mTLS authenticates service identity independent of IP registration. |
What to watch in production
- Measure registry quorum health, watch lag, reconnect rate, snapshot size, endpoint staleness, and failed identity handshakes.
- Chaos-test registry loss while clients continue from last-known-good snapshots.
- Canary endpoint drain and verify no client/runtime ignores intended connection rotation.
A scenario to reason through
A registry recovers after five minutes and 20,000 sidecars reconnect at once. How do you prevent a second outage?
One solid approach
Serve last-known-good data, require exponential backoff with full jitter, admit reconnects gradually, resume from revision when possible, and coalesce current membership into one snapshot rather than replaying every missed mutation.
Going deeper
Service discovery
The mechanism and its boundary
Discovery converts a logical service name into a healthy endpoint list. The hard part is freshness: clients need quick reaction to failures without turning every instance change into a global control-plane storm.
Discovery freshness loop
Instance registers
Registry watches
Client cache
Health refresh
Work through the sizing assumptions
When it belongs
Instances scale up and down frequently. Static host lists would make deploys and failover manual. Client-side or proxy-side load balancing needs current endpoint metadata.
Assumptions that matter
Endpoint churn is proportional to deploy velocity and autoscaling. Consumers cache discovery results and refresh asynchronously. Health checks include readiness, not only process liveness.
Easy mistakes
Registration proves that an instance announced itself, not that it can serve a request whose database or other dependencies are unavailable. A synchronous global registry lookup on every request turns a control-plane problem into an application-wide outage, so callers need a bounded local endpoint view. Clients that ignore refresh limits or readiness changes keep using retired instances even when the registry has correct information.
How it fails
Registry partition creates stale endpoint views. Mass restart produces watch fanout and reconnect storms. Over-eager eviction removes healthy nodes during latency spikes.
Trade-offs to name
Frequent refreshes shorten the time clients use removed endpoints, but increase registry reads and update fanout; choose a refresh policy from the tolerated stale-routing window. Client-side balancing can use local latency and failure observations, but every client implementation must correctly handle refresh, retries, and draining. Aggressive health checks detect failed instances sooner while also removing healthy instances during pauses, so combine readiness signals with bounded local endpoint retries.
Wrap-up
You have seen the churn problem, the mechanisms, and the stale-cache tax. Use these exercises to check the ideas stuck, then follow the references for primary docs.
Exercises
- Payment sees 4_000 QPS and DNS TTL is 20s. Estimate worst-case stale attempts after one pod dies at t=0 if the Endpoints update lands at t=8s. Would you lower TTL, add outlier ejection, or both, and why?
- Registry is partitioned from half the clients. Choose AP vs CP for everyday discovery vs for leader-election config, and name the user-visible failure if you pick wrong.
- Design a zero-downtime deploy: readiness failure until warm, preStop sleep vs terminationGracePeriodSeconds. Which metric pages if clients still hit draining pods?
Topic-specific references
- Kubernetes documentation: Services, Endpoints/EndpointSlices, readiness and liveness probes.
- Envoy Proxy documentation: EDS / xDS endpoint discovery and outlier detection.
- HashiCorp Consul documentation: service registry, health checks, and DNS interface.
Key takeaway
Discovery must propagate fast, prefer availability for everyday lookups, and coordinate with readiness and drain on deploy. DNS is the simplest starting point; mesh xDS is the richest when you need health-aware routing and policy. Always design for stale caches: discovery alone is not resilience.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Resolve healthy service instances through DNS or registry watches while handling stale caches, bootstrap, failover, watch storms, identity, and load balancing.
Explain the decision
Why does a 30-second DNS TTL not guarantee removal within 30 seconds?
Read the reasoning
Resolvers, clients, proxies, and connection pools cache independently; existing connections can outlive DNS records.
Apply it to a production scenario
The registry recovers and 50,000 clients reconnect their watch simultaneously. What protects it?
Read one approach
Snapshot plus revision resume, jittered reconnect, watch fan-out/cache tiers, quotas, and last-known-good endpoint sets.
Work through the numbers
Fifty thousand clients polling discovery every 10 seconds create what query load?
Read the worked calculation
5,000 QPS before retries; push watches reduce polling but add connection and fan-out state.
Reason through a failure
What must discovery during registry loss preserve?
Read the recovery reasoning
Clients use bounded last-known-good endpoints with health/outlier checks and cannot accept unauthenticated endpoint injection.
Sources and further study
Continue with Distributed Transactions.