System Design Reliability
Heartbeat mechanisms
Mid-ride, Uber's matching service stops answering. The load balancer has a choice: keep sending traffic and hope, or declare the node dead and move riders elsewhere. That choice is , and it starts with a : a small repeated "I am alive" signal.
At scale, the danger is not only missing a real failure. Declaring a healthy node dead during a GC pause (when the runtime stops the process briefly to reclaim memory) or a brief network blip can cascade into restart storms and : two sides both thinking they own the work.
Detection journey
Push, pull, or gossip carries a small “I am alive” pulse to monitors.
Missed beats start a soft clock so brief loss is not instant death.
Peers cross-check the silent node before anyone declares it failed.
Traffic moves away; stale owners lose write rights via fencing.
What actually happens
The matching-service story above is the everyday job of . A node, pod agent, or durable-log broker sends a small periodic . A monitor records the last arrival time. When silence stretches past the budget (interval times missed count, plus latency and pause buffers), the system marks the peer with or declares it dead.
Orchestrators separate from . Liveness asks whether the process is still running. Readiness asks whether this instance should receive user traffic. Mixing them so that every readiness probe hits a shared database can remove the whole fleet when that database hiccups.
For coordination, prefer a renewed by keep-alives. When the lease expires, ephemeral keys disappear and watches fire. Pair that with a so a paused old owner cannot resume and write, which is how you stop after a false death declaration.
How to choose a heartbeat style
Once you know you need detection, three questions decide push vs pull, fixed vs adaptive, and how hard you act on a miss.
1. Who owns timing? Push let the node control cadence. Pull health checks let the monitor decide when to probe. Gossip spreads membership without a central monitor.
2. How noisy is the path? Fixed timeouts work on quiet LANs. Cross-region paths need adaptive scores like or larger buffers (see Numbers to know below).
3. What happens on a false positive? If declaring death moves traffic, restarts pods, or promotes a replica, you need , quorum agreement, and a , not one missed ping.
Simple rule of thumb
Detect fast enough that users notice recovery, not the outage, but treat heartbeats as suspicion, not absolute truth. Separate shallow from , size timeouts with pause and p99 latency budgets, and fence any storage an old owner can still reach. In simple terms: miss a few beats before you act, and never let a "dead" node keep write authority.
From problem to mechanism: who carries the pulse?
The choice questions above pick a delivery style. Here is what push, pull, and gossip each change about state, traffic, and who decides a peer is silent.
Push
Node to monitor: "I am alive" every N seconds. The monitor tracks last_heartbeat. Simple, but it holds O(N) state.
Pull
Monitor to node: GET /health. Orchestrator probes and load balancers use this. Watch for many pollers overwhelming a node.
Gossip
Peers exchange state with random neighbors; no central monitor. Convergence is O(log N); detection can be slower than a probe.
Detection playbook: adaptive scores and timeout sizing
Style alone does not decide when silence means death. Fixed timeouts that feel right on a LAN become aggressive across data centers, so many systems use or an explicit buffer recipe instead of a hard cliff.
Phi accrual: suspicion as a score
keeps a sliding window of inter-arrival times and computes φ = −log₁₀(P(timeSinceLastHeartbeat)). Higher φ means rarer silence under normal conditions.
φ = -log₁₀(P(timeSinceLastHeartbeat))| φ value | Chance of being wrong | Interpretation |
|---|---|---|
| 1 | 10% | Node probably alive |
| 3 | 0.1% | Likely dead |
| 8 (default) | 0.000001% | Almost certainly dead |
| 12 (cross-DC) | ≈0 | Definitely dead |
A phi detector estimates the tail probability that the next heartbeat arrives later than the current silence interval, then takes its negative base-10 logarithm. A score of eight corresponds to an estimated tail probability of 10^-8 under that model, not the posterior probability that the machine is alive or dead. Sample-window size, distribution assumptions, and changing network conditions affect calibration; test false suspicions with realistic pauses and packet delay.
Timeout sizing recipe
When you stay with fixed timeouts, size them from interval, miss count, and the pause and latency budgets you already measured, then pick aggressiveness against the trade-off table below.
Sizing recipe
With a one-second heartbeat interval and a timeout of three seconds since the last accepted heartbeat, a crash just after a heartbeat is detected roughly three seconds later; one just before the next expected heartbeat may be detected sooner relative to the crash. Add scheduling, network, and processing delays when estimating the detection range. Use monotonic elapsed time for local intervals and treat the resulting value as an operational target, not proof of failure.
timeout =
interval × missed_count
+ p99.9 network latency
+ process pause budget
+ control-plane queue delay
+ jitter margin
Example:
interval 1s, missed_count 3
p99.9 latency 150ms, pause 500ms
queue delay 200ms, jitter 150ms
timeout ≈ 4sTrade-off
Short timeouts fail over quickly but false-positive during latency spikes and process pauses. Long timeouts protect against flapping but leave users on a bad node longer.
| Interval | Missed count | Detection before buffers | Traffic for 100K nodes | Use when |
|---|---|---|---|---|
| 500 ms | 3 | 1.5 s | 200K heartbeats/sec | Small LAN control plane |
| 1 s | 3 | 3 s | 100K heartbeats/sec | Common service discovery |
| 5 s | 3 | 15 s | 20K heartbeats/sec | Noisy network or non-critical workers |
| Choice | Failover time | False-positive risk | When it hurts |
|---|---|---|---|
| Aggressive (1s × 2) | ~2s | High during GC, jitter, brief loss | Flapping, restart storms, split-brain if unfenced |
| Balanced (1s × 3 + buffers) | ~4s | Moderate | Users wait a few seconds on real death |
| Conservative (5s × 3) | ~15s | Low | Bad node keeps receiving traffic too long |
Numbers to know
- Phi=8 corresponds to a modeled arrival-tail probability of 10^-8; it is not a universal default or a direct probability that a peer is healthy.
- Cross-DC often raises phi toward 12 because regional variance looks like death on LAN thresholds.
- Balanced example: 1s × 3 plus buffers lands near a ~4s declare timeout.
- At 100K nodes and 1 heartbeat/sec, a central monitor sees about 100K messages/sec.
Deep-check cascading risk
If depends on a shared database, a database outage can remove every instance from load balancing at once. Prefer shallow , separate readiness, circuit breakers around dependency checks, and degrade features instead of marking the whole fleet not-ready.
Shallow vs deep checks, and orchestrator probes
The cascading risk above is why production systems split "is the process alive?" from "should it take traffic?" and why orchestrators expose three probe kinds with different failure actions.
Purpose: is the process running?
A liveness probe asks whether the instance is stuck in a condition that a restart can reasonably fix. A successful TCP handshake proves less than a lightweight application response, while a deep database query may fail for a reason restarting this process cannot fix. Repeated failed liveness probes can trigger a local restart. Configure a startup probe or suitable initialization allowance so a slow but healthy startup is not killed before it becomes ready.
Purpose: can this instance serve traffic?
A readiness probe determines whether new serving traffic should reach this instance. Check completed initialization and dependencies strictly necessary for that traffic, with a short bounded probe. Optional cache or recommendation failures may justify degraded responses instead of removing every instance. Readiness failure removes an endpoint from normal service selection; it should not itself be treated as a request to restart the container.
Orchestrator probes at scale
| Probe | Purpose | On failure | Configuration |
|---|---|---|---|
| startupProbe | Slow-starting containers | Delay liveness/readiness | failureThreshold: 30, period: 10s |
| livenessProbe | Is process alive / not hung? | Container restarted | period: 10s, failure: 3, timeout: 3s |
| readinessProbe | Can pod serve traffic? | Removed from Service endpoints | period: 5s, failure: 3, success: 2 |
Node agents can run HTTP, TCP, command, or supported gRPC probes and apply configured success/failure thresholds. With thousands of pods, stagger intervals and bound probe work so health checking does not become the workload. Kubernetes disruption budgets constrain supported voluntary disruptions; they do not prevent every liveness-triggered restart or an involuntary node failure. Prevent restart storms through appropriate probes and resource controls as well.
Membership probes and lease-based coordination
Health probes answer "should this instance take traffic?" Membership and coordination answer a harder question: who is still in the cluster, and who still holds write authority after a miss.
(Scalable Weakly-consistent Infection-style Membership)
Used by service-discovery systems, schedulers, and gossip libraries.
Every T seconds, node picks random peer:
1. PING peer directly (UDP)
2. If no ACK: pick K random OTHER nodes, ask them to PING the suspect (indirect probe)
3. If indirect probes also fail: mark as SUSPECT (not immediately dead)
4. After suspicion timeout: mark as DEAD, disseminate via gossipDetection time: O(log N) protocol periods
Message complexity: O(N) per period (constant per node)
: time-bounded permission that must be renewed
Coordination sessions: client connects → server grants session TTL (e.g. 30s). Client heartbeats every TTL/3. No heartbeat in TTL → session expired → ephemeral nodes deleted → watches fire.
Store leases: create lease, associate keys, keepAlive periodically. On crash: no keepAlive → lease expires → keys deleted.
Why leases help coordination
A heartbeat timeout is a monitor’s suspicion based on missing communication. A lease adds a designated authority that decides when ownership expires, reducing ambiguity about who can renew or acquire next. However, a stale client can still believe it owns the lease after a pause; checking and then writing has a race if authority expires between those actions. A resource-side fence or equivalent atomic ownership check makes stale writes rejectable.
False positives, scale, and what still breaks
Even with probes and , a false positive can still declare a live node dead. Causes include long GC pauses, network partitions, CPU saturation delaying heartbeats, and clock skew making leases look expired.
Incident pattern
Primary pauses → timeout → replica promoted → original primary resumes and writes again → . , lease checks, and write rejection at storage prevent stale owners from corrupting data.
Still fails when…
- Timeouts skip and treat one miss as death under jitter.
- Failover promotes a replica but storage accepts the old primary's unfenced writes.
- Readiness probes a shared database, so one DB blip drains the whole fleet.
Mitigations and trade-offs
- Keep timeouts near ~3× interval plus buffers so brief pauses do not flap.
- Raise and probe peers before marking a node DEAD.
- Require quorum agreement so one noisy monitor cannot move traffic alone.
- Issue a so storage rejects stale owners after failover.
- Short intervals detect faster but raise traffic and false-positive risk.
- Adaptive detectors cut false positives but need more tuning than fixed timeouts.
Heartbeats at massive scale
A central monitor watching 100K nodes at 1 heartbeat/sec faces 100K messages/sec (see Numbers to know above) and becomes a bottleneck. Hierarchical monitors (node → rack → DC → global), piggybacking heartbeats on data messages, and batching statuses cut traffic.
Messaging platforms often keep 100M+ TCP connections alive with keepalive-style heartbeats on connection servers, still a pulse, just amortized across long-lived sockets.
Choosing and explaining heartbeats
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
Heartbeats detect suspected failure, not absolute truth.
Use missed-heartbeat thresholds or phi accrual detectors before moving traffic or leadership.
Pair detection with leases, fencing, or quorum so false positives do not corrupt state.
When it belongs
The design needs failure detection for nodes, workers, sessions, or leaders.
A load balancer or scheduler must remove unhealthy instances.
Persistent connections need liveness checks.
How it fails
Network jitter causes false death declarations.
Heartbeat storms overload a large cluster.
A paused node resumes and acts on stale authority.
Trade-offs to weigh
Short intervals detect faster but create more traffic and false positives.
Adaptive detectors reduce false positives but are harder to tune.
Central monitoring is simpler; gossip-based detection scales better.
Common misunderstandings
Treating one missed heartbeat as death.
Ignoring clock pauses, GC, and regional latency.
Failing over without fencing old owners.
Try it: heartbeat misses and failure detection
Advance logical time between heartbeats. After enough missed intervals, the monitor marks the node suspect then dead so load balancers stop sending new work.
Member node
Status: healthy
Simulated clock: 0s
Last beat at: 0s
- Status
- healthy
- Interval
- 5s
- Miss threshold
- 2
Send heartbeats on schedule. Skip beats or advance time to see suspect and dead transitions.
Going deeper
Heartbeat
The mechanism and its boundary
Heartbeat protocols infer liveness from repeated signals. A missing heartbeat is suspicion, not proof: the node may be dead, slow, paused, or separated by the network.
Suspicion versus proof
Send beat
Miss count
Suspect
Confirm or recover
Work through the sizing assumptions
When it belongs
The system needs membership, leader health, lease renewal, or worker supervision. Failure detection speed matters to failover time. Nodes can be slow without being truly dead.
Assumptions that matter
Heartbeat writes are cheap but continuous. Timeouts should be based on observed latency distributions. Jitter and backoff avoid synchronized spikes.
Easy mistakes
One missed heartbeat is evidence of delay, not proof of a crash, so an immediate takeover can overlap with a still-running original worker. A timeout suited to a local network can produce frequent false suspicions across a slower region, so the detector must account for the path it actually observes. Synchronizing every sender to the same instant creates bursts that delay the receiver itself; jitter spreads the load without changing the intended average interval.
How it fails
Network jitter triggers false failover. Control plane overload delays heartbeat processing. Paused processes miss beats and later resume with stale ownership.
Trade-offs to name
A shorter failure threshold detects silence sooner, but also interprets more network jitter and process pauses as failure, so choose it from observed delay distributions. A central heartbeat receiver gives operators one membership view while concentrating continuous traffic and processing load on the control plane. A conservative timeout reduces unnecessary membership changes, but extends the period before replacement work begins after a real crash, so measure the full failover objective as well as detection time.
Wrap-up
You have seen the detection path, the sizing numbers, and the failure modes. Use the references below for the primary papers and the probe docs that match production practice.
Topic-specific references
- Hayashibara et al.: φ Accrual Failure Detector : adaptive suspicion vs fixed timeouts.
- Das et al.: SWIM (DSN 2002) : probe/indirect probe failure detection.
- Kubernetes Docs: Container probes : heartbeat-like health signals in practice.
Key takeaway
Heartbeats are the pulse of distributed systems. Prefer adaptive over brittle fixed timeouts, gossip or hierarchy when a central monitor cannot scale, and clear separation of from . coordination plus turn "we think you are dead" into "you can no longer write."
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Detect suspicion rather than certainty using timeout distributions, phi accrual, incarnation, GC pauses, asymmetric partitions, and mandatory fencing.
Explain the decision
Why can a heartbeat timeout not prove a process is dead?
Read the reasoning
Delay, pause, packet loss, or asymmetric partition is indistinguishable from failure at the observer; act through epochs/fencing.
Apply it to a production scenario
A 10-second GC pause repeatedly evicts healthy nodes. What changes?
Read one approach
Use pause-aware/adaptive detection, tune distributions, expose pause metrics, reduce runtime stalls, and keep rejoin/incarnation safe.
Work through the numbers
Heartbeat every second and suspect after five misses gives what base detection delay?
Read the worked calculation
Roughly 5 seconds plus phase/network/scheduling delay; faster detection raises false positives.
Reason through a failure
What must happen after a suspected node resumes?
Read the recovery reasoning
It obtains a newer incarnation/lease and stale work is rejected; detection alone never restores write authority.
Sources and further study
Continue with Checksums & Data Integrity.