System Design Reliability

Fault Detection and Recovery Techniques

Retries and deadlines keep a single call honest. This note is the next layer: how the platform notices a broken replica, disk, or process—and how it recovers without creating a second writer.

Pair with timeouts/retries for request paths and heartbeats for liveness signals. Here we connect to , , checkpoints, and healing loops.

Detect → act safely

Probes, checksums, metrics, quorum votes.

Retire the old owner before promoting.

Restart, failover, rebuild, or degrade.

Automate routine repair with blast-radius caps.

What actually happens

Request-level fault tolerance (timeouts, bounded retries, breakers) handles transient remote errors. System-level watches processes, disks, and replicas over longer windows. The outputs feed : kill and restart, promote a standby, rebuild a shard, or serve degraded mode.

Models matter. is the happy assumption. Real networks produce behavior that looks like death. behavior needs heavier machinery and is rare as an explicit product requirement.

Silent corruption needs , not heartbeats. Leadership bugs need . Long jobs need . Platforms glue these into loops with rate limits.

How to choose detection + recovery

1. What fails? Process crash → restart. Primary unreachable → failover + fence. Bit rot → scrub and repair from replicas.

2. Cost of a false positive? High (dual writers) → suspicion, quorum, leases. Low (stateless pod) → aggressive restart OK.

3. What is the RPO/RTO? Dictates sync vs async replication and whether checkpoint intervals are seconds or minutes.

Simple rule of thumb

Detect with more than one signal when the action is destructive. Fence before promote. Prefer restart for stateless; failover for unique leadership; rebuild for under-replicated data. Cap automated healing so one bad config cannot loop the fleet.

Detection to recovery loop

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

flowchart TD
    Sig[Signals: heartbeat, probe, checksum, metrics] --> Dec{Detector}
    Dec -->|Healthy| Cont[Continue]
    Dec -->|Suspect| Soft[Soft state: retry probe / phi]
    Soft -->|Refuted| Cont
    Soft -->|Confirmed| Act[Recovery action]
    Act --> Fence[Fence stale owner]
    Fence --> Rec[Restart / failover / rebuild / degrade]
    Rec --> Ver[Verify: checksum, replica vote, smoke test]
    Ver -->|OK| Cont
    Ver -->|Fail| Esc[Escalate / page / quarantine]

plaintext

Complementary layers (do not collapse them):
  Request path: timeouts, retries, jitter, breakers, fallbacks
  Membership: heartbeats, SWIM, phi accrual, leases
  Data plane: CRC/xxHash, scrubbers, replica checksum compare
  Control plane: failover, re-replication, quarantines

Interview line:
  “Retries mask blips; detectors declare component failure;
   recovery changes topology; fencing makes that change safe.”

Detection techniques

Technique	Catches	Misses / risks
Heartbeat / lease	Crash, hang, partition (approx.)	GC pauses; false death
Active probe / SWIM	Peer failure with indirect checks	Probe traffic; partial nets
Watchdog timer	Stuck control loops	Must pet correctly or false reboot
Checksum / scrub	Silent corruption, bit rot	Not liveness; needs replicas to repair
Replica divergence	Wrong answers, stuck replicas	Needs quorum or merkle compare
SLO / anomaly metrics	Brownouts, hot shards	Noisy; correlation ≠ root cause
End-to-end canaries	“Works in healthz, dead for users”	Coverage gaps; synthetic bias

Heartbeat design details live in the Heartbeat note. Here the rule is compositional: liveness ≠ readiness ≠ data integrity. A green process heartbeat can still serve corrupt pages if you skipped checksums.

Failover with fencing

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

sequenceDiagram
    autonumber
    participant P as Primary
    participant M as Monitor / quorum
    participant S as Standby
    participant St as Storage

    P--xM: heartbeats stop
    M->>M: suspicion → confirmed failure
    M->>St: fence generation = G+1
    M->>S: promote with token G+1
    S->>St: writes carry G+1
    P->>St: stale write with G (rejected)
    Note over P,St: Fencing prevents split-brain after false death

has two clocks: how long until you are willing to declare death (detection), and how long until the standby can serve (warm-up, cache fill, replication catch-up). Quote both as RTO contributors.

mechanisms include: monotonically increasing epoch in every write, SCSI-3 persistent reservations, cloud “shoot the node” APIs, and lease-protected keys in etcd/ZooKeeper. Pick one at the durability boundary—not only in the load balancer.

plaintext

Failover checklist:
  1. Detector reaches confirmed-failed (not one missed ping)
  2. Acquire new epoch / fence token from quorum or coordinator
  3. Storage rejects lower epochs
  4. Promote standby; redirect clients
  5. Smoke test: write+read, checksum spot check
  6. Quarantine old node until human or automated forensics

RPO note:
  async replica may be behind → accept data loss or block promote until caught up
  sync/quorum replica → RPO≈0 for acknowledged writes, higher write latency

Checkpoint, log replay, and rebuild

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

flowchart LR
    W[Worker progress] --> C[Checkpoint every T]
    C --> Store[(Durable snapshot / offset)]
    Crash[Crash] --> R[Restart]
    R --> Store
    Store --> Replay[Resume from last checkpoint]
    Replay --> Redo[Redo durable log if needed]
    Redo --> W2[Continue]

plaintext

Patterns:
  Checkpoint/restart: batch jobs, stream operators, ML workers
  Write-ahead log + restart: databases (redo to last consistent point)
  Re-replication: HDFS/S3/erasure-coded objects under-replicated
  Rollback: bad deploy → previous artifact + data migrations caution

Interval math:
  checkpoint every 60s, 1k tasks → avg lost work 30s per task on crash
  too frequent → IO amplification; too rare → long recovery

Consistency:
  uncoordinated checkpoints need orphan handling or a global barrier
  prefer storing durable input offsets (Kafka) over ad-hoc memory dumps

Silent corruption and scrubbing

Disks and DRAM lie occasionally. An on every durable record (and ideally end-to-end from client) turns silent corruption into a detectable fault. Background scrubbers read cold data before users do. Repair pulls from another replica or reconstructs via erasure coding. If all copies match but are wrong, you needed application-level invariants or backups—checksums alone cannot invent truth.

Detect

Per-block CRC / xxHash
Merkle trees across ranges
ECC memory + scrubbing
Compare replica digests

Recover

Rewrite block from healthy replica
Erasure reconstruct
Quarantine bad disk / node
Restore from backup if N copies wrong

Self-healing with blast-radius caps

is automation around the loop above: Kubernetes restarts crashed pods; controllers re-replicate; autoscalers replace bad instances. Without caps, a bad config or cascading dependency outage becomes a reboot storm. Rate-limit restarts per node, require backoff, separate liveness from readiness, and page humans when failure rate exceeds a budget.

plaintext

Healing policy sketch:
  max restarts per pod per hour
  max % of shard replicas rebuilding concurrently
  circuit-break healing when correlated failures spike
  prefer remove-from-LB (readiness) before kill (liveness)

Correlate with request-path controls:
  breakers reduce load on a sick dependency
  detectors decide when to replace the dependency instance
  both are required; neither replaces the other

Mapping fault classes to actions

Observed fault	First recovery	Escalate if
Stateless pod crash	Restart; keep serving on others	Crash loop; page on budget burn
Primary silent	Suspect → fence → failover	Quorum loss; manual promote
Checksum mismatch	Repair from replica; quarantine disk	Multi-replica agreement on bad data
AZ partition	Serve majority side; fence minority	No majority → controlled outage
Bad deploy	Rollback binary; freeze writes if needed	Data migration irreversible
Overload brownout	Admit less; do not mass-restart	Healing would amplify load

SD use cases

Replicated key-value store

Lease-based leader, epoch fencing on every write, replica checksums, and automated re-replication when a node is marked dead. Client retries (request path) use the same idempotency keys before and after failover.

Consumer workers

Detect stuck consumers via lag and processing heartbeats. Recover by reclaiming partitions after session timeout. Checkpoint offsets only after side effects commit—or make side effects idempotent.

Object / blob storage

Scrubbers + erasure coding reconstruct missing shards. Mark disks unhealthy on CRC storms. Avoid declaring a whole node dead on one slow disk if isolation can evacuate only that disk.

Stateless API fleet

Liveness restart + readiness remove-from-LB. Prefer shedding and breakers under overload rather than detecting “failure” and restarting healthy pods that are merely slow.

Worked example: false death

Primary GC pauses 12 s. Detector timeout is 10 s. Standby promotes. Old primary wakes and accepts a client write on a sticky connection.

plaintext

Without fencing:
  two primaries → divergent data → hard incident

With fencing:
  promote bumps epoch to 42
  storage rejects epoch 41 writes from old primary
  sticky clients reconnect and learn new leader

Also improve detection:
  phi accrual or longer timeout + suspicion
  separate “remove from LB” (fast) from “promote leader” (careful)
  require quorum of monitors, not one flaky observer

Trade-offs

Choice	Gain	Cost
Aggressive detection	Low RTO	False positives, flapping
Conservative detection	Fewer split-brain risks	Longer user-visible outages
Auto failover	Hands-free RTO	Needs rock-solid fencing
Checkpoint often	Less lost work	IO and stall overhead
End-to-end checksums	Catch silent corruption	CPU; repair machinery
Self-healing	Operational leverage	Restart storms if uncapped
Byzantine protocols	Survive liars	High latency and complexity

Choosing and explaining fault detection and recovery

A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.

Explain the core idea

Separate request-path tolerance (retries) from component detection and topology recovery.

Name the fault model (crash vs silent vs corruption) and match detectors to it.

Always pair failover with fencing; quote RPO/RTO as detection + promotion + warm-up.

When it belongs

Designing leader election, multi-AZ databases, or re-replication.

Explaining how the system survives node loss or disk corruption.

Contrasting health checks with checksums and canaries.

How it fails

False death → dual writers without fencing.

Restart storms under overload mistaken for crashes.

Undetected bit rot until a rare read.

Failover to a lagging replica → silent data loss beyond RPO.

Trade-offs to weigh

Detection speed versus false-positive rate.

Automation versus blast radius.

Checkpoint frequency versus overhead.

Sync replication (RPO) versus write latency.

Common misunderstandings

Treating retries as recovery of a dead primary.

Equating /healthz success with correct data.

Promoting without epoch/lease fencing.

Healing by reboot during brownouts.

Wrap-up

Detection chooses when to stop trusting a component. Recovery chooses how to restore service. Fencing makes the handoff safe. Retries remain for transient calls—not for declaring a new primary.

Exercises

RTO budget. Detection 8 s, fence 1 s, promote 3 s, warm cache 20 s. What do users feel, and what can you overlap?
False positive. Design fencing for a primary that was only GC-paused.
Corruption. One of three replicas fails CRC. Steps to repair and to prevent serving the bad copy.

Primary sources

Key takeaway

Detect with signals matched to the fault, confirm before destructive action, fence stale owners, then recover and verify. Keep request retries for blips; keep detectors and healers for components—and never let healing amplify overload.