System Design Reliability
Redundancy
Friday night on Instagram: a photo lands on a disk that dies an hour later. If storage only had one copy, the photo is gone. With real , other disks, zones, or parity shards still hold the bits, and the feed keeps loading.
Redundancy is spare capacity and spare data placed in independent . It protects , , and recoverability: three different jobs that often need different spare designs.
Upload survival journey
Place spares where they fail independently.
/
Decide who serves traffic now vs later.
/
Survive disk and node loss efficiently.
Multi-zone / region
Survive metro and regional disasters.
Try it: lose a failure domain and keep capacity
Model three availability zones with equal capacity. Mark a zone unhealthy and compare how active-active and active-passive layouts absorb the same traffic spike.
- Healthy zones
- 3
- Serving capacity
- 3000 req/s
- Headroom
- 600 req/s
- Fleet shape
- N+0 (full fleet)
Zone A
healthyCarrying ~800 req/s (80% of local cap).
Zone B
healthyCarrying ~800 req/s (80% of local cap).
Zone C
healthyCarrying ~800 req/s (80% of local cap).
3 zone(s) serve 2400 req/s at 80% utilization (N+0 (full fleet)).
Three zones each contribute capacity. Fail a zone and see whether active-active peers absorb traffic or active-passive promotion leaves a gap.
Redundancy is not duplication. Spares must sit in independent failure domains. Two servers on one rack still share power and top-of-rack networking.
N+1 is a fleet plan. Losing one zone while running at 80% of three-zone capacity still leaves headroom. Running at 95% per zone before the failure guarantees an incident.
What actually happens
At hyperscale, redundancy stacks from network cards to regions: multiple disks, servers, availability zones, sometimes multiple providers. Two servers behind one power feed or one database primary still share a hidden single point of failure.
Extra components can keep the service online (), protect data from permanent loss (), or help restore after bad deletes and migrations (recoverability). Confusing those jobs leads to “we have replicas” that still lose money or data.
Patterns differ by layer: vs for traffic, and for storage, / for fleet headroom, and cold / warm / hot standbys for how fast you need to cut over.
How to choose a redundancy pattern
1. What are you protecting? Uptime, data survival, or restore-after-corruption need different spares. Backups alone do not keep the site up; live replicas alone may not undo a bad migration.
2. Are failures independent? Replicas in one rack, one DNS provider, or one control plane share fate. Place capacity across and verify with drills.
3. Can survivors absorb the load? After one zone fails, post-failover utilization must stay under a planned threshold, or you shed work. Paper N+1 that browns out is not redundancy.
Simple rule of thumb
Stack redundancy at network, compute, storage, and data. Prefer independent placement over more copies in the same domain. Prefer over naive 3× copies when storage tax dominates; prefer hot paths when must be seconds.
Fund redundancy when prevented downtime cost beats monthly spare cost: make the break-even explicit.
Redundancy map: remove single points of failure
Redundancy is useful only when spare parts fail independently. Two servers behind one power feed, one switch, or one database primary still share a hidden single point of failure.
Users
Anycast DNS
AZ-a
LB in AZ-a
App pool AZ-a
AZ-b
LB in AZ-b
App pool AZ-b
Cache cluster
shared by both app pools
Primary DB
writes from both AZs
Sync standby
Async cross-region copy
Immutable backups
Failure domain
A failure domain groups components that can be lost through one cause. Two servers may be separate machines but share a rack power supply, availability zone, identity provider, or software release. Draw these shared dependencies explicitly. Adding replicas improves resilience only for failures that do not remove all of them together, so placement and deployment strategy matter as much as replica count.
Independence
The math for parallel availability assumes failures are not perfectly correlated.
Verification
Chaos tests and failover drills prove that the spare path actually serves traffic.
What redundancy actually protects
Redundancy is not one thing. Extra components can keep the service online, protect data from permanent loss, or help restore after bad data changes.
Availability
Keep serving requests during faults, for example app replicas behind a healthy load balancer.
Durability
Avoid permanent data loss, for example replicated storage, parity fragments, and immutable backups.
Recoverability
Restore to a known good point after deletion, corruption, or a bad migration.
Active-active vs active-passive tradeoffs
shares live traffic across redundant nodes (great utilization, needs conflict handling). keeps spare capacity mostly idle until failover: simpler mentally, slower cutover.
| Aspect | Active-Active | Active-Passive |
|---|---|---|
| Traffic | All nodes serve traffic | Only primary serves |
| Failover Recovery Time Objective (RTO) | ~0 (instant redirect) | Seconds to minutes |
| Resource Usage | 100% utilized | 50% idle (standby) |
| Consistency | Conflict resolution needed | Simpler (one writer) |
| Complexity | High (conflict, split-brain) | Medium |
| Best For | Global services, CDN | Databases, stateful services |
Data redundancy: RAID comparison
stripes or mirrors disks so one drive failure doesn't destroy data: levels trade capacity, write speed, and how many simultaneous faults survive. Mirroring stores full copies on separate drives; parity-based arrangements reconstruct missing data from surviving fragments. Both can survive specified drive failures, but neither restores a file that the application deliberately deleted from every live copy. Backups retain earlier recovery points. Also consider the rebuild period, when surviving drives face extra reads and the array may have less remaining fault tolerance.
| RAID | Redundancy | Capacity | Tolerance |
|---|---|---|---|
| RAID 0 | None | 100% | 0 disks |
| RAID 1 | Mirror | 50% | 1 disk |
| RAID 5 | Parity | (N-1)/N | 1 disk |
| RAID 6 | Dual Parity | (N-2)/N | 2 disks |
| RAID 10 | Mirror+Stripe | 50% | 1 per mirror |
Erasure coding: beyond RAID
splits blobs into fragments with parity, like RAID across objects. So you get durability similar to many replicas with less raw storage, at the cost of CPU/network when rebuilding after faults.
At cloud scale, RAID is insufficient. Erasure coding provides tunable redundancy across distributed nodes while reducing raw storage overhead versus full copies.
Reed–Solomon RS(k, m): fragments & overheadk data shards + m parity shards · tolerate any m losses among k+m
In a Reed–Solomon layout with k data fragments and m parity fragments, any k available fragments can reconstruct the original object, so up to m missing fragments are tolerated under that coding model. A 6+3 layout stores nine fragment-sized pieces for six pieces of original data, a 1.5× raw storage factor. Spread fragments across failure domains; losing four co-located pieces would exceed its protection.
| Scheme | Overhead | Fault tolerance | vs 3× replication |
|---|---|---|---|
| RS(10, 4) | 14/10 = 1.4× | Any 4 of 14 shards missing | 3× copy ≈ 3.0× overhead for far less flexibility |
| Distributed file store | RS(6,3) ≈ 1.5× | 3 arbitrary shard failures | Default for large blobs vs legacy 3× blocks |
| Local reconstruction code | RS(12,4) + local parity | Similar durability with faster rebuild hot-spots | Optimizes reconstruction bandwidth |
| Tiered object store | RS(6,3) warm / RS(12,4) cold | Tiered coding by access temperature | Petabyte-scale savings vs naive replication |
Trade-off: reads/writes pay encoding math and rebuild traffic; operations teams tune (k, m) for durability targets vs storage tax.
Geographic redundancy
Copies in different cities or regions survive floods, power loss, or ISP cuts that hit one metro. Pay latency and consistency complexity when writes must span geography.
Multi-Region
Deploy across 2+ cloud regions. Protects against entire region failure (data center fire, power grid outage, natural disaster).
- Most mature platforms offer multiple fault-isolated zones per region
- Async replication: ms latency within region, 50-200ms cross-region
- Synchronous cross-region writes increase latency but reduce data loss
Multi-Cloud
Deploy across independent infrastructure providers. Protects against provider-wide outages. Extremely complex but used by the most critical services.
- Challenge: different APIs, networking, identity, and access models
- Container orchestration and infrastructure-as-code help with abstraction
Write ownership patterns
Single writer
One primary accepts writes; standby promotes only after fencing old primary.
Partitioned writers
Each region owns specific users, tenants, or shards to reduce conflicts.
Multi-writer
If several regions accept writes to the same logical data, specify which conflicts can occur and how they are resolved. Independently adding items to a wish list may support a merge; independently selling one remaining ticket does not. Use domain-aware merging for compatible operations, or route a record’s writes through one authority when its invariant needs serialization. Active-active compute does not force multi-writer storage.
Read-only pause
Freeze writes when ownership is uncertain to avoid split-brain.
N+1 and N+2 redundancy
For N units of needed capacity, run (one spare) or (two spares) so failure plus maintenance still leaves headroom: common in power, cooling, and server fleets.
N+1: One extra component
N+2: Two extra components
N+2 during deployment
If peak needs 10 units and you run 12, one unit can fail while one more drains for a rolling deploy, leaving exactly 10 useful units. With only N+1, routine maintenance plus one fault can overload the fleet.
Failover capacity formula
Redundancy is not enough if the spare path is too small. Calculate whether surviving nodes can absorb traffic after a fault, then decide whether to add headroom, shed load, or accept degraded quality.
| Topology | Normal utilization | After one failure | Outcome |
|---|---|---|---|
| 2 regions | 45% each | 90% | Survivable with autoscale and shedding. |
| 3 regions | 60% each | 90% on each survivor | Good compromise for one-region loss. |
| 2 regions | 75% each | 150% | Routing works, service browns out. |
Million-user failure mode
If 1M active users reconnect after a region loss, connection spikes, cache misses, and login refreshes can exceed steady-state traffic by several multiples.
Recovery control
Reserve failover headroom, stagger reconnects with jitter, and predefine which features degrade when survivor utilization crosses 85-90%.
| Worksheet item | Question to answer | Why it matters |
|---|---|---|
| Normal utilization | What percentage does each region run at peak? | Tells whether one failure is survivable. |
| Surviving capacity | How much capacity remains after one zone or region loss? | Routing alone does not create CPU, memory, or database connections. |
| Reconnect multiplier | Will clients reconnect at 1.2x, 1.5x, or 2x steady traffic? | Failover often creates a short surge before steady state. |
| Autoscaling delay | How many minutes before new replicas are warm? | Degraded mode must cover this gap. |
| Degradation threshold | At what utilization do we shed low-priority work? | Prevents a brownout from becoming a full outage. |
Redundancy at every layer
Reliability fails at the weakest layer: redundant app servers on one top-of-rack switch still share fate. Walk the stack: network, compute, storage, software, to find hidden single points.
| Layer | Redundancy Strategy | Failure Scenario |
|---|---|---|
| Network | Dual NICs, redundant switches, multiple ISPs | One transit provider drops routes. |
| DNS | Multiple NS records, anycast | One nameserver fleet is unreachable. |
| Load Balancer | Active-passive pair, anycast + ECMP | One balancer drains or misroutes traffic. |
| Compute | N+2 instances, auto-scaling, multi-AZ | One zone loses half the app pool. |
| Storage | Erasure coding, cross-region replication | Multiple disks or one zone disappear. |
| Database | Primary + sync replica + async read replicas | Primary writer fails or network-partitions. |
| Cache | Clustered cache with replicas, multi-tier | Hot shard restarts and causes a stampede. |
Failure-mode analysis guidance
In design reviews, walk each layer and ask: what fails, how do we detect it, who takes over, how much capacity remains, and what data might be stale or lost?
Common hidden gaps
- Replicas exist but are under-provisioned for failover load.
- Backups exist but encryption keys or restore order are missing.
- All replicas receive the same bad deploy at once.
- Health checks test process liveness, not real readiness.
Strong answers mention
- Independent AZ/region placement and anti-affinity rules.
- Automated failover plus manual override/fencing.
- Capacity headroom and load shedding during failover.
- Regular restore tests, game days, and postmortem action tracking.
Cold / warm / hot standby comparison
backups wake slowly but cheap; keeps partial stacks ready; mirrors production closely for fastest cutover: cost rises as shrinks.
Cold Standby
A cold recovery environment relies on stored backups and infrastructure definitions, with most serving resources created or started after an incident. Its cost can be lower because little compute runs continuously, but recovery includes provisioning, restoring data, obtaining keys, validating dependencies, and routing traffic. Recovery may take hours or longer depending on data volume and automation; the architecture label alone does not establish an RTO.
Warm Standby
A warm standby runs the required services at reduced capacity and keeps data reasonably current. Recovery expands capacity and validates it before taking full traffic. This reduces setup work compared with a cold restore, but depends on regional capacity, quotas, cache warm-up, and replication progress. Measure both the time to serve a small safe traffic slice and the time to carry peak load.
Hot Standby
A hot standby keeps a near-production serving environment ready for promotion. It can recover quickly because provisioning and much warm-up work are already complete, but health detection, writer fencing, replication state, and client reconnection still take time. Hot does not mean zero data loss or instant switch: synchronous versus asynchronous replication and the tested promotion procedure determine those properties.
Cost of redundancy vs cost of downtime
Frame capital and operating cost of replicas against revenue or SLA penalties per minute: leaders fund redundancy when the break-even math is explicit, not ideological.
Break-even formula
- Monthly redundancy cost: extra regions + extra replicas + operational staffing.
- Expected monthly outage cost: outage minutes × revenue impact/minute + support cost + SLA credits.
- Break-even: redundancy is justified when prevented outage cost is greater than extra monthly cost.
- Talking point: discuss formula and assumptions instead of quoting fixed vendor numbers.
Example: if downtime cost is C per minute and redundancy costs R per month, it pays for itself when it prevents more than R / C minutes of downtime in a month.
Redundancy pattern examples
Production case studies usually boil down to layered redundancy: multiple zones, synchronous replication, erasure-coded objects, and rehearsed failover. Extract the pattern rather than memorizing marketing numbers.
Object storage durability
An object-storage design can spread replicated copies or erasure-coded fragments across zones so that a specified number of failures still leaves enough data to read. Its durability and availability depend on placement, repair speed, and the supported failure model. Cross-region copies add geographic protection but may lag. Verify the chosen service’s actual storage-class guarantees rather than assuming every multi-zone object store tolerates the same losses.
Globally replicated database
Multiple replicas across 3+ zones by default. Synchronous quorum replication can enable strongly consistent reads. Survives zone and some regional failures when quorum and placement rules remain healthy.
Key takeaway
Redundancy is the foundation of everything reliable. Active-active maximizes utilization but requires conflict resolution. Erasure coding (RS codes) is more space-efficient than simple replication. Redundancy must exist at every layer: network, compute, storage, data. The cost of redundancy is often justified at billion-user scale when one hour of downtime costs more than a year of spare capacity.
Going deeper: correlated failure worksheet
Replicas in one rack, one DNS provider, or one control plane share fate. Add dependency diversity, restore verification, and a cost-versus-risk table before calling the system highly available.
- Control-plane redundancy is as important as data-plane replicas.
- Verify restores, not only backup success metrics.
- N+1 capacity must remain after deploy and zone loss, not just on paper.
Primary sources
- Google SRE: Service Level Objectives: nines via parallel composition.
- AWS Well-Architected: Reliability Pillar: multi-AZ redundancy patterns.
- Brewer: CAP Twelve Years Later: redundancy does not erase partition trade-offs.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Remove single points of failure without cloning common-mode defects, control-plane dependencies, bad data, or unusable backups.
Explain the decision
Why can identical redundant systems fail together?
Read the reasoning
They can share software bugs, credentials, configuration, upstreams, capacity assumptions, or deployment pipelines—correlated failure defeats replica count.
Apply it to a production scenario
Primary and backup regions use the same identity control plane, which is unavailable. What did the design miss?
Read one approach
Control-plane dependency and break-glass access were common-mode failures; diversify or pre-provision the minimal recovery path.
Work through the numbers
Two truly independent components each fail 0.1% of time. Approximate simultaneous unavailability.
Read the worked calculation
0.001² = 0.000001, or 99.9999% availability; independence is the strong and often false assumption.
Reason through a failure
What must backup redundancy prove?
Read the recovery reasoning
A clean, authorized, complete restore within RPO/RTO—not just that multiple backup files exist.
Sources and further study
Continue with Disaster Recovery.