System Design Reliability
High Availability
Friday 6:12 PM: sale traffic peaks, and a database primary dies mid-Pay. If the product is designed for , traffic moves to healthy capacity and the payment still completes: maybe a little slower, maybe without recommendations, but the money path stays alive.
is not “never fail.” It is picking a numeric uptime goal, measuring it, and building redundancy plus automatic so outages stay inside that budget.
Checkout reliability journey
Define what “good checkout” means in numbers.
Set the internal target the team owns.
Spend unreliability on launches, or freeze when burning.
Shift users to healthy zones or regions automatically.
What actually happens
At billion-user scale, even allows roughly 52.6 minutes of downtime per year, still millions of failed requests. Each extra nine shrinks allowed outage roughly 10× and usually costs much more in engineering and infra.
Teams translate that into an (what you measure), an (internal goal), and an (customer promise). The is the spendable gap between perfection and the SLO. When burn rate spikes, you pause risky deploys and fix reliability before shipping more features.
Architecture follows the budget: parallel redundancy behind load balancers, multi-zone or multi-region serving, health checks that remove sick nodes, and when surviving capacity is tight.
Try it: spend an error budget during failover
Translate nines into allowed outage minutes, then stress a checkout path. Fail the primary and compare active-active routing with an active-passive promotion that takes measurable time.
- Checkout SLI
- 100.000%
- Budget used
- 0.00 min
- Budget left
- 52.60 min
- Failover drills
- 0
Primary region
healthyAccepting writes and reads behind the load balancer.
Secondary region
servingAlways warm. Takes share of traffic when peers fail health checks.
Both paths healthy. Success rate 100.000% on 10,000 simulated requests.
Connect SLI to SLO. The success rate above is a coarse checkout SLI. Compare it with the nines target: four nines allow roughly 52.6 minutes of downtime per year, not zero failed requests.
Failover shape drives user pain. Active-active hides many machine deaths behind health checks. Active-passive saves cost but concentrates risk in promotion time, which is exactly what the failover delay slider models.
How to choose an availability target
1. Which journey owns the SLO? Login, checkout, feed reads, and admin tools often need different targets. Measure the user path that loses money or trust when it fails.
2. Translate nines into minutes. Four nines for payments may be worth the cost; four nines for avatar uploads probably is not. Cost jumps faster than the percentage looks.
3. Can survivors absorb the load? Duplicate regions only help if headroom, replication lag, and degraded mode are planned: routing alone does not create CPU.
Simple rule of thumb
Prefer parallel redundancy and automated over heroic manual runbooks. Pair every HA claim with measurement: error budgets, multi-region drills, and .
Call out correlated failures: two replicas on one rack do not give independent availability.
Service targets and error budget intuition
is the measurement, is the internal target, is the customer promise, and the is how much unreliability you may spend before slowing risky launches.
SLI: measured availability
SLO: internal goal
SLA: external contract
from SLO
Error budget
Budget healthy?
yes
Ship normally
burning fast
Freeze risky launches + improve reliability
SLI
For request-based availability, divide successful eligible requests by all eligible requests in the chosen window. Define success at the user boundary: a response that arrives after the deadline or contains unusable data may count as bad even if its HTTP status is 200. Define exclusions explicitly so health checks and internal retries do not distort the customer-facing measure.
SLO
Example: 99.95% checkout availability internally.
SLA
Example: 99.9% promised to customers with credits.
Error budget
The gap between perfection and the SLO; spend it on launches or incidents.
SLA math and the nines
"Three nines" or "four nines" are shorthand for allowed downtime per month or year. The table turns percentages into real minutes so you can reason about incidents.
| Availability | Downtime/Year | Downtime/Month | Downtime/Week |
|---|---|---|---|
| 99% (2 nines) | 3.65 days | 7.31 hours | 1.68 hours |
| 99.9% (3 nines) | 8.76 hours | 43.8 minutes | 10.1 minutes |
| 99.99% (4 nines) | 52.6 minutes | 4.38 minutes | 1.01 minutes |
| 99.999% (5 nines) | 5.26 minutes | 26.3 seconds | 6.05 seconds |
Error budget burn math
An converts reliability from vibes into spendable minutes or failed requests. If you burn it too fast, freeze risky changes and spend engineering time on reliability.
Monthly request budget
1B requests/month at 99.95% SLO allows 0.05% failures = 500K bad requests.
Burn rate
If 250K failures happen in 6 hours, you burned 50% of the monthly budget in 0.8% of the month.
Decision
High burn rate means pause deploys, shed low-priority traffic, and fix the highest-impact reliability gap before feature work resumes.
| Target | Allowed bad requests / 1B | Downtime / year | Operational posture |
|---|---|---|---|
| 99.9% | 1,000,000 | 8.76 hours | Good for non-critical surfaces. |
| 99.95% | 500,000 | 4.38 hours | Reasonable for important read paths. |
| 99.99% | 100,000 | 52.6 minutes | Needs active failover and strict change control. |
Reliability target worksheet
Start from the user journey, not from server uptime. Define what counts as good and bad requests, then convert the target into failed requests, downtime minutes, and launch decisions.
Critical journey: checkout
SLI numerator: successful checkouts
SLI denominator: valid checkout attempts
SLO: 99.95% over 30 days
Error budget: 0.05% bad
Burn rate high?
no
Ship normally
yes
Pause risky deploys + degrade non-critical features
Request budget
2B monthly checkout requests at 99.95% SLO allow 1M bad requests in that month.
Incident burn
A 30-minute incident at 20K failed requests/min creates 600K bad requests, or 60% of the monthly budget.
Decision
Freeze risky launches, shed enrichment work, and spend the next changes on the highest budget-burning failure mode.
| Burn signal | Meaning | Response |
|---|---|---|
| 1x burn | Budget is being spent at the planned monthly pace. | Monitor and keep normal release flow. |
| 5x for 1 hour | Small incident can eat days of budget. | Pause risky deploys and assign reliability owner. |
| 20x for 10 minutes | User-visible outage is active. | Incident response, feature shedding, rollback checks. |
| 100x | Budget will vanish quickly. | Emergency rollback, failover, or traffic isolation. |
Calculating composite availability
Think serial (everything in a row must work: availability multiplies and gets worse) vs parallel (backups take over: availability improves when paths are redundant). Real systems chain multiple components; modeling composition explains why long synchronous chains hurt uptime.
Serial chain
all must workAtotal = A₁ × A₂ × A₃ × …
- Example: Web → App → DB, each component target 99.9% (three nines).
- Combined: 0.999³ ≈ 0.997 → roughly two nines end-to-end (~26 hours downtime/year).
- Lesson: synchronous pipelines multiply outage probability: shorten chains or add redundancy inside each hop.
Parallel redundancy
any path can serveAtotal = 1 − (1 − A₁)(1 − A₂)(1 − A₃)…
- Two replicas at 99.9% each: 1 − (0.001)² = 0.999999 → about six nines (~31 seconds/year).
- Three replicas at 99% each: 1 − (0.01)³ = 0.999999 → same ballpark from weaker nodes: quantity helps.
- Lesson: independent failover paths improve availability, assuming failures aren't perfectly correlated.
Correlated failure caveat
Parallel availability math assumes independent failures. Two replicas behind the same bad deploy, shared database, shared control plane, shared config push, or shared network dependency can fail together, so do not count them as true redundancy.
User-visible availability also depends on the journey. If login works but checkout fails, browsing is available but the revenue-critical checkout journey is unavailable.
Active-active at global scale
: several regions serve real users at the same time (better utilization, faster redirect). : one region carries traffic while others mostly replicate: simpler, but failover takes longer and spare capacity idles.
Active-Passive
In active-passive serving, one environment handles the workload while another is prepared to take over. The standby may be fully provisioned, reduced in capacity, or require restore, and those choices strongly affect recovery time. Confirm that it has usable data, credentials, dependencies, and capacity. Paying for a second region is insufficient if its only database endpoint or key service still depends on the failed region.
- Failover takes minutes to hours
- Standby may have stale data
- 50% of infrastructure is idle
- Use case: cost-sensitive moderate HA
Active-Active (Multi-Region)
All regions serve traffic. If one fails, others absorb the load quickly, assuming data ownership and capacity planning were designed for regional loss.
- Near-zero , traffic shifts via DNS/anycast
- Requires conflict resolution (CRDTs, LWW, vector clocks)
- Full resource utilization across all regions
- Challenge: cross-region data consistency (CAP tradeoffs)
If multiple regions accept writes at once, somebody must decide how merges happen: naïve "last timestamp wins" can lose edits; fancier CRDTs or domain merges trade simplicity for correctness. For a counter, a merge rule may legitimately combine increments from different regions. For selling the last seat, merging two successful sales after the fact cannot preserve the no-double-sale promise. Assign one write owner, coordinate through a quorum, or reject writes during uncertainty for such invariants. Choose the consistency boundary from the business rule before choosing active-active routing.
Conflict resolution in active-active
Strategy 1: Last-Write-Wins (LWW)
Timestamp determines winner. Simple but can lose data when clocks skew or concurrent edits carry business meaning.
Strategy 2: CRDTs (Conflict-Free Replicated Data Types)
Automatically merge compatible data types without conflicts; useful for counters, sets, and collaborative state with predictable merge rules.
Strategy 3: Application-level merge
Custom logic per domain, such as merging shopping carts by item identity instead of blindly keeping one whole record.
Health monitoring and graceful shutdown
Load balancers and orchestrators need a reliable signal: "alive" vs "ready to take traffic." When you shut a machine down, drain in-flight work first so users don't see abrupt errors. A readiness failure should remove an instance only when redirecting traffic can improve the request outcome. If every replica depends on the same unavailable optional cache, marking all replicas unready can remove the fallback path too. Keep liveness focused on whether restarting this process would help, and use separate end-to-end probes to detect shared dependency or partial-correctness failures.
Multi-layer health checks
Layer 1: Liveness (is process running?)
GET /healthz returns 200 OK when the process is alive (minimal checks).
Layer 2: Readiness (can serve traffic?)
GET /readyz verifies dependencies (database, cache, critical RPCs) before receiving traffic.
Layer 3: Deep (end-to-end validation)
GET /health/deep runs an end-to-end probe (for example a canned read/write) less frequently than readiness.
Orchestrator probes
livenessProbe: fail → restart pod
readinessProbe: fail → remove from LB
startupProbe: fail → don't check othersGraceful shutdown
Zero-downtime deployment strategies
Ship new code without a global "site down" window: run two pools (blue/green), slice traffic gradually (canary), or replace instances in waves (rolling): each trades cost and rollback speed differently.
Blue-Green
Blue/green deployment runs old and new versions side by side and switches a routing rule after validation. That switch is not instantaneous for every existing connection, so both versions may serve during transition. Rollback also requires compatible schemas and events: new writes must remain readable by the old version. Test connection draining and data compatibility along with error and latency checks.
Cost: roughly 2× infrastructure during the deploy window.
Canary
Route 1-5% of traffic to new version. Monitor error rates, latency, business metrics. Gradually increase to 100% if healthy. Auto-rollback if metrics degrade.
Common rollout path: 0.1% → 1% → 10% → 100%, with automatic rollback on guardrail breach.
Rolling
Replace instances one at a time. At any point, both versions serve traffic. Lower cost than blue-green; slower rollback.
Typical orchestrator settings: max surge above target capacity, zero unavailable instances when strict uptime matters.
Chaos engineering
If you only test happy paths, production will surprise you. injects controlled failures (kill instances, drop AZs, add latency) to prove redundancy and runbooks actually work: preferably starting small with clear rollback.
Chaos practice intentionally injects small, controlled failures in production-like systems to validate that HA mechanisms actually work.
Chaos engineering principles
- Define "steady state" (normal behavior metrics)
- Hypothesize that steady state continues during failure
- Inject real-world failures (instance, zone, region)
- Observe. If system degrades, fix the gap.
- Automate and run continuously
Failover strategies at scale
is switching users or writes to healthy capacity when something breaks, via DNS, load balancer health checks, database promotion, or already-hot secondary regions. Faster failover usually costs more duplicated infra.
| Strategy | RTO | Cost | Use Case |
|---|---|---|---|
| DNS Failover | 1-10 min | Low | Simple web apps |
| LB Health Check | 10-30 sec | Medium | Stateless services |
| DB Auto-Failover | 30-60 sec | Medium | Managed primary/standby databases |
| Active-Active | ~0 sec | High | Global latency-sensitive services |
DNS health-check failover
DNS-based failover updates the answer returned for future lookups after health checks declare a destination unhealthy. Recovery includes failure detection, DNS propagation and cache expiry, client re-resolution, connection establishment, and capacity warm-up. Existing connections and clients with unusual caching behavior may not move immediately. Use a measured recovery distribution instead of treating the configured TTL as a guaranteed cutover time.
Database promotion failover
Writer fails → promote read replica → endpoint updates → apps reconnect with jitter and idempotency safeguards
Capacity during failover
Failover only works if surviving zones or regions have enough spare capacity. If each region normally runs at 80% CPU, losing one of two regions sends the survivor to 160% theoretical load, which means brownout unless you shed traffic or run more headroom.
2 regions at 45% each
One region fails → survivor handles 90%. Expensive but feasible with autoscale and shedding.
3 regions at 60% each
One fails → remaining regions carry 90% each. Common compromise for active-active global systems.
2 regions at 75% each
One fails → survivor needs 150%. Routing succeeds but users still see overload.
Failover surge calculation
Normal traffic: 900K req/s across 3 regions = 300K req/s each at 60% utilization. One region fails, so two survivors handle 450K req/s each = 90%. A 1.3x reconnect surge pushes each survivor to 585K req/s, or 117% required capacity. The response is to cap retries, shed low-priority 25%, serve cached data, and delay non-critical writes until utilization drops below 85%.
Availability is not just duplicate regions; mention headroom, autoscaling lag, data replication lag, and whether degraded mode protects core traffic while capacity catches up.
Million-user regional outage scenario
A regional outage is not solved by routing alone. The surviving regions need compute headroom, hot data, retry control, and for non-critical features.
1M active users
Region A fails
Traffic manager drains Region A
Regions B/C absorb users
Capacity below 90%?
yes
Serve normally + monitor lag
no
Enable degraded mode
Disable recommendations/search enrichments/media previews
Protect login, reads, writes, checkout
Failure-mode analysis
- Users reconnect at once, creating login and cache stampedes.
- Async replicas lag, so recent writes may be stale in surviving regions.
- Client retries amplify load unless deadlines and retry budgets are enforced.
Recovery plan
- Freeze risky deploys, declare incident, and route away from unhealthy capacity.
- Protect critical paths; serve cached recommendations, basic search, and compressed media.
- Reintroduce traffic gradually after replication catches up and error budget burn normalizes.
HA architecture patterns
Large-scale architectures combine globally routed traffic, replicated storage, orchestration, and cultural discipline around drills. Map examples back to primitives rather than memorizing names.
Global read-heavy service
Uses edge routing, replicated indexes, multi-region serving pools, and a strict freshness model so a single regional loss does not remove the core read path.
Global media service
Keeps playback metadata, session state, and media delivery decoupled so browsing can degrade while already-started playback remains protected.
How to talk about reliability targets
- Ask which user journey owns the SLO: login, checkout, feed reads, writes, or admin back office may need different targets.
- Translate nines to minutes and explain cost: four nines for payments may be worth it; four nines for avatar uploads probably is not.
- Call out correlated failures: two replicas on one rack do not give independent availability.
- Discuss failover headroom explicitly: can the remaining region absorb peak traffic, or do we need graceful degradation?
- Tell a graceful degradation story: preserve login, reads, and writes while recommendations, search enrichments, and media previews fall back.
- Close with blast radius: can one bad deploy, noisy tenant, or dependency outage take down every region at once?
What must never become false
Correctness invariant
The critical user journey stays inside its / definition: do not silently redefine success during an incident. After any single failure domain is lost (node, AZ, or region), remaining capacity must either absorb peak traffic or enter a named mode. Redundant copies that share a rack or dependency do not count as independent availability.
Failure timeline: one AZ gone
t=0 3 AZs active-active; 99.95% checkout SLO; error budget healthy t=1m AZ-a network partition; health checks fail; traffic steers away t=2m Survivors absorb ~50% more QPS; if utilization was >66%, pools saturate t=5m Clients reconnect + retry → stampede on auth/cache t=10m Without degradation: budget burns (e.g. 30m × 20K fail/min = 600K bad req) t=15m With headroom + shed enrichments + retry budgets: critical path survives Outcome HA is capacity after failover, not just "we have three zones"
Exercises
- 2B monthly checkouts at 99.95% SLO. A 20-minute outage fails 15K req/min. What fraction of the monthly error budget did you spend? Freeze launches?
- Three serial tiers at 99.9% each vs two parallel replicas at 99.9%. Compute composite availability and explain the interview talk track.
- Design graceful degradation for checkout when recommendations and search enrichments are down. Which SLIs still must stay green?
References
- Google SRE Book: Service Level Objectives / Error Budgets.
- Google SRE Book: Addressing Cascading Failures / Embracing Risk.
- AWS Well-Architected: Reliability Pillar (multi-AZ, blast radius, recovery objectives).
Key takeaway
HA at scale = redundancy (parallel composition) + automated failover + continuous chaos testing + zero-downtime deployments. Serial chains degrade availability; parallel redundancy multiplies nines. Active-active multi-region is the gold standard but requires conflict resolution. Every minute of downtime at massive scale affects millions: design accordingly.
Going deeper: dependency SLO allocation
Allocate the product’s failure allowance across the required dependency path. A 99.9% journey cannot require several 99.9% dependencies and assume the combined result is still 99.9%. Improve those dependencies, remove optional work from the path, or provide a correctness-preserving fallback. Failing open is appropriate only when the product can safely continue without that check; authorization and inventory ownership cannot be bypassed merely to improve an availability metric.
Gray failures
Detect elevated latency and partial correctness, not only hard downtime. Hedged reads and outlier ejection help when a replica is sick but not dead.
Stateful failover evidence
Prove failover and failback with automated tests: quorum loss, replica promotion fencing, and restore drills that verify user-visible recovery time.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Allocate dependency SLOs, remove correlated failure, detect gray faults, automate stateful failover/failback, and prove recovery with tests.
Explain the decision
Why are two replicas in one failure domain not highly available?
Read the reasoning
They share power, network, control plane, storage, and deployment risks; redundancy must cross the failures named in the availability target.
Apply it to a production scenario
One zone is slow rather than down and health checks stay green. What handles the gray failure?
Read one approach
Use user-outcome and outlier signals, bounded concurrency, latency-aware routing, zone evacuation gates, and dependency-specific probes.
Work through the numbers
Three serial dependencies are each 99.95% available. Estimate combined availability.
Read the worked calculation
0.9995^3 ≈ 99.85%, or about 65 minutes/month unavailable if independent; shared failures can be worse.
Reason through a failure
What must failback preserve?
Read the recovery reasoning
New and recovered replicas converge before write ownership changes, clients do not oscillate, and stale epochs cannot commit.
Sources and further study
Continue with Timeouts, Retries & Circuit Breakers.