System Design Reliability
Availability Concepts
A payment API that is “up” but times out at 3 seconds still fails the shopper. A dashboard that shows green while 2% of checkouts error is lying. is not a logo claim; it is a measured success rate for a named user journey, plus the architecture and process that keep that rate inside budget.
This note builds the vocabulary: , / / , , vs , and how dependency math and redundancy turn those numbers into design choices.
From slogan to budget
Name what “good checkout” means in numbers.
Pick the internal target and window.
Spend unreliability deliberately—or freeze.
Cut
Detect, failover, and degrade faster than failures get rarer.
What actually happens
Teams say “three nines” when they mean roughly 99.9% successful operation over a period. Translate that into downtime: about 8.8 hours per year, or ~43 minutes per month if the budget is flat. At high QPS, those minutes are millions of failed requests. The useful move is to define an on a critical journey (not raw machine ping), set an , and treat the as spendable currency for change velocity.
is broader than uptime. A system can answer quickly with wrong balances and still destroy trust. Availability work therefore pairs with correctness and durability goals. Architecturally, you raise availability by increasing (fewer failures) and decreasing (faster restore). Production experience usually shows that detection + + runbooks improve the user-visible number faster than chasing zero bugs.
Math bites when services call each other in series. A long multiplies independent failure probabilities. and break that chain: parallel capacity absorbs one AZ loss; shedding non-critical leaves keeps checkout alive when recommendations die.
How to choose the availability story
1. Which journey is critical? Checkout and login deserve stricter SLOs than recommendation carousels.
2. What counts as success? Include latency deadlines in the so “up but unusable” does not look green.
3. Where is the blast radius? Multi-AZ before multi-region; region-level only when RTO/RPO demand it.
Simple rule of thumb
Do not promise a nine you have not measured. Keep looser than , spend the consciously, and invest first in MTTR: paging, dashboards, and rehearsed failover beat vanity four-nines slides.
From SLI to degradation
Availability is a control loop: measure, budget, recover, and shed optional work when the budget burns.
Read the diagram definition
flowchart TD
A[User journey: checkout] --> B[Define SLI]
B --> C[Set SLO + error budget]
C --> D{Budget healthy?}
D -->|Yes| E[Ship features with risk]
D -->|Burning| F[Freeze risky deploys]
F --> G[Fix MTTR / redundancy]
G --> C
E --> H[Redundancy + health checks]
H --> I[Failover rehearsed]
I --> J[Graceful degradation]
J --> K[Observe burn rate]
K --> D
Failover as an availability mechanism
Redundancy only counts when traffic actually moves and the unhealthy writer is fenced.
Read the diagram definition
sequenceDiagram
participant U as Users
participant LB as Load balancer
participant AZ1 as Zone A primary
participant AZ2 as Zone B standby
participant HC as Health checks
U->>LB: Traffic
LB->>AZ1: Route writes/reads
HC--x AZ1: Missed heartbeats
HC->>LB: Mark AZ1 unhealthy
Note over AZ2: Promote / take traffic
LB->>AZ2: Shift traffic
AZ2-->>U: Serve with possible brief errors
Note over AZ1: Fence old primary so it cannot write
Nines table (and what they cost)
| Availability | Downtime / year | Typical posture |
|---|---|---|
| 99% (two nines) | ~3.65 days | Single region, manual recovery OK for many internal tools |
| 99.9% (three nines) | ~8.8 hours | Multi-AZ, automated failover for critical paths |
| 99.99% (four nines) | ~52.6 minutes | Harder: multi-region or very fast MTTR + strict change control |
| 99.999% (five nines) | ~5.3 minutes | Rare; often only for a few tightly scoped journeys |
Each extra nine is not a linear cost. Cross-region active-active, global traffic management, and continuous chaos testing appear when the budget shrinks to minutes.
SLI, SLO, SLA, error budget
An is a ratio you can compute from telemetry. An is the target for that ratio. An is what legal commits externally. The is 1 − SLO: if the SLO is 99.9% over 30 days, you may “spend” ~43 minutes of equivalent downtime (or the matching failed-request fraction) before you should slow feature launches.
Burn rate intuition
If a bad deploy burns a week of budget in two hours, page humans and roll back—even if absolute error rate still looks “small.” Budget is about rate of spend, not only the absolute remaining minutes.
MTBF, MTTR, and the simple availability equation
For a two-state (up/down) model, availability ≈ / ( + ). Doubling MTBF helps; halving MTTR helps the same percentage-wise when MTTR is the large term during incidents. Many teams underinvest in detect (good alerts), decide (runbooks), and act (one-click failover).
Example: MTBF = 720 h (one failure / month) MTTR = 1 h Availability ≈ 720 / 721 ≈ 99.86% → not yet three nines Cut MTTR to 15 min (0.25 h): Availability ≈ 720 / 720.25 ≈ 99.965% → past three nines without rarer failures
Dependency math
Independent serial dependencies multiply. Parallel alternatives improve the joint availability if failover works.
| Topology | Rough end-to-end | Design move |
|---|---|---|
| 3 serial deps at 99.9% each | ≈ 99.7% | Collapse path or make a leaf optional |
| 2 parallel AZs, each 99.9%, independent | ≈ 99.9999% if either can serve | Needs real failover; correlated failure still hurts |
| Critical + optional enrichment | Critical SLO stays if enrichment times out |
Correlated failures (shared power, bad deploy across all AZs, global DNS outage) destroy the independence assumption. Design for that with staged rollouts and blast-radius limits.
Failure timeline: green dashboard, red checkout
T0 SLI is "host ping OK" → 100% green while checkout p99 is 8s T1 Payment provider blips; retries amplify; thread pools exhaust T2 Error budget for real checkout SLI burns a week in 40 minutes T3 No degradation: homepage recommendations keep calling sick deps T4 Fix: journey SLI with latency, circuit break enrichment, rehearse AZ failover Outcome Availability is the journey metric + recovery path, not machine heartbeat
Break the improved design
Multi-AZ plus a good SLO still fails when health checks are shallow (process up, DB hung), when failover forgets fencing and creates dual writers, or when the is ignored and every Friday ships risk.
Multi-region active-active without a consistency plan trades availability for split-brain inventory. Measuring availability only on synthetic probes misses regional user pain. Partial outages that affect 5% of users can look fine on global averages.
Trade-offs
Extra nines vs engineering and infra cost
Each additional nine shrinks allowed downtime roughly 10× and usually demands multi-AZ or multi-region redundancy, stricter change control, and continuous testing. Promise nines only for journeys that justify the spend; keep looser SLOs for non-critical surfaces.
Higher MTBF vs lower MTTR
Preventing failures (safer deploys, better tests) raises MTBF. Faster detection and failover lowers MTTR. At scale both matter; many products get more user-visible availability from cutting MTTR first because some failure is inevitable.
Feature velocity vs error budget
Error budgets make the trade explicit: burn budget on launches when healthy; freeze when burning fast. Without a budget, reliability becomes either infinite caution or ignored until a customer SLA breach.
Full feature set vs graceful degradation
Serving a reduced product during incidents protects the availability SLI for critical journeys. Product and UX must accept “recommendations unavailable” states; otherwise every leaf dependency becomes a hard outage.
Operational availability vs CAP availability
Nines and SLOs measure successful user journeys. CAP availability is a partition-time formal property about answering without coordination. A CP inventory path can be “unavailable” under split while annual uptime still looks excellent. Keep the terms separate in interviews.
Choosing and explaining availability concepts
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
I define availability on a user journey SLI that includes a latency deadline, not host pings.
I set an SLO tighter than the SLA and manage an error budget so reliability and velocity share one dial.
I improve availability by cutting MTTR with redundancy and failover, and by degrading non-critical dependencies.
When it belongs
The design must state uptime goals or multi-AZ / multi-region needs.
The interviewer asks how you know the system is reliable enough to ship.
A dependency chain threatens to multiply downtime.
How it fails
Shallow health checks leave hung dependencies in rotation.
Failover without fencing creates dual writers.
Global averages hide regional or cohort outages.
Trade-offs to weigh
Extra nines versus cost and operational complexity.
Feature velocity versus error-budget burn.
Full UX versus graceful degradation during incidents.
Common misunderstandings
Confusing CAP availability with uptime SLOs.
Promising four nines without measuring a journey SLI.
Treating redundancy as enough without rehearsed failover.
Wrap-up
You can now turn “highly available” into numbers, budgets, and recovery design. Practice the dependency and MTTR exercises aloud.
Exercises
- Write the SLI. For mobile checkout, define success and the latency deadline. What is excluded (bots, client cancels)? What window (30d rolling)?
- Budget burn. SLO 99.9% / 30d. A 45-minute total outage happens on day 3. How much budget remains? Do you freeze deploys?
- Serial deps. API → auth → payments → fraud, each 99.95% independent. Estimate joint availability and name two design changes that raise the journey SLI without four-nines every leaf.
Primary sources
- Google SRE: Service Level Objectives : SLI/SLO/SLA and error budgets.
- SRE Workbook: Implementing SLOs : practical measurement.
- AWS Builders’ Library: Static stability : AZ redundancy patterns.
Key takeaway
Availability is a measured journey success rate managed with SLOs and error budgets. Raise it by shortening MTTR, adding real redundancy with failover, respecting dependency math, and degrading gracefully. Nines are a cost curve, not a marketing badge.