System Design Reliability
SLI, SLO, SLA & Error Budgets
Imagine Black Friday checkout on an ecommerce site. CPU graphs look fine, but one in a thousand payments hang for eight seconds. Users abandon carts. Support tickets spike. Leadership asks if you are "up", and the honest answer depends on whether you measured the journey, not the server.
Reliability is measured, not declared. An is what you measure, an is the target you hold yourself to, an is what you promise customers, and the is how much bad behavior you can afford before you stop shipping risky changes.
Decision path
Pick the journey
Measure checkout, login, or feed load success, not host CPU idle time.
Define good versus total attempts over a clear rolling time window.
Set the reliability target the team will defend when pressure rises.
Ship features while budget remains; freeze risky changes when it burns.
What actually happens when hosts look green
The Black Friday story above is the default whenever you measure boxes instead of journeys. Product wants to ship a payment redesign during peak season. Ops wants fewer pages. Finance wants fewer refunds. Without a shared definition of "good enough," every incident becomes a debate about feelings.
The naive system runs one dashboard: host CPU under 70%, process uptime 100%, HTTP 5xx under 1%. On-call pages when 5xx exceeds a static count. Releases ship whenever CI is green. Then a payment dependency slows to 8 s. The API still returns 200 after a soft timeout fallback that shows "processing." Hosts look idle. 5xx stays low. Cart abandonment spikes 12%. Leadership asks "are we down?" and the honest answer is yes for users, no for the naive dashboard.
A healthy CPU graph does not prove checkout works. You define an on the critical journey: successful payments, completed logins, feed loads under 500 ms: counting each against total attempts in a window. That indicator becomes an : "99.9% successful checkouts over 30 days." The remaining 0.1% is the . When burn is low, product can launch. When spikes, you page, roll back, and pause risky deploys.
The customer-facing sits one step looser. You want to burn internal budget and recover before anyone is owed credits. A load balancer can be 100% up while checkout fails for every user behind a hung dependency: measuring the box, not the journey, trains the team to ignore the real outage.
How to choose what to measure
Once you accept that host metrics can lie, three questions decide what belongs in the reliability control loop.
1. Would a user notice? Prefer journey success and latency over host metrics. A load balancer being "up" does not mean checkout completed.
2. Can you name numerator and denominator? Write them down for the . Ambiguous indicators become argumentative alerts during incidents.
3. What happens when budget is gone? Freeze launches, fix , then resume. An without a freeze rule is a poster, not a control loop.
Simple rule of thumb
Start with one critical journey and one . Translate it into an and alert on , not only raw error counts. Add more SLOs only when a second journey truly needs its own protection. Keep the customer-facing looser so you can remediate before credits are owed.
From problem to mechanism
Here is the same afternoon slowed down so you can see where the naive dashboard fails and what the control loop changes.
Problem. Product, ops, and finance disagree on "good enough" during peak season, so every incident becomes a debate about feelings instead of a shared number.
Naive system. Page on CPU, uptime, and static 5xx counts; ship whenever CI is green. Soft-timeout fallbacks that return HTTP 200 keep the error dashboard quiet while users wait.
Break it. Payment latency jumps to 8 s. Hosts stay green, 5xx stays low, and cart abandonment spikes 12%. Rollback of "server health" alerts cannot help because they never fired.
Mechanism. Name the journey event, define a , set a windowed from the , then encode an freeze rule. Alert on so pages reflect user impact before the monthly average looks bad.
Correctness invariant
Every counts the same user-visible outcome in numerator and denominator: good valid events / all valid events. Invalid client requests are excluded; server errors, timeouts, and policy-defined slow "successes" are not. An policy must change release risk when the budget is exhausted. Otherwise the SLO is a poster, not a control loop.
1. Name the journey event
Define the event before counting it. For example, count each logical checkout attempt that passes basic validation, including attempts that later time out or fail in a dependency. Repeated transport retries for the same checkout should not accidentally become several user journeys unless the indicator intentionally measures requests. State exclusions precisely and report their volume so a growing category of “invalid” traffic cannot hide real product failures.
2. Define good
Define “good” as an outcome a user recognizes, such as one confirmed order within two seconds and no duplicate charge. A response saying “processing” is not automatically good or bad: it depends on whether the product promises immediate completion or a clearly bounded asynchronous workflow. Measure the terminal outcome and pending duration when completion is asynchronous. Instrument abandoned and timed-out attempts so missing responses remain in the denominator.
3. Set the windowed target
Pick 99.9% over 30 rolling days from user tolerance and dependency reality, not from aspirational marketing copy.
4. Encode the freeze rule
Above 50% budget remaining: ship normally. Below 25%: reliability review. At exhaustion: freeze risky launches until burn stops.
Numbers to know: error budget worksheet
A request-based budget and a downtime budget answer different questions. At 99.9%, thirty days allow 43.2 minutes of complete unavailability if time is the measured denominator. The same target on 300 million valid events allows 300,000 bad events, including partial failures spread across the month. Do not exchange those values during uneven traffic without accounting for how many users were affected.
The mechanism above sets a 99.9% target; here is what that budget looks like in minutes and request counts, plus the tiers that should page before the month is already ruined.
Checkout receives 10M valid attempts/day. Target 99.9% over 30 days. Good = completed under 2 s with no duplicate charge (the same definition as above).
| SLO target | Bad budget / 30 days | At 10M req/day |
|---|---|---|
| 99% | ~7.2 hours | 3M bad requests |
| 99.9% | ~43 minutes | 300K bad requests |
| 99.99% | ~4.3 minutes | 30K bad requests |
Event budget
300M × 0.1% = 300,000
Thirty days × 10M/day. Event accounting catches partial failures that pure downtime minutes miss.
Fast burn (14.4×)
1.44% errors for 1h
For a 99.9% objective, the allowed bad-event share is 0.1%. An observed 1.44% bad-event share burns budget at 1.44/0.1=14.4 times the sustainable rate. With constant traffic, one hour consumes 14.4/720=2% of a thirty-day budget. Requiring both a long window and a short window to breach helps detect a sustained incident that is still active, rather than paging on one old burst.
Slow burn (6×)
0.6% errors for 6h
At a 0.6% bad-event share, burn is six times the allowed 0.1%. Six hours at that rate consumes 6×6/720=5% of a thirty-day budget under steady traffic. If sustained, six-times burn exhausts a full budget in five days; 14.4-times burn exhausts it in about fifty hours. Changing traffic volumes affect event-budget accounting, so calculate from actual good and total counts when evaluating a real incident.
Burn-rate alerting tiers
The multi-window approach and its budget-consumption arithmetic are explained in the Google SRE workbook on alerting from SLOs. Adapt the response policy to traffic volume and remaining budget instead of copying thresholds without their assumptions.
Use a fast multi-window condition for urgent impact, such as a high burn rate over one hour that also persists in the most recent five minutes. Use a lower burn threshold over longer windows for a slower incident, with a corresponding shorter confirmation window. The exact thresholds are policy choices; labels such as fast and slow do not by themselves define time to exhaustion.
Make the response actionable. A page should identify the affected journey, current bad-event rate, recent budget consumption, and a mitigation route. A lower urgency ticket can drive investigation of gradual degradation. On sparse APIs, a few failures make ratios noisy, so consider explicit event counts, synthetic journeys, or window-based indicators alongside the request ratio.
Vocabulary in one line for skimmers: an is good/total (or the fraction under a latency bound such as 300 ms); an is the internal target (for example 99.9% successful checkout attempts over 30 days); an is the looser customer promise with credits; the is the allowed bad share (0.1% ≈ 43 minutes or 300K bad requests at this traffic; see the table above).
Failure timeline, and what still breaks after the fix
Putting the worksheet numbers into motion, here is how the hung payment dependency plays out minute by minute, then the ways an improved design can still go wrong.
Timeline: dependency degrades while hosts stay green
T+0 Payment p99 jumps 200ms → 8s
Hosts green; 5xx still low (soft timeout returns 200)
T+5m Journey SLI: 2% bad events → 14.4× burn on 1h+5m windows
PAGE checkout on-call; halt progressive delivery
T+20m Failover to healthy processor; cap retries
Short-window burn falls; long window still elevated
T+2h Duplicate-charge reconciliation finishes
Incident stays open until correctness + short burn clear
T+24h 18% of monthly budget consumed
Freeze high-risk launches until policy exit criteria met
The naive alert never fired. The journey did, because timeouts and soft-fail "successes" counted as bad (see Numbers to know above for the 14.4× fast-burn math).
Still fails when…
- A monthly average hides peak-hour pain, so users suffer while the SLO still looks green.
- Too many SLOs dilute attention until nothing important is truly protected.
- Timeouts are excluded as "no response," recreating the naive blind spot on hung dependencies.
- Five nines are declared without dependency math, spending budget the system never had.
- Multi-window burn flaps on low-traffic APIs; prefer window-based availability for sparse control planes.
- An matching the internal SLO leaves no remediation room before credits.
Trade-offs
- Tighter targets raise trust and cost; looser ones preserve velocity but hide chronic pain.
- Request ratios fit busy APIs; calendar or window availability fits rare control-plane calls.
- Define availability and latency when slow equals broken; avoid double-counting unless the composite is explicit.
Try it: watch error budget burn against your SLO
Set the fraction of requests that fail your SLI definition, then advance simulated time. The budget is the bad-request share your 99.9% SLO allows. Burn rate compares your current failure rate to that sustainable rate; multi-window alerts use the same ratio at different horizons.
Burn rate
okCurrent burn 5.0×. At this rate budget exhausts in roughly 145 more hours at 1k req/h.
Budget ledger
Used 0 of ~720 allowed bad requests in this model.
- Simulated hours
- 0
- Total requests
- 0
- Bad requests
- 0
- Budget left
- 100%
100% error budget remaining · burn 5.0× vs sustainable rate · 0h simulated. You have a 99.9% monthly SLO. Advance simulated hours and raise the bad-request rate to see how fast the error budget burns.
Tie alerts to burn, not raw counts. At 0.5% bad requests the burn is about 5× the 0.1% SLO allows, so a short window exhausts budget even when a monthly average looks fine.
Define bad carefully. Timeouts and soft failures count if users experienced them. This lab uses a single bad-rate slider; production splits availability and latency SLIs explicitly.
Reliability control loop
What production needs to hold
A production design needs a way to recognize trouble and a rule it preserves while recovering. A trigger is the measurement or event that starts a response. An invariant is a statement that must remain true, such as never confirming the same reservation to two customers.
Trigger metric or condition
Use this discipline when teams argue about whether a service is healthy, alerts page on infrastructure noise, or release risk is disconnected from user impact.
Correctness invariant
Every SLI must count the same user-visible outcome in both numerator and denominator: good valid events / all valid events. Invalid client requests are excluded; server errors, timeouts, and policy-defined slow successes are not.
From a user journey to a release decision
Checkout attempt
valid user event
SLI pipeline
good / valid total
30-day SLO
99.9% success
Burn alerts
14.4x and 6x
Budget policy
ship, slow, or freeze
Worked checkout SLO worksheet
A checkout API receives 10 million valid attempts per day. Product accepts a 99.9% success SLO over a rolling 30-day window. A good event completes exactly once in under 2 seconds; user cancellations and syntactically invalid requests are excluded, while timeouts and duplicate charges are bad.
Event budget
300M × (1 − 0.999) = 300,000 bad events
Thirty days contain 300 million valid attempts. The same target would permit 43 minutes 12 seconds of complete unavailability, but event-based accounting also catches partial failures.
Fast burn
14.4 × 0.1% = 1.44% errors
A 14.4x burn sustained for one hour consumes about 2% of the monthly budget. Page only when both the 1-hour and 5-minute windows breach, which filters short spikes.
Slow burn
6 × 0.1% = 0.6% errors
A 6x burn sustained for six hours consumes about 5% of a 30-day budget. Pair 6-hour and 30-minute windows for a ticket or page based on remaining budget.
Decision
The team publishes the SLI query, ownership, exclusions, and window. Above 50% budget remaining it ships normally; below 25% it requires reliability review; at exhaustion it freezes risky launches until the burn is stopped and budget policy exit criteria are met.
How the mechanism holds
Follow how each part protects the rule above. The mechanism explains what the system does; the reasoning explains why that action is sufficient under the stated assumptions. If an assumption changes, the guarantee may change too.
Indicator specification
Record event source, good-event predicate, valid-event predicate, aggregation window, and known blind spots. Prefer edge or synthetic measurements when the application cannot observe failed requests.
Why it holds: Replay a labeled sample through the query and compare counts with raw request logs.
Objective selection
Choose a target from user tolerance and dependency reality, not aspiration. The internal SLO should be tighter than the contractual SLA so remediation starts before credits are owed.
Why it holds: Show that dependency SLOs and redundancy can plausibly meet the journey target.
Multi-window alerting
A short window detects catastrophic burn; a long window prevents a one-minute spike from paging. Require both windows for each severity and alert on burn, not raw error rate.
Why it holds: Backtest the rule against known incidents and harmless traffic dips.
Error-budget policy
Write product actions before an incident: release gates, exception owner, remediation allocation, and recovery conditions. Budget is a risk limit, not spare downtime to spend deliberately.
Why it holds: The change system can show which gate activated and who approved an exception.
Failure timeline: a checkout dependency degrades
Payment p99 rises
Timeouts turn 2% of otherwise valid checkout attempts into bad events while hosts remain green.
Response: Journey SLI captures the impact; dependency latency is supporting evidence, not the page condition.
Fast window confirms burn
Five-minute and one-hour rates cross the 14.4x threshold.
Response: Page checkout on-call, halt progressive delivery, and inspect exemplar traces.
Mitigation lowers errors
Traffic shifts to a healthy processor and retries are capped, but the long window remains elevated.
Response: Keep incident open until short-window burn is normal and duplicate-charge invariant is reconciled.
Policy drives recovery work
The incident consumed 18% of the monthly budget.
Response: Prioritize timeout isolation and failover tests; resume high-risk releases only at the documented threshold.
How to choose among alternatives
Request vs window SLI
Choose when: Use request-based SLIs for APIs and window-based availability for low-traffic control planes.
Caution: A single failure can dominate a low-volume request ratio; windows can hide repeated brief failures.
Rolling vs calendar window
Choose when: Rolling windows give continuous risk visibility; calendar windows align naturally with billing and SLAs.
Caution: Do not mix the alert window and the contractual accounting period without explaining it.
Availability vs latency
Choose when: Define both when slow responses are operationally equivalent to failure.
Caution: Do not double-count one request in combined budget math unless the composite SLI is explicitly defined.
Running it day to day
Measurements tell you what is happening; alerts tell an operator when action is needed. A rollout plan limits how many users encounter a change before it is checked. A runbook records the steps to diagnose and recover from a known failure.
Measure
- good events, valid events, and SLI ratio by journey
- budget remaining and burn rate by alert window
- measurement coverage and unknown-result rate
Alert
- 14.4x burn: 1h AND 5m
- 6x burn: 6h AND 30m
- SLI pipeline delayed or denominator unexpectedly zero
Roll out safely
- shadow the SLI against incident history
- review exclusions with product and support
- start policy in advisory mode before enforcing release gates
Runbook
- confirm the user journey and measurement are real
- stop the burn with rollback, isolation, or traffic shift
- reconcile correctness, record budget consumed, and apply policy
Practice scenario
Design availability and latency SLOs for checkout, including exclusions, alert windows, and what happens after 70% of the budget is consumed.
Questions worth asking next
- What if traffic drops to ten requests per hour?
- How do you allocate a 99.9% journey target across three serial dependencies?
- Would an enterprise SLA use the same target?
A solid answer sketch
Define valid attempts at the edge, success as exactly-one completed order, and latency as the fraction under 2 seconds. Use a rolling 30-day 99.9% objective, 14.4x/6x multi-window alerts, and a policy that pauses high-risk launches at 70% consumption while preserving emergency fixes.
Common mistakes
- • measuring host uptime instead of the journey
- • excluding timeouts because no application response exists
- • declaring five nines without dependency or budget math
Authoritative references
Last reviewed: 6 August 2026
Choosing and explaining SLI, SLO, and error budgets
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
Pick one critical user journey and define numerator, denominator, and measurement window.
Translate the SLO into an error budget and explain what happens when it is exhausted.
Tie alerts to burn rate so pages reflect user impact, not noisy CPU thresholds.
When it belongs
The design discusses availability, latency targets, or release velocity.
You need to justify redundancy cost versus acceptable failure.
Operations must decide freeze vs ship during an incident.
How it fails
SLI measures server health but not user-visible success.
Too many SLOs dilute attention: nothing is truly protected.
Budget exhausted but launches continue because alerts are ignored.
Trade-offs to weigh
Tighter SLOs improve trust but raise cost and slow feature delivery.
Looser SLOs preserve velocity but hide chronic user pain.
Customer SLA penalties should be stricter than internal SLO only at the margin.
Common misunderstandings
Equating uptime of one load balancer with journey availability.
Using monthly averages that hide peak-hour violations.
Declaring five nines without measuring or budgeting for them.
Wrap-up
You have seen the failure, the mechanism, and the burn-rate math. Use these exercises to check the ideas stuck, then follow the references for the primary sources.
Exercises
- Design the checkout pair. Write availability and latency SLOs for checkout: numerator, denominator, exclusions, 30-day window, and the action at 70% budget consumed. Decide whether soft-timeout "200 processing" responses are good or bad, and defend the choice.
- Allocate a journey across dependencies. Journey target is 99.9%. Three serial dependencies each claim 99.95%. Does the math work? If not, what do you change: targets, redundancy, or the journey definition?
- Pick alert windows for a low-traffic admin API. Ten requests per hour. Would 14.4× burn on a 1-hour window page too often? Propose a window-based alternative and say what you would still measure for correctness.
Topic-specific references
Topic sources for SLOs and burn-rate alerting, not generic system-design books.
Key takeaway
SLIs make reliability observable. SLOs make it negotiable. Error budgets connect engineering and product: when budget remains, ship boldly; when budget burns, prioritize stability, rollback, and removal of retry amplification.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Define user-outcome SLIs over valid events, achievable SLO windows, multi-window burn alerts, and an explicit error-budget decision policy.
Explain the decision
Why is “server uptime” often a weak availability SLI?
Read the reasoning
Users care about valid successful operations; a running server can return errors, time out, or omit critical traffic from the denominator.
Apply it to a production scenario
A 99.9% monthly checkout SLO burns 20% of its budget in two hours. What happens?
Read one approach
Page on fast/slow burn policy, stop risky rollout, mitigate user impact, and apply the agreed error-budget release rules.
Work through the numbers
How much bad time does 99.9% allow in a 30-day month?
Read the worked calculation
43.2 minutes if modeling time; event-based SLIs instead budget 0.1% of valid events.
Reason through a failure
What must the SLI denominator preserve?
Read the recovery reasoning
It includes all valid user attempts, including load-shed and timeout outcomes, and excludes only explicitly documented invalid events.
Sources and further study
Continue with Backpressure & Admission Control.