System Design Reliability
Chaos Engineering
The on-call page fires during Friday surge: matching, pricing, and payment each sit on different services. If one AZ hiccups in a real rush, you learn whether redundancy was real, or only drawn on a whiteboard.
schedules that lesson on purpose: define , state a , inject one fault inside a tiny , and abort on .
Experiment journey
Name the user journey and metrics.
Predict what stays healthy.
One pod, shard, or traffic slice.
Inject one fault; abort on guardrails.
Try it: inject one fault inside guardrails
Chaos engineering is a controlled experiment. State the steady-state metrics, inject latency, errors, or pod loss on a narrow blast radius, and stop immediately when SLO guardrails fire.
- Checkout success
- 99.95%
- p99 latency
- 180 ms
- Queue depth
- 12
- Experiment
- idle
Guardrail: success ≥ 99.2%
Current 99.95%
Guardrail: p99 ≤ 900 ms
Current 180 ms
Guardrail: queue ≤ 80
Current 12
Define steady state, pick one fault and a blast radius, then inject. Abort automatically when guardrails on success rate, latency, or queue depth break.
One variable at a time. Mixing a network partition with a code deploy makes learning impossible. This lab injects exactly one fault class per run.
Blast radius is a safety budget. Start with a single pod or shard during staffed hours. Production chaos without guardrails is indistinguishable from an incident you chose on purpose.
What actually happens
Chaos is not random sabotage. You pick a critical journey: request a ride, complete checkout, start a stream: define what looks like, write a measurable , then inject one realistic fault while humans watch dashboards.
If burn or queue age crosses a , you abort and roll back. Surprises become tickets with owners and a follow-up experiment that proves the fix. That learning loop is the product, not the fireworks.
Staging approximates topology but misses real traffic mixes. Production experiments need tighter blast caps, feature flags, and executive buy-in: start narrow during staffed hours, then widen only with evidence.
How to choose what to break
1. Which user journey must survive? Start with matching, checkout, auth, or playback: the promise customers actually buy.
2. How small can the blast be? One pod or one shard before one zone or region. Name the and abort button before injection.
3. What proves success? Tie the experiment to an and burn, not to “servers died and restarted.”
Simple rule of thumb
Never present chaos as destruction theater. Describe a hypothesis, scope, owner, dashboard, and abort button. Expand only after user journeys stay inside the SLO.
In simple terms: break small things on purpose so big things do not break by surprise.
Experiment lifecycle
A good chaos test has an owner, a hypothesis, a small blast radius, real-time guardrails, and a learning loop. The goal is not pain. It is converting unknown failure modes into planned engineering work.
Pick critical user journey
Define steady-state Service Level Indicator
Write hypothesis
Limit blast radius
Inject one fault
Guardrails healthy?
yes
Record evidence
no
Abort and roll back
Fix weakness
Repeat at wider scope
An abort condition is a measured threshold that ends the experiment before its impact exceeds the agreed budget. For example, stop if checkout errors remain above the chosen limit for a short evaluation window, or if queue age threatens order expiry. Test the stop mechanism before injection and make it independent of the component being disrupted. An operator must also be able to stop immediately when the metrics are incomplete.
One pod, one shard, one AZ, one percent of traffic, or one internal tenant before broader production tests.
Learning loop
Every surprise becomes a ticket, owner, deadline, and repeat experiment to prove the fix.
Core principles
Begin with a falsifiable claim about a customer journey, such as “losing one worker leaves accepted orders durable and clears the backlog within five minutes.” Establish normal behavior, inject one controlled fault, and compare the measured result with that claim. Record timing, traffic conditions, and unexpected effects so the result is interpretable. A successful experiment provides evidence for that scenario, not proof that all failures are safe.
1 · Define steady state
- What does "healthy" mean for users and the business?
- Technical guardrails (e.g. success rate > 99.9%, P99 latency < 200 ms).
- North-star KPIs: orders/minute, stream starts/sec, ride matches/min.
2 · Hypothesize
- Example: "Terminating 30% of API pods still returns to steady state via autoscale within 60 seconds."
- Hypotheses must be measurable, bounded in time, and agreed before anyone pulls levers.
3 · Inject realistic faults
- Infrastructure: host loss, AZ isolation, disk fullness, CPU steal.
- Platform: DNS misses, TLS expiry, clock skew, dependency brownouts.
- Workload: synthetic spikes, poison messages, partial data corruption drills.
4 · Observe divergence
- Compare dashboards to baseline; note latency, saturation, error budgets.
- Time-to-recovery and alerting fidelity matter as much as fault injection.
5 · Minimize blast radius
- Begin with one pod, one shard, or a sliver of traffic; automate abort if burn spikes.
- Always stage rollback paths first; graduate staging → canary → AZ → region only with evidence.
Chaos experiment template
A chaos experiment should read like a small scientific plan: who owns it, what user journey is protected, what fault is injected, how small the blast radius is, and when to abort.
Plan fields
- Owner and communication channel.
- User journey and steady-state metric.
- Hypothesis and exact fault.
- Scope: tenant, shard, feature flag, region, traffic percentage, and time window.
Safety fields
- Guardrails tied to SLO burn, P99 latency, queue age, and business metrics.
- Abort button and rollback owner.
- Expected dashboard movement.
- Follow-up proof that the fix worked.
Failure injection patterns
Each pattern teaches a lesson: kill instances, add latency, pretend a region vanished: forcing autoscalers, caches, and client retries to behave under stress.
Instance termination
Terminating a small number of instances tests replacement, load-balancer removal, in-flight request handling, and retry behavior. Select an initial scope whose loss the system is already designed to tolerate. Watch completed business operations as well as pod count: automatic replacement can appear healthy while requests are lost or survivors overload. Restore the baseline before using the result to justify a larger experiment.
Latency injection
Injects artificial delays into service-to-service calls. Simulates slow networks and overloaded databases. Exposes missing timeouts and . Inject delay into one named dependency or traffic cohort so the causal effect is visible. Check that deadlines stop useless work, retries stay within budget, and unrelated endpoints retain their resources. A slow dependency is often more revealing than a clean failure because it holds connections and workers open. Remove the fault and observe backlog recovery, not only the initial response.
Region isolation
Simulates an entire region failure. Redirects traffic to surviving regions and validates cross-region failover end-to-end.
Conformance scan
Checks instances follow best practices: auto-scaling groups, health checks, proper tagging. Non-conforming instances are flagged or terminated.
Security configuration scan
Scans for security misconfigurations such as public buckets and overly permissive roles. Alerts security responders for violations.
Resource cleanup scan
Identifies and cleans up unused resources: orphaned volumes, idle load balancers, and stale snapshots.
Chaos tooling categories
Managed experiment platforms, orchestration-native controllers, and cloud fault-injection APIs standardize experiments: pick tooling that matches where your workloads actually run.
Managed experiment platform
Hosted chaos platform with a UI for designing and executing experiments. Supports resource attacks (CPU, memory, disk), network attacks (latency, packet loss, DNS), and state attacks (process kill, time travel, shutdown).
Useful for regulated teams that want built-in safety controls, audit trails, and automatic halt when metrics breach thresholds.
Orchestration-native controller
Defines chaos experiments as custom resources and provides a library of pre-built experiments: pod-delete, node-drain, network-loss, disk-fill.
Integrates with CI/CD pipelines so chaos tests run in staging before production deploys. Supports scheduled workflows and automated steady-state validation with metrics queries.
Game days: organized chaos
Cross-functional align execs, support, and engineers around scripted failures: great for uncovering documentation gaps and brittle dependencies before customers notice.
Before game day
- Define scope: "Test payment service resilience"
- Identify steady state metrics and thresholds
- Prepare runbook: who does what when things break
- Brief all participating teams
- Ensure rollback mechanisms are tested and ready
During game daySample morning timeline: pair each fault with explicit observables
09:00 · Baseline
Capture at least one hour of steady metrics before touching anything.
10:00 · Experiment 1
Begin payment experiments with an internal test tenant or a very small, explicitly approved workload slice. Remove only capacity known to be redundant and confirm that accepted operations remain traceable and deduplicated. Measure surviving capacity, retry traffic, and pending-payment age. Larger pod-loss scenarios should follow evidence from smaller drills; “kill 50%” is not a safe default for an arbitrary production fleet.
10:30 · Experiment 2
Inject ~500 ms DB latency → validate timeouts, hedging limits, cache degradation banners.
11:00 · Experiment 3
Force primary DB failover → verify connection churn, replica promotion, writes draining safely.
11:30 · Combined fault
Combine faults only after individual experiments have established what each one does and how it is stopped. A compound scenario might pair a worker loss with a modest dependency delay to reveal retry amplification. Keep an explicit impact budget and timeline for each injection. If several faults begin together with no baseline, it becomes difficult to identify which change caused a failure or which repair addresses it.
After game day
- Document findings: what broke, what held
- Create action items with owners and deadlines
- Update runbooks with new failure modes discovered
- Schedule follow-up game day to verify fixes
Frequency at scale
Blast radius planning
The best answers name the blast radius before the fault: who can be affected, how the experiment stops, and which SLO proves the system stayed healthy.
| Scope | Example fault | Abort guardrail | Learning goal |
|---|---|---|---|
| One pod | Kill instance | P99 +20% for 2 min | Health checks and rescheduling. |
| One shard | Add 500 ms DB latency | Error budget burn > 5x | Timeouts, breakers, and fallback data. |
| One zone | Drop service discovery | Success rate < 99.5% | Zone failover and reconnect jitter. |
| One region | Block routing | Manual incident commander halt | DR plan, capacity headroom, data lag. |
Blast radius budget
Start at one instance or one shard, limit to 1% internal traffic, run during staffed hours, and abort if burn rate or queue age crosses the agreed threshold. Expand only after the previous level passes.
expected bad requests = traffic percentage × request rate × experiment duration × expected failure rate
Chaos in production vs staging
Staging approximates topology but misses traffic mixes and shadow configs. Production experiments demand tighter blast caps, feature flags, and organizational trust: start narrow during low-risk windows.
Staging chaos (good starting point)
Pros
- Zero customer impact if experiment goes wrong
- Lower stakes, easier organizational buy-in
- Good for validating new chaos experiments
Cons
- Staging ≠ production (different scale, data, traffic)
- Missing real traffic patterns, load, and dependencies
- "Works in staging" doesn't guarantee production resilience
Production chaos (the gold standard)
Pros
- Tests real infrastructure with real traffic
- Discovers failure modes staging can't reproduce
- Builds genuine confidence in production resilience
Cons
- Risk of customer impact (must minimize blast radius)
- Requires mature monitoring and quick rollback
- Organizational trust and blameless culture needed
Maturity ladder
- Level 1 · Manual staging drills: ad-hoc faults when someone remembers; good for learning tooling.
- Level 2 · Automated staging: pipelines replay faults nightly; catches regressions before prod.
- Level 3 · Production game days: humans supervise scripted failures with rollback ready; minimum bar for serious teams.
- Level 4 · Automated prod experiments: scheduled instance-termination jobs with guardrails.
- Level 5 · Continuous prod chaos: always-on injection paired with automatic halt triggers and mature operating discipline.
Most teams should prioritize reaching Level 3+ with measurable error budgets before chasing full automation.
What to measure during chaos
Measure user-visible behavior first, then technical symptoms. A server dying is acceptable if orders, login, and playback stay inside their SLOs. Also observe the recovery period. A five-minute fault can create a backlog that overloads the database for another hour after the injected fault is gone. Measure oldest work age, duplicate effects, restored capacity, and time to normal latency. Count the experiment successful only when the protected invariant holds and the system returns to a stable state under continuing traffic.
Availability
Success rate, failed checkouts, stream starts, auth success.
Latency
P95/P99, timeout rate, queue age, cache hit latency.
Saturation
CPU, DB connections, thread pools, retry traffic, breaker state.
Operations
Alert quality, time to acknowledge, runbook accuracy, rollback time.
Failure scenarios chaos should find
Stories become useful when you cite what broke, what detector fired, and what engineering changed, not vendor slogans.
Findings catalog
Strong chaos programs track repeatable failure themes: missing timeouts, retry amplification, under-provisioned standby capacity, cache stampedes, split-brain risk, alerts that did not fire, stale runbooks, and degradation flags that propagated too slowly.
Regional outage readiness
Region-isolation experiments prove traffic can move to surviving regions before a real regional outage. The signal is not "no servers failed"; it is that users stay inside the SLO.
Traffic spike readiness
Game days can reveal DB connection leaks. Under simulated 10x traffic, connection pools exhaust; teams fix leaks before launch events instead of discovering them during peak load.
Name resolution failure
A global name-resolution drill can uncover services with hardcoded resolvers or missing cache behavior. The fix is resilient discovery and sane TTLs.
Cache restart herd
Chaos testing can reveal that cache restarts trigger all services to query the database simultaneously. Add jittered warm-up and request coalescing to prevent cascading failure.
How to talk about safe chaos
- Never present chaos as random destruction; describe a hypothesis, scope, owner, dashboard, and abort button.
- Use chaos to validate earlier design claims: redundancy, failover, retry limits, graceful degradation, and alerting.
- Start with low-risk faults, then increase scope only after evidence shows user journeys stay inside SLO.
- Close the loop with action items and repeat experiments; otherwise game days become theater.
Key takeaway
At scale, failures are not a matter of if but when. Chaos engineering turns unknown failure modes into known, tested, handled scenarios. Start with game days in staging, graduate to guarded production experiments, and keep the learning loop closed. The investment pays for itself with the first prevented outage.
Deepening: experiment maturity
Write a steady-state hypothesis, define abort gates on SLO burn, and capture a remediation backlog after every experiment. Hidden coupling often appears only when a dependency is slow rather than fully down.
Primary sources
- Principles of Chaos Engineering: hypothesis-driven fault injection.
- Netflix: The Netflix Simian Army: production chaos origins.
- Google SRE: Embracing Risk: error budgets as the gate for experiments.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Test a steady-state hypothesis with scoped faults, automated abort gates, measurable learning, remediation ownership, and maturity progression.
Explain the decision
What makes chaos engineering different from random breakage?
Read the reasoning
It starts with a falsifiable steady-state hypothesis, controls blast radius, observes user outcomes, aborts safely, and turns findings into fixes.
Apply it to a production scenario
A zone-loss experiment starts increasing checkout errors. When do you abort?
Read one approach
At the predeclared user-impact or safety threshold; preserve evidence, restore capacity, and investigate instead of pushing through the gate.
Work through the numbers
A 1% cohort receives 80,000 requests/s. How much traffic is exposed to an experiment?
Read the worked calculation
800 requests/s; also cap affected tenants, regions, and duration because percentage alone does not bound critical-user impact.
Reason through a failure
What must an experiment controller guarantee?
Read the recovery reasoning
Abort and rollback remain available when the injected dependency/control path fails, and experiments cannot escape their declared scope.
Sources and further study
Continue with Stress and Chaos Testing Methods.