System Design Reliability
Disaster Recovery
Imagine checkout at peak hour: cards are authorizing, orders are writing, then an entire cloud region goes dark. Users do not care which Availability Zone failed. They care whether their paid order still exists and when the store comes back.
is the plan for that moment: how much data you may lose (), how fast traffic returns (), and which path you rehearse before the real fire.
Recovery clock
Last safe commit vs the disaster clock.
Declare incident
RTO starts when leadership calls DR.
Safe replica becomes writer.
Restore traffic
Verify, communicate, then open writes.
Try it: dial RPO and RTO through a regional failover
Move time forward, lose a region, and walk declare → promote → restore. Compare measured data loss and recovery duration against the RPO and RTO you chose.
- Business clock
- T+0 min
- Data loss window
- 0 min
- RPO met?
- —
- RTO met?
- —
Recovery timeline
- Last backup · T+0 min
Advance the business clock, then trigger a regional disaster. RPO sets how far back the last safe backup may sit; RTO is how long users wait after you declare DR.
RPO faces backward. It bounds how much committed business data you are willing to lose when the disaster clock stops. Backups alone do not shorten RPO unless they capture commits frequently enough.
RTO faces forward. It starts when leadership declares DR, not when the first pager fires. Runbooks, DNS, and fencing dominate the minutes you spend before users can write again.
What actually happens
A single server reboot is high availability. A region loss, ransomware event, or accidental delete is . You need off-site copies, a documented , and a rehearsed : preferably with so the old primary cannot keep writing.
looks backward: how far back is the last recoverable data point? looks forward: how long until users can work again? Confusing the two causes teams to buy backups when they needed hot capacity, or hot capacity when they needed better replication.
At scale, automation and quarterly matter as much as topology. A short partition without fencing can explode into a day of reconciliation. That is .
How to choose a DR posture
1. What is the business tier? Payments and medical records may need near zero. Analytics can often tolerate hours. Profile updates often sit in between.
2. How fast must traffic return? Minutes of push you toward or . Hours allow or .
3. Can you prove restore? A backup policy without a timed restore and a does not satisfy RTO. Untested copies are not protection.
Simple rule of thumb
Set RPO and RTO per workflow, not one number for the whole product. Automate promotion with fencing, keep runbooks out-of-band, and rehearse region failure until actual recovery matches the promise.
In simple terms: buy the recovery you can prove, not the architecture diagram that looks safest on a slide.
RPO and RTO in plain language
answers "how much data may we lose?" answers "how fast must we be back?" Together they drive sync vs async replication and how warm the standby must be.
RPO bands
Recovery point objective, or RPO, is the maximum acceptable gap between the failure and the newest data that can be recovered. If failure occurs at 10:00 and the last recoverable commit is 09:59:20, the actual loss window is 40 seconds. It says nothing about how long restoration takes. Measure the recoverable point itself; a healthy replication process with growing lag may already violate the objective.
RPO 0
Synchronous replication. Highest write latency cost. Payments and financial ledgers.
RPO ~1 min
Near-real-time async replication via or streaming. Orders and user data.
RPO ~1 hr
Hourly snapshots or WAL shipping. Analytics and reporting systems.
RPO ~24 hr
Daily backups. Lowest cost. Dev/staging and archival data.
RTO bands
Maximum acceptable downtime until service returns.
RTO ~0
multi-region. Traffic shifts almost instantly for globally critical journeys.
RTO <1 min
with auto-failover and near-real-time replication.
RTO <1 hr
, scale up on fail. Internal tools and many B2B services.
RTO <24 hr
Cold standby or restore from backup. Batch jobs and archival systems.
The recovery clock
RPO looks backward to the recoverable data point; RTO bounds how long recovery may take. Define the RTO clock’s start explicitly, normally from the disruption or loss of service, so a slow incident declaration cannot hide customer downtime. Teams may separately track declaration-to-restore execution time. In the example below, recovery execution takes nine minutes, but the user-visible outage lasts twelve minutes.
Disaster recovery clock
1
Last good backup or replica point: RPO window starts
2
Disaster happens: writes may be lost until recovered point
3
Incident declared: RTO clock starts
4
Restore data and promote services: runbooks, automation, verification
5
Traffic restored: RTO clock stops
Low RPO needs data work
Synchronous replication, WAL/binlog shipping, streams, immutable backups, and regular restore verification.
Low RTO needs serving work
Hot capacity, automated routing, pre-baked images, dependency order, status communications, and rehearsed .
Worked incident: regional database failure
Targets are promises; actual recovery is measured from the timeline. Separate first user impact, declaration, data recovery point, traffic restoration, and reconciliation.
Regional database failure example
09:59:20
Last replicated commit in DR
10:00:00
Primary region fails
10:03:00
Incident declared
10:09:00
DR database promoted
10:12:00
Traffic restored
10:20:00
Reconciliation starts
Actual RPO
Failure at 10:00:00, last DR commit at 09:59:20. Actual data loss window is 40 seconds.
Actual RTO
Declared at 10:03, traffic restored at 10:12. Recovery from declaration is 9 minutes.
User downtime
First impact at 10:00 and restore at 10:12. User-facing downtime is 12 minutes.
RPO and RTO by workflow
A single product may need near-zero loss for payments, minutes for profiles, and hours for analytics.
| Workflow | RPO | RTO | Design implication |
|---|---|---|---|
| Payments / orders | 0-5 seconds | 1-5 minutes | Synchronous or near-sync replication, idempotency keys, reconciliation. |
| User profile | 1-5 minutes | 15 minutes | Async change stream and temporary read-only mode. |
| Recommendations / search index | 15-60 minutes | 30-60 minutes | Rebuild from source of truth; serve stale or basic results. |
| Analytics | Hours | Next business day | Batch replay is acceptable; never block user traffic. |
DR strategies compared
Strategies range from "restore from backup" to "live traffic on two continents." Pick based on business tolerance for data loss and outage length, not every SKU needs hot standby.
| Strategy | RTO | RPO | Cost posture* | Use case |
|---|---|---|---|---|
| Backup / restore | Hours–days | Hours | Lowest steady cost | Non-critical |
| 1–2 hours | Minutes | Low-to-medium | Moderate priority | |
| 15–60 min | Minutes | Medium | Important services | |
| <1 min | Near-zero | High | Mission-critical | |
| ~0 | ~0 | Highest complexity | Billion-user journeys |
* Cost depends heavily on traffic, data size, staffing, and compliance; compare using business impact per outage minute.
Pilot light, warm standby, hot standby
Think of dimmer switches on spare infrastructure: each step reduces but raises the monthly bill.
Core data paths stay alive in DR; compute stays off or tiny until declaration. Failover boots capacity from gold images.
Typical sequence
- Promote read replica to writer (~2 min).
- Launch app tier from pre-baked machine images (~3 min).
- Scale workers to target footprint (~5 min).
- Flip DNS / traffic manager (~1 min).
Ballpark: ~10–15 min RTO; low steady cost but more recovery steps.
Always-on scaled-down stack: often ~10–20% of peak. So pools and configs stay warm; failover is mostly scale-out plus routing.
Typical sequence
- Raise auto-scaling floor toward full capacity (~5 min).
- Shift traffic via DNS / LB (~1 min).
Ballpark: ~5–10 min RTO; medium baseline cost because partial capacity is already running.
A hot standby already has the services, configuration, and substantial capacity needed to serve traffic. Its data may use synchronous or asynchronous replication depending on the design. Promotion still needs to establish safe write ownership, confirm recoverable data, update routing, and test the customer journey. Those steps determine actual recovery time; having idle servers does not make the switch instantaneous.
Typical sequence
- Health checks detect primary loss (~30 s).
- Automated DNS/LB shift to DR (~60 s).
Ballpark: ~1–2 min RTO; high steady cost because near-full capacity is already running.
Automated failover mechanisms
Humans typing DNS changes during a fire drill do not scale. Health checks, load-balancer policies, and managed database failover keep promotion repeatable, still with awareness.
Health checks + DNS failover
Monitor primary every ~10s → three consecutive failures mark unhealthy → DNS returns DR IP → with TTL ~60s most clients switch within a few minutes.
Cross-region database replication
A database recovery procedure detects primary failure, confirms that a candidate can be promoted safely, fences the old writer, and directs clients to the new one. With asynchronous replication, acknowledged writes not yet received by the candidate can be lost. Report measured replication lag and a recovery point instead of assuming every streaming setup achieves sub-second RPO or sub-minute recovery.
Active-active replicated tables
Any healthy region can accept writes; replication lag is typically under a second; no manual failover for many read paths.
Synchronous multi-zone database
Sync replication to a standby in another AZ → auto failover often 60–120s → DNS endpoint unchanged → RPO 0 within the zone pair.
Failover decision path
Declaring DR is a product and business decision, not only a monitoring event. Weigh customer impact, data safety, confidence in the backup region, and the cost of .
Major incident detected
Single component?
yes
Use local failover / rollback
no
Assess region / data plane
If not a single-component fault
Region or data plane impaired?
no
Degrade features + shed load
yes
Check DR readiness →
If region or data plane impaired
DR region data safe and caught up?
no
Freeze writes, preserve evidence, restore from known point
yes
Check survivor capacity →
If DR data is safe
Enough capacity after shift?
no
Enable degraded mode, scale standby, then shift
yes
Declare DR and route traffic
Verify, communicate, prevent split-brain
Million-user recovery plan
A good recovery plan names the first protected user journeys, expected data loss, blast radius controls, and when to run degraded mode instead of full failover.
0–5 min
Declare incident, freeze deploys, block unsafe writes if split-brain risk exists.
5–15 min
Promote safe replica, shift a small traffic slice, verify login/read/write success.
15–45 min
Scale standby, enable basic search and cached recommendations, limit media previews.
After recovery
Reconcile writes, rebuild indexes, restore normal flags gradually, publish lessons.
Data backup strategies
Backups are only real after a restore test: prove checksums, dependency order, and encryption keys. Automated snapshots still fail silently if nobody restores quarterly.
Full backup
A full backup records a complete recoverable copy at a defined point. It simplifies the restore chain but requires substantial storage and transfer work. For a hypothetical 10 TB database restored at a sustained 1 GB/s, reading the bytes alone takes about 2.8 hours before replay, index work, and validation. Use measured effective throughput and dependency startup time to estimate recovery, not database size alone.
Incremental
Only changes since the last backup (full or incremental). Smallest backup size. Restore needs full plus all incrementals in order: one corrupted link breaks later ones.
Differential
All changes since the last full backup. Grows over time but restore needs only full plus latest differential: a middle ground on size vs restore simplicity.
Restore validation checklist
- Backup can be read and checksum matches.
- Encryption keys and secrets are available.
- Schemas are compatible with application version.
- Indexes and derived tables are rebuilt or marked stale.
- Counts, checksums, and sampled records look plausible.
- Canary workflow can read and write before traffic shift.
Tier-1 backup rhythmFull snapshots + incrementals + streaming replication
- Weekly fullBaseline image to object storage with cross-region replication.
- Daily incrementalsWAL / binlog / block deltas shipped between fulls to shrink backup windows.
- Continuous CDCNear-real-time change streams into DR databases so stays sub-minute when required.
Point-in-time recovery
Point-in-time recovery restores a base backup and replays the required write-ahead log or binary log up to a chosen safe boundary. After an accidental deletion at 10:05, the goal may be to recover just before that change rather than replay it again. Retain an unbroken log chain and the encryption keys needed to read it, then validate business records in an isolated restore before replacing serving data.
3-2-1 rule
3 copies of data, 2 different storage types, 1 offsite copy.
Game days: prove the plan
rehearse pulling the plug on purpose: validate monitoring, communication templates, and leadership decisions without waiting for a real disaster.
An untested DR plan is not a DR plan. Mature teams regularly simulate disasters in production-like environments.
Game day structure
- Define scenario (region failure, DB corruption, ransomware)
- Assemble response team (on-call, leadership, comms)
- Execute controlled failure injection
- Time the response: detection → diagnosis → mitigation → recovery
- Measure actual RTO/RPO vs target
- Post-mortem: gaps found, runbook updates, automation opportunities
Region failure drill
Simulates an entire region going offline. Run quarterly to find failover gaps, stale runbooks, and hidden dependencies.
How often to rehearseCadence that keeps muscle memory fresh
- Tabletop / narrative drillMonthly: cheap, surfaces comms gaps.
- Single-component failoverWeekly via automation: prove detectors + runbooks.
- Whole-region failoverQuarterly: validates DNS, data promotion, staffing.
- Full DR reset testSemi-annual: timeboxed proof of RTO/RPO with executives observing.
Runbooks and communication plans
When primary collaboration tools are broken, maintain status pages, phone bridges, and premade customer wording so responders execute a instead of improvising prose under stress.
Runbook essentials
Every DR runbook must include
- Trigger criteria (when to activate)
- Step-by-step commands (copy-paste ready)
- Verification steps after each action
- Rollback procedure
- Escalation contacts + phone numbers
- Dependencies and order of operations
- Estimated time for each step
Store runbooks outside the production stack, version controlled, and available during outages.
Communication plan
Internal timeline
External timeline
Out-of-band: phone tree, SMS: don't depend on the system that's down.
Incident lessons
Real outages often emphasize , silent backup failures, and tooling dependence: extract actionable controls like , restore drills, and out-of-band access.
Network partition with split-brain
A short network partition caused the database primary to failover. Writes continued to the old primary. Result: divergent replicas. Recovery took 24+ hours to reconcile without losing user commits. Lesson: automated failover without fencing can cause split-brain.
Accidental deletion with silent backup failures
An operator accidentally deleted a production database. Multiple backup methods had all failed silently. Only one older snapshot worked, so several hours of data were lost. Lesson: test backups regularly. Untested backups are not backups.
Logical disaster path
If a bad deploy corrupts data, failover alone is unsafe because replicas may already contain the corruption. Freeze writes, restore to a point before corruption, replay safe events, rebuild derived indexes, and reconcile user-visible changes carefully.
How to talk about DR tradeoffs
- State the business tier first: payments may need RPO near zero; analytics can often tolerate hours.
- Explain why is not always best: conflict resolution, compliance, cross-region latency, and operational complexity can dominate.
- Call out restore testing: a backup policy without proven restore time does not satisfy RTO.
- Mention and : automatic failover can make recovery worse if old primaries keep accepting writes.
Key takeaway
DR at scale needs automated failover with fencing, continuous replication for tight RPO, regular game days, and battle-tested runbooks. Hot standby costs money; extended downtime costs more. Untested backups are not backups: and failover without fencing can create a worse problem than the outage you meant to fix.
Deepening: ransomware and failback
Immutable backups and dependency tiering
Tier services by recovery priority. Keep immutable, offline-capable backups so ransomware cannot encrypt the only restore path.
Failback and reconciliation
DR is incomplete without failback. After restore, reconcile ledgers, queues, and search indexes, then keep game-day audit evidence. Failback is another migration: the original environment may now be stale while the recovery environment owns all new writes. Rebuild it from the current authority, catch up replication, compare business state, and transfer ownership with fencing. Preserve the incident timeline and reconciliation results. Switching routing back before data catches up can lose the very writes the recovery environment successfully protected.
Primary sources
- Google SRE: Disaster Recovery: DiRT-style recovery drills.
- AWS: Disaster Recovery Options: backup/pilot-light/warm-standby/multi-site.
- NIST SP 800-34: Contingency Planning: RTO/RPO contingency framing.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Tier dependencies by RPO/RTO, keep immutable recoverable backups, automate regional recovery/failback, reconcile data, and capture game-day evidence.
Explain the decision
What is the difference between RPO and RTO?
Read the reasoning
RPO bounds acceptable data loss by recovery point; RTO bounds time to restore the service. Architecture and drills must demonstrate both.
Apply it to a production scenario
Ransomware encrypts production and online backups. What recovery path remains?
Read one approach
Use offline/immutable copies, separate credentials and control plane, known-clean restore environments, key recovery, and a tested business-priority order.
Work through the numbers
Restoring 240 TB over a sustained 20 Gb/s link takes how long ideally?
Read the worked calculation
240 TB × 8 / 20 Gb/s ≈ 96,000 seconds, about 26.7 hours, before protocol, validation, and index rebuild overhead.
Reason through a failure
What must a DR exercise record?
Read the recovery reasoning
Observed RPO/RTO, missing dependencies, manual steps, integrity checks, reconciliation results, owners, and remediation deadlines.
Sources and further study
Continue with Chaos Engineering.