System Design Data

Multi-Region Data

Mumbai edits an Instagram bio; New York updates the same profile caption at the same moment. Placing data near users makes the feed feel local, until both regions accept writes and you must decide whose edit survives.

Multi-region systems cut latency and survive outages. The hard part is consistency: sync cross-region commits pay ocean RTT (round-trip time) on every write; async copies add conflicts. Pick topology per entity, not one global slogan.

Decide per entity

Classify the data

Profile, ledger, and session each need a different write and merge story.

Name how much data loss and downtime you can accept before drawing boxes.

Choose a single writer region, or open active-active where latency demands it.

Conflict rules

Pick LWW, version vectors, or CRDTs before the outage, not during it.

Try it: resolve multi-region writes after a partition

Write in both regions while they cannot talk. Last-write-wins uses logical clocks and can drop an update on ties. A grow-only counter CRDT merges additive updates without losing increments. Pick the policy that matches your data type.

Region A

Value 10 · clock 1

Region B

Value 10 · clock 1

Global view

Merged value 10

Policy: LWW
Conflict: No
Merged: 10
Clock skew: 0

A=10 (c1) · B=10 (c1) · merged 10 · conflict no. Both regions start aligned. Write concurrently during a partition, then merge when the link returns.

Match structure to conflict. LWW fits single-writer fields with reliable fencing; counters and sets need CRDTs or operational transforms.

RPO follows topology. Async replication accepts loss on regional failure; sync quorum shifts latency and availability trade-offs.

What actually happens across oceans

The Mumbai/New York story is the default whenever you place copies near users. Not every table belongs in every region. A user profile might be with so either region can accept an edit. A payment ledger might stay single-region with an async replica for disaster recovery. A session might stick to a home region with cached reads elsewhere.

A synchronous write waits for the replicas required by its configured quorum, which may include a distant region depending on placement. If the majority is local, ordinary commits might avoid ocean latency while regional-loss durability has a different trade-off. Asynchronous single-writer replication can lag without creating concurrent writes; concurrency arises when multiple owners accept writes or an unfenced failover overlaps the old primary. Separate replication timing from write topology.

Failover is product and ops together: measure lag against , verify on promotion, rehearse traffic shift, and cap reconnect surge on survivors. DNS alone is not instant, and (how fast you must serve again) only holds if those steps are practiced.

How to choose a multi-region shape

Once you see that topology is per entity, three questions decide which shape to ship for each class of data.

1. What consistency does this entity need? Money and inventory hate silent merge. Display names and likes can tolerate or .

2. What are and ? Name them before drawing boxes. Minutes of lag may be fine for analytics and fatal for checkout.

3. Who merges conflicts? Storage config is not enough. Concurrent edits need product rules, not only a replication toggle.

Simple rule of thumb

Default to (or single-region primary) for high-stakes writes. Open only where latency SLOs demand it and you can name the merge strategy. Budget cross-region egress and p99 sync cost before promising global strong consistency. In simple terms: multi-region is a portfolio of patterns per data class, not one topology for the whole product.

From problem to mechanism

Here is the dual-edit scenario again, slowed down so you can see where the naive global-primary plan fails and what the fix changes.

Problem. Users in Mumbai and New York edit the same profile and cart. Leadership wants "active-active everywhere" for low latency and regional DR, including the payment ledger.

Naive system. One global primary in us-east; everyone else is async replica. Failover = flip DNS. No , no per-entity merge rules, unnamed.

Break it. Region A partitions; B is promoted while A still accepts late writes. Split-brain merge invents a third bio. Ledger lag was 90s; operators claimed 30s RPO and lost paid orders.

Mechanism. Classify entities: mergeable profiles/carts vs single-writer ledger. Use epoch fencing on promotion; measure against RPO before admitting writes; pick , , or only where the algebra matches the product. Treat failback as a new migration, not a blind DNS undo.

Correctness invariant

At any instant each single-writer entity has one fenced write owner. Multi-writer entities use a deterministic merge that preserves domain invariants (no silent ledger fork). Failover never silently exceeds declared RPO or residency boundaries.

Worked example: ocean RTT vs async RPO

Assume an illustrative EU-to-US round trip of 95 ms and a protocol that requires an acknowledgement across that link before commit. That requirement adds at least the relevant remote round trip plus processing and durable-storage time; it does not by itself prove p99 exceeds 200 ms. Measure the actual critical path. An async replica may have four seconds of typical lag and ninety during overload, so a thirty-second RPO cannot be justified from typical lag alone.

Sync global

+95 ms / commit

Stronger durability across regions; kills mobile checkout latency budgets that assume local writes.

Async + RPO 30s

promote only if lag ≤ 30s

If the candidate replica is known to lack ninety seconds of acknowledged writes, promoting it exceeds a thirty-second data-loss objective unless that history can be recovered elsewhere. Pause automatic promotion or obtain an explicit incident decision about the known risk. A “lag four seconds” sample taken before the source became unreachable is not proof of the final missing-write window; compare durable replication positions and establish what acknowledgements may be absent.

RTO budget

15m = 3+2+5+5

A fifteen-minute recovery target might allocate three minutes to detection, two to the decision, five to fencing and promotion, and five to validation and traffic movement. These are example budgets to rehearse. The recovery is not complete when a new database becomes writable: surviving capacity, reconnect storms, caches, queued jobs, and application correctness must also recover. Measure the whole user journey during the drill.

Numbers to know

EU↔US RTT ≈ 95 ms; sync quorum adds at least one ocean round trip per commit.
Async lag: roughly 4 s normal, up to 90 s under overload; compare to declared RPO.
Example RTO 15 minutes: detect 3, decide 2, fence/promote 5, validate/shift 5.

Counterexample to "active-active = always available": two regions accept ledger writes without and you get two balances that no can honestly merge into one bank account.

Topology patterns and conflict toolkit

With the latency and RPO numbers above in mind, compare write routing next, then pick the merge tool that matches each entity.

Topology comparison

Pattern	RPO / RTO	Conflicts
	Minutes typical	None on writes (single writer)
	Low	Concurrent writes need merge
Multi-master per cell	Regional	Cross-cell async + repair

Conflict resolution toolkit

For a display name, a deterministic last-writer policy may be acceptable if the product accepts one concurrent edit being replaced. For a cart, union can preserve additions but may resurrect removed items; quantities need more precise operation semantics. Collaborative text requires an algorithm designed for concurrent editing, not a generic map merge. A convergent merge can still violate a business invariant, so convergence and correctness must be checked separately.

with a logical clock: simple merge that can discard concurrent intent.
detect concurrency; the application still merges siblings.
auto-merge counters, sets, and registers when the algebra fits.
Operational transform for collaborative text, where character order must stay coherent.

Failover checklist

Before admitting writes on a new owner, prevent the old owner from successfully writing even if it later reconnects. Use the storage protocol’s authoritative epoch, fencing, or isolation mechanism rather than relying on DNS changes. Establish the recovered data boundary, validate residency and access rules, then move a bounded amount of traffic and observe errors and lag. On failback, synchronize and fence again; reversing DNS alone does not reconcile divergent histories.

Measure vs ; verify during promotion; rehearse DNS/traffic shift; cap reconnect surge on survivors.

Failure timeline, and what still breaks after the fix

Putting topology and fencing together, here is how a split-brain promote plays out minute by minute, then the signals that would have caught it, and the ways an improved design can still go wrong.

Timeline: split-brain merge

t0  Region A (epoch 71) primary; B async replica lag = 12s
t1  Partition: A unreachable from control plane; B healthy
t2  Ops promote B to epoch 72 but forget storage-level fence on A
t3  A still acks local writes (bio + cart) for 8 minutes
t4  Heal: A and B both have divergent profiles; LWW overwrites intent
t5  Ledger: two charge rows with same payment_id: finance reconcile queue
t6  User sees wrong balance; support merges manually; RPO claim was fiction
t7  Fix: epoch fence rejects A's writes; quarantine old-epoch log; rebuild A

What to watch

versus each entity's , so stale promotes are refused.
Dual write-epoch acknowledgements, which mean the old primary was never fenced.
Conflict and manual-merge queue depth, which show product rules are failing in production.
Residency violations and failback treated as a controlled migration, not a blind DNS undo.

Still fails when…

Global secondary indexes lag after failover, so search still shows ghost inventory.
sets preserve adds, but business caps (max qty) still need checkout rules.
Clients pin sticky sessions to a dead region longer than the DNS TTL allows.

Trade-offs

improves local UX but adds merge complexity and egress cost.
Spanner-style sync global consistency costs 100+ ms writes and placement constraints.
Data residency laws often block one unified global product model for every table.

Global data by entity

What production needs to hold

A production design needs a way to recognize trouble and a rule it preserves while recovering. A trigger is the measurement or event that starts a response. An invariant is a statement that must remain true, such as never confirming the same reservation to two customers.

Trigger metric or condition

Adopt a multi-region topology when measured user latency, regional continuity, data residency, or recovery objectives justify the replication latency, conflict semantics, and operational cost.

Correctness invariant

At any instant each single-writer entity has one fenced write owner, while multi-writer entities use a deterministic merge that preserves their domain invariants. Failover never silently exceeds declared RPO or residency boundaries.

Explicit regional failover state machine

Normal owner A

epoch 71 writes

Detect + freeze

stop old acknowledgements

Measure safe point

RPO and lag

Fence epoch 72

promote region B

Reconcile + failback

new migration, not undo

Per-entity topology and recovery math

A commerce platform serves Europe and the US. Profiles tolerate merge, carts require no lost items, and payments require one authoritative ledger. Inter-region RTT is 95 ms. Async replication normally lags 4 seconds and peaks at 90 seconds during overload.

Profile

active-active LWW per field + manual audit for protected fields

Display name may use a hybrid logical timestamp; legal name conflicts are retained for manual review rather than silently overwritten.

Cart

OR-Set of item IDs + quantity operation IDs

A CRDT-style add/remove set preserves concurrent item intent; business rules cap quantity and resolve unavailable inventory at checkout.

Ledger

home-region single writer + fenced async replica

Synchronous global quorum would add at least one 95 ms network round. The product instead declares RPO ≤ 30 seconds and stops promotion if verified lag is larger.

Recovery

RTO 15m = detect 3 + decide 2 + fence/promote 5 + validate/shift 5

Write down the time budget. If replication lag is 90 seconds, operators expose that potential loss instead of claiming a 30-second RPO.

Decision

EU personal data remains on EU-approved stores; only allowed aggregates replicate. Failback is a controlled resynchronization into a new epoch after reconciliation, never a blind DNS reversal to a stale former primary.

How the mechanism holds

Follow how each part protects the rule above. The mechanism explains what the system does; the reasoning explains why that action is sufficient under the stated assumptions. If an assumption changes, the guarantee may change too.

Per-entity decision

Record home/allowed regions, writer topology, read policy, consistency, conflict rule, RPO/RTO, retention, and egress. One diagram labeled “active-active” is not a data architecture.

Why it holds: Every critical entity has an owner and an executable partition/failover answer.

Regional fencing

Issue a monotonically increasing epoch from a quorum/control plane and require storage to reject old epochs. Stop or expire the former writer before accepting acknowledgements from the new owner.

Why it holds: Partitioned old region cannot commit after epoch promotion even if its process remains alive.

Conflict handling

Use LWW only where losing concurrent intent is acceptable. CRDT counters/sets merge algebraically; complex protected fields keep siblings and enter a human workflow with full provenance.

Why it holds: Pairwise merge tests prove commutativity, associativity, and idempotence where claimed.

Residency and reconciliation

Classify fields, replication destinations, encryption keys, operator access, backups, and derived indexes. After recovery compare source logs, counts, checksums, duplicate effects, and manual-conflict queues.

Why it holds: Residency policy tests block forbidden records and a drill reconciles to documented tolerances.

Regional outage and safe failback

Region A is unreachable

Health probes and user SLIs agree, but asynchronous lag is uncertain.

Response: Stop traffic, establish the last durable safe point, and begin incident RTO clock.

Promotion decision

Verified ledger lag is 18 seconds, within the 30-second RPO.

Response: Acquire epoch 72, fence epoch 71, promote B, then admit writes gradually.

Region A returns

Its local log contains unreplicated attempts from before fencing.

Response: Quarantine old-epoch records; reconcile idempotency keys and present ambiguous operations for review.

Failback later

A is rebuilt from B and catches current epoch.

Response: Treat failback as another planned migration with new fencing, validation, and traffic ramp.

How to choose among alternatives

Sync vs async replication

Choose when: Sync when lost acknowledged writes are unacceptable and RTT fits latency; async when local latency/availability wins with explicit RPO.

Caution: Quorum placement and tail RTT matter more than average distance.

Active-passive vs active-active

Choose when: Active-passive simplifies write ownership; active-active lowers local write latency for mergeable data.

Caution: Active-active without a product conflict rule merely postpones data loss.

Automatic vs manual failover

Choose when: Automate well-observed, fenced transitions; require humans when data-loss ambiguity or external side effects are material.

Caution: DNS changes alone neither fence writers nor guarantee clients move quickly.

Running it day to day

Measurements tell you what is happening; alerts tell an operator when action is needed. A rollout plan limits how many users encounter a change before it is checked. A runbook records the steps to diagnose and recover from a known failure.

Measure

replication lag in time/bytes and last safe position
write epoch, conflict rate, merge/manual queue, and stale reads
regional SLI, failover state duration, RPO/RTO consumption, and egress

Alert

lag threatens entity RPO
two regions acknowledge same single-writer epoch/entity
residency violation or reconciliation residue exceeds tolerance

Roll out safely

classify entities and shadow cross-region reads
test merge properties and old-epoch rejection
game-day failover, failback, and ambiguous-write reconciliation

Runbook

freeze and determine safe point before promotion
fence, promote, validate, and ramp traffic
quarantine old epoch, reconcile, then rebuild/fail back deliberately

Practice scenario

Design regional behavior for profile, cart, and payment ledger through a 20-minute regional outage.

Questions worth asking next

Which entity is active-active?
What if lag exceeds RPO?
How does old primary rejoin?

A solid answer sketch

Give each entity its own topology: mergeable profiles/cart near users, a fenced ledger writer with declared async RPO, and residency-aware replication. Promotion requires a safe point and new epoch. Old primary is quarantined, reconciled, rebuilt, and later reintroduced through a new migration.

Common mistakes

• one conflict policy for every table
• automatic promotion without fencing
• calling DNS failover an RTO plan while ignoring caches and data validation

Authoritative references

Azure Architecture Center: Data partitioning by geography AWS Well-Architected: Disaster recovery options

Last reviewed: 6 August 2026

Choosing and explaining multi-region data

A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.

Explain the core idea

Classify data by consistency need before drawing global active-active.

Cross-region sync writes pay RTT on every commit: budget p99.

Conflict resolution is product logic, not only storage config.

When it belongs

Latency SLO requires regional serving.

Regional disaster recovery is mandatory.

Users edit same object from multiple regions.

How it fails

Split-brain writes during partition.

Laggy replica promoted: lost writes.

Global index stale after failover.

Trade-offs to weigh

Active-active UX vs merge complexity.

Strong global consistency vs 100+ ms writes.

Data residency laws vs unified global product.

Common misunderstandings

Active-active for financial ledger without conflict design.

Assuming DNS failover is instant.

Ignoring cross-region egress cost for replication.

Wrap-up

You have seen the dual-edit failure, the fencing mechanism, and the operational signals. Use these exercises to check the ideas stuck, then follow the references for the primary sources.

Exercises

Profile vs cart vs ledger. For a 20-minute EU outage, pick topology and merge rule for each entity. What if verified lag is 90s against a 30s RPO?
Split-brain after promote. Old primary returns with unreplicated writes. Do you LWW, keep siblings, or quarantine? Who owns reconciliation SLAs?
Sync Spanner-style vs async home region. Checkout p99 budget is 150 ms; inter-region RTT is 95 ms. Argue for or against synchronous cross-region commits for payments only.

Topic-specific references

Spanner: Google's Globally-Distributed Database : TrueTime, external consistency, and the cost of sync global commits.
Shapiro et al.: A comprehensive study of Convergent and Commutative Replicated Data Types : when automatic merge is mathematically safe.
Dynamo (SOSP 2007) : eventual consistency, vector clocks, and application-level conflict resolution.

Key takeaway

Multi-region is a portfolio of patterns per data class. Define RPO/RTO, choose write routing, and name conflict rules before replication topology, not after the first regional outage.

Practice and further reading

Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.

Focus for this lesson: Choose topology per entity, enforce regional epochs, model failover/failback, apply CRDT/domain merge, quantify RPO/RTO, residency, and reconciliation.

Explain the decision

Why should topology be chosen per data entity rather than per application?

Read the reasoning

Profiles, ledgers, carts, and analytics have different write ownership, mergeability, residency, latency, and loss tolerance.

Apply it to a production scenario

Two regions accept offline cart edits, but payment authorization is single-writer. How do they coexist?

Read one approach

Merge cart items with domain rules/version vectors, then route checkout through a fenced authoritative region and reconcile on failover.

Work through the numbers

Replication lag is 8 s at 12,000 2 KiB writes/s. Estimate data at risk under asynchronous failover.

Read the worked calculation

About 187.5 MiB logical before log/index amplification; compare with the stated RPO.

Reason through a failure

What must regional promotion preserve?

Read the recovery reasoning

Only a region holding the newest enforced epoch accepts authoritative writes, and failback waits for reconciliation/convergence.

Sources and further study

Continue with Cost Engineering & Capacity Economics.