System Design Data
Eventual Consistency
You post a photo on Instagram in Mumbai. You see it instantly. A friend opening the feed on another continent might still see the old timeline for a moment, then the new post appears. That gap is in action, not a bug by default.
Asynchronous replication lets a write finish without waiting for every distant replica, often reducing coordination delay on the local path. It still pays local processing, durability, and replication costs. Cross-region latency depends on geography and the protocol; neither “100–300 ms for strong” nor “near zero for eventual” is a universal number. Specify which users may see older state, the desired freshness window, and what happens when the window is exceeded.
Post → world sees it
Write in region A
Local replica accepts the post where you uploaded it.
You see your own post immediately on that session.
Async replication
Region B briefly serves the old feed until catch-up.
Replicas settle on the same value through background repair.
What actually happens
The Mumbai story is the default for any global feed: a write is accepted where it lands, nearby reads may see it immediately, and distant catch up asynchronously. Until they do, two users can observe different truths (counts, profile text, or search hits) inside the allowed window.
Convergence is not automatic hope. It needs a mechanism such as , , gossip, or re-index from a source of truth. Concurrent writes need an explicit merge rule: , , or application compensation. Without that, updates silently disappear.
Not everything should be eventual. Payments and inventory often need or a strong path. Feeds and DNS-like lookups usually do not. Session guarantees like make eventual systems feel sane for owners even when distant friends lag.
How to choose consistency
Once you know reads can disagree briefly, three questions decide whether eventual is acceptable for a data class.
1. What breaks if the answer is stale? A lagging like count is fine. A lagging payment confirmation or seat reservation is not.
2. How do conflicts merge? Counters want . Profile fields may want field-level merge or owner-confirmed edits, not silent .
3. What does the owner need? Prefer for user-owned data even when distant readers stay eventual.
Simple rule of thumb
Default to for global feeds, indexes, and counters. Reserve strong paths for money, inventory, and auth state, and always name the repair SLO. In simple terms: eventual consistency is a product contract plus a convergence mechanism, not a hand-wave.
From problem to mechanism
The same Instagram-style feed, slowed down so you can see where a naive design fails and what the fix must guarantee.
Problem. A social feed must accept posts in Mumbai and serve readers worldwide with low latency. Cross-region writes often add 100-300ms of coordination per update.
Naive system. One primary database for the world, or "eventual consistency" with no repair SLO and no merge rule. Counterexample: without , a region that missed a write never converges, so eventual becomes "maybe never."
Break it. The single primary dies or partitions and writes go global-down. Or replicas accept concurrent profile edits and silently drops one user's change. Distant friends see stale feeds with no product contract explaining why.
Mechanism. Accept the write locally for availability, promise to the author, replicate asynchronously, and drive convergence with repair. Concurrent siblings need a declared merge policy (, field merge, or owner confirmation), not silent hope.
Correctness invariant
If writes to a key stop and the network recovers, every correct eventually returns the same value for that key (or the same merge of concurrent siblings under a declared policy). Until then, the user-visible consistency contract must hold: who may see stale data, for how long (SLO), how conflicts merge, and what the UI shows while replicas disagree. Session guarantees like read-your-writes are separate, explicit promises, not implied by "eventual."
Write accepted in region A
Local read sees new value
Write accepted in region A
Async replication
Region B briefly sees old value
Repair / anti-entropy
All replicas converge
User-visible contract and numbers to know
The mechanism above is incomplete as an engineering slogan. The product must state what users are allowed to see while replicas disagree. A useful contract names four things per data class, then the table and budgets below show how that looks in practice.
- Who: owner session vs distant readers (for example for the author only).
- How stale: a bound or SLO (seconds of lag, "pending" UI until confirm).
- On conflict: merge rule (, field merge, or reject), not silent hope.
- While diverging: what the UI shows (spinner, "updating," approximate counts) so stale is intentional, not broken.
| Data class | Allowed staleness | Conflict rule | Repair signal |
|---|---|---|---|
| Like/view counts | Seconds to minutes | Commutative counter merge | Counter drift and anti-entropy backlog |
| Profile edits | Usually read-your-writes for owner | Field-level merge or latest owner-confirmed edit | Sibling versions and merge failures |
| Inventory | Low or none for checkout | Reservation, escrow, or strong write path | Oversell rate and compensation count |
| Search index | Seconds to minutes | Re-index from source of truth | Index lag and missing-document sample checks |
| Payment state | No user-visible ambiguity after confirmation | Authoritative ledger and reconciliation | Ledger mismatch and stuck pending states |
Numbers to know
- Observed ≈ replication lag + cache TTL + client retry delay (700ms lag + 5s TTL ≈ 5.7s).
- Feed budget example: p99 lag 1.2s + TTL 3s ≈ 4.2s, inside a "friends see posts within 5s" SLO.
- Strong vs eventual write: ~15ms local vs ~180ms cross-region; at 50K QPS, cross-region quorum burns ~9 CPU-seconds of wait per second of wall time.
- with R=3 can add up to 2 repair writes per hot read; prefer Merkle for cold ranges.
Watch replication lag p50/p99, repair backlog age, conflict/sibling rate, and UX "pending" duration. Page when lag SLO breaches for user-visible entities. When lag breaches SLO, pause low-priority consumers, prioritize repair for user-visible entities, show pending UI where needed, and reconcile from the source of truth before declaring convergence.
Consistency models hierarchy
The contract section names how stale a read may be; this ladder places among stronger models so you can pick the weakest promise that still matches the product.
Linearizable → Sequential → Causal → Eventual
Linearizable (strongest)
Linearizability makes each operation appear to take effect at one instant between its invocation and response, while respecting real-time ordering. A read after a completed write must reflect that write or a later one; a read overlapping a write may return either valid state. The latency cost depends on replica placement and protocol, and simply contacting a node called “leader” is insufficient unless its leadership and read safety are established.
Sequential
Operations appear in a single global order. Often provided by a coordination service.
Causal
If A caused B, everyone sees A before B. Document stores often expose causal sessions.
Session guarantees
Read-your-writes, monotonic reads/writes, consistent prefix: UX-friendly without full linearizability.
Eventual
Eventual consistency promises convergence once updates stop and the required communication and repair complete. It does not itself bound lag or guarantee that a user reads their own latest write. Different databases, object stores, and individual APIs provide different guarantees, so avoid inferring consistency from a product category. A stronger session contract can be layered on an asynchronously replicated system by routing or waiting for a required version.
How replicas merge, repair, and tune strength
Hierarchy tells you how strong a promise is; these tools are how systems actually merge conflicts, heal divergence, and dial strength per query.
CRDTs: conflict-free merges
A merges concurrent updates without central coordination, so counters and sets do not silently lose work.
G-Counter
Each node stores {A:5, B:0}; merge takes max, total sums, never loses increments.
PN-Counter
A PN-counter represents increases and decreases with two grow-only component counters; merging each component by maximum preserves independent updates and subtracting totals yields the value. It is useful for replicated counts, but does not enforce a lower bound such as inventory never dropping below zero. Two disconnected regions can each decrement their locally available stock. Use reservations, escrowed rights, or a stronger coordination path when that bound is a business invariant.
LWW-Register
Timestamp per write; highest wins on merge, common in wide-column storage.
OR-Set
An observed-remove set tags each addition uniquely. A removal records the addition tags it has actually observed, so an independent concurrent addition can survive. This gives a defined add-versus-remove policy rather than eliminating the conflict by magic. Tombstone or causal-context cleanup requires care; discarding removal evidence before all replicas have incorporated it can resurrect an old element.
G-Set
Add-only set whose merge is union; fits seen articles and blocked users.
Collaborative and replicated systems use this style for counters, sets, notes, and design documents.
Anti-entropy protocols
Without , lagging copies never catch up. These four patterns drive replicas toward agreement.
Client reads several replicas, returns the newest, and writes repairs back to stale copies.
Merkle tree
Merkle-tree comparison finds differing ranges by descending through unequal hashes. Locating one divergent leaf in a balanced tree takes logarithmic depth, but repairing many changed ranges also requires work proportional to the differences and transferring their data. Tree agreement only summarizes matching versions under compatible range boundaries; version comparison and tombstones still determine which records survive repair.
Gossip
Exchange state with random peers for O(log n) rounds in gossip-based clusters.
Hinted handoff
When B is down, A stores a hint and delivers it when B returns, keeping writes available.
Quorum and tunable consistency
W + R > N → quorum-write overlap (not by itself)
N=3: W=2,R=2 observes completed quorum writes under Dynamo-style assumptions. W=1,R=1 is eventual. W=3,R=1 favors fast reads. W=1,R=3 favors fast writes.
Wide-column store
ONE, QUORUM, ALL, LOCAL_QUORUM, EACH_QUORUM: per-query tunable strength.
- Per-query CONSISTENCY QUORUM or ONE for read/write strength.
- conflict resolution can drop concurrent edits unless the product allows it.
- Read repair plus nodetool repair and Merkle trees drive convergence.
- Fits high write-rate consumer systems that tolerate brief staleness.
Managed key-value store
Eventually consistent by default (often cheaper); strong reads cost more.
- Default eventual consistency is usually cheaper than strong ConsistentRead paths.
- Strong path: ConsistentRead true, higher cost when freshness is required.
- Global tables are often always eventual and resolve with LWW.
Writes: QUORUM. Reads: ONE for most. Critical: QUORUM.
Vector clocks and causal consistency
A looks like [A:x, B:y, C:z]. On event: increment own. Send: include. Receive: merge + increment. Incomparable clocks mean concurrent updates that need a merge policy.
Concurrent
[A:2,B:0,C:0] vs [A:0,B:2,C:0] → CONFLICT
Causal
[A:2,B:1,C:0] dominates [A:1,B:0,C:0] → first after second
Version vectors support cart-like merges. Many modern systems use , , or hybrid logical clocks.
Failure timeline, and what still breaks after the fix
Putting the pieces together, here is how async replica lag and silent loss play out minute by minute, then the ways an improved design can still go wrong.
Timeline: async replica lag + LWW loss
T0 User A posts photo in region Mumbai; local replica accepts write T1 A reads locally → sees photo (read-your-writes OK) T2 Friend B in us-east hits lagging replica (replication lag 800ms + 5s CDN TTL) T3 B still sees old feed for ~5.8s: within or outside product SLO? T4 Meanwhile A edits caption in Mumbai; C edits same field in EU (partition) T5 Partition heals; LWW keeps EU caption, drops A's edit → silent data loss T6 Anti-entropy / read repair converge replicas on the surviving value
Lag is expected under . Silent loss under LWW is a product bug unless the contract allows it. Prefer field merge, , or owner confirmation when both edits matter. Stale-window math is in Numbers to know above.
Still fails when…
- Repair jobs fall behind hot partitions, so divergence lasts past the published SLO.
- Clients bounce between replicas and observe non-monotonic reads after failover.
- W+R>N still fails to give under sloppy membership.
- A regional partition still accepts writes, creating divergent siblings until heal.
- Cache TTLs stacked on replica lag hide a longer user-visible window than dashboards show.
Trade-offs
- Higher availability and lower latency versus accepting briefly stale reads.
- Conflict-resolution complexity versus simpler strong writes with coordination cost.
- Async repair cuts write latency but shifts burden onto background convergence jobs.
From the basic idea to a production design
Beginner explanation
Say which user actions can tolerate stale reads and which cannot. A like count can lag; a payment confirmation or password change should not.
Advanced answer
Define session guarantees, conflict-resolution rules, repair SLOs, and observability for replication lag. Eventual consistency is only credible when users and operators know the staleness contract.
What to watch in production
- Read-your-writes for user-owned data
- Explicit merge rule per entity type
- Alerts for lag, repair backlog, and conflict rate
- UX copy or refresh behavior for stale views
Questions worth answering
- Which operations need strong consistency?
- How do replicas detect and repair divergence?
- What happens during a regional partition?
Failure modes to cover
- Last-writer-wins silently drops updates
- Repair jobs fall behind hot partitions
- Clients observe non-monotonic reads after failover
Choosing and explaining eventual consistency
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
Use it when availability and latency matter more than immediate global freshness.
State the user-visible consistency contract: who can see stale data, for how long, how conflicts merge, and what the UI shows.
Add session guarantees such as read-your-writes when raw eventual consistency would confuse users.
When it belongs
Feeds, counts, search indexes, recommendations, DNS, or analytics can lag.
Cross-region strong consistency would dominate latency.
The system must remain writable during regional or replica failures.
How it fails
Last-writer-wins silently drops valid updates.
Replication lag exceeds product expectations.
Users bounce between replicas and observe time moving backward.
Trade-offs to weigh
Higher availability and lower latency versus stale reads.
Conflict resolution complexity versus simpler strong writes.
Asynchronous repair reduces write latency but shifts burden to background jobs.
Common misunderstandings
Saying eventual consistency without a convergence mechanism.
Forgetting UX states for pending, syncing, or stale data.
Using it for payments, password changes, or inventory without compensating controls.
Try it: eventual consistency and stale reads
Apply a write on one region, replicate asynchronously, and read from a replica that has not caught up yet to see temporary divergence.
US-East
Value: 0
Version: 0
EU-West
Value: 0
Version: 0
AP-South
Value: 0
Version: 0
- Max version
- 0
- Pending replicas
- 0
- Diverged
- No
Write on one replica, then propagate updates. Reads from a lagging replica can return an older value until convergence.
Going deeper
Production deep dive: make staleness a visible contract
Eventual consistency is not “anything may happen eventually.” Define which anomalies users can see, which session guarantees you preserve, and how quickly replicas must converge under normal and degraded conditions.
What must stay true
When updates stop and communication resumes, every healthy replica converges to a deterministic state; within one user session, acknowledged writes do not disappear behind older reads.
How the pieces fit
Write at v42
return session token
Replication log
ordered per key
Replica at v40
token not satisfied
Wait / primary route
read barrier
Read v42+
RYW + monotonic
Convergence SLO
99.9% < 5 s
Measured from authoritative commit to replica visibility.
Hard ceiling
< 60 s
Beyond this, page and remove the replica from stale-tolerant reads.
Session token
version 42
Reader may serve only a replica with applied_version ≥ 42.
User-visible contracts
A like count may be approximate for seconds; account balance, permissions, and “did my order submit?” need stronger paths. State stale labels, pending states, undo behavior, and when a refresh becomes authoritative instead of hiding anomalies as random UI bugs.
Read-your-writes sessions
Return a commit/version token. Subsequent reads carry it and either wait for a replica to catch up, choose a sufficiently advanced replica, or route to the primary. Sticky routing alone fails when that replica is replaced.
Monotonic reads
Track the greatest version observed in the session and never serve an older one. This prevents a user from moving between replicas and seeing a comment or status revert, even though global reads remain eventually consistent.
Conflict policy
Choose per field: add-wins set for tags, max for progress, sum/PN-counter for independent increments, explicit version conflict for documents, or domain merge for carts. Last-writer-wins is only safe when losing a concurrent value is acceptable.
Staleness observability
Record commit timestamp/version at the source and applied timestamp/version at each replica. Measure time lag and version lag per key/partition, plus read responses served below a carried session token and repair convergence after partitions.
Worked example: Concurrent-write merge exercise
- 1Cart starts {book}. Offline phone adds pen while web removes book and adds mug.
- 2Last-writer-wins could leave only {pen} or only {mug}. An observed-remove set records distinct add/remove operations.
- 3Merge retains pen and mug and removes the observed book add. A later unseen concurrent add of book would survive by the chosen add-wins policy.
Conflict resolution is a product decision expressed as a deterministic data type, not a generic database setting.
Consistency failure matrix
| Scenario | Consequence | How to contain it |
|---|---|---|
| User reads older replica after write | Acknowledged change appears lost. | Carry session version; wait, reroute, or read primary. |
| Replica partition lasts 20 minutes | Staleness SLO and retention headroom are at risk. | Remove it from reads, page, then incremental catch-up or snapshot repair. |
| Clock-based LWW under skew | Older physical clock can overwrite newer intent. | Use causal/logical versions or domain-specific merge. |
What to watch in production
- Dashboard convergence percentile, maximum staleness, version lag, unsatisfied session tokens, conflict rate, and anti-entropy repair age.
- Inject replication delay and verify UX pending/stale states rather than only backend convergence.
- Set separate consistency contracts per operation; do not average money and social counters into one SLO.
A scenario to reason through
A user updates a profile in region A and is routed to region B 200 ms later. Region B is one second behind. What should happen?
One solid approach
Carry the acknowledged version. Region B can wait within the read deadline, route to A/primary, or return a visible pending response; it must not silently serve an older version within that session.
Going deeper
Eventual consistency
The mechanism and its boundary
Eventual consistency means replicas may disagree for a while but converge if updates stop and repair continues. The interview answer should define the user-visible staleness window and the repair mechanism, not just say the data will sync.
Convergence loop
Local write
Async replicate
Conflict repair
Converged replicas
Work through the sizing assumptions
When it belongs
Availability and latency matter more than immediate global freshness. The product can explain or mask temporary disagreement. Conflicts have deterministic merge or reconciliation rules.
Assumptions that matter
Replication is asynchronous and can lag under load. Conflict rate rises with multi-writer topology. Background repair consumes measurable capacity.
Easy mistakes
Money movement, global uniqueness, and scarce inventory need additional coordination or explicit business controls when replicas can accept conflicting writes. A promise that data eventually converges gives users no useful waiting-time bound, so define and monitor the staleness the product accepts. Replicas do not repair themselves merely because the network recovers; durable replication and anti-entropy work need enough capacity to drain their backlog.
How it fails
Repair backlog grows after a region or node recovers. Conflicting writes merge incorrectly and lose intent. Clients read from one replica then write stale context to another.
Trade-offs to name
Serving a nearby replica can preserve availability and low latency during remote failures, but users may see state older than an acknowledged write. Allowing writes in several regions avoids waiting on one distant writer, while creating concurrent versions that require a deterministic merge rule or explicit reconciliation. A simpler local request path therefore moves complexity into product behavior, such as pending indicators, read-your-writes routing, or explaining why a value changed after conflicts were resolved.
Wrap-up
You have seen the failure, the mechanism, and the operational signals. Use these exercises to check the ideas stuck, then follow the references for the primary sources.
Exercises
- Write the contract for three data classes. Like count, profile bio, and checkout inventory. For each: who can see stale, max staleness, merge rule, and UX while diverging. Which one must leave the eventual path?
- Size the stale window. p99 replication lag = 2.5s, edge cache TTL = 10s, client retry delay = 1s. What is the worst-case user-visible staleness? How would you change caching without forcing every write through cross-region quorum?
- Concurrent cart edits. Two regions accept concurrent adds to the same cart. Compare LWW vs CRDT counter/set vs application union. Which invariant fails under LWW, and what metric would you alert on?
Topic-specific references
- Vogels, "Eventually Consistent" : product-facing framing of staleness vs availability.
- DeCandia et al., Dynamo (SOSP 2007) : quorum, sloppy membership, and repair as convergence mechanisms.
- Terry et al., Bayou : session guarantees and application-level conflict resolution.
Key takeaway
Eventual consistency enables global scale by choosing availability over immediate agreement. Pair it with an explicit user-visible consistency contract, CRDTs or merge rules, quorum knobs where needed, and anti-entropy that actually converges. Session guarantees keep UX honest. At scale, eventual is the default: strong consistency is the exception when correctness demands it.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Define user-visible convergence, read-your-writes/monotonic sessions, conflict semantics, staleness SLOs, repair, and observability.
Explain the decision
How can an eventually consistent UI provide read-your-writes?
Read the reasoning
Pin briefly to the writer, carry a version/session token to a caught-up replica, or merge the acknowledged local mutation until replication catches up.
Apply it to a production scenario
A user deletes a photo but another region serves it for minutes. Is “eventual” enough?
Read one approach
No without a bounded privacy contract; use a fast revocation authority/tombstone and measure propagation independently from ordinary content freshness.
Work through the numbers
If p99 replication lag is 4 s and the product promises 99.9% convergence within 3 s, is it meeting the contract?
Read the worked calculation
No; the measured tail exceeds the stated threshold. Change topology/capacity or renegotiate the explicit SLO.
Reason through a failure
What must conflict resolution guarantee?
Read the recovery reasoning
All replicas applying the same set of versions converge deterministically or escalate to a visible domain merge; arrival order alone is unsafe.
Sources and further study
Continue with DNS Deep Dive.