System Design Foundations
Distributed Systems Fundamentals
A checkout call times out after the payment service charged the card. Is the order paid? Retrying blindly double-charges. Giving up leaves a paid user without a receipt. That ambiguity is not a bug in one library; it is the default physics of a .
This note is the map: , partitions, clocks, delivery semantics, replication, sharding, consensus, and observability—the vocabulary every later system-design topic assumes.
Survive the timeout
Assume partial failure
Crash, slow, and lost reply look the same to the caller.
Retries must not double side effects.
Deadlines + budgets beat unbounded waits.
Trace one request across machines or you fly blind.
What actually happens
On one machine, a function either returns or the process dies with it. Across machines, is normal: Service B may have committed while Service A never saw the ack. Networks add variance and occasional . Wall clocks drift (), so “latest timestamp wins” without care silently drops work.
Systems cope with a small toolbox. copies data for durability and reads. splits load. elects leaders and orders critical logs when you need one truth. Messaging is usually ; effects come from and careful transactions, not wishful broker settings. reminds us that consensus needs timeouts and can stall under nasty timing—protocols are engineered compromises, not violations of physics.
Operations close the loop. Without , you cannot tell which hop failed. Without deadlines and backpressure, retries amplify outages. Distributed design is as much about these operational contracts as about boxes on a whiteboard.
How to think when designing
1. What if the reply is lost? Every write path needs an idempotency story before you draw the happy path.
2. What must be single-copy? Leaders, money, inventory → consensus or strong primary. Everything else can often replicate asynchronously.
3. How do you see it break? Correlation ids, RED/USE metrics, and traces are part of the design, not a later ops gift.
Simple rule of thumb
Assume at-least-once, design for idempotency, bound every wait, prefer replication for reads and consensus only for narrow critical state, and never trust wall clocks alone for cross-node truth.
Map of the fundamentals
Foundations force the tools; tools only work if you can operate them under failure.
Read the diagram definition
flowchart LR
subgraph foundations [Foundations]
PF[Partial failure]
CLK[Clocks / ordering]
NET[Partitions / latency]
end
subgraph tools [Building blocks]
REP[Replication]
SH[Sharding]
CON[Consensus]
ID[Idempotency + retries]
end
subgraph ops [Operate]
OBS[Observability]
DEG[Degradation / backpressure]
end
foundations --> tools --> ops
The timeout that might have succeeded
Partial failure in one picture: lost ack turns a successful charge into an ambiguous retry that must be idempotent.
Read the diagram definition
sequenceDiagram
participant C as Client
participant A as Service A
participant B as Service B
participant DB as Database
C->>A: Place order
A->>B: Charge payment
B->>DB: Commit charge
DB-->>B: OK
B--xA: Reply lost (network)
Note over A: Timeout: did charge happen?
A->>B: Retry charge
Note over B: Must be idempotent or double charge
B-->>A: Same payment id / already done
A-->>C: Order confirmed
Fallacies (still true)
The classic network fallacies are interview gold because teams still violate them: the network is reliable; latency is zero; bandwidth is infinite; the network is secure; topology does not change; there is one administrator; transport cost is zero; the network is homogeneous.
Design implication
Timeouts, retries with jitter, circuit breakers, TLS everywhere, topology-aware placement, and cost-aware chatty APIs.
Interview phrasing
“I assume messages can be delayed, duplicated, or lost, so every cross-service write is idempotent and deadline-bound.”
Time and order
means two nodes can timestamp events in the wrong order. Last-writer-wins on wall clocks can drop a valid concurrent update. Logical clocks (Lamport), vector clocks, and hybrid logical clocks restore causal or approximate real-time order without pretending NTP made machines identical.
When you truly need one global order for a shard, use a primary or log. When you only need cause-before-effect, track dependencies. When neither matters, eventual merge policies suffice—but name them.
Delivery semantics cheat sheet
| Claim | What it usually means | Your job |
|---|---|---|
| At-most-once | May lose messages; no retry | Only for loss-tolerant signals |
| At-least-once | Retries until ack; duplicates possible | |
| Effectively exactly-once | Deduped processing or transactional handoff | Outbox/inbox, idempotent keys, careful sinks |
Replication, sharding, consensus—roles
Replication
Copies for HA and read scale. Choose sync/async by RPO. Expect lag and define who may read stale data.
Sharding
Splits write and storage load. Pick keys that match access; plan reshard and hot-key escapes.
Consensus
Narrow tool for single leadership and strongly ordered logs. Do not put every user request through a global consensus round.
Numbers that shape intuition
| Scale | Order of magnitude | Design note |
|---|---|---|
| Same-AZ RTT | ~0.1–1 ms | Chatty RPCs still add up under fan-out |
| Cross-region RTT | ~50–150+ ms | Strong cross-region commits hurt p99 |
| Disk fsync | ~1–10 ms class | Durability is not free on the commit path |
| Fan-out 1→50 leaves | p99 ≈ max of leaves | Tail latency multiplies; hedge or bound width |
Failure timeline: distributed checkout
T0 Client → Order → Payment; payment commits; ack to Order lost T1 Order times out; retries without idempotency key → second charge T2 Partition isolates fraud service; sync call blocks all checkouts T3 Clocks skew; LWW inventory merge drops a valid reservation T4 Fix: idempotent payment ids, deadlines + degrade fraud, strong inventory path Outcome Fundamentals failures are ambiguous success, unbounded waits, and bad clocks
Break the improved design
Idempotency keys that expire too soon allow double effects on long retries. Consensus clusters that are too small (or span bad failure domains) lose majority and freeze writes. Shards without hot-key plans look fine in averages and melt on celebrities.
Observability that only samples happy paths misses the rare partition. Multi-layer retries (client + edge + worker) without a shared budget recreate thundering herds. Fundamentals are necessary; they are not automatic correctness.
Trade-offs
Coordination for truth vs independence for scale
Consensus and sync replication give clearer single-copy semantics and cost latency plus partition unavailability. Independent replicas and shards scale and stay available more easily but need merge rules and tolerate lag. Narrow the coordinated set.
Retries for reliability vs amplification
Retries heal transient faults; unbounded retries turn a slow dependency into a cascade. Pair deadlines, jitter, budgets, and idempotency—or you buy availability with a larger outage.
Wall-clock convenience vs logical time correctness
Timestamps are easy and wrong under skew for conflict resolution. Logical or hybrid clocks and primary sequencing are harder and safer when order matters. Use wall clocks for humans; use safer mechanisms for truth.
Broker “exactly-once” vs end-to-end effects
A broker feature that avoids duplicate log appends does not make every email, charge, or webhook exactly-once. End-to-end exactly-once is an application property built from idempotent handlers and transactional handoffs.
Feature architecture vs operable architecture
Beautiful service graphs fail without traces, SLOs, and runbooks for partial failure. Budget design time for observability and failure injection, not only for the happy-path boxes.
Choosing and explaining distributed systems fundamentals
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
I start from partial failure: timeouts are ambiguous, so writes are idempotent and deadline-bound.
I use replication and sharding for scale, and consensus only for narrow single-truth state.
I refuse wall-clock LWW as a correctness strategy and insist on observability across hops.
When it belongs
Any design spans multiple services, regions, or data stores.
The interviewer asks what happens when a call times out mid-write.
You need a vocabulary bridge before CAP, consensus, or queues deep-dives.
How it fails
Lost acks causing duplicate side effects.
Retry storms and unbounded fan-out tails.
Clock-skewed merges and unobservable partitions.
Trade-offs to weigh
Coordination versus independent scale.
Retry reliability versus load amplification.
Simple timestamps versus correct ordering machinery.
Common misunderstandings
Assuming the network is reliable or latency is negligible.
Believing broker exactly-once replaces application idempotency.
Putting every request through global consensus.
Wrap-up
You have the physics and the toolbox. Later notes specialize; this one keeps the shared assumptions honest.
Exercises
- Ambiguous timeout. Draw the payment lost-ack case. Where do you store the idempotency key, how long does it live, and what does the user see while pending?
- Fan-out budget. One API calls 20 dependencies with p99=20ms independent. Estimate rough end-to-end p99 if you wait for all; propose two mitigations.
- Where consensus? For a URL shortener, list state that needs consensus or a single primary versus state that can be eventually replicated—and why.
Primary sources
- Dynamo / distributed systems classics : replication and failure assumptions at scale.
- Lamport: Time, Clocks, and Ordering : logical time foundations.
- FLP impossibility paper : why async consensus needs help.
Key takeaway
Distributed systems are partial-failure machines. Bound waits, assume duplicates, copy and split data deliberately, coordinate narrowly, and instrument the path—or the timeout will invent a second truth for you.