SwadhyAI Notes

All System Design notes

System Design Reliability

Distributed System Testing

Phone retries; the ledger already charged. The user taps Pay, the card network times out, and the app tries again. Unit tests said the charge function works. Integration tests said the happy path works. Production still double-charges, because nobody injected a commit-then-timeout and proved under redelivery.

Distributed tests prove behavior under delay, duplication, and partial failure. If the design assumes retries, quorum loss, and fencing, the suite must inject those faults, or customers become the chaos experiment.

What to prove

1

Schemas dependents rely on still hold so CI blocks breaks before deploy.

2

Timeouts, drops, and duplicates on the wire match the failure modes you designed for.

3

Strong storage claims survive concurrent clients, not only single-thread demos.

4

Rehearse failure with humans watching so runbooks work before peak traffic.

Try it: inject nemeses and inspect a linearizability history

Distributed tests combine workload generators with failure injection. This toy register exposes split-brain and stale reads. Production suites record concurrent histories and check them with tools like Jepsen or linearizability oracles.

Operation history

  • No operations yet.
Value
v0
Nemesis
none
Checks
Pass
Ops
0

Register v0 · nemesis none · linearizable yes · 0 ops logged. Register increments a counter on three replicas. Inject failures, then run a read to see if history violates linearizability.

Test the property, not only happy paths. A passing unit test does not prove replicas agree under partition.

Gate releases. Run nemesis suites on staging before promoting images that touch consensus, fencing, or retention.

What actually happens when the network lies

The checkout story above is the default whenever the network can drop an ack after a commit. A payment service commits a charge, then crashes before the ack reaches the client. The client retries. Without , you charge twice. With an inbox key or dedupe store, the second attempt is a no-op. That story only shows up when tests inject "commit then silence."

A leader loses its majority in a . Writes must stop or fence, not invent a second leader. Clock skew on a lease holder must fail stale writers. Slow followers must not break read-your-writes promises you advertised.

A test is evidence about the executions it explores, not a mathematical proof of every possible distributed schedule. Start with a precise property, such as one charge per logical payment or no negative available stock, then drive concurrent operations while injecting relevant faults. Deterministic simulations and model checking can explore more schedules cheaply; tests against real processes and storage cover implementation behavior the model may omit.

How to choose what to test

Once you know the story fails under silence and cuts, three questions decide how deep the suite must go.

1. What did the design assume? Retries need duplicate delivery. Quorum needs majority loss. Leases need clock skew. If the slide mentions it, the suite should inject it.

2. Is the claim strong consistency? Then add / concurrency checks, not only single-thread demos.

3. Can staging lie? Prefer production-like latency variance and deploy-window runs. Flaky tests ignored become real bugs hidden.

Simple rule of thumb

Never say "exactly-once" or "safe under partition" without a test that injects duplicates or a cut. Start with and one fault catalog for the critical journey, then deepen with model checking where the state machine is clear. In simple terms: unit tests prove functions; distributed tests prove the failure modes you promised.

From problem to mechanism

Here is the payment path again, slowed down so you can see where a green CI badge still leaves you exposed and what a real suite changes.

Problem. A payment path claims "safe under retry and partition." Reviewers ask for evidence. The suite only runs single-threaded happy paths against mocked clocks and a perfect network.

Naive system. Unit tests for charge(), integration tests with retries mocked away, and a green CI badge. Staging never injects commit-then-silence, clock skew, or majority loss.

Break it. Client times out after the server commits; retry double-charges. A flaky checker is quarantined for "infra noise." A real later accepts split-brain writes that the ignored history already hinted at.

Mechanism. Name the safety/liveness property, record invoke/complete histories (including unknowns), run a schedule, check with / tools, and retain seed + build SHA so failures replay. Promote only when the checker and recovery both pass.

Correctness invariant

Every correctness test names the property it evaluates, records a history complete enough to check it (including timed-out unknowns), and can reproduce a failure from code version, workload, fault schedule, and random seed. A green dashboard alone is not a proof of linearizability or .

Worked example: how many histories do you need?

The mechanism above needs concrete budgets. Inventory service: 5 nodes, reserve(sku, request_id), invariant reserved ≤ stock and each request_id applies at most once.

Concurrent clients

8 clients × 50 ops = 400 ops

Record when each operation starts, its inputs and identifier, when it returns, and whether the result is success, definite failure, or unknown. A timeout is unknown when the server may have committed before the response was lost. The checker must consider legal placements of those ambiguous operations rather than dropping them from history; otherwise it can miss a duplicate or incorrectly report a lost update.

Nemesis budget

~12 partitions + 6 pauses / run

Choose a fault schedule tied to a claim. Drop the response after a committed charge to test deduplication; isolate a minority from a majority to test write authority; pause a lease holder until another owner writes, then resume it to test fencing. A random process kill alone may never reach the dangerous boundary. Retain the exact schedule, seed, configuration, and binary versions so a failure can be reduced and replayed.

CI cost

~45 min Jepsen-class / night

PR gate: deterministic model tests (thousands of seeds). Night: real cluster. Stage: one before release.

Counterexample to "integration tests are enough": single-thread reserve/read always passes while a concurrent history under shows stock going negative: the bug only exists in the interleaved schedule.

Layers, faults, and what linearizability means

With budgets in hand (see the workload sizing above), spend cheap checks on schema breaks first, then wire faults and concurrent histories where correctness claims live.

Testing pyramid

Use contract tests for producer/consumer compatibility, state-machine tests for business transitions, and concurrency tests for races on shared data. Load tests measure latency and overload behavior; they do not establish linearizability just because throughput is high. Real-cluster fault tests and production game days add operational evidence at increasing cost. Run the cheapest relevant layer on each change and broader failure drills on a planned cadence.

Consumer-driven API schemas catch breaking changes before dependents ship against them.

Integration +

Latency, drop, and duplicate on the wire between services under real schedules.

/

Concurrent history checkers for storage claims under partitions and crashes.

Explore reorderings of concurrent operations against a simple state-machine spec.

Fault injection catalog

Include recovery as part of the test. After healing a partition or restoring a paused process, stop new writes, allow the documented repair period, and check convergence, pending work, and persistent state after restart. A system that rejects unsafe operations during the fault may be safe but fail its liveness promise if it never resumes. Keep safety assertions active throughout the run and assess recovery deadlines separately.

Match each injection to a design assumption. If you never inject it, you have not tested it.

  • Kill process after commit before ack: client must retry safely via .
  • between leader and majority: refuse split-brain writes.
  • Clock skew on lease holder: fencing tokens must reject the stale writer.
  • Slow follower under load: read-your-writes and quorum behavior must still hold.
  • Message redelivery from queues: inbox or idempotency keys hold under duplicates.

Linearizability intuition

For a linearizable register, a completed write of 5 followed by a later non-overlapping read cannot return the old value 3 unless an intervening write explains it. Concurrent operations may be ordered either way if that ordering fits their invocation/response intervals and the sequential specification. A checker searches for such a valid ordering. If none exists for a complete, trustworthy history, the execution violates the claim; if one exists, that history passes without proving all future histories will.

Failure timeline, and what still breaks after the fix

Putting the pieces together, here is how a quarantined "flaky" checker turns into oversell, followed by the signals that would have caught it, and the ways an improved design can still go wrong.

Timeline: flaky test masks a linearizability bug

t0  CI: linearizability checker fails 1/20 runs: marked "flaky", quarantined
t1  Team ships "strong consistency" inventory based on green PR suite
t2  Client A: reserve(sku,id=7) → timeout (unknown). Server committed.
t3  Client A retries same id; Client B concurrent reserve on same sku
t4  Partition: leader minority still accepts write without fencing
t5  Histories: A sees ok, B sees ok, stock = -1, not linearizable
t6  Incident: oversell. Root cause: same violation the "flaky" seed already showed
t7  Fix: reinstate checker as blocking; retain seed+SHA; fence stale leaders

What to watch

  • Checker pass rate by seed class, so the same invariant failure cannot hide as noise.
  • Unknown-outcome ratio on timed-out ops, because dropped unknowns hide lost updates.
  • Time-to-replay a failing seed with retained SHA, so product vs harness ownership is clear.
  • Whether quarantined tests regain blocking status before the next correctness-critical release.

Still fails when…

  • The model omits clocks, leases, or disk full, so simulation never explores that bug class.
  • Chaos kills pods but never checks histories, leaving green SLIs with silent data loss.
  • Staging latency is flat; production variance triggers timeout and retry paths never tested.

Trade-offs

  • runs are slow and high signal versus fast mocks that miss schedules.
  • Deterministic simulation explores years of faults; real clusters catch kernel and cloud behavior.
  • Safety-only suites can "pass" by refusing all writes; add recovery and liveness bounds.

Proving behavior under faults

What production needs to hold

A production design needs a way to recognize trouble and a rule it preserves while recovering. A trigger is the measurement or event that starts a response. An invariant is a statement that must remain true, such as never confirming the same reservation to two customers.

Trigger metric or condition

Escalate beyond ordinary integration tests when correctness depends on ordering, retries, clocks, leases, quorum, recovery, or concurrency—especially for payments, inventory, schedulers, and replicated stores.

Correctness invariant

Every test names the safety or liveness property it evaluates, records a complete enough operation history to check it, and can reproduce a failure from code version, workload, nemesis schedule, and random seed.

From model to CI promotion evidence

Jepsen-style inventory reservation test

Five nodes expose reserve(sku, request_id) and available(sku). The invariant is reserved ≤ initial stock and one request_id changes stock at most once. Clients run concurrent reserve/read operations while a nemesis partitions leader from majority, pauses processes, skews clocks, and drops acknowledgements.

History record

invoke/completion + process + operation + value + monotonic time

Unknown outcomes remain indeterminate rather than guessed failed. A checker searches for a legal sequential order between each invocation and response.

State space

20 commands × many reorderings → model reduction required

Deterministic simulation controls network, timers, and scheduling; property-based generation shrinks a failing sequence to the smallest reproducible history.

Seed artifact

build SHA + seed + fault schedule + history + logs

A failure that cannot be replayed is still evidence, but it is expensive evidence. Preserve all nondeterministic inputs and virtual time.

Decision

The test fails if stock goes negative, a duplicate id reserves twice, or an acknowledged write disappears. It separately measures liveness: after healing, a new reservation must complete within the recovery objective.

How the mechanism holds

Follow how each part protects the rule above. The mechanism explains what the system does; the reasoning explains why that action is sufficient under the stated assumptions. If an assumption changes, the guarantee may change too.

Nemesis catalog

Partition majority/minority and asymmetric links; add loss, delay, reorder, process pause, disk full, clock jump, slow I/O, duplicate delivery, and dependency brownout. Match faults to actual architecture.

Why it holds: Each fault reaches the intended boundary and telemetry confirms injection rather than assumption.

Linearizability history

Capture invocation and completion intervals, including timeout/unknown results. A checker asks whether operations can appear atomic between those points while preserving real-time order.

Why it holds: The same checker rejects a deliberately broken implementation and accepts a known-correct model.

Model/property-based testing

Define commands, preconditions, state transitions, and invariants. Generate sequences and concurrency, then shrink failures. Examples: no negative stock, monotonic fencing token, dedup key applies once.

Why it holds: Every found sequence replays deterministically against model and system.

Deterministic simulation

Replace wall clock, randomness, network, and scheduler with controlled versions. Explore many years of virtual faults quickly, while retaining a smaller real-environment suite for kernel/cloud behavior.

Why it holds: A recorded seed reproduces the same event order and invariant failure.

CI promotion ladder

PR

Fast deterministic suite

Model and property tests explore thousands of seeds with virtual time.

Response: Any invariant failure blocks merge and stores the minimized seed.

Night

Ephemeral cluster faults

Real brokers/databases receive process, network, disk, and clock faults.

Response: Quarantine infrastructure failures only with evidence; correctness failures always block.

Stage

Production-shaped game day

Load, retries, failover, and observability are exercised with abort gates.

Response: Promotion requires recovery within RTO and reconciliation to zero.

Prod

Bounded experiment

A tiny blast radius validates assumptions unique to production.

Response: Automated abort on SLO/invariant signals; no first-time destructive hypothesis.

How to choose among alternatives

Simulation vs real cluster

Choose when: Simulation explores schedules cheaply; real clusters expose implementation and infrastructure behavior.

Caution: Use both—simulation fidelity gaps and real-test nondeterminism fail differently.

Safety vs liveness test

Choose when: Safety asserts bad things never happen; liveness asserts progress eventually resumes.

Caution: A system can preserve safety by refusing all writes, so test recovery bounds too.

Chaos vs correctness

Choose when: Chaos validates resilience outcomes; history checking validates data semantics.

Caution: A dashboard staying green does not prove linearizability or exactly-once effects.

Running it day to day

Measurements tell you what is happening; alerts tell an operator when action is needed. A rollout plan limits how many users encounter a change before it is checked. A runbook records the steps to diagnose and recover from a known failure.

Measure

  • tests/seeds explored, coverage by fault class, and shrink time
  • history completeness, checker result, unknown outcomes, and invariant deltas
  • recovery time, reconciliation residue, and flaky-infrastructure rate

Alert

  • promotion artifact missing or checker cannot parse history
  • previously passing seed regresses
  • game-day abort gate or recovery objective breaches

Roll out safely

  • start with a written model and deliberately broken control
  • add one deterministic fault class at a time
  • promote to staging and bounded production only after replay is reliable

Runbook

  • freeze seed/history/log artifacts
  • replay against same build and minimized model
  • classify product vs harness fault, fix, and retain seed as permanent regression

Practice scenario

Prove a distributed lock prevents stale writers after lease expiry and a long process pause.

Questions worth asking next

  • Which nemeses run?
  • What history is recorded?
  • What blocks deployment?

A solid answer sketch

Model monotonically increasing fencing tokens and require storage to reject lower tokens. Generate acquire/write/release operations, pause the holder beyond lease, partition it, and let another holder acquire. Record invocations/results and block promotion on any accepted stale write; preserve the seed.

Common mistakes

  • • checking only HTTP success rate
  • • dropping timed-out operations from history
  • • running random chaos without a named invariant or reproducible schedule

Last reviewed: 6 August 2026

Choosing and explaining distributed system testing

A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.

Explain the core idea

Idempotency and timeouts need duplicate-delivery tests, not just unit mocks.

Chaos experiments should match game-day scenarios with rollback criteria.

Strong consistency claims need concurrency tests, not single-thread demos.

When it belongs

Design includes leader election, queues, or replicated storage.

Incidents repeat the same retry or partition class.

Team ships correctness-critical paths (payments, inventory).

How it fails

Staging lacks production-like latency variance.

Tests never run during deploy window.

Flaky tests ignored: real bugs hidden.

Trade-offs to weigh

Jepsen-style tests are slow but high signal.

Chaos in prod finds real issues; needs blast radius controls.

Model checking requires simplified spec of system.

Common misunderstandings

Mocking away network failure in integration tests.

Testing only happy path saga.

Assuming exactly-once without duplicate injection.

Wrap-up

You have seen the failure, the mechanism, and the operational signals. Use these exercises to check the ideas stuck, then follow the references for the primary sources.

Exercises

  1. Payment charge with client timeout after commit. Design the test: what fault do you inject, what history fields do you record, and what blocks merge? Decide whether HTTP 200 rate is enough evidence.
  2. Quarantined "flaky" Jepsen seed. A linearizability failure reproduces on 3 of 40 seeds. Do you ship? What artifact must CI retain, and who owns proving harness vs product fault?
  3. Simulation vs real cluster budget. You have 1 engineer-week before launch. Allocate time across model tests, ephemeral cluster faults, and one game day. Which claim remains unproven if you skip one layer?

Topic-specific references

Key takeaway

Distributed correctness is an empirical claim. Inject faults that match your design assumptions, automate contract tests, and run game days before customers become the chaos experiment.

Practice and further reading

Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.

Focus for this lesson: Test invariants through model/property checks, deterministic simulation, linearizability histories, Jepsen-style faults, reproducible seeds, and promotion gates.

Explain the decision

Why is fault injection without history checking insufficient?

Read the reasoning

The system may stay up while violating ordering or data invariants; record operations and verify them against the stated consistency model.

Apply it to a production scenario

A seeded partition test finds one lost acknowledged write in 10,000 runs. Ship?

Read one approach

No for a durability invariant; preserve seed/history, minimize and reproduce, fix, add regression, then rerun the promotion suite.

Work through the numbers

A deterministic test explores 200 seeds across 12 fault schedules and 5 workloads. How many runs?

Read the worked calculation

12,000 runs; parallelize while retaining seed, build, topology, and history artifacts.

Reason through a failure

What must a linearizability checker prove?

Read the recovery reasoning

Completed operations can be ordered consistently with real-time precedence and the sequential object specification; ambiguous operations are modeled explicitly.

Sources and further study

Continue with Multi-Region Data & Conflict Resolution.