System Design History
Event Sourcing & Projection Rebuilds
A support agent opens a disputed Instant Payment. They need every status change, who approved it, and why the balance flipped, not just the final row. Mutable CRUD (create, read, update, delete that overwrites the current row) cannot answer that without bolting on an audit trail after the fact.
That is the problem this page solves. stores state as an append-only log of facts in an . Current views are built by replaying those events, so you can audit history and rebuild a wrong read model. The cost is schema evolution, , and enough replay capacity to finish before your incident budget runs out.
Write → read journey
Command accepted
Validate business rules, then append domain facts to the aggregate stream.
Keep an immutable, ordered history per aggregate that writers never rewrite in place.
Consume the log into read tables optimized for balances, timelines, and search.
Fix projector bugs by replaying from a checkpoint, not by hand-patching rows.
Try it: grow the event log faster than a projection rebuilds
The write model appends immutable events. The read model is a derived projection that replays from a checkpoint. Pause the projector or rebuild from scratch to see lag, catch-up, and why snapshots plus checkpoints matter in production.
Event store (source of truth)
- #1 AccountOpened · not projected yet
Balance projection (query model)
$0
1 event(s) ahead of the projector. Queries return stale balance until catch-up.
- Events in log
- 1
- Applied through
- #0
- Projection lag
- 1
- Projector
- Running
Log length 1 · projection through event #0 · lag 1 · balance $0. Append ledger events on the left. The read model catches up when the projector runs.
Rebuild is replay, not magic. Rebuilding reapplies every event after the checkpoint. Long logs need snapshots so rebuild time stays bounded.
Upcasters belong in the replay path. When event schemas change, transform old payloads during projection, not only on new writes.
What actually happens when the log is truth
The Instant Payment story above is the default whenever regulators, disputes, or analytics ask temporal questions: "What was the balance at 14:02?" "Who approved the override?" "Rebuild the fee report after a pricing bug." A single mutable row answers only "what is true now?" So teams reach for : commands append facts; turn those facts into read-optimized tables.
The naive alternative looks simpler. CRUD updates overwrite accounts.balance. A separate audit table is appended "when we remember." Wrong read models are hand-patched. Schema changes mutate historical meaning in place. Then a projector bug undercounts fees for 30 million orders, engineers "fix" production rows with SQL, and two days later dispute tooling disagrees with finance exports because nobody can replay the same history twice. The audit table is incomplete for the week the bug shipped.
With the log as truth, you fix the consumer and replay from a instead of patching corrupted rows by hand. Users still feel as "I just paid, why is my balance old?" Plan lag budgets, version events with (converting old payloads into the shape modern projectors expect), and rehearse rebuilds before promising same-day recovery mid-incident.
Append-only logs are not always better. Simple CRUD with three columns and no temporal questions does not need this pattern. Without rebuild drills, lag SLOs (service level objectives for how far the read model may trail), and upcasters, it becomes a liability.
How to choose event sourcing
Before you commit to an append-only write path, run the decision through three questions and the fit checklist below. If you cannot name a rebuild playbook and a lag budget, prefer ordinary storage plus an audit log.
1. Do you need immutable history? Audit, disputes, and temporal queries are strong signals that pays off; a settings page with rare disputes usually does not.
2. Multiple read shapes from one write model? pay off when balances, timelines, and search indexes diverge; simple CRUD rarely needs that fan-out.
3. Can the team operate replay? Without rebuild drills, SLOs, and , the pattern becomes a liability rather than an audit win.
Good fit
- Audit or regulatory replay needs every status change, not just the final row.
- Temporal queries ask what was true at a past time for disputes or reports.
- Several read models share one write stream of domain facts.
- Complex domain lifecycles make current state hard to explain without history.
Poor fit
- Simple CRUD tables with few columns and no temporal questions to answer.
- Team lacks rehearsed replay ops, lag budgets, and versioned event handling.
- UI promises instant reads without a budget for how far projections may trail.
Rule of thumb
Events are facts; are disposable views. If you cannot name a rebuild playbook and a lag budget, prefer ordinary storage plus an audit log.
From problem to mechanism
Here is the fee-bug scenario again, slowed down so you can see where hand-patching fails and what the log-as-truth design changes.
Problem. Product analytics and dispute tooling need rebuildable history after a pricing mistake, not a best-effort audit table that was incomplete the week the bug shipped.
Naive system. Overwrite accounts.balance, append audit rows when someone remembers, and SQL-patch wrong projection rows in place when a bug undercounts fees for 30 million orders.
Break it. Two days later finance exports and dispute tooling disagree because nobody can replay the same history twice, and the incomplete audit trail cannot settle the argument.
Mechanism. Append to an ; build disposable ; accelerate replay with a ; evolve old payloads with ; and treat as an SLO. Rebuild into a new namespace (blue/green), dual-read sample keys, cut over an alias, and keep the old view for rollback. Projector rollback changes the alias; it does not rewind the event log.
Correctness invariant
Deterministic replay requires recording the facts a decision used. A projector that calls today’s exchange-rate API while replaying a five-year-old payment can produce a different answer on every rebuild. Store the relevant rate, effective time, or stable reference in the event contract and keep projection logic deterministic. Retaining an event forever is not useful if its meaning depends on external data that no longer exists.
An aggregate stream is append-only and version-monotonic. A checkpoint advances only after all effects through that position are durable, so replaying any prefix produces the same projected state. Events are facts; projections are disposable views. Never treat the projection as the write-side source of truth.
Command → stream → project → rebuild
The mechanism above names four operational pieces. Each one exists so two writers cannot diverge, replay stays safe, and a bad projector can be replaced without rewriting history.
Optimistic append
A command first reconstructs the aggregate’s current state and validates its rule, then appends new events only if the stream still has the version it read. If two callers both read version 12, only one append at expected version 12 succeeds; the other must reload and re-evaluate. This protects one aggregate’s history, not arbitrary invariants across many streams. An event such as PaymentCaptured records an accepted fact rather than a request that still might fail.
Idempotent projectors
A projector transforms events into a queryable view and stores its progress in the same transaction as its output writes where possible. A crash before commit repeats safe work; a crash after commit resumes after the saved position. If a projector writes multiple stores, one checkpoint cannot atomically certify all of them without extra coordination or reconciliation. Historical replay must be isolated from email, payment, and other non-repeatable external actions.
as acceleration
An aggregate snapshot stores enough state at stream version N to avoid replaying the entire aggregate on every command. It has its own schema version and can be discarded or replaced if invalid, provided the required underlying history is retained. A snapshot of an account balance may help reconstruct that account but cannot necessarily rebuild a new report needing each historical transaction’s category. Choose snapshot contents for the replay task rather than assuming every snapshot accelerates every projection.
Blue/green projections
Rebuild into a new namespace, dual-read sample keys, cut over an alias, and keep the old view for rollback. The event log stays untouched.
Rebuild playbook
Deploy the corrected projector into a separate namespace with independent checkpoints. Read the retained history or a compatible snapshot plus its following events, while the live projector continues serving. Run old-version events through tested upcasters that translate their representation without inventing facts missing from the original event.
Compare aggregate totals, invariants, and representative queries, then catch the replacement up to the live boundary before switching readers. Keep the previous projection and its code version during the rollback window. Replaying the log should repair a wrong derived view; correcting a wrong historical business fact normally requires a new compensating or correction event rather than editing history invisibly.
Numbers to know: rebuild capacity before you promise it
The playbook only works if you can finish replay inside the incident budget. Worked example: a fee bug omitted events from account summaries. The holds 600M events averaging 700 bytes, with every 500 events, while live ingestion continues at 10K events/s.
Raw history
600M × 700 B ≈ 420 GB
Bandwidth and deserialization, not only event count, set rebuild capacity.
Historical replay
600M / 80K/s ≈ 2.1 hours
Reserve headroom so the live path and old keep their SLOs.
Catch-up after history
80K − 10K = 70K/s net
At 80,000 processed events per second with 10,000 new events per second arriving, net backlog reduction is 70,000 per second. Clearing an initial 600 million-event backlog while ingest continues therefore takes about 600M/70K=8,571 seconds, or 2.38 hours, before validation and cutover. The 2.1-hour figure above counts only processing the original history; events arriving during that time still need to be consumed.
Smaller drill: 100M events at 50K/s ≈ 33 minutes, still too slow if you never rehearsed and promised "same-day" mid-incident.
Failure timeline, and what still breaks after the fix
Putting the pieces together, here is how a wrong projector cutover plays out, followed by the ways an improved design can still fail and the trade-offs you accept when you choose this pattern.
Timeline: projector v8 is wrong
Read the diagram definition
sequenceDiagram
participant Cmd as Command API
participant Store as Event store
participant V7 as Projection v7
participant V8 as Projection v8
participant UI as Read path
Cmd->>Store: append PaymentSettled (v=N)
Store-->>V7: project (live)
Note over V8: Bug found in fee handling
V8->>Store: replay from snapshot / offset 0
V8->>V8: catch up to live high-water mark
Note over V7,V8: dual-read sample keys
UI->>V8: atomic alias cutover
Note over V7: keep for rollback windowA Shadow compare finds 0.02% negative balances on v8 Do not cut over; retain checkpoint + mismatch samples B Root cause: upcaster misread legacy FeeWaived Ship projector v9 into a fresh namespace (do not mutate v8) C v9 catches live head; sampled balances match event totals Brief alias freeze; drain ordered tail; switch atomically D Post-cutover: latency rises, correctness holds Point alias back to v7 while indexes tune: log untouched
Still fails when…
- violates expectations if the UI promises instant reads without read-your-writes.
- Breaking event schemas without blocks replay of long-lived streams.
- Missing make full rebuilds longer than the incident budget allows.
- Emitting email or payment effects during replay duplicates side effects users already saw.
- Editing historical events to "fix" a schema destroys the audit integrity you bought.
- Sharing output tables between live and rebuilding projectors corrupts both read models.
- Deleting events to save space without an archive policy erases the only truth.
Trade-offs
- vs an audit table: choose ES when rebuilds are routine; CRUD stays simpler otherwise.
- cadence: snapshot when replay latency crosses an objective; too frequent raises write cost.
- Many multiply consumer fan-out; blue/green doubles cost during rebuild.
Replayable state
What production needs to hold
A production design needs a way to recognize trouble and a rule it preserves while recovering. A trigger is the measurement or event that starts a response. An invariant is a statement that must remain true, such as never confirming the same reservation to two customers.
Trigger metric or condition
Event sourcing earns its complexity when immutable audit, temporal questions, and several independently rebuildable views are core requirements—not merely because an append-only log is fashionable.
Correctness invariant
An aggregate stream is append-only and version-monotonic. A projection checkpoint advances only after all effects through that position are durable, so replaying any prefix produces the same projected state.
Command path and rebuild path
Command
expected stream version
Event stream
immutable facts
Snapshot
state at version N
Projector
idempotent checkpoint
Versioned view
atomic alias cutover
Rebuild a ledger projection while reads stay live
A bug omitted fee events from account summaries. The store has 600 million events averaging 700 bytes. Snapshots exist every 500 events. Production reads use projection_v7; a corrected v8 projector writes a fresh namespace and tracks a durable partition checkpoint.
Raw history
600M × 700 B ≈ 420 GB
Read bandwidth, deserialization, and write amplification—not only event count—set rebuild capacity.
Rebuild time
600M / 80,000 events/s ≈ 2.1 hours
At 80K events/s sustained. Reserve headroom so live ingestion and the old projection do not miss their SLOs.
Catch-up
live 10K/s, projector 80K/s → 70K/s net
After historical replay, subscribe from the captured high-water mark, close lag to seconds, then dual-read samples before cutover.
Decision
The team compares v7/v8 balances and known invariants, atomically changes a read alias to v8, preserves v7 for rollback, and never rewinds the canonical event log. A projector rollback means changing the view alias, not deleting events.
How the mechanism holds
Follow how each part protects the rule above. The mechanism explains what the system does; the reasoning explains why that action is sufficient under the stated assumptions. If an assumption changes, the guarantee may change too.
Snapshots
Snapshot aggregate state plus stream version and serializer version. Verify checksum and ignore a corrupt snapshot by replaying earlier history; snapshots are an optimization, never the source of truth.
Why it holds: Delete the newest snapshot and recover the same aggregate state from events.
Upcasters and event versions
Keep stored facts immutable. Convert old payload shapes into the current in-memory representation through deterministic upcasters, and version semantic changes as new event types.
Why it holds: Golden fixtures from every historical version deserialize during CI and full replay.
Projection idempotency
Use event_id or monotonic stream position, apply writes and checkpoint atomically, and reject version gaps. Side effects leave through an outbox rather than running directly during replay.
Why it holds: Processing every event twice produces an identical view and no external notifications.
Temporal queries
Replay to an event position or occurred-at boundary while preserving the distinction between event time and ingestion time. Use immutable correction events rather than rewriting disputed history.
Why it holds: An as-of query can explain its exact event prefix and projector version.
Recovery drill: projector v8 is wrong
Shadow comparison fails
Invariant checks find v8 negative balances for 0.02% of accounts.
Response: Do not cut over; stop v8 and retain its checkpoint and mismatch samples.
Bug is fixed
Upcaster interpreted a legacy FeeWaived event incorrectly.
Response: Publish projector v9; rebuild into a new namespace rather than mutating v8 in place.
v9 catches live head
Lag reaches the cutover threshold and sampled balances match event-derived totals.
Response: Freeze alias briefly, apply remaining ordered events, and switch atomically.
Post-cutover regression
Read latency rises even though correctness holds.
Response: Return alias to v7 while v9 indexes are tuned; the event log and write path remain untouched.
How to choose among alternatives
Event sourcing vs audit table
Choose when: Choose event sourcing when current state is defined by the event sequence and rebuilds are routine.
Caution: An audit table is simpler when CRUD state remains authoritative and history is only for compliance.
Snapshot cadence
Choose when: Snapshot when replay latency or stream length crosses an objective, not every arbitrary N events.
Caution: Too frequent snapshots increase write cost and schema-version burden.
Blue/green projections
Choose when: Use versioned outputs for safe rebuild and rollback while serving traffic.
Caution: They temporarily double storage and downstream load; capacity-plan both.
Running it day to day
Measurements tell you what is happening; alerts tell an operator when action is needed. A rollout plan limits how many users encounter a change before it is checked. A runbook records the steps to diagnose and recover from a known failure.
Measure
- projector lag by partition and oldest unprojected event
- events/sec, checkpoint age, and version-gap count
- shadow mismatch rate and invariant violations
Alert
- checkpoint stuck while input advances
- rebuild threatens live path capacity
- new projector mismatch exceeds cutover gate
Roll out safely
- ship upcasters with golden history fixtures
- rebuild a shadow namespace and dual-read samples
- cut over alias gradually and retain old projection for rollback
Runbook
- pause faulty projector without changing the log
- capture high-water marks and mismatch evidence
- fix, rebuild clean output, validate, cut over, and reconcile
Practice scenario
A pricing projection is wrong for 30 million orders. Rebuild it without stopping writes or emitting duplicate emails.
Questions worth asking next
- Where is the checkpoint committed?
- How do old event versions load?
- What is the rollback boundary?
A solid answer sketch
Use deterministic upcasters, an idempotent projector, and a new projection namespace. Capture a log high-water mark, bulk replay, catch up live traffic, validate invariants, and switch an alias. External effects are disabled during replay or emitted through a separately gated outbox.
Common mistakes
- • editing historical events to fit the new schema
- • sharing output tables between old and rebuilding projectors
- • running email or payment effects during replay
Authoritative references
Last reviewed: 6 August 2026
Choosing and explaining event sourcing and projections
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
Events are facts; projections are disposable views.
Rebuild capacity and projection lag are first-class SLOs.
Upcasting or versioned events are required for long-lived streams.
When it belongs
Audit, replay, or multiple read shapes justify append-only history.
CQRS already separates write and read models.
Regulatory or dispute workflows need immutable history.
How it fails
Projection consumer falls behind user expectations.
Breaking event schema without upcaster blocks replay.
Snapshot missing: full replay too slow for recovery.
Trade-offs to weigh
Perfect audit trail vs operational complexity.
Many projections vs consumer fan-out cost.
Snapshots speed recovery but add storage and consistency steps.
Common misunderstandings
Using event sourcing for a table with three columns.
Deleting events to save space without archive policy.
Treating projection as source of truth for writes.
Wrap-up
You have seen the failure, the mechanism, and the rebuild numbers. Use these exercises to check the ideas stuck, then follow the references for the primary sources.
Exercises
- Rebuild 30M wrong price rows. Pricing is wrong. Rebuild without stopping writes and without emitting duplicate emails. Where is the checkpoint committed? How do old event versions load via ? What is the rollback boundary?
- Set a lag SLO for "instant" balance UI. Users expect balance updates within 500 ms of a successful payment. Choose among wait-for-projection, read-your-write from the write model, or honest staleness. Quantify the capacity each option needs at 5K writes/s (see Numbers to know for how catch-up rate relates to live ingest).
- Decide event sourcing vs CRUD + audit. A settings page with five fields and rare disputes. A ledger with daily reconciliations and regulatory replay. Which gets , and what operational prerequisites would you demand before approving either?
Topic-specific references
Event-sourcing pattern sources: Fowler, Microsoft Architecture Center, and Greg Young's CQRS materials.
Key takeaway
Event sourcing trades mutable rows for replay power. Success depends on budgets, versioned events with , and rehearsed rebuilds, not on the novelty of append-only storage.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Version immutable events, snapshots, upcasters, idempotent projection checkpoints, live rebuild generations, temporal queries, and rollback.
Explain the decision
Why should old events usually not be rewritten?
Read the reasoning
They are the audit/source history; evolve interpretation with versioned schemas/upcasters and create corrective events when domain facts change.
Apply it to a production scenario
A projection rebuild must run while traffic continues. How?
Read one approach
Build a new generation from snapshot/checkpoint, consume live tail, compare invariants, then atomically switch serving and retain rollback.
Work through the numbers
Replaying 4 billion events at 100,000/s takes how long ideally?
Read the worked calculation
40,000 seconds, about 11.1 hours, before throttling, state writes, and live catch-up.
Reason through a failure
What must a projection checkpoint guarantee?
Read the recovery reasoning
State effects through offset N and recorded offset N commit atomically or idempotently, preventing skip/double-count after restart.
Sources and further study
Continue with Data Modeling & Schema Evolution.