System Design Communication
Event-Driven Architecture
An Uber ride request is one API call; Matching, pricing, fraud, receipts, and ops all still need the fact. The request path should not chat synchronously with every domain on the critical path. The product moves forward by publishing facts: RideRequested, DriverMatched, TripCompleted.
architecture means services publish what already happened, and other services consume those events later. Producers and consumers scale separately, if every event still has an owner, schema, retry plan, and key.
Ride journey
Ride request
Commit local state + outbox fact.
Same transaction as the business row.
Publish after commit via relay/CDC.
Consumers
Pricing, fraud, receipts, dashboards.
Try it: split outbox lag from broker consumer lag
Event-driven checkout writes the order row and an outbox row in one database transaction, then a relay publishes integration events. Outbox lag is inserted minus published rows. Downstream consumers can lag even when the relay is healthy, and the relay can fall behind even when consumers are idle.
Transactional outbox
after DB commitPending rows: 0
Relay → broker
Published total: 0
Outbox not draining at these rates
Downstream consumers
900/s fixedConsumer lag: 0
- Outbox inserts total
- 0
- Outbox lag rows
- 0
- Published total
- 0
- Broker consumer lag
- 0
Orders commit to the database and the outbox in one transaction. The relay publishes afterward; broker consumer lag is separate. Advance time to compare outbox backlog with downstream lag.
Commit without publish. Pause the relay with writes at 1,200/s. Outbox lag grows by 1,200 rows per second while the API still returns success. Resume relay at 1,000/s to drain, but only if writes slow below that rate.
Monitor both gaps. Page on outbox relay lag for the dual-write gap and on consumer lag for downstream processing. They fail in different places and need different runbooks.
Model scope: one relay, one consumer group at 900 events/s, and no poison messages.
Problem → naive → break
Concrete problem. A ride request must wake matching, pricing, fraud, receipts, and ops, without chaining every domain on the request latency budget.
Naive system. After INSERT rides, call bus.publish(RideRequested) as a second step. Or fan out synchronous RPCs to every domain before returning 200.
Break. The DB commits; the process dies before publish. Matching never starts. Or fraud times out and the whole request fails even though the ride row is durable. Dual-write gaps and sync fan-out both produce ghost tickets.
Counterexample to "wrap DB and Kafka in one distributed transaction." Most brokers are not participants in your database ACID boundary. Pretending they are invents a 2PC you do not operate: use an (or ) plus idempotent consumers instead.
Correctness invariant
What must never become false
Every committed business transition eventually yields its corresponding at least once; no uncommitted transition is published. Every consumer side effect is applied at most once per stable key. Multi-step money/inventory flows end in a product-legal terminal state (succeeded, compensated, or manual review), never silent half-commits. Delivery on the wire remains at-least-once; do not claim broker "exactly once" makes external effects exactly once.
What actually happens
The safe pattern is not "write database, then hope publishing works." Save the business change and event record together with a , then let a relay or publish after commit. Billing, search, and inventory react from the bus with idempotent handlers.
Separate , , and -style events: contracts differ. Multi-step money paths use a with compensation instead of pretending distributed ACID is free.
Your design process should stay concrete: explicit contracts, outbox publishing, sagas for multi-step business flows, and plans for fixing bugs.
Read the diagram definition
sequenceDiagram
autonumber
participant API as Order API
participant DB as Orders DB
participant OB as Outbox table
participant R as Relay or CDC
participant B as Event bus
participant C as Consumers
API->>DB: Write order row
API->>OB: Write OrderPlaced event in same transaction
API-->>API: Commit local transaction
R->>OB: Poll or stream committed rows
R->>B: Publish integration event
B->>C: Deliver to billing, search, inventory
C-->>B: Ack after idempotent handlingFailure timeline: dual-write crash (naive)
Read the diagram definition
sequenceDiagram
autonumber
participant API as Order API
participant DB as Orders DB
participant B as Event bus
API->>DB: COMMIT order
Note over API: Crash before publish
API--xB: publish never sent
Note over DB,B: Order exists; no consumer ever notifiedT0 BEGIN; INSERT orders(...); COMMIT → durable order T1 process crash / timeout before bus.publish T2 bus empty; billing/search/inventory never notified T3 client retry may create a second order OR leave the first orphaned
What to watch
Every event should have an owner, schema, purpose, retention policy, and sample payload.
Correlation and causation ids connect the original command to all downstream reactions.
Projection rebuilds should be idempotent and bounded by retained history or snapshots.
How to choose an event-driven design
1. Who needs the fact? Cross-team reactions belong on . Keep rich inside the boundary that owns them.
2. Can publish fail after commit? If yes, ship the (or CDC) before claiming atomicity.
3. Do multiple services commit money or inventory? Prefer a with when steps, timeouts, and compensations must be auditable.
Simple rule of thumb
Prefer explicit integration events over synchronous chats between domains; pair with idempotent consumers and sagas for money-adjacent workflows. Ship the outbox before pretending publishes are atomic with SQL commits.
Version schemas, publish replay-friendly keys, and instrument lag. You will thank yourself during incidents.
Capacity math: event-driven systems
Event-driven capacity starts with user actions, then multiplies by emitted facts and downstream reactions. The outbox, bus, consumers, DLQs, and retained history each need separate budgets.
User actions
Domain events
Outbox relay
Integration stream
Projection consumers
Notification consumers
Analytics consumers
Lag and replay dashboards
| Scenario | Estimate | Design implication |
|---|---|---|
| Notifications | 1M users × 8 triggers/day × 10× peak ≈ 930 events/sec | Partition by user id for per-user order; DLQ permanent template or token failures. |
| Feed events | 1M users × 5 posts/reactions/day × 8× peak ≈ 463 events/sec before fan-out | Separate immutable feed facts from materialized timeline writes. |
| Analytics clicks | 1M users × 100 clicks/day × 6× peak ≈ 6.9k events/sec | Use batching, compression, sampling policy, and long retention for replayable aggregates. |
| Chat events | 1M users × 20 messages/day × 20× peak ≈ 4.6k events/sec | Key by conversation id; keep presence and receipts on separate streams if they dominate volume. |
How to choose
Interview capacity answer for events
- Start with product actions: Translate users and actions/day into average and peak events/sec before picking any bus shape.
- Separate facts from side effects: One event may drive notifications, feeds, analytics, and search; size each consumer group independently.
- Bound replay: Retention storage and rebuild time decide how far back you can safely fix projection bugs.
Outbox relay sizing
If order writes insert 1,500 outbox rows/sec and the relay publishes 2,000 rows/sec, a 900k-row lag drains in900,000 / (2,000 - 1,500) = 1,800 seconds(~30 minutes). During that window search and inventory are stale by design: page on oldest unpublished age as an SLO, not only broker consumer lag. Track outbox lag rows separately from broker consumer lag because they fail in different places.
Event shapes
Domain events stay inside a bounded context; integration events cross wires with stable versioning; notification events stay thin when consumers should fetch authoritative state via API.
A domain event describes a meaningful change in the model owned by one service. For example, OrderPlaced may contain the line items and the pricing decision that were accepted together. Internal handlers can react to that fact without asking the order service to know every reaction. Publishing domain events does not require event sourcing: an ordinary relational application can emit events after changing its current-state tables.
An integration event is the versioned contract that other teams consume. The order team might publish OrderConfirmed with an order ID, customer ID, total, currency, and schema version while keeping internal pricing details private. Consumers depend on what these fields mean, so changing a currency field from major units to minor units is a breaking change even when its data type stays unchanged.
Pointers like { type, id } so payloads stay small and you avoid stale embedded snapshots. A thin notification trades message size for a dependency on the source service. If an order changes twice before a consumer fetches it, the fetch returns the latest state and may skip the intermediate transition. That is fine for refreshing a search document, but insufficient for reconstructing every historical price change. Include a version or a richer event when the intermediate fact matters.
Event catalog card
Event name, owner, producer service, schema version, sample payload.
Topic, partition key, consumer list, retention, replay policy.
Idempotency key, sensitivity flag, deprecation owner, DLQ handling.
Event sourcing sketch
Store the append-only history as truth; materialize current state by (plus periodic snapshots so replay stays bounded).
| Seq | Event | Δ | Balance |
|---|---|---|---|
| 1 | AccountCreated | $0 | $0 |
| 2 | MoneyDeposited | +$1000 | $1000 |
| 3 | MoneyWithdrawn | -$200 | $800 |
That is the heart of : operational cost rises with schema and projection discipline.
CQRS + events
Commands mutate authoritative write models (often event streams); asynchronous projectors build read models tuned for queries: accept eventual consistency between sides. That split is .
WRITE
READ
A projection is a derived view, such as orders indexed by customer or searchable product documents. Its worker reads an event, updates the derived record, and records progress so it can resume after a crash. Rebuild a changed projection into a separate destination, catch it up to live traffic, compare results, and switch readers only after validation. Replaying directly over the serving index can expose a half-rebuilt view.
Schema evolution
Treat published events like API contracts: additive changes by default, compatibility modes enforced by registry or CI checks. That is .
Registry-guided compatibility
// v1: additive evolution keeps readers tolerant
type OrderPlacedV1 = { orderId: string; userId: string; total: number };
// v2: new optional field when registry allows forward compatibility
type OrderPlacedV2 = OrderPlacedV1 & { currency?: string };Explicit version field
{ "version": 2, "type": "OrderPlaced", ... }Upcast while reading
def upcast_v1_to_v2(evt):
evt.setdefault("currency", "USD")
evt["version"] = 2
return evtHow to choose
How to discuss event schema evolution
- Start with contracts: Published events are APIs. Name owners, compatibility mode, examples, and rollout policy.
- Avoid breaking changes: Add optional fields, keep old fields through a deprecation window, and tolerate unknown enum values.
- Replay implication: New code must handle old retained events, so defaults and upcasters belong in the design.
Sagas
Replace distributed ACID with coordinated local commits plus compensating actions: via events or via a workflow engine.
Happy path
Failure + compensation
Orchestration improves readability for long flows; choreography minimizes coupling: pick per bounded context.
Read the diagram definition
sequenceDiagram
autonumber
participant O as Order service
participant P as Payment service
participant I as Inventory service
participant N as Notification service
O->>P: PaymentRequested
P-->>O: PaymentCaptured
O->>I: ReserveInventory
I-->>O: InventoryFailed
O->>P: RefundPayment
O->>N: OrderCancelledRead the diagram definition
stateDiagram-v2
[*] --> Pending
Pending --> Succeeded: all steps committed
Pending --> Failed: timeout or rejection
Failed --> Compensating: start undo commands
Compensating --> Compensated: undo committed
Compensating --> ManualReview: compensation retry exhausted
ManualReview --> Compensated: operator fixes stateSaga failure matrix
| Failure | Required design response |
|---|---|
| Step succeeds, ack lost | Retry command with idempotency key and read current step state before repeating side effects. |
| Step times out but later completes | Accept late completion only if saga state still permits it; otherwise compensate. |
| Compensation fails | Retry with backoff, alert, and move to manual review after bounded attempts. |
| Duplicate event arrives | Ignore using processed-event id or monotonic saga version. |
| Manual review required | Expose enough correlation metadata and audit trail for an operator to resolve safely. |
How to choose
Interview guidance for event-driven transactions
- Do not claim 2PC: Across services, prefer local transactions plus compensation unless the system truly controls all participants.
- Money paths: Use orchestration or a durable workflow when steps, timeouts, and compensations must be explicit and auditable.
- Idempotency: Every command and event handler in the saga needs an idempotency key because retries are normal.
Transactional outbox
Persist domain rows and outbox rows in one database transaction; a relay or publisher ships events downstream after commit. This closes the "database committed but publish failed" gap. The relay can still publish an event twice if it crashes after the broker accepted the message but before recording completion. At the consumer, insert the event ID and apply the database change in the same transaction; a duplicate ID then prevents repeating that change. For an external payment or email API, use its own idempotency contract because your local transaction cannot make the external call atomic.
BEGIN;
INSERT INTO orders ...;
INSERT INTO outbox (type, payload, created_at) ...;
COMMIT;
-- Relay / CDC ships outbox rows to the busReplay
Fix projection bugs
To repair a projection bug, start a new consumer at a known checkpoint and recompute the affected view from retained events. Keep the old view serving until the replacement catches up. Disable real-world side effects during replay: rebuilding an order dashboard must not resend every receipt or capture every payment. Compare counts and business totals before switching readers.
New downstream consumer
Start from earliest retained offset or snapshot boundary: budget storage and time.
Debug timelines
Replay entity stream in order inside a sandbox to compare intermediate states.
Replay and event storm calculation
One user action can emit 3 facts, each read by 4 consumers, and each consumer can create 2 side-effect calls:side-effect calls/sec = actions/sec * 3 * 4 * 2. Also check rebuild time: if 600M retained events replay at 50k/sec, rebuild takes about 3.3 hours before validation and cutover.
Illustrative domains
Mobility marketplace
Lifecycle events (requested, matched, completed) fan into pricing, fraud, receipts, and ops dashboards: each subscriber scales independently.
Streaming product telemetry
UI and playback signals feed experimentation, personalization, and reliability pipelines with strict schema governance.
Operating EDA
Invest in correlation ids, consumer lag dashboards, playbooks, and tracing that stitches async spans. Otherwise failures feel like ghosts.
Correlation metadata
{
"correlationId": "corr-abc",
"causationId": "evt-payment-456",
"type": "InventoryReserved"
}Minimum tooling
- Event catalog with owners
- Lag + DLQ alerts
- Trace propagation across producers/consumers
Outbox failure drills
| Failure | What breaks | Guardrail |
|---|---|---|
| DB commit succeeds, relay is down | Business row exists but event is delayed | Monitor outbox row age and drain rate separately from consumer lag |
| Relay publishes twice | Consumers may repeat side effects | Require idempotency keys and dedupe on consumer write boundary |
| Schema change ships early | Old consumers fail or silently drop fields | Backward-compatible rollout plus catalog ownership and version gates |
Break the improved design
Outbox closes the dual-write gap, then fails differently: a stuck relay grows unpublished age until projections silently diverge; relay republish after lost ack creates duplicates that double-charge without ; a compensation that itself fails leaves money captured and inventory free until manual review.
Schema changes that are not additive break old consumers during . hides the end-to-end graph so timeouts and partial failures become ghost tickets: prefer when money paths need an auditable state machine.
Trade-offs
Outbox vs CDC
Outbox and CDC solve different parts of the same workflow and can be combined. The outbox defines which business event is committed with the application change; CDC is one way to carry that committed row to the broker by reading the database change log. Polling is another relay mechanism. Direct CDC from business tables requires additional translation because a row-level update does not automatically express a stable business event.
Choreography vs orchestration
Choreography lets each service react independently, but the complete workflow is spread across handlers and subscriptions. Orchestration stores the workflow state in one place and explicitly decides which step or compensation comes next. Choose based on the difficulty of understanding and recovering the whole process: adding a durable coordinator is often worthwhile when several steps can time out or require human resolution.
Fat events vs notifications
Carried state eases consumers; thin pointers force lookups but avoid stale embedded snapshots and privacy deletion pain.
Exercises
1. Design RideRequested → matching + fraud + receipt
No distributed transactions. What is in the first local TX? What happens if broker ack is lost after publish? How long do consumer dedupe keys live? Where does the SMS live if the provider cannot share the receipt TX?
2. Pick choreography or orchestration
Payment → inventory → ship with a 15-minute inventory hold. Argue for orchestration or choreography, name the terminal states, and list the alerts that page a human into ManualReview.
3. Outbox lag SLO
Product promises search is stale by at most 2 minutes. Given 1,500 writes/s and a relay that drops to 800/s for 10 minutes, compute peak lag and say whether you scale the relay, shed analytics consumers, or pause writes, and why.
Going deeper
Production deep dive: events as owned contracts
Event-driven architecture is safe when facts, requests, and delivery plumbing are distinguished. An event reports something that already happened; a command asks one named owner to decide whether something may happen.
What must stay true
A committed domain change must eventually produce its integration event, and replaying that event must not repeat the resulting business side effect.
How the pieces fit
Command
intent to one owner
Domain transaction
state + outbox row
CDC relay
committed rows only
Event bus
versioned fact
Inbox + handler
dedupe + projection
Outbox input
1,500 rows/s
One integration event per committed order.
Relay capacity
2,000 rows/s
Only 500 rows/s of net recovery while traffic continues.
900k backlog
30 min
900,000 ÷ 500 net rows/s.
Event versus command
Name facts in past tense (OrderPlaced) and do not reject them downstream. Name commands as imperatives (ReserveInventory), route them to one authority, and return an accepted/rejected outcome. Treating a broadcast event as a command creates multiple accidental owners.
Transactional outbox + CDC
Write the aggregate and outbox record in one local transaction. CDC reads only committed log entries and publishes at least once. The consumer inserts event_id into an inbox in the same transaction as its projection update; a duplicate event then becomes a no-op.
Compatibility rules
Prefer additive optional fields, stable meanings, tolerant readers, and unknown-enum handling. A rename is add → dual-publish → migrate consumers → deprecate → remove, never an in-place field swap while retained history still contains the old shape.
Catalog and ownership
The catalog records schema, producer, domain owner, data classification, partition key, retention, known consumers, compatibility mode, and deprecation date. Producer owns semantic correctness; each consumer owns its lag, inbox, retries, DLQ, and projection.
Tracing asynchronous work
Propagate trace context when useful, plus durable correlation_id, causation_id, event_id, aggregate_id, and schema_version. A new consumer span should link to the producer span because processing can occur long after the original trace has ended.
Worked example: Crash-window timeline
- 1T0: Order row and outbox row commit together; the API returns success.
- 2T+4 s: CDC publishes event E42, but crashes before recording its own acknowledgement.
- 3T+9 s: CDC publishes E42 again. The consumer inbox already contains E42, so the projection transaction exits without charging twice.
At-least-once transport plus an atomic consumer inbox produces an effectively-once business outcome without claiming impossible global exactly-once delivery.
Contract and ownership failures
| Scenario | Consequence | How to contain it |
|---|---|---|
| Command broadcast as an event | Several consumers perform the same authoritative action. | Route commands to one owner; publish the accepted fact afterward. |
| Consumer deploys before additive field | Projection expects data that old events lack. | Default missing fields and test against retained schema versions. |
| Trace context expires | Incident has disconnected spans. | Query durable correlation/causation IDs in logs and event catalog. |
What to watch in production
- Alert separately on outbox age, broker publish errors, consumer lag, inbox conflicts, and DLQ age.
- Run schema compatibility checks in CI against every supported consumer version.
- Practice rebuilding a shadow projection from the retained log before relying on replay in an incident.
A scenario to reason through
The database commit succeeds, the process dies before publishing, and the customer already saw “order placed.” What makes the event appear later?
One solid approach
The durable outbox row survived the process. A restarted poller or CDC reader publishes it. Consumers dedupe with event_id; alerting uses outbox oldest-row age so this gap is visible.
What to remember
Prefer explicit integration events over synchronous chats between domains; pair with idempotent consumers and sagas for money-adjacent workflows. Ship the outbox (or CDC) before pretending publishes are atomic with SQL commits. Version schemas, publish replay-friendly keys, and instrument lag. You will thank yourself during incidents.
Topic-specific references
- Transactional Outbox pattern : same-commit publish capture without claiming DB↔broker 2PC.
- Saga pattern (choreography and orchestration) : local commits plus compensations instead of distributed ACID.
- Confluent: Designing Event-Driven Microservices : contracts, schemas, and operational concerns for event streams.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Separate events from commands, publish atomically with state, evolve schemas, own catalogs, trace causality, and reconcile asynchronous workflows.
Explain the decision
What is the difference between an event and a command?
Read the reasoning
A command asks a named owner to act and may be rejected; an event records a past fact that subscribers may independently react to.
Apply it to a production scenario
An order commits but OrderCreated is never published. How is the dual-write gap closed?
Read one approach
Write an outbox record in the same database transaction, relay through CDC, and let consumers deduplicate a stable event ID.
Work through the numbers
A service emits 8,000 events/s and average end-to-end fan-out is six. Estimate consumer deliveries/s.
Read the worked calculation
About 48,000 deliveries/s, then model retry amplification, retention, and payload size.
Reason through a failure
What must event schema evolution preserve?
Read the recovery reasoning
Existing consumers continue decoding and interpreting fields correctly; incompatible semantics require a new version and measured migration.
Sources and further study
Continue with Stream Processing.