System Design Integration

CDC, Transactional Outbox & Inbox

Checkout just saved a paid order in the database. The next line of code was supposed to tell inventory, email, and search. The process crashed before that publish ran. The order row is durable; inventory never heard. That silent split brain is a : two systems updated as separate steps with no shared atomic boundary.

Three patterns close that gap. streams committed row changes from the database log. A writes a named event in the same commit as the business row. An makes consumer retries safe when the bus delivers the same message more than once.

Closing the gap

Tail the commit log so every durable row change becomes a bus event.

Insert a named event row in the same commit as the business write.

Record each processed event id so retries never double-apply side effects.

Publish unpublished outbox rows after commit; page when lag grows.

What actually happens on a dual write

You need a local commit to cause an external effect (reserve stock, send mail, update search) without a distributed transaction across the database and the broker. Most brokers are not participants in your database's ACID boundary (the all-or-nothing commit rules). Pretending they are invents a two-phase commit you do not operate.

The naive path is two calls in application code: INSERT order then bus.publish(OrderPlaced). Crash after the commit and before publish: the database has a paid order; inventory never hears. Reverse the order and you publish phantom events about rows that never committed.

Invariant. Every committed business transition is eventually published at least once. No uncommitted transition is published. Consumer side effects apply at most once per stable key. The wire stays at-least-once: broker "exactly once" does not make SMTP or a payment API exactly-once.

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

sequenceDiagram
    participant App as Order service
    participant DB as Orders DB
    participant Bus as Message bus
    participant Inv as Inventory

    App->>DB: BEGIN; insert order
    App->>DB: COMMIT order
    Note over App,DB: Crash before publish
    App--xBus: publish OrderPlaced (never sent)
    Note over Inv: Inventory never learns
    Note over DB,Bus: Split brain: DB yes, bus no

T0  BEGIN; INSERT orders(...); COMMIT          → durable order
T1  process crash / network timeout before publish
T2  bus has nothing; inventory / email / search never notified
T3  client retries → second order, or a silent orphan

Which pattern fits this write path?

Once you see the dual-write gap, pick a producer pattern and always pair it with consumer idempotency. The table below is the fast filter; the sections after it show how each mechanism actually works.

Pattern	Mechanism	Best when
	Same TX: business row + event row; relay publishes after commit	You own the schema and want explicit events
	Connector tails WAL/binlog and streams row changes	Many tables, analytics, search projections
	Consumer stores processed message id with the side effect	At-least-once delivery with safe retries

Rule of thumb

Use an outbox when you control the publish contract and want named domain events. Use when many tables must feed search, warehouse, or caches with less application code. Put an inbox (or equivalent idempotency) on every consumer, regardless of how events were produced.

From problem to CDC

is Change Data Capture: notice that a row changed and emit that fact as an event, without the application calling publish() on every write path. The source of truth is the database's own commit log: , MySQL binlog, or MongoDB oplog.

The application commits through its normal database API. A connector reads committed changes from the database’s supported change stream or transaction log and records a durable resume position, such as a log sequence number. The representation and ordering details vary by database, so a connector must understand transaction boundaries rather than publish uncommitted row mutations.

For each captured change, emit a stable key, operation type, and the available before/after data. The connector checkpoints after the downstream handoff is acknowledged; a crash near that boundary may repeat an event. Consumers therefore use versions or deduplication. Deletes and tombstones must propagate too, or a search index can retain records long after the source removed them.

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

sequenceDiagram
    participant App as Order service
    participant DB as Orders DB
    participant Log as WAL / binlog
    participant CDC as CDC connector
    participant Bus as Message bus
    participant Search as Search / cache

    App->>DB: UPDATE order SET status=placed
    DB->>Log: append committed change
    DB-->>App: COMMIT
    CDC->>Log: read next LSN / offset
    CDC->>Bus: emit row-change event
    Bus->>Search: update projection

What CDC captures, and what it does not

CDC communicates storage changes such as a product price changing from 12 to 15. That is useful for reconstructing a search document, but does not always identify the business reason for the change. An OrderPlaced domain event expresses a completed transition with an order identifier and relevant version. Publishing named events from a transactional outbox provides that deliberate contract while a CDC connector handles transport from the committed outbox row.

A CDC pipeline needs a safe boundary between its initial snapshot and ongoing changes. If it copies a row at version 3 while version 4 commits during the scan, the stream must apply version 4 after that snapshot representation without missing or incorrectly reversing the change. Use the connector’s supported snapshot protocol and resume offsets rather than independently exporting rows and later guessing where to start the log.

Log retention must cover connector lag. Some databases retain log segments for a stalled replication slot, protecting recoverability while increasing source disk usage. Others may eventually remove history needed to resume. Monitor both oldest unconsumed position and retained log bytes; decide whether recovery resumes normally or requires a new consistent snapshot.

A common hybrid: put named events in an outbox table, then let CDC publish that table (Debezium's outbox event router). You keep one atomic commit and skip writing your own poll loop. Typical tools include Debezium on Postgres/MySQL, cloud equivalents such as AWS DMS, or a vendor CDC product.

Outbox and inbox playbook

When you own the schema and want explicit domain events, use a instead of (or beside) raw row CDC. In one local transaction you update business state and insert an outbox row with a stable event_id. Commit both or neither. Never call the broker inside that transaction.

Producer: outbox + relay

Begin transaction.
Update orders set status = 'placed'.
Insert into outbox (aggregate_id, event_type, payload, created_at).
Commit: both succeed or neither.
A (or CDC on the outbox table) publishes with that event_id as the idempotent key, then marks the row published. Advance the durable offset only after the broker acknowledges. Crash between ack and offset update means republish the same id.

Consumer: inbox

An inbox stores a uniqueness key for each event this consumer has applied, commonly scoped by consumer and event identifier. Insert that key and update local inventory in one database transaction. If the transaction rolls back, neither is durable and retry can safely try again; if it commits and acknowledgement is lost, redelivery finds the key and skips the already applied mutation.

An external email or charge cannot share this local transaction. Record a new outgoing intent and use the downstream service’s idempotency and reconciliation facilities. A second outbox makes the handoff durable but cannot alone guarantee that SMTP or an arbitrary remote API executes exactly once. Carry the original business identifier across those boundaries.

Numbers to know

Concern	Rule of thumb	Why
Outbox volume	2,000/s × 900 B × 86,400 ≈ 155 GB/day	Partition and purge only after broker retention and replay policy are met.
Inbox retention	≥ max(redelivery, replay), e.g. 7 days	A 1-day inbox with a 7-day replay will double-apply late messages.
Ordering	partition = order_id	Per-aggregate version N follows N−1. Global order is unnecessary.

Evolve schemas with : version the event envelope, keep consumers on N and N−1 during rollout, add optional fields first, and remove old fields only after every consumer upgraded.

When the relay loses the ack

Suppose event E42 is committed with order 123. The relay sends E42, the broker stores it, and the acknowledgement is lost. The relay cannot distinguish that result from a failed send, so it sends E42 again with the same identifier. A consumer that atomically recorded E42 with its first mutation now treats the second delivery as a duplicate. Generating a new event identifier on resend would defeat the inbox and turn one transition into two.

Outbox does not remove duplicates on the wire. It makes duplicates the same event, so the can ignore the second copy. That duplicate is expected at-least-once behavior if event_id is stable.

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

sequenceDiagram
    participant App as Order service
    participant DB as Orders + outbox
    participant Relay as Outbox relay
    participant Bus as Message bus
    participant Inv as Inventory + inbox

    App->>DB: BEGIN; order + outbox row
    App->>DB: COMMIT both
    Relay->>DB: read unpublished outbox
    Relay->>Bus: publish event_id=E1
    Note over Relay,Bus: Ack lost / relay crashes
    Relay->>Bus: republish event_id=E1
    Inv->>Inv: inbox insert E1 wins once
    Note over Inv: Duplicate delivery; one side effect

1  DB commits order + outbox at the same log position
2  Relay publishes; broker stores E1; ack is lost
3  Relay restarts; source offset not advanced → republishes E1
4  Consumer inbox unique(E1) wins once; reservation commits with inbox
5  Duplicate delivery is a successful no-op

Failure timeline, and what still breaks after the fix

Putting the pieces together, each boundary owns a different recovery. If the side effect cannot share the inbox transaction (email), record a second local outbox for that hop.

Failure matrix

Ordering needs an explicit scope. Partition events by aggregate identifier when updates to one order must be processed in sequence, and include an aggregate version so consumers can detect gaps or stale updates. One broker partition does not impose a global order across every order or transaction. Quarantining a poison event also has semantic cost: later events for that aggregate may need to wait until the missing predecessor is repaired.

Failure point	Symptom	Safety	Recovery
Before DB commit	No order or event	Single local transaction	Client retries with an idempotency key
After commit, before capture	Outbox lag grows	Durable WAL / outbox	Restart relay from last offset
After publish, before offset	Duplicate event	Stable event_id + inbox	Republish and deduplicate
Poison payload	Partition retry loop	Bounded retry + quarantine	Fix or upcast, then replay
After effect, before ack	Broker redelivery	Effect + inbox in one TX	Duplicate becomes a no-op

What to watch

Watch	Page when
Oldest unpublished outbox row / CDC source-offset lag	Lag threatens the business SLO
Publish duplicates, inbox conflicts, consumer lag	Connector offset cannot progress
Poison count, schema rejections, version-gap age	Quarantine grows, or one aggregate stays stuck

Still fails when…

A stuck relay grows unpublished age until search and inventory silently diverge.
CDC schema changes break deserializers mid-rollout before consumers finish expand-contract.
An inbox kept shorter than the replay window lets late redelivery double-charge.
Marking outbox published before broker ack recreates the dual-write gap.
Deleting outbox rows before confirmed capture removes evidence for reconciliation.
One poison payload stalls a partition unless you quarantine and upcast it.

Trade-offs

Outbox is explicit and operable; CDC cuts app code but couples to log retention.
Polling a relay is simple; log-based capture preserves commit order with less scanning.
Thin notification events force a source lookup; fat events complicate privacy deletion.
Ordering guarantees depend on partition key choice, not on global bus order.

How to explain this in an interview

Use this when you need to explain why the concept belongs in the design, not only how the mechanism works. Name the dual-write gap first, then pick outbox or CDC, then insist on inbox.

Core idea to name

Dual writes without outbox or CDC leave gaps between database and bus.
Outbox names events in one commit; CDC tails the log for wide table fan-out.
Consumers still need inbox or idempotency under at-least-once delivery.

When it belongs

A service must publish reliable events after its local commit succeeds.
Search, warehouse, or cache must track database changes across many tables.
You cannot tolerate lost notifications or duplicate side effects on retry.

How it fails

Relay stuck: outbox backlog grows; page on oldest unpublished row age.
CDC schema change breaks the deserializer before consumers finish upgrading.
Inbox missing means duplicate charges appear whenever the broker redelivers.

Trade-offs and traps

Outbox adds table and relay ops; CDC couples you to database log retention.
CDC sees every column change, so filter noise early in the pipeline.
Never publish before commit, or delete outbox rows before confirmed publish.
Never break the event schema without a version field and expand-contract.

Try it: reproduce the dual-write race, then fix it with outbox

Place an order, publish to the message bus, and crash between steps. Dual-write exposes order-without-event or event-without-order states. The transactional outbox commits both sides of the boundary in one database transaction; only the relay crosses to the bus.

Order database

No order row yet.

Message bus

No event on the bus.

Consistency

Both sides agree on whether the order happened.

Pattern: Dual write
In flight step: idle
Outbox backlog: 0
Mismatch: No

DB order: no · Bus event: no · Outbox pending: no. Consistent. Dual-write mode: database and bus are separate steps. Crash between them to see the gap.

Run the failure you fear. In dual-write mode, commit the database then crash before publish. The lab keeps the order row and leaves the bus empty.

CDC is the other half. When many tables fan out, tail the WAL instead of application publish calls. Consumers still need inbox dedupe under at-least-once delivery.

Wrap-up

You have seen the dual-write failure, the CDC and outbox mechanisms, and the operational signals. Use these exercises to check the ideas stuck, then follow the references for the primary sources.

Exercises

OrderPlaced → inventory + email. No distributed transactions. What is in the first local TX? What happens if the broker ack is lost? How long do inbox keys live? Where does email live if SMTP cannot share the inventory transaction?
Dual-write crash vs outbox crash. For each failure point (before commit, after commit before capture, after publish before offset, after effect before ack), name the symptom, the safety mechanism, and the recovery step.
Outbox or CDC for search. Forty tables feed Elasticsearch. Product wants domain events for five of them and raw row change for the rest. Propose a hybrid and list the ops alerts you would page on.

Topic-specific references

Outbox, CDC, and delivery-semantics sources: pattern docs and broker semantics, not generic textbooks.

Key takeaway

Capture changes from the same commit that made them durable: CDC from the database log, or an outbox row in the same transaction. Evolve event schemas with expand-contract. Make every consumer retry-safe with an inbox. Outbox plus inbox is the default answer when checkout spans an order database and a notification bus.

Practice and further reading

Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.

Focus for this lesson: Close database/broker dual writes with transactional outbox, ordered CDC offsets, idempotent inbox retention, poison handling, schema rules, and replay.

Explain the decision

Which transaction makes the outbox pattern safe?

Read the reasoning

The business mutation and outbox record commit atomically in the same source database transaction; relay publication may repeat.

Apply it to a production scenario

The relay publishes an event then crashes before marking it sent. What happens?

Read one approach

It republishes; consumers use stable event IDs and an inbox/business idempotency record to suppress duplicate effects.

Work through the numbers

At 15,000 events/s and seven-day inbox retention, how many dedup rows?

Read the worked calculation

About 9.07 billion before compaction/partitioning; choose retention from maximum replay horizon and storage budget.

Reason through a failure

What must CDC offset management preserve?

Read the recovery reasoning

A committed source-log position advances only with durable downstream publication/checkpoint state, so restart neither skips nor reorders beyond contract.

Sources and further study

Continue with Event Sourcing & Projection Rebuilds.