System Design Communication
Publish-Subscribe
One Instagram photo lands. Friends start seeing it in feeds, push notifications fire, search indexes update, and moderation scans the media, without the upload API calling every team one by one. One fact, many independent reactions.
is that pattern: a writes once to a , and each gets its own copy or cursor, lag, retries, and .
Post journey
Emit PostCreated once.
Durable stream of post facts.
Feed, notify, search, moderate.
One publish becomes many deliveries.
Try it: fan-out multiplies downstream work
A publisher writes once per fact, but each subscription maintains its own cursor. Every matching subscriber receives a copy, so total handler load is published events times subscriptions times filter hit rate. One slow subscriber does not block others, yet it still consumes storage, retries, and on-call attention while its lag grows.
- Delivered to each sub/s
- 1,000
- Total handler calls/s
- 3,000
- Max subscription lag
- 0
Feed projector
Private lag: 0 events
Push notifier
Private lag: 0 events
Search indexer
Private lag: 0 events
One publish can fan out to every subscription. Advance time to see independent lag per subscriber and total downstream load.
Isolate a slow subscriber. Keep publish at 1,000/s and drop the push notifier to 700/s. After twenty seconds its private lag grows by about 300/s while feed and search stay near zero lag at 950 and 1,000/s.
Filters change cost, not publish volume. Lower filter hit rate to 0.4 and watch delivered events fall even though the publisher rate is unchanged. That is why server-side filters need enough excluded traffic to justify the policy complexity.
Model scope: independent cursors, no shared DLQ back-pressure across subscriptions, and uniform event size.
Problem → naive → break
Concrete problem. One post must notify feed, push, search, and moderation. The upload API cannot call every team on the request path. That couples deploy schedules and multiplies timeouts.
Naive system. After saving the post, loop for each subscriber: http.post(...). Or use one shared work queue and hope every team somehow shares the same consumer.
Break. Search is down for 40 seconds. The loop retries, holds the upload thread, and the user sees a 504, even though the photo already landed. Or email is slow and the shared queue backs up so inventory never drains.
Counterexample to "the broker gives exactly-once." Retries, visibility timeouts, and rebalances redeliver. Delivery is usually ; safe effects need . Do not treat a vendor checkbox as end-to-end exactly-once.
Correctness invariant
What must never become false
For the durable at-least-once design on this page, an accepted event remains available to each matching subscription within the configured retention and recovery contract. Each subscription advances independently, so one slow subscriber does not prevent another from progressing. Ordering is scoped to the declared partition or ordering key. Ephemeral pub/sub systems have a weaker contract and may drop events for disconnected subscribers.
What actually happens
Queues load-balance work: one winner per message. Pub/sub duplicates logical delivery across subscriptions: each or subscription advances its own cursor. The upload API stays thin; inventory-style side effects become independent consumers that can lag without blocking the publish.
cost dominates social timelines and observability pipelines at scale. A single publish can become many deliveries; celebrity posts can explode further with into follower mailboxes. Combine write and read strategies thoughtfully rather than assuming one pattern everywhere.
Plan durable subscriptions, , schema discipline, DLQs, and lag per consumer group before copying feed architectures from blog posts. Brokers usually offer ; handlers need . Casual "exactly once" usually means exactly-once effects, not once-on-the-wire delivery.
Publisher
Topic: order-events
Subscription: inventory
Inventory workers
Cursor and DLQ
Subscription: billing
Billing workers
Cursor and DLQ
Subscription: search
Indexing workers
Cursor and DLQ
Publisher: 1k events/sec
Topic
Search subscription: healthy
Lag stable
Billing subscription: healthy
Lag stable
Email subscription: slow
Consumes 700/sec
Lag grows 300/sec
Retry, DLQ, replay plan
What to watch
One slow subscriber should not block another; monitor each cursor independently.
A topic can publish successfully while one subscription repeatedly fails and fills its DLQ.
Server-side filters save work only when they drop enough irrelevant traffic to justify the extra policy surface.
How to choose queue vs pub/sub
1. One worker or many reactions? Use a queue when one worker should claim each job. Use pub/sub when several independent consumers need the same fact.
2. Is replay a product requirement? Logs and durable subscriptions make common; classic queues often delete after ack.
3. Have you counted delivery amplification? Size subscribers and DLQs separately instead of quoting only topic ingress.
Simple rule of thumb
Publish immutable integration events with versioned schemas; attach durable queues per downstream when you need retries and DLQs. Design around real consistency boundaries, filter early, and monitor subscription lag like an SLI.
Mega fan-out paths almost always need hybrid write/read strategies, not naive full materialization.
Pub/sub vs queue
Pub/sub is a delivery topology, not a durability guarantee. An ephemeral subscription may receive only events published while it is connected. A durable subscription retains a delivery obligation or log position across disconnects, within configured retention. Several workers inside one subscription can compete to process its deliveries, while a separate subscription receives its own copy. Name the subscription boundary when describing “every consumer gets the event”.
Point-to-point queue
- Exactly one competing consumer typically wins each message
- Great for job runners and background tasks
Pub/sub
- Each subscription receives (or can seek through) the stream
- Great for domain events and telemetry multiplexing
| Criterion | Queue | Pub/sub |
|---|---|---|
| Recipients per message | One worker pattern | Many subscriptions |
| Replay | Usually none after ack | Common with logs |
| New consumer | Steals capacity | Catches up from cursor |
Capacity math: fan-out topics
If a publisher emits 2,000 events per second and five subscriptions each match every event, downstream processing starts at 10,000 deliveries per second before retries. Adding a sixth subscriber increases total work even though producer traffic is unchanged. With a shared retained log, payload storage need not be copied six times, but broker egress and consumer processing still increase. Filters change delivery volume, so estimate each subscriber separately.
Pub/sub capacity is not just publish rate. Multiply by subscriptions, filter hit rate, and feed fan-out amplification to expose the real downstream work.
1M users create feed events
Fan-out strategy
Write fan-out
Write one mailbox item per follower
Queue partitions sized by amplified writes/sec
Read fan-out
Store once, merge at read time
Read caches sized by timeline reads/sec
Track lag, DLQ, and replay window
| Metric | How to estimate | Million-user example |
|---|---|---|
| Published events/sec | users × events/user/day ÷ 86,400 × peak factor | 1M users × 5 feed events/day × 8× peak ≈ 463 events/sec |
| Delivered events/sec | published events/sec × matching subscriptions × average filter hit rate | 463/s × 4 subscribers × 60% hit rate ≈ 1.1k deliveries/sec |
| Feed write amplification | posts/sec × average follower recipients | 200 posts/sec × 150 recipients ≈ 30k mailbox writes/sec |
| Slow subscriber lag | subscription input rate - consumer drain rate | 1k/s in, 700/s out grows by 300 messages/sec |
| Retention budget | publish rate × payload size × retention × replicas | 500/s × 2 KB × 3 days × 3 ≈ 778 GB raw |
Delivery amplification worksheet
Start with published events/sec, then list matching subscriptions, filter hit rate per subscription, delivered events/sec per subscriber, and total downstream handler calls/sec. One new subscriber can increase system work even when publisher throughput does not change.
Worked example: lag age vs retention
Email subscription takes 700/s against 1,000/s ingress → lag grows 300 msg/s. After 2 hours: 300 × 7,200 = 2.16M messages behind. At 2 KB each that is ~4.3 GB of unread backlog for one subscription alone. If topic retention is 24 hours and lag age hits 20 hours, a further outage risks permanent loss: page on oldest unacked age, not only depth.
How to choose
Queue vs pub/sub answer pattern
- Queue: Use when one worker should claim each job, capacity is pooled, and replay after ack is not a core requirement.
- Pub/sub: Use when several independent consumers need the same fact, with their own lag, retries, and replay windows.
- Capacity caveat: A single publish can become many deliveries, so size subscribers and DLQs separately instead of quoting only topic ingress.
Fan-out on write vs read
Push fan-out materializes feeds eagerly (writes explode for mega-followed accounts). Pull fan-out merges at read time (cheap posts, heavier reads). Hybrids are normal.
Insert into each follower's mailbox when the author posts: reads stay cheap; viral accounts need special casing or rate limits.
Store posts once; timeline queries merge recent activity from followed ids: writes cheap, reads heavier and cache-sensitive.
Topics, partitions, ordering
Ordering is normally scoped to a partition or ordering key, such as order_id. All events that must be interpreted sequentially should share that key, but one hot key then limits parallel processing. Independent subscriptions can be at different positions without violating the ordering inside each one. Publishing globally ordered events requires a stronger coordination point and can constrain throughput.
Ordering travels with the or ; subscribers in one group split partitions for scale while independent groups each observe the whole stream at their pace. After fan-out, ordering is scoped separately per subscription, so a slow subscriber can be behind while another subscriber sees fresh events.
| Concept | What happens |
|---|---|
| Partitions | A topic splits into P ordered shards; scale by adding partitions and consumers. |
| One consumer group | Members divide partitions: each partition consumed by one active consumer for load balancing. |
| Independent groups | Each group keeps its own cursor; same stream can feed analytics, inventory, and billing at different speeds. |
| Ordering key | Route events with the same entity id to one partition so related updates stay contiguous. |
Durable fan-out (topic → queues)
Bare topics may drop subscribers that are offline; pairing each subscriber with its own durable queue preserves retries, DLQs, and independent pacing.
- inventory-qpull worker, retries, DLQ
- analytics-qbatch-friendly consumer
- notifications-qpush adapters / third-party APIs
Filtering and schemas
An integration event describes a durable fact, such as “order accepted,” with a stable identifier, event time, schema version, and the data consumers need. Avoid making every consumer call back immediately for missing fields, which recreates coupling through synchronous requests. Evolve schemas so older consumers can continue interpreting existing fields, and distinguish optional additions from changes in meaning. Filtering must be compatible with schema evolution so new versions do not silently disappear from subscriptions.
Server-side filters
Drop irrelevant events before they cross regions or hit fragile handlers: subscription policies differ by platform but the goal is the same.
# subscription filter sketch (pseudo-YAML)
conditions:
- region IN ["us-east", "eu-west"]
- order_value >= 1000Central compatibility checks stop producers from shipping fields that break downstream decoders; pairs naturally with schema-based encodings.
How to choose
Schema evolution is the hidden pub/sub interview trap.
- Backward compatibility: New producers should not break old consumers; prefer additive optional fields and stable enum handling.
- Forward compatibility: New consumers may read old events during replay, so defaults and upcasters matter.
- Operational answer: Mention schema registry or CI contract tests, owner metadata, and a deprecation window for removing fields.
DLQ, replay, flow control
Dead-letter paths
Failed deliveries after retry budgets route aside for inspection; alert on depth spikes.
Logs let you rewind consumers for backfills: bounded by retention policy and storage cost.
Flow control
Cap outstanding pulls or bytes per subscriber so slow handlers cannot exhaust memory.
Replay safety checklist
- Consumer can read old schema versions and unknown enum values.
- Writes use natural keys, upserts, or processed-event ids for .
- Replay range is bounded by stream retention and product risk.
- User-visible notifications and external calls are suppressed, deduplicated, or dry-run.
- Projection rebuilds can run in dry-run mode before switching traffic.
Subscription lifecycle
Creating a subscription requires choosing its starting position: only new events, a retained historical point, or a full backfill followed by live updates. Deleting or recreating it can discard progress depending on the broker. Record ownership, retention, failure handling, and replay permission as part of the subscription contract. A new subscriber should catch up within a bounded resource budget so its historical replay does not disrupt live consumers.
- Create subscription with owner, filter, retention, and DLQ policy.
- Bootstrap from earliest retained point or latest point depending on product need.
- Catch up while watching lag age and downstream write rate.
- Switch to live processing, then alert on retry exhaustion and oldest unprocessed event age.
Managed pub/sub (illustrative)
Fully managed buses scale out throughput and retention knobs for you, still design idempotent handlers and monitor end-to-end lag per subscription.
- Publish to a regional topic; subscribers are pull workers or HTTP push endpoints depending on product.
- Use ordering keys where the vendor supports them to serialize delivery per business entity.
- Tune flow control so slow handlers cap buffered messages or bytes on the client side.
Managed buses still need idempotent handlers and lag dashboards: "serverless" does not remove back-pressure design.
Many consumer groups on one log
A single high-volume topic often feeds search, billing, fraud, and warehouse loaders independently: adding a group replays or tails without rewiring producers.
- High-churn activity topics often fan out to many downstream consumer groups.
- Geo replication and failover use vendor-specific mirroring: plan offset semantics and ordering guarantees explicitly.
How to choose
Consumer lag answer pattern
- Define it: Lag is log-end offset minus committed offset, usually tracked per partition and consumer group.
- Diagnose it: Separate slow processing, failed commits, rebalance churn, skewed keys, and insufficient partitions.
- Mitigate it: Scale consumers up to partition count, tune batches, split hot keys, or shed low-value consumers before retention catches up.
Failure timeline: rebalance mid-ack
T0 Consumer A processes PostCreated#42; side effect starts (push send) T1 Side effect succeeds; process dies before offset commit / ack T2 Rebalance assigns partition to Consumer B T3 B redelivers PostCreated#42 (at-least-once delivery) T4 Without dedupe key → second push; with processing idempotency → no-op
This is expected broker behavior under retries and rebalances, not a rare bug. Design handlers for T4 before promising notification SLOs.
Break the improved design
Independent subscriptions fix upload coupling, then fail differently: celebrity saturates mailbox writers while the topic looks healthy; a poison payload fills one while search stays fresh; hot pin one partition and create lag islands.
Schema breaks during when old retained events lack fields new code requires. Multi-layer retries (client + broker + handler) amplify load into dependency outages: budgets, backoff, and jitter still apply to consumers.
Trade-offs
Queue vs pub/sub
Queues pool capacity for one winner; pub/sub pays delivery amplification for independent reactions and replay.
Write vs read fan-out
Write fan-out makes reads cheap until celebrity posts explode writes; read fan-out flips the cost to merge latency.
More partitions
Raises parallelism and weakens per-key ordering scope; never assume total order across the topic.
Going deeper
Production deep dive: durable fan-out without surprises
A production pub/sub design is a retained log plus independently operated subscriptions. Delivery is not finished when the broker accepts a publish; it is finished when every required subscriber has either committed its cursor or entered an owned recovery path.
What must stay true
Deleting retained data is safe only when every required subscription has advanced beyond it or has an independently verified snapshot from which it can resume.
How the pieces fit
Publisher
immutable event + key
Partitioned log
retention + ordering
Subscription cursor
independent progress
Idempotent handler
commit after effect
DLQ / replay
owned recovery
Ingress
50 MB/s
25,000 events/s × 2 KB before replication.
Logical fan-out
400 MB/s
Eight subscriptions each receive the full stream.
24 h retained
≈ 13 TB
50 MB/s × 86,400 s × three replicas, before overhead.
Retention and replay
Time/size retention belongs to the topic, while progress belongs to each subscription. A replay creates duplicate deliveries by design: pause the consumer, choose an offset or timestamp, reset only its cursor, then rate-limit the rebuild so it cannot starve live traffic.
Slow-subscriber isolation
Bound in-flight messages and bytes per subscriber. Alert on oldest-unprocessed age, not only message count; one giant message can be more dangerous than many tiny ones. A slow email consumer must never hold back billing or search cursors.
Ordering scope
Ordering is normally guaranteed only inside one partition or ordering key. Retrying a failed item can reorder later items unless the consumer blocks that key, buffers successors, or writes with a monotonic entity version.
Subscription lifecycle
Provision an owner, filter, schema compatibility mode, retention requirement, lag SLO, DLQ, and access policy. New subscriptions backfill behind a quota; retired subscriptions are first disabled and observed, then their cursor and DLQ are deleted after an agreed recovery window.
Exactly-once boundary
Broker transactions may make consume-transform-publish exactly-once inside that broker. They cannot atomically cover an external email, payment API, or unrelated database. Those side effects still need an idempotency key, dedupe record, or transactional inbox/outbox.
Worked example: Replay economics
- 1A subscriber is six hours behind at 25,000 events/s: backlog = 540 million events.
- 2At a safe catch-up rate of 75,000 events/s while 25,000 new events/s continue, net drain = 50,000 events/s.
- 3Catch-up time = 540M ÷ 50k = 10,800 seconds, or three hours.
A six-hour incident needs at least nine hours of usable retention from the oldest failed event, plus margin for a slower-than-planned rebuild.
Failure and recovery matrix
| Scenario | Consequence | How to contain it |
|---|---|---|
| Consumer commits before side effect | Crash loses the business action. | Commit after an idempotent effect or use an inbox transaction. |
| Poison event retries forever | Partition head blocks and lag grows. | Bound attempts, quarantine to DLQ, and preserve ordering policy per key. |
| Retention expires first | Cursor points to deleted history. | Page before the retention wall; restore from snapshot or source-of-truth backfill. |
What to watch in production
- Dashboard publish failures, oldest-event age, per-partition skew, redeliveries, DLQ age, and retention headroom.
- Test a single-subscription rewind in staging and document who authorizes production replay.
- Quota replay traffic separately from live consumption and verify downstream write capacity first.
A scenario to reason through
Search has bad projection logic for six hours, while billing is healthy. How do you repair only search without duplicating invoices or dropping live updates?
One solid approach
Deploy an idempotent fixed projector, create a shadow index, reset only the search subscription to the first bad offset, throttle replay, compare counts/checksums, then atomically switch the read alias. Billing keeps its untouched cursor.
Delivery vs processing (no casual "exactly once")
Independent delivery does not imply independent business correctness. An email subscriber and an inventory subscriber may both receive “order accepted,” but their effects can complete at different times. Keep each effect idempotent, expose workflow status when users need it, and reconcile missed or failed reactions. A publisher should only announce a committed fact, using an outbox or equivalent atomic publication mechanism when its database and broker are separate systems.
Separate two claims. is what the broker promises after retries and rebalances: often more than once. is what your handler promises so duplicate deliveries do not create duplicate side effects. Saying the pipeline is "exactly once" without that split usually hides a bug.
Delivery
Publish ack, consumer redelivery, visibility timeout, rebalance: the same offset or message id can appear again.
Processing
Dedupe keys, upserts, or transactional outbox so effects are applied once even when delivery is not.
What to remember
Publish immutable integration events with versioned schemas; attach durable queues per downstream when you need retries and DLQs. Design ordering keys around real consistency boundaries, filter early, and monitor subscription lag like an SLI. Mega fan-out paths almost always need hybrid write/read strategies, not naive full materialization. Treat at-least-once delivery plus idempotent handlers as the honest default, not casual "exactly once."
Topic-specific references
- Apache Kafka: Message Delivery Semantics : at-most-once, at-least-once, and what "exactly-once" means in Kafka's transactional model.
- Google Cloud Pub/Sub subscribers : at-least-once delivery and ack deadlines in a managed bus.
- Transactional Outbox pattern : publishing reliably without dual-write gaps; still needs idempotent consumers.
Exercises
1. Replay without disturbing money paths
Search had a bug for six hours. Rewind only that consumer group, rebuild safely, and keep billing/inventory on current offsets. What retention, idempotency, and dry-run checks do you require before cutover?
2. Celebrity write fan-out budget
200 posts/sec average, but one account has 40M followers. Full write fan-out is impossible. Propose a hybrid write/read strategy and name the SLIs you page on during a viral spike.
3. Exactly-once claim in review
A design doc says "Kafka exactly-once so push notifications are unique." Write the precise delivery vs processing rebuttal and the minimal handler change you would require before merge.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Design fan-out around retention, replay, slow subscribers, ordering scope, subscription lifecycle, filtering, and delivery economics.
Explain the decision
How does pub-sub differ from a work queue?
Read the reasoning
Each subscription receives its own logical copy, whereas competing queue consumers share work; retention and replay policies are per contract.
Apply it to a production scenario
A subscriber falls six hours behind and retention is four hours. What happens?
Read one approach
Expose lag against retention, pause unsafe truncation or rebuild from an archive/source of truth, and make the gap visible rather than silently resuming.
Work through the numbers
Publishing 10,000 1 KiB events/s to 40 independent subscribers creates what logical fan-out?
Read the worked calculation
400,000 deliveries/s and about 390.6 MiB/s payload before filtering, batching, compression, and broker replication.
Reason through a failure
What ordering promise should be documented?
Read the recovery reasoning
Ordering is scoped to a key/partition, and subscriber concurrency, retries, and repartitioning must not imply a stronger global order.