Case study
Design a social microblog feed
You open X or Threads on the train, pull to refresh, and your fills with short posts from people you follow. Saving one post is straightforward. Serving a fast, correct feed for millions of readers without making the author wait is the real design problem.
This case study matches products like Twitter, X, and Threads: short posts, a follow graph, likes, reposts, search, and notifications. The hard part is into timelines, especially when a posts.
What happens on a post?
Allocate the id, store the post, and return 201 so the author is not blocked.
Async event
Publish post-created so workers can fill follower timelines without holding the request.
Push for normal authors; skip full push for accounts with huge follower counts.
Home feed reads ids from cache, then batch-loads posts and users for the page.
Try it: fan-out vs pull on a social feed
Choose push, pull, or hybrid fan-out. Celebrities force hybrid or pull-heavy paths because push writes scale with follower count.
- Author
- Post service
- Fan-out
- Timeline store
- Reader
- Followers
- 500
- Fan-out writes
- 0
- Timeline reads
- 0
- Strategy
- hybrid
Write amplification
fan-outTotal fan-out writes: 0.
Read path
latencyPull merges increase read cost but cap write spikes for celebrities.
Ordering
consistencyPer-user timelines are eventually consistent; ranker may briefly reorder posts.
Post from a normal user vs a celebrity. Push fan-out explodes for huge follower counts.
Related building blocks
Fan-out, caching, and queues show up in almost every feed design.
What actually happens
Those related notes are the building blocks. Here is the end-to-end story for post and home feed.
On write, the post service allocates a , persists the post, and returns quickly. Kafka (or a similar log) carries post-created so workers can update follower without holding the author's request open.
On read, the timeline service loads candidate ids, merges followees for hybrid authors, then does : batch-fetching posts and users, filtering blocks and deletes, before returning a page. Reads dominate writes, so cache and batching shape the design.
How to choose fan-out
Once write and read are clear, three product choices shape the design.
1. How many followers does a typical author have? Push into the works when amplification stays manageable.
2. What about celebrities? Skip full push and merge on read with so one viral account cannot melt your write path.
3. Must deletes and blocks apply immediately? Treat timeline caches as accelerators; recheck visibility on against authoritative state.
Simple rule of thumb
Persist first, fan out asynchronously, read from a precomputed timeline, hydrate in batches. Use as soon as follower counts make full push unsafe. Do not make the author wait for every follower's cache update. Exact rates live in Numbers to know.
With those choices set, size the system for millions of daily active users and billions of posts. Exact capacity math lives in Numbers to know; the diagram below is the map, then architecture layers unpack each box.
High-level design diagram
Entry → write path → fan-out → read path → search → storage → ops. Match colors to write (orange) and read (blue) in the drawing.
Treat this drawing as the map for everything below. The legend shows sync versus async edges; highlights call out the decisions the boxes encode.
Client waits for API response
Kafka / background workers
Create post, media, publish
Home timeline fetch + hydrate
Fan-out, storage, replication
Design highlights (from the diagram)
Kafka carries fan-out off the HTTP thread so “posted” means durable, not fully distributed.
Load timeline IDs, batch-hydrate details, then filter, never store full cards in every timeline.
Push for normal authors; merge celebrity posts on read so write storms stay bounded.
Redis for live timelines, Cassandra for durable feeds, object storage for archives.
CDN serves images and video; multi-region keeps the app available when one region falters.
Notifications and search must not own the feed p99: run them asynchronously.
Walk the same left-to-right order as the HLD so each layer below maps to a box you already saw in the drawing.
Architecture layer by layer
Same order as the HLD: entry, gateway, write, fan-out, read, search, storage, ops.
Walk the stack the same way a request does: edge and auth first, then the write path that makes a post durable, then the background that fills timelines, and finally the read path that a home feed. Each layer explains the state it owns, the work it performs, and how failures affect the next step.
01Clients, DNS, and CDNGet users to the nearest edge with static assets already warm.
Get users to the nearest edge with static assets already warm.
Think of it like: Like opening the local edition of a newspaper instead of flying to the printing plant.
Web and mobile clients, plus authorized third-party API consumers, reach a healthy serving location through DNS and edge routing. Geographic proximity is one input, alongside capacity and routing policy; the closest site is not always the best destination. The CDN serves reusable JavaScript bundles, avatars, images, and video so those bytes do not pass through the feed API on every read.
TLS protects the connection up to its termination point, and internal hops need their own protection when traffic is decrypted at the edge. Static media caches can be widely shared only when visibility and cache-key rules permit it. Personalized feed responses need a separate cache scope so one user never receives another user’s private timeline.
02API gateway and authOne front door for security, limits, and routing.
One front door for security, limits, and routing.
Think of it like: A bouncer checks your ID, stops spam, then points you to the right room.
The gateway validates credentials such as a session cookie or an access token, applies per-user and per-IP admission rules, and routes requests to Post, Timeline, User, Search, or Notification services. An identity provider may use OAuth and OIDC during login, while the authentication service manages device sessions, refresh-token rotation, and account recovery.
Authentication establishes who is calling; resource authorization decides whether that caller may view or mutate this post or relationship. Keep those checks at the owning service as well as relevant edge controls, including tenant or audience boundaries. Rate limits should also account for expensive operations and shared-IP users, rather than treating every request or IP address as equivalent.
03Write path (post creation)The user should see “posted” quickly; fan-out happens in the background.
The user should see “posted” quickly; fan-out happens in the background.
Think of it like: You drop a letter in the mailbox; the post office sorts it to every subscriber later.
The post service validates text, mentions, links, visibility, and applicable abuse rules. Media is uploaded separately to object storage and referenced by a media ID after the required processing state is reached. The content service assigns a stable post ID, for example , so the post can be addressed consistently by every downstream consumer.
Commit the post and a publishable event record in one durability boundary, then let an outbox relay deliver post-created to Kafka. Return success once that accepted state is durable under the promised failure model. A stable client operation key makes a retried create return the same post; the author does not wait while workers copy it into thousands of follower timelines.
Simple rule of thumb: never fan out to millions of timelines inside the HTTP request. Return 201 after the post and its outgoing event are recoverably committed; let a relay and workers do the heavy lifting. A separate unprotected database write followed by publish still has a crash gap.
04Fan-out and feed generationCopy post IDs into follower timelines, or skip for celebrities.
Copy post IDs into follower timelines, or skip for celebrities.
Think of it like: A newspaper printing one extra line on 500 subscriber copies, unless the author has 20M subscribers. Then readers pick up that edition at the newsstand.
A reads post-created and retrieves eligible follower IDs from the social graph, such as a followee-indexed Cassandra table or graph store. For authors whose amplification is affordable, it inserts post_id into each follower’s in bounded batches. Storing IDs avoids copying full text and profile data repeatedly.
For , the worker can skip full copying and arrange instead. Choose that threshold from posting rate, active follower count, read traffic, and worker capacity; one million followers is only an illustrative value. Interaction workers update likes and reposts separately, with duplicate-safe state transitions so event redelivery does not inflate counters.
Timelines often use Redis (score = time, member = post_id). Reads use ZREVRANGE; workers batch writes in a pipeline.
05Read path (home timeline)Fetch IDs fast, hydrate details in batches, filter, then return JSON.
Fetch IDs fast, hydrate details in batches, filter, then return JSON.
Think of it like: You get a table of contents first, then the library fetches the actual book pages in one trip.
The timeline service starts with a bounded candidate list and . It merges recent posts from authors skipped during , then deduplicates overlapping IDs. A ranked feed needs a defined cursor contract, such as a short-lived candidate snapshot or ranking version, because changing scores can otherwise move items across page boundaries.
To the candidates, batch-fetch post bodies, author records, and media metadata from caches with bounded database fallback. Apply current deletion, block, mute, private-audience, and content-policy checks before returning anything. Over-fetch within a limit to replace filtered entries, but never relax access checks just to fill a page when a policy dependency is unavailable.
06Search and discoveryFull-text search and trending stay off the hot feed path.
Full-text search and trending stay off the hot feed path.
Think of it like: The library catalog desk is separate from the “today’s headlines” rack.
A separate indexing consumer reads post events and builds full-text search over posts, hashtags, and user discovery fields. This makes search query patterns independent of the home timeline’s storage layout. Indexing may lag behind posting under an explicit freshness target, while deletion and visibility need serving-time protection and eventual cleanup of derived documents.
Trending combines observations within a chosen time window. Exact counters or a Count-Min Sketch estimate how often a hashtag appeared; HyperLogLog estimates how many distinct users or other unique values contributed. Those answer different questions and are not interchangeable. Recommendations can then use these aggregates with relationship and relevance signals without adding a warehouse query to every feed open.
07Storage and supporting servicesDifferent stores for timelines, posts, graph, analytics, and cold archives.
Different stores for timelines, posts, graph, analytics, and cold archives.
Think of it like: Sticky notes for hot feeds, filing cabinets for posts, warehouse for old analytics.
Partition durable timeline candidates by reader and posts by an appropriate author or post key, with bounded buckets for long histories. Keep the follower graph indexed by followee for fan-out and, where needed, by follower for read-time queries. These indexes encode different access paths; one giant graph query is not required for each ordinary feed request.
Redis can cache timeline sorted sets, hot post objects, and derived like counts, while Cassandra or Scylla may hold durable history and relationships. Send analytical events to ClickHouse or a warehouse and retain cold archives in object storage according to policy. User, notification, and backup services support the core paths, but cache or analytics state must not become the only recoverable record of an accepted post.
08Infrastructure and observabilityYou cannot debug a stale feed without fan-out lag metrics.
You cannot debug a stale feed without fan-out lag metrics.
Think of it like: Air traffic control watching every runway, not just the ticket counter.
A scheduler such as Kubernetes runs service replicas; a service mesh is an optional way to apply transport identity, traffic policy, and bounded retries. Deploy with canaries or blue/green routing, and manage secrets through a controlled service such as Vault. These tools support operations but do not by themselves make a post transaction, permission check, or replay correct.
Measure feed p99, cache hit rate, and alongside business freshness and failed delivery counts. Logs and traces should connect the original gateway request to the durable post and later worker batches. Test cache loss, hot-author spikes, and rolling upgrades under load; otherwise a healthy create-post latency graph can hide followers waiting minutes for new content.
Interviewers expect you to split post traffic from feed traffic. Capacity figures live in Numbers to know below; here focus on durability, caches, and what stays off the author's hot path.
Write path vs read path
Interviewers want you to separate these. Different traffic shape, different caches, different failure modes.
Write path (post)
The user sees success when the post is durable. , search indexing, and push notifications happen after that response. Commit a stable post ID and an event in the same local durability boundary, then let workers distribute that ID. If the acknowledgement is lost, a client operation key prevents a second post on retry. Worker retries also need duplicate-safe timeline insertion. A durable authoring response does not mean every follower, search index, or push provider has already seen the post.
Read the diagram definition
sequenceDiagram
autonumber
participant U as User app
participant GW as API gateway
participant P as Post service
participant M as Media service
participant C as Content service
participant DB as Post store
participant K as Kafka
participant F as Fan-out worker
participant T as Timeline cache
U->>GW: POST /posts (JWT)
GW->>P: create post
opt has media
P->>M: upload / attach media_ids
M-->>P: S3 URLs
end
P->>C: allocate post_id (Snowflake)
P->>DB: INSERT post (durable)
P-->>K: post-created (async)
P-->>U: 201 Created
K->>F: consume event
alt normal author
F->>T: ZADD post_id to follower timelines
else celebrity author
F->>F: skip full fan-out
end- Idempotent create with a client request id so network retries do not duplicate posts.
- Upload media via a presigned S3 URL in parallel so text create is not blocked.
- Allocate a before persist so ids sort by time without a central sequence.
- Check status before workers run so the Post HTTP handler stays fast.
Read path (home feed)
The stores post IDs only in the ; loads author, text, and media in one or two batch calls. Storing IDs avoids copying the full post and author profile into thousands of timelines. Hydration fetches those objects in batches, filters unavailable or unauthorized candidates, and fills enough entries to make a page. Bound batch size and missing-object work so one stale timeline cannot generate an unbounded series of lookups. Cache objects by version where updates or moderation can change them.
Read the diagram definition
sequenceDiagram
autonumber
participant U as User app
participant GW as API gateway
participant TL as Timeline service
participant R as Redis timeline
participant H as Hydration cache
participant DB as Cassandra / SQL
U->>GW: GET /timeline/home?cursor=
GW->>TL: fetch feed
TL->>R: ZREVRANGE timeline:user_id
alt cache hit
R-->>TL: post_ids[]
else miss
TL->>DB: load timeline row
DB-->>TL: post_ids[]
TL->>R: warm cache
end
TL->>TL: merge celebrity followees
TL->>H: MGET posts + users (batch)
H-->>TL: hydrated objects
TL->>TL: filter blocks / deleted
TL-->>U: JSON feed page- Paginate with a (timestamp, post_id) cursor so deep scrolls stay cheap, not OFFSET.
- Fall back to chronological order if ranking is down so feeds still load for users.
- Allow eventual consistency on like counts, but never serve a deleted post as live.
End-to-end feed pipeline
One story from write through read: post → Kafka → → merge → batch → rank. Normal authors use push ; large accounts use instead of full push.
Read the diagram definition
sequenceDiagram
autonumber
participant U as User app
participant P as Post service
participant K as Kafka
participant F as Fan-out worker
participant T as Timeline cache
participant TL as Timeline service
participant H as Hydration cache
participant R as Ranking service
U->>P: POST /posts
P->>K: post-created (async)
P-->>U: 201 Created
K->>F: consume event
alt normal author
F->>T: ZADD post_id to follower timelines
else celebrity author
F->>F: skip push fan-out
end
U->>TL: GET /timeline/home
TL->>T: ZREVRANGE timeline:user_id
T-->>TL: post_ids[]
TL->>TL: merge celebrity followees
TL->>H: MGET posts + users (batch)
H-->>TL: hydrated objects
TL->>R: rank candidates
R-->>TL: scored feed
TL-->>U: JSON feed pageFan-out on write pays once per follower when an author posts, making later home-feed reads cheap. Fan-out on read stores the author’s posts and merges them when followers request a feed, saving writes for followers who never return. A hybrid pushes ordinary authors while pulling high-amplification authors. Choose the threshold from posting rate, active follower fraction, feed-read rate, and acceptable freshness, not follower count alone.
Fan-out: the interview centerpiece
Hybrid push + pull is how real timelines survive celebrities.
How to choose a fan-out strategy
Start from follower count and read latency. Push when the write cost is small; pull when a single author would overwhelm workers; combine both at production scale.
Fan-out on write
Copy the post ID into each follower timeline as soon as it is created. Precomputed candidate IDs make the first feed lookup cheap; hydration, visibility checks, and ranking still add work.
Each post can require one insert per eligible follower, including people who never return to read it. Batch and checkpoint the work so one author cannot occupy every worker. Use when posting rate × eligible followers fits the worker and storage budget and makes enough future reads cheaper.
Fan-out on read
Store the post once, then merge followee posts when the feed opens. Storing an author’s post once avoids immediate copies into inactive readers’ timelines and makes the write cost less sensitive to follower count.
A naive read must fetch and merge many followee streams. Caches, candidate indexes, and bounded retrieval reduce that fan-out, but the read still needs a measured latency and capacity budget. Use for manageable follow graphs, infrequent readers, or indexed candidate retrieval whose measured read cost fits the freshness goal.
Hybrid (production)
Push for normal authors; merge celebrities on read. Affordable authors use precomputed candidates, while expensive authors use bounded read-time retrieval. This lets the design spend work where followers are likely to consume it.
Both paths need deduplication and the same visibility rules. Changing an author’s strategy requires a transition plan so old copied entries and newly pulled posts do not create gaps or repeats. Use when the audience and activity distribution make neither universal push nor universal pull economical.
Fan-out workers consume post-created and use to insert each post_id with score = timestamp. Reads use ZREVRANGE (newest first), then and optionally merge celebrity posts.
# Hybrid fan-out (simplified)
def on_post_created(post):
author = get_author(post.author_id)
if author.follower_count > CELEBRITY_THRESHOLD:
mark_for_read_time_merge(post.author_id)
return
for batch in chunks(get_followers(post.author_id), 1000):
pipe = redis.pipeline()
for follower_id in batch:
pipe.zadd(f"timeline:{follower_id}", {post.id: post.ts})
pipe.execute()
def get_home_feed(user_id):
ids = redis.zrevrange(f"timeline:{user_id}", 0, 799)
ids += fetch_recent_posts_from_celebrity_follows(user_id)
visible_ids = filter_visible(user_id, dedupe(ids))
return rank_and_hydrate(visible_ids)Before naming Redis or Cassandra, tell the user story: post, open the home feed, then engage.
What users experience
Tell this story before naming Redis or Cassandra.
Post a thought
You: You type a short post (maybe add a photo) and tap Post.
System: Validate, upload media if needed, allocate a Snowflake ID for time-ordered uniqueness, persist the post, and return 201. Kafka carries post-created so fan-out workers can fill follower timeline caches without making the author wait.
Open home feed
You: You pull to refresh and see posts from people you follow.
System: Load home timeline post ids from the timeline cache, merge celebrity posts on hybrid fan-out so huge accounts do not melt writes, batch-load text and avatars, then filter mutes before the page returns.
Like or search
You: You heart a post or search a hashtag.
System: Likes and notifications update asynchronously so the feed stays fast. Search hits Elasticsearch and stays off the home timeline hot path.
Lock scope with the interviewer before capacity math. Clarify home feed versus profile versus search; rates and fan-out amplification live in Numbers to know.
Requirements
Clarify home feed vs user profile vs search before drawing boxes.
Functional
- Let users post short text with optional media so the product matches a microblog.
- Support follow and unfollow so each home timeline has a clear follow graph.
- Serve a personalized and a per-user profile timeline for browsing.
- Support like, repost, and reply so engagement stays first-class beside posting.
- Search posts, hashtags, and users so discovery is not limited to follows.
- Deliver push and in-app notifications so users learn about relevant social activity.
Non-functional
- Keep home feed latency low enough that pull-to-refresh feels instant (see Numbers to know).
- Target very high availability on the read path because a blank feed looks like total outage.
- Accept eventual consistency on counts and search so the write path stays fast for authors.
- Survive accounts without melting push into every follower cache.
Numbers to know
Requirements set the latency bar; these rates are the traffic shape you design against.
Each write can amplify into many timelines
Read bias forces cache-first feed design
Write amplification is the true bottleneck
Home feed must feel instantly responsive
Plan from 200M DAU, about 5 posts per user per day, and about 20 feed loads per user per day. That yields the card rates above (11.6K writes/s average, 46K feed reads/s average, and about 5.8M fan-out inserts/s at ~500 followers per post). Hold feed p99 under 200ms as on the cards; a 10× peak can push writes near 116K/s. Hydration dominates read cost (~50 posts per feed at peak QPS), so batch and cache. Later sections should cross-reference this block rather than restating these rates.
Case-study upgrade
Social Feed Fan-Out Math
A social feed delivers recent posts from people a user follows. The hard part is deciding whether to pay fan-out cost during writes or reads.
Assumptions and formulas
- Daily active users: 200M. Average posts/user/day: 5. Average feed loads/user/day: 20.
- Timeline cache stores post ids first; full post objects are hydrated separately from caches or databases.
- Celebrity accounts are treated as hot publishers and should not blindly fan out to every follower.
Concrete math
- Posts: 200M x 5/day = 1B posts/day = 11.6K writes/sec average; 10x peak is 116K writes/sec.
- Traffic split (approximate): original posts ~4.6K/s, reposts ~2.9K/s, replies ~2.3K/s, media-attached posts ~1.7K/s, likes ~116K/s (10B/day), feed reads 46K QPS average (peak 150K-500K QPS).
- Feed reads: 200M x 20/day = 4B/day = 46K QPS average; peak can reach 150K to 500K QPS.
- Write amplification: normal account with 500 followers = 11.6K posts/sec x 500 = 5.8M timeline insertions/sec (500x). Celebrity with 20M followers bypasses fan-out; merge on read instead.
- Fan-out workers: 5.8M inserts/sec / 10K inserts/sec per worker batch = ~580 workers minimum; at 50K inserts/sec per worker, ~116 workers.
- Fan-out: 11.6K posts/sec x 500 followers average = 5.8M timeline insertions/sec.
- Timeline cache: 200M users x 800 post ids x 8 bytes = 1.28 TB raw; with sorted-set overhead, plan 5 to 20 TB.
- Hydration: 50 posts/read x 150K peak QPS = 7.5M post lookups/sec before batching and cache hits.
- Hydrated post-object cache: ~2-5 KB/post (text, user, media refs). 1M hot posts x 3 KB = 3 GB raw; 100M hot posts x 3 KB = 300 GB raw before replicas and overhead.
Design decisions forced by numbers
- Use fan-out-on-write for normal accounts and merge celebrity posts on read.
- Store candidate ids separately from hydrated post, user, media, and privacy data.
- Monitor fan-out lag, cache hit rate, hydration QPS, and stale/deleted candidate rate.
Scaling to millions
- At 1M users: chronological feed cache and simple fan-out workers are enough.
- At 10M users: shard timelines by reader id and batch fan-out writes.
- At 100M+ users: hybrid fan-out, hot-publisher bypass, regional caches, and degraded chronological fallback.
Readable flow
Interview follow-ups to expect
- Where do we pay fan-out cost: write path or read path?
- How do privacy changes and deleted posts get filtered before display?
- How fresh must the feed be during fan-out backlog?
Operational readiness checklist
After the math, explain how this capacity plan is measured, protected, launched, and evolved.
Observability
- Attach each capacity estimate to a dashboard metric: p99 latency, queue age, cache hit rate, shard skew, error rate, and freshness lag.
- Alert on rate-of-change and saturation, not only hard failures; backlogs and hot keys usually warn before an outage.
Security and abuse
- Budget for auth checks, rate limits, quota isolation, audit logging, privacy filters, and abuse-review side paths.
- Separate trusted internal events from user-controlled input so retries and backfills cannot bypass policy checks.
Degraded modes
- Define the fallback when a dependency is slow: stale cache, partial results, read-only mode, delayed async work, or admission control.
- Protect the invariant first; reduce freshness, personalization, or analytics quality before risking duplicate or corrupt state.
Testing and rollout
- Test average, peak, and hot-object traffic; include retry storms, cold caches, lagging workers, and regional dependency loss.
- Use canaries, shadow reads, dual-write validation, feature flags, and explicit rollback criteria for risky path changes.
Rollout, backfill, and runbooks
- Make migrations and backfills idempotent, resumable, rate-limited, and visible through progress and error metrics.
- Write the runbook before launch: owner, dashboards, paging thresholds, mitigation switches, replay steps, and repair commands.
10x evolution
- For social feed fan-out math, state which number forces the next move: cache memory, shard count, write amplification, queue lag, egress, or failover reserve.
- Add the next layer only when the metric forces it: sharding, regional reads, async replay, hot-key isolation, or stricter backpressure.
Operational depth and 10x plan
What separates a feed sketch from a production answer.
With the rates in Numbers to know and the capacity panel above, this checklist ties planning numbers to SLOs, failure modes, tests, and a celebrity write-path drill interviewers often push on.
SLOs and metrics
Feed p99 under 200ms, post create p99 under 300ms, and 99% of within 60s. Track fan-out lag, hit rate, batch size, ranking p99, stale-feed age, and privacy-filter rejects.
Degraded modes
If ranking is unavailable, serve eligible candidates in chronological order and communicate any material limitation. If fan-out lags, a bounded read-time merge can retrieve recent posts from selected followees, but pulling every followee for every request can overload the source stores. Keep privacy checks active in all fallback modes. A reduced feed may be less personalized; it must not expose a deleted or unauthorized post.
Testing
Load test with skewed follower counts, replay post-created events into a shadow timeline store, chaos test cache loss, and canary ranking or fan-out threshold changes by region.
10x evolution
Add regional timeline caches, celebrity read-merge indexes, pre-ranked candidate stores, hot-post replicas, privacy recheck at hydration, and backfill workers for fan-out rule changes.
Hard-part drill
A 20M-follower account posts during a regional event. Do not write 20M timelines synchronously; store the post, mark the author for , precompute a small celebrity candidate list, and let reads merge it with cached follower timelines before ranking and visibility filtering.
Fan-out makes timelines fast, but copied post IDs are derived state. Walk rebuilds, invalidation, and cross-device consistency before naming APIs.
Timeline lifecycle: rebuilds, invalidation, and device consistency
Fan-out makes timelines fast, but copied post IDs are derived state, not the source of truth.
Read the diagram definition
sequenceDiagram
autonumber
participant C as User or moderator
participant P as Policy service
participant DB as Authoritative post store
participant K as Priority event stream
participant T as Timeline caches
participant R as Timeline read
C->>P: delete, block, or visibility change
P->>DB: persist new state and policy version
DB-->>K: post-invalidated event
K->>T: remove post id from cached timelines
Note over T: Best-effort acceleration
R->>T: load candidate ids
R->>DB: batch recheck visibility and tombstones
DB-->>R: authoritative allow or deny
R-->>C: serve only allowed postsStable ranked pagination
A cursor identifies where pagination should continue under a defined ordering contract. For chronological feeds, it may contain the last timestamp and post ID; ranked feeds may need a ranking version or short-lived candidate snapshot. Sign or validate the cursor, bind it to the viewer, and define expiry. New posts and rescoring otherwise move boundaries between requests, causing repeated or skipped items with simple numeric offsets.
Online feed recomputation
Replay post and graph events into a versioned shadow , compare fill rate and sampled diffs, then atomically switch the reader pointer. Retain the previous version for rollback instead of rewriting the live cache in place.
Delete, privacy, and moderation
Persist a tombstone or policy version first, publish a high-priority invalidation, and evict copied IDs asynchronously. must still recheck deletes, blocks, mutes, audience membership, and quarantine state because cache invalidation can lag. Authorization data used during hydration must itself meet the revocation promise. If the check reads a five-minute stale permission cache, the check’s existence does not make deletion immediate. Use policy versions, bounded freshness, or an authoritative check for sensitive transitions. Purge copies and caches as a separate cleanup path while enforcing the visibility boundary at serving time.
Hot authors
Bucket a follower set, first to recently active followers, and merge the rest on read. Devices share server-issued cursor/session state; a privacy change is authoritative across every device even when their cached versions differ.
Invariant
A copied timeline entry may be stale; an unauthorized post must never be served. Cache eviction improves freshness, while the authoritative read-time policy check preserves correctness during retries, partitions, backfills, and moderation races.
With lifecycle and rebuild rules set, name the HTTP contracts interviewers expect on the whiteboard.
APIs worth naming
Method, path, and one-line purpose: enough for most interview time boxes.
These are the contracts interviewers usually expect you to name. Keep create and home-feed separate: one optimizes for durable writes, the other for fast, paginated reads.
/api/v1/postsCreate post (auth, optional media_ids)/api/v1/timeline/homeCursor-based home feed/api/v1/timeline/user/{id}Profile timeline (author shard)/api/v1/followFollow user; graph + optional timeline seed/api/v1/posts/{id}/likeIdempotent like; async counters/api/v1/search?q=Elasticsearch-backed searchFor POST /posts and GET /timeline/home, mention (not OFFSET) and an idempotency key on create. Point back to the write/read path diagrams for handler behavior.
Those APIs stay thin; Kafka carries , search, and notifications off the post response.
Async pipeline
Kafka decouples posting from fan-out, search, and notifications.
Message bus (Kafka topics)
Posting feels instant because fan-out, search indexing, and notifications run in the background, not inside the create-post request.
Partition by author_id when related records for one author need partition order; post_id instead groups one post’s records and does not order all of that author’s posts. Ordering applies within a topic partition, not across independent topics or arbitrary parallel side effects. Consumer parallelism is also bounded by the available partitions.
Publishers
Services → Kafka
are the online services (or outbox relay) that emit events after their database write succeeds. They do not update follower timelines directly.
post-createdtopicPublish service
Fired after the post row is committed.
Payload: post_id, author_id, created_at. These fields identify the durable post for and search indexing. Commit an outgoing event with the post, then publish through a recoverable relay. The HTTP acknowledgement means accepted state is durable; it need not wait for every consumer to receive the event.
like-eventstopicEngagement API
User liked or unliked a post.
Persist the desired relationship state and a versioned event before returning success. “Set liked” and “set unliked” can be repeated safely; a blind toggle changes the result again on retry. Downstream workers project the resulting transition into counters and optional notifications without blocking the click on every derived count.
follow-eventstopicSocial graph service
New follow or unfollow relationship.
Updates the graph store (who follows whom). Optional: seed the follower's timeline with recent posts from the followee. Unfollow may trigger timeline cleanup async so the follow API stays quick.
repost-eventstopicEngagement API
User shared someone else's post to their followers.
Product-dependent: may create a new post row and run fan-out again, or only attach metadata. Still published async so repost spikes do not stall the gateway.
Consumers (workers)
Kafka → workers
run out of band. Each subscribes to one or more topics, processes messages in batches, and must tolerate delivery (dedupe on post_id). Different worker types can share a topic (fan-out vs search) or use separate topics.
post-createdtopicFan-out workers
Copy post_id into follower timelines, or skip for celebrities.
load followers, batch into timeline:{follower_id}, or mark celebrity authors for merge-on-read. Scale with consumer group partitions; alert on .
post-createdtopicSearch indexer
Make the post searchable in Elasticsearch.
Denormalize content, hashtags, and author handle into the search index. Search can lag seconds behind posting; that is acceptable for most products.
like-eventstopicCounter worker
Approximate or exact like counts + notify author.
A raw Redis INCR on likes:{post_id} repeats its effect on redelivery. Under processing, deduplicate a stable transition ID or conditionally apply a newer relationship version before changing the count. This preserves a legitimate like → unlike → like sequence and lets notifications use their own operation identities.
user-eventstopicAnalytics / warehouse
Trending topics, BI, abuse signals.
Stream into ClickHouse, Spark, or BigQuery for aggregates (top hashtags, DAU). No impact on feed p99; used for dashboards and ranking features later.
Simple rule of thumb: if consumers lag while durable publication still works, posting and existing cached feeds can continue under their contracts. Monitor and oldest event age. A bounded recent-post merge can improve freshness, but pulling every followee for every request during a backlog can overload source storage.
With the async path named, show how a and media stay off the critical read path.
Post IDs and media
A Snowflake-style layout combines time, worker identity, and a per-time-unit sequence into a roughly time-ordered ID. The illustrated 41 + 10 + 12 bits use 63 bits, leaving the sign bit reserved in a 64-bit representation. Uniqueness requires exclusive worker IDs and a defined response to clock rollback or sequence exhaustion. These IDs are convenient for lookup and chronological ordering; they do not establish causal or ranked-feed order.
Media handling
- Presigned S3/GCS upload stores media URLs on the post so create stays fast.
- Async WebP and HLS transcode keeps media encoding off the write path.
- CDN fronts object storage while clients lazy-load thumbnails to cut bandwidth.
- Feed reads show placeholders until transcode finishes so never waits.
IDs and media only matter if the tables match the write and read split already described.
Data model (core tables)
users
Stores profile and auth fields; cache hot users for frequent .
posts
primary key; shard by user_id for fast profile timelines.
followers
Shard by followee_id so workers read follower lists efficiently.
home_timeline
Per-reader as Cassandra or Redis ZSET of post ids.
likes
Keep likes off the post row and aggregate into counters asynchronously.
key: timeline:{user_id}: sorted set of post_id by timestamp. with post:{id} and user:{id} caches in batch.
Schema alone will not meet the hit-rate and safety goals from Numbers to know above; cache layers and abuse checks close that gap.
Caching, search, and safety
Caching layers
- L1 in-process LRU on the timeline service cuts repeat local reads.
- L2 Redis cluster caches timelines, posts, users, and trending keys.
- CDN serves media and static assets so origin storage stays cool.
- Cache-aside with single-flighter on hot post misses prevents stampedes.
Abuse and privacy
- Rate-limit post, follow, and like APIs so abuse cannot flood writes.
- Apply block and mute filters at read time on the hydrated feed.
- Score spam asynchronously and shadow-ban without blocking the write path.
- Route reports to a queue with human review for escalations.
When traffic skews or a worker stalls, say which failure stays user-visible and which you degrade safely. Peak write and math live in Numbers to know above.
Bottlenecks and degraded modes
Celebrity fan-out
Risk: One post can trigger millions of writes.
Mitigation: Skip push ; merge celebrity posts on read instead.
Hot partition
Risk: A trending user overloads one shard so reads and writes pile onto one machine.
Mitigation: Add read replicas, dedicated cache replicas, and rate-limit writes to that shard.
Fan-out lag
Risk: Friends' fresh posts go missing from the feed when lags.
Mitigation: Show cached feed and pull recent from followees; alert on consumer lag.
Hydration storm
Risk: One feed read turns into hundreds of DB calls during .
Mitigation: Batch MGET; cap posts per page; coalesce duplicate authors.
Close the design by showing both the five-minute sketch and the deep-dive stack without inventing new components.
Beginner vs advanced answers
Beginner (5 min)
Users, posts, follows, and a from a DB query or simple cache. Mention media on S3 plus CDN.
Advanced (deep dive)
, Kafka workers, Redis/Cassandra , , batch , Elasticsearch search, merge, fan-out lag metrics, degraded chronological feed.
Choosing and explaining Social microblog feed
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
This is Twitter/X/Threads-shaped: short posts, follow graph, home timeline.
Separate write (post + Kafka) from read (timeline IDs + hydrate).
Hybrid fan-out: push for normal users, pull celebrities on read.
When it belongs
Classic “design Twitter” or “design a news feed” question.
They ask about celebrities, fan-out cost, or feed latency.
They probe consistency of likes vs durability of posts.
How it fails
Celebrity post tries to write 20M timelines synchronously.
Feed read does N+1 queries per post.
Fan-out workers fall behind with no read-path fallback.
Trade-offs to weigh
Fan-out on write vs read vs hybrid.
Chronological vs ranked feed (ranking can fail open).
Strong consistency on counts vs availability.
Common misunderstandings
Storing full post bodies inside every follower timeline row.
Blocking POST /posts until all followers are updated.
OFFSET pagination on billions of rows.
Deeper walkthrough
Social microblog deeper walkthrough
Three paths: post writes, timeline reads, and side paths (search, notifications, ranking). The timeline path is the interview centerpiece.
Requirement refinements
Clarify home timeline vs user profile vs search vs DMs. They differ in access patterns.
Posts must be durable quickly; likes, counts, and search may lag (eventual consistency).
API design details
Cursor pagination on (timestamp, post_id), never OFFSET on feeds.
Split create post from like/repost APIs for independent rate limits and aggregation.
Data model and storage
Home timeline = precomputed post_id list; hydrate post/user/media separately.
Shard posts by author; followers by followee; timelines by reader.
Scaling strategy
Hybrid fan-out with celebrity threshold (~1M followers).
Batch hydration; monitor fan-out lag, feed p99, cache hit rate.
Reliability, recovery, and operations
If fan-out lags, merge recent posts from followees on read.
If ranking fails, serve chronological candidates.
Deepening reference
Microblog Feed: realistic production notes
Beginner mental model: A timeline system is mostly a fan-out and ranking problem, not just a post table.
Production scope: The system must handle normal users, celebrity accounts, media, search, notifications, abuse controls, and ranking. The main interview distinction is home timeline versus user timeline.
Choose what the first version must guarantee
Start by deciding which microblog feed capabilities are essential. Describe the main read and write operations before choosing infrastructure. For each operation, identify the stored records it uses and the condition that makes its result correct. A fast response is useful only if it satisfies that condition.
Next, separate work the user must wait for from work that can finish later. Estimate requests per second, storage growth, and the busiest object or customer. Use those estimates to choose one bottleneck to investigate. The calculations below are example workloads with stated assumptions; they are not measured limits of a particular product.
Concrete capacity calculations
Traffic: 200M DAU x 5 posts/day = 1B posts/day = 11.6K writes/sec average; 10x peak is 116K writes/sec.
Feed reads: 200M DAU x 20 loads/day = 4B/day = 46K QPS average; peak can reach 150K to 500K QPS.
Fan-out: 11.6K posts/sec x 500 followers average = 5.8M timeline insertions/sec. Workers must batch and partition by follower timeline owner.
Celebrity exception: one author with 20M followers would create 20M inserts for one post. Skip full fan-out and merge that author on read.
Timeline cache: 200M users x 800 post ids x 8 bytes = 1.28 TB raw; with sorted-set overhead, plan 5 to 20 TB.
Hydration: 50 posts/read x 150K peak QPS = 7.5M post lookups/sec before batching and cache hits.
Scaling to millions of users
Partition post storage by author_id or time-bucketed post_id; partition home timelines by reader_id because reads fetch one user timeline.
Use hybrid fan-out: push normal authors into follower timelines, pull very high-follower authors during feed read, then rank and hydrate.
Cache timeline IDs separately from post/user/media hydration. Batch hydration calls to prevent one feed read from becoming hundreds of backend reads.
Run fan-out through durable queues with lag metrics. If workers fall behind, reads still return cached and recent pull candidates with a freshness warning internally.
When ranking fails, degrade to recent chronological candidates rather than failing the feed.
Sequence and data-flow diagram
Walk this path after the high-level design so it is clear what is synchronous, what is asynchronous, and where the system can degrade safely.
Read the diagram definition
sequenceDiagram
participant A as Author
participant P as Post service
participant L as Event log
participant F as Fanout workers
participant T as Timeline cache
participant R as Reader
A->>P: Create post
P->>L: Publish post event
L->>F: Consume event
alt Normal author
F->>T: Insert post ID into follower timelines
else Very high-follower author
F->>T: Mark for read-time merge
end
R->>T: Read home candidates
T-->>R: Ranked hydrated feedCore flow to walk through
Create post: write post, publish event, update normal followers timelines, skip full fan-out for celebrities.
Read home feed: fetch precomputed timeline, merge celebrity/recent candidates, hydrate post/user/media data, rank and page.
Engagement: likes and reposts update counters asynchronously with anti-spam checks.
Production readiness
Microblog Feed: ship, operate, and evolve
Now examine the same request flow during failures and deployments. Tie each operational control to the user behavior it protects.
Observability
Track microblog feed p95/p99 latency, error rate, saturation, and user-visible freshness separately for read, write, and async paths.
Alert on leading indicators such as queue age, hot-key skew, cache miss storms, reconciliation mismatches, and error-budget burn.
Security and abuse
Enforce authentication, authorization, rate limits, input validation, audit trails, and privacy checks at the service boundary.
Treat abuse controls as a first-class path: throttling, quota isolation, moderation hooks, revocation, and safe handling of sensitive data.
Degraded modes
Name what still works if cache, search, ranking, stream processing, object storage, or a regional dependency is slow.
Prefer stale-but-safe reads, partial results, async retry, admission control, and feature disablement over corrupting core state.
Testing and rollout
Load test average, peak, and hot-object traffic; add failure-injection for retries, timeouts, backlog growth, and dependency outages.
Ship with canaries, shadow reads or dual writes when useful, rollback switches, schema compatibility checks, and replay validation.
Runbooks
Document dashboards, paging thresholds, owner actions, rollback steps, data repair commands, and customer-impact language.
Keep backfills and replays idempotent, resumable, rate-limited, and observable so repair work does not create a second incident.
10x evolution
Say which metric forces the next architecture change: shard count, cache footprint, write amplification, queue lag, p99 latency, or regional failover reserve.
Evolve one bottleneck at a time: partition by access pattern, isolate hot tenants, regionalize reads, add replayable async pipelines, then harden multi-region behavior.
Hard parts worth deepening
Hybrid fan-out: fan-out on write for normal accounts and fan-out on read for celebrity accounts.
Timeline hydration must batch fetches to avoid N+1 database reads.
Counters are eventually consistent and should tolerate delayed aggregation.
Ranking requires candidate generation first; do not rank the entire corpus on every read.
Failure modes and recovery
Celebrity post overloads fan-out workers; bypass fan-out and merge on read.
Timeline cache loss; rebuild from post store and follower graph with degraded freshness.
Media service slows feed; return text-first feed with placeholders.
Abuse spam floods writes; rate limit by account, IP, device, and reputation.
Signals to monitor
Home-feed p99 measures slow feed loads, while fan-out lag measures how long new posts wait to enter follower timelines. A feed can load quickly and still omit recent posts, so both matter.
Compare timeline cache hit rate with the cost of merging celebrity posts during reads. Track post-write success separately: a successful publish does not prove every follower has received the post.
| Choice | Best when | Cost |
|---|---|---|
| Fan-out on write | Most authors have manageable follower counts. | Write amplification for popular users. |
| Fan-out on read | Authors have massive follower counts. | Read path becomes more expensive. |
| Chronological feed | Simplicity and explainability matter. | Lower engagement than ranked feed. |
Try this scenario
Design the exact read path for a user who follows 1,000 normal accounts and 20 celebrity accounts.
Wrap-up
The interview angle, deeper walkthrough, and upgrade checklist above show how to talk the design. Use this card to lock invariants, rehearse a failure story, drill the fan-out math, and leave with one clean takeaway.
Invariants
A muted or deleted post must never reach the viewer, even when fan-out lags and caches still hold post IDs. Recheck visibility at ; best-effort cache invalidation alone is not enough.
Failure timeline: post without
Contrast this spike with average fan-out cost in Numbers to know: one 20M-follower account can drown the write path that average amplification already stresses.
T+0 Celebrity with 20M followers hits POST /posts; handler tries sync fan-out T+30s Worker queue depth explodes toward tens of millions of timeline writes T+2m Post create p99 collapses; ordinary authors cannot publish T+8m Home feeds for some followers show the post; others stale for hours T+20m Privacy delete races with lagging fan-out: unauthorized copy still in cache Outcome Without hybrid skip + read-time policy check, write path and authz both break
Exercises
- Using the average math in Numbers to know, at what follower threshold would you switch an author to pull-on-read, and which metric shows the write path stayed healthy?
- Fan-out lag is 15 minutes. Design the read-path merge that still never serves a muted or deleted post (defend the authorization invariant above).
- Ranking is down. Argue chronological cached candidates vs an empty feed using product goals and SLO language.
References
Feed fan-out and social graph scale sources.
Key takeaway
A microblog feed is the same class of system as Twitter, X, or Threads: short posts, a follow graph, and a personalized home timeline. Draw the HLD, then walk two paths clearly.
Write: persist the post, return quickly, and fan out post IDs through Kafka workers, not inside the HTTP handler. Skip full push for celebrities and merge them on read (hybrid fan-out).
Read: load timeline post IDs from cache or Cassandra, batch-hydrate posts and authors, then filter mutes and deletes. Cite capacity and amplification in Numbers to know; name Redis, Cassandra, Kafka, and Elasticsearch for the usual store map.
Resilience: if fan-out lags, posting and cached feeds still work; reads can pull recent followee posts until workers catch up. If ranking fails, serve chronological candidates.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Build timeline fan-out around follower skew, celebrity hot paths, stable pagination, privacy/deletion, ranking, recomputation, and multi-device consistency.
Explain the decision
Why use hybrid fan-out for a social timeline?
Read the reasoning
Push ordinary authors into follower inboxes for fast reads, but pull celebrity content at read time to avoid enormous write amplification.
Apply it to a production scenario
A private account revokes a follower after posts were fanned out. What happens?
Read one approach
Authorization is rechecked during hydration, cached entries are invalidated/filtered, and removal propagation is measured and replayable.
Work through the numbers
A user with 8 million followers posts twice/hour. What fan-out write rate does eager push create per post?
Read the worked calculation
Eight million writes per post; even spread over 60 seconds is about 133,000 writes/s, motivating a pull/hybrid path.
Reason through a failure
What must timeline pagination preserve?
Read the recovery reasoning
A versioned ranking boundary prevents duplicates/skips across pages while privacy and deletion checks remain fresher than cached candidates.
Sources and further study
Continue with Design Chat System.