SwadhyAI Notes

All System Design notes

System Design Case Study

Designing Recommendation Systems

A home feed, a “because you watched,” or a product carousel must choose a handful of items from a huge catalog in tens of milliseconds. The classic shape is then —retrieve broadly, score carefully, then blend.

Quality is not only model accuracy. You need a , defenses against , deliberate , and honesty about the your logs create.

Request → list journey

1

ANN, collab, trending pools.

2

Online lookups for user/item.

3

Heavier model scores the pool.

4

Diversity, business rules, slots.

What actually happens

On request, the service builds context (user id, device, locale, time, recent actions). Several generators produce candidates: a search over an , co-visitation, neighbors, and editorial/trending lists.

Candidates are merged and capped (hundreds). The returns fresh aggregates. A ranker scores for the product objective—watch time, CTR, revenue, or a weighted mix—then filters (already seen, blocked, out of stock) and reorders for diversity. Explorer slots inject learning traffic.

Offline, event logs train new models and rebuild indexes. If you only train on what you showed, the narrows the catalog. Log position bias and keep some exploration so learning remains possible. When the ranker is down, serve cached popular or last-good retrieval lists rather than empty shelves.

How to choose the design moves

1. What is the objective? Click, long watch, purchase, and retention pull ranking in different directions. Multi-objective blending and business constraints (ads, inventory) belong in the product spec, not as afterthoughts.

2. Catalog size? Thousands of SKUs can brute-force more; millions need ANN retrieval and staged ranking.

3. Latency budget? Home feed p99 often sits in the low tens to low hundreds of ms. Precompute item towers; parallelize generators; budget feature fetch carefully.

4. How sparse are interactions? Sparse catalogs lean on content features and cold-start policies; dense social/video graphs lean on collaborative signals.

Simple rule of thumb

Two stages almost always: cheap recall, expensive precision. Everything else—features, exploration, filters—hangs off that spine.

Case-study upgrade

Social Feed Fan-Out Math

A social feed delivers recent posts from people a user follows. The hard part is deciding whether to pay fan-out cost during writes or reads.

Assumptions and formulas

  • Daily active users: 200M. Average posts/user/day: 5. Average feed loads/user/day: 20.
  • Timeline cache stores post ids first; full post objects are hydrated separately from caches or databases.
  • Celebrity accounts are treated as hot publishers and should not blindly fan out to every follower.

Concrete math

  • Posts: 200M x 5/day = 1B posts/day = 11.6K writes/sec average; 10x peak is 116K writes/sec.
  • Traffic split (approximate): original posts ~4.6K/s, reposts ~2.9K/s, replies ~2.3K/s, media-attached posts ~1.7K/s, likes ~116K/s (10B/day), feed reads 46K QPS average (peak 150K-500K QPS).
  • Feed reads: 200M x 20/day = 4B/day = 46K QPS average; peak can reach 150K to 500K QPS.
  • Write amplification: normal account with 500 followers = 11.6K posts/sec x 500 = 5.8M timeline insertions/sec (500x). Celebrity with 20M followers bypasses fan-out; merge on read instead.
  • Fan-out workers: 5.8M inserts/sec / 10K inserts/sec per worker batch = ~580 workers minimum; at 50K inserts/sec per worker, ~116 workers.
  • Fan-out: 11.6K posts/sec x 500 followers average = 5.8M timeline insertions/sec.
  • Timeline cache: 200M users x 800 post ids x 8 bytes = 1.28 TB raw; with sorted-set overhead, plan 5 to 20 TB.
  • Hydration: 50 posts/read x 150K peak QPS = 7.5M post lookups/sec before batching and cache hits.
  • Hydrated post-object cache: ~2-5 KB/post (text, user, media refs). 1M hot posts x 3 KB = 3 GB raw; 100M hot posts x 3 KB = 300 GB raw before replicas and overhead.

Design decisions forced by numbers

  • Use fan-out-on-write for normal accounts and merge celebrity posts on read.
  • Store candidate ids separately from hydrated post, user, media, and privacy data.
  • Monitor fan-out lag, cache hit rate, hydration QPS, and stale/deleted candidate rate.

Scaling to millions

  • At 1M users: chronological feed cache and simple fan-out workers are enough.
  • At 10M users: shard timelines by reader id and batch fan-out writes.
  • At 100M+ users: hybrid fan-out, hot-publisher bypass, regional caches, and degraded chronological fallback.

Readable flow

Post write
Event stream
Fan-out workers
Timeline cache
Feed read
Hydrate and rank

Interview follow-ups to expect

  • Where do we pay fan-out cost: write path or read path?
  • How do privacy changes and deleted posts get filtered before display?
  • How fresh must the feed be during fan-out backlog?

Operational readiness checklist

After the math, explain how this capacity plan is measured, protected, launched, and evolved.

Observability

  • Attach each capacity estimate to a dashboard metric: p99 latency, queue age, cache hit rate, shard skew, error rate, and freshness lag.
  • Alert on rate-of-change and saturation, not only hard failures; backlogs and hot keys usually warn before an outage.

Security and abuse

  • Budget for auth checks, rate limits, quota isolation, audit logging, privacy filters, and abuse-review side paths.
  • Separate trusted internal events from user-controlled input so retries and backfills cannot bypass policy checks.

Degraded modes

  • Define the fallback when a dependency is slow: stale cache, partial results, read-only mode, delayed async work, or admission control.
  • Protect the invariant first; reduce freshness, personalization, or analytics quality before risking duplicate or corrupt state.

Testing and rollout

  • Test average, peak, and hot-object traffic; include retry storms, cold caches, lagging workers, and regional dependency loss.
  • Use canaries, shadow reads, dual-write validation, feature flags, and explicit rollback criteria for risky path changes.

Rollout, backfill, and runbooks

  • Make migrations and backfills idempotent, resumable, rate-limited, and visible through progress and error metrics.
  • Write the runbook before launch: owner, dashboards, paging thresholds, mitigation switches, replay steps, and repair commands.

10x evolution

  • For social feed fan-out math, state which number forces the next move: cache memory, shard count, write amplification, queue lag, egress, or failover reserve.
  • Add the next layer only when the metric forces it: sharding, regional reads, async replay, hot-key isolation, or stricter backpressure.

Scope you should nail down

Product surface

  • Personalized home / feed / “recommended for you”
  • Related items and “customers also bought”
  • Filters: seen items, blocked creators, stock, age rating
  • Explanation hooks optional (“because you liked X”)
  • Offline evaluation + online A/B

Scale targets (example)

  • 100M MAU; 20M DAU home-feed opens
  • Catalog 10M–100M items
  • Serving p99 under ~100–200ms for the recs RPC
  • Training daily (ranker) / hourly (hot features) as needed

Capacity picture

~230/s
Avg feed opens
~2K/s
Peak recs QPS
500
Candidates before rank

Feed opens: 20M/day ≈ 230/s average; evening peak ~8–10× → ~2K/s.

Rank cost: 2K QPS × 500 candidates = 1M score ops/s. Batch on GPU/CPU carefully; prune before deep models.

Feature fetch: Often the real bottleneck—coalesce keys, cache hot user features, avoid per-candidate RPCs.

Events: Impressions can dwarf clicks; sample or aggregate before warehouse load while keeping full fidelity for training windows you care about.

Online serving path

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.
Read the diagram definition
sequenceDiagram
    autonumber
    participant U as User client
    participant GW as API gateway
    participant RS as Recs service
    participant CG as Candidate generators
    participant ANN as ANN index
    participant FS as Feature store
    participant RK as Ranker
    participant F as Filters / blender

    U->>GW: GET home / feed / product-recs
    GW->>RS: user_id, context
    RS->>CG: Parallel retrieve
    CG->>ANN: User embedding kNN
    CG-->>RS: Candidates (multi-source)
    RS->>FS: Batch feature lookup
    FS-->>RS: User + item features
    RS->>RK: Score candidates
    RK-->>RS: Scores
    RS->>F: Filter, diversify, explore slots
    F-->>U: Ranked list + request_id

Parallel generators and batched feature lookup are how you stay inside a latency budget while still personalizing.

Learning and feedback loop

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.
Read the diagram definition
sequenceDiagram
    autonumber
    participant U as User
    participant S as Serving
    participant L as Event log
    participant J as Training jobs
    participant FS as Feature pipelines
    participant M as Model registry
    participant ANN as ANN / indexes

    U->>S: Impressions, clicks, buys
    S->>L: Log with position, props
    L->>FS: Aggregate features
    L->>J: Train retrieval + ranker
    J->>M: Register model versions
    M->>S: Canary / promote ranker
    J->>ANN: Rebuild item embeddings
    Note over U,ANN: Feedback loop: what you showed becomes tomorrow's labels

APIs worth sketching

typescript
getHomeFeed(user_id, context, cursor?, limit) → items[], request_id
getRelated(item_id, user_id?, limit) → items[]
recordImpression(request_id, item_ids, positions)
recordEngagement(user_id, item_id, type, value?)
// Internal
rebuildItemEmbeddings(model_version)
publishRanker(model_version, canary_percent)

Core data model

Users

sql
PK user_id
traits, locale
onboarding prefs
created_at

Items

sql
PK item_id
title, tags, media
status, created_at
creator_id

Interactions

sql
user_id, item_id
type, value, ts
request_id, position

Embeddings

sql
entity_id, type
vector, model_ver
updated_at

Features

sql
entity_key
feature_map
as_of (online store)

Models

sql
model_id, stage
metrics, artifact_uri
rolled_out_at

High-level architecture

Clients
↓
Recs / Feed service
↓
GeneratorsANNFeature storeRankerBlender / filtersEvent log
↓ offline
training warehousemodel registryindex builder

Hard parts: retrieval, features, and bias

Retrieval quality. If the right item never enters the candidate set, the ranker cannot save you. Monitor recall of held-out engagements against generators; add specialized generators for niches the ANN misses.

Training/serving skew. The same feature name must mean the same computation offline and online. Point-in-time correctness during training prevents label leakage from the future.

Position bias. Top slots get clicks regardless of relevance. Log position; use debiasing or listwise methods; do not treat raw CTR as ground truth for every slot equally.

Blender sketch

plaintext
candidates = merge(ann_hits, collab, trending, related)
candidates = dedupe_cap(candidates, 500)
features = feature_store.batch_get(user, candidates)
scores = ranker.score(features)  // multi-objective → scalar
ranked = sort_by(scores)
ranked = filter(ranked, seen, blocked, ineligible)
ranked = diversify(ranked, by=category|creator, strength=λ)
ranked = inject_exploration(ranked, quota=5%)
return ranked[:limit]

Algorithms in the toolbox

Item-item co-visitation graphs are simple and strong for related products. Matrix factorization / two-tower models scale personalization when interactions are dense enough.

Text/image embeddings and metadata neighbors help new items and explainability. Hybrid systems almost always beat pure CF or pure content alone.

Precompute item vectors into ANN; encode the user online. Great candidate generator; usually not the final ranker.

  • Onboarding preference quizzes
  • Popular-in-locale and trending backfills
  • Content embeddings for new items
  • Explorer quota until enough interactions

Failure modes

Empty or near-empty lists

Over-aggressive filters or ANN outage. Keep a popular fallback shelf per market; never return a blank home.

Feature store timeout

Rank with partial features or a lighter model; degrade gracefully rather than failing the page.

Stale indexes

New items invisible; deleted items still recommended. Track index lag SLOs and tombstone quickly.

Runaway feedback / popularity bias

Head content dominates. Inject exploration, diversity constraints, and propensity-aware training.

Bad canary model

Auto-rollback on online metric drops (CTR, watch time, complaints). Shadow traffic before full promote.

Trade-offs

Single-stage vs two-stage ranking

Single-stage scoring of huge catalogs is rarely feasible in interactive latency. Two-stage (or three-stage with a light pre-ranker) is the industry default; accept approximate recall for speed.

Exploit vs explore

Exploitation maximizes short-term engagement. Exploration costs a little CTR now to improve future models and long-tail coverage. Product should set an explicit exploration budget.

Realtime features vs batch

Realtime recent-click features feel magical for session personalization but are operationally expensive and skew-prone. Batch features are stabler; hybrid systems use both with clear freshness tiers.

Personalization vs diversity / fairness

Maximizing predicted affinity can create filter bubbles and creator inequity. Blender constraints and multi-objective losses encode values the business is willing to pay engagement points for.

Bottlenecks to call out

Feature store fan-out: batch and cache; avoid N RPCs per candidate.

ANN rebuilds: incremental upserts vs periodic rebuild; watch recall drift.

Ranker CPU/GPU: candidate caps, distilled models, early-exit scoring.

Event volume: sample impressions; keep joins keyed by request_id.

Choosing and explaining recommendation systems

A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.

Explain the core idea

Draw retrieve → rank → filter/blend, and name at least two candidate sources.

Mention feature stores and training/serving consistency unprompted.

Call out cold start, exploration, and feedback loops as product-system issues, not only ML trivia.

When it belongs

Home feed, related products, “for you,” or personalization prompts appear.

You need to show ANN, caching, and offline/online ML serving together.

How it fails

Empty feed on dependency failure.

Stale indexes hiding new content.

Popularity collapse from uncorrected feedback loops.

Trade-offs to weigh

Latency versus model complexity.

Exploitation versus exploration.

Personalization versus diversity and fairness constraints.

Common misunderstandings

Scoring the entire catalog online.

Optimizing offline AUC while ignoring position bias and online metrics.

No fallback when the ranker or feature store blips.

Review the design

Two stages: candidate generation (ANN, CF, content, trending) then ranking.

Features: shared definitions offline/online via a feature store.

Blender: filters, diversity, business rules, exploration slots.

Learning: log with position; train; canary; watch feedback loops.

Resilience: popular/cached fallbacks beat empty shelves.

Compare a starting answer with a more complete design

A starting answer

Use user history to find similar items, rank them, and show a list; fall back to popular items for new users.

Developing the reasoning

Discuss multi-source candidate generation, two-tower ANN retrieval, feature-store skew, multi-objective ranking, exploration, position bias, index lag SLOs, and canary rollbacks.

Deeper walkthrough

recommendation systems deeper walkthrough

Recommenders couple online serving latency with offline learning. Retrieval, ranking, features, and logging form one loop that can amplify bias if left unchecked.

Requirement refinements

Clarify surface (home vs related), objective metrics, catalog size, latency SLO, and whether ads or inventory constraints apply.

State invariants: never recommend ineligible items; degraded mode must still return a non-empty policy-approved list when possible.

API design details

Return request_id with every list so impressions and engagements join cleanly.

Separate related-item and home-feed endpoints if contexts and generators differ.

Data model and storage

Immutable interaction events with position and context; versioned embeddings; online feature documents with timestamps.

Model registry records metrics and artifact pointers for rollback.

Scaling strategy

Precompute item towers; shard ANN; cache hot user features; cap candidates before deep ranking.

Horizon scale with more generators and a light pre-ranker rather than unbounded scoring.

Reliability, recovery, and operations

Timeouts on feature fetch degrade to lighter scoring; ANN failure falls back to trending/collab caches.

Monitor p99 latency, empty-rate, index lag, online metric deltas on canaries, and exploration fill rate.

Shipping beyond the first design

The request flow explains normal operation. The next sections examine how you keep that behavior dependable after deployment: measuring user outcomes, limiting access, recovering from failures, and changing the system while people are using it.

Degraded modes and graceful fallback

Skip personalization and serve popular-by-locale; hide modules that hard-fail rather than blocking the page.

Testing and validation strategy

Offline recall@k and ranking AUC/NDCG; online A/B with guardrail metrics; replay shadow traffic for new rankers.

Primary sources