System Design Case Study
Real-time analytics systems
Product, ops, and growth teams want counts, funnels, and alerts within seconds of user activity— not tomorrow's batch warehouse refresh. The hard part is not drawing a dashboard; it is defining freshness, correctness under late data, and cost when explodes.
Treat analytics as a pipeline: durable , time-aware , , and an for slice-and-dice—each with an explicit consistency story.
Event → insight journey
Validate, quota, append durably.
Watermarks close buckets.
Dashboards hit views, not raw.
Ad-hoc queries + pages.
What actually happens
Clients and services emit events through an ingest API that authenticates, schema-validates, applies quotas, and appends to a partitioned . Acceptance means "durably logged," not "already on the CEO dashboard."
Stream processors assign , advance , and update . An may hold both rollups and sampled raw rows for exploration. Alerts subscribe to anomalies on the same aggregates—not to a separate snowflake pipeline that disagrees at 3am.
, retries, and are normal. Define whether closed windows receive corrections, and how applies to sinks versus alerts.
How to choose the design moves
1. What is "real-time"? Seconds for ops dashboards, minutes for product funnels, sub-second for fraud scores—each tier can share the log but use different jobs and SLOs.
2. Exact or approximate? Unique users and high-cardinality group-bys often need HyperLogLog or top-k sketches. Exact distincts at global scale are a cost decision, not a moral failing.
3. One pipeline or lambda? Prefer a durable log plus streaming materializations; add batch for corrections. Reach for classic only when you can staff two merge paths.
Simple rule of thumb
The stream is the source of truth for what happened; serving stores are derived. If you cannot replay, you cannot fix yesterday's bug.
In simple terms: write once to a log, compute views you can rebuild, and never let a dashboard query scan every raw event live.
Case-study upgrade
Realistic Case Study Math
Realistic problems are product stories with one hidden bottleneck. First find what must be exact, then calculate the stream, storage, and worker load.
Assumptions and formulas
- For collaboration, count edit operations, hot documents, snapshots, and presence fan-out.
- For marketplace and matching systems, count state transitions, candidate scans, and timeout windows.
- For telemetry and metrics, count messages/sec, series cardinality, compressed blocks, and alert evaluator work.
Concrete math
- Collaborative editor: 1B edit operations/day = 11.6K ops/sec average; a hot document with 1K editors can create thousands of presence messages/sec.
- Delivery marketplace: 100K orders/minute in a region = 1.67K orders/sec; scanning 50 candidates/order gives 83K candidate scores/sec.
- Device telemetry: 10M devices x 1 message/minute = 167K messages/sec; a 5x reconnect burst becomes 835K messages/sec.
Design decisions forced by numbers
- State-machine cases need idempotency keys, audit logs, and clear timeout transitions.
- Ingestion cases need partitioned streams, backpressure, replay, and late-event handling.
- Hot collaborative objects need a single sequencer or explicit conflict-merge policy.
Scaling to millions
- At 1M users: one region and simple queues are enough for most practice cases.
- At 10M users: isolate hot tenants, shard streams, and separate online serving from analytics.
- At 100M+ users: design replay, regional failover, quota protection, and partial-result serving.
Readable flow
Interview follow-ups to expect
- What must be exactly correct, and what can lag?
- How do retries avoid duplicate state changes?
- How large can the backlog become during a regional or worker outage?
Operational readiness checklist
After the math, explain how this capacity plan is measured, protected, launched, and evolved.
Observability
- Attach each capacity estimate to a dashboard metric: p99 latency, queue age, cache hit rate, shard skew, error rate, and freshness lag.
- Alert on rate-of-change and saturation, not only hard failures; backlogs and hot keys usually warn before an outage.
Security and abuse
- Budget for auth checks, rate limits, quota isolation, audit logging, privacy filters, and abuse-review side paths.
- Separate trusted internal events from user-controlled input so retries and backfills cannot bypass policy checks.
Degraded modes
- Define the fallback when a dependency is slow: stale cache, partial results, read-only mode, delayed async work, or admission control.
- Protect the invariant first; reduce freshness, personalization, or analytics quality before risking duplicate or corrupt state.
Testing and rollout
- Test average, peak, and hot-object traffic; include retry storms, cold caches, lagging workers, and regional dependency loss.
- Use canaries, shadow reads, dual-write validation, feature flags, and explicit rollback criteria for risky path changes.
Rollout, backfill, and runbooks
- Make migrations and backfills idempotent, resumable, rate-limited, and visible through progress and error metrics.
- Write the runbook before launch: owner, dashboards, paging thresholds, mitigation switches, replay steps, and repair commands.
10x evolution
- For realistic case study math, state which number forces the next move: cache memory, shard count, write amplification, queue lag, egress, or failover reserve.
- Add the next layer only when the metric forces it: sharding, regional reads, async replay, hot-key isolation, or stricter backpressure.
Scope you should nail down
Product surface
- Event ingest SDK/API with schemas and quotas
- Real-time dashboards (KPIs, funnels, funnels by segment)
- Alerting on thresholds and anomaly scores
- Ad-hoc exploration over recent + historical windows
- Backfill / metric versioning when definitions change
Scale targets
- ~500K–2M events/sec peak (state your assumed product)
- Dashboard p99 <200ms for pre-aggregated KPIs
- End-to-end freshness: e.g. 15–60s for ops, 1–5m for product analytics
- Retention: hot days in OLAP, cold in object storage / lake
Capacity picture
Partitions: 1M/s ÷ 10K msg/s/partition ≈ 100 partitions minimum; plan headroom for rebalance and hot keys.
State: 5-minute sliding unique users across 10K segments needs sketches, not giant hash sets per key.
Serving: 10K dashboard QPS should hit rollup tables/caches; OLAP scans are for explorers, not every page refresh.
Cardinality: unbounded tag keys at 1M/s will OOM aggregators long before disk fills—enforce allow-lists.
Ingest to dashboard path
Read the diagram definition
sequenceDiagram
autonumber
participant C as Clients / services
participant G as Ingest gateway
participant K as Event stream
participant SP as Stream processor
participant MV as Materialized views
participant OL as OLAP store
participant UI as Dashboard / API
C->>G: POST events (batch)
G->>G: validate + enrich + quota
G->>K: append (partition key)
G-->>C: 202 accepted
K->>SP: consume by offset
SP->>SP: window + watermark
SP->>MV: upsert rollups
SP->>OL: sink aggregates / raw samples
UI->>MV: low-latency KPI read
UI->>OL: ad-hoc slice queryKeep the client path async: 202 after durable append. Dashboards never query the ingest API's memory.
Late events and watermarks
Read the diagram definition
sequenceDiagram
autonumber
participant K as Event stream
participant SP as Stream job
participant W as Window state
participant SO as Side output / DLQ
participant MV as Serving view
K->>SP: event (event_time=T1)
SP->>W: assign to window [T0,T0+5m)
Note over SP: watermark advances past T0+5m
SP->>MV: close window → publish aggregate
K->>SP: late event (event_time=T1)
alt within allowed lateness
SP->>W: update window
SP->>MV: corrective upsert
else beyond lateness
SP->>SO: emit late record
Note over MV: aggregate unchanged; alert on late rate
endInterviewers listen for + allowed lateness + side outputs. Saying "exactly-once Kafka" without a late policy is incomplete.
APIs worth sketching
ingestEvents(project_id, batch[], idempotency_key?) → { accepted, rejected[] }
defineMetric(name, expression, dimensions[], window) → metric_id
queryKPI(metric_id, range, filters) → timeseries // hits materialized view
queryExplore(sql_or_builder, range) → rows // OLAP; rate-limited
createAlert(metric_id, predicate, notify) → alert_id
triggerBackfill(metric_id, from, to, version) → job_idCore data model
Event
Stream partition meta
Metric definition
Rollup row
Alert
Backfill job
High-level architecture
Ingest, schema, and partitioning
Batch small events from mobile to cut connection overhead. Reject unknown event names and unbounded property keys early. Partition by entity when order matters; randomize when hot users would pin one partition.
Schema evolution sketch
// Compatible changes: add optional field with default
// Breaking: rename/remove required field → new event_name or version
props.schema_version = 3
if (!allowList(props.keys)): reject
if (props.bytes > MAX): reject or truncate documented fieldsStream processing deep dive
- Tumbling: non-overlapping buckets for periodic KPIs
- Sliding: overlapping for smoother charts
- Session: activity gaps define visits
Derive from event_time with a bounded out-of-orderness estimate. Idle sources need idle watermarks or windows stall forever.
Idempotent upserts keyed by (metric, window, dims) give practical exactly-once aggregates. Alerts need cooldown keys so retries do not page twice.
Cap dimension enums, hash unknown into "other," use HLL for uniques, and isolate abusive projects with quotas.
Serving layer
KPI pages read (or a thin cache in front). Explorers hit the with scan budgets and result-size limits. Do not let a single "GROUP BY user_id" query take down the cluster— admission-control expensive queries.
Cold history lives in a lake; tiered storage moves aged segments out of hot OLAP. Document freshness: "ops board ≤60s delay" vs "finance daily reconcile."
Backfill and metric changes
When a funnel definition changes, bump metric_version, run into a new table or partition, compare checksums on overlapping ranges, then flip the read pointer. Avoid mutating the live view in place under concurrent streaming writes.
// Cutover
1. Start stream job writing metric_v2 (live)
2. Backfill v2 for [T_hist, T_cut)
3. Validate: sum/count within ε vs v1 sample
4. Atomically point query router to v2
5. Retain v1 until rollback window endsFailure modes
Consumer lag spike
Scale processors, shed non-critical sinks first, and show "data delayed" in UI rather than serving silently stale numbers as fresh.
Hot partition
One celebrity key saturates a shard. Salt keys for unordered aggregates; keep unsalting only where order is required.
Cardinality bomb
A new tag with millions of values OOMs state. Kill the metric, reject at ingest, and page the owning team.
Duplicate alerts
At-least-once delivery without cooldown keys. Store alert fingerprints with TTL.
Backfill vs live clash
Two writers to one mutable aggregate. Use versioned sinks or exclusive ownership per range.
Trade-offs
Freshness vs cost
Sub-second everything is expensive. Most product analytics tolerate minutes; spend budget on the few metrics that drive pages.
Exact vs approximate
Sketches save memory at the price of error bars. Finance-facing numbers may need exact batch reconciliation overnight.
Lambda vs kappa
doubles logic; a single replayable stream simplifies ops if the log retention and compute allow backfills.
Push alerts vs pull checks
Stream-side detectors react fast; periodic evaluators are easier to reason about. Many systems do both with shared metric definitions.
Production readiness
SLOs and metrics
- Ingest accept success, schema reject rate, quota throttles
- Consumer lag by job; end-to-end event_time → view freshness
- Late-event rate, watermark stall, OLAP query error/p95
- Alert precision/recall sampled; page fatigue signals
Degraded modes
- Lag high → mark UI stale; keep accepting ingest
- OLAP down → serve cached KPIs only; disable explore
- Cardinality breach → drop offending dimensions automatically
Security
- PII minimization at ingest; tokenize where possible
- Project-level ACLs on query; row filters for tenants
- Audit metric definition changes and backfill cutovers
Choosing and explaining real-time analytics
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
Start from a durable event log, then derive windows and materialized views with explicit freshness SLOs.
Explain watermarks, allowed lateness, and how late events correct or side-output.
Call out cardinality controls and approximate structures for unique counts.
When it belongs
The prompt asks for live dashboards, streaming metrics, or operational alerting on user events.
You need to separate transactional databases from analytical serving paths.
Interviewers probe exactly-once, late data, or backfills after metric bugs.
How it fails
Serving stale rollups without labeling delay when consumers lag.
Unbounded tags causing aggregator OOM.
Alert storms from at-least-once sinks without fingerprint cooldowns.
Trade-offs to weigh
Seconds-level freshness versus cluster cost.
Exact distincts versus HyperLogLog error bounds.
Lambda dual pipelines versus replayable single-stream (kappa-style) designs.
Common misunderstandings
Querying raw Kafka from the dashboard on every page load.
Claiming exactly-once without defining the sink idempotency key.
Ignoring event time and building windows on processing time only.
Takeaway
Real-time analytics is a replayable log plus disciplined derivations. Windows need watermarks, views need versions, and cardinality needs a budget—or the system will fail loudly at the first viral tag.
Review the design
Ingest: validate, quota, append to a durable partitioned stream; return 202.
Process: event-time windows, watermarks, late-event policy, idempotent sinks.
Serve: materialized KPIs for dashboards; OLAP for exploration with admission control.
Cardinality: allow-lists, sketches, and kill switches for runaway dimensions.
Change: version metrics and backfill into new views before cutover.
Ops: lag, freshness, and stale banners beat silent wrong numbers.
Compare a starting answer with a more complete design
A starting answer
Collect events into a queue, aggregate counts in stream jobs, store rollups for dashboards, and alert on thresholds.
Developing the reasoning
Discuss event-time watermarks, late-event corrections, exactly-once sink idempotency, cardinality governance, metric versioning, and backfill cutover.
Deeper walkthrough
Real-time analytics deeper walkthrough
Product and ops need near-live metrics from high-volume event streams. Correctness under delay and cost under cardinality matter as much as chart latency.
Requirement refinements
Define freshness tiers, exact vs approximate metrics, retention, multi-tenant isolation, and alert semantics.
State invariants: durable ingest before ack; dashboards never require scanning the raw log; late policy is documented.
API design details
Batch ingest with idempotency keys; separate KPI query from exploratory query with different quotas.
Metric definitions are versioned resources; backfill is an explicit job API.
Data model and storage
Raw events in the log/lake; rollups keyed by metric, window, and dimension hash; alert fingerprints with TTL.
Keep serving tables rebuildable from the log for any metric version.
Scaling strategy
Partition the log for ingest parallelism; scale stream jobs on lag; isolate heavy OLAP scans.
Bound cardinality at ingest and in aggregators; salt hot keys when order is not required.
Reliability, recovery, and operations
On lag, mark UI stale and keep accepting events; on OLAP outage, degrade to cached KPIs.
Monitor lag, watermark progress, late rate, schema rejects, and alert duplicate rate.
Primary sources
- Apache Kafka documentation: durable event logs and consumer offsets.
- Apache Flink: event time and watermarks: window closing under out-of-order data.
- ClickHouse docs: columnar OLAP serving patterns (illustrative).