System Design Case Study

Designing IoT Backends

Millions of sensors wake, publish a few bytes, and sleep. Operators flip a desired temperature while a plant is offline. The backend must ingest a firehose of , keep a trustworthy , and still push safely.

IoT backends fail differently from web APIs: , huge cardinality, flaky radios, and tenants that can DDoS you with misconfigured firmware. Design for backpressure and identity first.

Device → insight journey

TLS + per-device credentials.

Broker → durable stream fast.

TSDB write; update reported.

Desired state, acks, rollouts.

What actually happens

Devices (or an ) maintain a long-lived session to an broker over TLS. Each publish carries a device identity. Ingest authenticates, applies per-tenant quotas, validates payload schema, and appends to a partitioned stream. The device should get an ack without waiting on dashboards.

Consumers write to a , update the reported section, and feed rule engines. Commands and desired config flow the other direction with correlation ids. Offline devices reconcile twin deltas on reconnect. A broker helps mark abrupt disconnects, but application heartbeats remain the source of “is it healthy?”

The operational nightmare is correlated failure: power restoration, bad firmware, or a broken DNS entry. Without jittered backoff and , auth and brokers collapse. Plan shed policies for low-priority metrics before paging humans about missing charts.

How to choose the design moves

1. Direct devices or gateways? Constrained leaf devices often need a local gateway for protocol translation and WAN buffering. Direct-to-cloud works for capable devices with good connectivity.

2. What is the write rate? 10M devices × 1 msg/min is already ~167K msg/s. That forces partitioned streams, horizontal ingest, and careful TSDB cardinality design—not a single relational table of points.

3. Twin vs raw commands? Use the twin for config that can wait. Use explicit commands for time-critical actions with timeouts and operator-visible status.

4. How multi-tenant? Quotas, topic namespaces, and encryption boundaries per customer. One noisy factory cannot exhaust shared broker capacity.

Simple rule of thumb

Fast durable ingest in front; everything else is a consumer. Identity and quotas are load-bearing security and reliability features, not afterthoughts.

Case-study upgrade

Realistic Case Study Math

Realistic problems are product stories with one hidden bottleneck. First find what must be exact, then calculate the stream, storage, and worker load.

Assumptions and formulas

For collaboration, count edit operations, hot documents, snapshots, and presence fan-out.
For marketplace and matching systems, count state transitions, candidate scans, and timeout windows.
For telemetry and metrics, count messages/sec, series cardinality, compressed blocks, and alert evaluator work.

Concrete math

Collaborative editor: 1B edit operations/day = 11.6K ops/sec average; a hot document with 1K editors can create thousands of presence messages/sec.
Delivery marketplace: 100K orders/minute in a region = 1.67K orders/sec; scanning 50 candidates/order gives 83K candidate scores/sec.
Device telemetry: 10M devices x 1 message/minute = 167K messages/sec; a 5x reconnect burst becomes 835K messages/sec.

Design decisions forced by numbers

State-machine cases need idempotency keys, audit logs, and clear timeout transitions.
Ingestion cases need partitioned streams, backpressure, replay, and late-event handling.
Hot collaborative objects need a single sequencer or explicit conflict-merge policy.

Scaling to millions

At 1M users: one region and simple queues are enough for most practice cases.
At 10M users: isolate hot tenants, shard streams, and separate online serving from analytics.
At 100M+ users: design replay, regional failover, quota protection, and partial-result serving.

Readable flow

Product event

State check

Durable write

Async stream

Worker pool

Read model

Interview follow-ups to expect

What must be exactly correct, and what can lag?
How do retries avoid duplicate state changes?
How large can the backlog become during a regional or worker outage?

Operational readiness checklist

After the math, explain how this capacity plan is measured, protected, launched, and evolved.

Observability

Attach each capacity estimate to a dashboard metric: p99 latency, queue age, cache hit rate, shard skew, error rate, and freshness lag.
Alert on rate-of-change and saturation, not only hard failures; backlogs and hot keys usually warn before an outage.

Security and abuse

Budget for auth checks, rate limits, quota isolation, audit logging, privacy filters, and abuse-review side paths.
Separate trusted internal events from user-controlled input so retries and backfills cannot bypass policy checks.

Degraded modes

Define the fallback when a dependency is slow: stale cache, partial results, read-only mode, delayed async work, or admission control.
Protect the invariant first; reduce freshness, personalization, or analytics quality before risking duplicate or corrupt state.

Testing and rollout

Test average, peak, and hot-object traffic; include retry storms, cold caches, lagging workers, and regional dependency loss.
Use canaries, shadow reads, dual-write validation, feature flags, and explicit rollback criteria for risky path changes.

Rollout, backfill, and runbooks

Make migrations and backfills idempotent, resumable, rate-limited, and visible through progress and error metrics.
Write the runbook before launch: owner, dashboards, paging thresholds, mitigation switches, replay steps, and repair commands.

10x evolution

For realistic case study math, state which number forces the next move: cache memory, shard count, write amplification, queue lag, egress, or failover reserve.
Add the next layer only when the metric forces it: sharding, regional reads, async replay, hot-key isolation, or stricter backpressure.

Scope you should nail down

Product surface

Device registry, provisioning, and credential rotation
Telemetry ingest, dashboards, and alerting
Device twin desired/reported sync
Command API with delivery status
Optional OTA firmware campaigns

Scale targets (example)

10M devices, 1–5 messages/minute average
Peak reconnect 5–10× after regional outage
Command p99 delivery to online devices under a few seconds
Retention: hot 7–30 days, cold months/years by contract

Capacity picture

167K/s

Msgs @ 1/min × 10M

~835K/s

5× reconnect burst

10M

Twin documents

Payload: 200-byte average × 167K/s ≈ 33 MB/s raw before fan-out replication.

Cardinality: 10M devices × 20 metrics = 200M series if naive. Aggregate, limit tag keys, and downsample or you will not afford indexes.

Partitions: Hash by device_id or tenant+device so one industrial site cannot pin one hot partition forever—unless you intentionally isolate whales.

Connections: Millions of concurrent MQTT sessions need regional brokers and sticky routing; plan file-descriptor and memory budgets explicitly.

Telemetry ingest path

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

sequenceDiagram
    autonumber
    participant D as Device / gateway
    participant B as MQTT broker
    participant I as Ingest service
    participant S as Stream log
    participant V as Validator / enricher
    participant TS as Time-series store
    participant A as Alert evaluator
    participant T as Device twin

    D->>B: Publish telemetry (auth TLS)
    B->>I: Forward / bridge message
    I->>I: Authz, schema, rate limit
    I->>S: Append durable event
    I-->>B: Ack path (QoS dependent)
    S->>V: Consume partition
    V->>TS: Write points / aggregates
    V->>T: Update reported properties
    V->>A: Evaluate rules
    A-->>A: Page or ticket if firing

The critical SLA is “message durably accepted.” Downstream lag should page on consumer delay, not block device publishers.

Command and twin path

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

sequenceDiagram
    autonumber
    participant App as Operator / app
    participant API as Device API
    participant T as Twin store
    participant Q as Command queue
    participant B as MQTT broker
    participant D as Device

    App->>API: Set desired config / send command
    API->>T: Update desired state
    alt Device online
        API->>Q: Enqueue command (correlation_id)
        Q->>B: Publish command topic
        B->>D: Deliver
        D-->>B: Ack + result
        B->>API: Result event
        API->>T: Merge reported
    else Device offline
        Note over T,D: Desired waits; device pulls twin on reconnect
        D->>B: Reconnect + subscribe
        D->>API: Fetch twin delta
        API->>T: Read desired vs reported
        T-->>D: Delta
        D->>API: Report applied
    end

APIs worth sketching

typescript

registerDevice(tenant_id, hardware_id, cert_csr) → device_id, credentials
getTwin(device_id) → desired, reported, metadata, version
updateDesired(device_id, patch, if_match_version?) → twin
sendCommand(device_id, name, payload, timeout_ms) → command_id
getCommandStatus(command_id) → pending|acked|succeeded|failed|timed_out
queryTelemetry(device_id, metric, from, to, step) → points
createAlertRule(tenant_id, expression, channels) → rule_id
startOtaCampaign(image_id, cohort, pace) → campaign_id

Core data model

Devices

sql

PK device_id
tenant_id, hardware_id
status, firmware_ver
created_at

Credentials

sql

device_id
cert_serial / token_hash
expires_at, revoked_at

Twins

sql

device_id
desired JSON
reported JSON
version, updated_at

Telemetry

sql

device_id, metric
ts, value, tags
(in TSDB, not OLTP)

Commands

sql

PK command_id
device_id, name
status, expires_at
result JSON

AlertRules

sql

PK rule_id
tenant_id, expr
channels, enabled

High-level architecture

Devices / Edge gateways

↓ MQTT / HTTPS

Broker / Device gateway

↓

IngestRegistry / IdentityTwinCommandRules / AlertsOTA

↓

stream logTSDBmetadata DBobject store (firmware)

Hard parts: identity, cardinality, storms

Identity. Issue unique certs or tokens per device. Mutual TLS to the broker is common. Revocation lists or short-lived credentials limit blast radius. Never share one credential across a SKU batch in production.

Cardinality. Unbounded tag values (session ids as labels) explode TSDB indexes. Enforce allow-lists of tags at ingest; aggregate high-frequency points on the edge or in stream processors.

Reconnect storms. Devices must backoff with jitter after disconnect. Brokers and auth should shed with clear errors so firmware can slow down instead of retrying immediately.

Quota and shed sketch

plaintext

// Per-tenant token bucket at ingest
if !quota.allow(tenant_id, bytes, messages):
  if message.priority == LOW:
    drop_or_downsample(message)   // count as shed
  else:
    reject_with_retry_after(jittered_seconds)

// Device reconnect policy (firmware)
delay = min(cap, base * 2^attempt) + random_jitter()
sleep(delay); reconnect()

Edge, OTA, and alerting

Protocol translation and local buffering
Aggregate 1Hz sensors to 1-minute summaries when WAN is thin
Fail closed on unsafe commands if local policy requires it

Sign artifacts, stage canaries, pace by cohort, auto-pause on crash or disconnect spikes. Store images in object storage; devices download with integrity checks.

Alerting

Evaluate rules on a stream of enriched points or short windows. Deduplicate flaps; page on sustained threshold breaches. Missing-data alerts need explicit “expected heartbeat” configuration or you drown in false pages when radios flake.

Failure modes

Poison payloads

Oversized or schema-invalid messages. Validate early; quarantine to a DLQ; never block a partition forever on one bad device.

Clock skew

Device clocks lie. Prefer server receive time for ordering when absolute accuracy is impossible; store both timestamps.

Duplicate commands

At-least-once delivery re-runs actuators. Devices must key idempotency on command_id.

Twin version conflicts

Concurrent desired updates need version checks (optimistic concurrency). Last-writer-wins without versions loses operator intent.

TSDB lag / disk full

Apply : shed low-priority metrics, keep safety-critical streams, page on consumer lag and disk.

Trade-offs

MQTT QoS 0 vs 1 vs 2

QoS 0 is cheapest and lossy—fine for high-frequency noisy sensors you aggregate anyway. QoS 1 gives at-least-once with duplicates. QoS 2 is rarely worth the complexity for telemetry; reserve stronger guarantees for critical commands via application acks.

Raw retention vs downsampling

Keeping every point forever is expensive. Downsample for dashboards but keep raw windows where forensics or ML need them. Document retention per metric class in the contract with tenants.

Cloud twin vs edge autonomy

Cloud-desired state is convenient for fleets. Safety-critical local loops (industrial shutoff) must work when the WAN is down— edge policy wins over a stale desired temperature that never arrived.

Shared brokers vs dedicated stacks

Multi-tenant brokers are cost-efficient with strict quotas. Regulated or whale customers may need dedicated broker pools for noisy-neighbor and compliance isolation.

Bottlenecks to call out

Broker connections: memory and file descriptors; regionalize and shed.

Hot tenants: isolate partition keys; apply per-tenant rate limits.

TSDB cardinality: enforce label policies at ingest.

OTA bandwidth: pace downloads; use CDNs; avoid syncing with peak telemetry hours.

Choosing and explaining IoT backends

A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.

Explain the core idea

Lead with fan-in: durable ingest, then consumers for TSDB, twin, and alerts.

Call out device identity, quotas, and reconnect jitter as first-class design.

Separate twin config sync from urgent command delivery semantics.

When it belongs

The problem mentions sensors, fleets, MQTT, telemetry, or device control.

You need to show stream ingestion, backpressure, and multi-tenant isolation together.

How it fails

Reconnect storms after outages.

Cardinality explosions and partition hot spots.

Duplicate actuator commands without idempotency.

Trade-offs to weigh

QoS and battery/bandwidth versus delivery guarantees.

Raw retention cost versus forensic/ML value.

Shared platform efficiency versus dedicated isolation.

Common misunderstandings

Putting telemetry inserts on a synchronous request path to a warehouse.

One shared device credential for a product line.

Ignoring offline desired-state reconciliation.

Review the design

Ingest: authenticate, quota, validate, append to a stream; ack fast.

Twin: desired vs reported with versions; sync on reconnect.

Commands: correlation ids, timeouts, device-side idempotency.

Storage: TSDB with cardinality control and tiered retention.

Ops: jittered reconnect, backpressure sheds, staged OTA.

Compare a starting answer with a more complete design

A starting answer

Devices publish MQTT telemetry to a broker; a backend stores time-series, keeps device state, and sends commands.

Developing the reasoning

Discuss per-device identity, partitioned durable ingest, twin concurrency, idempotent commands, cardinality limits, reconnect jitter, multi-tenant quotas, and OTA canaries.

Deeper walkthrough

IoT backend deeper walkthrough

IoT platforms combine long-lived connections, write-heavy telemetry, eventual device state sync, and safety-sensitive command delivery across flaky networks.

Requirement refinements

Clarify device types, message sizes, online command latency, offline config needs, alert freshness, and retention SLAs.

State invariants: only the owning tenant can command a device; command_id is idempotent on the device; twin updates are versioned.

API design details

Expose twin patch with If-Match versions; command APIs return status resources under correlation ids.

Ingest rejects unknown schemas and over-quota tenants with Retry-After guidance.

Data model and storage

Registry and credentials in OLTP; twins document-oriented; telemetry in TSDB; firmware blobs in object storage.

Keep an audit log of desired changes and command outcomes for support and compliance.

Scaling strategy

Partition streams by device or tenant; scale brokers regionally; isolate noisy tenants.

Downsample and aggregate early; bound label cardinality at the gate.

Reliability, recovery, and operations

Load-test reconnect storms; chaos the TSDB consumer lag path; verify command idempotency.

Monitor connection counts, ingest ack latency, consumer lag, twin conflict rates, command timeout rate, and shed counts.

Shipping beyond the first design

The request flow explains normal operation. The next sections examine how you keep that behavior dependable after deployment: measuring user outcomes, limiting access, recovering from failures, and changing the system while people are using it.

Security, privacy, and abuse controls

mTLS or equivalent per device; rotate and revoke; encrypt twins at rest for sensitive configs; authorize every command.

Degraded modes and graceful fallback

Accept and buffer only; delay non-critical dashboards; pause OTA; keep critical command path if capacity allows.

Primary sources

MQTT specification: device pub/sub semantics and QoS.
MQTT 5.0: reason codes, user properties, shared subscriptions.
Apache Kafka documentation: durable fan-in buffering for telemetry.