System Design Patterns

Logging and monitoring

Checkout fails for a slice of Uber-scale traffic at 2 a.m. The user sees a spinner; on-call sees a page. Without the right signals, engineers guess which hop broke: gateway, payments, inventory, or a bad deploy.

means production can explain itself through , , and . Huge fleets need structure, sampling, and -oriented alerts, not infinite raw noise.

Incident journey

User impact wakes on-call.

Which service and region?

Which hop ate the latency?

Exact failure with the same trace id.

Problem → naive → break

Problem: checkout fails for a slice of traffic at 2 a.m. On-call must find which hop broke without SSH folklore.

Naive move: log everything as free text, page on CPU > 80%, and skip traces because "metrics are enough." Counterexample: CPU looks fine while payment p99 burns the error budget: symptom alerts never fire.

Break: uncorrelated signals. Metrics without an SLI wake nobody useful; logs without trace ids cannot join hops; explosions make the metrics backend itself the outage.

Correctness invariant

Every user-impact page maps to a named and (not a raw host gauge alone). Metric label sets stay inside a declared series budget. Logs, metrics, and traces that describe the same request share a correlation id so triage can move signal → exemplar → span → structured event without guessing.

What actually happens

Every user request can emit three kinds of evidence. cheaply show trends and wake pagers. show where time went across hops. hold the detailed story for one failing order, but only if you use with a shared trace id.

Teams define an (what “good” means) and an (the target). The leftover is the . Alert on , not every noisy CPU blip.

Cost shows up as explosions, unsampled verbose logs, and dashboards nobody uses during incidents. Treat telemetry as a product: name conventions, sampling policies, retention tiers, and runbooks maintained like code.

User request

Service emits telemetry

Which signal?

Logs

Structured logs

Metrics

SLO alert

Traces

Correlate by trace id

Incident triage

Runbook and fix

Back to service telemetry

RequestTelemetryAlertFix loop

Try it: walk a correlated debugging path

Start from a symptom, then move through metrics, traces, and logs in an order that matches production triage. Toggle high-cardinality labels to see how a metric change can break the same dashboards you rely on during incidents.

Starting symptom

Add unbounded user_id label to checkout metrics

Step: 1/5
Active signal: metrics
Estimated series: 5,000
Path: slo burn

Triage timeline

1. SLO burn alert · metrics
2. RED metrics by route · metrics
3. Trace exemplar · traces
4. Structured logs · logs
5. Mitigate · metrics

SLO burn alert: Error budget is draining faster than allowed for the checkout route.

Correlation. Trace ids belong in structured logs so you can jump from a burn alert to the exact requests that consumed budget.

Cardinality budget. Route and status class labels are usually safe. Raw user ids as metric labels are not; they belong in logs with sampling policies.

How to choose what to collect

1. What question are you answering? Trends and paging → metrics. Cross-service latency → traces. Exact event detail → logs.

2. Can the label set stay bounded? Route, status class, region, and version are usually fine. User id and raw URLs are not metric labels.

3. Will someone act on the alert? Page only when there is user impact, an owner, a likely action, and a runbook. Everything else is a ticket.

Simple rule of thumb

Instrument on user-facing services, on nodes, propagate trace context everywhere logs emit. Tier retention: hot days for debugging, warm weeks for incidents, cold archive for compliance.

In simple terms: if an alert lacks a runbook action, it is noise.

Intent, forces, and consequences

Intent

Make production behavior explainable without reproducing every issue locally or logging into individual service hosts.

Forces

More data improves diagnosis, but cardinality, retention, privacy, and paging fatigue cap how much you can collect.

Consequences

Teams need naming conventions, ownership labels, sampling policies, dashboards, and runbooks maintained like code.

Metrics vs logs vs traces (pick by question)

Signal	Answers	Cost driver	SLI linkage
Metrics	Is rate/error/latency changing? Should we page?	Series cardinality (label product)	Numerator/denominator of the SLI; burn-rate alerts
Logs	What exactly happened for this order/request?	Lines/sec × bytes × retention × index	Evidence after SLI burn; not the primary pager
Traces	Which hop ate the latency on this path?	Spans/sec × sampling policy × buffer	Exemplars from bad histogram buckets → spans

Three pillars

Logs

A structured log records one event with named fields, such as order_created, service, severity, and trace_id. Stable field names let an operator filter all failures for one operation without parsing changing prose. Record enough context to explain the decision, while excluding secrets and unnecessary personal data at the source. Sampling verbose success logs reduces volume, but critical audit records require their own explicit durability policy.

Metrics

A counter accumulates events, such as completed requests; use its change over time to calculate a rate. A gauge records a current level, such as active connections. A histogram counts observations in value ranges so latency distributions can be estimated across instances. Labels create separate series for each combination, so route templates and status classes are useful while unbounded user IDs make storage and queries grow without control.

Traces

Span trees show dependency drag; preserves rare slow paths affordably. Each span represents a timed operation, with parent relationships or links connecting related work. A long checkout trace can show that most time was spent waiting for inventory rather than executing checkout code. Trace propagation must cross queue boundaries as well as HTTP calls. Tail sampling needs the relevant spans to reach the sampling system before its decision; incomplete traces and late spans limit what it can preserve.

Structured logging

Avoid

Unstructured prose without machine fields

Prefer

Schema-stable JSON per line

json

{
  "ts": "2026-05-03T10:30:00.123Z",
  "level": "INFO",
  "service": "order-service",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "event": "order.created",
  "order_id": "67890",
  "latency_ms": 47
}

Ingestion pipeline (sketch)

Agents on nodes → durable append-only buffer → processors → hot search store + cold object archive + metric derivations.

A buffer separates application log production from slow indexing. If indexing pauses for ten minutes while applications emit 10 MB/s, at least 6 GB of additional raw buffer space is needed, before overhead. When the buffer fills, explicitly choose which data to drop or spool locally. Blocking business requests on a failed telemetry destination can turn a diagnostic outage into an application outage.

App emitters

Local buffer

Collector

Processor and sampler

Where does telemetry go?

Hot

Hot query store

Cold

Cold archive

replay for investigation

Alerts

Alert evaluator

Incident triage

Telemetry volume math

Logs per second

200 services × 20 pods × 50 log lines/sec = 200k lines/sec. At 800B each, that is about 160MB/sec raw before indexing.

Daily storage

160MB/sec × 86,400 sec/day is about 13.8TB/day raw. A 2.5× index multiplier makes hot indexed storage about 34.5TB/day before compression.

Metric cardinality

Metric cardinality is the number of distinct label combinations that actually appear. For one metric, 50 routes × 20 status codes × 200 tenants × 4 methods gives an upper-bound illustration of 800,000 combinations before replicas and histogram buckets. Not every combination necessarily exists, but adding an unbounded label removes any useful ceiling. Enforce per-service budgets and inspect the largest contributors before queries and alert evaluation slow down.

plaintext

log_bytes_per_day = lines_per_sec * avg_bytes * 86_400
indexed_storage ~= raw_bytes * index_multiplier
series = product(label_value_counts)
sampled_traces = request_rate * sample_rate
backlog_seconds = queued_bytes / drain_bytes_per_sec

200_000 lps * 800B * 86_400 ~= 13.8TB/day raw
13.8TB * 2.5 index multiplier ~= 34.5TB indexed/day
50 routes * 20 statuses * 200 tenants * 4 methods = 800_000 series

Telemetry budget and field rules

Required fields

Logs should carry service, environment, severity, event name, trace id, request id, tenant tier, route template, status, and latency. Use route templates, not raw URLs with ids.

Forbidden labels

Keep user id, order id, raw path, request body, email, token, and high-cardinality error strings out of metric labels. Put them in sampled logs after privacy redaction.

Series budget per service

Budget time series before launch. Example: route × status × method × region × version may be acceptable; adding tenant_id to every metric can multiply storage, query latency, and alert evaluation cost.

Tracing and context propagation

Propagation

Carry context across gateways, service meshes, queues, and worker pools using standard carriers: typically a traceparent header so hops stay linked without custom IDs per backend.

Instrumentation and backends

Standardized instrumentation can feed multiple tracing backends from one codebase: prefer that over ad hoc tracing headers per language.

Metrics stacks

Types

Counter: monotonic totals
Gauge: point-in-time level
Histogram: latency buckets for percentiles server-side

Query sketches

Example expressions you will see in dashboards and alerts: adapt syntax to the metrics backend.

rate(http_requests_total[5m])

histogram_quantile(0.99,
  rate(http_duration_seconds_bucket[5m]))

RED, USE, golden signals

Rate, Errors, Duration: per request-serving component.

Utilization, Saturation, Errors: per resource (CPU, disk, network interface).

Four golden signals

Latency, traffic, errors, saturation: simple service health shorthand.

SLIs, SLOs, error budgets

Measured ratio reflecting user-perceived goodness.

Target over window; is remainder for planned risk. For a request-based SLO of 99.9% success across one million eligible requests, the error budget permits 1,000 failed requests in that window. Define the eligible requests and what counts as success before calculating it. A low-volume endpoint and a high-volume endpoint should not accidentally hide each other’s customer impact behind one aggregate percentage.

SLA

Service Level Agreement: customer-facing contract, often looser than internal SLO.

Alerting

Burn-rate alerts

Burn rate compares the observed bad-request fraction with the fraction allowed by the SLO. Under a 99.9% success objective, a 1% error rate burns budget at 10 times the sustainable rate. A short window detects urgent incidents quickly, while a longer window confirms that the signal persists. Connect the page to a user-facing symptom and a concrete action; one brief CPU spike does not necessarily justify waking someone.

Anomaly detection

Seasonal traffic breaks static fences: ML monitors help capacity teams when baselines swing wildly.

yaml

# Illustrative multi-burn alert idea: tune windows to your SLO math
expr: fast_error_budget_burn > threshold

Debugging path

Start from SLO burn or a user-impact alert, not a random noisy graph.
Inspect service metrics for rate, errors, duration, saturation, deploy version, and dependency health.
Jump to trace from the bad latency or error bucket.
Use the trace id to filter structured logs and identify the failing component or request path.
Rollback, shed traffic, or fix config, then validate recovery on the same SLO dashboard.

Good page vs ticket rule

Page only when the alert shows user impact, owner, likely action, rollback path, and runbook. Create tickets for slow-burn capacity or cleanup signals.

Anti-patterns

Dashboard theater

Beautiful charts that no one uses during incidents are decoration. Every critical graph should answer a runbook question.

Alerting on symptoms and causes equally

Page on user-visible SLO burn; ticket causes like disk growth unless they are immediately dangerous.

Unbounded labels

User IDs, URLs with raw IDs, or request bodies as metric labels can explode cost and degrade query performance.

Logging secrets

Tokens, passwords, personally identifiable information (PII), and payment data require redaction at source plus retention and access controls.

How to talk about this in an interview

Selection: metrics for alerting and trends, logs for event detail, traces for cross-service latency. Do not claim one replaces the others.

Tradeoffs: cardinality, sampling, hot vs cold retention, backend portability, privacy, and agent overhead.

Failure modes: telemetry pipeline lag, dropped spans during incidents, high-cardinality outages, and missing trace context over queues.

Rollout: for millions of users, instrument golden paths first, define SLOs, add burn-rate alerts, then require dashboards and runbooks for new services before broad launch.

Production deep dive: a correlated telemetry system

Treat telemetry as a queryable data product. Metrics detect a user-impact change, an exemplar selects a representative trace, and the trace/span IDs retrieve the precise structured logs, without putting unbounded identities in metric labels.

Head vs tail sampling

Head sampling is cheap and immediate. Tail sampling can retain errors, slow paths, or chosen tenants after outcome is known, but collectors must buffer and assemble complete traces.

Exemplars and cardinality

Attach trace IDs to selected histogram buckets. Keep route, status class, region, and bounded version labels; move user, order, and request IDs to trace/log attributes.

Incident workflow

Page on multi-window SLO burn → confirm affected region/version → open a slow/error exemplar → inspect dependency spans → pivot to redacted logs → link the finding in the incident timeline.

plaintext

# Tail-sampling capacity worksheet
incoming_spans_per_sec = 18_000
average_span_bytes = 900
decision_wait_sec = 12

raw_buffer = incoming_spans_per_sec * average_span_bytes * decision_wait_sec
           = 194_400_000 bytes  # ~185 MiB before indexes and headroom

# Example policy order
keep(error_or_timeout) = 100%
keep(latency_ms > route_p99_target) = 100%
keep(debug_tenant_allowlist) = 100%  # short-lived and audited
keep(other) = 2%

Checkout dashboard that answers a decision

Valid-event success SLI and fast/slow burn rates.
p50/p95/p99 by region, route, and release, not averages alone.
Admission/retry volume, dependency latency, queue age, and pool wait.
Top bounded error codes with deploy and feature-flag annotations.

Observe the observability pipeline

Export failure, dropped spans/logs, local buffer fill, and collector queue age.
Trace completeness, sampling decision latency, series count, and ingestion cost.
Redaction failures and forbidden-field detection block unsafe telemetry at source.
Telemetry loss never blocks application traffic; it raises its own visible alert.

Failure timeline: cardinality, sampling bias, missing trace context

T+0      Engineer adds label user_id to http_requests_total "for debugging"
T+10m    Series explode (routes × status × user_id); scrape/query latency ↑
T+25m    Tail sampler keeps only errors; rare slow-success paths vanish from traces
T+40m    Checkout p99 burns SLO, but spans lack baggage / traceparent across hop
T+55m    Logs cannot join metrics → incident without a pivot id
Outcome  Cardinality + biased sampling + broken context = blind observability

Watch the watchers: series count per service, scrape duration, dropped spans/logs, collector queue age, sampling decision rates, and burn-rate alert freshness. Telemetry loss must raise its own visible alert without blocking app traffic.

Exercises

Write the checkout SLI as good_events / valid_events. Which status codes belong in the denominator? Which belong only in logs?
Budget 50k series for one service. Propose a label set (route, status class, region, version, …) and show the product stays under budget.
Given a p99 spike with an exemplar trace id, list the exact pivots: metric bucket → trace → structured logs. Where does the chain break if queues drop context?

Primary references

Topic sources for telemetry and SLI-driven monitoring, not generic system-design books.

OpenTelemetry: Concepts Google SRE: Monitoring Distributed Systems OpenMetrics: Specification

Takeaway

Metrics page on SLI burn; traces locate the hop; logs hold the event: linked by one id. Bound cardinality like capacity. If an alert lacks a runbook action, it is noise.

Practice and further reading

Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.

Focus for this lesson: Correlate logs, metrics, traces, and exemplars through OpenTelemetry while budgeting sampling, cardinality, retention, privacy, and incident usability.

Explain the decision

When is tail sampling preferable to head sampling?

Read the reasoning

When rare errors or slow traces must be retained after their outcome is known; it costs collector buffering and requires trace-wide decisions.

Apply it to a production scenario

During an incident, traces disappear exactly when traffic peaks. What do you fix?

Read one approach

Decouple telemetry with bounded local buffers, reserve collector capacity, shed low-value spans first, alert on export loss, and retain unsampled RED metrics.

Work through the numbers

A service emits 18,000 spans/s at 900 bytes each. What daily raw volume and 5% sampled volume result?

Read the worked calculation

About 1.40 TB/day raw and 70 GB/day at 5%, before compression, indexes, replicas, and tail-sampling buffers.

Reason through a failure

What must telemetry degradation preserve?

Read the recovery reasoning

Loss of observability must never block the request path, and the system must expose dropped telemetry so missing evidence is not mistaken for health.

Sources and further study

Continue with Leader Election.