System Design Observability
Observability Concepts
Checkout fails for one percent of users in one region. CPU graphs are green. Without a coherent signal model, on-call guesses which hop broke. With one, they follow burn → golden metric → exemplar trace → logs—and fix the right dependency.
This note frames as the / / triad guided by / / . It complements deeper logging, OpenTelemetry sampling, and SLO notes rather than replacing them.
Debug path
User impact pages on-call.
Which service, route, region?
One concrete request path.
Exact failure detail.
What actually happens
Monitoring used to mean “uptime check + CPU graph.” That fails the moment a dependency hangs while your process stays up. Naive stacks also drown in unstructured logs, alert on every spike, and store every span until the bill or the backend melts. The problem is not a missing tool; it is a missing model of what questions production must answer.
is that model: export enough correlated signal to diagnose novel failures. The triad is complementary. are cheap aggregates for trends and pages. show causal structure across hops. hold high-cardinality detail for a single incident. Without , they are three disconnected piles of data.
Reliability framing keeps the triad honest. An measures user-visible goodness; an sets the target; the decides whether to ship or freeze. Alert from budget burn, investigate with metrics, drill with exemplars and traces, confirm with logs, verify recovery on the same SLI—not on a random host graph.
Cost is a first-class constraint. explosions and unsampled trace floods make observability itself an outage. Sampling, retention tiers, and label allowlists are design choices, covered in depth in OpenTelemetry and logging notes; here we keep the conceptual map.
How to choose what to instrument
1. Would a user notice? Prefer journey success and latency over host idle time. Start with one critical path SLI.
2. Is the label bounded? If values grow with users or raw URLs, keep them out of metric dimensions—use logs/traces instead.
3. Can you walk burn → exemplar → logs? If any hop lacks correlation ids, fix propagation before adding more dashboards.
Simple rule of thumb
Instrument on edges and on scarce resources. Define one journey SLO before twenty vanity panels. Keep metrics unsampled for SLIs; sample traces; structure logs; never put user ids on every histogram.
Triad meets SLO framing
Signals without reliability goals become decorative. Goals without signals become arguments. The useful system connects both.
Read the diagram definition
flowchart LR
subgraph signals [Telemetry triad]
M[Metrics]
L[Logs]
T[Traces]
end
subgraph framing [Reliability framing]
SLI[SLI]
SLO[SLO]
EB[Error budget]
end
M --> SLI
SLI --> SLO --> EB
EB -->|burn alerts| OnCall[On-call]
OnCall --> M
M -->|exemplar| T
T -->|trace_id| L
The three signals—roles, not rivals
Population truth for rates, errors, and latency histograms. Drive burn alerts and capacity plans. Keep label sets allowlisted. Attach so a bad bucket opens a trace.
Causality across services, queues, and DBs. Expensive at full fidelity—sample with bias toward errors and slow paths (see OTel sampling notes). Useless if context dies at a gateway or worker.
High-detail evidence: error codes, payloads (redacted), feature flags. Structure fields; include trace_id; sample chatty success paths. Logs are not a substitute for metrics when paging.
Incident journey in one picture
This is the operational contract your instrumentation should enable—independent of vendor.
Read the diagram definition
sequenceDiagram
participant U as User journey
participant M as Metrics / SLO
participant OC as On-call
participant X as Exemplar
participant T as Trace UI
participant L as Structured logs
U->>M: Error or latency burn
M->>OC: Page on burn rate
OC->>M: Find bad route / region
M->>X: Open exemplar trace id
X->>T: Inspect slow / failing span
T->>L: Grep same trace_id
L-->>OC: Root cause detail
RED, USE, and golden signals
Google's four golden signals (latency, traffic, errors, saturation) and the / mnemonics are checklists so you do not invent a unique dashboard religion per service. Request-serving components need RED; resources and pools need USE. Saturation—queue depth, thread pool wait, connection checkout time—often moves before hard errors.
Golden signals still need SLO framing. A latency graph without a user journey definition invites debates about “is p99 300ms bad?” Write the SLI: which requests count, what “good” means, which window. Then the same metrics feed both dashboards and budget burn.
| Lens | Applies to | Example |
|---|---|---|
| RED | Request services | checkout QPS, 5xx ratio, p99 latency |
| USE | CPU, disk, pools | CPU util, run-queue, disk wait, pool wait |
| SLI/SLO | User journeys | % checkouts succeeding under 300ms / 30d |
Correlation, cardinality, and cost
is the difference between a trace UI and folklore. Propagate context on RPC, HTTP, and queue payloads. Stamp the same ids on log lines. Exemplars bridge metric buckets to traces without stuffing tenant ids into labels.
is the difference between an affordable metrics tier and a pager about the observability stack. series_count roughly multiplies label values. Route templates beat raw URLs; status classes beat every code; bounded service/region labels beat user ids. High-cardinality debugging belongs in traces and access-controlled logs.
Retention tiers close the loop: hot metrics for pages, short hot traces for incidents, cold storage for audit. Sampling policy belongs with OpenTelemetry notes; the concept here is that 100% of everything is rarely a strategy—it is a future outage.
How this complements sibling notes
| Note | Goes deep on | This note's role |
|---|---|---|
| Logging & monitoring | Pipelines, structure, RED/USE practice | Places logs in the triad + SLO path |
| OpenTelemetry & sampling | Head/tail sampling, exemplars, cardinality math | Why sampling exists in the concept model |
| SLI/SLO/error budgets | Budget math, burn alerts, freeze rules | How SLOs steer which signals matter |
Failure timeline: pretty but blind
T0 Host CPU dashboards green; no journey SLO; traces sampled 0.1% head-only
T1 Payment dependency slow; API returns 200 with empty body after soft timeout
T2 Error-rate metrics look fine; cart abandonment spikes; support floods
T3 On-call has no exemplar; rare failure traces were never kept
T4 Fix: checkout SLI/SLO + burn alert, unsampled success metrics, tail-keep errors,
readiness that fails soft-timeouts, trace_id on logs
Outcome Observability without user SLIs measures the wrong system
Break the improved design
Perfect SLOs still fail if the SLI excludes timeouts users waited through, or if multi-window burn is misconfigured into alert spam. Exemplars help only when the linked trace was retained. Structured logs with secrets still create compliance incidents.
Vendor lock-in and three overlapping agents can recreate cardinality chaos under a prettier UI. Treat telemetry pipelines as production systems: SLOs for the observability stack itself (ingest lag, drop rate, query latency).
Trade-offs
Debuggability vs telemetry cost
More spans and verbose logs shorten MTTR until cardinality and storage dominate. Prefer biased retention (errors, slow, allowlist) over 100% forever. Budget telemetry like any other capacity.
Symptom (SLO) alerts vs cause alerts
Burn alerts catch user pain; cause alerts (disk full) catch known precursors. Too many cause alerts recreate pager fatigue. Lead with symptoms; keep a few high-precision cause pages.
Vendor suite vs open instrumentation
Integrated suites are fast to start. OpenTelemetry-style instrumentation preserves optionality and forces cleaner semantic conventions. Either way, cardinality policy travels with you.
Forensic detail vs privacy
Full payloads debug faster and leak PII. Redact at source, minimize retention, and gate access. Observability that violates privacy is a product risk, not a virtue.
Wide instrumentation vs focused golden paths
Instrumenting every internal function creates noise. Start at user edges and scarce resources; deepen where burn and incidents prove need. Breadth without correlation is expensive folklore.
Choosing and explaining observability
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
I instrument the triad—metrics, traces, logs—correlated by context ids.
SLIs/SLOs and error budgets decide what pages; exemplars connect burn to a real trace.
Cardinality and sampling are capacity plans so telemetry stays affordable.
When it belongs
Any multi-service design needs an incident story.
The interviewer asks how you know the system is healthy.
You must justify metrics vs logs vs traces without hand-waving.
How it fails
Green host metrics while journeys fail.
Broken propagation → orphan spans.
Unbounded labels melting the metrics tier.
Trade-offs to weigh
Debuggability versus cost and privacy.
Symptom burn alerts versus noisy cause alerts.
Vendor speed versus portable instrumentation.
Common misunderstandings
Deriving SLOs from sampled traces.
Alerting on every metric without a user journey.
Treating more dashboards as more observability.
Wrap-up
Observability is a correlated triad steered by SLOs. Use sibling notes for pipeline, sampling, and budget math depth; use this map to keep them consistent.
Exercises
- Journey SLI. Pick login or checkout. Write numerator, denominator, and one burn-alert policy. Which triad signals support each step of the debug path?
- Cardinality review. A teammate wants user_id on every RED metric. Explain the failure mode and propose an exemplar/log alternative.
- Missing hop. Traces break at a queue worker. What propagation and log fields do you require before calling the system observable?
Topic-specific references
Key takeaway
Observability = correlated triad + SLO framing + cost discipline. Debug path: budget burn → golden metric → exemplar trace → logs → fix → verify the same SLI.