System Design Observability
OpenTelemetry, Sampling, Exemplars & Cardinality Control
Friday evening, checkout latency trips the SLO. The on-call needs one real slow request path, not a bill for storing every from Black Friday traffic.
standardizes metrics, logs, and traces. At scale you sample, attach , and police so the observability stack stays useful and affordable.
Debug journey
SLO burn
Latency or error budget is on fire; start from the golden signal, not a random trace.
Find the bad route or dependency in an unsampled histogram before opening traces.
Open one concrete request path linked from the hot histogram bucket.
Fix + verify
Confirm recovery on the same golden signal after the fix ships.
Try it: balance head sampling with label cardinality
Lower head sampling to cut trace volume, then attach a high-cardinality user_id label to every metric. Observability cost often tracks series count and indexed attributes, not only stored spans. This model separates the two levers.
Trace volume
~1,200 spans/s exported at 6 spans per kept trace.
Metrics cardinality
~6,000 series for 12 routes × label dimensions in this toy model.
- Traces kept/s
- 200
- Spans/s
- 1,200
- Series (est.)
- 6,000
- Cost index
- 324
200 traces/s kept · ~1,200 spans/s · ~6,000 active metric series · cost index 324. Head sampling keeps one trace per N requests. High-cardinality labels explode metric series even when traces are sampled.
Sample traces, bound labels. Tail sampling can keep errors while dropping happy paths, but unbounded label values still exhaust the metrics backend.
Redact before export. Drop or hash identifiers at the SDK; never rely on the backend to refuse PII you already emitted.
What actually happens when volume explodes
The Friday story above is the default at high QPS (queries per second): you cannot store every span. Keep nothing and rare payment failures vanish. Keep everything and collectors and backends melt under volume and from unbounded labels. The design goal is correlated evidence: unsampled golden metrics for SLOs, sampled traces for causality, and that bridge the two, without PII in metric labels.
A naive claim sounds tempting: capture 100% of traces, put user_id and raw URL on every metric, and compute error rate from sampled traces. At 40K req/s with 12 spans × 600 B you ingest ~288 MB/s (~25 TB/day) before indexing. Adding tenant_id as a metric label can create hundreds of millions of series. Sampling bias then makes dashboards lie about true error rate.
Services record aggregate metrics, structured logs, and trace spans with propagated context. Choose where trace decisions happen: head sampling reduces traffic early, while tail sampling can select errors and slow traces from the spans it actually receives. A dropped head trace cannot be restored by a later tail decision. Allowlist bounded attributes and keep personal identifiers out of metric dimensions; exemplars can link an interesting metric observation to a retained trace.
Propagate valid trace context through calls and represent asynchronous relationships with appropriate parent context or span links. Do not treat arbitrary incoming baggage as trusted telemetry attributes. Keep SLO measurements independent of trace-retention bias, and measure dropped, late, or incomplete telemetry. At 50,000 requests per second, a 1% head sample records about 500 traces per second; retaining every later error instead requires a pipeline that observes those candidates before discarding them.
How to choose sampling and labels
Once you accept that 100% capture is unaffordable, three questions decide how aggressive to sample and which labels are safe.
1. Must every request be perfect? Payments and rare admin paths may need higher sample rates or forced-on debug flags so the few bad requests are not lost to chance.
2. Do errors matter more than averages? Prefer (or always-on for failures) over blind 1% alone.
3. Is the label bounded? If values grow with users or raw URLs, drop or aggregate at the collector before they hit the metrics tier and explode .
Simple rule of thumb
Use unsampled application counters and histograms for the SLO population, then choose trace retention that preserves useful diagnostic evidence within a capacity budget. Favor errors and slow traces when the pipeline can observe them, and report what was dropped or never recorded. Exemplars link a metric observation to a retained trace; they do not turn a biased sample into a complete estimate of request error rate.
Three signals and two sampling strategies
The distinction between deciding before the outcome and deciding after observing spans is described in the OpenTelemetry sampling documentation. Choose the acquisition pipeline and capacity before promising a retention policy.
The questions above choose a policy; this is the mechanism that policy rides on. treats metrics, logs, and traces as one correlated system, and sampling decides which traces survive.
Metrics
Aggregates such as counters and stay cheap to query and are the source of truth for SLOs.
Logs
High-volume events: structure them, sample noisy paths, and correlate each line with a trace id.
Traces
Request paths built from : sample widely, but keep error and slow paths via tail policy.
Head sampling decides near the start of a trace whether to record and export it, commonly using a deterministic decision derived from the trace identifier. Parent-based propagation keeps downstream spans aligned with that decision. It saves recording and transport cost early, but a request rejected at the head cannot later be recovered merely because it became slow or failed. A 1% head sample therefore cannot promise to retain every error trace.
Tail sampling waits for spans to arrive, groups them by trace identifier, and selects traces using observed duration, errors, or approved debugging criteria. To consider every error, it must receive the relevant spans for all candidate traces; an earlier 1% drop would have removed most candidates permanently. Routing all spans of a trace to the same decision worker, bounding wait time, and handling late spans are central parts of the design.
Worked example: volume, buffer, and cardinality
For this worked design, send candidate traces to a tail-sampling tier that retains observed errors and slow requests plus a 1% probabilistic baseline of ordinary requests. At 40,000 requests per second, twelve spans per trace, and 600 bytes per span, the tier receives about 288 MB/s before overhead. The 1% selection occurs at the tail decision, not as an earlier head drop, so the collector still pays the full ingress and buffering cost.
Numbers to know
Unsampled ingress
40K × 12 × 600 B = 288 MB/s ≈ 24.9 TB/day
The 24.9 TB/day figure represents candidate span payload arriving at the tail tier, not just the retained storage. Network, decode CPU, and temporary buffers must handle that volume even if only a small fraction is exported onward. The “retain errors” goal also depends on spans arriving before the decision and capacity not dropping them. Monitor late spans, incomplete traces, and telemetry loss instead of treating the policy as an unconditional guarantee.
Baseline storage @ 1%
24.9 TB/day × 1% ≈ 249 GB/day
Error and slow-trace policies add on top. Record effective sample rate per rule so analysts avoid biased population estimates.
Tail buffer (10s wait)
288 MB/s × 10s ≈ 2.88 GB raw
A ten-second decision window at 288 MB/s requires about 2.88 GB of raw payload buffering in steady state, plus data-structure overhead, partially completed traces, and bursts. Long-running traces may outlive the window; define whether late spans attach to a retained decision, are dropped, or require longer buffering. Route consistently by trace identifier and rebalance carefully so a collector rollout does not split every in-flight trace.
Cardinality trap
100×5×4×10×50K tenants ≈ 1B series
Removing a fifty-thousand-value tenant label reduces this illustrative combination count from one billion to twenty thousand, before histogram buckets, resource labels, and additional instruments. Use route templates instead of raw paths, status classes where sufficient, and bounded service/region labels. Tenant-specific debugging belongs in access-controlled logs, traces, or deliberately bounded aggregate views rather than an unbounded dimension on every metric.
Cardinality budget
series_count ≈ product of label cardinalities. route × status × tenant × region × version can explode from thousands to millions of series (see Numbers to know above). Follow and keep on slow buckets only.
- Allowlist labels: service, route template, status class; never raw user id.
- Drop or aggregate high-cardinality labels at the collector before metrics storage.
- Attach a trace id exemplar to histogram buckets for slow requests only.
- Codify semantic-convention lint and per-service series budgets in CI.
Failure timeline, and what still breaks after the fix
The numbers above show how a bad label burns the budget; here is the same incident minute by minute, what to watch next time, and the ways a careful design can still fail.
Timeline: cardinality explosion
Release adds customer_id label
Series creation rate jumps 500× while request traffic is normal. Collector allowlist should drop the label and increment a policy-violation counter.
Metrics backend slows
Ingestion queue and memory rise from already-created series. Rate-limit the offending source; preserve golden SLO metrics; shed debug telemetry first.
Rollback completes
New series stop but backend compaction remains loaded. Keep alerting on telemetry pipeline health.
Guardrail codified
Semantic-convention lint and per-service series budgets enter CI. Document the sampling blind spot and cost impact: do not "backfill" exploded series.
ASCII sketch
bad label ──► series churn spike ──► ingest queue / OOM risk
│ │
▼ ▼
collector drop + alert protect golden SLOs
│ │
└────────── rollback ──► CI allowlist ──► budget per serviceWhat to watch
- Collector receive, export, and drop rate plus queue depth, decision latency, and memory.
- Effective sampling rate and how many traces each retention policy keeps.
- Active series, series churn, bytes per signal, and cost per service.
- Propagation coverage: a synthetic request traverses every hop with one trace id.
Still fails when…
- Missing on a queue hop leaves orphan spans sampling cannot reconnect.
- Spans for one trace_id land on different collectors, so sees an incomplete tree.
- Availability computed from retained traces biases toward errors you chose to keep; use unsampled metrics.
Trade-offs
- improves debuggability versus collector RAM and pipeline complexity.
- Higher head sample on critical routes only is simpler, but still misses rare failures elsewhere.
- Agent collectors isolate apps; gateway collectors centralize tail policy; large systems often use both.
- A single gateway pool is a telemetry blast radius: shard it and apply backpressure.
Observable and affordable
What production needs to hold
A production design needs a way to recognize trouble and a rule it preserves while recovering. A trigger is the measurement or event that starts a response. An invariant is a statement that must remain true, such as never confirming the same reservation to two customers.
Trigger metric or condition
Design the telemetry pipeline when incidents cannot be followed across services, rare failures vanish under uniform sampling, user-controlled labels create unbounded series, or observability cost grows faster than traffic.
Correctness invariant
Trace context remains consistent across every asynchronous and synchronous hop, while no untrusted PII or unbounded attribute reaches storage. Sampling decisions are measurable and do not silently invalidate SLO metrics.
Signal path from SDK to correlated evidence
SDK + context
W3C trace context
Agent collector
batch + redact
Gateway tier
tail policy + budgets
Signal stores
metrics / logs / traces
SLO → exemplar
one-click evidence
Telemetry volume and sampling budget
A platform serves 40,000 requests/second. Each trace averages 12 spans at 600 bytes after encoding. Errors are 0.2%. All candidate traces reach a tail-sampling tier, which keeps a 1% probabilistic baseline plus observed errors and requests slower than one second. No earlier head-sampling drop removes candidates from this worked path; retention is still bounded by complete delivery and the tail decision window.
Unsampled traces
40K × 12 × 600 B = 288 MB/s ≈ 24.9 TB/day
This is collector ingress before indexing and replication. Tail sampling still receives full traces, so network and buffer capacity differ from stored volume.
Baseline storage
24.9 TB/day × 1% ≈ 249 GB/day
Error and slow-trace policies add to this. Record the effective sample rate per rule so analysts can avoid biased population estimates.
Tail buffer
288 MB/s × 10s ≈ 2.88 GB
A ten-second decision wait needs roughly this raw memory across the gateway tier plus overhead and burst headroom; shard consistently by trace_id.
Cardinality trap
100 routes × 5 statuses × 4 regions × 10 versions × 50K tenants = 1B series
Remove tenant_id and the same metric has 20,000 combinations. Investigate individual tenants in traces/logs or bounded top-N views.
Decision
SLOs come from unsampled counters/histograms. Exemplars attach selected trace IDs to latency buckets. Gateway collectors redact email/payment attributes, cap dimensions, and route debug traces to short retention while aggregates remain longer.
How the mechanism holds
Follow how each part protects the rule above. The mechanism explains what the system does; the reasoning explains why that action is sufficient under the stated assumptions. If an assumption changes, the guarantee may change too.
Context propagation
Propagate traceparent/tracestate plus approved baggage over HTTP, gRPC, queues, and scheduled jobs. Create producer/consumer spans and link fan-out work when one strict parent is misleading.
Why it holds: A synthetic request traverses every hop with one trace and no orphan-span spike.
Head vs tail sampling
Head sampling is cheap and predictable but misses unknown rare failures. Tail sampling retains errors/slow or policy-matched traces after buffering, at the cost of complete trace routing, memory, and decision delay.
Why it holds: Sampling-rule metrics reconcile received, decided, dropped, and exported traces.
Sampling bias and exemplars
Never compute request-rate SLOs from sampled traces. Use unsampled metrics, expose rule/effective probability, and use exemplars to pivot from a histogram bucket to representative traces.
Why it holds: Dashboard totals match edge counters while exemplars resolve to retained traces.
PII and cardinality budgets
Allowlist attributes, template routes, hash only when re-identification policy permits, redact in SDK and collector, and reject labels such as user_id, raw URL, or exception message.
Why it holds: CI semantic checks and collector drop counters detect budget violations before backend OOM.
Failure timeline: a new label floods metrics
Release adds customer_id
Series creation rate jumps 500× while request traffic is normal.
Response: Collector allowlist drops the label and increments a policy-violation counter.
Metrics backend slows
Ingestion queue and memory rise from already-created series.
Response: Rate-limit offending source, preserve golden SLO metrics, and shed debug telemetry first.
Rollback completes
New series stop but backend compaction remains loaded.
Response: Keep alerting on telemetry pipeline health and extend query timeouts only for non-paging dashboards.
Guardrail is codified
Semantic-convention lint and per-service series budgets enter CI.
Response: Backfill no data; document sampling blind spot and cost impact.
How to choose among alternatives
Agent vs gateway collector
Choose when: Agents isolate apps and batch locally; gateways centralize tail sampling, routing, and credentials. Large systems commonly use both.
Caution: A single gateway pool is a telemetry blast radius and must be sharded and backpressured.
Sampling vs retention
Choose when: Sample broad low-value traffic and keep errors; tier retention by operational value.
Caution: Short retention can make slow regressions impossible to compare; full capture can bankrupt storage.
Logs vs span events
Choose when: Use spans for request causality and structured logs for detailed domain/operator evidence.
Caution: Duplicating every payload in both doubles cost and privacy exposure.
Running it day to day
Measurements tell you what is happening; alerts tell an operator when action is needed. A rollout plan limits how many users encounter a change before it is checked. A runbook records the steps to diagnose and recover from a known failure.
Measure
- collector receive/export/drop rate, queue, CPU, memory, and decision latency
- effective sampling rate and retained traces by policy
- active series, series churn, bytes per signal, and cost per service
Alert
- golden telemetry or trace propagation coverage drops
- collector queue risks data loss
- cardinality or PII-policy violation exceeds budget
Roll out safely
- publish semantic attribute allowlists
- load-test collectors at burst plus backend outage
- canary sampling rules and compare retained incident traces
Runbook
- protect SLO metrics and security audit first
- drop/debug-sample low-value signals and isolate offender
- restore export, quantify gaps, and tune budgets with owners
Practice scenario
Design an OTel pipeline for 40K requests/second that keeps rare payment failures without storing every trace.
Questions worth asking next
- Where does tail sampling happen?
- How much memory does it need?
- Why not put tenant_id on metrics?
A solid answer sketch
Use SDK propagation, agent batching/redaction, and trace-id-sharded gateway collectors. Keep a 1% head baseline plus tail rules for errors/slow payments, but calculate SLOs from unsampled metrics. Allowlist bounded labels and use exemplars to reach retained traces.
Common mistakes
- • treating sampled traces as an unbiased request counter
- • tail sampling after spans have been routed to different collectors
- • putting raw URLs, users, or exception text in metric labels
Authoritative references
Last reviewed: 6 August 2026
Choosing and explaining OpenTelemetry and sampling
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
Propagate trace context on every outbound call.
Sample for cost and prioritize observed errors and SLO violations; report the coverage lost through early sampling or pipeline drops.
Cardinality is a capacity plan: one bad label can OOM the metrics tier.
When it belongs
Microservices need cross-service debugging.
SLO dashboards must tie to exemplar traces.
Telemetry cost threatens observability budget.
How it fails
Broken context propagation → orphan spans.
100% trace capture at high QPS.
user_id label on every metric.
Trade-offs to weigh
Tail sampling improves debuggability vs collector RAM.
Structured logs cost storage vs grepability.
Vendor-agnostic OTel vs integrated proprietary agents.
Common misunderstandings
Logging PII into traces.
Alerting on every metric threshold without SLO framing.
No retention tier: hot vs cold storage.
Wrap-up
You have seen the volume math, the sampling choices, and the cardinality failure path. Use these exercises to check the ideas stuck, then follow the references for the primary sources.
Exercises
- Pipeline at 40K req/s. Design an OTel pipeline that keeps rare payment failures without storing every trace. Where does run, and how much buffer memory do you need for a 10s decision window (see Numbers to know above)?
- tenant_id on histograms. A teammate wants
tenant_idon every HTTP latency histogram. Compute the series growth and propose an alternative for debugging one tenant. - SLO from samples. Explain why an SLO dashboard must not use sampled traces as the request counter. What signal and path should on-call use instead?
Topic-specific references
Key takeaway
OTel unifies instrumentation; sampling and discipline make it affordable. Debug path: SLO burn → golden signal → exemplar trace → trace id in logs → fix → verify recovery metric.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Propagate context, size collector tiers, choose head/tail sampling, control bias, attach exemplars, redact PII, budget cardinality, and model cost.
Explain the decision
What bias does uniform head sampling introduce?
Read the reasoning
Rare errors may be dropped before outcome is known; tail sampling can retain error/slow traces but needs complete-trace buffering and capacity.
Apply it to a production scenario
A user_id metric label creates millions of series. What is the repair?
Read one approach
Remove it from metrics, use traces/logs for per-user investigation, enforce label allowlists/budgets, and expire the cardinality explosion safely.
Work through the numbers
Thirty thousand spans/s at 700 bytes with 10-second tail buffer requires what raw memory?
Read the worked calculation
About 210 MB before trace indexes, metadata, replication, and burst headroom.
Reason through a failure
What must context propagation prevent?
Read the recovery reasoning
Untrusted external trace fields cannot inject unlimited baggage or sensitive labels, and async boundaries preserve valid trace/causal linkage.
Sources and further study
Continue with Distributed System Testing.