System Design · Performance

Performance Profiling Methods

Checkout p99 jumped after a deploy. CPU is at 40%. Scaling pods will not help if the time is spent waiting on a lock or allocating JSON trees. Profiling is how you find the real cost center before you rewrite the wrong layer.

A builds a of or . Pair that with and profiles, and use so the next incident already has a baseline.

Profile journey

Name the symptom

p99, error rate, or CPU burn—pick one golden signal.

CPU, wall, alloc, or lock—not guesswork.

Find the wide plateau on the user path.

Prove the fix moved the same metric.

What actually happens

Metrics tell you that something is wrong. Traces show which request path was slow. A profile answers which code inside that process burned CPU, allocated, or waited. Without profiles, teams “optimize” by rewriting the wrong service or adding caches that never hit the .

is the default: every few milliseconds, record the stack of running (or, for wall profiles, interesting) threads. Aggregate millions of samples into a flame graph. Rare micro-spikes need longer windows or a brief higher sample rate—not a permanent instrumentation tax.

When CPU is low but latency is high, switch to and contention profiles. When GC or RSS climbs, start with allocation profiles. Interview answer: “I pick the profile type from the symptom, not from the tool I know.”

How to choose a profiling method

1. Is the core busy? High CPU → . Low CPU, high latency → wall / off-CPU / locks.

2. Is memory or GC the story? Use an before you tune GC flags in the dark.

3. Do you need exact counts? Sampling first; targeted only on the named hotspot, watching .

Simple rule of thumb

Sample under the same load shape that hurts users. Read the widest user-path plateau. Fix one thing. Diff the profile against the baseline and confirm the golden percentile—not a microbenchmark alone.

The profiling loop

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

flowchart TD
    Symptom["SLO / p99 / CPU alert"] --> Choose["Choose signal: CPU, wall, alloc, lock"]
    Choose --> Sample["Sample under load window"]
    Sample --> Flame["Flame graph / top stacks"]
    Flame --> Hot{"Wide hotspot on user path?"}
    Hot -->|No| Off["Check off-CPU / I/O / queue"]
    Hot -->|Yes| Fix["Change code or config"]
    Off --> Fix
    Fix --> Diff["Differential profile vs baseline"]
    Diff --> Verify["Golden metric recovered?"]
    Verify -->|No| Choose
    Verify -->|Yes| Done["Document + keep continuous profile"]

plaintext

Checklist before you attach a profiler:
  1. Which SLO / percentile / host is bad?
  2. Reproduce or catch continuous profile for that window
  3. Prefer production-like load (or continuous profiling store)
  4. Capture ≥30–60s under load (longer for rare tails)
  5. Record build SHA, flags, GOMAXPROCS / heap size
  6. After fix: same window length, differential view

Anti-patterns:
  - Profiling an idle canary
  - Optimizing a background job that is not on the SLO path
  - Trusting a 10-second sample for a 1-in-10k spike

On-CPU vs off-CPU

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

flowchart LR
    Req["Request"] --> Run["On-CPU work"]
    Req --> Wait["Off-CPU wait"]
    Run --> Parse["Parse / compute / serialize"]
    Wait --> IO["Disk / net / DB"]
    Wait --> Lock["Mutex / pool"]
    Wait --> Sched["Runnable queue"]
    Parse --> CPUProf["CPU / sampling profile"]
    IO --> Wall["Wall / off-CPU profile"]
    Lock --> Cont["Contention profile"]
    Sched --> Wall

Symptom	Start with	Often finds
High CPU, rising cost	CPU sampling + flame graph	Regex, JSON, crypto, N+1 pure compute
Low CPU, high p99	Wall / off-CPU / traces	Slow DB, DNS, disk, GC pause, queue
Threads blocked	Contention / lock profile	Global mutex, pool starvation
GC / RSS pressure	Allocation profile	Hot alloc sites, retained caches

Reading a flame graph

In a classic ICP (inverted call pyramid) flame graph, the bottom frame is the root (often main or a thread start). Frames above are callees. A plateau that is 40% wide means roughly 40% of samples landed in that subtree. Flip icicle charts top-down if your tool defaults that way—same idea.

Common “gotchas”: inlined frames missing in some runtimes; JIT renaming; samples during GC safepoints; attributing time to a framework dispatcher that only relays to your handler. Drill until you see your package names dominating.

plaintext

Mental model:
  samples_in_frame / total_samples ≈ fraction of CPU (or wall) time

Example story:
  55%  json.Marshal  ← hotspot
  20%  db.driver.Query  ← also check off-CPU for wait
  10%  crypto/tls
   8%  business handlers
   7%  other

Interview move:
  “I would open the flame graph, find the widest unexpected plateau on the
   request path, confirm with a differential profile after the change.”

Sampling vs instrumentation

Sampling profilers

pprof (Go), async-profiler / JFR (JVM), perf / eBPF (Linux), py-spy, and continuous agents (Parca, Pyroscope, Datadog, Google Cloud Profiler) all lean on sampling. Typical production rates: tens to a few hundred Hz per core equivalent.

plaintext

Go (CPU):
  go tool pprof -http=:8080 https://host/debug/pprof/profile?seconds=30

Go (heap / allocs):
  go tool pprof https://host/debug/pprof/heap
  go tool pprof https://host/debug/pprof/allocs

Linux (on-CPU):
  perf record -F 99 -g -- sleep 30
  perf script | stackcollapse-perf.pl | flamegraph.pl > out.svg

JVM:
  async-profiler: -e cpu / -e alloc / -e lock
  JFR: profile settings with low overhead continuous recording

Instrumentation profilers

Exact call graphs and timings for instrumented methods. Useful in local microbenchmarks or after sampling named a package. Avoid leaving method-level instrumentation on in production at high QPS—the can dominate the profile and inflate latency you are trying to fix.

Continuous profiling

Always-on low-rate profiles turn “we cannot reproduce” into a queryable history. Store by service, version, and zone. Redact labels that embed user data. Use differentials: bad canary vs baseline, or this deploy vs previous SHA. Continuous profiling complements traces; it does not replace request-scoped causality.

Allocation and contention profiles

Allocation profiles show stacks that create objects or bytes. A handler that builds huge intermediate trees will show up even if each allocation is tiny. Fix: reuse buffers, stream, avoid accidental copies, move large caches off-heap or out-of-process.

Contention profiles highlight waiters on a mutex. Classic designs: one process-wide map guarded by a lock; a connection pool of size 1; synchronized logging on the request path. Remedies: shard, lock-free read paths, enlarge pools, move work off the critical section.

plaintext

Worked story — “CPU fine, p99 800 ms”:
  CPU flame: thin
  Wall profile: wide plateau in mutex.Lock / park
  Contention: sync.Mutex on global feature-flag map
  Fix: atomic snapshot / RCU-style reload every 5s
  Diff: lock wait samples collapse; p99 → 90 ms

Worked story — “GC thrash”:
  CPU shows runtime.gc*
  Alloc profile: 40% in json.Unmarshal of large payloads
  Fix: protobuf or streaming decoder; size limits
  Diff: alloc/s down; GC CPU and p99 recover

Worked example: deploy regressed p99

Version B adds “helpful” debug logging that stringifies every request body. Average latency barely moves; p99 doubles at peak.

plaintext

Evidence path:
  1. Histogram p99 for route /checkout worsens after SHA B
  2. Continuous CPU profile: log/slog or fmt.Sprintf widens
  3. Alloc profile: []byte growth in logging path
  4. Trace exemplar: slow spans sit in “log fields”
  5. Fix: sample logs, structured fields without full body, async logger
  6. Differential profile: logging plateau shrinks
  7. Same golden histogram recovers

Why averages lied:
  Most requests are small; the tail has large bodies.
  Profiling under peak mix catches what a tiny load test missed.

Trade-offs

Choice	Gain	Cost
Sampling profiler	Low overhead, finds hotspots	Statistical; may miss rare spikes
Instrumentation	Exact counts for a region	High overhead if global
CPU-only view	Simple, cheap	Misses I/O and lock waits
Wall / off-CPU	Matches user latency	Noisier; needs good filters
Continuous profiling	History + differentials	Storage, privacy, agent cost
Local repro only	Safe iteration	Misses prod mix and data shape

Choosing and explaining performance profiling

A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.

Explain the core idea

Pick the profile type from the symptom: CPU, wall/off-CPU, allocation, or contention.

Sampling plus flame graphs finds hotspots; differentials prove a fix.

Profiles explain cost inside a process; metrics and traces still frame which process and request.

When it belongs

Latency or cost regressions after a deploy.

Designing observability so continuous profiles exist before the incident.

Choosing whether to optimize compute, I/O, locks, or allocations.

How it fails

Profiling idle traffic and “fixing” nothing.

Optimizing a wide plateau that is not on the SLO path.

Instrumentation overhead distorting results.

Ignoring off-CPU waits when CPU looks fine.

Trade-offs to weigh

Sample rate versus overhead and resolution.

Always-on continuous profiles versus storage and privacy.

Local microbenchmarks versus production mix.

Common misunderstandings

Reading flame height as “slowness.”

Averaging away the tail that profiling should explain.

Equating “we added a cache” with “we profiled.”

Tuning GC without an allocation profile.

Wrap-up

Profiling turns “the service is slow” into a stack you can change. Keep continuous low-overhead samples, match the profile kind to the symptom, and always verify with the same golden percentile.

Exercises

Choose the tool. CPU 15%, p99 2s, many threads in futex_wait. Which profile first, and why?
Read the graph. A flame graph shows 60% width in a JSON library under your handler. List three fixes and how you would differential-verify.
Continuous vs ad hoc. Argue when a one-shot perf record is enough and when you need always-on profiles.

Primary sources

Key takeaway

Profile the signal that matches the pain: CPU for busy cores, wall/off-CPU for waits, alloc for GC pressure, contention for lock convoys. Flame graphs show where; differentials prove you fixed it.