System Design · Performance
Latency Optimization Techniques
Latency vs throughput explained what to measure. This note is about how to shrink the slow tail: budget the , cut , and stop from turning rare stragglers into common bad pages.
Target and with caches, locality, coalescing, hedging, deadlines, and shedding—each chosen because it attacks a specific part of the wait.
Tail-cut journey
Assign ms to each hop; find the overspender.
Headroom, faster workers, or shed.
Shorten the path
Cache, colocate, async the rest.
Hedge, cancel, degrade optional fan-out.
What actually happens
End-to-end latency is queueing delay plus service time along the critical path. Throughput optimizations (batching, compression) often add wait for the first byte—fine for pipelines, wrong for a 200 ms page SLO. Here we optimize percentiles the user feels, especially p99.
Tails come from variance: GC pauses, noisy neighbors, slow disks, lock waits, cold caches, cross-AZ hops, and retries that amplify load. makes a 1% slow dependency into a frequent slow page. Techniques either remove variance (cache hits, colocation), hide it (hedging), or refuse to wait (deadlines, degradation, shedding).
Interview framing: “I would write a latency budget, measure each hop with histograms, then apply the cheapest technique that attacks the overspending hop—not a random cache.”
How to choose a technique
1. Is the time in queue? Raise capacity, cut work per request, or —do not micro-optimize a handler that never runs.
2. Is a hop on the critical path optional? Make it async or degrade; keep the page fast.
3. Is variance the issue? Cache, colocate, coalesce, hedge, or fix the slow leaf with profiles and EXPLAIN.
Simple rule of thumb
Budget → measure → attack the largest overspend. Prefer removing work from the critical path over clever retries. Bound every hedge and retry by the parent deadline.
Critical path and latency budget
Read the diagram definition
flowchart TD
User["User click"] --> Edge["Edge / CDN"]
Edge --> API["API gateway"]
API --> Auth["Auth"]
API --> Core["Core handler"]
Core --> Cache["Cache lookup"]
Cache -->|miss| DB["Primary read"]
Cache -->|hit| Merge["Merge response"]
DB --> Merge
Core -.->|async| Analytics["Analytics enqueue"]
Core -.->|optional| Rec["Recommendations"]
Rec --> Merge
Merge --> User
Analytics -.->|not on critical path| Sink["Worker"]Example budget for p99 ≤ 300 ms page:
Edge / TLS resume ........ 40 ms
Auth ..................... 20 ms
Handler + validation ..... 30 ms
Cache get ................ 5 ms (hit path)
Dependency A (required) .. 80 ms
Dependency B (optional) .. 50 ms ← degrade if over budget
Merge + serialize ........ 20 ms
Slack .................... 55 ms
────────────────────────────────
Total target ............. 300 ms
Rules:
- Optional deps get a tight timeout; never block the SLO
- Propagate remaining deadline downstream
- Revisit budget when you add a hopTail toolkit
Read the diagram definition
flowchart LR
Tail["High p99"] --> Q["Cut queueing"]
Tail --> C["Shorten critical path"]
Tail --> S["Tame stragglers"]
Q --> Q1["Headroom + sheds"]
Q --> Q2["Faster workers"]
C --> C1["Cache / colocate"]
C --> C2["Async non-critical"]
S --> S1["Hedging / backup"]
S --> S2["Deadlines / cancel"]
S --> S3["Fix slow leaf"]| Technique | Attacks | Watch out |
|---|---|---|
| Caching tiers | Slow storage / compute | Stale data, stampede |
| Colocation | Network RTT | Blast radius, imbalance |
| Coalescing / singleflight | Thundering herds | Latency coupling of waiters |
| Hedging | Stragglers | Duplicate load |
| Deadlines + cancel | Wasted work | Too-tight budgets → errors |
| Load shedding | Saturation queues | Fairness / which traffic dies |
| Async + degrade | Non-critical path length | Eventual consistency UX |
Caching, coalescing, and locality
belongs on the critical path only when hit rate and freshness requirements match. In-process L1 removes network entirely for ultra-hot keys; distributed L2 shares across pods; CDN/edge helps read-mostly public responses. Pair with so a miss storm does not amplify origin load and create a worse tail.
is underrated in interviews: a 2 ms same-AZ Redis get can become 30–80 ms cross-region. Pin data near compute for chatty paths; replicate read-only data to each region rather than calling “home” on every request.
Stampede pattern:
without coalescing:
500 pods × cache expire → 500 origin queries at once → DB p99 explodes
with singleflight / request collapsing:
500 waiters × 1 origin query → fan-in result
Locality checklist:
- App and primary DB in same AZ for write path?
- Cross-AZ only for HA failover, not every read?
- Regional caches populated, not empty on cold start?
- Connection pools warmed to avoid handshake tails?Hedging, deadlines, and fan-out
Hedged requests
If P(slow) is small but painful, send a second request after a percentile-based delay (for example after the expected p95 wait). Cancel the loser. Cap concurrent hedges cluster-wide. Idempotent reads only unless you have dedupe.
Pseudo:
primary = start(rpc)
sleep(hedge_after) // e.g. observed p95
if primary not done and budget remains:
backup = start(rpc)
return first_success(primary, backup)
cancel the other
Anti-patterns:
- Hedge immediately on every call (2× load always)
- Hedge non-idempotent payments
- Hedge after parent deadline expiredDeadline propagation
Fan-out amplification math
Independent calls, each slow with probability p:
P(page slow) ≈ 1 - (1-p)^n
n=20, p=0.01 → ≈ 18% of pages see at least one slow call
n=50, p=0.01 → ≈ 39%
Mitigations:
- Fewer critical calls (compose backends)
- Optional parallel with short timeout + empty fallback
- Hedging only the flakiest dependency
- Bound concurrency; prefer batched APIsQueueing, utilization, and shedding
When arrivals outrun service rate, dominates. Speeding the handler by 10% helps; shedding the overload helps more. Use concurrency limits, admission control, and priority: interactive user traffic over batch scrapers.
Retries without jitter and budgets are a tail factory: they re-inject load exactly when the system is sick. Prefer idempotent retries with exponential backoff, capped attempts, and respect for .
Little's Law reminder (steady state):
L = λ × W
concurrency ≈ arrival_rate × latency
If λ = 5_000 req/s and W = 0.2 s → L ≈ 1_000 in-flight
If a pause or stall pushes W → 1 s → L ≈ 5_000
→ pools, threads, and memory blow up → worse p99
Shedding policy sketch:
if in_flight > soft_limit: reject low priority
if queue_wait_p50 > budget: fail fast 503 + Retry-After
never accept work you cannot finish before deadlineWorked example: homepage p99
Product wants homepage p99 under 400 ms. Current: p50 120 ms, p99 1.8 s. Traces show optional recommendations at 1.5 s on the tail; cache miss storms on the product card service after deploys.
Plan (ordered by leverage):
1. Budget: mark recommendations optional; timeout 80 ms; empty shelf OK
2. Coalesce + cache product cards; warm cache on deploy
3. Colocate card service with its Redis in one AZ
4. Propagate deadline from gateway (400 ms absolute)
5. Hedge only inventory read after 60 ms, cancel loser
6. Shed bot traffic when gateway in_flight > soft limit
7. Verify: homepage histogram p99; keep unsampled metrics
Expected:
Optional timeout removes 1.5 s straggler from critical path
Cache hit path drops card service from ~100 ms to ~5 ms
p99 falls near budget; remaining tails investigated with profilesTrade-offs
| Choice | Gain | Cost |
|---|---|---|
| Aggressive caching | Lower p50/p99 on hits | Staleness, invalidation complexity |
| Hedging | Cuts straggler tails | Extra load; needs cancel |
| Tight deadlines | Protects queues | More user-visible errors |
| Load shedding | Stable latency for accepted work | Rejected clients; fairness debates |
| Degraded UI | Keeps page inside budget | Product compromise |
| Cross-region active-active | HA / locality for users | Replication lag, conflict design |
| Batching on the user path | Higher throughput | Adds wait—usually wrong for p99 pages |
Choosing and explaining latency optimization (p99)
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
Separate critical-path work from async work, then budget milliseconds per hop.
Attack queueing, path length, and stragglers with different techniques.
Fan-out amplifies tails: optional deps need tight timeouts and degradation.
When it belongs
User-facing SLOs on p95/p99.
Designs with deep dependency graphs or multi-region reads.
Explaining why more servers did not fix latency (queues vs service time).
How it fails
Unbounded retries and hedges melting the dependency.
Cache stampedes after expiry.
Optional features left on the critical path.
Cross-region chatter eating the budget.
Trade-offs to weigh
Freshness versus cache hit latency.
Hedging success versus duplicate load.
Fail-fast errors versus slow successes.
Throughput batching versus per-request latency.
Common misunderstandings
Optimizing averages while p99 burns the SLO.
Adding caches without coalescing or invalidation.
Flat timeouts that ignore remaining deadline.
Confusing this topic with pure throughput scaling.
Wrap-up
Latency optimization is budgeted engineering on the critical path. Complement the latency-vs-throughput mental model with concrete tail techniques—and always verify on the percentile you promised.
Exercises
- Budget. SLO p99 250 ms. List hops for a checkout API and assign numbers; mark one hop optional.
- Fan-out. 30 parallel calls, each with 0.5% chance of >500 ms. Approx. probability the page exceeds 500 ms? Name two mitigations.
- Hedge safely. Write rules for when hedging is allowed (idempotency, delay, cancel, global cap).
Primary sources
Key takeaway
Shrink p99 by budgeting the critical path, keeping queues short, and neutralizing stragglers with cache, locality, coalescing, hedging, deadlines, and degradation—not by chasing mean latency alone.