System Design Foundations
Latency vs Throughput
One Google Search waits a couple hundred milliseconds. Meanwhile the fleet behind it may finish hundreds of thousands of queries per second for everyone else. Those are two different measurements of the same system, and confusing them is the most common capacity mistake in system design.
is that one person's wait. is how many requests the fleet finishes together. The two are linked by a queue: if arrivals outrun servers, requests wait in line, and latency jumps even when "average capacity" looked fine on paper. This page explains that link, shows how to measure the slow tail with percentiles, and gives you the two laws that turn latency and throughput into quick capacity arithmetic.
One search click
Request arrives
It joins a queue whenever every worker is already busy serving someone else.
Workers serve
The app, a cache, the database, or an external API does the actual work.
/
Watch the slow tail of requests, not only the average, because users feel the tail.
Protect capacity
Timeouts, load shedding, and async work keep queues short when traffic bursts.
What actually happens
Start with the two definitions, because the rest of the page builds on them. is the time one operation takes, measured in milliseconds (ms), microseconds (μs), or nanoseconds (ns); one HTTP request completing in 50 ms is a latency. is how many operations complete per unit time, in requests per second (req/s), megabytes per second (MB/s), or transactions per second (txn/s); a fleet finishing 100K HTTP requests per second is a throughput. A highway makes the difference concrete: latency is one car's trip time, throughput is cars per hour past a point, and bandwidth (the lane count) is the upper bound on throughput. Throughput can never exceed bandwidth, no matter how fast each car drives.
Latency itself has two parts: time waiting in line plus time being served. The waiting part is what grows. As (how close the fleet runs to full capacity) approaches saturation, queueing pushes latency up nonlinearly: tiny bursts create long queues and spikes. That is why many teams target roughly 60–70% steady load rather than running servers flat out; the spare headroom is what keeps the slow tail tame.
Averages hide all of this. A system can report a fine mean latency while freezing some users, or high (queries per second) with a sluggish product because queues backlog during a launch. Neither metric replaces the other, so talk about and (the delay the slowest 5% and 1% of requests see), where queues build, and whether the page to many dependencies, since one slow dependency can hold up the whole page.
Before buying bigger servers, sanity-check capacity with and ask whether parallelism can even help under . Both laws get a worked section below; first, here is how to decide which metric you should be optimizing at all.
Try it: turn arrival rate and workers into latency
Adjust service time, worker count, and arrival rate, then advance simulated time. Throughput is bounded by workers divided by service time; once arrivals exceed that, queues grow and latency rises even though each request still takes the same CPU work.
Waiting requests
0
Each waiting request adds roughly service time divided by workers to latency.
- Max capacity
- 100 req/s
- Utilization
- 60%
- p50 latency
- No completions yet
- p95 latency
- No completions yet
~0 completions in recent window · queue 0 · ~0 ms estimated wait · utilization 60%. Four workers, 40 ms service time, 60 arrivals per second. Step time to watch queue wait separate from service time.
Optimize the right metric. Shorter service time helps both latency and throughput. More workers raise throughput capacity; they only shrink latency once a queue already exists.
How to choose what to optimize
Now that latency and throughput are separate ideas, the practical question is which one is actually hurting. Three questions sort almost every case.
1. Are users waiting too long? That is a problem. Cut work, cache results, colocate data (keep it on the same machine or zone as the code), parallelize safe so independent calls overlap, or keep queues shallow so requests are served rather than waiting.
2. Is the fleet falling behind arrivals? That is a problem. Add workers, batch small items into one operation, shard state (split it across machines), or, before any of those, find the serial bottleneck that every request still passes through.
3. Are only the tails bad? Then the average is fine and a few requests suffer. Protect them with timeouts, circuit breakers (stop calling a dependency that keeps failing), bounded , admission control (accept only as much work as you can serve), and before queues explode.
Simple rule of thumb
Optimize the percentiles that match the user journey, not vanity averages. Use for quick capacity algebra and before overspending on parallelism. Deep dependency graphs amplify tails, so budget end-to-end latency with fan-out in mind.
Queueing mental model
All three questions above come from one picture. Requests arrive, wait in a queue if every worker is busy, get served by a worker that may itself call a database, cache, or external API, and finally leave as responses. The single decision that governs the whole system is how close sits to 100%.
Incoming requests
Queue or backlog
Workers or servers
DB / cache / external API
Completed responses
Utilization near 100%?
Yes
p95 and p99 rise sharply
No
Latency stays predictable
Map the three questions onto this diagram. Lowering shortens the time a request spends in the queue and at the worker. Raising widens the worker stage so the queue drains faster. Protecting tails keeps the system from ever reaching the "Yes" branch, where and climb sharply. The next section explains what those two percentiles actually measure.
Percentiles: p50, p95, p99
A percentile belongs to a specific measurement population and time window. A p99 of 500 milliseconds means approximately 99% of those observed requests completed within that value; it is not the latency of every request in the slowest 1%. Do not average p99 values across servers to obtain fleet p99. Combine a suitable distribution, such as mergeable histogram counts, and compute the percentile over the combined observations.
The queue diagram explains why a slow tail exists; percentiles are how you measure it. A percentile says what fraction of requests finished faster than a given time: is the median, is the time 95% of requests beat, and asks what delay the worst 1% see. That last number is critical when pages fan out to dozens of calls, because the worst 1% of any one call becomes a common event at the page level.
Why averages lie
An average latency of 50 ms can coexist with 5% of requests at 430 ms. Fixing the average barely helps those users, which is why the table below reads the same distribution at several points instead of one.
| Percentile | Meaning | Example |
|---|---|---|
| p50 (median) | Half of requests faster than this | 45 ms |
| p90 | 90% faster | 120 ms |
| p95 | 95% faster | 250 ms |
| p99 | 99% faster | 1200 ms |
| p99.9 | "Three nines" tail | 5000 ms |
Why p99 can dwarf p50
In the table, p99 is more than twenty-five times the median. That gap matters because at huge daily volume the rare tail still touches many sessions, and because a page that issues dozens of parallel RPCs (remote procedure calls to other services) behaves roughly like repeated draws from the same latency distribution: the odds that at least one call lands in the tail rise quickly. User latency then tracks something closer to the slowest dependency than the average one, which is why percentile budgets per dependency are essential. Typical targets to anchor on are listed in Numbers to know below, and the fan-out section works through the arithmetic.
Two laws for quick capacity algebra
Percentiles tell you how bad the tail is; these two laws tell you whether more hardware would help. connects latency to throughput through concurrency, and says how much parallelism can buy.
Little's Law
Use matching boundaries and averages. If a stable service completes 500 requests per second and the average request spends 0.2 seconds inside it, average in-flight work is 100 requests. If that time includes waiting in a queue, the concurrency includes queued requests as well as active execution. A p99 latency cannot replace the average in this equality, and an ever-growing backlog violates the steady-state assumption used for simple capacity estimates.
Under steady state, average concurrency equals arrival rate times average time in system. Read it both ways: halving average latency doubles the throughput you can sustain with the same concurrency budget.
- L: average items in the system at once, which is your concurrency (in-flight requests).
- λ (lambda): average arrival rate, which in steady state equals your throughput.
- W: average time each item spends in the system, which is its latency.
Amdahl's Law
Suppose 20% of a job is necessarily serial and 80% can run in parallel. With four ideal workers, normalized time is 0.2 + 0.8 / 4 = 0.4, a 2.5-fold speedup. Even infinitely many workers cannot reduce time below the serial 0.2, so maximum speedup is fivefold. Real systems add scheduling and communication overhead, making these optimistic bounds rather than promised benchmark results.
Little's Law tells you how much concurrency you need; Amdahl's Law warns that adding workers only speeds up the fraction of work that can run concurrently. A stubborn serial section caps theoretical speedup no matter how many machines you rent.
- P: the fraction of the work that can run in parallel across workers.
- N: the number of parallel workers or machines you add to the job.
Example: P = 0.9 (90% parallel), 10% serial
Attack the serial slice first; widening N alone yields diminishing returns.
End-to-end latency budget and Numbers to know
With the two laws in hand, you can turn a product target into a budget. If the target is under 200 ms, every dependency on the request path needs its own slice of that budget, because one slow database query or cross-region call can consume the whole request. The table gives example slices; the async row is the point of the exercise, since work that leaves the critical path costs the user nothing.
| Part | What it includes | Example budget |
|---|---|---|
| CDN hit path | Edge lookup + response | 20-50 ms |
| Network and TLS | DNS/TLS/edge routing when not warm | 30-80 ms |
| App processing | Auth, validation, business logic | 30-60 ms |
| Cache lookup | Distributed cache in same region | 1-5 ms |
| Database query | Indexed local read/write | 20-60 ms |
| Async work | Email, analytics, media processing | Outside critical path |
Most sizing starts with unit conversion: requests per day become , payload size becomes bandwidth, and a latency budget becomes concurrency via . The numbers below are the worked conversions and the reference targets the rest of the page points back to.
Numbers to know
- Illustrative p50 targets: search under 200 ms, inbox under 100 ms, maps under 300 ms.
- Requests per day to QPS: 100M/day ÷ 86,400 ≈ 1,157 average; assume 4× peak, so ~4.6K peak QPS.
- Payload to bandwidth: 4.6K QPS × 20 KB ≈ 92 MB/s ≈ 736 Mbps before protocol overhead.
- QPS × latency to concurrency: 4.6K QPS × 120 ms average ≈ 552 in-flight requests.
- Concurrency check: 1.2K QPS × 200 ms ≈ 240 in flight; at 800 ms p99 that becomes about 960 slots.
- Cache offload: 80% cache hits means only 20% of read QPS reaches the database.
- Edge offload: 90% of bytes served at the edge leaves origin bandwidth at one-tenth of user-facing bandwidth.
- SLO sketch: 99.9% of requests under 300 ms over 30 rolling days; error budget ≈ 0.1% may violate.
Fan-out tail amplification
The formula below assumes independent chances of being slow and a user operation that waits for every dependency. Shared infrastructure can correlate delays, and optional calls may have different completion rules. For a recommendation panel, returning partial results by a deadline may be acceptable; for a transaction requiring all participants, omitting a response can violate correctness. Tail mitigation must follow the user-visible contract.
The budget above assumed each dependency is called once. Real pages , and that multiplies the tail. If one dependency has 1% slow requests and a page calls 20 dependencies, the chance that at least one call is slow is about 1 − 0.99^20, or 18%. The user waits for the slowest critical call, so a event for one service becomes a nearly one-in-five event for the page.
One page request
Service A
Service B
Service C
Service D … 20 calls
User waits for slowest critical call
1 − 0.99^20 ≈ 18% chance of one slow call
The takeaway is that p99 for one service is not the same as p99 for the whole user journey. This is exactly why the tail-protection tools from question 3 above exist: they keep one slow dependency from holding the critical path hostage, and moving non-critical work async takes it off the path entirely. The next section turns that into concrete moves and their costs.
Strategies and their trade-offs
Here is the toolbox for the three questions from earlier. wins usually remove work or move data closer to the code. wins usually add parallelism, batching, or partitions, and only pay off once the serial bottleneck is understood (recall Amdahl's Law above).
Reduce latency
- Caching tiers (database, then distributed cache, then in-process L1) so hot reads skip slow storage.
- CDN or edge servers (machines placed near users) for static or cacheable API responses.
- Connection pooling reuses open TCP/TLS connections instead of paying a handshake per request.
- Async non-critical work off the critical path so the user never waits for it.
- Data locality (same availability zone or rack) cuts network hops between app and data.
- HTTP/2+ and gRPC binary payloads shrink bytes and round trips where appropriate.
Increase throughput
- Horizontal scale behind a load balancer spreads arrivals across more identical workers.
- Batching (producer linger, bulk writes) amortizes per-request overhead across many items.
- Compression on the wire moves more payload through the same bandwidth.
- Queues for absorbable work smooth bursts so workers run at a steady pace.
- Sharding or partitioning state splits data across machines so each handles a slice.
- Read replicas serve query-heavy workloads so the primary database is not the only machine answering reads.
- Multiplexed protocols run many streams over one connection, avoiding per-request connection cost.
Latency vs throughput trade-offs
Several moves in those lists help one metric while hurting the other. Say the trade-off out loud instead of claiming every change is always faster.
Batching
Improves throughput because one round trip carries many items
Adds wait time before the first item starts processing
Compression
Reduces bandwidth by shrinking every payload on the wire
Uses CPU and may hurt p95 when servers are already loaded
Caching
Reduces latency and database QPS by answering from memory
Allows stale data unless invalidation is handled carefully
Hedging
Reduces tail latency by racing a backup request
Increases duplicate load if hedges are not bounded
Tail-latency drill, and what to watch
Load generation must keep sending according to the intended arrival pattern when the service slows. A test that waits for each response before issuing its next request reduces offered load during the slowdown and can hide the worst queueing. Record offered requests, admitted requests, completed requests, errors, and wait time together. This shows whether improved latency came from more capacity, less work, or simply rejecting most traffic.
Knowing the strategies is not the same as knowing they work. Tail latency needs a failure drill, because a system can look healthy on averages while one slow dependency makes whole pages violate the user (service level objective, the measurable latency promise you made). Run the drill in four steps, then keep the monitoring that would catch a regression.
Degraded-mode test
1. Budget the path
For p95 under 200 ms and p99 under 800 ms, assign budgets to edge, app, cache, DB, and each downstream call. Anything outside the critical path should move async.
2. Inject slowness
Add 500 ms delay to 1% of one dependency and test a 20-call page. The page-level slow probability climbs the way the fan-out arithmetic above predicts, so user p95 can fail even when service p99 looks fine.
3. Degrade response
Return partial cards, stale cache, empty recommendations, or queued work rather than blocking the whole response on a non-critical slow dependency.
4. Control retries
Retries and hedging can reduce tail latency but add load. Cap duplicate work and stop retrying once the end-to-end deadline is nearly spent.
The drill only counts if it can block a release: a useful load test fails the build when page-level or violates its budget, not only when individual service averages regress.
What to watch
- Collection: standardized traces and metrics, so every service reports latency the same way.
- Storage: a time-series metrics store plus a distributed tracing store for slow spans (individual timed steps).
- Dashboards: histograms, heatmaps, or percentile lines, never means alone, so tails stay visible.
- Alerting: error-budget burn on latency or availability SLOs, rather than single threshold spikes.
- Tracing: follow slow spans to see which dependency dominates the tail before optimizing anything.
An example objective to instrument against is the SLO sketch in Numbers to know above; the error budget is the small slice of requests allowed to violate it, and alerts fire when that slice is being spent unusually fast.
Reference latencies (orders of magnitude)
Every budget and conversion above depends on knowing roughly how long each tier takes. These numbers drift with hardware generation; use them to compare tiers (RAM vs SSD vs cross-region round trip), not as benchmarks for your laptop.
| Operation | Latency (ballpark) |
|---|---|
| L1 cache reference | ~0.5 ns |
| L2 cache reference | ~7 ns |
| Main memory reference | ~100 ns |
| SSD random read | ~16 μs |
| Read ~1 MB sequential SSD | ~1 ms |
| Disk seek (HDD) | ~10 ms |
| Transcontinental RTT | ~150 ms |
| Distributed cache GET (same zone) | ~0.1–0.5 ms |
| Relational indexed query | ~1–10 ms |
| Object storage GET | ~20–100 ms |
| TLS 1.3 handshake | ~30–80 ms |
How to explain this in an interview
Everything above compresses into a short talk track. Lead with the capacity arithmetic, then ask the questions that reveal which metric the interviewer actually cares about.
Capacity estimate hooks
- Open with to size in-flight requests, then state your assumptions out loud.
- State peak separately from average QPS, because capacity is planned for peaks.
- Budget per dependency, especially when one request fans out to many services.
- Name the bottleneck: CPU, locks, database I/O, network, or downstream quotas behave very differently.
Follow-up questions
- Is the about p50, p95, p99, or end-to-end page load time?
- Can we shed low-priority work when queues grow, and who decides what counts as low priority?
- Which calls sit on the critical path, and which could run async without the user noticing?
- Will batching improve throughput while hurting per-request latency, and is that acceptable here?
Wrap-up
What to remember
Track / that match user journeys, not vanity averages. Use for capacity algebra and before overspending on parallelism. Deep graphs amplify tails, so budget end-to-end latency with in mind.
Primary sources
- Jeff Dean: Numbers Everyone Should Know: the latency hierarchy behind the reference table above, used in back-of-envelope design.
- Google SRE Workbook: Implementing SLOs: how to define SLIs (the measured indicators) that mix latency and availability.
- Dean & Barroso: The Tail at Scale: the paper explaining why p99 tails dominate user experience in fan-out systems.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Measure latency distributions and sustainable throughput without coordinated omission, hidden queues, or closed-loop load-test bias.
Explain the decision
Why can average latency stay low while users suffer?
Read the reasoning
A small tail can contain the most valuable or fan-out-heavy requests; p95/p99 histograms and per-operation slices reveal it.
Apply it to a production scenario
A closed-loop load test reports stable latency while production overloads. What changed?
Read one approach
Closed-loop clients wait before sending more and suppress offered load; use an open-loop arrival model and record omitted waiting time.
Work through the numbers
By Little’s Law, what concurrency corresponds to 8,000 requests/s and 250 ms average time in system?
Read the worked calculation
L = λW = 8,000 × 0.25 = 2,000 concurrent requests.
Reason through a failure
What must a latency SLO measurement include?
Read the recovery reasoning
All valid user attempts, including timeouts and load-shed responses, with a histogram that does not erase queue wait or coordinated omission.
Sources and further study
Continue with Latency Optimization Techniques.