System Design Overload
Backpressure, Admission Control & Load Shedding
Imagine Uber surge at New Year's Eve: ride requests triple in minutes. If every request queues forever, the app hangs, retries multiply, and even payments start failing. A healthy system says no early so critical paths stay fast.
When demand exceeds capacity, something must give. signals overload upstream; rejects work before it enters; drops lower-priority work to protect login and checkout. then tunes how much work stays in flight from live latency feedback.
Control order
Cap concurrency at the edge so work is rejected before unbounded queues form.
Slow producers before buffers explode so retries do not amplify the storm.
Keep P0 journeys alive by dropping optional enrichment and background work first.
Tune in-flight limits from latency feedback so capacity matches what downstream can handle.
What actually happens under overload
The New Year's surge above is the default shape of overload: capacity is finite, arrivals are not. The real question is not "how do we accept everything?" but "which work survives, and how do we tell the rest to wait without melting the stack?"
The naive system assumes unbounded request queues, generous timeouts, and clients that "retry forever," with autoscaling catching up in time. Every request is equal: recommendations and checkout share one FIFO (first-in, first-out) queue. That fails as soon as arrivals jump from 1,000 to 1,300 req/s for 20 seconds while service time stays 80 ms. Clients time out and retry twice, offered load climbs further, p99 (the 99th-percentile latency) explodes, and even admitted checkout fails because workers are busy on doomed work (see Numbers to know below for the backlog math).
A larger queue cannot invent capacity. It only converts a capacity error into a latency outage and memory risk: work expires after the client already left. As approaches 100%, wait time grows without bound, so a fast 503 with retry guidance beats a slow hang that burns error budget on every layer. Healthy systems reject or shed early, reserve headroom for auth and checkout, return Retry-After so clients do not amplify the storm, and use to raise and lower in-flight limits from latency and error feedback.
How to choose a control
Once you accept that something must give, three questions decide which control to apply first.
1. Can you reject before queueing? Prefer edge over deep unbounded buffers, because dropping cheap work early protects the workers that still have a chance to finish.
2. Which journeys are sacred? Rank P0/P1/P2 and apply to optional work first so auth, checkout, and writes keep reserved capacity.
3. Will clients retry safely? Pair shedding with Retry-After, jitter, and idempotency keys so a 503 does not become a retry storm that raises offered load.
Simple rule of thumb
Fail fast beats fail slow. Cap admitted work, propagate , shed by priority, and watch queue depth as a first-class signal, not only CPU.
From problem to mechanism: admit, bound, shed, adapt
The questions above name the controls; here is how they stack into one mechanism, and the invariant that must stay true while they run.
Correctness invariant
Accepted work must remain bounded by service capacity and deadline. Once admitted, P0 work receives enough reserved capacity to complete. Rejected work must not create a retry loop that increases offered load: pair every shed with Retry-After, jitter, and a retry budget.
A token bucket limits how quickly work enters over time and may permit a bounded burst. A concurrency cap limits how many operations remain active at once, adapting naturally to slower service because occupied slots are released later. Use both when request rate and execution cost vary independently. Reject before acquiring expensive database sessions or allocating large payload buffers, and distinguish a client quota response such as 429 from temporary service overload such as 503.
Bounded queues + deadlines
A queued request consumes time even before useful work begins. Track its arrival time and propagated deadline, then compare remaining time with a realistic service-time estimate before dispatch. If it cannot plausibly finish, expire it rather than executing work whose caller has already left. A bound based on average service time is a sizing approximation; tail latency, varying request costs, and worker scheduling require load testing and headroom.
Rank journeys. Drop P2 analytics and enrichment first; protect auth, checkout, and write paths with reserved capacity.
Adaptive concurrency changes the in-flight limit using measured latency and error signals. In additive-increase/multiplicative-decrease, the system cautiously raises the limit while healthy and reduces it sharply when overload appears. Use a minimum and maximum limit, a smoothing window, and hysteresis so noise does not repeatedly double and halve capacity. Separate materially different workloads; mixing a two-millisecond cache read with a one-second export obscures what the latency signal means.
is the signal that ties the stack together: slow consumers tell producers to reduce send rate or pause pulls before buffers explode. Without it, admission alone still fills every buffer between the edge and the dependency.
Try it: watch a queue grow, then try to drain it
A service can complete 1,000 equally expensive requests each second, but 1,300 arrive. Decide whether to admit everything or limit the waiting queue. Advance the clock to see where the excess work goes. Then reduce arrivals: returning demand to capacity stops further growth, while spare capacity is what removes the backlog already accepted.
- Waiting now
- 0
- Queue drain time
- 0.00s
- Completed total
- 0
- Rejected total
- 0
The queue starts empty. Advance time to apply the selected arrival and service rates.
With the current settings, demand exceeds capacity by 300 requests/second; that excess must wait or be rejected.
Read the exact history as a table
| Time | Admitted | Completed | Rejected | Waiting |
|---|
Work the surge through. With the starting settings and admission unlimited, advance twenty seconds. The service finishes 20,000 requests, but 26,000 arrive, leaving 6,000 waiting. That is about six seconds of work ahead of a new arrival at 1,000 completions per second. Set arrivals to 1,000/s and advance again: the backlog stays at 6,000. Set arrivals to 700/s and run another twenty seconds: the 300/s spare capacity drains it.
Trade delay for explicit rejection. Reset the queue, restore 1,300 arrivals/s, and enable the 300-request limit. After twenty seconds, 300 wait, 20,000 completed, and 5,700 were rejected before acceptance. In this model, the shorter queue keeps its drain time near 0.3 seconds. It cannot make all demand succeed: the service still lacks capacity. A client can retry a rejected request only if its deadline and a bounded retry policy permit it; immediate retries would become new arrivals and recreate the overload.
Understand what the numbers omit. This is a deterministic fluid model: work arrives uniformly, every request has equal cost, and one-second service capacity is available throughout that second. It does not predict p99 latency, bursts within a second, request cancellation, or memory exhaustion. Queue drain time is waiting work divided by throughput; it excludes the new request’s own service time. In production, bound queue age and active concurrency too, and measure useful completions as dependency latency changes.
The control rejects new work before promising to execute it. Turning on a smaller limit never deletes accepted work. Backpressure additionally asks an upstream producer to slow down; rejection alone cannot guarantee it obeys. See Google SRE’s treatment of queues and overload for the operational reasoning behind bounded queues and shedding.
Numbers to know
The mechanism needs sizing, not slogans. ρ equals arrival rate divided by service capacity for a comparable workload. In common stochastic queueing models, expected waiting time rises sharply as ρ approaches 1; exact behavior depends on arrival and service-time variability. The deterministic lab has no such variability, so equal arrival and service rates leave its existing queue unchanged. says L = λW for stable long-run averages with consistent boundaries: include queue time in W when L includes waiting requests. A timeout limits how long a caller waits; it frees backend capacity only when cancellation actually stops the work.
Steady-state in-flight
L = 1,000/s × 0.08s = 80
At a stable arrival rate of 1,000 requests per second and mean active service time of 80 milliseconds, average active work is about eighty requests. This describes the observed workload; it does not prove that a limit of exactly eighty will meet p99 under bursts. Measure the capacity curve, account for variability, and reject excess work when increasing concurrency starts increasing latency without increasing useful throughput.
Queue bound for 120 ms wait SLO
1,000/s × 0.12s = 120 queued
At an assumed service rate of 1,000 per second, a queue of 120 corresponds to roughly 120 milliseconds of drain time when work costs are similar. Adding eighty active requests yields an illustrative admitted-work budget near two hundred. Variable job sizes and a slowing dependency can invalidate that estimate. Enforce both queue depth and age, and recalculate admission from the actual remaining deadline.
Uncontrolled 20 s burst
(1,300 − 1,000) × 20 = 6,000
Backlog implies ~6 s of waiting. Shed ≥300 req/s or add capacity that is already warm: autoscaling that arrives later cannot rescue expired work.
Retry amplification
1 + 0.2 + 0.04 = 1.24×
If each attempt independently fails with probability 0.2 and at most two retries are allowed, expected attempts are 1+0.2+0.2²=1.24 per logical request. During a correlated outage the independence assumption fails and nearly every request may make all three attempts. Jitter spreads retry timing but does not reduce the maximum attempt count; a shared retry budget and deadline limit how much extra load the system can create.
Shedding priority table
Priority must reflect dependency cost as well as business importance. Reserve concurrency for checkout, but also cap how much inventory and database work each checkout can start. Otherwise protected requests can overload the same backend that optional requests were shed to protect. Fairness across tenants and classes needs bounded queues or scheduling shares, with an explicit rule for reclaiming capacity that lower priorities temporarily borrow.
Numbers size the queue; this table decides which journeys those limited slots belong to when starts.
| Priority | Protect | Shed first |
|---|---|---|
| P0 | Auth, checkout, write path | Never (reserved capacity) |
| P1 | Core reads, search | Recommendations, previews |
| P2 | None reserved | Analytics, exports, batch reindex |
Failure timeline, and what still breaks after the fix
Putting the stack and priorities together, here is how a dependency slowdown plays out minute by minute, then the ways an improved design can still fail and the trade-offs you accept.
Timeline: downstream latency triples
0s Dependency service time 80ms → 240ms
Safe concurrency requirement collapses; arrivals unchanged
2s Queue age hits 50% of client deadline
Expire queued P2; pause broker pulls; adaptive limit freezes
5s Edge returns 503 + Retry-After for shed traffic
Gateways enforce shared retry budget + jitter
15s Checkout (P0) still completing; recommendations disabled
Reserved capacity holds the critical path
60s Latency hysteresis clears; raise limits gradually
Do not snap back to pre-incident concurrency
Without and shedding, the same timeline ends with every priority timing out, including P0, because workers stay busy on work that already missed its deadline.
Still fails when…
- Shedding the wrong tier drops checkout while recommendations keep running and burning trust.
- One global concurrency limit treats cheap reads and expensive writes equally and starves both.
- oscillates when feedback is noisy and hysteresis is missing.
- Clients that ignore Retry-After still amplify load after every shed response.
- Multi-tenant systems without fair queues let one whale exhaust the shared pool.
- Raising timeouts instead of reducing offered load only lengthens the death spiral.
Trade-offs
- Rate limits protect quota over time; concurrency limits protect time-varying cost, so often use both.
- Queue only when wait still fits the deadline; otherwise reject early and protect admitted tail latency.
- Static caps are operable; adaptive caps track reality but need hysteresis and clean signals.
Overload control
What production needs to hold
A production design needs a way to recognize trouble and a rule it preserves while recovering. A trigger is the measurement or event that starts a response. An invariant is a statement that must remain true, such as never confirming the same reservation to two customers.
Trigger metric or condition
Introduce overload controls before sustained utilization drives queue wait beyond the latency SLO, or when retries and one large tenant can consume capacity needed by critical traffic.
Correctness invariant
Accepted work must remain bounded by service capacity and deadline; once admitted, P0 work receives enough reserved capacity to complete. Rejected work must not create a retry loop that increases offered load.
A bounded overload path
Client deadline
retry budget
Edge admission
rate + priority
Bounded queue
deadline ordered
Concurrency limit
adaptive in-flight cap
Dependency
propagates pressure
Worked overload and queue-sizing timeline
A service can complete 1,000 requests/second with an 80 ms average service time. The end-to-end p99 budget leaves at most 120 ms for queueing. A promotion sends 1,300 requests/second for 20 seconds, and clients retry twice without jitter.
In-flight work
L = λW = 1,000/s × 0.08s = 80
Little’s Law says roughly 80 requests execute concurrently at capacity. This is a starting limit, not a guarantee; adaptive control tracks observed latency.
Queue bound
1,000/s × 0.12s = 120 queued requests
A larger queue cannot meet the deadline. Cap total accepted work near 200, expire work before dispatch, and reserve a portion for critical priority.
Uncontrolled burst
(1,300 − 1,000) × 20s = 6,000 requests
The backlog implies about six seconds of waiting. Shed at least 300 requests/second or add immediately available capacity; autoscaling that arrives later cannot rescue expired work.
Retry amplification
1 + 0.2 + 0.04 = 1.24× offered load
At a 20% failure rate with two retries, naive clients add about 24% traffic. Retry-After, exponential jitter, idempotency, and a retry budget are part of admission design.
Decision
The edge reserves 30% capacity for checkout, applies weighted fair queues per tenant, returns HTTP 429 for quota exhaustion and 503 for unavailable capacity, and publishes Retry-After. Workers pause pulls when their bounded queue is full.
How the mechanism holds
Follow how each part protects the rule above. The mechanism explains what the system does; the reasoning explains why that action is sufficient under the stated assumptions. If an assumption changes, the guarantee may change too.
Protocol signaling
HTTP uses 429/503 plus Retry-After; gRPC uses RESOURCE_EXHAUSTED or UNAVAILABLE with client backoff; streams use credits, paused consumption, or reduced TCP receive windows.
Why it holds: A load test shows upstream send rate falls rather than retrying immediately.
Adaptive concurrency
Use gradient or AIMD feedback from minimum latency, queue wait, and errors. Separate cheap and expensive routes so one global limit does not treat them equally.
Why it holds: During a latency injection the in-flight cap falls and admitted p99 recovers without oscillation.
Tenant and priority fairness
Combine per-tenant token buckets, weighted fair queues, and reserved capacity. Permit borrowing when idle but reclaim it quickly when the owner needs it.
Why it holds: A whale-tenant stress test cannot violate a small tenant’s protected SLO.
Deadline propagation
Carry remaining time through every hop and do not start work that cannot finish. Shed optional enrichment before core correctness work.
Why it holds: Expired requests consume negligible downstream CPU and never commit after the client deadline unless semantics require it.
Failure timeline: downstream latency triples
Dependency slows
Service time jumps from 80 ms to 240 ms; the safe concurrency requirement changes while arrivals remain constant.
Response: Adaptive limit detects rising minimum latency and stops increasing in-flight work.
Queue reaches deadline bound
New background and low-priority work cannot meet its deadline.
Response: Expire queued work, pause broker pulls, and shed P2 before P0.
Clients receive explicit pressure
Capacity rejection reaches callers.
Response: Return typed status plus Retry-After; gateways enforce a shared retry budget and jitter.
Service stabilizes degraded
Checkout remains available while recommendations are disabled.
Response: Hold the lower concurrency limit until latency hysteresis and queue depth satisfy recovery gates.
How to choose among alternatives
Rate vs concurrency limit
Choose when: Rate protects a quota over time; concurrency protects a service whose cost is time-varying.
Caution: Use both when traffic entitlement and instantaneous saturation are different concerns.
Queue vs reject
Choose when: Queue only when work remains valuable after waiting and the bound fits its deadline.
Caution: An unbounded queue converts a capacity error into a latency outage and memory risk.
Drop vs degrade
Choose when: Drop expendable background work; degrade optional pieces of an interactive response.
Caution: Never degrade across a correctness boundary such as charging without durable order state.
Running it day to day
Measurements tell you what is happening; alerts tell an operator when action is needed. A rollout plan limits how many users encounter a change before it is checked. A runbook records the steps to diagnose and recover from a known failure.
Measure
- arrival, admit, reject, and completion rates by priority and tenant
- queue depth, queue age, in-flight work, and deadline expiry
- retry amplification and dependency service-time histogram
Alert
- oldest queue age exceeds 50% of deadline
- P0 rejection or fairness SLO breach
- retry traffic exceeds the retry budget
Roll out safely
- shadow decisions and label would-reject traffic
- canary static bounds before adaptive control
- fault-test slow dependencies and reconnect storms
Runbook
- identify the saturated resource and highest-value work
- lower admission, stop pulls, and disable optional fan-out
- coordinate Retry-After, then recover limits gradually with hysteresis
Practice scenario
Protect a shared search service from a 2× burst while premium interactive queries and free bulk exports use the same workers.
Questions worth asking next
- Where do you place the limit?
- How do Kafka consumers express pressure?
- What prevents retry amplification?
A solid answer sketch
Separate priority queues and concurrency pools, reserve capacity for premium traffic, cap each tenant, expire work by deadline, and pause bulk pulls. Return protocol-specific overload signals and make retries consume a bounded client budget.
Common mistakes
- • raising timeouts when capacity is exhausted
- • assuming autoscaling is instantaneous
- • using a global FIFO that lets cheap and expensive work block each other
Authoritative references
Last reviewed: 6 August 2026
Choosing and explaining backpressure and load shedding
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
When utilization approaches 100%, latency is not linear: queue wait dominates.
Reject or shed before unbounded queueing; protect P0 journeys with reserved capacity.
Pair shedding with Retry-After and idempotency so clients do not amplify overload.
When it belongs
Traffic is bursty or retry-heavy.
Downstream has hard connection or CPU limits.
Optional features must not take down core flows.
How it fails
Shedding the wrong tier breaks trust.
Global concurrency limit blocks cheap reads and expensive writes equally.
No signal to clients → retry storm.
Trade-offs to weigh
Early rejection improves tail latency for admitted work.
Adaptive limits add complexity versus static caps.
Fair queuing helps multi-tenant fairness but needs tenant identity.
Common misunderstandings
Increasing timeouts instead of reducing offered load.
Autoscaling lag during a 5-minute surge.
Shedding without metrics. You will not know what users lost.
Wrap-up
You have seen the failure, the mechanism, and the operational signals. Use these exercises to check the ideas stuck, then follow the references for the primary sources.
Exercises
- Protect search under a 2× burst. Premium interactive queries and free bulk exports share workers. Where do you place limits, how do you reserve capacity, and what status codes plus Retry-After values do clients see?
- Size the queue from an SLO. Service rate 2,000/s, service time 40 ms, p99 queue-wait budget 80 ms. What is the maximum useful queue depth, and what happens if product asks for a 10,000-deep buffer "just in case"?
- Express pressure from a Kafka consumer. Downstream RPC latency triples. Do you pause polls, reduce max.poll.records, or both, and how do you avoid a consumer-group rebalance storm while applying ?
Topic-specific references
Overload-control sources on , flow control, and SRE handling of overload.
Key takeaway
Overload is a product decision encoded in systems: rank work, enforce admission at the edge, propagate backpressure, and shed with observability. A fast 503 with guidance beats a slow hang that burns error budget on every layer.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Bound queues and concurrency, propagate backpressure, allocate tenant/priority capacity, avoid retry amplification, and shed before collapse.
Explain the decision
What is the difference between backpressure and admission control?
Read the reasoning
Backpressure asks producers to slow as capacity fills; admission rejects or defers work at a boundary before unsafe resource consumption.
Apply it to a production scenario
Queue age rises while CPU is only 50%. What do you inspect?
Read one approach
Downstream latency, concurrency limit, per-tenant skew, connection pools, blocked I/O, retry loops, and service-time distribution.
Work through the numbers
At 2,000 jobs/s and 400 ms average time, what concurrency follows from Little’s Law?
Read the worked calculation
800 in-flight jobs; set a bounded limit with tail/headroom evidence rather than an unbounded queue.
Reason through a failure
What must overload handling preserve?
Read the recovery reasoning
Accepted work stays within resources and priority/fairness policy; rejection is explicit, retryable only when safe, and never fabricates success.
Sources and further study
Continue with CDC, Outbox & Inbox Patterns.