System Design Production Engineering

Load Testing Techniques

Marketing promises a Black Friday sale. Capacity planning says the checkout cluster can handle 12k checkouts per minute. Without a , that number is a spreadsheet hope—until the first real surge proves otherwise.

Load testing is how you turn capacity claims into measured evidence: a realistic , a controlled , and dashboards that show and errors before customers do.

Test journey

Name journeys from production, not invent them.

Match how real arrivals keep coming under slowdown.

Find the knee before you hold a plateau.

P99 + success rate beat average RPS.

What actually happens

A load generator runs scripts that act like clients. Each follows steps with , or the generator schedules an of independent requests. Responses are timed; failures are counted; backend metrics show where queues form.

Good tests change one variable at a time: intensity, mix, or data shape. Bad tests max out a single hot endpoint with empty payloads and declare victory because average latency stayed low while the real checkout path never ran.

Results only transfer to production if the environment is close enough: similar instance sizes, connection pools, caches warmed (or deliberately cold), and a data volume that exercises indexes the way production does.

How to choose a load-test design

1. What claim are you proving? Peak RPS for a launch, overnight soak for leaks, or “can we survive 2× Black Friday?” Each claim needs a different and duration.

2. Open or closed? If production arrivals keep coming when you slow down, use an . A with fixed VUs can understate overload risk.

3. Which SLO line must not move? Tie pass/fail to success rate and on the critical journey—not to generator CPU or raw throughput.

Simple rule of thumb

Never trust a load number without the mix, model (open/closed), ramp shape, and the percentile you measured. “We did 50k RPS” is meaningless if 50k were health checks.

Open vs closed arrivals

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

flowchart LR
  subgraph Open["Open model"]
    A[Arrival process] -->|fixed RPS| Q[Request queue]
    Q --> S[Servers]
    S -->|slow| Q
  end
  subgraph Closed["Closed model"]
    V[N virtual users] --> R[Request]
    R --> S2[Servers]
    S2 -->|response| V
  end

Prefer open when…

You are validating flash-sale or viral spikes.
You need to see queue growth and overload behavior.
Autoscaling and shedding must prove themselves under pressure.

Closed is fine when…

You model a fixed agent or device fleet.
You are exploring capacity with a known concurrency budget.
You explicitly document that throughput will self-limit under latency.

Ramp shapes that answer different questions

The is the experimental design. A flat plateau answers “are we stable at X?” A staircase answers “where is the knee?” A sudden jump answers “what happens in the first 30 seconds of a spike?”

Shape	Question it answers	Watch for
Linear ramp	Where does P99 leave the SLO?	Knee of the latency curve; first
Step ramp	Is each level stable for N minutes?	Warmup artifacts; GC and cache fill between steps
Spike	Do we shed or thrash on a jump?	Retry storms; connection pool exhaustion
Soak (hours)	Do leaks or fragmentation appear?	Memory growth; disk fill; replica lag drift

From problem to mechanism

Problem. Leadership asks whether checkout survives 3× baseline. The team ran a closed Locust script of 500 VUs hitting GET /health and reported “fine.”

Naive system. Average latency looked great because health checks are cheap and the closed pool slowed itself when anything else stalled.

Break it. Real traffic is an open arrival of cart reads, inventory holds, and payment authorizations. Under 3×, the DB connection pool saturates; P99 checkout exceeds 5s; clients retry and double the arrival rate.

Mechanism. Encode the checkout funnel as a , drive an at 1× → 2× → 3× with a step , and gate on checkout success rate plus P99—not on health-check RPS.

Correctness invariant for the test itself

A passing load test must state the journey, intensity, model, duration, environment SHA, and pass criteria. Without those, the green badge is not evidence.

Worked example: sizing a checkout load test

plaintext

Business target: 12,000 successful checkouts / minute at peak
= 200 checkouts / second

Funnel mix (from analytics):
  browse SKU     8× checkout  → 1,600 RPS
  add-to-cart    3× checkout  →   600 RPS
  start checkout 1.4×         →   280 RPS
  pay            1.0×         →   200 RPS
  confirm        1.0×         →   200 RPS

Open-model generator:
  target_arrival = mix_weighted RPS
  think_time from p50 session gaps (do not zero it out)

Pass criteria (example SLO):
  checkout success ≥ 99.5%
  checkout P99 ≤ 800 ms
  payment dependency error rate ≤ 0.2%
  primary DB CPU < 70%, pool wait P99 < 20 ms

Intensity

200 checkout RPS

Derived from the business minute rate, not from VU folklore.

Supporting reads

~1.6k browse RPS

Cache hit rate and CDN behavior only show under the real mix.

Gate

P99 + success

Reject “throughput won” if the tail or errors moved.

Generator pipeline

Whether you use k6, Gatling, Locust, JMeter, or a custom client farm, the logical pipeline is the same: schedule arrivals, execute the scenario, collect timings, and correlate with system metrics.

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

sequenceDiagram
    participant Sched as Arrival scheduler
    participant VU as Scenario runner
    participant SUT as System under test
    participant Obs as Metrics + traces

    Sched->>VU: Open arrival tick (RPS)
    VU->>VU: Apply think time / auth / data
    VU->>SUT: HTTP or gRPC call
    SUT-->>VU: Status + body
    VU->>Obs: Latency sample, outcome
    SUT->>Obs: Saturation signals
    Note over Sched,Obs: Abort if SLO burn or error budget guardrail trips

Technique catalog

Baseline then stress the claim

Establish 1× production-like load and record the steady dashboard. Only then move to 1.5× / 2× / claim×. Without a baseline, you cannot tell whether a bad P99 is new or always there.

Warm vs cold start

Run once with warm caches and connection pools, once after a cold deploy. Launch days often look like cold starts; nightly soaks look like warm. Report both.

Data realism

Skewed keys, large carts, and rare SKUs matter. Uniform random IDs miss hot partitions and optimistic lock contention that real catalogs create.

Client-side limits

Generator CPU, DNS, TLS session churn, and shared load-balancer IPs can bottleneck before the SUT does. Scale generators horizontally and watch generator saturation as a first-class signal.

Correlate, do not admire averages

Join request IDs to traces when P99 spikes. A rising average with a flat P50 often means a small cohort is dying—exactly the cohort your SLO cares about.

Reading results without fooling yourself

plaintext

Good report card
  arrival_rate_achieved ≈ arrival_rate_target (±5%)
  success_rate on critical journey
  P50 / P95 / P99 / P999 (or max) for that journey
  error taxonomy: 4xx vs 5xx vs client timeout
  saturation: CPU, mem, disk, pool waits, queue depth
  headroom: first resource that would fail at +20%

Red flags
  generator could not reach target RPS → under-loaded SUT
  closed model + rising latency → hidden overload
  only average latency reported → tail invisible
  test data all fits in cache → disk path untested
  pass on staging 4× smaller than prod → false confidence

Failure timeline: the “50k RPS” mirage

t0  Team runs closed test: 2k VUs, GET /api/ping, reports 50k RPS
t1  Launch marketing: real open arrivals on checkout mix
t2  Inventory service pool saturates; P99 checkout → 6s
t3  Mobile clients retry; effective arrival ≈ 1.8× planned
t4  Payment timeouts cascade; success rate drops below SLO
t5  Postmortem: load test never exercised pay path or open model
t6  Fix: open-model funnel test gated on checkout P99 + success

Still fails when…

Staging lacks production-like data volume or replica lag.
Third-party payment sandboxes do not rate-limit like production.
Autoscaling cooldowns differ between test and prod accounts.

Trade-offs

reveal overload; they cost more generator capacity and harder analysis.
Full-mix tests are slower to author than single-endpoint blasts, but transfer better to launches.
Prod canary load is highest fidelity and highest blast risk—use tiny cohorts and hard abort gates.

Where load testing sits among other tests

Unit and contract tests prove logic and schemas. Distributed fault tests prove retries and partitions. Chaos engineering proves resilience hypotheses under injected failure. Load testing proves capacity and latency under intensity. Stress and soak push past or through the claim to find the cliff and the slow leak. Use each for the question it answers; do not substitute one for another.

Choosing and explaining load testing

A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.

Explain the core idea

A load test is evidence for a capacity claim: mix, arrival model, ramp, and percentile gates—not a raw RPS trophy.

Open models keep arrivals coming when the system slows; closed models can hide overload by self-throttling.

Pass/fail belongs on the critical user journey’s success rate and tail latency, correlated with the first saturating resource.

When it belongs

A launch, sale, or migration claims a peak QPS or concurrency number.

Autoscaling, caches, or connection pools were resized and need proof.

Incidents show overload, retry storms, or “it was fine in staging.”

How it fails

Testing a cheap endpoint while the expensive funnel path never runs.

Using a closed VU pool and mistaking self-throttle for headroom.

Ignoring generator saturation so the SUT never sees the target arrival rate.

Trade-offs to weigh

Realistic open-mix tests cost more to build and run than ping floods.

Staging fidelity versus production canary risk must be explicit.

Long soaks find leaks; short ramps find knees—budget for both when stakes are high.

Common misunderstandings

Reporting average latency without P99 and success rate.

Zero think time that invents robot traffic shapes.

Declaring victory because the load generator hit a high RPS on health checks.

Wrap-up

Exercises

Convert a peak. Product wants 30k orders/hour. Derive checkout RPS and a plausible supporting mix. Which model—open or closed—would you use for the launch gate?
Find the knee. Design a step ramp from 0.5× to 2× claimed peak. What pass criteria and abort signals stop the run before you melt a shared database?
Debug a mirage. A report shows 40k RPS and 12 ms average latency, but on-call still fears Black Friday. List five questions you ask before trusting the report.

Topic-specific references

ACM Queue: Performance Testing — framing what performance tests can and cannot prove.
k6 documentation — open-model scripting, thresholds, and executors as a concrete generator pattern.
Google SRE: Handling Overload — why arrival behavior under slowdown matters for test design.

Key takeaway

Load testing is experimental evidence for a capacity claim. Match the mix and arrival model to production, ramp with intent, and gate on the journey’s tail—not on a generator’s RPS counter.