System Design · Performance

Throughput Optimization Strategies

Black Friday traffic doubles. Latency SLOs still hold for most users, but the write path falls behind: queue age climbs, retries amplify load, and the fleet is “busy” without completing more useful work. That is a problem—completed operations per second—not a prettier p50 chart.

This note is the playbook after you already know latency versus throughput as definitions. Find the , apply , , , pooling, and copy reduction in the right order, and protect the path with and so optimization does not become self-inflicted overload.

Raise sustained rate

CPU, IOPS, pool, lock, or hot key—measure first.

One trip for N items when SLOs allow linger.

Keep workers and wires busy without unbounded depth.

More machines only help if keys are even.

What actually happens

Every request pays a fixed tax (parse, auth, TLS, RPC framing) and a variable tax (bytes, rows, CPU). When arrivals rise, the stage with the least leftover capacity becomes the . Queues form in front of it. Under , more queueing raises average time in system, so the same concurrency budget supports less useful λ—or λ stays high while user-visible latency collapses.

Optimization strategies attack the tax or the serial section. and cut fixed costs. and async I/O hide wait. and horizontal scale raise parallel capacity—bounded by and by . Compression and raise MB/s when the wire or memcpy is the cliff.

The last mile is control. Unbounded retries and optimistic accept rates turn a throughput win into a meltdown. Pair every capacity increase with and so the system stays in the region where Little’s Law still describes steady state, not a growing backlog.

How to choose what to optimize

1. Is the fleet falling behind arrivals? That is throughput. Profile which resource saturates—do not start with “add Redis.”

2. Is the cost per item mostly fixed? Batch, multiplex, and pool. If cost is proportional to bytes or rows, cut work, compress, or push filters down.

3. Is one key or lock serial? No amount of horizontal scale helps until you remove or isolate the hotspot.

Simple rule of thumb

Measure the bottleneck, amortize fixed cost, parallelize only the parallel part, and admit work you can finish. Throughput optimizations that ignore latency SLOs and hot keys just move the cliff.

Find the bottleneck before you “optimize”

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

flowchart TD
    In[Arrivals λ] --> Q[Queue / backlog]
    Q --> App[App workers]
    App --> Pool[Connection pool]
    Pool --> DB[(Database / cache)]
    App --> Net[Network egress]
    DB --> Sat{Which saturates first?}
    Sat -->|CPU| C[Scale compute / cut work]
    Sat -->|Pool / DB| D[Batch, shard, or raise pool carefully]
    Sat -->|Network| N[Compress, coalesce, CDN]
    Sat -->|Lock / hot key| H[Remove serial path]

Instrument stage latency, queue depth, pool wait, and saturation metrics (CPU%, disk util, NIC, lock wait). A high app CPU with idle DB means you need less work or more app capacity—not a bigger primary. A saturated connection pool with idle CPU means you are waiting on the next hop; raising threads without raising pool or shard count just burns RAM.

plaintext

Steady-state sanity (Little):
  L ≈ λ × W
  5,000 req/s × 0.08 s ≈ 400 in-flight
  If pool max = 100 to DB → you are oversubscribed:
    either cut W (cache, batch), raise pool carefully,
    or lower λ (admission) until queues stabilize

Saturation checklist:
  app CPU ~100%, DB idle     → app work / GC / crypto
  pool wait high, DB CPU high → DB bottleneck / missing index
  one shard CPU hot           → hot key / skew
  NIC saturated               → payload size / fan-out
  lock wait / singleflight    → Amdahl serial section

Batching and pipelining

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

sequenceDiagram
    autonumber
    participant P as Producer
    participant B as Batcher
    participant W as Writer
    participant S as Store

    P->>B: item
    P->>B: item
    Note over B: linger or max-size flush
    B->>W: batch of N
    W->>S: one round trip / one fsync
    par pipeline next batch
        P->>B: more items
    and
        S-->>W: ack
    end
    W-->>B: ready for next

wins when fixed cost dominates: Kafka produce linger, Redis pipelines, JDBC batch inserts, bulk Elasticsearch index, gRPC streaming. Choose linger and max bytes so p95 still meets the product promise. Infinite linger maximizes MB/s and breaks interactive writes.

keeps the wire and CPU busy by overlapping. Cap in-flight batches with a semaphore sized from Little’s Law and downstream capacity. Without a cap, “async” is an unbounded queue with a marketing name.

Technique	Raises	Risks
Producer linger / bulk write	Items/s, lower RPC/s	Added wait; larger failure blast
Request coalescing	Effective QPS under stampede	Correctness if results differ by caller
Pipeline / async fan-out	Utilization of wait-heavy stages	Head-of-line if depth unbounded
Group commit / group fsync	Durable writes/s	Durability window; tail latency

Parallelism: pools, shards, and Amdahl

Horizontal scale and multiply capacity only for the parallel fraction. A global lock, a single leader assigning ids, or a celebrity is the serial fraction in . Measure skew: p99 shard QPS versus median shard QPS. If one shard does 40% of writes, buying ten more replicas of the cold shards does nothing.

is parallelism with a budget. Size from concurrent need (Little) and from the database’s max connections ÷ number of app instances. Prefer fewer, busier connections with batched statements over a forest of idle sockets.

plaintext

Amdahl (speedup):
  S = 1 / (s + (1-s)/N)
  s = serial fraction, N = workers
  s = 0.2 → even infinite N caps at 5×

Hot-key mitigations (pick by write semantics):
  - local + regional cache for read-heavy keys
  - salt key into K buckets, re-aggregate async
  - write buffer / singleflight coalesce
  - move counter to approximate structure (HLL, Redis INCR sharded)

Pool sizing sketch:
  needed ≈ λ × W_db
  per_instance = floor(db_max_conn / instances × safety)
  set pool = min(needed_per_instance, per_instance)
  if needed > db_max_conn → batch, cache, or shard DB

Bytes on the wire and in memory

Compression (and binary protocols) raise effective MB/s when bandwidth or serialization CPU dominates. Measure both directions: CPU to compress versus bytes saved. At already-high CPU, compression can lower request throughput even as it lowers network utilization.

paths (sendfile for static files, buffer pooling, arena allocators, avoiding JSON re-encode) cut CPU per byte. Columnar and page-aligned formats help analytics throughput the same way. Prefer “do less work” over micro-optimizing a chatty schema.

Usually worth it

HTTP/2 or gRPC multiplexing, response compression for large JSON, CDN for cacheable GETs, protobuf/avro on internal high-QPS links, buffer reuse in proxies.

Often a trap

Compressing tiny messages, premature zero-copy rewrites, SELECT * over the wire, and N+1 chatty APIs that no codec can save.

Protecting throughput under overload

Peak capacity is not the same as overload behavior. When λ exceeds sustainable service rate, queues grow and useful throughput can fall (retry storms, GC, context switch thrash). and load shedding keep accepted work inside the stable region. propagates “slow down” to producers instead of buffering until OOM.

plaintext

Overload playbook:
  1. Cap concurrency / queue depth at each tier
  2. Shed low-priority or expensive routes first
  3. Bound retries with jitter; honor Retry-After
  4. Prefer fail-fast 503 over multi-second timeouts
  5. Watch goodput (successful useful work/s), not just QPS

Goodput example:
  Offered 20k QPS, 40% timeouts → ~12k “QPS” of waste
  Admit 12k, 99% success → ~11.9k goodput, healthier tails

Worked micro-estimate

plaintext

Write API: 8k events/s peak, each event 1.2 KB
  Naive: 1 INSERT + 1 fsync path per event
  DB: ~2k durable single-row commits/s → backlog grows

Batch of 64 with group commit every 10 ms:
  batches/s ≈ max(8000/64, 100) ≈ 125 (linger-bound)
  still within DB group-commit capacity
  added wait ≤ 10 ms — check against write SLO

Wire before compression: 8k × 1.2 KB ≈ 9.6 MB/s
  3× compress → ~3.2 MB/s (CPU cost must fit)

If one user_id is 15% of writes:
  shard capacity effectively ÷ skew
  salt into 16 buckets or coalesce writes for that key

Pool: W_db after batching ≈ 15 ms, λ=8k
  L ≈ 120 → size app↔DB concurrency near that, not 2k threads

Trade-offs to weigh

Batch size versus first-byte latency

Larger batches raise items/s and enlarge the failure and latency blast. Tie linger to the write SLO; offer a flush path for interactive paths that cannot wait.

More shards versus skew complexity

Sharding multiplies capacity when keys are even. Hot keys, cross-shard joins, and rebalancing cost operational complexity. Fix skew before you celebrate shard count.

Pool size versus database lifetime

Bigger pools raise app concurrency until the database melts. Cap on the scarcer resource, then batch or cache to lower required L.

Compression versus CPU headroom

Compression buys network and storage throughput and spends CPU. On CPU-bound services it can lower request throughput; measure goodput both ways.

Accept rate versus goodput

Accepting everything maximizes vanity QPS and can minimize successful work. Admission control trades refusal for stable goodput and intact latency SLOs for accepted traffic.

Choosing and explaining throughput optimization

A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.

Explain the core idea

Throughput is completed work per second at a named boundary; raise it by removing the bottleneck, not by guessing a cache.

Batching and pooling amortize fixed cost; sharding and parallelism help only for the parallel fraction and even keys.

Protect gains with backpressure and admission control so overload does not erase goodput.

When it belongs

The system falls behind arrivals while you still need higher sustained rate.

You must justify batching, sharding, pool sizing, or compression in a capacity plan.

Latency is fine for accepted work but queues or retries show the fleet is saturated.

How it fails

Optimizing a non-bottleneck stage while the real limiter saturates.

Unbounded batches, pools, or async depth that recreate queues and OOMs.

Hot keys and serial locks that make horizontal scale look like it failed.

Trade-offs to weigh

Batching and compression trade latency or CPU for higher items/s or MB/s.

More shards and pools trade operational complexity and downstream risk for parallel capacity.

Admission control trades some refusals for higher goodput under overload.

Common misunderstandings

Confusing high QPS with high goodput when timeouts dominate.

Assuming more replicas fix a single hot partition.

Ignoring Little’s Law when sizing pools and in-flight limits.

Wrap-up

Throughput optimization is bottleneck science plus amortization and controlled parallelism. Measure the limiter, cut fixed cost with batches and pools, split only even work, move fewer bytes, and refuse work you cannot finish. Pair this page with the latency-versus-throughput foundations note when you need percentile budgets and Little/Amdahl arithmetic in more depth.

Primary sources

Key takeaway

Raise completed work/s by fixing the real bottleneck, amortizing fixed cost, and parallelizing only what can scale—then admit and backpressure so goodput survives the peak.