System Design Internals
Asynchronous Programming Concepts
A gateway holds 40k open connections. Almost none of them are doing CPU; they are waiting on sockets. If each wait occupied a thread, the process would drown in stacks and context switches. An plus keeps a handful of threads busy while thousands of requests sit cheaply in memory.
This note is about how a service waits inside one process: loops, pools, , and . It complements async vs sync communication (queues, 202, DLQs) without repeating that product-level choice.
Inside one replica
Park on readiness, not on a stuck thread.
Resume short continuations; offload CPU.
Blocking DNS, disk, native libs live here.
Fan-out dies with the request deadline.
What actually happens
A request handler hits await db.query(). The runtime captures a , registers the socket, and returns the loop to other ready work. When the kernel signals readability, the continuation runs—often on a different thread than the one that started the request.
That is a different axis from “should checkout email go on a queue?” You can have a fully synchronous HTTP API implemented with async I/O inside the replica, or an async product path whose workers still block a thread per job. Mixing them up in an interview sounds like you only memorized “async scales.”
Capacity still obeys Little’s Law: concurrent in-flight ≈ arrival rate × latency. Async changes the cost per waiter (stack vs heap task), not the number of in-flight operations the downstream can absorb. Without , you just OOM later with prettier syntax.
How to choose a wait model
1. Mostly socket waits? Prefer an or a modest async runtime. Thread-per-connection at 10k waits is usually the wrong tax.
2. Blocking libraries you cannot replace? Isolate them on a sized . Never call them on the loop.
3. CPU in the request? Offload encode/compress/JSON-of-doom. The loop is ; a 30 ms burst is a latency incident for everyone else on that core.
Simple rule of thumb
Async I/O on the hot path, bounded pools for blocking/CPU, a deadline on every wait, and a scope so children cannot outlive the request. If you cannot cancel it, you cannot safely hedge or time it out.
Event-loop request path
Read the diagram definition
sequenceDiagram
autonumber
participant C as Client
participant L as Event loop
participant K as Kernel epoll
participant DB as Database
C->>L: Accept connection
L->>K: Register socket
L->>L: Parse request (short CPU)
L->>DB: Non-blocking query start
L->>K: Wait for DB socket
Note over L: Loop serves other clients
K-->>L: DB socket ready
L->>DB: Read rows
L-->>C: Write response
Note over L: Thread count stays tinyInterview usefulness: when connections are high and CPU is low, you are I/O-bound and an evented model is the default. When CPU is high and connections are modest, threads or a CPU pool dominate. Measure both; slogans do not size pods.
The blocking-on-the-loop failure
Read the diagram definition
sequenceDiagram
autonumber
participant A as Request A
participant Loop as Loop thread
participant B as Request B
participant DNS as Blocking DNS
A->>Loop: Handler starts
Loop->>DNS: getaddrinfo (blocks 80ms)
B->>Loop: Socket ready
Note over Loop,B: B waits: loop is stuck in DNS
DNS-->>Loop: Returns
Loop-->>A: Continue
Loop-->>B: Finally runs
Note over A,B: p99 of B is now A's blocking callSymptoms that the loop is blocked:
- event-loop lag / libuv lag / reactor delay climbing
- p99 rises while process CPU looks “fine” (one core pegged)
- other connections idle in ESTABLISHED with data ready
- thread dump shows the loop in a syscall, not epoll_wait
Typical culprits:
- DNS via blocking libc
- JDBC / ORM used from an async framework without a pool
- gzip, JSON, crypto on huge payloads
- synchronized / file I/O / random native SDK
Fix:
- async DNS / Happy Eyeballs clients
- dedicated blocking executor with a queue bound
- time-box CPU slices; reject oversized payloads
- never hold a mutex across an await (continuation may resume elsewhere)Fan-out needs a scope
Read the diagram definition
flowchart TD
Req[Incoming request] --> Scope[Task scope / structured wait]
Scope --> Auth[Auth future]
Scope --> Inv[Inventory future]
Scope --> Price[Price future]
Auth --> Join[Join with deadline]
Inv --> Join
Price --> Join
Join --> Cancel[On timeout: cancel siblings]
Cancel --> Resp[Respond or fail closed]Unstructured go func() / Promise.all without cancellation leaks work: the user already got 504, but three downstream calls keep running and retrying. That is how a timeout storm is born inside one replica.
makes the join the API: the scope cannot close until children finish or are cancelled. Pair it with a remaining-time deadline so auth + inventory + price cannot each take the full 200 ms budget.
Capacity math for in-process async
Little's Law still holds:
L = λ × W
8k req/s × 25 ms = 200 in-flight requests (mean)
Cost per waiter:
Thread model: ~1–8 MB stack + kernel thread
10k waiters → gigabytes of stacks (usually fatal)
Async task: tens of KB of captured locals + socket buffers
10k waiters → tens–hundreds of MB (often viable)
You still pay:
- per-connection RAM (TLS buffers, HTTP parser)
- downstream pool slots (DB max_connections)
- file descriptors
- timer heap / task-queue CPU
Design:
concurrent_in_flight_cap = min(
memory_budget / bytes_per_conn,
fd_limit_headroom,
downstream_pool_budget,
product_latency_SLO via shedding
)| Knob | If too small | If too large |
|---|---|---|
| Loop count | One core saturates; HOL blocking | Cache-cold hops; extra context switches |
| Blocking pool | Queue delay; timeouts | Oversubscribe CPU; smash DB |
| Accept backlog | SYNs dropped under burst | Admitted work already doomed |
| In-flight cap | Under-utilization | Latency cliff, then OOM |
Cancellation, timeouts, and hedging
Propagate a deadline (absolute time, not a fresh 200 ms at every hop). When the parent cancels, close the outbound socket or send HTTP/2 RST, abort the DB query if the driver supports it, and stop retries. Hedged requests (send a second copy after p95) without cancel-on-first-success double the downstream load during the exact moment it is already slow.
Cooperative means you must poll the token between chunks of CPU and use interruptible I/O. A tight hash loop will ignore the deadline until it finishes hashing.
Deadline remaining-time (serial hops):
budget = 200ms, three serial awaits
hop1 deadline = now+200
hop2 deadline = original budget - hop1 elapsed
hop3 ...
If remaining < min_useful_work: fail fast, do not start hop3
Hedging:
start primary
after delay d (often p95), start hedge
first success cancels the other
both count against retry budget
Never:
timeout the inbound request while leaving outbound RPC running
Promise.race without aborting the loserIn-process backpressure
Async pipelines (parse → enrich → write) need bounded channels. Unbounded queue.add converts a slow sink into RSS growth. Credit-based streams (reactive pull, TCP-like windows) are the same idea as service-level load shedding, one layer down.
Signals the pipeline is drowning
- Task-queue depth climbing while QPS is flat
- Allocation rate / GC time rising with lag
- Downstream pool wait, not business CPU
- Event-loop delay correlating with queue depth
Typical fixes
- Bounded queue + drop/503 when full
- Slow down acceptors (listen backlog, 429)
- Separate pools per dependency (bulkhead)
- Apply backpressure to websocket ingest
Models you will actually ship
Reactor / Netty / Node / asyncio
Proactor / IOCP / io_uring completions
Virtual threads / goroutines / fibers
Actors / single-threaded mailboxes
Worked example: “async API still times out”
A Node/Netty service is rewritten from a servlet thread pool to async. Load test still fails p99. The handler still calls a blocking Redis client and JSON-parses 2 MB bodies on the loop.
Before rewrite:
200 threads × 40 ms wait = 8k in-flight theoretical
Reality: context-switch tax, 512 MB of stacks
After naive async:
40k sockets accepted
loop blocked 15–40 ms per fat JSON
event-loop lag ≈ 200 ms under peak
downstream Redis still 1 connection? or a huge pool?
Repair:
1. Move Redis to async client; cap pipeline depth
2. Bound body size; parse off-loop if large
3. In-flight semaphore = 2k (from λW + headroom)
4. Cancel outbound on client abort
5. Track loop lag as an SLO, not only HTTP latencyTrade-offs
| Choice | Gain | Cost |
|---|---|---|
| Event loop | Huge connection density | HOL blocking; hard to mix blocking libs |
| Thread-per-request | Simple blocking code | Poor wait density; switch storms |
| Virtual threads | Blocking style, cheap waits | Pinning, still need caps |
| Wide in-process fan-out | Lower latency via parallelism | Amplifies downstream; cancel is mandatory |
| Unbounded async queues | Smooths short blips | Turns slowness into OOM |
| Hedged RPCs | Cuts tail latency | Extra load unless losers cancel fast |
Choosing and explaining asynchronous programming in services
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
Async programming multiplexes waits inside a process so thousands of socket waits do not need thousands of OS threads.
It is not the same as async communication: a sync HTTP API can still use an event loop, and a queue worker can still block.
Safety comes from bounded in-flight work, offloading CPU/blocking calls, deadlines that cancel children, and structured scopes.
When it belongs
Sizing gateways, websocket fans, or I/O-bound APIs.
Explaining event-loop lag, thread-pool exhaustion, or timeout storms after a “we went async” rewrite.
Designing fan-out of dependent RPCs with a remaining-time budget.
How it fails
Blocking or heavy CPU on the loop → head-of-line stalls.
Fire-and-forget tasks that outlive the request and keep hitting dependencies.
Unbounded pipelines that OOM under a slow sink.
Timeouts that do not cancel, especially with hedging.
Trade-offs to weigh
Connection density versus programming complexity and HOL risk.
Blocking-style virtual threads versus explicit futures/callbacks.
Parallel fan-out latency versus downstream amplification.
Common misunderstandings
Saying “async” when you mean a message queue.
Assuming await magically adds capacity.
Holding locks across await (resume on another thread).
Copying Node/Netty slogans without an in-flight cap and loop-lag SLO.
Wrap-up
Treat the concurrency runtime as part of the architecture: how you wait, what you cap, and what you cancel. Product-level async (queues, 202) is a different decision with its own durability rules.
Exercises
- Split the wait. A handler does auth RPC, Redis GET, and a 15 ms image thumbnail. Sketch loop vs pool placement and a 100 ms deadline split.
- Cancel the hedge. Primary + hedge after 50 ms. Draw what happens if the loser is not aborted, at 2k req/s during a 30% slowdown.
- Bound the pipeline. Parser can produce 50k events/s; writer does 10k/s. Pick a queue bound from a 200 ms lag SLO and say what you return when full.
Primary sources
Key takeaway
Asynchronous programming makes waits cheap inside a replica. It does not replace bounds, cancellation, or the separate choice of sync vs async communication. Keep CPU and blocking calls off the loop, join children in a scope, and cap in-flight work before memory does it for you.