System Design Foundations
Threading and Multithreading Models
Ten thousand open chat sockets and eight CPU cores. If each socket owns a kernel stack, the box spends its life switching. If one event loop also JPEG-encodes thumbnails, every ping waits on the encoder. The threading model is how those two facts share hardware.
The OS basics note maps processes, threads, and async I/O onto file descriptors and cgroups. This note is the catalog of mappings: 1:1 versus , versus , pools, virtual threads, actors, and the sizing math. Pick the model from the wait, then bound it from Little's Law and the downstream connection budget.
Same 10k connections
Thread per connection
10k kernel stacks. Context-switch storm. Simple code, poor density.
Admit only N blocking workers. Excess waits or 503s.
Few threads, many sockets. CPU work must leave the loop.
Blocking-style code, runtime parks on I/O. Still cap DB pools.
What actually happens
Many-to-one (historical green threads on a single kernel thread) multiplexes cheaply and dies on one blocking syscall. One-to-one maps each to a : pthreads, typical JVM platform threads. keeps thousands of tiny tasks and a handful of carriers. Runtimes that look different in marketing—goroutines, virtual threads, Erlang schedulers—are M:N with different preemption stories.
Request handling then picks a programming style on top: blocking pools, reactors, proactors, or actors. Style does not erase Little's Law. 2,000 QPS at 200 ms still means ~400 in-flight tasks. The model only changes whether those 400 are 400 stacks of 1 MB or 400 4 KB user stacks on 8 carriers—and whether a ninth blocking JDBC call can still stall someone who should have been independent.
How to choose a threading model
1. Mostly sockets waiting? Reactor, proactor, or M:N virtual threads. Do not pay a kernel stack per idle connection.
2. Blocking libraries you cannot change? A dedicated (or virtual threads that unmount) around that library. Never run it on the accept loop.
3. CPU-heavy or GIL-bound? Process pool or native workers. Extra Python threads will not give extra bytecode cores.
Simple rule of thumb
I/O density → async or M:N. Blocking drivers → isolated pool. CPU → cores-sized workers. Isolate pools by wait type. Size each with and then cut to the downstream max connections.
1:1 versus M:N
Read the diagram definition
flowchart TB
Q[Runnable user tasks]
Q --> S[Runtime scheduler]
S --> K1[Kernel thread / carrier 1]
S --> K2[Kernel thread / carrier 2]
S --> K3[Kernel thread / carrier 3]
K1 --> CORE1[CPU 0]
K2 --> CORE2[CPU 1]
K3 --> CORE3[CPU 2]
K1 -.->|blocking syscall unmount| PARK[Parked task]
PARK --> Q
| Mapping | Strength | Failure mode |
|---|---|---|
| Many:1 | Tiny tasks, simple runtime | One blocking syscall freezes all tasks |
| 1:1 | True overlap of blocking I/O; OS preemption | Stack + switch cost; oversubscription |
| M:N | Density of many:1, overlap of 1:1 | Pinning, hidden blocking, scheduler bugs |
Preemption matters. Cooperative user threads that never yield starve others (old Node-style CPU hog, or a goroutine in a tight loop before Go added better preemption). Virtual threads that stay mounted while holding a native mutex pin a carrier: you accidentally rebuilt many:1 for everyone sharing that carrier.
Reactor, proactor, and the forbidden long loop
Read the diagram definition
sequenceDiagram
autonumber
participant L as Event loop
participant K as Kernel epoll
participant D as DB pool
participant C as CPU worker pool
L->>K: Wait for ready fds
K-->>L: Socket readable
L->>L: Parse HTTP, auth
alt I/O bound query
L->>D: Async or hand off blocking JDBC
D-->>L: Completion callback
else CPU encode
L->>C: Offload
C-->>L: Future done
end
L->>K: Write response when writable
A demultiplexes readiness: “this fd can read.” The application does the read. A demultiplexes completion: “your read finished into this buffer.” Both are how you get C10k on a few threads. Both fail the same way: a 50 ms JSON parse or image resize on the loop thread adds 50 ms to every other connection on that loop. The OS basics note states the hybrid fix; here the implementation is a second pool with its own queue and timeout, plus metrics for loop lag (time between event and handler start).
Loop health (reactor):
event_wait → callback_start = loop lag
callback_start → callback_end = handler CPU
If handler CPU p99 > 1 ms on a shared loop,
you are already taxing neighbors.
nginx / Node / Netty: many connections, tiny callbacks
CPU work: bounded worker pool, never unbounded Thread.startPool sizing and the blocking factor
CPU pools want about one runnable worker per core (plus a little for stalls). I/O pools want more because workers are parked. The textbook formula is a starting point, not a license to open 500 connections to Postgres.
Little: L = λW
400 QPS × 0.040 s DB = 16 in-flight DB calls
Blocking-factor sketch:
threads ≈ cores × (1 + wait/compute)
8 cores, wait 36 ms, compute 4 ms → 8 × (1+9) = 80
Reconcile with the database:
30 app replicas × 80 = 2,400 connections (usually disaster)
Downstream cap 900 → per replica ≈ 20–25
Admit only 25 blocking DB workers; queue or shed the rest
Split pools:
cpuPool = cores (encodes, crypto)
jdbcPool = min(formula, DB budget / replicas)
httpClient = separate cap (do not share jdbc threads)One pool, two wait types
If image encode and JDBC share a pool of 32, a burst of encodes occupies every worker and checkouts cannot get a DB thread. Bulkheads exist so one wait class cannot eat the other. Actors and event loops need the same idea: a stuck mailbox is a bulkhead failure.
Virtual threads, goroutines, and pinning
A (or goroutine) is cheap because the runtime owns the stack growth and the park points. You can spawn one per request again—without 1 MB stacks—as long as every blocking call is a known park point. Synchronized blocks around native I/O, uninstrumented syscalls, and CPU hogs pin or hog carriers. You then have eight kernel threads and 50,000 tasks, of which eight are runnable forever and the rest never get a carrier.
Goroutines
Java virtual threads
Work stealing
Actors, CSP, and the GIL
Actors and CSP
An serializes its own mailbox. CSP channels pass ownership of messages. Both push shared-memory races out of the default path. Design pressure moves to mailbox bounds, deadlocks in request-reply cycles, and hot keys that hash to one actor. Shard on identity; supervise children; do not unbounded-buffer.
GIL and processes
CPython's means threads help I/O overlap and hurt CPU overlap. Prefork workers (gunicorn-style) or a native/Rust/Go sidecar take the CPU path. Async Python (asyncio) is a reactor in one thread: same “do not block the loop” rule, plus watch for sync SDK calls that freeze everyone.
Worked example: chat, transcode, mixed API
Three services, three models. The mistake is copying one model into all three because the company “standardized on threads.”
Chat fanout (100k idle sockets, tiny messages):
Model: reactor or M:N
Carriers ≈ cores
Cap: memory per connection, not thread count
Danger: JSON broadcast on the loop; offload fanout batches
Video transcode (CPU minutes per job):
Model: 1:1 workers ≈ cores (or GPU workers)
Queue jobs; never one thread per upload socket doing encode
Danger: work-steal of 10-minute tasks; isolate encode pool
Mixed HTTP API (JDBC + light JSON):
Model: virtual threads or modest JDBC pool + async server
jdbcPool sized from L = λW and DB max_connections / replicas
cpuPool for reports
Danger: shared executor; report query occupies checkout workersStructured concurrency belongs on the mixed API: a checkout scope starts inventory, pay, and fraud children, cancels siblings on failure, and cannot return until the tree is silent. Leaked background threads after the HTTP response are how you double-charge on retry.
Hazards the model does not erase
| Hazard | Looks like | Design move |
|---|---|---|
| Deadlock | Pools waiting on each other (A needs B's thread) | Never call pool B while holding pool A; timeouts |
| Starvation | Low-priority tasks never run | Separate pools; avoid unbounded FIFO with giants |
| Priority inversion | High-priority waits behind a lock held by idle low | Priority inheritance, or do not mix priorities on one lock |
| Livelock | Everyone retries, nobody progresses | Jittered backoff, retry budgets (see isolation note) |
| Oversubscription | Runnable threads ≫ cores; CPU cache dies | Cap CPU pools at cores; park I/O instead |
Trade-offs to weigh
1:1 blocking style
Easy to read, easy to explode. Fine for CPU workers and small services. Bad as the only model for huge idle connection counts.
Reactor density
Superb for idle sockets. Callbacks and offload boundaries are easy to get wrong. Loop lag is the SLO.
M:N / virtual threads
Blocking syntax with high density. Pinning, unaware native libs, and unbounded spawn remain. Connection pools still rule.
Actors
Clear isolation and supervision. Hot mailboxes and unbounded queues recreate the problems you left shared memory to escape.
Choosing and explaining threading models
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
Name 1:1 versus M:N, then say whether the request path is a reactor, a pool, or virtual threads.
Size CPU pools near cores and I/O pools from Little’s Law, then cut to the downstream connection cap.
Keep wait classes in separate bulkheads so JDBC and encode cannot starve each other.
When it belongs
C10k connections, mixed blocking drivers, or a GIL language enters the design.
Someone proposes thread-per-request at high idle concurrency.
You must offload CPU from an event loop.
How it fails
Event-loop stalls from CPU or sync I/O.
Thread-pool deadlock between two executors.
Virtual-thread stampede of a 20-connection database.
CPython threads expected to scale CPU.
Trade-offs to weigh
Kernel-thread simplicity versus stack and switch cost.
Callback density versus blocking-style M:N complexity.
Actor isolation versus hot-mailbox bottlenecks.
Common misunderstandings
Treating virtual threads as unbounded database connections.
One shared executor for CPU and blocking I/O.
Assuming more threads mean more parallelism on a GIL.
Forgetting that M:N still needs preemption and park points.
Wrap-up
Map user tasks onto kernel threads on purpose. Density for waits, isolation for CPU, caps everywhere a downstream has a hard limit. The OS still schedules the carriers; you still own the queues.
Exercises
- Size three pools for a service that does 300 QPS of 30 ms JDBC, 20 QPS of 200 ms encodes, on 8 cores, 25 replicas, DB max 800 connections.
- Pinning. A virtual-thread service holds a synchronized lock around a DNS call. What happens to carriers, and how would you see it?
- Pick a model for 50k idle WebSockets plus occasional thumbnail CPU. Sketch loop, workers, and what you measure for loop lag.
Primary sources
- Kegel: The C10K problem : why thread-per-connection broke, and readiness I/O.
- Oracle: Virtual threads (Java) : M:N carriers, pinning, and when platform threads remain.
- Go blog: Goroutines : user tasks on a small set of OS threads.
Key takeaway
Threading models are mappings from tasks to kernel schedulable entities. Use 1:1 where blocking and CPU isolation matter, reactors or M:N where connections idle, and always bulkhead wait classes against a real connection budget. Virtual threads change stack cost, not Little's Law.