System Design Internals
Memory Management Concepts
A cache that “fits in 16 GB” still misses the CPU on every pointer chase. A JVM that “has enough heap” still takes a 40 ms pause and blows a 20 ms SLO. Memory management for system design is not malloc trivia: it is latency, density, and blast radius.
This note covers , request , locality, , and caches—the knobs that show up when you size replicas and explain p99 cliffs.
Where the bytes go
Request junk should die in the nursery.
Bump-alloc, reset at request end.
Pack hot scans; avoid pointer soup.
Fit RAM/cgroup or pay page faults.
What actually happens
A handler allocates headers, JSON trees, and temporary buffers. In a managed runtime those objects land in a nursery. If nothing retains them, a minor collection reclaims the whole nursery cheaply. If you stuff them into a global map “just in case,” they promote, the old generation grows, and later collections scan a larger live set.
Native services (C++, Rust, custom stores) often use an per request or per RPC: pointer bump, then reset. Databases mix arenas, slab allocators, and buffer pools so the of pages stays hot.
Throughput designs care about and fragmentation. Latency designs care about pause, cache misses, and hops. Density designs care whether 8 replicas of a 6 GB heap fit the node after native overhead.
How to choose a memory strategy
1. Request-scoped junk? Arenas or short-lived GC objects. Do not intern it.
2. Multi-GB cache? Prefer or a dedicated cache process so the GC does not scan every cached byte.
3. Tight p99? Budget STW, huge-page TLB, and allocation rate. Throughput GCs that pause 100 ms are the wrong collector for user-facing RPCs.
Simple rule of thumb
Size for live set + headroom, not peak allocation. Keep request garbage ephemeral. Pack data you scan. Put huge byte arrays off-heap or in a cache service. Treat cgroup memory as the real ceiling.
Allocation and promotion
Read the diagram definition
flowchart TD
Req[Request starts] --> A[Allocate hot objects]
A --> Y[Young / nursery]
Y --> Die{Reachable after request?}
Die -->|No| Free[Reclaim in minor GC / arena reset]
Die -->|Yes, still reachable| Old[Promote to old gen / global heap]
Old --> Major[Infrequent major / concurrent mark]
Major --> Live[Still live: cost of scanning]
Live --> OldGenerational intuition:
survive_minor ≈ objects still reachable at minor GC
promotion_rate high → old gen grows → concurrent mark / full GC more often
Nursery sizing:
too small: constant minor GCs, CPU tax
too large: longer minor pauses, worse locality after the collection
Live-set math for a replica:
heap_need ≈ live_set × (1 + fragmentation) + nursery + headroom
cgroup_need ≈ heap_need + native + thread stacks + page cache you still want
If live_set is 4 GB and you set -Xmx 4g:
first burst promotes, then Full GC / OOM
leave 30–50% headroom unless you have measured otherwiseGC pauses become queueing
Read the diagram definition
sequenceDiagram
autonumber
participant U as Users
participant App as App threads
participant GC as Collector
U->>App: 4k req/s, 8 ms work
Note over App: In-flight ≈ 32
GC->>App: STW pause 40 ms
Note over App: Threads frozen; arrivals queue
U->>App: extra ~160 requests during pause
App->>App: restart + catch-up alloc spike
GC->>App: another young GC
Note over U,GC: Tail latency is pause + queueing, not pause aloneInterview line: “I would pick a low-pause collector (or C++/Rust arenas) for the user-facing path, keep allocation rate down, and isolate batch jobs so a throughput-oriented heap does not sit on the same SLO.” Concurrent marking still uses CPU: you trade pause for cores.
| Collector style | Fits | Watch |
|---|---|---|
| Throughput / parallel STW | Batch, offline, analytics | Multi-100 ms pauses |
| Concurrent low-pause | User RPCs, gateways | Mark CPU, allocation spikes still pause |
| Reference counting | Predictable, cyclic-graph caution | Cycles, atomic refcount ping-pong |
| Manual + arenas | Stores, proxies, HFT-ish paths | Lifetime bugs if pointers escape |
Arenas: free in one stroke
RPC servers, game frames, and parsers love arenas. Locality is excellent: objects allocated together are used together. Fragmentation inside the arena is mostly “we reserved 2 MB and used 700 KB,” which you cap by pooling arenas of fixed size.
The contract is lifetime. If you stash a request-arena string in a long-lived cache, the next reset corrupts it. Copy-out (or intern on a different allocator) anything that must survive. Mixing GC heaps with native arenas means you also pin or copy when crossing the boundary.
Request arena:
buf = mmap or pooled 2 MB slab
alloc(n) = bump; fail if bump+n > end
reset() = bump = start // O(1)
Patterns:
- per-request arena (HTTP)
- per-message arena (queue consumer)
- generation arena (swap two regions: allocate in A, freeze, allocate in B, drop A)
Do not:
- return interior pointers to callers that outlive the handler
- hold mutexes while other threads allocate from the same unsynchronized arena
- assume reset zeroes memory (explicitly clear secrets)Cache locality and false sharing
Read the diagram definition
sequenceDiagram
autonumber
participant C0 as Core 0
participant L as Cache line 64B
participant C1 as Core 1
Note over L: counterA at offset 0, counterB at offset 8
C0->>L: increment counterA
C1->>L: increment counterB
L-->>C0: line invalidated
L-->>C1: line invalidated
Note over C0,C1: False sharing: two logical counters, one physical lineData-oriented layout (struct-of-arrays for scans, array-of-structs for per-entity access) is a storage-engine concern that leaks into API design: a row-oriented JSON blob that you always filter by one field is a cache-miss machine. Columnar layouts, packed protobufs, and flatbuffers exist to keep the scan in cache.
shows up in “I sharded my counter” designs that still pack twelve atomics into one object. Pad to 64 bytes, or use thread-local counters and harvest.
Miss cost intuition (order-of-magnitude, not a spec):
L1 hit ~ 1 ns
L3 hit ~ 10–20 ns
DRAM ~ 70–100 ns
NUMA remote ~ extra tens of ns + interconnect
Major fault ~ milliseconds
Layout:
sequential 64-byte records × 1M ≈ 64 MB working set
pointer graph of 1M objects with 24 B headers + random next
→ more bytes, worse prefetch, GC must visit each object
False sharing fix:
struct alignas(64) PaddedAtomic { atomic<uint64_t> v; };Off-heap caches and huge pages
A 12 GB on-heap cache forces the collector to visit (or at least card-mark) a huge live set. Moving values or to Redis/memcached keeps the process heap to code + request garbage. You then need bounds, eviction, and CRC/checksums yourself.
When off-heap helps
- Large byte[] / String caches
- Network buffer pools (Netty, Envoy-style)
- mmap of immutable data (features, FST)
- You can bound and checksum the region
Costs
- Manual lifetime; double-free / leak
- Serialization to get objects in/out
- RSS still counts against the cgroup
- Huge pages: fewer TLB misses, harder reclaim, startup cost
Pools, reuse, and allocation rate
Buffer pools for I/O are usually worth it. Pooling tiny POJOs in a generational GC often loses: you pay bookkeeping and still confuse escape analysis. Measure with a profiler before you invent a pool.
Unbounded pools are caches in disguise and become the live set. Cap them. Clear sensitive fields on release. Never put a pooled object into a structure that outlives the checkout.
NUMA and multi-socket replicas
On a two-socket host, a thread on socket 0 touching memory allocated on socket 1 pays interconnect latency and can saturate QPI/UPI before DRAM. Latency-critical stores pin reactor threads, allocate from the local node, and avoid a single shared allocator lock. Most CRUD APIs on one socket VMs can ignore this until profiles show remote DRAM.
When NUMA starts to matter:
- multi-socket bare metal / large VMs
- in-process caches of tens of GB
- very high QPS with small payloads (overhead dominates)
Design moves:
- one replica per socket rather than one replica spanning both
- partition the cache by shard pinned to a loop
- avoid a global malloc lock; per-thread caches (jemalloc/tcmalloc)SD use cases
User-facing API
In-process cache vs Redis
LSM / buffer pool
Stream processors
Worked example: cache killed p99
Team adds an on-heap Caffeine/Guava cache of 8 GB of JSON strings to “save Redis.” Heap is 12 GB. Young GCs stay short; old-gen concurrent mark uses 30% CPU; occasional remark pauses hit 25–40 ms. p99 SLO is 30 ms.
What happened:
live_set ≈ cache + 1 GB request junk
collector must mark 8 GB of Strings
allocation rate unchanged → frequent young GC + expensive marking
STW remark + queueing > SLO
Repair options (pick with numbers):
1. Move cache off-heap / to Redis; heap back to ~2–3 GB live
2. Cache compact bytes (protobuf), not JSON strings
3. Cap cache by RSS, not entry count
4. Isolate cache in its own process (failure domain + GC domain)
5. If you keep on-heap: larger heap, pause-sensitive collector, lower hit-rate target
Do not:
- “just raise -Xmx” without raising cgroup (OOMKill)
- pool JSON trees that still sit in old genTrade-offs
| Choice | Gain | Cost |
|---|---|---|
| Managed GC heap | Safety, fast iteration | Pauses, mark CPU, promotion traps |
| Request arenas | O(1) free, locality | Escape bugs; pooling complexity |
| Off-heap cache | GC ignores payload | Manual lifetime, copy in/out |
| Packed / columnar layout | Scan speed, smaller RSS | Awkward random field access |
| Object pooling | Lower alloc rate for fat buffers | Leaks, stale data, live-set growth |
| Huge pages | Fewer TLB misses | Reclaim pain; startup; waste |
Choosing and explaining memory management in system design
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
Talk about live set, allocation rate, and pause—not only “we have enough RAM.”
Request garbage should die young or in an arena; large caches should not sit on the GC’s critical path.
Locality (cache lines, packing) and false sharing are why two designs with equal RAM have unequal p99.
When it belongs
Sizing heaps, cache replicas, or cgroup limits.
Explaining latency cliffs correlated with GC or page faults.
Choosing in-process vs sidecar cache, or GC vs native store.
How it fails
Promotion of request objects into a huge old generation.
STW pause plus arrival queueing blowing the SLO.
Working set exceeding RAM/cgroup → fault storm / OOMKill.
False sharing on hot counters; pointer-chasing graphs.
Off-heap leaks that still kill the pod.
Trade-offs to weigh
GC safety versus pause and mark CPU.
Arena speed versus lifetime discipline.
On-heap convenience versus off-heap isolation.
Packed scans versus flexible object models.
Common misunderstandings
Equating host free memory with the container’s limit.
Raising -Xmx without raising the cgroup or leaving nursery headroom.
Calling every latency spike “GC” without pause logs.
Pooling everything in a generational runtime as a reflex.
Wrap-up
Memory architecture is how long a request may live in RAM, how packed its bytes are, and who pays to reclaim them. Put that in the design before you promise a p99.
Exercises
- Size a heap. Live set 3.2 GB, nursery 512 MB, native 800 MB, cgroup 8 GB. Propose -Xmx and say what happens if the cache grows 2 GB.
- Pause math. 5k req/s, 40 ms STW. How many extra in-flight requests arrive, and how could that trigger another GC?
- Layout. A feed ranks 10k items by one int score. Contrast array-of-objects vs packed score array plus id array for cache behavior.
Primary sources
Key takeaway
Design the live set. Let request garbage die in a nursery or an arena, keep huge caches off the collector’s path, and pack data the CPU will scan. p99 is often memory behavior wearing an application costume.