System Design Foundations

Containerization Concepts

Containers package an application with its libraries so “works on my machine” becomes “works on this host kernel.” They are not miniature virtual machines by default: they isolate processes with Linux primitives, then rely on orchestration to place, heal, and expose those processes at scale.

This note explains , , , , layering, resources, and how changes design assumptions—without turning into a single-vendor tutorial.

From build to run

Layered filesystem + entrypoint, pinned by digest.

Private PID, net, and mount views on a shared kernel.

CPU/memory limits become real failure modes.

Schedule, health-check, roll, and rediscover.

What actually happens

When you run a , the runtime creates namespaces, applies cgroup limits, sets up a from image layers plus a writable top layer, then starts the entrypoint process. Networking is usually a virtual interface into a bridge or CNI plugin—not magic privacy from the host.

Compared with VMs, containers start faster and pack denser because they skip a guest kernel. Compared with bare processes, they ship dependencies and give weaker-but-useful isolation. Stronger isolation (microVMs, gVisor-like sandboxes, Kata) exists when the threat model demands it—mention them when multi-tenant untrusted code appears.

At fleet scale, decides which node runs which container, restarts crashes, and rolls new images. Your app must tolerate death, reschedule, and changing IPs.

How to choose containers vs VMs vs bare metal

1. Packaging and density? Stateless services and sidecars thrive in containers. Kernel modules, custom schedulers, or extreme isolation may need VMs or bare metal.

2. Trust boundary? Untrusted customer code often needs stronger than default container isolation. Same-team microservices usually accept shared-kernel density with good IAM and network policy.

3. Stateful systems? Databases can run in containers with careful storage classes, but many teams keep them on managed services or VMs to reduce blast radius—state the reason explicitly.

Simple rule of thumb

Use containers for portable app packaging and orchestrated fleets. Treat cgroup limits as capacity truth. Pin image digests, keep state off ephemeral layers, and design for rescheduling from day one.

Images, layers, and registries

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

flowchart TB
    BASE[Base layer: minimal OS]
    DEPS[Deps layer: language runtime / libs]
    APP[App layer: binaries / config]
    META[Config: entrypoint, env, user, ports]
    REG[Registry]
    NODE[Node image cache]
    RUN[Running container writable layer]

    BASE --> DEPS --> APP --> META
    META --> REG
    REG --> NODE --> RUN

Fat images slow pulls and widen the CVE surface. Multi-stage builds keep compilers out of the runtime image. Run as non-root when possible; drop Linux capabilities you do not need.

belongs in design interviews now: digest pins, admission controllers that reject unsigned images, and scanned bases. Floating :latest is how Friday’s deploy becomes Monday’s mystery binary.

Namespaces and cgroups

Primitive	Isolates / limits	Design implication
PID namespace	Process IDs	PID 1 responsibilities; zombie reaping
NET namespace	Interfaces, ports, routes	Localhost shared with sidecars in a pod
MNT namespace	Mounts / filesystem view	Volumes vs image layers
USER namespace	UID mapping	Root-in-container ≠ root-on-host when mapped
CPU cgroup	Shares / quotas	Throttle → latency under “idle” host CPU
Memory cgroup	RSS / working set caps	OOMKill; size heaps to the limit

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

sequenceDiagram
    participant Sched as Orchestrator scheduler
    participant Kubelet as Node agent
    participant Runtime as Container runtime
    participant Kernel as Host kernel
    participant App as App process

    Sched->>Kubelet: Assign pod to node
    Kubelet->>Runtime: Pull image digest, create container
    Runtime->>Kernel: Create namespaces + cgroups
    Runtime->>Kernel: Set up layered mounts + net iface
    Runtime->>App: exec entrypoint as PID 1 in namespace
    Note over App,Kernel: Memory limit hit → OOMKill → orchestrator restarts

Requests, limits, and noisy neighbors

influence placement; enforce hard caps. Mis-set either and you get eviction, throttle, or stranded capacity.

plaintext

Packing intuition:

  Node: 8 CPU, 32 GiB
  Pod A request 2 CPU / 4 GiB, limit 4 CPU / 8 GiB
  Pod B request 2 CPU / 4 GiB, limit 4 CPU / 8 GiB
  ...

  Scheduler packs by requests (roughly).
  At runtime, bursts up to limits compete.
  If many pods burst together → CPU throttle / mem OOM.

Design habits:
  - Set requests near steady p95 usage
  - Set memory limits with headroom above heap + native
  - Separate batch from latency-sensitive nodes or priorities
  - Watch throttling metrics, not only host idle CPU

Pods, sidecars, and shared fate

A shares the network namespace with the app—localhost calls to a proxy are the classic pattern. An runs first for setup. Both consume CPU/memory requests that the scheduler must honor.

Shared fate means if the pod is evicted, every container goes. That is desirable for “proxy must die with app” and painful if you stuffed an unrelated batch job into the same unit.

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

flowchart LR
    subgraph POD["Pod / scheduling unit"]
      INIT[Init: migrate wait]
      APP[App container]
      SIDE[Sidecar: proxy / logs]
    end
    VOL[(Volume: config / secrets)]
    NET[Shared network namespace]
    INIT --> APP
    APP --- SIDE
    VOL --> APP
    VOL --> SIDE
    APP --> NET
    SIDE --> NET

Storage and networking realities

Storage

vanishes with the container/node
Durable volumes need explicit lifecycle and backup stories
Do not write critical state only to the writable image layer
Image layers + emptyDirs compete for node disk

Networking

Pod IPs change; clients need stable service discovery
Network policies are soft isolation, not a full security boundary
Egress and DNS failures dominate “my container can’t start” tickets
Service meshes add sidecars—budget CPU and complexity

Health checks must match reality: a process that accepts TCP but cannot reach its database is not ready. Liveness that is too aggressive kills healthy pods under load; readiness that is too loose sends traffic into death.

Orchestration changes the contract

Once an orchestrator owns placement, your binary is cattle. Graceful shutdown, idempotent startups, and externalized config stop being optional polish.

plaintext

App contract under orchestration:

  - Listen for SIGTERM; drain; exit quickly
  - Bind to 0.0.0.0; do not assume stable IP
  - Config via env/files mounted at start; avoid image rebuilds for flags
  - Connections: retry with jitter; pool sizes fit downstream caps
  - Rollouts: readiness gates before taking traffic
  - Observability: logs to stdout; metrics scrapeable; trace context propagated

Anti-patterns:
  - Special snowflake node affinity for every service without reason
  - Storing sole data copy on emptyDir
  - Privileged containers "to make it work"
  - Ignoring image CVEs because "we are behind a VPC"

Security baseline

Prefer minimal bases, non-root users, read-only root filesystems where feasible, dropped capabilities, and no privileged mode. Secrets should mount from a sealed store—not bake into layers. Admission policies enforce what busy humans forget.

Multi-tenant platforms that run stranger workloads should discuss stronger sandboxes. Same-org services still need network policy, workload identity, and least privilege to cloud APIs.

Trade-offs

Choice	Gain	Cost
Containers vs VMs	Density, fast start, portable packaging	Shared kernel; weaker default isolation
Many small images	Clear ownership; smaller blast radius	More registry ops; more networking
Sidecar per concern	Reuse proxies/agents without app changes	CPU/mem tax; coupled failure domains
Tight memory limits	Protect neighbors	OOM under real peaks
Loose / no limits	Absorb bursts	Noisy neighbor; node death
Pin digests + admit sign	Reproducible, safer supply chain	Process overhead; rebuild discipline
Stateful sets in-cluster	Unified ops model	Storage/failover complexity

Worked example: “scale-out made it worse”

A team doubles replicas during a launch. Each pod requests little memory but the JVM heap defaults high. Nodes look empty by requests, pack densely, then OOMKill under traffic. Restarts amplify DB connection storms.

plaintext

Root causes:
  - requests << actual working set → overpacking
  - memory limit below heap + metaspace + native
  - no connection budget across new replicas
  - readiness passed before warm pools filled

Fix narrative:
  1. Measure RSS under load; set request/limit from data
  2. Align JVM/container memory flags with cgroup limit
  3. Cap pools; add pooler if needed
  4. Readiness requires dependencies + warm state
  5. Load-test rolling updates and scale events

Choosing and explaining containerization

A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.

Explain the core idea

Define a container as a namespaced, cgrouped process on a shared kernel started from an image—not as a tiny VM.

Tie resource requests/limits to scheduling and failure modes (throttle, OOM).

Explain orchestration as changing the app contract: ephemeral IPs, rolling updates, graceful shutdown.

When it belongs

Choosing deploy units for microservices or batch workers.

Discussing multi-tenant isolation strength.

Sizing fleets, sidecars, or rollout strategy.

How it fails

OOMKill and CPU throttle from bad limits.

Data loss on ephemeral storage.

Supply-chain compromise via floating tags.

Thundering herds on reschedule or rolling restart.

Trade-offs to weigh

Density versus isolation strength.

Sidecar reuse versus resource and complexity cost.

In-cluster state versus managed data services.

Common misunderstandings

Assuming containers equal strong security isolation.

Treating host free memory as the container’s budget.

Writing durable state into the writable layer.

Ignoring digest pinning and image admission.

Wrap-up

Containerization is packaging plus kernel isolation plus—usually—orchestration. Master the primitives and the operational contract; tool UIs will keep changing.

Exercises

Draw isolation. For a multi-tenant SaaS running customer plugins, argue when default containers are enough versus when microVMs or separate nodes are required.
Size a pod. Given p95 CPU 400m and RSS 700Mi with spikes to 1.2Gi, propose requests/limits and justify headroom.
Rollout plan. Sketch readiness/liveness and SIGTERM drain for a service with in-flight WebSockets.

Key takeaway

Containers ship dependencies as images and isolate processes with namespaces and cgroups on a shared kernel. Design for cgroup truth, ephemeral compute, pinned provenance, and orchestrated failure—not for eternal SSH pets.