System Design Infrastructure
Container Orchestration Tools
Black Friday traffic doubles overnight. You need more checkout pods in two availability zones, a safe image rollout, and a stable hostname while old pods die. SSH-ing into fifty hosts to start containers is not a design—it is an outage waiting for human error.
platforms encode three ideas: a places work, plus heal drift, and a abstraction keeps clients offline from churning backends. This note is about those mechanisms—not a kubectl tutorial.
Scale checkout
Replicas, image, CPU/memory, zones.
Pack or spread onto capable workers.
Only healthy pods join the service.
Dead node → reschedule from desire.
What actually happens
A fleet of container hosts without orchestration looks fine until change velocity rises. Humans place containers by habit, forget a zone, leave a bad binary running, and discover it when traffic shifts. Autoscaling becomes a spreadsheet. Failover is a pager playbook. The naive fix—"run Docker Compose on every box"—still leaves placement, healing, and service discovery as tribal knowledge.
Orchestrators invert the contract. You write into a durable . Controllers run : count running replicas, start missing ones, stop extras, and update endpoints. A binds pending work to that have capacity and satisfy constraints. When a node dies, desire still says "three replicas," so the loop creates replacements elsewhere.
Networking finishes the story. A gives clients a stable name while endpoints churn. Readiness keep half-started pods out of the VIP. Deployments become gated on those probes, not "hope SSH finished."
At interview scale the point is not product names. Kubernetes, Nomad, ECS, and Mesos-era systems differ in APIs, but they share schedule → run → observe → reconcile → expose. If you can explain those loops and their failure modes, you can map any vendor onto the whiteboard.
How to choose an orchestration shape
1. How often do instances move? High churn and multi-tenant packing need a real scheduler and service abstraction. A handful of long-lived VMs may only need a process supervisor.
2. Who owns the control plane? Managed offerings trade ops cost for less API surface control. Self-managed clusters demand HA etcd-like stores and upgrade discipline.
3. What fails loudly? Prefer readiness-gated rollouts and explicit over "best effort forever." Plan for under node pressure.
Simple rule of thumb
Declare desire, size requests honestly, spread critical replicas with anti-affinity, gate traffic on readiness, and treat Pending / CrashLoop / NotReady as first-class alerts. Orchestration multiplies mistakes as easily as capacity—make probes and limits part of the design.
Desired state and reconciliation
The control loop is the product. Everything else—CLI, YAML, dashboards—is how humans edit desire and watch status.
Read the diagram definition
flowchart TD
Desire[Desired state objects] --> API[Control plane API]
API --> Store[(Durable cluster store)]
Store --> Ctrl[Controllers]
Ctrl -->|create / update / delete| Workers[Worker agents]
Workers -->|status / events| Store
Store --> Ctrl
Note1[Reconciliation: actual vs desired]
Ctrl -.-> Note1
Controllers must be level-based: "make it so there are three Ready replicas of image v2," not a brittle script of create / sleep / create. Idempotent reconciliation survives duplicate events, network blips, and controller restarts. Status fields (Ready, Available, Progressing) are how operators know whether desire has converged or is stuck.
Stuck states matter as much as happy paths: ImagePullBackOff, InsufficientCPU, Unschedulable due to , PVC pending, or probe failures. A mature platform surfaces these as objects and events; a thin wrapper over Docker leaves you grepping logs on the wrong host.
Scheduling: placement as a system design problem
The scheduler is a constrained bin-packer with topology awareness. Filter, then score—then live with the consequences for months.
Read the diagram definition
sequenceDiagram
participant Dev as Operator
participant CP as Control plane
participant Sch as Scheduler
participant N1 as Worker A
participant N2 as Worker B
participant Svc as Service VIP
Dev->>CP: Desire: 3 replicas of checkout:v2
CP->>Sch: Pending pods need nodes
Sch->>Sch: Filter capacity + affinity
Sch->>N1: Bind 2 pods
Sch->>N2: Bind 1 pod
N1-->>CP: Ready after probes
N2-->>CP: Ready after probes
CP->>Svc: Endpoints updated
Note over Svc: Clients dial stable name; backends churn
| Constraint | Why it exists | Failure if wrong |
|---|---|---|
| CPU/memory requests | Reserve capacity for packing | Overpack → throttle/OOM; underpack → waste |
| Taints / tolerations | Reserve nodes for special work | General pods starve specialised pools |
| Zone anti-affinity | Survive AZ loss | All replicas die with one zone |
| Topology spread | Balance load / failure domains | Hot nodes + cold capacity |
Bin-pack vs spread
Packing densifies and frees empty nodes to scale down. Spreading reduces correlated failure and noisy neighbors. Batch jobs often pack; user-facing replicas usually spread.
Priority and preemption
When the cluster is full, something yields. Priority classes let payments preempt batch. Without them, critical pods sit Pending while scrapers hog the nodes.
Services: stable names over unstable pods
Pods are cattle with ephemeral IPs. A maps a DNS name (and often a virtual IP) to a set of ready endpoints. Implementations vary: kube-proxy iptables/IPVS, cloud load balancers, eBPF dataplanes, or service meshes that add retries and mTLS. The design contract is the same: clients should not cache a single pod IP across deploys.
Readiness is the gate. If you only check process uptime, a pod that accepts TCP but returns 500s still receives traffic. Couple readiness to dependency health carefully: if every instance waits on a shared DB that is slow, you can empty the entire service and create a restart storm. Prefer local readiness ("I can serve") plus separate SLO alerts on dependencies.
Headless services, sticky sessions, and externalName aliases exist for special cases. Sticky sessions fight the cattle model; prefer shared session stores. External aliases paper over migration; they are not a long-term discovery plan.
Rollouts, probes, and resource contracts
A is reconciliation with a rate limit: surge new pods, wait for Ready, then terminate old ones. maxUnavailable protects capacity; maxSurge protects cluster headroom. Blue/green and canary are the same idea with sharper traffic splits—still driven by desire objects and endpoint weights.
tell the scheduler what to reserve; limits tell the runtime when to throttle or OOMKill. Guaranteed QoS (request=limit) resists eviction; BestEffort dies first under . Lying about requests ("set them low so we pack more") creates silent contention and mysterious latency until memory pressure starts killing random pods.
Graceful shutdown completes the contract: receive SIGTERM, stop accepting new work, finish in-flight requests within terminationGrace, then exit. Without it, rollouts drop connections and amplify retries into the remaining pods.
| Probe | Purpose | Misuse |
|---|---|---|
| Startup | Delay other probes until warm | Too short → restart loops on slow JVMs |
| Readiness | Join / leave service endpoints | Coupled to global deps → empty VIP |
| Liveness | Restart wedged processes | Hits busy lock → kill healthy load |
Numbers that shape intuition
| Situation | Order of magnitude | Design note |
|---|---|---|
| Image pull cold start | seconds–minutes | Pre-pull, smaller images, local caches |
| Probe interval × failureThreshold | often 10–90s to declare dead | Too aggressive thrash; too slow bleeds traffic |
| terminationGracePeriod | often 30–120s | Must exceed longest in-flight request |
| Control-plane RTO | minutes without HA | Existing pods may run; no heal/schedule |
Failure timeline: bad rollout
T0 Desire: checkout image v2, maxUnavailable=50%, readiness = TCP:8080 only T1 New pods accept TCP before app finishes warming DB pool T2 Service endpoints flip; p99 errors spike; SLO burns T3 Liveness hits a contended /health that takes locks → restart storm T4 Scheduler packed all new pods into one AZ (no anti-affinity) T5 Fix: readiness = app-level OK, surge limited, zone spread, longer startup Outcome Orchestration automated the outage as efficiently as capacity
Break the improved design
Perfect probes still fail if the new binary breaks a wire protocol that old clients need—rollbacks must remain possible, and schema changes need expand/contract. Resource requests that match yesterday's load still OOM when a feature doubles memory. Cluster autoscalers that wait minutes leave Pending pods during a flash sale.
Multi-cluster and multi-region add another layer: local reconciliation cannot place pods in a dead region. You need global traffic management and replicated desire. Orchestration inside one cluster is necessary; it is not a DR plan by itself.
Trade-offs
Density vs isolation
Packing raises utilization and noisy-neighbor risk. Spreading and dedicated node pools buy isolation with lower density. Match packing policy to blast-radius tolerance, not only to cost dashboards.
Managed control plane vs self-operated
Managed offerings remove etcd pager duty and add provider limits, release cadence, and less low-level tunability. Self-managed clusters are flexible and easy to misconfigure into split-brain or undrainable upgrades.
Platform networking vs service mesh
Cluster services give discovery and L4/L7 basics. Meshes add retries, mTLS, and traffic shaping with sidecar or ambient cost and operational surface. Introduce a mesh when policy needs exceed the platform VIP—not by default.
Imperative ops vs declarative GitOps
Click-ops and one-off CLI fixes diverge from desired state quickly. GitOps makes desire reviewable and rollback-friendly but needs sealed secrets, promotion pipelines, and clear ownership of who may merge prod drift.
Stateless cattle vs stateful sets
Stateless services are the sweet spot for orchestration. Databases need ordered pods, stable identity, and volume attachment—possible, but failure modes (split brain, attach storms) deserve dedicated operators and runbooks.
Choosing and explaining container orchestration
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
I declare desired replicas and resources; controllers reconcile actual pods to that desire.
The scheduler places work under capacity and topology constraints; services keep a stable name over Ready endpoints.
Rollouts are rate-limited reconciliation gated by readiness—not SSH and hope.
When it belongs
The design needs auto-healing, multi-AZ placement, or frequent deploys across many hosts.
The interviewer asks how you scale or roll out without downtime.
You must contrast VMs, single-host containers, and clustered orchestration.
How it fails
Bad probes causing restart storms or traffic to cold pods.
Unschedulable pods from over-constrained affinity or lied-about requests.
Control-plane outage freezing heal/schedule while data plane limps on.
Trade-offs to weigh
Packing density versus blast radius and noisy neighbors.
Managed control plane convenience versus operational flexibility.
Simple cluster services versus mesh complexity.
Common misunderstandings
Treating orchestration as a kubectl cheat sheet instead of control loops.
Assuming the platform replaces graceful shutdown and idempotent apps.
Using liveness probes as a substitute for fixing deadlocks.
Wrap-up
Orchestration is desired state, scheduling, and services under continuous reconciliation. Master the loops and failure modes; vendor CLIs become details.
Exercises
- Pending forever. Three replicas stay Pending. List five distinct root causes across capacity, affinity, taints, volumes, and quotas—and what status field would distinguish them.
- Safe rollout. Design maxSurge / maxUnavailable and probe settings for a service with 10 pods, 2s warm-up, and 30s p99 requests. What breaks if terminationGrace is 10s?
- AZ loss. One zone disappears. Explain how desire, the scheduler, and the service VIP behave if anti-affinity was missing versus enforced.
Topic-specific references
Key takeaway
Orchestrators automate placement, healing, and discovery from declared desire. They also automate bad probes and bad packing—so design requests, topology, and readiness as carefully as the application code.