Auto-scaling

What actually happens

Try it: autoscaling cooldown and warmup

Raise CPU past the target, then advance simulated time. Scale-out starts warmup and cooldown timers; traffic still lands on ready replicas until warmup completes. Scale-in waits for low CPU and an empty cooldown.

How to choose a scaling policy

The feedback loop: reactive or predictive

The three questions above pick the inputs; this is the loop they feed. Metrics come in, a capacity change goes out, and the loop observes again. Notice the amber steps on each branch: scaling out is not finished when a pod starts but when its caches and connections are warm, and scaling in is not finished until in-flight work has drained.

The decision diamond can be triggered in two ways. Reactive waits until a threshold breaches and then provisions, which is honest about real demand but can cost minutes of lag. Predictive uses calendars or models to pre-warm before the spike arrives. The two cards compare what each one triggers on and where it falls short.

Scaling metrics

A scaling metric should respond predictably when replicas are added. Average CPU often falls when parallel CPU work is distributed, but a downstream database bottleneck may keep latency high regardless of application replica count. Queue age signals user delay, while queue depth and arrival rate help estimate the work required to remove it. Use the delay to detect pain and a capacity model to choose the replica count.

Whichever trigger style you use, the loop is only as good as the metric feeding it, and scaling on CPU alone can miss queue backlogs, downstream saturation, or p99 latency spikes. Pick the signal that matches your bottleneck: CPU for compute-heavy APIs, queue lag for consumers (the age of the oldest job often matters more than raw depth), p99 when users feel pain before averages move, and active connections for WebSocket servers (long-lived two-way connections). The last row shows a composite scale-in rule: shrinking only when several signals agree is how you avoid deleting capacity on a brief dip.

Worked example: numbers to know

Assume each ready worker completes 40 jobs per second, arrivals are 240 jobs per second, and 12,000 jobs are waiting. Six workers only match arrivals and never clear the backlog. To clear it in five minutes, add 12,000 / 300 = 40 jobs per second of spare processing capacity, requiring seven workers in total under this simple model. Provisioning delay, retries, and uneven job sizes require additional reserve or a revised drain target.

A threshold like "70% CPU" means nothing until it connects to a capacity model. The worksheet below runs one million-user API through the steps: pick the metric closest to pain, load-test the safe per-replica capacity, divide peak QPS (queries per second) by it, add headroom for surges, reserve capacity for a zone failure, then set cooldowns around the real warmup time. Thresholds should come from load tests and user-facing latency targets, not from round numbers, and later sections refer back to these figures rather than repeating them.

Autoscaling lag and degraded mode

The delay includes collecting a metric window, evaluating policy, scheduling instances, obtaining machines if needed, starting the application, warming dependencies, and passing readiness checks. During that interval, arrivals continue. At 1,000 requests per second above current capacity, a two-minute delay represents 120,000 excess requests. A finite queue cannot make that demand disappear; decide which requests wait, receive a cheaper response, or are rejected.

How platforms implement the loop

In Kubernetes, HPA changes desired pod replicas, while node provisioning is a separate capacity concern. A desired pod that remains pending adds no serving capacity. CPU utilization targets are measured relative to configured CPU requests, so unrealistic requests distort the policy. Watch desired, scheduled, ready, and actually serving replicas separately to see which stage prevents scaling from helping. See Kubernetes HPA documentation for the precise contract.

Everything so far is policy; this is the machinery that runs it. On Kubernetes, the HPA changes the replica count from CPU or custom metrics, the VPA changes each pod's CPU and memory requests instead (and usually restarts the pod to apply them), and an event-driven autoscaler scales from external signals such as stream lag or HTTP queue depth, including down to zero for batch workloads. When those want more pods than the nodes can hold, a cluster autoscaler adds nodes, and modern provisioners can choose suitable instance types quickly. On cloud instance groups the same ideas appear as target tracking (the recommended default), step scaling for graduated responses to severe breaches, simple scaling for one action at a time, scheduled scaling for predictable calendars, and cooldowns that stop the group thrashing between add and remove.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  scaleTargetRef:
    kind: Deployment
    name: api-server
  minReplicas: 10
  maxReplicas: 500
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "1000"

Databases vs stateless services

All the machinery above assumes the thing being scaled can be copied freely. That is true for a web tier, where any replica can answer if session state lives elsewhere, and false for a data tier, which carries partitions (slices of the data pinned to nodes) and replication lag. Managed storage scaling exists, but migration is never "just add pods."

Cost optimization

Scale-in is a controlled removal, not just a smaller number. Stop routing new requests, drain bounded work, checkpoint or release worker ownership safely, and let persistent clients reconnect with backoff. Stabilization windows reduce repeated add-remove cycles around a threshold. Keep enough minimum capacity for expected bursts and failure recovery; the lowest idle bill is not necessarily the lowest total cost when repeated cold starts cause incidents.

Scaling in is where the savings live: auto-scaling saves money only when idle capacity actually disappears. Combine spot or preemptible nodes (spare cloud capacity sold cheaply but reclaimable at short notice) for fault-tolerant work, right-sizing for steady waste, and a reserved baseline (capacity committed for a year or more at a discount) for predictable cores.

Failure timeline, and what must never become false

t=0      Steady 231 avg QPS; 13 replicas sized for ~1.2K peak with headroom
t=1m     Spike to 1.2K; CPU still looks OK while p99 rises (I/O wait)
t=3m     HPA finally trips; new pods start; 3 min cold start + readiness
t=4m     Users feel pain; DB connections multiply as half-ready pods join
t=5m     Without shedding: cascade. With shedding + predictive pre-warm: survive
Outcome  Autoscaling is late by definition: budget the gap and guard dependencies

Wrap-up