System Design Operations
Cloud Infrastructure Management Tools
Clicking through a console does not scale, and neither does a shell history that only one engineer understands. Cloud infrastructure management is the discipline of expressing desired topology as code, reconciling it safely through the provider control plane, and preventing humans and bots from fighting over the same resources.
This note covers , , , reconcilers, immutable vs mutable hosts, , and —concepts that transfer across tool brands, not a product manual.
Management loop
Networks, IAM, data stores as reviewed artifacts.
Plan / diff
Show what will create, update, or destroy.
Call provider APIs with locks and limits.
Reconverge or consciously adopt live changes.
What actually happens
comes in declarative flavors (describe end state) and imperative ones (scripts that create things). Declarative wins for most cloud topology because plans are reviewable and converges are repeatable. Imperative still appears for one-off migrations and glue.
Underneath, everything is a . Creates are often eventually consistent; IAM propagation lags; deletes can be soft. Your “apply failed halfway” story is a distributed-systems story wearing a DevOps badge.
Continuous reconcilers—platform —keep running after the first apply. That is powerful and dangerous: a bad spec will be reasserted forever until you change the source of truth.
How to choose an approach
1. Who owns the source of truth? If multiple teams edit the same live resources, pick one owner and one state boundary. Competing IaC roots are a reliability bug.
2. How wide is one apply? Shrink with separate state per environment and domain. Production IAM should not live in the same unlock as a marketing sandbox.
3. Push or pull? CI push-applies are simple; pull agents excel when many clusters must stay aligned—and need secret and promotion discipline.
Simple rule of thumb
Prefer declarative IaC with reviewed plans, small blast-radius state files, drift detection, and policy checks in CI. Use controllers when continuous reconciliation is the product; use push pipelines when change is human-gated and infrequent.
Desired state to live cloud
Read the diagram definition
flowchart LR
SRC[Desired state in git]
CI[CI review + policy checks]
PLAN[Plan / diff]
LOCK[State lock]
API[Cloud control plane APIs]
LIVE[Live resources]
DRIFT[Drift detector]
SRC --> CI --> PLAN --> LOCK --> API --> LIVE
LIVE --> DRIFT
DRIFT -->|reconverge or alert| SRCState storage (remote backends) records what the tool thinks it manages. Corrupt or duplicated state is an outage class of its own: protect it like a database, encrypt it, and restrict who can unlock.
Plans are the human contract. Destroy actions on stateful resources deserve stronger review than adding a tag. Encode that in CODEOWNERS and policy, not hope.
Declarative vs imperative vs controllers
| Style | Strength | Failure mode |
|---|---|---|
| Declarative IaC | Reviewable diffs; converge to desired | State drift; halfway applies; provider quirks |
| Imperative scripts | Flexible migrations | Not idempotent; tribal knowledge |
| Continuous reconciler | Self-heal toward spec | Bad spec reapplied forever; fight loops |
| Click-ops console | Fast emergency poke | Undocumented drift; unreproducible envs |
Read the diagram definition
sequenceDiagram
participant Dev as Engineer
participant Git as Git desired state
participant Agent as Reconciler / apply job
participant CP as Cloud control plane
participant Res as Resources
Dev->>Git: Merge change to VPC / IAM / DB
Agent->>Git: Read desired spec
Agent->>CP: Describe current resources
CP-->>Agent: Observed state
Agent->>Agent: Diff desired vs observed
Agent->>CP: Create / update / delete
CP->>Res: Mutate
Note over Agent,Res: On failure: retry with backoff, surface status, do not silent-skip destroysImmutable vs mutable infrastructure
means a new image or artifact rolls out as new instances; old ones drain. Config changes are new deploys. Rollbacks are traffic shifts. This shrinks configuration drift on hosts.
still exists—database parameter groups, appliance firmware, emergency hotfixes. The design mistake is pretending an entire estate is immutable while runbooks quietly SSH everywhere.
Immutable app roll:
build image v42 → launch new group → health check → shift traffic → terminate v41
Mutable snowflake risk:
SSH fix on box-7 → forgotten → AMI rebuild lacks fix → next scale-out recreates outage
Hybrid that works:
immutable compute + declarative cloud resources +
narrowly scoped mutable break-glass with expiry and auditGitOps and promotion
makes git the desired-state store and agents the appliers. Promotion becomes merging (or rendering) manifests from staging to prod paths—not “someone ran apply from a laptop.”
Read the diagram definition
flowchart TB
DEV[Dev overlays]
STG[Staging overlays]
PRD[Prod overlays]
AGENT_S[Staging reconciler]
AGENT_P[Prod reconciler]
CLUS_S[Staging cluster / account]
CLUS_P[Prod cluster / account]
DEV -->|PR + tests| STG
STG -->|promotion PR + approvals| PRD
AGENT_S --> STG
AGENT_P --> PRD
AGENT_S --> CLUS_S
AGENT_P --> CLUS_PSecrets are the hard part. Never commit long-lived credentials. Use : references in git, values from a vault or cloud secret store at runtime, short-lived where possible.
Progressive delivery (canaries, traffic splits) sits beside GitOps. Declaring “100% prod now” in git is still a risk decision; pair with health signals and automatic rollback criteria.
Policy as code and least privilege applies
Policies worth encoding
- Deny public object ACLs / open security groups by default
- Require encryption and deletion protection on data stores
- Restrict regions and instance types for cost/compliance
- Block wildcard IAM on human-driven modules
Pipeline privileges
- Separate plan roles from apply roles
- Short-lived OIDC federation beats static cloud keys in CI
- Environment locks and mandatory reviewers for destroy
- Break-glass roles audited and time-boxed
is how organizations scale review. It does not replace architecture judgment—someone still decides whether a public CDN origin is intentional.
Drift, halfway applies, and recovery
Real systems fail mid-apply. Design for it: idempotent resources, explicit dependencies, and runbooks that refresh state from the live world when the tool’s memory lies.
Common failure timeline:
T0 Plan looks good: replace subnet, recreate ASG
T1 Subnet delete blocked by ENIs still attached
T2 Apply aborts; some resources new, some old
T3 On-call “fixes” in console → drift from git
T4 Next apply tries to undo the console fix
Better path:
- smaller changesets
- create-before-destroy where supported
- import / refresh state from live when needed
- forbid silent console edits in prod (or adopt them into code same day)
- rehearse rollback: revert git + apply, not tribal SSHTrade-offs
| Choice | Gain | Cost |
|---|---|---|
| One giant state file | Simple mental model early | Huge blast radius; slow locks |
| Many small states | Safer applies; team autonomy | Cross-stack references; sprawl |
| Push apply from CI | Explicit human gates | Drift between applies; laptop temptation |
| GitOps pull agents | Continuous alignment | Secret & promotion complexity |
| Strict policy gates | Fewer foot-guns | False blocks; exception process needed |
| Immutable fleets | Reproducible hosts | Heavier rollout machinery |
| Console break-glass | Fast incident response | Drift unless adopted back to code |
Worked example: the “helpful” console edit
During an incident, an engineer opens a security group to debug. Traffic recovers. Days later an IaC apply “fixes” the group back to the coded rules and recreates the outage—or worse, the open rule stays forever because applies are paused.
Interview-quality answer:
1. Break-glass is allowed with time-boxed access and ticket link
2. Every console change opens a follow-up: adopt into IaC or revert
3. Drift detection alerts within hours, not weeks
4. Prod apply pipelines stay unlocked enough to ship the fix
5. Policy as code would have blocked a 0.0.0.0/0 SSH rule in the PR
that permanently "fixed" the fire drillChoosing and explaining cloud infrastructure management
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
Describe IaC as desired-state management over the cloud control plane, not as a brand name.
Call out drift, state blast radius, halfway applies, and who owns the source of truth.
Separate immutable app rollout from declarative cloud resources and from emergency mutation.
When it belongs
Explaining how environments stay reproducible across regions or accounts.
Discussing deploy safety, IAM for pipelines, or multi-team ownership.
Comparing GitOps pull reconciliation to CI push applies.
How it fails
Competing automation fighting over the same resource.
Secrets in git or in plaintext state.
Monolithic state enabling one PR to destroy prod data stores.
Silent console edits that never return to code.
Trade-offs to weigh
Autonomy of many stacks versus coordination overhead.
Continuous reconciliation versus human-gated applies.
Strict policy versus delivery velocity.
Common misunderstandings
Treating tool choice as architecture without ownership and blast-radius design.
Assuming apply success means the system is healthy.
Equating GitOps with “no reviews needed.”
Baking long-lived credentials into images or pipelines.
Wrap-up
Infrastructure management tools are how you industrialize cloud change. The concepts—desired state, drift, reconciliation, policy, and blast radius—matter more than any one CLI.
Exercises
- Split state. Sketch boundaries for network, data, and app stacks for a three-environment product. List cross-stack outputs and who can apply each.
- Drift drill. Write the runbook when live IAM differs from git after a break-glass incident.
- Policy set. Propose five policy-as-code rules for a payments account and one documented exception process.
Key takeaway
Manage cloud infrastructure as reviewed desired state with small blast radii, explicit reconciliation, and policies that catch dangerous diffs before they hit the control plane—then treat drift as a first-class operational signal.