System Design Production Engineering

Build and Deployment Automation Tools

Shipping is a reliability problem wearing a tooling costume. proves a change is buildable; keeps a releasable ; progressive traffic shifts decide whether production stays healthy.

This note is for system-design interviews: pipelines, promotion, , , config/secrets handoff, and supply-chain basics—not a Jenkins or GitHub Actions cookbook.

Release journey

Every change gets a shared, repeatable verdict.

Promote digests; do not rebuild at deploy time.

Canary or blue/green with abort gates.

Prior digest must still be runnable.

What actually happens

Humans write code; machines should own the path from commit to running process. The is that path: deterministic stages, pinned toolchains, and a published identified by digest.

Environments differ only by , not by “we rebuilt with different flags.” Promotion is a decision about which digest the router points at—and how much traffic it gets.

When something fails after go-live, speed of is often more valuable than the cleverness of the forward fix. That speed is designed in: compatible schemas, retained prior artifacts, and automated cutback.

CI vs CD (and the third D)

Practice	What “done” means	Human in the loop?
	Main stays green; feedback in minutes	Merge judgment; not deploy judgment
	Every green build is a prod candidate	Yes—promote when ready
	Eligible greens ship without a ticket	Policy + automation; rare manual hold

Interview phrasing

Do not say “we use CI/CD” as if it were one switch. Say what is automated, what requires approval, and how you limit blast radius when automation ships a bad change.

The pipeline as a system

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

flowchart LR
    C[Commit] --> CI[CI: build + test]
    CI --> A[Immutable artifact]
    A --> Scan[Scan + sign]
    Scan --> Reg[Registry]
    Reg --> Stg[Deploy staging]
    Stg --> Gate{Promote gate}
    Gate -->|approve / auto| Prog[Progressive delivery]
    Prog --> Prod[Production traffic]
    Prog -->|SLO burn| RB[Rollback to prior digest]

Latency matters. A 90-minute CI queue destroys the point of continuous integration. Parallelize tests, cache dependencies carefully, and split slow suites—without sacrificing the tests that catch real regressions.

Flakes are production incidents for the platform. Quarantine or fix flaky tests; never teach engineers to “rerun until green.” That trains the org to ignore red signals.

Artifacts are the contract. Staging and production should run the same digest. Environment drift comes from config and data, not from “we rebuilt with a newer base image.”

From problem to mechanism

Problem. A checkout bug reaches production every few weeks. Deploys are manual SSH and “copy the jar from Jenkins’ workspace.” Nobody can say which commit is live.

Naive system. More code review and longer change freezes. Incidents still happen; freezes just concentrate risk into big-bang releases.

Break it. An engineer hotfixes prod by editing a file on the box. The next “official” deploy overwrites the hotfix. Audit cannot reconstruct the timeline.

Mechanism. CI produces an ; the deploy system only accepts signed digests; traffic moves via ; rollback points at the previous digest. Hotfixes are commits, not live edits.

Correctness invariant for releases

At any moment you can answer: which digests serve which traffic slices, which config version they loaded, and how to cut back to the last known-good pair—without rebuilding from source under pressure.

Progressive delivery shapes

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

sequenceDiagram
    autonumber
    participant Dev as Developer
    participant Pipe as Pipeline
    participant Reg as Artifact registry
    participant Router as Traffic router
    participant Obs as SLO monitors

    Dev->>Pipe: Merge to main
    Pipe->>Pipe: Build, test, package
    Pipe->>Reg: Push digest-pinned image
    Pipe->>Router: Deploy canary (1%)
    Router->>Obs: Canary vs baseline metrics
    alt Error or latency burn
        Obs-->>Pipe: Abort
        Pipe->>Router: Rollback to previous digest
    else Healthy for soak window
        Obs-->>Pipe: Widen 10% → 50% → 100%
        Pipe->>Router: Promote full
    end

shines when…

You can afford a second full environment briefly.
Cutover must be atomic for a sticky session or protocol reason.
Smoke tests on green catch most issues before any user traffic.

shines when…

You want real production traffic as the final test.
Capacity cost of a full twin stack is painful.
You can compare canary vs baseline with clean metrics.

Shared hazard: database migrations. If green/canary writes a schema the old version cannot read, is blocked. Expand/contract migrations and dual-write windows are part of deployment design, not “DBA homework.”

Worked example: unsafe vs safe promote

plaintext

Unsafe path (common in early teams)
  1. CI runs tests on commit abc123
  2. Deploy job "git pull && mvn package" on prod hosts
  3. Load balancer set to 100% new instances
  4. On failure: SSH, guess which jar, hope

Problems
  - Prod binary ≠ what CI tested (clock, dep cache, uncommitted hotfix)
  - No gradual exposure; blast radius = all users
  - Rollback = tribal knowledge

Safer path
  1. CI builds image registry.example/checkout@sha256:9f3a…
  2. Same digest → staging → soak
  3. Canary 1% for 20 min; gate on checkout success + P99
  4. Widen or auto-rollback to previous digest sha256:71bc…
  5. Config from secret store; image has no prod passwords

Invariant
  deploy(target) = point traffic at known digest + config version

Config, secrets, and environment handoff

Bake code into the ; inject environment through . If your image contains prod database passwords, every registry reader is a potential breach—and every environment needs a different image.

Prefer short-lived credentials (workload identity, IAM roles, rotated tokens) over long-lived static secrets in CI variables that print into logs. Feature flags belong next to config: they let you decouple “binary deployed” from “behavior enabled.”

plaintext

Handoff checklist (interview-ready)
  Image:     environment-agnostic, digest-pinned
  Config:    non-secret knobs (timeouts, pool sizes, flag defaults)
  Secrets:   vault / KMS / cloud secret manager; never git
  Identity:  runtime assumes a role; CI uses OIDC to cloud, not forever keys
  Flags:     kill-switch for risky behavior without a full rollback
  Provenance: who built this digest, from which commit, with which policy

Supply-chain basics for SD interviews

At interview depth you are not expected to recite every SLSA level. You are expected to name risks and compensating controls for the :

Pin dependencies and base images by digest; avoid floating latest.
Least-privilege CI runners: no shared god tokens that can push anywhere.
Sign artifacts; verify signature at admission/deploy time.
Scan for known CVEs—and treat critical runtime vulns as release blockers when exposure is real.
Separate “build” identity from “deploy” identity so a stolen PR token cannot empty production.

Rollback is a product feature

fails for predictable reasons: forward-only migrations, deleted prior images, incompatible protocols between clients and servers, or config that was mutated in place. Design for N and N−1 coexistence for at least one release cycle.

Automated rollback needs a clear signal: error-rate delta, latency burn, or synthetic probe failure—not a human staring at a dashboard hoping to notice. Pair abort with a soak window so noise does not thrash versions.

Failure	Why rollback stalls	Design fix
Schema drop	Old binary needs dropped column	Expand → migrate → contract across releases
Image GC	Prior digest deleted from registry	Retain last N prod digests policy
Client forced upgrade	Mobile app only speaks new API	Versioned APIs; dual support window
Config drift	Live-edited flags not versioned	Versioned config + same promote model

Where automation fits in a larger design

Microservices fleet

Per-service pipelines with shared templates beat one mega-Jenkins job. Independent deployability is the point—so contract tests and compatibility windows matter more than lockstep releases.

Monolith with many teams

One artifact, many owners. Feature flags and modular boundaries reduce “everyone merges, nobody can ship.” Pipeline time and test ownership become org design problems.

Stateful / data plane

Binaries can canary; data migrations cannot “rollback” as casually. Separate schema change pipelines, expand/contract, and backfill jobs with their own abort criteria.

ML model rollout

Treat models as versioned artifacts with shadow traffic and KPI gates (not only unit tests). Same progressive delivery ideas; different success metrics.

Trade-offs

Choice	Gain	Cost
Continuous deployment	Tiny changes, fast learning	Requires strong tests, canary, rollback
Manual promote (delivery)	Human judgment, change windows	Batching risk; slower MTTR culture
Blue/green	Clean cutover; easy router rollback	2× capacity briefly; shared-data surprises
Canary	Real traffic proof; lower twin cost	Needs good metrics; partial dual-version load
Rebuild at deploy	Feels simple	Non-reproducible; breaks audit & rollback
Strict supply-chain gates	Harder tampering; clearer provenance	Pipeline friction; false-positive scans

Choosing and explaining build and deployment automation

A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.

Explain the core idea

Separate CI (prove the change) from delivery/deployment (promote a digest under progressive traffic).

Deploy immutable, digest-pinned artifacts; inject config and secrets at runtime; keep N and N−1 compatible for rollback.

Progressive delivery plus automated abort is how automation stays safer than big-bang manual releases.

When it belongs

Designing multi-service release strategy, MTTR, or change-failure rate.

Discussing how a marketplace or payments system ships without weekend freezes.

Explaining blast radius for a bad release or a poisoned dependency.

How it fails

Rebuilding at deploy time so prod ≠ what CI tested.

100% cutover with no health gate; rollback blocked by schema.

Secrets in images or CI logs; stolen runner tokens with god privileges.

Trade-offs to weigh

Continuous deployment speed versus gated delivery judgment.

Canary fidelity versus blue/green simplicity and cost.

Supply-chain strictness versus engineer throughput.

Common misunderstandings

Saying “CI/CD” without specifying promote policy or rollback plan.

Treating feature flags as optional when continuous deployment is claimed.

Ignoring data migrations as part of the deployment design.

Wrap-up

Exercises

Classify the practice. A team merges to main ten times a day, builds signed images, and requires two approvers to promote to prod. Is this CI, continuous delivery, continuous deployment—or a mix? What would you change for a high-risk payments service?
Design the abort. Spec canary gates for a feed API: which metrics, what soak window, and what happens to in-flight clients when you roll back?
Unblock rollback. Version N writes a new column that N−1 does not know. Sketch an expand/contract sequence so a bad N can still cut back.

Topic-specific references

Martin Fowler: Continuous Delivery — definitions that interviewers still expect you to separate cleanly.
Google SRE Workbook: Canarying Releases — progressive delivery as a reliability practice.
SLSA — vocabulary for supply-chain levels without tool lock-in.

Key takeaway

Build automation is a control plane for change: prove once, package immutably, shift traffic progressively, and keep rollback as a first-class path—not an after-hours prayer.