System Design Security
Secrets Management, Workload Identity & Key Rotation
A checkout service is about to charge a card. It needs a payment-provider credential, but that key must not live in the Dockerfile, the git history, or a Slack paste from last year's onboarding.
prove identity or unlock data. lets the service authenticate without long-lived passwords in config. limits blast radius when something still leaks.
Credential journey
Platform issues a short-lived identity so the pod never stores a long-lived password.
Policy and audit live in the vault; secrets never sit in env files committed to git.
Each service fetches only the secrets it needs under least privilege.
Keep dual-active keys during overlap, then retire the old version after refresh.
Try it: compare static env secrets with workload identity
This lab uses fictional references only. No real passwords, API keys, or vault paths appear. Step through fetch, expiry, leak, and rotation to see why short-lived, identity-bound tokens shrink blast radius compared with months-long env vars.
Credential reference (fictional)
vault://demo/app/database-binding
Displayed token: (not issued yet)
Blast radius
okNo leak simulated.
- Auth mode
- Identity
- TTL (model)
- 300s
- Rotation gen
- 1
- Leaked
- No
Mode workload-identity · credential pending · display (not issued yet) · TTL 300s · leak no. Workload identity: the pod proves who it is via the platform; the secret store returns a short-lived token. No long-lived password is baked into the image.
Never embed production secrets in labs or repos. Bind credentials to pod/service identity, scope audience and TTL, and automate rotation without copying material into chat or tickets.
Break-glass is audited. Emergency human credentials should be rare, time-boxed, and alert differently from routine workload identity fetch.
What actually happens when credentials outlive every deploy
Long-lived API keys in environment variables are easy to ship and hard to revoke. They leak into logs, crash dumps, container layers, and CI artifacts. When rotation requires synchronized redeploys of every consumer, teams postpone it until an incident forces a painful cutover. The checkout story above fails for that reason: the payment key becomes forever-config instead of a leased credential.
The modern pattern replaces stored cloud keys with attested , short-lived leases from a or , and audience-bound federation via or . At startup or on a refresh timer, the service authenticates with platform identity (a cloud IAM role, a Kubernetes service account, or a SPIFFE document) and asks the vault or KMS for what it needs. Often it first uses WIF to exchange an OIDC token for a cloud role without a static access key. The vault checks policy, returns a short-lived secret or data key, and writes an audit event.
Nothing durable should treat that value as forever config. When runs, you issue a new version, keep valid through the overlap, redeploy or hot-reload consumers, then disable the old version after metrics stay green. Failure modes are operational: vault outage needs a bounded cache and a documented path; renewal alerts must leave enough time to retry within the actual remaining credential lifetime.
Naive design, and the invariant that must hold
Before choosing tooling, pin down why the "just put it in env vars" habit fails, and what must stay true after you fix it.
Naive claim
"Put the payment key in Kubernetes secrets / env vars, rotate manually every quarter, and share one admin key across checkout and reporting."
Counterexample
A support bundle includes a process dump with the env var. The same key still works for 90 days. Rotating it without drops live charges for pods that have not restarted. One shared key turns a reporting compromise into a payment outage.
Correctness invariant
What must never become false
A credential is issued only to an attested for an explicit audience and short lifetime. Rotation and revocation never create an interval where neither old nor new credential works, or where a revoked identity remains trusted indefinitely. are never logged in plaintext.
How to choose a secrets approach
With the invariant in mind, three questions decide whether you need a vault-backed identity path or can stay simpler for a while.
1. Who needs the credential? Humans, CI, and services get different paths, never one shared admin key that violates .
2. How long can it live? Prefer short-lived tokens bound to over static API keys in env vars.
3. Can you rotate without an outage? If not, redesign for and consumer refresh before the next incident forces it.
Simple rule of thumb
Inject secrets at runtime from a , bind them to workload identity, rotate on a schedule and on incident, and audit every fetch. Treat secrets as volatile credentials, not as static config that outlives the deploy.
Three layers that make the pattern work
SPIFFE distinguishes a workload’s identifier from the verifiable identity document carrying it. The SPIFFE concepts documentation explains those roles; authentication still needs resource-specific authorization after identity is verified.
The rule of thumb above depends on three cooperating pieces. Skip any one and you are back to long-lived keys or outage-prone rotation.
A secrets manager stores credentials with access policies, version history, encryption at rest, and auditable retrieval. A key-management service primarily protects cryptographic keys and performs operations such as wrapping a data-encryption key; it is not automatically the place every application password should live. Define which component issues a credential, which stores it, which workloads may fetch it, and which resource ultimately validates it.
Workload identity answers which running service is requesting access without embedding a reusable cloud password in its image. The platform attests properties of the workload and supplies an identity document; an exchange service verifies its issuer, subject, audience, and validity before issuing a narrowly scoped credential. A Kubernetes service-account name alone is not proof unless the receiving system validates the actual token and its policy binding.
For a planned rotation, make the new credential usable before instructing healthy clients to adopt it, observe old-version use, and retire the old credential after the supported refresh window. Overlap preserves availability only when both credentials remain safe to trust. During a compromise, explicitly deny the compromised identity or credential as quickly as the resource supports, even if some requests must fail; keeping a stolen credential valid for a routine overlap defeats containment.
Worked example and rotation playbook
Here is how those layers behave under a short vault outage, then the five steps you run for every scheduled . Numbers below are the budgets you size before the first real cutover.
Checkout receives a 15-minute SVID (a identity document) and exchanges it for a 10-minute database credential. Sidecars renew at one-third lifetime with jitter. The is unreachable for eight minutes during a planned rotation; current and next database roles overlap for 20 minutes.
Numbers to know
Safe cache window
TTL 10m; refresh ≈ 3m20s
A newly issued ten-minute credential has nearly ten minutes of validity. If its next refresh is attempted after three minutes twenty seconds, about six minutes forty seconds remain when that attempt fails. An eight-minute issuer outage beginning then can exceed the remaining lifetime. The service must either refresh successfully before expiry, use another authorized path, or stop protected operations; overlapping database roles do not extend an already expired credential.
Overlap budget
20m > TTL + rollout skew
A twenty-minute overlap is a design example, not a general formula. It must cover the planned client refresh behavior, deployment skew, clock tolerance, and verification window while respecting the credential’s own expiry. Track which valid credential versions resources still accept and which healthy clients still use the old version. Revocation, role disablement, and token expiry may occur at different layers, so test the actual end-to-end behavior.
Revocation bound
≈ token TTL (offline JWT)
Short lifetime is the practical revocation control when resources do not introspect. High-risk paths may require live checks at availability cost.
Rotation playbook
Inventory all consumers and the resource-side validation behavior before rotating. Issue the new version, test it with a small healthy cohort, expand adoption while observing authentication failures and version use, then disable the old version and verify it is rejected. Include background jobs, idle connections, disaster-recovery environments, and rarely invoked tasks; a successful web deployment does not prove every credential consumer has refreshed.
- Issue a new key version in ; keep the old valid with .
- Deploy consumers that read the
currentalias so they pick up the new version. - Re-encrypt data at rest if needed, using a background job rather than a blocking cutover.
- Disable the old key only after max TTL plus a green verification metric (see Numbers to know above).
- Document rollback for a healthy planned rotation; never re-enable a known-compromised credential merely to restore availability.
Failure timeline, and what still breaks after the fix
Putting the pieces together: how a leaked is contained minute by minute, what still fails with vault and identity in place, and the trade-offs you accept for that safety.
Timeline: secret leak and containment
Leak detected
A secret scanner finds a token in a support bundle. Classify identity, audience, scope, issue time, expiry, and resources touched; preserve audit evidence.
Containment
Issuer denies new exchange for the compromised selectors. Revoke lease/session where supported; rotate downstream credentials; isolate the workload.
Replacement identity rolls
Clean instances receive a new identity and policy version. Overlap applies only to healthy identities; the compromised subject is rejected explicitly.
Recovery proven
Audit shows last old-version use and all affected actions. Reconcile resources, rotate derived secrets, close , add detection tests.
ASCII sketch
leak ──► classify ──► revoke / deny exchange ──► rotate downstream
│ │
▼ ▼
preserve audit dual-active for healthy only
│
▼
zero old use ──► close incidentWhat to watch in production
- Issuance and renewal latency plus remaining TTL, so refresh fails before expiry surprises you.
- Use by identity, audience, policy, and key version, without ever logging secret values.
- sealed state, errors, audit lag, and unexpected break-glass activation need paging.
Still fails when…
- Cache TTL is infinite during a sealed vault, leaving stale credentials forever.
- Old key disables before the slowest consumer refreshes, causing mid-flight 401 storms.
- Break-glass lacks short TTL and dual control, becoming a standing privileged backdoor.
- Compromised tokens remain valid until TTL or introspection catches up after revocation.
- Credentials can expire during live traffic if automated renewal has no retry budget before expiry.
Set renewal timing relative to the actual credential lifetime and allow jittered retries with a bounded remaining-validity budget. A ten-minute credential needs alerts on minute-scale renewal failures; “renew thirty days before expiry” applies only to a much longer-lived certificate. New instances should not claim readiness for protected operations without valid identity. Existing instances may use valid cached credentials within policy, but must not silently extend expired credentials during an issuer outage.
Trade-offs
- Vault adds latency and a control-plane dependency versus plain env vars.
- Deploy-time secret push is simpler at startup but worse after compromise.
- JWT identities fit federation; X.509 SVIDs fit mTLS between services.
- Live introspection speeds revocation; local caches preserve availability in brownouts.
Example brownout: vault sealed mid-rotation
T+0 Vault sealed / KMS errors spike; pods use cached DB tokens (TTL 12m) T+4m Rotation job disables old key version early (overlap too short) T+5m 18% of fleet still on old version → auth failures T+9m Break-glass opened "to unblock"; no TTL ticket filed T+40m Compromised subject still exchanges until session TTL: revocation lag Outcome Availability vs safety trade-off without audit becomes permanent risk
Short-lived machine trust
What production needs to hold
A production design needs a way to recognize trouble and a rule it preserves while recovering. A trigger is the measurement or event that starts a response. An invariant is a statement that must remain true, such as never confirming the same reservation to two customers.
Trigger metric or condition
Move to workload identity when static keys appear in images or CI, rotation requires synchronized redeploys, services share credentials, or incident responders cannot identify which workload used a secret.
Correctness invariant
Issue a credential only to an attested workload with an explicit scope and bounded lifetime. Planned rotation overlaps authorized credentials so healthy workloads can switch. Reject expired or revoked credentials according to the stated validation policy, even when doing so reduces availability.
SPIFFE/OIDC federation without a stored cloud key
Workload attests
node + runtime evidence
SPIFFE SVID
short-lived identity
OIDC exchange
audience-bound token
Vault / cloud STS
least-privilege lease
Database / KMS
authorizes identity
Rotate database credentials through a control-plane outage
A checkout workload receives a 15-minute SVID and exchanges it for a 10-minute database credential. Sidecars renew at one-third lifetime with jitter. Vault becomes unreachable for eight minutes during rotation; current and next database roles overlap for 20 minutes.
Safe cache
credential TTL 10m; refresh at ~3m20s; outage tolerance ~6m40s
Jitter prevents a renewal herd. The application uses an in-memory credential only until expiry; it never converts temporary failure into an unbounded static secret.
Overlap
20m > max credential TTL + rollout skew
Old and new roles remain valid long enough for cached credentials and slow instances to converge. Usage of the old version must fall to zero before disablement.
Revocation bound
worst case ≈ token TTL unless resource checks online
Short lifetime is the practical revocation control for offline JWT validation. High-risk resources can require introspection or denylist checks at added availability cost.
Decision
Existing instances ride through most of the outage on bounded caches; new instances fail readiness instead of starting without identity. The relevant limit is remaining lifetime at outage start, not the original TTL. Under these assumptions, six minutes forty seconds remain but recovery takes eight minutes, so the credential expires one minute twenty seconds before recovery. Checkout must enter its documented degraded mode; overlapping database roles does not extend an expired credential.
How the mechanism holds
Follow how each part protects the rule above. The mechanism explains what the system does; the reasoning explains why that action is sufficient under the stated assumptions. If an assumption changes, the guarantee may change too.
Attestation and federation
A SPIFFE workload API issues an X.509-SVID or JWT-SVID after node/workload attestation. OIDC federation or token exchange maps that identity to a cloud role without distributing a cloud access key.
Why it holds: The token audience, issuer, subject, expiry, and workload selectors are checked at every trust boundary.
Vault and KMS behavior
Vault issues leased dynamic credentials; KMS keeps key material non-exportable and performs cryptographic operations. Define bootstrap, sealed/control-plane outage, rate limit, audit delivery, and regional failover behavior.
Why it holds: A dependency game day shows what existing and new workloads do through an outage.
Rotation overlap
Publish new trust first, issue new credentials, watch old-version use decay, then revoke old trust. For encryption, separate rotating key-encryption keys from background data re-encryption.
Why it holds: Telemetry names credential version without exposing values and proves zero old use before revocation.
Break glass
Require two-person approval, short TTL, narrow scope, separate identity, immutable audit, and immediate post-use revocation. It is never embedded in deployment manifests.
Why it holds: Quarterly drill retrieves and revokes access, then reconciles every action.
Incident drill: workload credential is exposed
Leak is detected
Secret scanner identifies a token in a support bundle.
Response: Classify identity, audience, scope, issue time, expiry, and resources touched; preserve audit evidence.
Containment
Issuer denies new exchange for the compromised workload selectors.
Response: Revoke lease/session where supported, rotate downstream credential, and isolate the workload.
Replacement identity rolls
Clean instances receive a new identity and policy version.
Response: Use overlap only for healthy identities; reject the compromised subject/version explicitly.
Recovery is proven
Audit shows last old-version use and all affected actions.
Response: Reconcile resources, rotate derived secrets, close break-glass access, and add detection/regression tests.
How to choose among alternatives
Push vs fetch secret
Choose when: Fetch/lease enables rotation and audit; push can simplify startup for low-risk static configuration.
Caution: Environment variables leak through process dumps and remain stale until restart.
JWT vs X.509 identity
Choose when: JWT fits audience-bound federation; X.509 fits mutual TLS and connection identity.
Caution: Both require issuer trust distribution, expiry handling, and clock-skew bounds.
Cache vs live validation
Choose when: Bounded local caches preserve availability; live introspection speeds revocation.
Caution: Choose per resource risk and state explicitly what happens when the identity plane is down.
Running it day to day
Measurements tell you what is happening; alerts tell an operator when action is needed. A rollout plan limits how many users encounter a change before it is checked. A runbook records the steps to diagnose and recover from a known failure.
Measure
- credential issuance/renewal latency, failures, and remaining TTL
- use by identity, audience, policy, and key version
- Vault sealed state, KMS errors, and audit delivery lag
Alert
- credential will expire before next safe renewal
- old version still used near revocation deadline
- unexpected identity/audience or break-glass activation
Roll out safely
- add new trust/policy before issuing credentials
- canary identity exchange and dual-trust overlap
- revoke only after version-use and readiness gates are green
Runbook
- identify and contain principal, not merely the host
- revoke/rotate downstream leases and inspect audit
- restore with clean identity, reconcile actions, and reduce future TTL/scope
Practice scenario
Design service-to-database authentication with no static application secret and explain behavior during an identity-control-plane outage.
Questions worth asking next
- How does OIDC federation work?
- What is the revocation bound?
- How is break-glass access controlled?
A solid answer sketch
Attest the workload, issue a short-lived SPIFFE identity, exchange it for an audience-bound dynamic database lease, renew early with jitter, and cache only until expiry. New pods fail readiness during a long outage; existing traffic follows a declared degradation plan. Rotation uses overlap and version telemetry.
Common mistakes
- • putting the emergency credential in every pod
- • rotating issuer and verifier simultaneously with no overlap
- • logging a token while diagnosing renewal failure
Authoritative references
Last reviewed: 6 August 2026
Choosing and explaining secrets and workload identity
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
Prefer short-lived tokens over static API keys in services.
Rotation must be automated with overlap window.
Least privilege: each workload gets only the secrets it needs.
When it belongs
Microservices call cloud APIs or databases with credentials.
Compliance requires key rotation and access audit.
Zero-trust design removes network-perimeter trust.
How it fails
Secret in git history forever.
Shared admin key across all services.
Rotation without dual-key breaks live traffic.
Trade-offs to weigh
Vault adds latency and dependency vs env vars.
Per-service identity multiplies policy management.
Frequent rotation improves security; increases ops load.
Common misunderstandings
Logging secret values on fetch failure.
Using root cloud keys in application code.
Skipping rotation because manual process is scary.
Wrap-up
You have seen the failure, the mechanism, and the operational signals. Use these exercises to check the ideas stuck, then follow the references for the primary sources.
Exercises
- No static DB secret. Design service-to-database auth with no static application secret. What happens to new vs existing pods during an 8-minute identity-plane outage?
- Size the overlap. Size a dual-active overlap for a 10-minute credential TTL and a 15-minute rolling deploy. When is it safe to disable the old version?
- Break-glass policy. Write who approves, TTL, scope, audit fields, and how you prove revocation after the drill.
Topic-specific references
Topic sources for and secrets, not generic system-design books.
Key takeaway
Treat secrets as volatile credentials with identity, audit, and rotation, not as static config. Workload identity plus KMS is the default pattern for services that outlive any single deploy.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Issue short-lived workload identity through attestation/federation, cache safely, rotate with overlap, survive control-plane faults, revoke, and audit break-glass.
Explain the decision
Why is workload identity preferable to static service credentials?
Read the reasoning
It binds short-lived credentials to an attested workload and automates rotation, scope, revocation, and audit without distributing long-lived secrets.
Apply it to a production scenario
The identity issuer is down for 20 minutes and certificates last 30 minutes. What happens?
Read one approach
Use cached valid credentials/last-known trust bundles, alert on expiry horizon, shed new workload starts if needed, and never disable verification.
Work through the numbers
Fifty thousand workloads rotate 1-hour credentials evenly. Average issuance rate?
Read the worked calculation
About 13.9 issuances/s; restart storms and overlap require much higher peak capacity.
Reason through a failure
What must rotation overlap preserve?
Read the recovery reasoning
Peers trust both old and new issuers/keys only for a bounded window, new credentials are preferred, and revocation remains possible.
Sources and further study
Continue with OpenTelemetry, Sampling & Cardinality.