System Design · Security
Encryption
Coffee-shop Wi‑Fi carries a checkout payment and a ChatGPT paste. Even if traffic is copied, a disk is stolen, or a backup leaks, the bytes should stay useless without the right keys.
is layered: in transit, and at rest, and sometimes field-level crypto when admins must not read sensitive columns.
Checkout protection journey
Encrypt the browser path; verify the server.
Edge termination
Cert automation, then re-encrypt to origin.
DEK encrypts data; KEK wraps the DEK.
Key plane + audit
Rotate, revoke, and log every unwrap.
What actually happens
The browser opens : ClientHello, certificate check, shared secrets, then encrypted HTTP. cuts handshake RTTs; can go even faster on resume but is replayable: keep it off money paths.
At the edge, TLS often terminates for latency and . Plaintext exists briefly inside that proxy. Then traffic should continue with private TLS or toward origin. Treating “TLS outside, plaintext inside” as free is a common mistake.
At rest, a random encrypts the object; a wraps that DEK. Rotate KEKs by re-wrapping, not rewriting petabytes. When disk encryption is not enough, add ; when privacy law demands delete, consider .
Try it: stage a TLS handshake and envelope wrap
Walk TLS 1.3 from ClientHello through encrypted application data, then wrap a disk object with envelope encryption. The model shows defensive layering: wire confidentiality first, then at-rest key hierarchy.
- TLS stage
- 0 / 5
- Record encryption
- Not yet
- DEK wrapped
- No
- Trust check
- Pending
- 1ClientHello
- 2ServerHello + certificate
- 3Verify + key agreement
- 4Finished messages
- 5Encrypted application data
Browser to edge
TLS 1.3Protected channel + storage
AEAD + KMSAdvance through a TLS 1.3 full handshake. Each step adds confidentiality and integrity on the wire before HTTP leaves the browser.
Transit. Certificate verification and Finished messages matter as much as cipher choice. A broken chain aborts before secrets are used.
At rest. Envelope encryption separates bulk data keys from the central KEK so rotation and audit concentrate in the key plane.
How to choose encryption layers
1. Where can plaintext exist? Wire, proxy memory, queues, logs, backups, and support tools each need an explicit answer.
2. Who holds keys? Platform-managed is fine for defaults; customer-controlled KEKs when tenants or auditors demand isolation.
3. What must survive key loss or rotation? Plan overlap windows, handshake monitoring, and restore drills before retiring old material.
Simple rule of thumb
Default TLS everywhere, automate cert renewal, envelope-encrypt data at rest, tighten keys per tenant or sensitivity, and rehearse rotation before auditors ask.
In simple terms: lock the road, lock the disk, and treat keys as the real product.
TLS 1.3 handshake
Fewer round trips · 0-RTT replay caveats · cipher selection
TLS 1.2: two round trips
- Flight 1: ClientHello.
- Flight 2: ServerHello, certificate chain, key exchange material.
- Flight 3: Client finishes key agreement and sends Finished.
- Flight 4: Server Finished → application data can flow after ~2 RTT.
TLS 1.3: one round trip
- ClientHello carries key-share guesses; server replies with selected cipher, its key-share, and encrypts certificate/finished when possible in the same flight.
- Result: first application bytes after ~1 RTT: big win for mobile and cold connections.
TLS 1.3 session resumption: 0-RTT
TLS 1.3 early data can send application bytes before the new handshake finishes by using a previously established secret. This reduces a round trip, but early data can be replayed under the protocol’s threat model. Permit it only for operations whose repetition is safe under an explicit anti-replay policy. A payment capture or state-changing request should not become replayable merely to save handshake latency.
Handshake cost at scale
- Fewer RTTs → fewer wasted CPU milliseconds per connection: matters when you open millions of TLS sessions per second at the edge.
- Session resumption + QUIC combine to keep tail latency flat; still monitor handshake failures and cert expiry SLOs.
TLS termination & scale planning
Where plaintext may exist, what gets re-encrypted, and which metrics prove it works
Read the diagram definition
sequenceDiagram
autonumber
participant Client
participant Edge as Edge terminator
participant Gateway
participant Service
participant KeyPlane as Key control plane
participant Audit
Client->>Edge: TLS handshake and HTTP request
Edge->>KeyPlane: Fetch cert handle, staple status, policy
Edge->>Gateway: Re-encrypt with private TLS or mTLS
Gateway->>Service: Forward signed request context
Service-->>Gateway: Response
Gateway-->>Edge: Response over encrypted hop
Edge-->>Client: Response over original TLS session
Edge->>Audit: Handshake errors, cert age, protocol, SNITermination boundary
Edge termination improves latency and certificate operations, but plaintext exists briefly inside the proxy process. Treat that host, memory, and logs as sensitive. At TLS termination, encrypted network records are decrypted so the proxy can inspect and route HTTP. Re-encrypting the next hop protects that link, but the terminating proxy can still see plaintext. End-to-end encryption is a different boundary: intermediaries do not hold the message-decryption keys. Draw where keys are available and where plaintext is processed before claiming which attackers encryption excludes.
Re-encryption
Use private TLS or mTLS from edge to gateway/service. Avoid “TLS outside, plaintext inside” unless the network segment is isolated and explicitly risk-accepted.
Scale signals
Track handshakes/sec, resumption hit rate, certificate days remaining, protocol mix, and handshake CPU. A 1% resumption drop can become real edge cost at peak.
TLS termination capacity math
- 2M new TLS handshakes/minute = ~33.3k new handshakes/sec.
- If resumption drops from 90% to 80%, full handshakes double from ~3.3k/sec to ~6.7k/sec.
- At 3 ms CPU/full handshake, added cost is ~10 CPU-sec/sec, or roughly 10 extra dedicated cores before safety margin.
- Roll certificate reloads in small batches and watch handshake failures, not only successful requests.
Plaintext boundary checklist
- Where does TLS terminate: edge proxy, gateway, service, or database client?
- Where can plaintext exist: proxy memory, debug dumps, queues, logs, metrics, analytics, backups, and support tools?
- Is traffic re-encrypted with private TLS or mutual TLS (mTLS) before it leaves the trusted process?
- Which decrypt and key-unwrap events are audited with actor, service, resource, and reason code?
Key answer pattern
Terminate TLS at controlled edges for latency and certificate automation, immediately re-encrypt to origin, keep private keys behind a managed key boundary, and alert on expiry plus handshake anomalies before users see failures.
TLS lifecycle architecture
Certificates are operational systems, not one-time files
Read the diagram definition
sequenceDiagram
autonumber
participant Browser
participant Edge as CDN or ingress
participant CA as Certificate authority
participant KeyPlane as Managed key service or hardware key module
participant Origin
Edge->>CA: Prove domain control with certificate automation
CA-->>Edge: Issue short-lived certificate
Edge->>KeyPlane: Load private key material or hardware key module handle
Browser->>Edge: TLS ClientHello with SNI and key share
Edge-->>Browser: Certificate, ServerHello, Finished
Browser->>Edge: Encrypted HTTP request
Edge->>Origin: mTLS or private TLS to service
Note over Edge,KeyPlane: Renew early, monitor expiry, rotate keys safelyIssue
Automate certificate automation/DNS validation, pin ownership, and alert on unexpected CT log entries.
Serve
Terminate at edge for browsers, then use mTLS/private TLS from proxy to origin.
Renew and revoke
Track days-to-expiry, safe reloads, emergency key rotation, and client pin fallout.
Certificate lifecycle
Certificate automation renewals · internal PKI, workload identity, transparency logs
Public TLS automation
- certificate automation (automated public certificate authorities): agent proves domain control, receives short-lived certs (~90 days), and renews early through an orchestrated job.
- At thousands of hostnames, centralize issuance policies and audit certificate automation account keys.
- Reload proxies (proxy fleet) safely: stagger to avoid thundering herds.
Internal PKI & mesh
workload identity identities
Platform-native RPC encryption concept
Certificate Transparency
Encryption at rest
AES-GCM and disk crypto · envelope encryption and SSE modes
AES-256-GCM (AEAD)
Envelope encryption + managed key service
Data
A data-encryption key, or DEK, encrypts a payload using a symmetric authenticated-encryption construction such as AES-GCM. Authenticated encryption also detects tampering, provided its nonce requirements are respected. The same key must not reuse a nonce where the algorithm forbids it. Store the ciphertext, authentication tag, nonce, and key-version metadata needed to decrypt later; a missing field can make otherwise intact backups unusable.
Wrap DEK
A key-encryption key, or KEK, wraps the DEK, allowing the encrypted DEK to be stored beside the ciphertext. A managed key service applies access policy and records use of the wrapping key, while the application handles bulk data with the temporary plaintext DEK. Whether the key is hardware-backed and non-exportable depends on the selected service and key type. Protect application memory and minimize how long plaintext keys remain there.
Why bother
Wrapping-key rotation can rewrap existing DEKs under a new KEK without re-encrypting every payload byte. Keep old key versions available until all required objects and backups have a valid recovery path. If a plaintext DEK or the data itself was compromised, merely changing its wrapper does not undo that exposure; the incident may require a new DEK and re-encryption. Revocation policy and cryptographic rotation solve related but different problems.
Object storage modes (cloud provider-shaped)
| Mode | Who holds keys | Typical use |
|---|---|---|
| provider-managed server-side encryption | Platform-managed | Default durability with minimal ops |
| SSE-managed key service | Customer-controlled managed key service key | Audit trails, per-tenant isolation |
| customer-supplied server-side encryption | Customer-supplied per request | Legacy hardware key module integrations |
| CSE | Client encrypts before upload | Storage platform never sees plaintext |
Key management planes
hardware key module vs managed key service vs secret broker
hardware key module
Tamper-resistant hardware where master keys perform crypto without export. Use for root CA, payment hardware key modules, or regulated workloads (hardware assurance tiers).
Managed key service
Managed APIs wrapping hardware key module clusters: IAM policies, automatic rotation hooks, audit streams. Ideal envelope encryption at planetary scale.
Secret broker services
Dynamic credentials, transit encryption APIs, and Public Key Infrastructure (PKI) issuance become one control plane when multiple environments or legacy services need consistent policies.
Key rotation math
If 500M objects each store a 96-byte wrapped DEK, the wrapped-key metadata is ~48 GB. Re-wrapping 25k keys/sec takes ~5.6 hours before retries, throttling, and regional safety windows. Keep old KEKs enabled until all readers prove they can decrypt with the new version.
Threat model card
- Assets
- Private keys, DEKs, KEKs, plaintext fields, backups.
- Actors
- Network observer, compromised proxy, support user, stolen disk holder.
- Trust boundaries
- TLS terminator, service memory, key service, backup system, support console.
- Attack paths
- Expired certs, bad termination, key unwrap misuse, plaintext logs.
- Controls
- TLS, mTLS, envelope encryption, key policy, least-privilege decrypt APIs.
- Detection
- Handshake errors, cert age, decrypt spikes, key-policy changes.
- Residual risk
- Plaintext inside trusted services and metadata leakage from deterministic fields.
Failure drill: bad cert or key rollout
- Canary one edge slice, then watch handshake failures, OCSP stapling, and certificate age before widening.
- Keep the old certificate/key version readable through the overlap window so rollback is one config flip, not a reissue.
- Block plaintext logging during emergency debug; require a ticketed decrypt reason for any key-unwrap surge.
- Test restore from encrypted backups with the rotated key set before retiring old key material.
Field-level & app-level crypto
When disk encryption isn't enough
Field-level encryption
- Encrypt selective columns (SSN, card artifacts) with tenant-specific keys: DBAs see ciphertext only.
- Drivers (e.g. client-side field encryption) can encrypt client-side; queries often limited to equality on deterministic tokens.
- Use for regulated-data defense-in-depth beyond transparent disk encryption.
Application-level encryption
Homomorphic encryption (emerging)
Compliance snapshots: payment security
Card data minimization + crypto controls
Key payment-security encryption expectations
- Strong TLS for transmission (modern configs disallow legacy SSL).
- Strong symmetric crypto at rest (AES-256-class).
- Protect keys: dual control, split knowledge for manual procedures.
- Rotate keys on policy; document ceremonies.
- PAN unreadable via encrypt/hash/truncate; never store CVV/PIN/full track.
Payment processor pattern
Real-world layering
Defense in depth vs single-point failures
Layered encryption (hyperscaler shape)
- 1Hardware/disk encryption across SSD fleets.
- 2Storage subsystem encrypts chunks with chunk keys.
- 3Optional file/object keys (unique DEKs per object).
- 4managed key service wraps DEKs; human-readable policies gate unwrap.
- 5Root keys live in hardware key modules with quorum ceremonies.
Choose key boundaries according to the damage one compromised identity should be able to cause. Separate keys or access policies by tenant or sensitivity can limit which data one service may decrypt. They also add key metadata, quotas, recovery work, and authorization complexity. A shared administrator with decrypt permission across every tenant can defeat intended isolation even when each tenant technically has a different key.
Crypto-shredding / privacy-law deletes
Case studies
TLS bugs, mass HTTPS adoption, and how breaches teach key-management discipline
TLS · memory safety
TLS parser memory leak: optional protocol feature leaked live server memory
A bug in TLS library let attackers ask “echo this padding” and receive extra bytes the server never meant to send: sometimes cookies, private keys, or other sessions still in RAM. HTTPS still showed the padlock while memory leaked; victims often learned from headlines, not dashboards.
Beginners should remember: cryptography can be “correct on paper” yet unsafe if adjacent parser code mishandles lengths: TLS brings together ASN.1, X.509, record fragmentation, and heartbeat extensions in one hot path.
Simplified attack / failure chain
- Client sends TLS heartbeat with small payload but claims a huge payload length.
- Server copies that many bytes out of process heap into the response without bounds-checking.
- Attacker repeats across connections to harvest adjacent buffers (sessions, occasionally key material).
- Operators disable heartbeat, upgrade TLS library, and often mass-rotate certificates fearing key compromise.
Advanced controls & design pressure
- Prioritize rapid patch SLOs for TLS stacks; chaos-test reload paths on busy proxies.
- Assume ephemeral session bytes could leak: rotate certs and scrutinize long-lived RSA keys after suspected exposure.
- Prefer TLS 1.3 + memory-safe TLS implementations where feasible; shrink optional extensions.
- Combine transport crypto with session binding, theft detection, and mutual auth inside the app: TLS alone does not stop logic bugs.
- Fuzz parsers continuously; treat heartbeat-like optional features as higher-risk surface.
Aftermath, signals & interview questions
Global scramble to patch; many orgs learned they lacked inventory of every proxy and load-balancer linked against TLS library.
- Mock interview: “How would you prove no vulnerable binaries remain in prod within 24 hours?” (dependency inventory + package queries + edge config dumps.)
- Metric spike: handshake errors vs baseline during forced rotations: signals clients pinning old chains.
Certificates · automation
automated certificate issuance: encryption became boring default, not a quarterly ticket
Before free automated certs, many teams shipped HTTP or ran brittle spreadsheets tracking expiry. automated public certificate authorities proved you can issue short-lived certificates via the certificate automation protocol: HTTPS became “cron + observability,” not a quarterly outage roulette.
Beginners benefit from noticing why short lifetimes help: stolen private keys age out faster, but only if renewal automation is more reliable than humans restarting web framework at 3am.
Simplified attack / failure chain
- Agent on each cluster proves DNS or HTTP-01 control to the CA.
- CA mints cert (~90-day validity); agent installs into ingress controllers.
- Automated renewals at ~60 days avoid last-minute human drama.
- If renewal fails, paging fires before users hit browser warnings.
Advanced controls & design pressure
- Staging certificate automation endpoints + jittered retries prevent thundering herds across millions of hosts.
- Centralize account keys; dual-sign DNS updates when using DNS-01 challenges.
- Monitor CT logs + issuance APIs: detect rogue certs for your apex domains.
- Private PKI for meshes (workload identity) vs public PKI for browsers, never reuse operational assumptions across both.
- Plan warm reload semantics so ECDSA chain swaps do not drop QUIC 0-RTT pools blindly.
Aftermath, signals & interview questions
Incident flavor shifts from “expired cert outage” to “rate-limit bans from CA”: teams learned to respect certificate automation backoff policies and multi-region challenge routing.
- Interview question: “Sketch renewal architecture for 50k micro-ingresses.”
- SLO: percentage of certs with >14 days remaining; alert if any drop below 7.
At-rest crypto · deduplication trap
Consumer sync & convergent encryption: dedupe metadata can leak file existence
Consumer sync historically hashed chunks to skip re-uploading identical files: massive bandwidth savings. The beginner twist: if chunk hashes or “file fingerprints” are visible globally, attackers may learn that a sensitive file exists somewhere on the provider without decrypting it: an existence oracle.
End-to-end encryption protects message content from intermediaries that do not hold the necessary keys. It does not automatically hide who talks to whom, message sizes, timestamps, or device identifiers. Search, recovery, and multi-device access become harder because the server cannot freely read content. Explain how devices receive keys and what happens when a user loses them; a server recovery path that can decrypt everything changes the trust claim.
Simplified attack / failure chain
- Client computes hashes of blocks before upload.
- Server responds “already have block X”: upload skipped.
- Attacker with upload capability probes hashes for known contraband/sensitive files.
- Legal/compliance teams ask whether existence leakage violates policy: engineering revisits per-user keys.
Advanced controls & design pressure
- Document threat models for confidentiality vs integrity vs existence oracles separately.
- Per-user data keys shrink cross-tenant dedupe but restore clearer isolation.
- Use managed key or hardware key envelopes when wrapping keys per tenant; audit unwrap policies.
- Pair crypto choices with logging policies: support consoles often undo theoretical E2E guarantees.
Aftermath, signals & interview questions
- Regulators increasingly ask “who can query metadata?”, not only ciphertext possession.
- Design review question: “Show me the data model row storing dedupe keys.”
payment-security · retail POS
Retail point-of-sale breach archetype: card-swipe malware vs modern tokenization
Attackers stole partner remote-access credentials, pivoted onto point-of-sale networks, and deployed RAM scrapers while magnetic stripe data was briefly plaintext in memory. Disk encryption did little because malware sat after decryption inside the trusted process.
Beginner mantra: encryption follows trust boundaries, if malware shares the same privilege as the POS app, AES only protects stolen drives, not live checkout. Disk encryption protects stored bytes when the storage device is separated from the authorized runtime. Once an application has legitimately decrypted data, malware with that application’s privileges may read it from memory or request it through normal APIs. Pair encryption with least-privilege access and workload isolation. This distinction explains why encryption at rest cannot compensate for an application that authorizes the wrong customer.
Simplified attack / failure chain
- Phished or leaked credentials for low-security partner remote access.
- Lateral movement onto POS VLAN lacking micro-segmentation.
- Custom malware scrapes track data from RAM or intercepts drivers.
- Exfiltration blends into “normal” nightly FTP habits until fraud spikes trigger banks.
Advanced controls & design pressure
- Tokenize PAN at earliest hop; keep decryption confined to hardware-backed payment-security zones.
- Default-deny mesh + workload identity: partner connectivity becomes explicit policy, not flat VLAN trust.
- File integrity monitoring + application allow lists on POS firmware paths.
- Crypto-shredding + key tiers tie consumer deletion rights to wrapped DEKs.
- Shift to chip/contactless + point-to-point encryption reduces mag-stripe exposure windows.
Aftermath, signals & interview questions
chip-card adoption accelerated; executives learned payment-security scope reduction (tokens, delegated vaults) beats arguing over checkbox audits alone.
- Interview: “Where does PAN first appear in plaintext in your architecture?”
- encrypted messaging: sudden TLS termination inside POS subnets without matching inventory tickets.
Supply chain · compression library
Upstream package compromise: trust anchors extend beyond TLS wire encryption
A volunteer-maintained compression library accepted patches that ultimately attempted to inject a build-time backdoor toward privileged remote access: proving TLS on downloads is insufficient if maintainer accounts and release tarballs are the weak link.
Beginners: encryption secures bytes in motion or at rest, but provenance secures the dependency graph: signature chains, reproducible builds, and code review bandwidth are part of crypto posture.
Simplified attack / failure chain
- Long-con social pressure on unpaid maintainers (“please help”).
- Suspicious commits merge into release branches despite subtle obfuscation.
- Autoconf/build scripts alter behavior only for specific targets.
- Researchers spot anomalies before widespread rollout: response hinges on dependency inventory + distro rebuild velocity.
Advanced controls & design pressure
- Mandatory two-person review on critical upstream deps; fund maintainers you rely on.
- Reproducible builds + signed attestations (provenance-style) comparing artifact hashes.
- Canary deployments per distro channel; differential analysis on tarball expansions.
- Runtime egress controls, even perfect TLS cannot stop a malicious binary once executed.
Aftermath, signals & interview questions
- Executive tabletop: “Which ten packages brick the internet if poisoned?”
- Metric: mean time to rebuild AMIs after upstream revoke, not just vulnerability severity scores.
Audit deepening: agility and key lifecycle
Plan algorithm agility and post-quantum migration paths before a single algorithm becomes frozen into every stored ciphertext. Track nonce uniqueness, workload-certificate lifetime, and rotation failure modes where old and new keys must coexist safely.
What must never become false
Sensitive plaintext must not leave an explicitly named trust boundary without encryption, and a retired or revoked key must not keep decrypting production data after the overlap window closes. Disk encryption alone does not satisfy this if support tools, logs, or queues still hold plaintext fields.
Failure timeline: cert expiry + KEK cutover race
T+0 Edge cert hits 0 days remaining; renewal job stuck on DNS-01 rate limit T+12m 18% of PoPs serve expired chain → browser TLS failures spike T+40m On-call force-rotates KEK while readers still cache old unwrap handles T+55m Object reads fail with decrypt errors; backups restore only with retired KEK T+2h Overlap re-enabled; old KEK readable until re-wrap job proves 100% coverage Outcome Availability + confidentiality controls collided: rotation without overlap breaks readers
Counterexample to "we encrypt everything": encryption without monitored renewal and key overlap is a scheduled outage machine.
Exercises
- Resumption falls from 90% to 80% at 33k new handshakes/sec. How many extra full handshakes/sec appear, and at 3 ms CPU each how many cores do you need before safety margin?
- 500M wrapped DEKs at 96 B each. Re-wrap at 25k/sec: how long until done? Where do you keep the old KEK enabled, and which metric says it is safe to disable?
- Product wants TLS 1.3 0-RTT on checkout. Argue for or against using the invariant and replay risk: pick an alternative for payment POSTs.
Primary references
Transport and key-management primary sources.
Short takeaway
Default TLS everywhere, automate cert renewal, envelope-encrypt data at rest, tighten keys per tenant/sensitivity, and rehearse rotation + deletion stories before auditors ask.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Protect data in transit and at rest with clear trust boundaries, algorithm agility, nonce safety, key rotation, workload certificates, and failure plans.
Explain the decision
Why is “encrypted with AES” not enough?
Read the reasoning
Mode, key size, nonce uniqueness, authentication, key storage, rotation, access policy, and boundary placement determine security.
Apply it to a production scenario
A certificate authority is unavailable while workload certificates approach expiry. What is safe?
Read one approach
Use short but buffered lifetimes, cached valid credentials, alert before exhaustion, protected emergency issuance, and never silently disable verification.
Work through the numbers
Ten thousand workloads rotate 24-hour certificates evenly. What average issuance rate?
Read the worked calculation
About 0.116 certificates/s, but synchronized restarts can create bursts, so jitter and peak capacity matter.
Reason through a failure
What must key rotation preserve?
Read the recovery reasoning
Readers can decrypt data written under active prior keys during overlap, while new writes use the new key and compromised keys can be revoked.
Sources and further study
Continue with OAuth & JWT.