System Design Security
Application Security Hardening
A thumbnail worker accepts customer uploads. The product feature is a square image. The systems feature is a parser that has historically been a remote-code-execution magnet. Hardening is how that worker can fail without taking the API, the secrets, or the VPC with it.
This note is about shrinking , shipping , isolating dangerous code, and proving what you run. It is not an API-injection catalog and not a TLS tutorial: those live in sibling security notes.
Hardening journey
Asset, attacker, entry, blast radius.
Close ports, flags, and debug paths.
Untrusted bytes leave the API process.
Signed build, SBOM, non-root runtime.
What actually happens
Start with a for the component: who benefits from breaking it, what identity it holds, and which parsers or admin planes it exposes. Then shrink the —remove unused listeners, bind admin to the control network, disable debug profiles in production images, and delete leftover “temporary” feature flags.
Remaining risky work (decoders, native extensions, template engines) moves into a under . The API process should not mmap untrusted files with the same credentials that can dump the database.
What you deploy must be the thing you reviewed. plus signed provenance ( ) lets admission controllers reject unsigned or rebuilt-in-the-wild images. Runtime policy then enforces non-root, read-only root FS, and egress allow lists as .
Naive design, and the invariant that must hold
Naive claim
“Run the monolith as root in a fat image, parse uploads in-process, keep a debug servlet for emergencies, and put a in front so we do not have to change code.”
Counterexample
A crafted image crashes the codec and executes in the API user. That user can read the service account token, reach metadata, and talk to internal admin ports the WAF never saw because the call never left the pod.
Correctness invariant
What must never become false
Untrusted input is parsed with a privilege set that cannot mint production side effects. Production images are signed, non-debug, and non-root. A config or WAF miss cannot grant a capability the binary and OS policy already forbade.
How to choose hardening depth
1. What bytes do you parse? Image, document, and proto parsers want sandboxes and . JSON-from-your-own-clients still wants size limits and schema, but not necessarily a VM per request.
2. What can this process already do? If it can refund, mint tokens, or reach a data warehouse, treat a crash as a potential pivot. Split the privileged plane.
3. How fast must you disable a feature? Without a , every parser bug is a full rollback. Switches must default off when the flag store lies.
Simple rule of thumb
Delete unused surface, run as non-root with a read-only filesystem, isolate parsers, sign what you ship, and keep a fail-closed kill switch. Use a WAF as a time-buyer after a CVE, not as the architecture.
The hardening loop
Read the diagram definition
flowchart TD
TM[Threat model and asset class] --> AS[Shrink attack surface]
AS --> BL[Secure defaults in the binary]
BL --> DP[Drop privileges and sandbox parsers]
DP --> SC[Signed, reproducible build with SBOM]
SC --> FZ[Fuzz and regression crash tests]
FZ --> RL[Progressive roll out plus kill switch]
RL --> RT[Runtime: WAF as patch, not source of truth]
RT --> TMThe loop is product work, not a once-a-year questionnaire. New endpoints, new native libraries, and new admin tools re-open surface. Revisit the model when blast radius changes—even if the feature looks small.
Isolate the dangerous work
Read the diagram definition
sequenceDiagram
autonumber
participant Client
participant API as API process
participant Queue
participant Worker as Sandboxed worker
participant OS as OS policy
participant Store
Client->>API: Upload untrusted file
API->>API: Authz, size, type allow list
API->>Queue: Job with object id
Queue->>Worker: Pull job
Worker->>OS: seccomp, no-net, read-only except scratch
alt parse succeeds
Worker->>Store: Write derived artifact
Worker-->>API: Done
else crash or timeout
OS-->>Worker: Kill
Worker-->>API: Poison flag, no retry storm
endPut codecs in a worker with no cloud credentials, no cluster network, a timeout, a memory cap, and a scratch directory that dies with the process. Poison a crashing object so retries do not become a fork-bomb. Prefer languages for new parsers; still sandbox native codecs you cannot rewrite this quarter.
OS policy (seccomp, AppArmor/SELinux, capability dropping) is the last referee if the language runtime is wrong. Measure: worker crash rate, kill timeout hits, and whether any worker packet ever reached an unexpected subnet.
Build provenance and runtime shape
Read the diagram definition
flowchart LR
Src[Reviewed source] --> CI[Hermetic builder]
Dep[Pinned dependencies] --> CI
CI --> Art[Artifact + SBOM]
CI --> Prov[Signed provenance]
Art --> Adm[Admission controller]
Prov --> Adm
Adm --> Run[Runtime with non-root, RO FS]
Run --> Egress[Egress allow list]Pin dependencies, generate an , and sign provenance so admission can reject “same tag, different bytes.” Hermetic builders reduce “it compiled on my laptop with a different OpenSSL.” Distroless or minimal base images shrink package CVE volume, but only if you actually rebuild when the base moves.
Runtime: non-root, readOnlyRootFilesystem, dropped capabilities, no privileged pods, and egress that cannot scrape instance metadata unless the job truly needs it. These controls survive an application bug that a never sees.
Secure defaults in the application itself
Frameworks should ship production profiles that refuse to boot with debug, default secrets, or open CORS-all. Error pages must not dump stacks to strangers. Admin listeners bind internally. Feature flags default off. Timeouts and body size limits exist before the first customer.
Browser hardening headers (CSP, frame ancestors, referrer policy) are useful last-mile depth for web surfaces. They are not the primary control for APIs or workers, and they are not a replacement for safe rendering—covered in API-security notes. Here they are one more default you compile in so a forgotten nginx snippet is not the only line of defense.
Dependency scanning gates builds on fixable high CVEs in reachable code. Unreachable transitive noise should not page you; reachable parsers in the upload path should. Pair SBOMs with actual call-graph or at least path-based triage.
Worked numbers
Upload service at 200 images/s, p99 decode 40 ms, 2 vCPU workers: you need about 200 × 0.04 = 8 concurrent decodes, plus headroom for slow bombs. Cap concurrency explicitly (say 16) so a zip-bomb cannot occupy the node. Memory cap 256 MB per worker; 16 × 256 MB = 4 GB reserved just for isolation—budget it, do not share with the API heap.
Kill switch: flag evaluation p99 5 ms at the edge. If the flag store is down, fail closed for new risky features (uploads off) while keeping read-only serving. That is an availability trade you choose in the threat model, not at 2am.
Provenance: verifying a signature at admission in 2 ms is cheaper than incident response after a poisoned cache. If you roll 50 services × 20 deploys/day, that is 1,000 verifications/day—trivial until you skip it “just this hotfix.”
Trade-offs and failure modes
Isolation vs latency. A sandbox hop and queue add milliseconds and operational parts. In-process decode is faster until it is an RCE. Put the hop on untrusted bytes, not on every JSON field from your own service mesh.
Minimal images vs debugability. Distroless images shrink surface and also shrink shells. Keep a separately built debug variant, never the production tag.
WAF vs code fix. Virtual patching buys hours or days. If the WAF rule is the only control, the next bypass is an outage of your own making.
Fail closed vs availability. Kill switches and missing-flag behavior can disable a feature globally. Document which features are allowed to fail open (rare) versus must go dark.
Memory-safe rewrites vs native speed. Rewrites take time; sandboxes are the bridge. Leaving C codecs in-process because “rewrite next year” is a schedule, not a control.
Still fails when…
- Workers inherit the API service account “for convenience.”
- Privileged sidecar shares the pod network with the sandbox.
- Hotfix images skip admission and provenance.
- Fuzz crashes are filed as “flaky” and never triaged.
What to measure
- Unsigned or privileged admissions (should be zero).
- Worker escape probes: unexpected egress, unexpected files.
- Time from CVE disclosure to rebuilt artifact in prod.
- Kill-switch drill success and fail-closed behavior.
Choosing and explaining application security hardening
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
Hardening is reducing attack surface, isolating untrusted work, and proving the binary you run.
Privileges and parsers do not belong in the same process identity.
WAFs and headers are depth, not the design. Kill switches must fail closed for risky features.
When it belongs
You parse untrusted files, documents, or codecs.
A service holds a powerful identity or sits on an admin plane.
You need a story for supply-chain and runtime containment after a bug.
How it fails
In-process parsers with production credentials.
Root containers, writable filesystems, and open debug ports.
Unsigned hotfixes and SBOMs nobody can query.
Kill switches that fail open when the flag store is wrong.
Trade-offs to weigh
Sandbox latency and cost versus in-process speed.
Minimal images versus production debugging.
Fail-closed feature gates versus availability.
Rewrite to memory-safe code versus sandboxing native islands.
Common misunderstandings
Calling a WAF “application hardening.”
Confusing this topic with API injection/XSS catalogs or TLS setup.
Assuming a service mesh identity makes a root container safe.
Shipping debug profiles because they were handy in staging.
Wrap-up
Hardening is a loop: model threats, delete surface, isolate parsers, sign builds, and keep a fail-closed switch. Depth only works when layers do not share one over-privileged identity.
Primary sources
Key takeaway
Treat untrusted bytes as hostile, run them without production keys, and only admit signed images into a non-root, read-only runtime. Hardening is containment and provenance, not a longer WAF rule list.