System Design · Security
Network Security Controls
A checkout POST leaves a café laptop, hits a public anycast IP, and should never be able to scan the payroll database two hops later. The packet is not “trusted” because it reached a VPC.
stacks filters that fail independently: edge , subnet , host , mesh identity, and so a stolen pod cannot phone home freely.
Checkout packet journey
TLS, WAF, origin lock-down.
Stateful allow to app ports only.
Identity, not private-IP trust.
Allow-listed destinations only.
What actually happens
traffic meets capacity and semantics first: absorb floods, terminate TLS at a controlled edge, and inspect HTTP with a . Origin addresses should not be reachable around that edge.
Inside the VPC, a is a coarse, stateless fence. A is the stateful NIC filter. Neither proves caller identity. That is why paths add mesh mTLS and .
only works if health checks, sidecars, DNS, and admin jump hosts are inventoried. watches; blocks inline and can take the site down if it fails closed on a bad signature.
How to choose controls
1. Which layer is the threat? Bandwidth and spoofed packets need provider and L3/L4 filters. Injected HTTP needs a WAF and app validation. Stolen credentials need identity, not another CIDR rule.
2. Where can you enforce without lying? Put the control on a choke point the attacker cannot walk around: private subnets, no public RDS, no alternative hostname.
3. What fails open? Name fail-open vs fail-closed for WAF, IPS, and egress proxy. Availability SLOs and security SLOs collide here.
Simple rule of thumb
Default deny, lock origins behind the edge, segment east–west by identity plus port, and treat egress as an equal control to ingress. Network location is reachability, never authorization.
Packet path through layered filters
Every hop answers a different question
Read the diagram definition
flowchart LR
Client["Client"] --> Edge["Anycast edge / WAF"]
Edge --> LB["Load balancer"]
LB --> SG["Security group"]
SG --> Mesh["Service mesh"]
Mesh --> App["App pod"]
App --> NACL["Subnet ACL"]
NACL --> DB["Private data store"]
App -.-> Egress["Egress proxy"]
Egress --> Internet["Allowed destinations"]| Control | Sees | Good at | Blind to |
|---|---|---|---|
| WAF / edge | HTTP after TLS terminate | Known web exploits, bot scores | Non-HTTP, encrypted tunnels to origin |
| NACL | IP, protocol, port, stateless | Coarse subnet quarantine | Identity, HTTP method, ephemeral replies unless mirrored |
| Security group | Stateful 5-tuple on ENI | App-to-DB port allow lists | Which principal sent the packet |
| Mesh / app | Cert, JWT, route, method | Least-privilege RPC | Raw packet floods before handshake |
Stateful groups vs stateless ACLs
Order, return traffic, and who owns the rule
Read the diagram definition
sequenceDiagram
autonumber
participant Pkt as Inbound packet
participant NACL as Subnet ACL
participant SG as Host security group
participant WAF as HTTP WAF
participant Mesh as Mesh authz
participant App as Application
Pkt->>NACL: Stateless allow/deny
alt subnet deny
NACL-->>Pkt: Drop
else subnet allow
NACL->>SG: Stateful 5-tuple
alt group deny
SG-->>Pkt: Drop
else group allow
SG->>WAF: HTTP inspect
WAF->>Mesh: Identity + route
Mesh->>App: Authorized RPC
end
endSecurity group pattern
Allow inbound 443 from the load-balancer group; allow outbound 5432 to the database group. Connection tracking admits replies. Prefer group-to-group references over raw CIDRs so scale-out does not rewrite IPs.
NACL gotcha
Ephemeral client ports (1024–65535) must be allowed on the return path or healthy TCP dies mysteriously. Keep NACLs as emergency quarantine and broad “no internet in the data subnet,” not a 200-line port novel.
Capacity math: 50k new connections/sec with 40-byte conntrack entries is ~2 MB/sec of table growth; a 30-minute idle timeout at that rate needs ~3.6 GB of state before expiry. Exhausted tables look like a network outage.
North–south vs east–west placement
Do not copy the edge policy inward
North–south
- Anycast + WAF + origin allow-list from edge IPs only.
- Private for cloud APIs; no public bucket URLs from browsers if you can avoid them.
- Admin consoles on a separate hostname with stronger posture, not “same VPC = fine.”
East–west
- so a front-end pod cannot SYN-scan Redis in another namespace.
- mTLS principals on payment RPCs; security groups still limit who can even attempt the handshake.
- Kubernetes NetworkPolicy is L3/L4 intent; it does not replace method-level authz.
Egress, IDS/IPS, and fail modes
Exfil and inline boxes are first-class design
Egress as a product
IDS vs IPS
Bypass inventory
Trade-offs
Default deny vs delivery speed. Tight groups break health checks and DNS until inventory is honest. Observe-only NetworkPolicy for two weeks beats a Friday enforce that pages everyone.
Central WAF vs per-service filters. One WAF is cheaper to staff; a bypassed listener still needs service-level validation. Do both for money paths.
Stateful tracking vs scale. Conntrack is RAM and timeout policy. Stateless filters scale easier but punish return-path mistakes.
Fail-open vs fail-closed. Egress proxy down: freeze the fleet or leak exfil? Pick per sensitivity class, not one global toggle.
Network vs identity. Groups shrink the attacker’s scanning surface. They never replace “this workload may charge a card.”
What must never become false
A packet that is merely “inside the VPC” must not be sufficient to read or mutate sensitive stores. Data-plane listeners must not be reachable on a public or forgotten path that skips the documented edge and identity checks.
Failure timeline: public origin + flat groups
T+0 WAF blocks SQLi on the pretty hostname T+5m Attacker hits the origin A record still in DNS history T+12m Origin security group allows 0.0.0.0/0:443 "for cert renewal" T+20m Compromised app pod scans 10.0.0.0/8; Redis has no password, SG is wide T+40m Egress HTTPS to a paste host: no proxy, no SNI allow list Outcome Layers existed on the slide; routing and groups did not enforce them
Exercises
- Draw north–south and east–west for a three-tier shop. Name one control that would still hold if the WAF were bypassed via a stale origin IP.
- 50k new connections/sec, 40-byte conntrack, 30-minute idle. Estimate table size. What timeout and SYN-cookie policy keep the invariant during a flood?
- Write observe-then-enforce stages for default-deny NetworkPolicy in a namespace that talks to DNS, a sidecar, and Postgres.
Choosing and explaining network security controls
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
Place independent filters on choke points: edge for HTTP and floods, subnet/host for reachability, mesh for identity, egress for exfil.
Security groups are stateful NIC rules; NACLs are stateless subnet fences. Neither is authorization.
Default deny only after you inventory health checks, DNS, and admin paths—and after you prove origins cannot be reached around the edge.
When it belongs
Any design with a public edge, private data stores, and more than one service namespace.
When the interviewer asks how a compromised pod is contained.
When multi-tenant or payment tiers share a cluster.
How it fails
Forgotten public IPs and alternate listeners bypass the WAF.
Conntrack exhaustion looks like a total outage.
Fail-open egress during proxy failure turns into silent exfil.
Stateless ACLs that forget ephemeral return ports break production.
Trade-offs to weigh
Operational friction of default deny versus blast radius of default allow.
Inline IPS latency and outage risk versus detect-only IDS.
Central WAF economics versus per-service defense in depth.
Common misunderstandings
Treating VPC membership as authentication.
Copying north–south WAF rules and calling east–west done.
IP allow lists for SaaS egress that rot behind CDNs.
Confusing NetworkPolicy (L4) with method-level authorization.
Primary references
Short takeaway
Network controls buy you a smaller door. Identity and data policy still decide who may walk through it. Draw bypasses as carefully as you draw arrows.