System Design · Security
Security Models in Distributed Systems
Payments, warehouse ETL, and a 3 a.m. support session share one company and several threat models. A warehouse job that can unwrap card keys has already lost, even if every TLS box is green.
Pick a model per : properties, , then or (or capabilities) so one compromised identity cannot act as the whole system.
Model journey
Name which property each hop protects.
Edge, app, data, keys, analytics.
RBAC, ABAC, or capabilities.
Blast radius of one credential.
What actually happens
Distributed systems add untrusted networks, partial failure, and multiple operators. The says hop-by-hop TLS and checksums do not replace endpoint integrity of a transfer or a payment intent.
Access control is not one knob. lets owners share. enforces labels even against owners. Product SaaS usually combines for humans with or for tenant-scoped APIs.
Classic failure shapes still apply: the (privileged proxy uses caller input as tenant), and (ACL changed between check and write). is how you keep those bugs from becoming company-wide incidents.
How to choose a model
1. Which CIA property is non-negotiable? Card data leans confidentiality and integrity; a status page leans availability. Do not encrypt yourself into an unrecoverable outage without a named break-glass model.
2. Who is the adversary inside the box? Network observer, curious DBA, compromised worker, malicious tenant. Each implies a different domain and key story.
3. How dynamic is authorization? Stable org charts fit RBAC. Tenant + time + ticket + device need ABAC or capabilities, plus policy tests in CI.
Simple rule of thumb
Draw trust domains first, pick CIA priorities per domain, then choose RBAC for coarse human roles and ABAC/capabilities for tenant-scoped machine actions. Never let analytics identities unwrap payment keys.
Trust domains
Compromise of one box should not imply compromise of all
Read the diagram definition
flowchart TB
subgraph Internet["Untrusted internet"]
User["User / partner"]
end
subgraph Edge["Edge domain"]
Proxy["Identity-aware proxy"]
end
subgraph App["Application domain"]
API["API pods"]
Worker["Workers"]
end
subgraph Data["Data domain"]
OLTP["Primary DB"]
Analytics["Warehouse"]
end
subgraph Keys["Key domain"]
KMS["Key service"]
end
User --> Proxy
Proxy --> API
API --> OLTP
Worker --> Analytics
API --> KMS
Worker -.->|"must not unwrap PAN keys"| KMS| Domain | Adversary you still fear | Typical control |
|---|---|---|
| Edge | Internet + stolen browser session | TLS, WAF, identity-aware proxy |
| Application | RCE in one service | mTLS, scoped roles, no shared SA |
| Data | SQL as app role, backup theft | Row policy, envelope encryption |
| Keys | Unwrap API misuse | KMS policy, reason codes, split duties |
Access-control families
DAC, MAC, RBAC, ABAC, capabilities
Read the diagram definition
flowchart LR
Sub["Subject"] --> DAC["DAC: owner ACL"]
Sub --> MAC["MAC: labels vs clearance"]
Sub --> RBAC["RBAC: role grants"]
Sub --> ABAC["ABAC: attributes + policy"]
DAC --> Dec{"Allow?"}
MAC --> Dec
RBAC --> Dec
ABAC --> Dec
Dec -->|"yes"| Res["Resource action"]
Dec -->|"no"| Deny["Deny + audit"]Bell–LaPadula and Biba, translated
Clark–Wilson (integrity via well-formed transactions)
RBAC explosion vs ABAC
Distributed-system wrinkles
Deputies, races, partitions, Byzantine lies
Confused deputy
An image-resize service with a file:// or internal URL fetch is the textbook deputy. So is an admin GraphQL resolver that takes tenantId from the JSON body instead of the signed session. Bind the resource to the token’s audience and tenant claim.
TOCTOU and replicas
Check ACL on replica A, write on replica B after a grant was revoked. Use the same transaction, a fencing token, or a resource version in the authz ticket. Eventual consistency of IAM is a security bug until you measure max revocation lag.
Crash vs Byzantine
Most product systems assume crash-stop plus authenticated channels (mTLS). If a replica may lie about committed data, you need Byzantine-aware protocols or an external source of truth—not “trust the majority of pods.”
End-to-end vs hop-by-hop
Mesh mTLS protects against the network, not against a logging sidecars that prints bodies. If the threat is the operator or a compromised proxy, only endpoint encryption and field-level crypto match the claim.
Scale sketch: policy decisions
80k RPS × 3 authorization checks = 240k PDP lookups/sec. At 800 bytes per decision log, raw audit is ~192 MB/sec before indexes. Cache allow decisions for 2 seconds on idempotent GETs if policy version is in the cache key; never cache “charge card” allows. Measure p99 policy latency as part of the API SLO, not a side dashboard.
Trade-offs
Simplicity of RBAC vs precision of ABAC. Roles onboard humans quickly. Attributes encode reality. Un-tested ABAC is a new outage class.
Capability passing vs ambient authority. Capabilities shrink confused-deputy bugs and make revocation a token problem. They are harder to debug than “user is admin.”
Isolation vs cost. Separate key domains and clusters raise the price of a breach and the price of every deploy. Share only what you can revoke.
Confidentiality vs operability. Field encryption that blocks all support queries will be bypassed with a shared break-glass that undoes the model.
Availability vs strict MAC-style labels. Failing closed on missing attributes is correct for money and wrong for a public read-only status page. Split the models.
What must never become false
A credential minted for one trust domain must not be sufficient to act in a higher-sensitivity domain (analytics must not unwrap payment DEKs; a tenant token must not name a different tenant in the body and succeed). Revocation lag must be a measured SLO, not a rumor.
Failure timeline: deputy + shared role
T+0 Support role can impersonate any tenant "for debugging" T+8m Ticket bot posts an internal URL; image proxy fetches it (deputy) T+15m Warehouse SA reuses the OLTP role because IAM was "too many roles" T+25m ETL job unwraps KEKs meant for checkout; dumps land in a world-readable bucket T+1d Audit shows user=etl, action=Decrypt, reason=none Outcome CIA confidentiality collapsed because domains shared ambient authority
Exercises
- Map edge, app, OLTP, warehouse, and KMS for a payments API. For each domain, name the adversary that is still in-scope after the previous domain is honest.
- Replace a 40-role RBAC matrix with 6 job roles plus ABAC tenant/env attributes. What tests prove a support engineer cannot read another tenant after hours?
- 80k RPS × 3 checks, 800-byte logs. Daily audit volume? Where is caching allowed, and what is in the cache key?
Choosing and explaining security models in distributed systems
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
Name CIA properties per trust domain, then AAA: authenticate, authorize, account.
RBAC for coarse human jobs; ABAC or capabilities for tenant-scoped machine actions; MAC-style labels when owners must not overshare.
Call out confused deputy and TOCTOU explicitly; hop-by-hop TLS is not end-to-end integrity.
When it belongs
Multi-tenant APIs, payments, or analytics that share a platform.
When asked how a compromised worker is contained.
When IAM diagrams show one role used by many pipelines.
How it fails
Ambient authority and shared service accounts across domains.
Authz on a stale replica while grants change.
Policy caches that outlive revocation.
Claiming end-to-end confidentiality while proxies log plaintext.
Trade-offs to weigh
RBAC simplicity versus ABAC precision and test burden.
Compartmentalization cost versus blast radius.
Fail-closed integrity versus availability of read paths.
Common misunderstandings
Equating encryption with a complete security model.
Treating VPC or cluster membership as a MAC label.
Confusing authentication (“who”) with authorization (“may”).
Copying Bell–LaPadula jargon without a mapped product control.
Primary references
Short takeaway
Models are how you argue that a compromise stays in its box. Draw domains, pick CIA and authz per domain, and refuse shared ambient authority between analytics and money.