System Design · Security
Authentication & authorization
You sign into ChatGPT, Uber, or Instagram and expect the app to remember you, yet still block a stolen laptop from draining payments. Login is only the first second of that story; every later API call must re-check proof and permission.
proves who is calling. decides what that caller may do. At scale you also need revocation, blast-radius limits, and auth latency treated like product latency.
Login → request journey
Cookie, session, JWT, or mTLS cert.
Edge / gateway
Validate freshness before the app.
RBAC, ABAC, or ReBAC for this action.
Enforcement + audit
Allow, deny, or step-up MFA; log why.
What actually happens
After login, the browser or app holds a cookie or bearer proof. On the next request the edge asks: is this proof still valid, and may this principal touch this resource? Humans use browsers, , and ; services use or short-lived workload tokens: mixing those worlds breeds shortcuts.
Authorization models match product shape: for corporate roles, for attribute-heavy clouds, when sharing graphs beat role matrices. removes “inside the VPN” as a free pass.
Scale math is blunt: millions of users times dozens of authenticated calls means hundreds of thousands of auth checks per second at peak. Cache cheap identity facts near the gateway; keep revocation and high-risk decisions on an observable policy path.
Try it: walk an authenticated request through the gateway
This model separates authentication from authorization. Advance through session validation, policy, enforcement, and audit. Expired proof stops before policy; a deny never reaches the app; step-up holds the request for MFA.
- Stage reached
- 0 / 5
- Decision
- Not decided
- Scenario
- valid
- HTTP to client
- —
- 1Edge gateway
- 2Session validation
- 3Policy engine
- 4Enforcement
- 5Audit stream
Session tier
AuthNValidates cookie or bearer proof, token version, and freshness.
Policy engine
AuthZRBAC, ABAC, or ReBAC for this action on this resource.
Enforcement + audit
OutcomeForward signed context, return challenge, or block with audit.
Choose a proof condition, then advance one gate at a time. The highlighted panel is the stage you just reached.
Valid path. Fresh session, allow policy, signed context forwarded, audit row written. That is the happy path every API expects at peak.
Fail fast. Expired tokens should fail at the session tier so policy CPU stays available for legitimate traffic.
How to choose auth building blocks
1. Human or service? Browser flows need sessions, MFA, and federation. Service-to-service needs rotating workload identity, not long-lived passwords in git.
2. What is the sharing model? Start with RBAC; move to ABAC or ReBAC when tenant isolation and dynamic shares explode role lists.
3. Can you revoke under load? Short TTLs, central session kill, and step-up on risk matter more than a fancy login screen.
Simple rule of thumb
Separate identity proof from every-request authorization. Prefer phishing-resistant for high-risk humans, hashed scoped for partners, and continuous verification over VPN-only trust.
In simple terms: show ID at the door, check the badge for each room, and practice taking badges away fast.
Where controls sit
Name the proof, decision point, enforcement point, and audit loop.
User or service
Edge or gateway
Authentication check
Proof
Session cookie, JWT, mTLS cert
Policy decision point
Policy enforcement point
Application or data API
Audit and risk signals
Step-up MFA or revoke
loops back to AuthN
Authn proof
A credential is evidence presented by a caller, not an identity merely because a header contains a name. A session cookie references server-side state; a signed token carries claims from an issuer; an API key identifies a permitted integration; a client certificate can identify a workload. Verify the evidence through its trusted mechanism before using the resulting identity in an authorization decision.
Authz decision
RBAC/ABAC/ReBAC evaluates action + resource + context, not just login state.
Enforcement
Gateway, sidecar, middleware, and database policy each catch different bypass paths.
Threat model & auth traffic math
Every request is an auth decision, but not every request should hit the deepest control plane.
Read the diagram definition
sequenceDiagram
autonumber
participant Client
participant Edge as Edge gateway
participant Session as Session tier
participant Policy as Policy engine
participant App
participant Audit as Audit stream
Client->>Edge: Request with cookie or bearer proof
Edge->>Session: Validate session id or token version
Session-->>Edge: Principal, tenant, freshness
Edge->>Policy: Can principal perform action on resource?
Policy-->>Edge: Allow, deny, or step-up
alt allowed
Edge->>App: Forward signed identity context
App-->>Client: Response
else denied or risky
Edge-->>Client: 401, 403, or challenge
end
Edge->>Audit: Decision, reason, latency, correlation idTraffic estimate
20M daily active users × 40 authenticated calls/day averages ~9.3k auth checks/sec; multiply by 10-20× for regional peaks, retries, and launch traffic.
Session storage
100M live sessions × 0.7-1.2 KB metadata is ~70-120 GB before replicas and indexes. Keep session blobs lean, shard by stable user/session hash, and expire aggressively.
Threat pressure
Model credential stuffing, session theft, token replay, confused-deputy authz bugs, and insider policy changes. Each needs detection and revocation, not only login success.
Auth capacity worksheet
- 50M active users × 60 authenticated calls/day ÷ 86,400 ≈ 34.7k average auth checks/sec.
- 34.7k × 12× peak × 1.3× retries ≈ 541k peak checks/sec at the edge and gateway.
- Two policy checks/request means ~1.08M policy decisions/sec before cache hits.
- One 900-byte audit event/request at peak is ~487 MB/sec raw ingest before compression.
Threat model card
- Assets
- Sessions, refresh tokens, roles, admin policies.
- Actors
- Users, services, attackers with stolen passwords, insiders.
- Trust boundaries
- Browser cookie jar, token issuer, gateway, policy engine, audit stream.
- Attack paths
- Credential stuffing, token replay, confused-deputy authorization.
- Controls
- MFA, short token lifetime, central revocation, object-level policy checks.
- Detection
- Impossible travel, deny spikes, refresh-token reuse, admin policy changes.
- Residual risk
- Delayed revocation during cache outages and stale regional policy copies.
Say it clearly
Authentication establishes a principal; authorization decides whether that principal can perform this action on this resource under this context. Cache cheap identity facts close to the gateway, but keep revocation and high-risk decisions observable through a central policy path.
Session management at scale
Sticky pods vs a shared session store · TTL, rotation, and blast-radius revocation
Anti-pattern
In-memory sessions on Server A break horizontal scaling: Server A restarts and everyone on that node loses auth mid-flow.
Centralized store pattern
A shared session store lets any application replica retrieve the same authenticated session using a high-entropy opaque session ID. This removes dependence on one app process, but the store becomes part of the authentication path. Replicate and shard it according to the required availability and revocation behavior. Avoid treating a stale replica as authoritative after logout, and define what happens if session lookup is unavailable.
- Key
- session:{session_id}
- Payload
- user_id, roles, created_at, last_active (~500 B typical)
- TTL
- ~30 min sliding window (refresh on activity)
- Footprint math
- 1B sessions × 500 B ≈ 500 GB raw: shard in-memory datastore across many primaries
Token hygiene
- Rotate session tokens ~every 15 minutes with ~60 s overlap so races don't log users out.
- On suspicious activity, revoke all sessions for that principal immediately.
Session store sizing worksheet
120M live sessions × 900 B metadata × 2 replicas × 1.4 index/fragmentation overhead ≈ 302 GB. If average Time To Live (TTL) is 30 minutes and 10% churns per minute at peak, expect ~200k create/delete operations/sec; keep shard headroom so failover does not double hot-key pressure on surviving primaries.
RBAC, ABAC, and ReBAC
Roles vs rules vs relationships · PDP latency, tuple growth, and condition keys
RBAC: roles
Role-based access control groups permissions into roles, such as editor or billing administrator, and assigns users to those roles. This is easy to explain when responsibilities are stable. Still check the resource boundary: an editor in tenant A should not edit tenant B’s posts. A role answers which kinds of actions are allowed; tenant membership and ownership determine which records those actions may target.
alice → role editor
editor → posts:create, posts:edit, posts:delete
- Pros: audits, enterprise familiarity.
- Cons: role explosion unless layered with ABAC/ReBAC.
ABAC: attributes
Attribute-based access control evaluates properties of the caller, resource, action, and environment. A rule might allow a support engineer to read an order only when the engineer belongs to the right region and has an active support case. Attributes must come from trusted sources, and their freshness matters. A stale employment or device-compliance attribute can keep access open after the reason for granting it has disappeared.
- Matchuser.department = resource.department
- Matchuser.clearance ≥ resource.classification
- Windowbusiness hours + corporate IP range
Common in cloud provider IAM condition keys, cloud policy stacks, internal PDPs.
ReBAC: relationships
Relationship-based access control models facts such as “Alice is a member of Team A” and “Team A can edit Document D.” The decision follows those relationships to determine access, including inherited sharing. Define how quickly a removed relationship must stop granting permission and how cycles or large graphs are bounded. Caching speeds decisions, but revocation must invalidate or expire the cached relationship result.
Because bob ∈ eng and eng → editor, bob inherits edit rights.
large-scale relationship authorization; use a reviewed tuple store and policy engine instead of hand-written graph checks in each service.
Multi-factor authentication
Possession vs knowledge factors · passkeys, TOTP threats, and guess throttles
TOTP (RFC 6238)
- Shared secret synchronized once; both sides hash time buckets (~30 s).
- Typical code: HMAC-SHA1(secret, floor(epoch/30)) → six digits.
- ±1 window for skew; weak against realtime phishing proxies.
phishing-resistant passkeys
- Per-origin key pairs; private keys stay inside hardware-backed key module/hardware-backed key store.
- Phishing-resistant challenges bind credentials to the RP domain.
- Broad deployment at large provider, platform provider, developer platform for high-risk flows.
Passkeys
- passkey protocol credentials synced via platform password managers.
- Removes shared passwords while retaining device-bound crypto.
- Rollout + UX still maturing across ecosystems.
Operating at hyperscale
- Shard verification POPs close to users: latency dominates perceived login quality.
- Encrypt TOTP seeds at rest with managed key or hardware key-backed wrapping keys.
- Throttle brute force (~5 tries / 15 min / principal) with device-aware escalation.
Service-to-service authentication
Why pods need identities · workload identity, JWKS rotation, audience pinning
Mutual TLS (mTLS)
Both peers present X.509 certs rooted in a mesh Certificate Authority (CA): service mesh and proxy fleets verify workload identity before forwarding.
- Client hello: workload presents workload identity/workload identity document identity.
- Server hello: downstream proves itself to prevent blind proxies.
- Verify: pin to mesh CA; map SAN/workload identity IDs to RBAC.
spiffe://cluster.local/ns/payments/sa/order-service
Automate certificate issuance and renewal using a lifetime chosen from the threat model and the reliability of the identity infrastructure. Renew before expiry with enough overlap for rollout and clock uncertainty, and alert on failed renewals. Short lifetimes reduce the usefulness of stolen credentials, but can cause broad outages if workloads cannot obtain replacements. Test recovery from issuer outages as well as normal rotation.
JWT workload tokens
Client-credentials style exchanges for compact JWT access tokens.
- Caller authenticates to STS (grant_type=client_credentials).
- Issuer returns JWT with tight TTL, explicit audience, fine scopes.
- Receiver verifies signature (JWKS), enforces aud, scope, nbf/exp.
- sub
- order-service
- aud
- payment-api
- scope
- payments:create
- TTL
- ≈ 5 minutes (refresh proactively)
Identity federation (SAML & OIDC)
Redirects and tokens · metadata trust, nonce/state, JWKS caching
SAML 2.0: enterprise SSO
- User hits service provider (enterprise applications).
- SP redirects browser to IdP (identity provider, cloud identity, …).
- User completes MFA at IdP; IdP posts signed XML assertion.
- SP verifies signatures/certs, maps attributes → groups.
Mature connectors; heavier XML stacks than JSON-native OIDC.
OpenID Connect
- Authorization-code flow at consumer IdP (large provider, developer platform, platform provider).
- Browser returns auth code to backend redirect URI.
- Exchange code for ID token (JWT) + optional access token.
- Validate iss, aud, exp, fetch JWKS.
Claims like sub, email: pair with nonce/state hardening.
Throughput narratives
- OIDC dominates consumer onboarding at planetary scale.
- Enterprise suites federate thousands of SAML tenants behind regional POPs.
- Managed CIAM proves horizontal auth, not every app needs that footprint.
Zero Trust architecture
VPN vs continuous verification · device posture and PDP placement
A large-enterprise access model story: perimeter controls failed during a targeted intrusion: identity + device posture became the movable perimeter.
Design pillars
- Authorize using user + device signals, not subnet membership alone.
- Mandatory authentication, authorization, and encryption on every path.
- Prefer policy brokers + application connectors instead of hairpin VPNs.
- Continuous device compliance scoring (patching, encryption, EDR).
- Step-up MFA when risk spikes mid-session.
Every request checkpoint
- Identity: SSO session + MFA tier by sensitivity.
- Device: MDM cert + posture telemetry.
- Policy: PDP / authz sidecars at enforcement points.
- Telemetry: centralized logs + SIEM workflows.
identity-aware access, private-access connector, conditional access platforms bundle overlapping patterns: pick integration depth vs operational burden.
API key management
Secret vs publishable keys · hashing, per-key limits, overlap rotation
Baseline controls
- Persist salted hashes / hardware key module envelopes, never plaintext secrets.
- Prefix keys (sk_live_, pk_test_) for leak grep-ability.
- Least-privilege scopes; separate keys per integration.
- Expiry (~90 days) with painless rotation UX.
- Per-key rate limits beat IP-only throttles.
- Structured audit logs for SOC workflows.
Public/private key split
- Secret keys (sk_live_…) server-side only.
- Publishable keys (pk_live_…) with domain restrictions.
- Restricted keys for partner services and microservices with narrow verbs.
- Overlap validity (~72 h) during rotation to avoid deploy cliffs.
Millions of keys online
- Shard metadata (owner_id, scopes, rpm) beside hashes.
- Cache negative lookups against brute-force sprays.
- Benchmark validation hot paths: serialization dominates naive stacks.
Case studies
Password dumps, MFA bypass tricks, and federation outages: auth failures are rarely “only” crypto bugs
Credential stuffing · billions of leaked passwords
large breach-corpus scale dumps: reused passwords beat complexity rules
Crackers aggregate billions of credential pairs from historic dumps: often sorted by mail provider, and replay them slowly against login APIs to bypass naive IP blocks. Because humans reuse passwords, correct passwords succeed without phishing.
Beginners see MFA as economics: attackers amortize compute cents per thousand tries; defenders must raise marginal cost per account takeover beyond resale value of stolen loyalty points.
Simplified attack / failure chain
- Combo lists curated per vertical (streaming, food delivery, crypto).
- Bots distribute attempts across residential proxies: rate limits mislead.
- Successful logins trigger payout fraud or crypto draining APIs.
- Victims discover via bank alerts, not your auth logs, unless ATO models fire.
Advanced controls & design pressure
- Risk-based MFA + step-up on unseen device/ASN combos.
- breach-corpus checks on enrollment + periodic forced resets for VIP cohorts.
- Per-identity throttle curves + shadow blocking suspicious refresh reuse.
- Device-bound sessions detecting sudden geo hops within token TTL.
- Partner with fraud teams: chargebacks often precede auth telemetry spikes.
Aftermath, signals & interview questions
Regulatory guidance (modern identity guidance) deprecates composition rules: push phishing-resistant factors instead.
SIM swap · social engineering
SMS OTP bypass: carriers become the weak MFA factor
Fraudsters bribe or socially engineer telecom reps into porting numbers or swapping SIMs: intercepting SMS OTPs and password resets meant for victims. Crypto exchanges historically bore disproportionate losses.
Beginners internalize SMS as account recovery surface, not cryptographic possession proof: carriers optimize churn reduction, not banking-grade assurance.
Simplified attack / failure chain
- OSINT gathers victim carrier + birthday trivia from leaks.
- Imposter convinces retail/support to activate new SIM.
- Attacker triggers “forgot password” flows across fintech stack.
- Victim regains number after assets are gone; investigation trails lag.
Advanced controls & design pressure
- phishing-resistant factors bound to platform hardware-backed key module/Hardware keys.
- Number-change alerts + delayed wires/crypto withdrawals after SIM events.
- SMS OTP allowed only with concurrent push approval on enrolled mobile.
- Treat telecom portability as hostile: monitor ASN/carrier metadata hops.
- Legal partnerships expediting carrier fraud affidavits: operations matter.
Aftermath, signals & interview questions
carrier process reforms move slowly: assume SMS MFA is unacceptable for privileged tiers indefinitely.
Federation · availability
identity-provider-class outages, when IdP downtime bricks every downstream application login
Enterprises wired dozens of application consoles through one SSO plane: DNS blips, certificate rotations, or regional outages simultaneously denied access to collaboration and incident tools precisely when incident responders needed them.
Beginners map federation uptime to business continuity: authentication latency belongs next to payment latency on exec dashboards.
Simplified attack / failure chain
- IdP deploy/regression spikes error budgets.
- SP-initiated flows fail cascade: some apps lacking offline caches.
- Engineers attempt break-glass secrets stored who-knows-where.
- Customer trust erodes when status pages disagree with user reality.
Advanced controls & design pressure
- HA IdP cells + health-checked regional stacks, not single-region vanity URLs.
- Break-glass MFA-enrolled hardware safes with tamper alarms.
- Cached JWKS + configurable skew buffers preventing thundering herds.
- Synthetic probes hitting SAML/OIDC endpoints every minute from multiple vantage points.
- Graceful degradation specs (“read-only mode”) encoded per tier-1 app.
Aftermath, signals & interview questions
Availability evidence now competes on SSO minutes-down: good benchmark questions for procurement reviews.
enterprise ticketing · enterprise desktop OS
Golden / Silver ticket narratives: forged tickets when domain keys leak
Once attackers capture KRBTGT hashes or service principal keys, they forge enterprise ticketing tickets granting prolonged domain dominance: often stealthier than dropping new local accounts that hunters grep quickly.
Beginners mapping cloud-era parallels: long-lived API signing keys or symmetric JWT secrets replicate this jackpot dynamic cross-ecosystem.
Simplified attack / failure chain
- Initial foothold escalates to DC sync rights or backup exposures.
- KRBTGT dumped offline crack/feeds Golden Ticket tooling.
- Skeleton keys authenticate silently across services trusting DC.
- Incident responders rotate KRBTGT twice: painful org-wide password resets.
Advanced controls & design pressure
- hardened admin workstations + hardened Tier-0 segmentation.
- cloud directory / cloud bridges shortening legacy enterprise ticketing reliance.
- Telemetry on abnormal ticket lifetimes + encryption types.
- Practice KRBTGT rotation drills, not theoretical wiki pages.
- Translate lessons to cloud STS: detect rogue signing key uploads.
Aftermath, signals & interview questions
platform provider publishes recurring guidance: pair reading with lab time forging tickets ethically in isolated AD labs.
passkey protocol · phishing-resistant MFA · 2023 onward
Passkeys adoption: killing shared secrets for consumer-scale login
Passkeys (passkey standard/passkey protocol) bind authentication to device-held private keys: phishing sites cannot replay challenges cross-domain the way OTP SMS fails. platform provider/large provider ecosystems syncing keys lowered UX friction that blocked earlier security-key pilots.
Beginners differentiate something you have (device hardware-backed key module) from “something interceptable” (OTP codes): deployment complexity shifts to account recovery design.
Simplified attack / failure chain
- Rolling passkeys per tenant reduces password stuffing viability sharply.
- Attackers pivot toward customer support / recovery flows: weak secondary factors.
- Mishandled cloud key backups become next trophy target.
- Observability catches surge in recovery tickets correlating with fraud.
Advanced controls & design pressure
- Pair passkeys with hardware-bound PRF extensions where regs demand.
- Device publickeys enrolled with risk scoring + geo variance alerts.
- Recovery relies on split-knowledge schemes, not reusable SMS alone.
- Federation bridges translating passkey protocol assertions into legacy SAML consumers.
- UX research on fallback pathways: security dies if users hate enrollment.
Aftermath, signals & interview questions
security teams increasingly prefer phishing-resistant MFA: expect interview questions comparing passkey protocol vs OTP budgets.
Short takeaway
Sketch proof, policy decision, and enforcement. Then defend latency vs assurance vs ops cost.
When someone asks how auth works, label three parts: identity proof (cookie, session, JWT, or certificate), policy decision (RBAC, ABAC, or ReBAC), and enforcement (gateway, mesh sidecar, middleware, or database policy).
Tie threats to mitigations: phishing → MFA/passkeys; stolen JWT → short TTL + rotation; lateral movement → Zero Trust + mTLS; scrapers → scoped keys + rate limits, not vague “we use OAuth.” Mature teams rehearse revocation under load, not only happy-path login.
- Humans: SAML/OIDC federation, MFA/passkeys, centralized sessions (no sticky RAM).
- Services: mTLS + workload IDs and/or STS-issued JWTs with tight TTL.
- Authorization: RBAC defaults; ABAC/ReBAC when sharing graphs explode matrices.
- Perimeter: Zero Trust posture brokers instead of VPN-only trust.
- Developer APIs: hashed secret keys, publishable-only in browsers, overlap rotations.
Verify continuously, minimize long-lived shared secrets, measure auth like product latency, and assume phishing succeeded somewhere this quarter.
Audit deepening: tenancy and revocation
Tenant isolation and policy-as-code
Encode authorization policy as versioned code, test deny/allow cases in CI, and ensure tenant IDs are mandatory on every authz decision.
Cache invalidation and recovery
Revocation must reach every place that can continue granting access. Deleting the browser cookie logs out that browser, but a stolen token or cached permission decision may remain valid elsewhere. Revoke the server session or refresh-token family, propagate a policy or session version where immediate invalidation is required, and bound cache lifetimes. Test the time until an already-issued credential stops working at each protected endpoint.
What must never become false
A principal that has been revoked (logout, force-logout, compromised session, or role removal) must not keep acting with prior privileges after the revocation window. Authentication proves who; authorization must re-check action + resource + tenant on every sensitive request: login success alone is not an entitlement.
Failure timeline: stolen session vs stale cache
T+0 Attacker steals session cookie from shared laptop T+1m Victim hits "log out all devices"; central session store deletes row T+1m+5s Edge still serves from 60s identity cache → attacker reads inbox T+2m Cache expires; next request 401s, but refresh-token reuse already minted a new session T+5m Without refresh reuse detection, attacker keeps a live principal Outcome Invariant broken until revocation + refresh reuse detection share one timeline
Counterexample to "logout is enough": if refresh tokens and gateway caches are not versioned together, the victim's logout is theater.
Exercises
- 80M live sessions × 1 KB metadata × 3 replicas. How much session RAM before indexes? If peak auth checks are 600k/sec and identity cache hit rate drops from 95% to 70%, how many extra session-store reads appear?
- You must revoke a compromised admin within 30 seconds worldwide. Choose session store kill, JWT short TTL, or both, and state which metric proves the invariant held.
- Product wants ReBAC for shared folders but RBAC for payroll. Sketch where the PDP lives and what breaks if folder-tuple replication lags 10 minutes behind the source of truth.
Primary references
AuthN/AuthZ primary sources, not generic system-design books.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Separate authentication, session lifecycle, authorization policy, tenant isolation, revocation, recovery, audit, and cache invalidation.
Explain the decision
Why is authentication at the gateway insufficient for authorization?
Read the reasoning
Services still need resource/action/tenant policy using trusted identity context; gateway authentication cannot know every domain object rule.
Apply it to a production scenario
An employee is removed from a tenant but cached permissions last 30 minutes. What changes?
Read one approach
Use versioned policy/session claims, event-driven invalidation, short high-risk cache TTLs, and authoritative checks for sensitive actions.
Work through the numbers
Five million active sessions require 160 bytes each with three replicas. Estimate raw session state.
Read the worked calculation
2.4 GB before index, allocator, expiration, and audit overhead.
Reason through a failure
What must account recovery preserve?
Read the recovery reasoning
Recovery cannot be an easier path than authentication; bind it to verified identity, revoke old sessions, rate-limit attempts, and audit changes.
Sources and further study
Continue with Encryption (TLS/SSL).