SwadhyAI Notes

All System Design notes

System Design · Security

Authentication & authorization

You sign into ChatGPT, Uber, or Instagram and expect the app to remember you, yet still block a stolen laptop from draining payments. Login is only the first second of that story; every later API call must re-check proof and permission.

proves who is calling. decides what that caller may do. At scale you also need revocation, blast-radius limits, and auth latency treated like product latency.

Login → request journey

1

Cookie, session, JWT, or mTLS cert.

2

Edge / gateway

Validate freshness before the app.

3

RBAC, ABAC, or ReBAC for this action.

4

Enforcement + audit

Allow, deny, or step-up MFA; log why.

What actually happens

After login, the browser or app holds a cookie or bearer proof. On the next request the edge asks: is this proof still valid, and may this principal touch this resource? Humans use browsers, , and ; services use or short-lived workload tokens: mixing those worlds breeds shortcuts.

Authorization models match product shape: for corporate roles, for attribute-heavy clouds, when sharing graphs beat role matrices. removes “inside the VPN” as a free pass.

Scale math is blunt: millions of users times dozens of authenticated calls means hundreds of thousands of auth checks per second at peak. Cache cheap identity facts near the gateway; keep revocation and high-risk decisions on an observable policy path.

Try it: walk an authenticated request through the gateway

This model separates authentication from authorization. Advance through session validation, policy, enforcement, and audit. Expired proof stops before policy; a deny never reaches the app; step-up holds the request for MFA.

Stage reached
0 / 5
Decision
Not decided
Scenario
valid
HTTP to client
—
  1. 1Edge gateway
  2. 2Session validation
  3. 3Policy engine
  4. 4Enforcement
  5. 5Audit stream

Session tier

AuthN

Validates cookie or bearer proof, token version, and freshness.

Waiting

Policy engine

AuthZ

RBAC, ABAC, or ReBAC for this action on this resource.

Waiting

Enforcement + audit

Outcome

Forward signed context, return challenge, or block with audit.

Waiting

Choose a proof condition, then advance one gate at a time. The highlighted panel is the stage you just reached.

Valid path. Fresh session, allow policy, signed context forwarded, audit row written. That is the happy path every API expects at peak.

Fail fast. Expired tokens should fail at the session tier so policy CPU stays available for legitimate traffic.

How to choose auth building blocks

1. Human or service? Browser flows need sessions, MFA, and federation. Service-to-service needs rotating workload identity, not long-lived passwords in git.

2. What is the sharing model? Start with RBAC; move to ABAC or ReBAC when tenant isolation and dynamic shares explode role lists.

3. Can you revoke under load? Short TTLs, central session kill, and step-up on risk matter more than a fancy login screen.

Simple rule of thumb

Separate identity proof from every-request authorization. Prefer phishing-resistant for high-risk humans, hashed scoped for partners, and continuous verification over VPN-only trust.

In simple terms: show ID at the door, check the badge for each room, and practice taking badges away fast.

Where controls sit

Name the proof, decision point, enforcement point, and audit loop.

Actor / proofControl planeEnforcementRisk loop

Authn proof

A credential is evidence presented by a caller, not an identity merely because a header contains a name. A session cookie references server-side state; a signed token carries claims from an issuer; an API key identifies a permitted integration; a client certificate can identify a workload. Verify the evidence through its trusted mechanism before using the resulting identity in an authorization decision.

Authz decision

RBAC/ABAC/ReBAC evaluates action + resource + context, not just login state.

Enforcement

Gateway, sidecar, middleware, and database policy each catch different bypass paths.

Threat model & auth traffic math

Every request is an auth decision, but not every request should hit the deepest control plane.

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.
Read the diagram definition
sequenceDiagram
    autonumber
    participant Client
    participant Edge as Edge gateway
    participant Session as Session tier
    participant Policy as Policy engine
    participant App
    participant Audit as Audit stream

    Client->>Edge: Request with cookie or bearer proof
    Edge->>Session: Validate session id or token version
    Session-->>Edge: Principal, tenant, freshness
    Edge->>Policy: Can principal perform action on resource?
    Policy-->>Edge: Allow, deny, or step-up
    alt allowed
        Edge->>App: Forward signed identity context
        App-->>Client: Response
    else denied or risky
        Edge-->>Client: 401, 403, or challenge
    end
    Edge->>Audit: Decision, reason, latency, correlation id

Traffic estimate

20M daily active users × 40 authenticated calls/day averages ~9.3k auth checks/sec; multiply by 10-20× for regional peaks, retries, and launch traffic.

Session storage

100M live sessions × 0.7-1.2 KB metadata is ~70-120 GB before replicas and indexes. Keep session blobs lean, shard by stable user/session hash, and expire aggressively.

Threat pressure

Model credential stuffing, session theft, token replay, confused-deputy authz bugs, and insider policy changes. Each needs detection and revocation, not only login success.

Auth capacity worksheet

  • 50M active users × 60 authenticated calls/day ÷ 86,400 ≈ 34.7k average auth checks/sec.
  • 34.7k × 12× peak × 1.3× retries ≈ 541k peak checks/sec at the edge and gateway.
  • Two policy checks/request means ~1.08M policy decisions/sec before cache hits.
  • One 900-byte audit event/request at peak is ~487 MB/sec raw ingest before compression.

Threat model card

Assets
Sessions, refresh tokens, roles, admin policies.
Actors
Users, services, attackers with stolen passwords, insiders.
Trust boundaries
Browser cookie jar, token issuer, gateway, policy engine, audit stream.
Attack paths
Credential stuffing, token replay, confused-deputy authorization.
Controls
MFA, short token lifetime, central revocation, object-level policy checks.
Detection
Impossible travel, deny spikes, refresh-token reuse, admin policy changes.
Residual risk
Delayed revocation during cache outages and stale regional policy copies.

Say it clearly

Authentication establishes a principal; authorization decides whether that principal can perform this action on this resource under this context. Cache cheap identity facts close to the gateway, but keep revocation and high-risk decisions observable through a central policy path.

Session management at scale

Sticky pods vs a shared session store · TTL, rotation, and blast-radius revocation

Anti-pattern

In-memory sessions on Server A break horizontal scaling: Server A restarts and everyone on that node loses auth mid-flow.

Centralized store pattern

A shared session store lets any application replica retrieve the same authenticated session using a high-entropy opaque session ID. This removes dependence on one app process, but the store becomes part of the authentication path. Replicate and shard it according to the required availability and revocation behavior. Avoid treating a stale replica as authoritative after logout, and define what happens if session lookup is unavailable.

Key
session:{session_id}
Payload
user_id, roles, created_at, last_active (~500 B typical)
TTL
~30 min sliding window (refresh on activity)
Footprint math
1B sessions × 500 B ≈ 500 GB raw: shard in-memory datastore across many primaries

Token hygiene

  • Rotate session tokens ~every 15 minutes with ~60 s overlap so races don't log users out.
  • On suspicious activity, revoke all sessions for that principal immediately.

Session store sizing worksheet

120M live sessions × 900 B metadata × 2 replicas × 1.4 index/fragmentation overhead ≈ 302 GB. If average Time To Live (TTL) is 30 minutes and 10% churns per minute at peak, expect ~200k create/delete operations/sec; keep shard headroom so failover does not double hot-key pressure on surviving primaries.

RBAC, ABAC, and ReBAC

Roles vs rules vs relationships · PDP latency, tuple growth, and condition keys

RBAC: roles

Role-based access control groups permissions into roles, such as editor or billing administrator, and assigns users to those roles. This is easy to explain when responsibilities are stable. Still check the resource boundary: an editor in tenant A should not edit tenant B’s posts. A role answers which kinds of actions are allowed; tenant membership and ownership determine which records those actions may target.

alice → role editor
editor → posts:create, posts:edit, posts:delete

  • Pros: audits, enterprise familiarity.
  • Cons: role explosion unless layered with ABAC/ReBAC.

ABAC: attributes

Attribute-based access control evaluates properties of the caller, resource, action, and environment. A rule might allow a support engineer to read an order only when the engineer belongs to the right region and has an active support case. Attributes must come from trusted sources, and their freshness matters. A stale employment or device-compliance attribute can keep access open after the reason for granting it has disappeared.

  • Matchuser.department = resource.department
  • Matchuser.clearance ≥ resource.classification
  • Windowbusiness hours + corporate IP range

Common in cloud provider IAM condition keys, cloud policy stacks, internal PDPs.

ReBAC: relationships

Relationship-based access control models facts such as “Alice is a member of Team A” and “Team A can edit Document D.” The decision follows those relationships to determine access, including inherited sharing. Define how quickly a removed relationship must stop granting permission and how cycles or large graphs are bounded. Caching speeds decisions, but revocation must invalidate or expire the cached relationship result.

doc:readme#viewer@user:alice
doc:readme#editor@group:eng
group:eng#member@user:bob

Because bob ∈ eng and eng → editor, bob inherits edit rights.

large-scale relationship authorization; use a reviewed tuple store and policy engine instead of hand-written graph checks in each service.

Multi-factor authentication

Possession vs knowledge factors · passkeys, TOTP threats, and guess throttles

Something you know
Passwords, PINs, recovery questions
Something you have
TOTP apps, SMS (weak), security keys
Something you are
Biometrics (face biometric, fingerprint templates)

TOTP (RFC 6238)

  • Shared secret synchronized once; both sides hash time buckets (~30 s).
  • Typical code: HMAC-SHA1(secret, floor(epoch/30)) → six digits.
  • ±1 window for skew; weak against realtime phishing proxies.

phishing-resistant passkeys

  • Per-origin key pairs; private keys stay inside hardware-backed key module/hardware-backed key store.
  • Phishing-resistant challenges bind credentials to the RP domain.
  • Broad deployment at large provider, platform provider, developer platform for high-risk flows.

Passkeys

  • passkey protocol credentials synced via platform password managers.
  • Removes shared passwords while retaining device-bound crypto.
  • Rollout + UX still maturing across ecosystems.

Operating at hyperscale

  • Shard verification POPs close to users: latency dominates perceived login quality.
  • Encrypt TOTP seeds at rest with managed key or hardware key-backed wrapping keys.
  • Throttle brute force (~5 tries / 15 min / principal) with device-aware escalation.

Service-to-service authentication

Why pods need identities · workload identity, JWKS rotation, audience pinning

Mutual TLS (mTLS)

Both peers present X.509 certs rooted in a mesh Certificate Authority (CA): service mesh and proxy fleets verify workload identity before forwarding.

  1. Client hello: workload presents workload identity/workload identity document identity.
  2. Server hello: downstream proves itself to prevent blind proxies.
  3. Verify: pin to mesh CA; map SAN/workload identity IDs to RBAC.

spiffe://cluster.local/ns/payments/sa/order-service

Automate certificate issuance and renewal using a lifetime chosen from the threat model and the reliability of the identity infrastructure. Renew before expiry with enough overlap for rollout and clock uncertainty, and alert on failed renewals. Short lifetimes reduce the usefulness of stolen credentials, but can cause broad outages if workloads cannot obtain replacements. Test recovery from issuer outages as well as normal rotation.

JWT workload tokens

Client-credentials style exchanges for compact JWT access tokens.

  1. Caller authenticates to STS (grant_type=client_credentials).
  2. Issuer returns JWT with tight TTL, explicit audience, fine scopes.
  3. Receiver verifies signature (JWKS), enforces aud, scope, nbf/exp.
sub
order-service
aud
payment-api
scope
payments:create
TTL
≈ 5 minutes (refresh proactively)

Identity federation (SAML & OIDC)

Redirects and tokens · metadata trust, nonce/state, JWKS caching

SAML 2.0: enterprise SSO

  1. User hits service provider (enterprise applications).
  2. SP redirects browser to IdP (identity provider, cloud identity, …).
  3. User completes MFA at IdP; IdP posts signed XML assertion.
  4. SP verifies signatures/certs, maps attributes → groups.

Mature connectors; heavier XML stacks than JSON-native OIDC.

OpenID Connect

  1. Authorization-code flow at consumer IdP (large provider, developer platform, platform provider).
  2. Browser returns auth code to backend redirect URI.
  3. Exchange code for ID token (JWT) + optional access token.
  4. Validate iss, aud, exp, fetch JWKS.

Claims like sub, email: pair with nonce/state hardening.

Throughput narratives

  • OIDC dominates consumer onboarding at planetary scale.
  • Enterprise suites federate thousands of SAML tenants behind regional POPs.
  • Managed CIAM proves horizontal auth, not every app needs that footprint.

Zero Trust architecture

VPN vs continuous verification · device posture and PDP placement

A large-enterprise access model story: perimeter controls failed during a targeted intrusion: identity + device posture became the movable perimeter.

Design pillars

  1. Authorize using user + device signals, not subnet membership alone.
  2. Mandatory authentication, authorization, and encryption on every path.
  3. Prefer policy brokers + application connectors instead of hairpin VPNs.
  4. Continuous device compliance scoring (patching, encryption, EDR).
  5. Step-up MFA when risk spikes mid-session.

Every request checkpoint

  • Identity: SSO session + MFA tier by sensitivity.
  • Device: MDM cert + posture telemetry.
  • Policy: PDP / authz sidecars at enforcement points.
  • Telemetry: centralized logs + SIEM workflows.

identity-aware access, private-access connector, conditional access platforms bundle overlapping patterns: pick integration depth vs operational burden.

API key management

Secret vs publishable keys · hashing, per-key limits, overlap rotation

Baseline controls

  1. Persist salted hashes / hardware key module envelopes, never plaintext secrets.
  2. Prefix keys (sk_live_, pk_test_) for leak grep-ability.
  3. Least-privilege scopes; separate keys per integration.
  4. Expiry (~90 days) with painless rotation UX.
  5. Per-key rate limits beat IP-only throttles.
  6. Structured audit logs for SOC workflows.

Public/private key split

  • Secret keys (sk_live_…) server-side only.
  • Publishable keys (pk_live_…) with domain restrictions.
  • Restricted keys for partner services and microservices with narrow verbs.
  • Overlap validity (~72 h) during rotation to avoid deploy cliffs.

Millions of keys online

  • Shard metadata (owner_id, scopes, rpm) beside hashes.
  • Cache negative lookups against brute-force sprays.
  • Benchmark validation hot paths: serialization dominates naive stacks.

Case studies

Password dumps, MFA bypass tricks, and federation outages: auth failures are rarely “only” crypto bugs

Credential stuffing · billions of leaked passwords

large breach-corpus scale dumps: reused passwords beat complexity rules

Crackers aggregate billions of credential pairs from historic dumps: often sorted by mail provider, and replay them slowly against login APIs to bypass naive IP blocks. Because humans reuse passwords, correct passwords succeed without phishing.

Beginners see MFA as economics: attackers amortize compute cents per thousand tries; defenders must raise marginal cost per account takeover beyond resale value of stolen loyalty points.

Simplified attack / failure chain

  1. Combo lists curated per vertical (streaming, food delivery, crypto).
  2. Bots distribute attempts across residential proxies: rate limits mislead.
  3. Successful logins trigger payout fraud or crypto draining APIs.
  4. Victims discover via bank alerts, not your auth logs, unless ATO models fire.

Advanced controls & design pressure

  • Risk-based MFA + step-up on unseen device/ASN combos.
  • breach-corpus checks on enrollment + periodic forced resets for VIP cohorts.
  • Per-identity throttle curves + shadow blocking suspicious refresh reuse.
  • Device-bound sessions detecting sudden geo hops within token TTL.
  • Partner with fraud teams: chargebacks often precede auth telemetry spikes.

Aftermath, signals & interview questions

Regulatory guidance (modern identity guidance) deprecates composition rules: push phishing-resistant factors instead.

SIM swap · social engineering

SMS OTP bypass: carriers become the weak MFA factor

Fraudsters bribe or socially engineer telecom reps into porting numbers or swapping SIMs: intercepting SMS OTPs and password resets meant for victims. Crypto exchanges historically bore disproportionate losses.

Beginners internalize SMS as account recovery surface, not cryptographic possession proof: carriers optimize churn reduction, not banking-grade assurance.

Simplified attack / failure chain

  1. OSINT gathers victim carrier + birthday trivia from leaks.
  2. Imposter convinces retail/support to activate new SIM.
  3. Attacker triggers “forgot password” flows across fintech stack.
  4. Victim regains number after assets are gone; investigation trails lag.

Advanced controls & design pressure

  • phishing-resistant factors bound to platform hardware-backed key module/Hardware keys.
  • Number-change alerts + delayed wires/crypto withdrawals after SIM events.
  • SMS OTP allowed only with concurrent push approval on enrolled mobile.
  • Treat telecom portability as hostile: monitor ASN/carrier metadata hops.
  • Legal partnerships expediting carrier fraud affidavits: operations matter.

Aftermath, signals & interview questions

carrier process reforms move slowly: assume SMS MFA is unacceptable for privileged tiers indefinitely.

Federation · availability

identity-provider-class outages, when IdP downtime bricks every downstream application login

Enterprises wired dozens of application consoles through one SSO plane: DNS blips, certificate rotations, or regional outages simultaneously denied access to collaboration and incident tools precisely when incident responders needed them.

Beginners map federation uptime to business continuity: authentication latency belongs next to payment latency on exec dashboards.

Simplified attack / failure chain

  1. IdP deploy/regression spikes error budgets.
  2. SP-initiated flows fail cascade: some apps lacking offline caches.
  3. Engineers attempt break-glass secrets stored who-knows-where.
  4. Customer trust erodes when status pages disagree with user reality.

Advanced controls & design pressure

  • HA IdP cells + health-checked regional stacks, not single-region vanity URLs.
  • Break-glass MFA-enrolled hardware safes with tamper alarms.
  • Cached JWKS + configurable skew buffers preventing thundering herds.
  • Synthetic probes hitting SAML/OIDC endpoints every minute from multiple vantage points.
  • Graceful degradation specs (“read-only mode”) encoded per tier-1 app.

Aftermath, signals & interview questions

Availability evidence now competes on SSO minutes-down: good benchmark questions for procurement reviews.

enterprise ticketing · enterprise desktop OS

Golden / Silver ticket narratives: forged tickets when domain keys leak

Once attackers capture KRBTGT hashes or service principal keys, they forge enterprise ticketing tickets granting prolonged domain dominance: often stealthier than dropping new local accounts that hunters grep quickly.

Beginners mapping cloud-era parallels: long-lived API signing keys or symmetric JWT secrets replicate this jackpot dynamic cross-ecosystem.

Simplified attack / failure chain

  1. Initial foothold escalates to DC sync rights or backup exposures.
  2. KRBTGT dumped offline crack/feeds Golden Ticket tooling.
  3. Skeleton keys authenticate silently across services trusting DC.
  4. Incident responders rotate KRBTGT twice: painful org-wide password resets.

Advanced controls & design pressure

  • hardened admin workstations + hardened Tier-0 segmentation.
  • cloud directory / cloud bridges shortening legacy enterprise ticketing reliance.
  • Telemetry on abnormal ticket lifetimes + encryption types.
  • Practice KRBTGT rotation drills, not theoretical wiki pages.
  • Translate lessons to cloud STS: detect rogue signing key uploads.

Aftermath, signals & interview questions

platform provider publishes recurring guidance: pair reading with lab time forging tickets ethically in isolated AD labs.

passkey protocol · phishing-resistant MFA · 2023 onward

Passkeys adoption: killing shared secrets for consumer-scale login

Passkeys (passkey standard/passkey protocol) bind authentication to device-held private keys: phishing sites cannot replay challenges cross-domain the way OTP SMS fails. platform provider/large provider ecosystems syncing keys lowered UX friction that blocked earlier security-key pilots.

Beginners differentiate something you have (device hardware-backed key module) from “something interceptable” (OTP codes): deployment complexity shifts to account recovery design.

Simplified attack / failure chain

  1. Rolling passkeys per tenant reduces password stuffing viability sharply.
  2. Attackers pivot toward customer support / recovery flows: weak secondary factors.
  3. Mishandled cloud key backups become next trophy target.
  4. Observability catches surge in recovery tickets correlating with fraud.

Advanced controls & design pressure

  • Pair passkeys with hardware-bound PRF extensions where regs demand.
  • Device publickeys enrolled with risk scoring + geo variance alerts.
  • Recovery relies on split-knowledge schemes, not reusable SMS alone.
  • Federation bridges translating passkey protocol assertions into legacy SAML consumers.
  • UX research on fallback pathways: security dies if users hate enrollment.

Aftermath, signals & interview questions

security teams increasingly prefer phishing-resistant MFA: expect interview questions comparing passkey protocol vs OTP budgets.

Short takeaway

Sketch proof, policy decision, and enforcement. Then defend latency vs assurance vs ops cost.

When someone asks how auth works, label three parts: identity proof (cookie, session, JWT, or certificate), policy decision (RBAC, ABAC, or ReBAC), and enforcement (gateway, mesh sidecar, middleware, or database policy).

Tie threats to mitigations: phishing → MFA/passkeys; stolen JWT → short TTL + rotation; lateral movement → Zero Trust + mTLS; scrapers → scoped keys + rate limits, not vague “we use OAuth.” Mature teams rehearse revocation under load, not only happy-path login.

  • Humans: SAML/OIDC federation, MFA/passkeys, centralized sessions (no sticky RAM).
  • Services: mTLS + workload IDs and/or STS-issued JWTs with tight TTL.
  • Authorization: RBAC defaults; ABAC/ReBAC when sharing graphs explode matrices.
  • Perimeter: Zero Trust posture brokers instead of VPN-only trust.
  • Developer APIs: hashed secret keys, publishable-only in browsers, overlap rotations.

Verify continuously, minimize long-lived shared secrets, measure auth like product latency, and assume phishing succeeded somewhere this quarter.

Audit deepening: tenancy and revocation

Tenant isolation and policy-as-code

Encode authorization policy as versioned code, test deny/allow cases in CI, and ensure tenant IDs are mandatory on every authz decision.

Cache invalidation and recovery

Revocation must reach every place that can continue granting access. Deleting the browser cookie logs out that browser, but a stolen token or cached permission decision may remain valid elsewhere. Revoke the server session or refresh-token family, propagate a policy or session version where immediate invalidation is required, and bound cache lifetimes. Test the time until an already-issued credential stops working at each protected endpoint.

What must never become false

A principal that has been revoked (logout, force-logout, compromised session, or role removal) must not keep acting with prior privileges after the revocation window. Authentication proves who; authorization must re-check action + resource + tenant on every sensitive request: login success alone is not an entitlement.

Failure timeline: stolen session vs stale cache

T+0      Attacker steals session cookie from shared laptop
T+1m     Victim hits "log out all devices"; central session store deletes row
T+1m+5s  Edge still serves from 60s identity cache → attacker reads inbox
T+2m     Cache expires; next request 401s, but refresh-token reuse already minted a new session
T+5m     Without refresh reuse detection, attacker keeps a live principal
Outcome  Invariant broken until revocation + refresh reuse detection share one timeline

Counterexample to "logout is enough": if refresh tokens and gateway caches are not versioned together, the victim's logout is theater.

Exercises

  1. 80M live sessions × 1 KB metadata × 3 replicas. How much session RAM before indexes? If peak auth checks are 600k/sec and identity cache hit rate drops from 95% to 70%, how many extra session-store reads appear?
  2. You must revoke a compromised admin within 30 seconds worldwide. Choose session store kill, JWT short TTL, or both, and state which metric proves the invariant held.
  3. Product wants ReBAC for shared folders but RBAC for payroll. Sketch where the PDP lives and what breaks if folder-tuple replication lags 10 minutes behind the source of truth.

Primary references

AuthN/AuthZ primary sources, not generic system-design books.

Practice and further reading

Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.

Focus for this lesson: Separate authentication, session lifecycle, authorization policy, tenant isolation, revocation, recovery, audit, and cache invalidation.

Explain the decision

Why is authentication at the gateway insufficient for authorization?

Read the reasoning

Services still need resource/action/tenant policy using trusted identity context; gateway authentication cannot know every domain object rule.

Apply it to a production scenario

An employee is removed from a tenant but cached permissions last 30 minutes. What changes?

Read one approach

Use versioned policy/session claims, event-driven invalidation, short high-risk cache TTLs, and authoritative checks for sensitive actions.

Work through the numbers

Five million active sessions require 160 bytes each with three replicas. Estimate raw session state.

Read the worked calculation

2.4 GB before index, allocator, expiration, and audit overhead.

Reason through a failure

What must account recovery preserve?

Read the recovery reasoning

Recovery cannot be an easier path than authentication; bind it to verified identity, revoke old sessions, rate-limit attempts, and audit changes.