System Design · Security

DDoS protection

Black Friday checkout, a viral Instagram upload spike, or a quiet Tuesday. Then the login page starts timing out for everyone. Sometimes it is a flash crowd. Sometimes a botnet is trying to make Uber or your API unreachable.

defense assumes always-on mitigation: absorb early with and a , filter with a , and refuse to auto-scale expensive origins into attacker cost.

Defense journey

Classify the spike

bps, pps, rps, or cost per request?

Absorb bandwidth far from origin.

Challenge or block suspicious HTTP.

Protect origin budgets

Cache, queue, shed exports; keep login.

What actually happens

floods fill pipes; protocol/state attacks exhaust connection tables; abuse looks like users. makes small spoofed queries into huge inbound storms. showed small botnets can burn CPU without matching bandwidth.

Healthy designs spray traffic across PoPs, divert dirty bits through , cache reads on a , and apply at the gateway. Preserve login and checkout; shed exports and expensive search first.

Watch for : if attackers can force GPU or database scale-out, your bill becomes the outage. Hard caps, fast 429 paths, and edge absorb are part of the same story as packet filters.

Try it: layer defensive filters in front of origin

Compare normal read traffic with an elevated spike. Each layer removes work before the next: anycast absorption, CDN cache, WAF challenge, rate limit, then origin within budget.

Batches sent: 0
Absorbed at edge: 0
Served from origin: 0
Total modeled: —

Anycast edge

Layer

cumulative in lab

CDN cache

Layer

cumulative in lab

WAF challenge

Layer

cumulative in lab

Rate limit 429

Layer

cumulative in lab

Origin served

Layer

cumulative in lab

Send a batch of requests through edge absorption, CDN caching, WAF scoring, and per-route rate limits. Elevated traffic stays mostly off the origin.

Measure the axis. Bandwidth spikes need edge absorption; expensive RPS needs challenges and route budgets together.

Protect the bill. Rate limits and caps prevent auto-scale from becoming the outage when traffic is mostly junk.

How to choose controls

1. Which axis is spiking? Bandwidth and packets need edge absorption; expensive RPS needs challenges and route clamps.

2. What must stay up? Reserve capacity for login, session refresh, and payment; degrade everything else deliberately.

3. Can autoscaling hurt you? Cap scale, absorb at CDN/WAF, and page finance when cost curves go vertical.

Simple rule of thumb

Assume permanent background radiation. Automate mitigations, rehearse provider failover, measure time-to-mitigate, and treat customer communication as part of defense.

In simple terms: stop garbage early, protect scarce origin capacity, and never promise infinite scale to attackers.

Attack taxonomy & scale signals

Name the layer before picking controls

Class	Layer	Example signal	Mechanism
Volumetric	L3/L4	Terabit/sec UDP floods	Saturate pipes unless absorbed at edge
Protocol/state	L3/L4	SYN floods, RST storms	Exhaust connection/state tables
Application	L7	HTTP floods, credential stuffing	Looks like users: needs semantics + ML

Amplification families

DNS: tiny queries bouncing huge responses: guard open resolvers.
NTP/memcached: historic mega-amplification sources: patch or firewall reflectors.

Representative L7 patterns

SYN flood / Slowloris-style slow headers.
Credential stuffing on auth endpoints.
Expensive API abuse (search export, graph fan-out).

DDoS control placement

Rate limiting, WAF, and CDN solve different layers

Botnet and reflectors

Anycast edge

Scrubbing center

CDN cache

WAF and bot score

API gateway limits

Origin services

Block, challenge, or tarpit (from WAF)

Degrade to queue or stale reads (from gateway)

AttackEdge absorbControlOrigin / degrade

CDN / anycast

A volumetric attack consumes bandwidth or packet-processing capacity before business code receives a useful request. If the origin link is saturated, adding application replicas behind that same link does not restore access. Upstream filtering, distributed edge capacity, and caching can keep much of that traffic away from the origin. Ensure the origin is not directly reachable through an alternate address that bypasses those protections.

WAF / bot layer

Application-layer filtering examines the request’s meaning and cost: route, method, authentication state, cache behavior, and abuse signals. A small number of expensive exports can consume more resources than many cached reads. Start with measured baselines and tune rules using false-positive outcomes. Browser challenges are one option for suspicious interactive traffic, but machine API clients need a different admission and authentication path.

Gateway limits

Budget scarce resources at the boundary where they are consumed. Login protects password-verification workers; exports protect database scans and background slots; checkout protects inventory and payment concurrency. Combine tenant or account quotas with broader safety caps because one IP may contain many legitimate users and one attacker may use many IPs. Rejection should be cheaper than fully executing the protected request.

Origin protection budget & runbook

Preserve real-user paths while shedding attacker cost

Origin protection budget

Max clean origin traffic: 80k requests/sec with 30% failover headroom.
Cacheable read target: 95% served at edge during attack mode.
Login preservation: reserve 10k requests/sec for login, session refresh, and recovery.
Expensive endpoints: exports, search, and graph fan-out clamp first.

Attack runbook

Detect spike and classify layer: bits/sec, packets/sec, requests/sec, or cost/request.
Activate edge controls and route-specific limits with pre-approved rollback.
Protect origin by serving stale cache, queueing writes, and disabling non-critical exports.
Verify legitimate-user success rate, especially login and payment flows.
Communicate status, then review rules, false positives, and cost after recovery.

DDoS traffic estimation

Separate bandwidth, packet rate, request rate, and origin cost before choosing controls

Normal baseline

Observed spike

Compare by region, ASN, path, method

Which layer dominates the spike?

L3/L4 (bps / pps)

Edge absorb or divert

L7 (rps / cost per request)

Challenge, throttle, queue, or shed

Only shaped traffic reaches origin

Volumetric math

1M bots × 50 KB/sec is ~400 Gbps before amplification. Your app cannot code its way out if the upstream pipe is already full.

Application math

200k requests/sec at 30 ms CPU each asks for ~6,000 CPU cores if you process every request fully. Cheap denial paths matter.

Mitigation target

Define “good enough” as preserving login, checkout, and status reads while shedding expensive exports, search, and low-priority writes.

Four-axis DDoS calculator

Bandwidth: 500k HTTP requests/sec × 4 KB ≈ 16 Gbps before response traffic.
Requests: 500k requests/sec × 15 ms app CPU ≈ 7,500 CPU cores if origin handles all.
Database: 2 cache misses/request means 1M database reads/sec before blocking.
Origin savings: block 90% at edge and origin falls to 50k requests/sec, ~750 CPU cores, and 100k database reads/sec.

Threat model card

Assets: Edge capacity, origin CPU, connection tables, login endpoints.
Actors: Botnets, paid booter panels, compromised clients, noisy tenants.
Trust boundaries: Internet edge, scrubbing tier, challenge layer, gateway, origin, queue.
Attack paths: Bandwidth flood, packet flood, TLS handshakes, expensive API routes.
Controls: Anycast, WAF, route budgets, challenges, stale cache, queues.
Detection: RPS, packets/sec, cache-miss rate, login success, CPU/request.
Residual risk: False positives and upstream saturation before traffic reaches your edge.

Millions of users, layered defense

Start with baselines, then defend in layers: absorb packets before origin, challenge suspicious browsers, enforce route-specific budgets, queue non-critical writes, and communicate graceful degradation instead of promising infinite scale.

Layered defense funnel

Stop garbage before it touches precious CPUs

Edge / BGP anycast

Spray bits across hundreds of PoPs; withdraw unhealthy prefixes; sinkholes for obviously spoofed sources.

Scrubbing / CDN

Protocol sanity checks, reputation lists, geo fences: cheap classification before expensive stacks.

WAF + bot scores

HTTP semantics, web exploit signatures, behavioral clustering: block replay bots mimicking clients.

Gateway rate limits

Per IP / key / user token buckets: 429 is cheaper than spinning pods.

Application gates

Auth for costly ops, adaptive CAPTCHA, fairness queues for VIP tenants.

Clean traffic only reaches origin tiers ↓

Anycast economics

Divide impossible floods into manageable shards

Unicast choke point

A multi-Tbps burst aimed at one facility saturates uplinks: routing cannot rescue insufficient ingress capacity.

Anycast dilution

Anycast advertises the same address from multiple locations, and network routing policies determine which location receives a client’s traffic. This can spread incoming load across a provider’s edge and enable filtering before origin delivery. It does not guarantee perfectly even geographic distribution or unlimited capacity, and health-aware route changes require supporting operations. The destination selected by BGP is not necessarily the geographically nearest site.

Bots, challenges, scoring

Friction only where risk spikes

Challenge ladder

JS / invisible checks: prove browser execution cheaply before showing CAPTCHA.
Human challenge: reserve for worst bots: measure abandonment.
Proof-of-work tokens: burn attacker CPU per attempt.
Fingerprints: JA4/TLS client hints + HTTP/2 settings clustering.

Behavioral scoring

Contrast steady-state RPS, unique paths, geo entropy vs baseline.
Feed supervised models labels from verified human cohorts.
Expose scores to automation: auto challenge below threshold, block deterministic bots.

Economic denial-of-service

When scaling becomes the vulnerability

Attack traffic triggers autoscaler → attacker burns your cloud bill (cost-control incident worse than bandwidth outage).

Absorb at CDN/WAF so malicious sessions never reach costly GPU/DB tiers.
Hard caps + budgets with paging, not infinite scale curves.
429 fast paths cheaper than full request handling.
Protection contracts may credit attack-driven scale bursts: read Service Level Agreement (SLA) fine print.

Case studies

What volumetric floods, IoT botnets, and protocol quirks taught operators, beyond blocking IPs

DNS · correlated dependency · 2016

Authoritative DNS outage, when everyone shares one authoritative DNS tier

The consumer-device botnet weaponized consumer IoT with default passwords to generate enormous UDP/TCP floods aimed at a DNS provider’s authoritative DNS: sites still ran healthy origin stacks, yet DNS recursion failures blanketed whole regions.

Beginners internalize DNS as Tier-0 dependency: multi-region orchestrator means nothing if recursive resolvers cannot obtain AAAA/A answers. Your status page should monitor resolution paths, not only `/healthz`. Name resolution happens before the client can reach the application’s healthy replicas. Monitor authoritative DNS and representative external resolution paths, and document how a DNS incident affects failover. Keep status and incident-access dependencies sufficiently independent to remain useful during the same failure. Moving application traffic across regions cannot help a client that cannot discover any reachable address.

Simplified attack / failure chain

Infected cameras bootstrap C2 and synchronize attack slots.
Volumetric floods saturate authoritative edges / peers.
Major online services sharing providers face correlated brownouts.
Mitigation blends provider absorption + anycast steering + emergency TTL tweaks.

Advanced controls & design pressure

Dual independent DNS providers + IaC-tested delegation swaps.
Anycast authoritative stacks with isolation between customer zones.
Recursive resolver diversity inside corp VPN vs hotspot reliance.
Telemetry on NXDOMAIN spikes vs legitimate traffic drops: often faster than HTTP monitors.
Quarterly game-days fail provider A intentionally during business hours.

Aftermath, signals & interview questions

Wake-up call on IoT governance; many ISPs improved scanning for Telnet defaults.

Interview: “Which dependency graph ties your top 10 revenue APIs to one DNS ASN?”

HTTP/2 · vulnerability-class · 2023

HTTP/2 reset flood: cancelling streams faster than servers could bookkeep

Attackers opened immense HTTP/2 concurrency then issued RST_STREAM bursts: forcing proxies to allocate/deallocate state faster than steady legitimate workloads ever would. Small botnets achieved headline request-per-second figures without matching bandwidth spend.

Beginner framing: L7 attacks weaponize algorithmic complexity, not only Mbps. Your hottest CPUs die before optics saturate.

Simplified attack / failure chain

Bot opens multiplexed connection honoring SETTINGS generously.
Rapid stream cancel loops churn allocator/free paths on proxies.
Worker pools exhaust → latency cascades to benign traffic.
Vendors ship patches + interim connection/stream throttle knobs.

Advanced controls & design pressure

Subscribe to CDN/WAF rapid-response lists, not every vulnerability hits public vulnerability database same-day.
Adaptive concurrent-stream ceilings per IP + JA3-style fingerprints.
Canary clusters comparing CPU jiffies per accepted byte ratio vs baseline.
Origin shields + caching minimize blast when edges degrade.
Formal verification backlog on custom parsers, especially QUIC stacks hot patching HTTP/3 parity.

Aftermath, signals & interview questions

mitigation timelines became a good interview question for “coordination with infrastructure partners during private fixes.”

cache service · amplification · late 2010s

1.3 Tbps-class amplifications: tiny spoofed queries → giant UDP responses

Attackers sent tiny spoofed UDP queries to tens of thousands of misconfigured cache service instances: each replied with payloads orders of magnitude larger to the forged victim source IP. Amplification factors dwarfed DNS/NTP classics briefly.

Beginners: UDP services that reply larger than queries belong behind strict ACLs: reflection economics punish victims asymmetrically.

Simplified attack / failure chain

Scanner finds UDP/11211 exposed globally.
Spoofed GET primes mega payloads aimed at victim /32 or /128.
Transit providers see inbound spikes dwarfing victim commit bandwidth.
Mitigation = sinkholes + ACL tightening + ISP uRPF enforcement debates.

Advanced controls & design pressure

BCP38/uRPF near edges; document unavoidable asymmetric routing gaps honestly.
Automated scans for UDP amplification inventories weekly.
Flow telemetry baselines per ASN: detect “tiny egress / huge ingress” imbalance.
Contracts clarifying scrubbing center activation SLAs + communication trees.
Disable UDP entirely where TCP suffices: cache service rarely needs WAN UDP.

Aftermath, signals & interview questions

A developer platform famously sustained ~1.35 Tbps combined vectors: proved large service edges could absorb but budgets matter.
Interview: “What’s your provider’s packet-per-second ceiling vs bandwidth ceiling?”

Booters · abuse market

Stresser/Booter takedowns: teenager-priced “tests” hammer schools & rivals

Subscription booters packaged UDP floods + TCP SYN storms + application probes behind slick dashboards: priced cheaper than pizza. Victims included game hosts, school districts, and municipal portals, not headline APTs.

Beginners internalize baseline telemetry: without knowing normal SYN ratios per ASN, you cannot distinguish adolescent fireworks from precursors to ransomware distraction waves.

Simplified attack / failure chain

Buyer tops up wallet anonymously.
Panel orchestrates bot farms across compromised VPS/resolvers.
Attack rotates vectors every minutes: cheap evasion vs static ACLs.
Law enforcement traces payouts eventually, but enterprises cannot wait.

Advanced controls & design pressure

Automated SOC runbooks tiering volumetric vs L7 spikes.
Share IOC fingerprints with ISP abuse desks: many booters reuse templates.
Insurance questionnaires demand rehearsal logs: document tabletop outcomes.
Captcha/TLS fingerprint escalation pathways only after benign-user measurement.
Canary endpoints detecting asymmetric ICMP/TCP option fingerprints.

Aftermath, signals & interview questions

Global arrests disrupted major panels yet hydra effect persists: defense remains architectural, not purely legal.

public-service edge · application volumetric · 2018

Developer platform record-setting flood: proving edges absorb multi-vector terabit storms

A large developer-facing service experienced a multi-vector cache-service amplification flood yet stayed reachable thanks to massive anycast absorption and upstream coordination.

Beginners learn DDoS defense is partly capacity economics: hyperscale bandwidth buffers blunt amplification spikes while engineers tune scrubbing rules behind the scenes.

Simplified attack / failure chain

Attack peaks beyond typical commit webhook footprint orders.
Edges auto-invoke scrubbing tiers + reroute anycast announcements.
SRE validates legitimate git-over-SSH/Web flows preserved.
Post-incident notes educate ecosystem on cache service exposure hygiene.

Advanced controls & design pressure

Contract scrubbing SLAs with explicit PPS ceilings, not only Gbps.
Measure tail latency during mitigation: availability ≠ usability.
Regional failover drills assuming partial PoP brownouts.
Clear customer communication builds trust during recovery.

Aftermath, signals & interview questions

Interview gold: compare proprietary scrubbing, always-on CDN/WAF bundles, and Border Gateway Protocol (BGP) diversion architectures for your workload mix.

Commercial mitigation stacks

Illustrative, not exhaustive pricing

Offering	Notes	Sweet spot
CDN + WAF bundles	Often unlimited volumetric absorption at edge tiers	Websites & APIs fronted publicly
Baseline network protection	Baseline L3/L4 included with many services	Lift-and-shift starting posture
Advanced protection class	DRT support + cost protection clauses	Regulated / revenue-critical workloads
cloud edge protection / edge provider / etc.	Rule engines tied to each hyperscaler / CDN POP architecture	Multi-cloud + contractual SLAs

Audit deepening: economic and origin protection

Layer-7 economics and flash crowds

Expensive search or export endpoints can be exhausted without large bandwidth. Distinguish flash crowds from attacks, and automate triggers from error rate, origin CPU, and cache-bypass ratios. Use saturation and customer-impact signals to trigger protection, while examining cache-hit ratio and authenticated goodput to understand the traffic. A successful defense preserves useful completed operations, not merely a low origin request count. Track the cost of retries, logs, challenges, and autoscaling too: processing rejected traffic can itself consume a substantial share of the system.

Origin and DNS dependencies

Keep origin locked down, protect cache-bypass paths, and treat DNS providers as part of the DDoS surface with an explicit failover runbook. Restrict origin access to the intended edge or authenticated forwarding path, with a tested process for provider changes and emergency access. Include upload endpoints, administrative routes, alternate hostnames, and uncached query variants in the inventory. A single unprotected expensive path can bypass otherwise effective CDN caching and route-level controls.

What must never become false

During attack mode, reserved capacity for login, session refresh, and checkout must remain reachable for legitimate users, even if exports, search, and low-priority writes are shed. Autoscaling must not chase attacker traffic without a hard spend and origin RPS ceiling.

Failure timeline: L7 flood + economic DoS

T+0      Botnet opens 500k RPS against /search (30 ms CPU/request)
T+3m     Origin autoscaler adds pods; bill and DB read QPS climb 10×
T+8m     Login shares the same pool → real users time out (invariant at risk)
T+12m    Edge finally challenges; cache hit rises; origin falls toward budget
T+20m    Finance pages on spend; without hard caps the outage was economic
Outcome  Bandwidth looked fine; scarce CPU and shared login path were the real targets

Counterexample to "scale out until it stops": attacker-driven scale can bankrupt you while still starving login.

Exercises

500k RPS × 4 KB request bodies ≈ how many Gbps before responses? If you block 90% at the edge, what origin RPS and DB reads remain at 2 cache misses/request?
You have 80k clean origin RPS budget and must reserve 10k for login. Design shed order for export, search, and writes: name the metric that flips attack mode on.
DNS provider A is under volumetric attack. Sketch dual-provider failover without breaking the login invariant mid-cutover.

Primary references

DDoS and edge-defense primary sources.

RFC 4732: Internet Denial-of-Service Considerations BCP 38 / RFC 2827: Network Ingress Filtering Google SRE: Handling Overload

Short takeaway

Assume permanent background radiation: automate mitigations, rehearse provider failover, measure time-to-mitigate, and treat finance plus customer communication as part of defense.

Practice and further reading

Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.

Focus for this lesson: Layer anycast/edge absorption, origin isolation, cache strategy, bot economics, admission, quotas, and an automated attack-versus-flash-crowd runbook.

Explain the decision

Why is a cache-bypass endpoint dangerous during DDoS?

Read the reasoning

Small attacker requests can force expensive origin/database work, creating economic and computational amplification despite edge bandwidth capacity.

Apply it to a production scenario

Traffic rises 20x after a product launch. How do you distinguish a flash crowd from an attack?

Read one approach

Compare identity, reputation, request diversity, conversion, cache hit rate, geographic/ASN distribution, protocol behavior, and origin cost before blocking broadly.

Work through the numbers

A 3 kB request triggers 6 MB of origin work. What is amplification?

Read the worked calculation

Roughly 2,000x by bytes, before CPU/database cost; prioritize authentication, caching, and admission around that endpoint.

Reason through a failure

What must emergency filtering preserve?

Read the recovery reasoning

Critical authenticated users and control-plane access retain bounded capacity; mitigation cannot expose the hidden origin or lock out operators.

Sources and further study

Continue with API Security Best Practices.