System Design Networking
DNS Deep Dive
At 02:03 the primary in us-east stops answering. App servers in eu-west are healthy, the load balancers are fine, and the on-call dashboard still shows green for every HTTP check. Users see timeouts anyway, because phones and browsers are still dialing the old IP that handed them minutes ago.
That is the whole problem this page solves. A name like chatgpt.com is useless until DNS turns it into an address, and that answer is cached worldwide. Failover, canaries, CDN edges, and multi-region HA all hitch rides on and which IP returns. When DNS fails or lags, the site feels dead even if every app server is healthy.
Name → first byte
Client / OS cache
Stale local answers can keep sending traffic to a retired IP.
On a miss, walks root → TLD → your authoritative nameservers.
Returns the record you publish, plus the TTL that gates caching.
How long resolvers and clients may keep serving that answer.
What actually happens when a name is resolved
The browser asks the OS, which asks a (usually at the ISP or a public DNS provider). On a cache miss, that resolver walks the hierarchy: root, then the TLD (the .com or .org zone), then your . The answer includes a ; every cache between you and the authority may keep serving it until that timer expires.
DNS caching means a new authoritative answer does not immediately replace answers already held by recursive resolvers, operating systems, or applications. In an illustrative failover, a health check detects failure after thirty seconds and a resolver that fetched the old answer just before detection may retain a sixty-second TTL, giving roughly ninety seconds from outage to a fresh lookup. Other clients refresh sooner, while persistent connections or stale-serving resolvers can take longer. Treat this as a timeline with assumptions, not a fixed propagation delay.
("this name does not exist") answers are cached too, via . Create records before clients probe them, or a premature lookup can fail until the negative TTL expires. Providers scale with , multiple NS records, and DDoS capacity; steers regions; helps when the resolver is far from the user; and , , and harden integrity and privacy later on this page.
How to choose DNS settings
Once you see that is both a phonebook and a slow global cache, three questions decide your settings.
1. How fast must traffic move? Active failover endpoints often use a ~60s so most clients can shift within a couple of minutes. Static infrastructure can live at 3600s because the name barely changes.
2. Is DNS the right lever? Use it for coarse region or provider steering (), not fine-grained per-request balancing. Pair it with connection draining or multi-value healthy IPs when you need smoother handoff.
3. Did you wait out the old TTL? Lower TTL days before a migration, wait for the old cache to expire, then cut over. Assume some clamp or ignore you.
Simple rule of thumb
Short TTL where agility matters; long TTL where names barely change. Dual-provider DNS when outage risk is existential. In simple terms: design the phonebook and the cache, not just the A record.
From problem to mechanism
Here is the 02:03 outage slowed down so you can see where the naive flip fails and what a safe design changes.
Problem. Users must reach a healthy region or CDN edge before HTTP starts. When us-east dies, traffic must move, but the internet's phonebook is a multi-layer cache you only partly control.
Naive system. One nameserver, =3600, flip the A record at cutover and expect instant failover. There is no dual-answer window, no plan for sticky caches, and no budget for the propagation tail.
Break. Primary dies; health checks take ~30s; a 60s TTL means many clients stick ~90s; a fat tail lasts hours. Premature queries cache via , and the fixed record still fails until that negative timer expires.
Mechanism. Publish health-driven answers from resilient (often plus multiple NS records), keep dual healthy IPs during cutover, and treat TTL as a clock you must wait out. Lower TTL ahead of migrations; create records before clients probe them; rehearse the fat tail with draining and multi-value answers.
Correctness invariant
A DNS cutover must remain safe while some clients still use the old destination. Ordinary caches count down the TTL on the answer they received, but RFC 8767 permits serving stale data under defined resilience policies, and application connection reuse can outlive a DNS cache entry. Negative answers and delegation records have separate caching behavior. Therefore TTL is an important planning input, not a universal hard deadline after which no client can reach the old endpoint.
Failover playbook and numbers to know
The mechanism above names health-driven answers and dual IPs; these are the concrete patterns, TTL knobs, and sizing numbers that make them real. Cross-reference this card whenever a later section mentions ~90s shift or authoritative QPS.
DNS failover patterns
Pattern 1: Health Check + Failover (managed DNS)
Primary: A record → 10.0.1.1 (us-east-1)
Secondary: A record → 10.0.2.1 (eu-west-1)
Health check: GET /health every 10s
Primary healthy → return 10.0.1.1
Primary unhealthy (3 failures) → return 10.0.2.1
Failover time: ~30s (3 checks × 10s) + TTL propagation
Pattern 2: Weighted Routing (Canary)
api.example.com → 10.0.1.1 weight=90
api.example.com → 10.0.2.1 weight=10
→ 90% traffic to primary, 10% to canary
→ Gradually shift: 90/10 → 50/50 → 0/100
Pattern 3: Latency-Based Routing
api.example.com → 10.0.1.1 (us-east-1, measured latency)
api.example.com → 10.0.2.1 (eu-west-1, measured latency)
→ managed DNS returns lowest-latency endpoint per client
Pattern 4: Geolocation Routing
api.example.com → 10.0.1.1 (if client in US)
api.example.com → 10.0.2.1 (if client in EU)
api.example.com → 10.0.3.1 (if client in Asia)
→ Route based on client country/continent
Pattern 5: Multi-value Answer
api.example.com → [10.0.1.1, 10.0.1.2, 10.0.1.3]
→ Return up to 8 healthy IPs
→ Health-checked: only healthy IPs returnedstrategies
Short TTL (60s)
A short TTL can reduce how long ordinary cached answers retain an old endpoint, but total failover also includes failure detection, answer publication, resolver policy, and reconnect behavior. Use a sixty-second TTL as an example setting to test, not a promise that every client switches within sixty seconds. Pair DNS steering with a healthy destination and bounded connection recovery.
More DNS queries (higher load on authoritative). Higher latency (more DNS lookups).
Long TTL (3600s / 1 hour)
Fewer DNS queries (lower load, lower cost). Lower latency (cached locally longer).
Slow failover (clients stuck on old IP for up to 1 hour). Slow deployment propagation.
Recommended TTLs & migration strategy
Recommended TTLs:
Active failover endpoints: 60s (fast failover, managed DNS health checks)
CDN edge routing: 300s (5 min, acceptable for edge switching)
Static infrastructure: 3600s (1 hour, rarely changes)
NS records: 86400s (24 hours, very stable)
Migration strategy:
1. Normal TTL: 3600s (1 hour)
2. 48 hours before change: lower TTL to 60s
3. Wait 48h for old TTL to expire everywhere
4. Make DNS change
5. 95%+ traffic shifts within 60 seconds
6. After migration stable: raise TTL back to 3600s
DNS propagation delays:
→ Some resolvers ignore TTL (cache longer than specified)
→ Some clients cache aggressively (Java: default infinite cache!)
Fix: networkaddress.cache.ttl=60 in Java security properties
→ ISP resolvers may override TTL (minimum 300s common)
→ Full propagation: assume 24-48h worst caseNumbers to know
- Failover budget: 3×10s health checks plus 60s TTL ≈ 90s for many clients.
- Resolver multiplier: 1M clients behind 20K resolvers ≈ 20K authoritative queries/min at TTL=60s.
- DDoS headroom: plan for 10K QPS normal rising to 10M+ under attack.
- Negative cache trap: create records first; a 300s cache can still fail launches.
Resolver walkthrough
For a cold lookup of an example host, a recursive resolver first obtains the responsible top-level-domain servers from a root server, then the domain’s authoritative servers, then the host’s records. These intermediate delegations are cached, so later lookups usually skip much of the hierarchy. If the answer contains an alias, resolving its target may require additional queries and introduces another independently cached record.
An authoritative server answers for a zone; a recursive resolver performs the lookup on the client’s behalf. Neither is the application load balancer that handles the eventual TCP or QUIC connection. Separating these roles makes an incident easier to diagnose: a valid cached address does not establish that the destination is healthy, and a healthy destination does not help a client whose name lookup fails.
- Client and OS cache check first; a stale local answer can ignore your new IP.
- asks root, TLD, then authoritative on a cache miss.
- returns the record TTL; the resolver caches until it expires.
- carries a negative TTL, so premature queries can cache "does not exist."
Migration checklist
Lower the relevant TTL early enough for records carrying the previous longer TTL to expire. Publish and test the new endpoint before shifting traffic, retain the old endpoint during a measured overlap window, and verify both authoritative answers and real client success. Switching back is also a DNS change with caching delay, so prepare application-level compatibility for both destinations instead of treating rollback as instantaneous.
- Lower TTL days ahead, then wait the old TTL to expire everywhere.
- Publish both old and new answers, observe traffic, keep a rollback window.
- Remove old records only after sticky and clamped caches have drained.
Authoritative DNS at scale
Resolver behavior during an authority outage can include serving expired cached data under the conditions in RFC 8767, Serving Stale Data. This improves availability while reinforcing why a migration must remain safe for clients retaining the previous destination.
The playbook assumes someone answers for your domain under load. That someone is , scaled with , multiple NS records, and DDoS headroom (see Numbers to know above for the 10K→10M QPS jump).
Authoritative DNS (your domain's nameservers)
Receives queries for YOUR domain from recursive resolvers worldwide.
Scaling strategies
1. Anycast: same IP announced from 100+ locations
→ Query goes to nearest nameserver
→ Public resolver networks: 300+ cities, < 10ms global
→ Managed DNS: dozens of edge locations
2. Horizontal scaling: multiple NS records
example.com. NS ns1.example.com.
example.com. NS ns2.example.com.
example.com. NS ns3.example.com.
example.com. NS ns4.example.com.
→ Resolvers distribute queries across NS records
3. Over-provisioning for DDoS
Normal load: 10K queries/sec
DDoS attack: 10M+ queries/sec (1000× normal)
→ Must have capacity for attack traffic
→ Managed DNS providers handle this at fleet scaleManaged DNS & public resolvers
Managed DNS:
- Manages huge numbers of DNS zones for many customers at once.
- Global edge locations via anycast so queries stay nearby.
- High availability SLAs that treat names as critical infrastructure.
- Handles DNS DDoS with provider-scale shielding and capacity.
Public resolver networks:
- Trillions of queries per day across hundreds of PoPs worldwide.
- Low teens of ms average global response time for most lookups.
- Free tiers handle millions of queries per month for end users.
Integrity, privacy, steering, and other jobs DNS does
Scale keeps names answering; these layers decide whether answers are trustworthy, private, and useful for routing beyond a single A record.
and encrypted lookups ( / )
DNSSEC signs DNS records:
→ Resolver verifies signature
→ Detects tampered responses
→ Chain of trust: Root → TLD → Domain
Without DNSSEC: Attacker intercepts DNS → returns fake IP
With DNSSEC: Response includes RRSIG (signature), resolver checks validity
Adoption: ~30% of .com domains (2024)
Managed DNS: full DNSSEC support
Public DNS providers: often one-click DNSSECDNS over HTTPS (DoH):
DNS queries over HTTPS (port 443)
→ Encrypted, ISP can't see queries
→ Used by modern browsers
DNS over TLS (DoT):
DNS queries over TLS (port 853)
→ Used by: Android Private DNS
Privacy: Traditional = ISP sees all queries; DoH/DoT = encrypted only
Performance: Traditional ~1ms; DoH ~5-20ms (connection reuse helps)and
How CDNs route users to a nearby edge
1. User queries: cdn.example.com
2. Authoritative DNS sees resolver IP (e.g., US-East ISP resolver)
3. Returns IP of nearest healthy edge PoP (e.g., Newark PoP)
4. User connects to Newark edge → low latencyEDNS Client Subnet (ECS)
Problem: resolver IP ≠ user IP (e.g., resolver and user may be in different regions)
ECS: resolver sends truncated client IP to authoritative
→ Authoritative can route based on actual user location
dig @8.8.8.8 cdn.example.com +subnet=203.0.113.0/24
→ Authoritative sees client is in 203.0.113.0/24 → returns Asia PoPDNS + Anycast + BGP & hybrid approach
Public resolvers and CDNs often combine DNS + Anycast:
1. DNS returns anycast IP (same IP globally)
2. BGP routes to nearest PoP automatically
3. Simpler, faster failover (BGP converges in seconds)
→ But anycast doesn't work well for stateful TCP
Hybrid approach (most common):
DNS GSLB for coarse routing (region-level)
Anycast for fine routing within region (PoP-level)
Health checks for removing unhealthy regionsDNS as an infrastructure building block
DNSSEC validates DNS data through signatures and a chain of trust; it does not encrypt the query. DNS over HTTPS or TLS encrypts the connection to the chosen resolver but does not prove every authoritative record is signed. Geographic steering uses information such as resolver location or optional client-subnet hints, which can affect cache sharing and privacy. Choose each feature for the specific property it supplies.
Service Discovery (container orchestrator):
my-service.namespace.svc.cluster.local → ClusterIP
CoreDNS resolves within cluster, no external DNS needed
Blue-Green Deployments:
api.example.com CNAME → api-blue.example.com (current)
Deploy green → test → switch CNAME → api-green.example.com
Rollback: switch CNAME back to blue (TTL delay!)
Multi-CDN:
static.example.com CNAME → cdn.cloudflare.com (primary)
Failover: CNAME → cdn.fastly.com (backup)
Email Deliverability (SPF, DKIM, DMARC):
example.com TXT "v=spf1 include:_spf.google.com ~all"
selector._domainkey.example.com TXT "v=DKIM1; k=rsa; p=..."
_dmarc.example.com TXT "v=DMARC1; p=reject; rua=..."
Certificate Validation:
_acme-challenge.example.com TXT "..."
→ Let's Encrypt uses DNS to prove domain ownership
→ Required for wildcard certificatesFailure timeline, and what still breaks after the fix
Putting the pieces together, here is how a region failover plays out under , the signals that catch a bad cutover, and the ways an improved design can still go wrong.
Timeline: region failover under TTL
T0 us-east primary healthy; A record TTL=60s → 10.0.1.1 T1 Primary dies; health check every 10s, need 3 failures T2 T+30s managed DNS starts returning eu-west 10.0.2.1 T3 Resolvers with fresh cache still serve 10.0.1.1 until their TTL expires T4 Most clients shift by ~T+90s; sticky/clamped caches stretch the tail T5 Ops assumed "DNS failover = instant" → error budget already burned
Pair DNS with connection draining, multi-value healthy IPs, or where stateful TCP allows, and rehearse the fat tail (see Numbers to know above).
What to watch
- NXDOMAIN and SERVFAIL rates, so bad publishes or signing failures show up early.
- Authoritative latency and health-check flap, which precede wrong answers under load.
- Traffic still hitting retired IPs after cutover, proving sticky caches remain.
Still fails when…
- Dual-provider DNS publishes inconsistent answers, or Java clients ignore TTL entirely.
- plus a long CNAME chain multiplies TTLs and delays every flip.
- improves geo but leaks prefix privacy and can fragment caches.
- Health checks probe a healthy LB while the app is dead, steering into a black hole.
- misconfiguration (expired signatures, broken DS) turns resolvers into hard failures.
Trade-offs
- Low TTL improves agility but raises lookup volume on authoritative servers.
- Anycast fails over fast, yet needs strong route-health automation to stay correct.
- GeoDNS improves locality, but public resolvers can still misroute distant users.
Choosing and explaining DNS
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
DNS is both name resolution and a global cache hierarchy, so TTL is the main design lever.
Use DNS for coarse traffic steering and failover, not per-request load balancing.
Plan for resolver caching behavior you do not fully control.
When it belongs
Global users need to reach the nearest or healthiest region.
The system needs provider, region, or CDN failover.
A design starts before the first HTTP request reaches your edge.
How it fails
Long TTLs delay failover.
Short TTLs increase authoritative DNS load.
Recursive resolvers ignore or clamp TTLs.
Trade-offs to weigh
Low TTL improves agility but can raise lookup volume.
Anycast is fast but needs strong route health automation.
GeoDNS improves locality but can misroute through public resolvers.
Common misunderstandings
Assuming DNS changes are instant.
Using DNS alone for fine-grained load balancing.
Forgetting negative caching and stale resolver behavior.
Try it: DNS TTL and resolver cache
Resolve a hostname, cache the RRset, then change the authoritative target or wait past TTL. Compare what a cached resolver returns with a fresh upstream lookup.
Authoritative zone
api.example.com → 203.0.113.10, 203.0.113.11
Resolver cache
Empty for this name.
- Cache entries
- 0
- Cached valid
- —
- Stale risk after IP change
- No
Resolve a name against cache or authoritative DNS. Change authoritative records and advance TTL to see stale cached answers.
Going deeper
Production deep dive: DNS controls names, not packet routes
DNS resolution traverses caches before reaching authoritative servers. Its TTL-based convergence is useful for coarse traffic steering, but it cannot instantly revoke cached answers or control the network path after an address is chosen.
What must stay true
Authoritative DNS may be unavailable without breaking cached traffic, and every planned record change has a rollback that remains valid through the maximum observed cache lifetime.
How the pieces fit
Application stub
local / runtime cache
Recursive resolver
positive + negative cache
Root / TLD
delegation
Authoritative DNS
signed RRset
A / AAAA answer
connect by IP
Previous TTL
300 s
Lower it before migration, not at the cutover instant.
Negative TTL
60 s
NXDOMAIN may remain cached after the record is created.
Measured tail
≈ 8 min
Some clients/connections exceed declared TTL; observe reality.
Recursive and authoritative flow
The stub asks a recursive resolver. On cache miss, the resolver follows root and TLD delegations to authoritative nameservers, validates DNSSEC when enabled, caches the RRset, and returns it. The application then opens a connection; DNS is normally absent from subsequent packets.
Negative caching
NXDOMAIN and some empty answers are cached using the zone SOA negative-cache policy. Creating a missing record does not instantly fix clients that cached its absence; pre-create names and include negative TTL in incident timelines.
TTL limitations and failover timing
Resolver, OS, runtime, browser, and connection caches can differ. Lower TTL at least one old-TTL window before a cutover, keep old endpoints healthy for the measured tail, shift weight gradually, and restore old weights for rollback rather than deleting records.
DNSSEC failure modes
Expired signatures, bad DS/DNSKEY rollover, clock skew, or oversized fragmented responses can produce SERVFAIL for validating resolvers. Monitor validation from multiple resolvers and stage key rollover with overlapping signatures.
DNS versus Anycast and BGP
DNS selects an address. Anycast lets many sites advertise the same address, while BGP chooses the network route to that announcement. With Anycast failover the DNS answer may not change at all; BGP withdrawal and connection retry drive recovery.
Worked example: Safe regional DNS failover timeline
- 1T−10 min: lower a 300-second record to 30 seconds and confirm old TTL has aged out; keep both regions healthy.
- 2T0: shift 10%, 50%, then 100% weighted answers while monitoring resolver and application success.
- 3T+8 min: retain old capacity because measured cache/connection tail is eight minutes, despite the advertised 30-second TTL.
DNS failover RTO is the observed resolver + connection tail and application retry behavior, not simply the new TTL.
DNS failure matrix
| Scenario | Consequence | How to contain it |
|---|---|---|
| Authoritative outage | New cache misses fail while cached answers work. | Independent authoritative providers, anycast, long-enough emergency cache, monitor delegation. |
| Bad DNSSEC rollover | Validating resolvers return SERVFAIL. | Prepublish keys, overlap signatures, validate chain externally, keep rollback material. |
| NXDOMAIN cached | Freshly created service still appears absent. | Pre-provision name or wait negative TTL; do not churn names during incident. |
What to watch in production
- Monitor authoritative latency/errors, delegation, DNSSEC expiry, resolver-specific answers, cache-hit behavior, and connection success by returned IP.
- Test changes through several public and enterprise resolvers before and after cutover.
- Maintain a record-change runbook containing old TTL wait, gradual weights, endpoint drain, and reversible rollback.
A scenario to reason through
You change an A record from region A to B with TTL 30 seconds, but traffic still reaches A five minutes later. Is DNS broken?
One solid approach
Not necessarily. A resolver may have cached the previous 300-second TTL, or clients may reuse live connections. Inspect answer TTLs and connection age; leave A draining until the measured tail ends.
Going deeper
DNS deep dive
The mechanism and its boundary
DNS is a distributed cache hierarchy for names. Its power is scale and loose coupling; its danger is slow control because TTLs, resolver behavior, and negative caching delay changes.
Name resolution cache chain
Client cache
Resolver cache
Authority
Address answer
Work through the sizing assumptions
When it belongs
Users need to enter a stable name while services move across addresses or regions. Global traffic steering or failover happens before application routing. The design needs cacheable, decentralized name resolution.
Assumptions that matter
Resolvers cache aggressively and may not honor very low TTLs exactly. Clients retry and cache differently by platform. Application health checks still need to remove bad targets.
Easy mistakes
Changing an authoritative record cannot instantly replace answers already cached with a long TTL, so planned migrations lower the TTL before the cutover. A cached negative answer can keep a newly created name unresolved until its negative-cache lifetime expires, so test both successful and failed lookup paths. DNS alone does not choose a backend afresh for every request, because resolvers cache answers and applications reuse connections.
How it fails
Stale answers continue routing to a failed region. Authoritative outage prevents cache misses from resolving. Misconfigured records break every client before traffic reaches the app.
Trade-offs to name
A long TTL allows repeated lookups to reuse cached answers and reduces authoritative query load, but also delays planned routing changes for clients with cached records. A shorter TTL creates more opportunities to learn a new answer while increasing refresh traffic, and still does not force every application or existing connection to switch immediately. DNS is useful for broad regional steering, while an application proxy makes finer decisions using request details and backend health at the cost of another serving dependency.
Wrap-up
You have seen the failure, the mechanism, and the operational signals. Use these exercises to check the ideas stuck, then follow the references for the primary sources.
Exercises
- Plan a blue-green cutover. Current TTL is 3600s; cutover is in 72 hours. Write the TTL schedule, dual-answer window, and rollback rule. What fraction of traffic might still hit blue 30 minutes after the flip?
- Size authoritative QPS. 5M MAU, ~40K recursive resolvers, failover endpoint TTL=30s. Estimate steady authoritative QPS and argue whether TTL=5s is worth the load.
- Negative cache incident. A canary hostname was queried before the record existed; NXDOMAIN cached 300s. Propose process and SOA negative-TTL policy so this cannot recur on launch day.
Topic-specific references
- RFC 1035: Domain names (implementation) : record types, TTL, and the resolver model.
- RFC 2308: Negative caching : NXDOMAIN / NODATA caching and SOA TTL rules.
- RFC 4033: DNSSEC introduction : chain of trust; pair with DoH/DoT RFCs 8484 and 7858.
Key takeaway
DNS is the foundation of global routing and a cache you only partially control. Use anycast for authoritative scale, short TTLs for failover endpoints, and ECS when geo accuracy matters. Plan TTL changes days ahead; dual providers beat a single DNS SPOF.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Trace stub/recursive/authoritative resolution, positive/negative caches, delegation, TTL, failover timing, DNSSEC, and separation from routing.
Explain the decision
Why can lowering TTL five minutes before a migration fail?
Read the reasoning
Resolvers may still hold older records until their prior TTL expires; lower it at least one old-TTL window in advance.
Apply it to a production scenario
A mistaken NXDOMAIN is cached after recovery. What controls duration?
Read one approach
Negative TTL from SOA/RFC rules, recursive resolver behavior, and client caches; monitor both positive and negative paths.
Work through the numbers
Two million clients resolve every 300 s with perfect staggering. Estimate average queries/s reaching their resolvers.
Read the worked calculation
About 6,667 QPS; authoritative load depends heavily on recursive cache sharing and TTL.
Reason through a failure
What must DNSSEC validation preserve?
Read the recovery reasoning
Bogus signatures fail closed as designed, while key rollover maintains a valid chain of trust through overlap and propagation.
Sources and further study
- RFC 1034 — DNS Concepts and Facilities
- RFC 2308 — Negative Caching of DNS Queries
- RFC 4033 — DNS Security Introduction
Continue with Saga Pattern.