SwadhyAI Notes

All System Design notes

System Design Networking

DNS Deep Dive

At 02:03 the primary in us-east stops answering. App servers in eu-west are healthy, the load balancers are fine, and the on-call dashboard still shows green for every HTTP check. Users see timeouts anyway, because phones and browsers are still dialing the old IP that handed them minutes ago.

That is the whole problem this page solves. A name like chatgpt.com is useless until DNS turns it into an address, and that answer is cached worldwide. Failover, canaries, CDN edges, and multi-region HA all hitch rides on and which IP returns. When DNS fails or lags, the site feels dead even if every app server is healthy.

Name → first byte

1

Client / OS cache

Stale local answers can keep sending traffic to a retired IP.

2

On a miss, walks root → TLD → your authoritative nameservers.

3

Returns the record you publish, plus the TTL that gates caching.

4

How long resolvers and clients may keep serving that answer.

What actually happens when a name is resolved

The browser asks the OS, which asks a (usually at the ISP or a public DNS provider). On a cache miss, that resolver walks the hierarchy: root, then the TLD (the .com or .org zone), then your . The answer includes a ; every cache between you and the authority may keep serving it until that timer expires.

DNS caching means a new authoritative answer does not immediately replace answers already held by recursive resolvers, operating systems, or applications. In an illustrative failover, a health check detects failure after thirty seconds and a resolver that fetched the old answer just before detection may retain a sixty-second TTL, giving roughly ninety seconds from outage to a fresh lookup. Other clients refresh sooner, while persistent connections or stale-serving resolvers can take longer. Treat this as a timeline with assumptions, not a fixed propagation delay.

("this name does not exist") answers are cached too, via . Create records before clients probe them, or a premature lookup can fail until the negative TTL expires. Providers scale with , multiple NS records, and DDoS capacity; steers regions; helps when the resolver is far from the user; and , , and harden integrity and privacy later on this page.

How to choose DNS settings

Once you see that is both a phonebook and a slow global cache, three questions decide your settings.

1. How fast must traffic move? Active failover endpoints often use a ~60s so most clients can shift within a couple of minutes. Static infrastructure can live at 3600s because the name barely changes.

2. Is DNS the right lever? Use it for coarse region or provider steering (), not fine-grained per-request balancing. Pair it with connection draining or multi-value healthy IPs when you need smoother handoff.

3. Did you wait out the old TTL? Lower TTL days before a migration, wait for the old cache to expire, then cut over. Assume some clamp or ignore you.

Simple rule of thumb

Short TTL where agility matters; long TTL where names barely change. Dual-provider DNS when outage risk is existential. In simple terms: design the phonebook and the cache, not just the A record.

From problem to mechanism

Here is the 02:03 outage slowed down so you can see where the naive flip fails and what a safe design changes.

Problem. Users must reach a healthy region or CDN edge before HTTP starts. When us-east dies, traffic must move, but the internet's phonebook is a multi-layer cache you only partly control.

Naive system. One nameserver, =3600, flip the A record at cutover and expect instant failover. There is no dual-answer window, no plan for sticky caches, and no budget for the propagation tail.

Break. Primary dies; health checks take ~30s; a 60s TTL means many clients stick ~90s; a fat tail lasts hours. Premature queries cache via , and the fixed record still fails until that negative timer expires.

Mechanism. Publish health-driven answers from resilient (often plus multiple NS records), keep dual healthy IPs during cutover, and treat TTL as a clock you must wait out. Lower TTL ahead of migrations; create records before clients probe them; rehearse the fat tail with draining and multi-value answers.

Correctness invariant

A DNS cutover must remain safe while some clients still use the old destination. Ordinary caches count down the TTL on the answer they received, but RFC 8767 permits serving stale data under defined resilience policies, and application connection reuse can outlive a DNS cache entry. Negative answers and delegation records have separate caching behavior. Therefore TTL is an important planning input, not a universal hard deadline after which no client can reach the old endpoint.

Failover playbook and numbers to know

The mechanism above names health-driven answers and dual IPs; these are the concrete patterns, TTL knobs, and sizing numbers that make them real. Cross-reference this card whenever a later section mentions ~90s shift or authoritative QPS.

DNS failover patterns

plaintext
Pattern 1: Health Check + Failover (managed DNS)
  Primary:  A record → 10.0.1.1 (us-east-1)
  Secondary: A record → 10.0.2.1 (eu-west-1)
  Health check: GET /health every 10s
  Primary healthy → return 10.0.1.1
  Primary unhealthy (3 failures) → return 10.0.2.1
  Failover time: ~30s (3 checks × 10s) + TTL propagation

Pattern 2: Weighted Routing (Canary)
  api.example.com → 10.0.1.1 weight=90
  api.example.com → 10.0.2.1 weight=10
  → 90% traffic to primary, 10% to canary
  → Gradually shift: 90/10 → 50/50 → 0/100

Pattern 3: Latency-Based Routing
  api.example.com → 10.0.1.1 (us-east-1, measured latency)
  api.example.com → 10.0.2.1 (eu-west-1, measured latency)
  → managed DNS returns lowest-latency endpoint per client

Pattern 4: Geolocation Routing
  api.example.com → 10.0.1.1 (if client in US)
  api.example.com → 10.0.2.1 (if client in EU)
  api.example.com → 10.0.3.1 (if client in Asia)
  → Route based on client country/continent

Pattern 5: Multi-value Answer
  api.example.com → [10.0.1.1, 10.0.1.2, 10.0.1.3]
  → Return up to 8 healthy IPs
  → Health-checked: only healthy IPs returned

strategies

Short TTL (60s)

A short TTL can reduce how long ordinary cached answers retain an old endpoint, but total failover also includes failure detection, answer publication, resolver policy, and reconnect behavior. Use a sixty-second TTL as an example setting to test, not a promise that every client switches within sixty seconds. Pair DNS steering with a healthy destination and bounded connection recovery.

More DNS queries (higher load on authoritative). Higher latency (more DNS lookups).

Long TTL (3600s / 1 hour)

Fewer DNS queries (lower load, lower cost). Lower latency (cached locally longer).

Slow failover (clients stuck on old IP for up to 1 hour). Slow deployment propagation.

Recommended TTLs & migration strategy

plaintext
Recommended TTLs:
  Active failover endpoints:  60s  (fast failover, managed DNS health checks)
  CDN edge routing:           300s (5 min, acceptable for edge switching)
  Static infrastructure:      3600s (1 hour, rarely changes)
  NS records:                 86400s (24 hours, very stable)

Migration strategy:
  1. Normal TTL: 3600s (1 hour)
  2. 48 hours before change: lower TTL to 60s
  3. Wait 48h for old TTL to expire everywhere
  4. Make DNS change
  5. 95%+ traffic shifts within 60 seconds
  6. After migration stable: raise TTL back to 3600s

DNS propagation delays:
  → Some resolvers ignore TTL (cache longer than specified)
  → Some clients cache aggressively (Java: default infinite cache!)
  Fix: networkaddress.cache.ttl=60 in Java security properties
  → ISP resolvers may override TTL (minimum 300s common)
  → Full propagation: assume 24-48h worst case

Numbers to know

  • Failover budget: 3×10s health checks plus 60s TTL ≈ 90s for many clients.
  • Resolver multiplier: 1M clients behind 20K resolvers ≈ 20K authoritative queries/min at TTL=60s.
  • DDoS headroom: plan for 10K QPS normal rising to 10M+ under attack.
  • Negative cache trap: create records first; a 300s cache can still fail launches.

Resolver walkthrough

For a cold lookup of an example host, a recursive resolver first obtains the responsible top-level-domain servers from a root server, then the domain’s authoritative servers, then the host’s records. These intermediate delegations are cached, so later lookups usually skip much of the hierarchy. If the answer contains an alias, resolving its target may require additional queries and introduces another independently cached record.

An authoritative server answers for a zone; a recursive resolver performs the lookup on the client’s behalf. Neither is the application load balancer that handles the eventual TCP or QUIC connection. Separating these roles makes an incident easier to diagnose: a valid cached address does not establish that the destination is healthy, and a healthy destination does not help a client whose name lookup fails.

  1. Client and OS cache check first; a stale local answer can ignore your new IP.
  2. asks root, TLD, then authoritative on a cache miss.
  3. returns the record TTL; the resolver caches until it expires.
  4. carries a negative TTL, so premature queries can cache "does not exist."

Migration checklist

Lower the relevant TTL early enough for records carrying the previous longer TTL to expire. Publish and test the new endpoint before shifting traffic, retain the old endpoint during a measured overlap window, and verify both authoritative answers and real client success. Switching back is also a DNS change with caching delay, so prepare application-level compatibility for both destinations instead of treating rollback as instantaneous.

  • Lower TTL days ahead, then wait the old TTL to expire everywhere.
  • Publish both old and new answers, observe traffic, keep a rollback window.
  • Remove old records only after sticky and clamped caches have drained.

Authoritative DNS at scale

Resolver behavior during an authority outage can include serving expired cached data under the conditions in RFC 8767, Serving Stale Data. This improves availability while reinforcing why a migration must remain safe for clients retaining the previous destination.

The playbook assumes someone answers for your domain under load. That someone is , scaled with , multiple NS records, and DDoS headroom (see Numbers to know above for the 10K→10M QPS jump).

Authoritative DNS (your domain's nameservers)

Receives queries for YOUR domain from recursive resolvers worldwide.

Scaling strategies

plaintext
1. Anycast: same IP announced from 100+ locations
   → Query goes to nearest nameserver
   → Public resolver networks: 300+ cities, < 10ms global
   → Managed DNS: dozens of edge locations

2. Horizontal scaling: multiple NS records
   example.com.  NS  ns1.example.com.
   example.com.  NS  ns2.example.com.
   example.com.  NS  ns3.example.com.
   example.com.  NS  ns4.example.com.
   → Resolvers distribute queries across NS records

3. Over-provisioning for DDoS
   Normal load: 10K queries/sec
   DDoS attack: 10M+ queries/sec (1000× normal)
   → Must have capacity for attack traffic
   → Managed DNS providers handle this at fleet scale

Managed DNS & public resolvers

Managed DNS:

  • Manages huge numbers of DNS zones for many customers at once.
  • Global edge locations via anycast so queries stay nearby.
  • High availability SLAs that treat names as critical infrastructure.
  • Handles DNS DDoS with provider-scale shielding and capacity.

Public resolver networks:

  • Trillions of queries per day across hundreds of PoPs worldwide.
  • Low teens of ms average global response time for most lookups.
  • Free tiers handle millions of queries per month for end users.

Integrity, privacy, steering, and other jobs DNS does

Scale keeps names answering; these layers decide whether answers are trustworthy, private, and useful for routing beyond a single A record.

and encrypted lookups ( / )

plaintext
DNSSEC signs DNS records:
  → Resolver verifies signature
  → Detects tampered responses
  → Chain of trust: Root → TLD → Domain

Without DNSSEC: Attacker intercepts DNS → returns fake IP
With DNSSEC: Response includes RRSIG (signature), resolver checks validity

Adoption: ~30% of .com domains (2024)
Managed DNS: full DNSSEC support
Public DNS providers: often one-click DNSSEC
plaintext
DNS over HTTPS (DoH):
  DNS queries over HTTPS (port 443)
  → Encrypted, ISP can't see queries
  → Used by modern browsers

DNS over TLS (DoT):
  DNS queries over TLS (port 853)
  → Used by: Android Private DNS

Privacy: Traditional = ISP sees all queries; DoH/DoT = encrypted only
Performance: Traditional ~1ms; DoH ~5-20ms (connection reuse helps)

and

How CDNs route users to a nearby edge

plaintext
1. User queries: cdn.example.com
2. Authoritative DNS sees resolver IP (e.g., US-East ISP resolver)
3. Returns IP of nearest healthy edge PoP (e.g., Newark PoP)
4. User connects to Newark edge → low latency

EDNS Client Subnet (ECS)

plaintext
Problem: resolver IP ≠ user IP (e.g., resolver and user may be in different regions)
ECS: resolver sends truncated client IP to authoritative
→ Authoritative can route based on actual user location

dig @8.8.8.8 cdn.example.com +subnet=203.0.113.0/24
→ Authoritative sees client is in 203.0.113.0/24 → returns Asia PoP

DNS + Anycast + BGP & hybrid approach

plaintext
Public resolvers and CDNs often combine DNS + Anycast:
1. DNS returns anycast IP (same IP globally)
2. BGP routes to nearest PoP automatically
3. Simpler, faster failover (BGP converges in seconds)
→ But anycast doesn't work well for stateful TCP

Hybrid approach (most common):
  DNS GSLB for coarse routing (region-level)
  Anycast for fine routing within region (PoP-level)
  Health checks for removing unhealthy regions

DNS as an infrastructure building block

DNSSEC validates DNS data through signatures and a chain of trust; it does not encrypt the query. DNS over HTTPS or TLS encrypts the connection to the chosen resolver but does not prove every authoritative record is signed. Geographic steering uses information such as resolver location or optional client-subnet hints, which can affect cache sharing and privacy. Choose each feature for the specific property it supplies.

plaintext
Service Discovery (container orchestrator):
  my-service.namespace.svc.cluster.local → ClusterIP
  CoreDNS resolves within cluster, no external DNS needed

Blue-Green Deployments:
  api.example.com CNAME → api-blue.example.com (current)
  Deploy green → test → switch CNAME → api-green.example.com
  Rollback: switch CNAME back to blue (TTL delay!)

Multi-CDN:
  static.example.com CNAME → cdn.cloudflare.com (primary)
  Failover: CNAME → cdn.fastly.com (backup)

Email Deliverability (SPF, DKIM, DMARC):
  example.com TXT "v=spf1 include:_spf.google.com ~all"
  selector._domainkey.example.com TXT "v=DKIM1; k=rsa; p=..."
  _dmarc.example.com TXT "v=DMARC1; p=reject; rua=..."

Certificate Validation:
  _acme-challenge.example.com TXT "..."
  → Let's Encrypt uses DNS to prove domain ownership
  → Required for wildcard certificates

Failure timeline, and what still breaks after the fix

Putting the pieces together, here is how a region failover plays out under , the signals that catch a bad cutover, and the ways an improved design can still go wrong.

Timeline: region failover under TTL

T0  us-east primary healthy; A record TTL=60s → 10.0.1.1
T1  Primary dies; health check every 10s, need 3 failures
T2  T+30s managed DNS starts returning eu-west 10.0.2.1
T3  Resolvers with fresh cache still serve 10.0.1.1 until their TTL expires
T4  Most clients shift by ~T+90s; sticky/clamped caches stretch the tail
T5  Ops assumed "DNS failover = instant" → error budget already burned

Pair DNS with connection draining, multi-value healthy IPs, or where stateful TCP allows, and rehearse the fat tail (see Numbers to know above).

What to watch

  • NXDOMAIN and SERVFAIL rates, so bad publishes or signing failures show up early.
  • Authoritative latency and health-check flap, which precede wrong answers under load.
  • Traffic still hitting retired IPs after cutover, proving sticky caches remain.

Still fails when…

  • Dual-provider DNS publishes inconsistent answers, or Java clients ignore TTL entirely.
  • plus a long CNAME chain multiplies TTLs and delays every flip.
  • improves geo but leaks prefix privacy and can fragment caches.
  • Health checks probe a healthy LB while the app is dead, steering into a black hole.
  • misconfiguration (expired signatures, broken DS) turns resolvers into hard failures.

Trade-offs

  • Low TTL improves agility but raises lookup volume on authoritative servers.
  • Anycast fails over fast, yet needs strong route-health automation to stay correct.
  • GeoDNS improves locality, but public resolvers can still misroute distant users.

Choosing and explaining DNS

A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.

Explain the core idea

DNS is both name resolution and a global cache hierarchy, so TTL is the main design lever.

Use DNS for coarse traffic steering and failover, not per-request load balancing.

Plan for resolver caching behavior you do not fully control.

When it belongs

Global users need to reach the nearest or healthiest region.

The system needs provider, region, or CDN failover.

A design starts before the first HTTP request reaches your edge.

How it fails

Long TTLs delay failover.

Short TTLs increase authoritative DNS load.

Recursive resolvers ignore or clamp TTLs.

Trade-offs to weigh

Low TTL improves agility but can raise lookup volume.

Anycast is fast but needs strong route health automation.

GeoDNS improves locality but can misroute through public resolvers.

Common misunderstandings

Assuming DNS changes are instant.

Using DNS alone for fine-grained load balancing.

Forgetting negative caching and stale resolver behavior.

Try it: DNS TTL and resolver cache

Resolve a hostname, cache the RRset, then change the authoritative target or wait past TTL. Compare what a cached resolver returns with a fresh upstream lookup.

Authoritative zone

api.example.com → 203.0.113.10, 203.0.113.11

Resolver cache

Empty for this name.

Cache entries
0
Cached valid
—
Stale risk after IP change
No

Resolve a name against cache or authoritative DNS. Change authoritative records and advance TTL to see stale cached answers.

Going deeper

Production deep dive: DNS controls names, not packet routes

DNS resolution traverses caches before reaching authoritative servers. Its TTL-based convergence is useful for coarse traffic steering, but it cannot instantly revoke cached answers or control the network path after an address is chosen.

What must stay true

Authoritative DNS may be unavailable without breaking cached traffic, and every planned record change has a rollback that remains valid through the maximum observed cache lifetime.

How the pieces fit

Previous TTL

300 s

Lower it before migration, not at the cutover instant.

Negative TTL

60 s

NXDOMAIN may remain cached after the record is created.

Measured tail

≈ 8 min

Some clients/connections exceed declared TTL; observe reality.

Recursive and authoritative flow

The stub asks a recursive resolver. On cache miss, the resolver follows root and TLD delegations to authoritative nameservers, validates DNSSEC when enabled, caches the RRset, and returns it. The application then opens a connection; DNS is normally absent from subsequent packets.

Negative caching

NXDOMAIN and some empty answers are cached using the zone SOA negative-cache policy. Creating a missing record does not instantly fix clients that cached its absence; pre-create names and include negative TTL in incident timelines.

TTL limitations and failover timing

Resolver, OS, runtime, browser, and connection caches can differ. Lower TTL at least one old-TTL window before a cutover, keep old endpoints healthy for the measured tail, shift weight gradually, and restore old weights for rollback rather than deleting records.

DNSSEC failure modes

Expired signatures, bad DS/DNSKEY rollover, clock skew, or oversized fragmented responses can produce SERVFAIL for validating resolvers. Monitor validation from multiple resolvers and stage key rollover with overlapping signatures.

DNS versus Anycast and BGP

DNS selects an address. Anycast lets many sites advertise the same address, while BGP chooses the network route to that announcement. With Anycast failover the DNS answer may not change at all; BGP withdrawal and connection retry drive recovery.

Worked example: Safe regional DNS failover timeline

  1. 1T−10 min: lower a 300-second record to 30 seconds and confirm old TTL has aged out; keep both regions healthy.
  2. 2T0: shift 10%, 50%, then 100% weighted answers while monitoring resolver and application success.
  3. 3T+8 min: retain old capacity because measured cache/connection tail is eight minutes, despite the advertised 30-second TTL.

DNS failover RTO is the observed resolver + connection tail and application retry behavior, not simply the new TTL.

DNS failure matrix

ScenarioConsequenceHow to contain it
Authoritative outageNew cache misses fail while cached answers work.Independent authoritative providers, anycast, long-enough emergency cache, monitor delegation.
Bad DNSSEC rolloverValidating resolvers return SERVFAIL.Prepublish keys, overlap signatures, validate chain externally, keep rollback material.
NXDOMAIN cachedFreshly created service still appears absent.Pre-provision name or wait negative TTL; do not churn names during incident.

What to watch in production

  • Monitor authoritative latency/errors, delegation, DNSSEC expiry, resolver-specific answers, cache-hit behavior, and connection success by returned IP.
  • Test changes through several public and enterprise resolvers before and after cutover.
  • Maintain a record-change runbook containing old TTL wait, gradual weights, endpoint drain, and reversible rollback.

A scenario to reason through

You change an A record from region A to B with TTL 30 seconds, but traffic still reaches A five minutes later. Is DNS broken?

One solid approach

Not necessarily. A resolver may have cached the previous 300-second TTL, or clients may reuse live connections. Inspect answer TTLs and connection age; leave A draining until the measured tail ends.

Going deeper

DNS deep dive

The mechanism and its boundary

DNS is a distributed cache hierarchy for names. Its power is scale and loose coupling; its danger is slow control because TTLs, resolver behavior, and negative caching delay changes.

Name resolution cache chain

1

Client cache

2

Resolver cache

3

Authority

4

Address answer

Work through the sizing assumptions

A 300-second TTL means many clients can keep an old answer for up to 5 minutes after failover.
If 20M clients refresh a 60-second TTL uniformly, authoritative systems may see around 333,000 queries/sec before resolver caching reduces it.

When it belongs

Users need to enter a stable name while services move across addresses or regions. Global traffic steering or failover happens before application routing. The design needs cacheable, decentralized name resolution.

Assumptions that matter

Resolvers cache aggressively and may not honor very low TTLs exactly. Clients retry and cache differently by platform. Application health checks still need to remove bad targets.

Easy mistakes

Changing an authoritative record cannot instantly replace answers already cached with a long TTL, so planned migrations lower the TTL before the cutover. A cached negative answer can keep a newly created name unresolved until its negative-cache lifetime expires, so test both successful and failed lookup paths. DNS alone does not choose a backend afresh for every request, because resolvers cache answers and applications reuse connections.

How it fails

Stale answers continue routing to a failed region. Authoritative outage prevents cache misses from resolving. Misconfigured records break every client before traffic reaches the app.

Trade-offs to name

A long TTL allows repeated lookups to reuse cached answers and reduces authoritative query load, but also delays planned routing changes for clients with cached records. A shorter TTL creates more opportunities to learn a new answer while increasing refresh traffic, and still does not force every application or existing connection to switch immediately. DNS is useful for broad regional steering, while an application proxy makes finer decisions using request details and backend health at the cost of another serving dependency.

Wrap-up

You have seen the failure, the mechanism, and the operational signals. Use these exercises to check the ideas stuck, then follow the references for the primary sources.

Exercises

  1. Plan a blue-green cutover. Current TTL is 3600s; cutover is in 72 hours. Write the TTL schedule, dual-answer window, and rollback rule. What fraction of traffic might still hit blue 30 minutes after the flip?
  2. Size authoritative QPS. 5M MAU, ~40K recursive resolvers, failover endpoint TTL=30s. Estimate steady authoritative QPS and argue whether TTL=5s is worth the load.
  3. Negative cache incident. A canary hostname was queried before the record existed; NXDOMAIN cached 300s. Propose process and SOA negative-TTL policy so this cannot recur on launch day.

Topic-specific references

Key takeaway

DNS is the foundation of global routing and a cache you only partially control. Use anycast for authoritative scale, short TTLs for failover endpoints, and ECS when geo accuracy matters. Plan TTL changes days ahead; dual providers beat a single DNS SPOF.

Practice and further reading

Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.

Focus for this lesson: Trace stub/recursive/authoritative resolution, positive/negative caches, delegation, TTL, failover timing, DNSSEC, and separation from routing.

Explain the decision

Why can lowering TTL five minutes before a migration fail?

Read the reasoning

Resolvers may still hold older records until their prior TTL expires; lower it at least one old-TTL window in advance.

Apply it to a production scenario

A mistaken NXDOMAIN is cached after recovery. What controls duration?

Read one approach

Negative TTL from SOA/RFC rules, recursive resolver behavior, and client caches; monitor both positive and negative paths.

Work through the numbers

Two million clients resolve every 300 s with perfect staggering. Estimate average queries/s reaching their resolvers.

Read the worked calculation

About 6,667 QPS; authoritative load depends heavily on recursive cache sharing and TTL.

Reason through a failure

What must DNSSEC validation preserve?

Read the recovery reasoning

Bogus signatures fail closed as designed, while key rollover maintains a valid chain of trust through overlap and propagation.