Caching

What actually happens

Try it: follow a cache entry through its life

This is one application process caching one profile. A version number represents the profile’s contents: v2 is a newer edit than v1. Requests and time move only when you press a button, so you can inspect the gap between a miss starting a database read and that read filling the cache.

How to choose a cache strategy

Cache-aside invalidation flow

Deleting after a database commit is a useful baseline, but it still has a race. A reader misses and reads old version 7; a writer commits version 8 and deletes the cache key; then the first reader populates version 7 after the deletion. A TTL eventually repairs this stale fill, but strict read-after-write behavior needs additional coordination, version-aware fills, or a read routed to the authoritative store. Explain which guarantee the chosen pattern actually provides.

The rule of thumb starts with cache-aside, so here is that pattern slowed down into individual messages, first for a read that misses and then for a write that must invalidate.

sequenceDiagram
    autonumber
    participant U as User
    participant A as App
    participant C as Cache
    participant D as Database

    U->>A: Read product 123
    A->>C: GET product:123
    C-->>A: Miss
    A->>D: SELECT product 123
    D-->>A: Fresh row
    A->>C: SET product:123 with TTL+jitter
    A-->>U: Response

    U->>A: Update product 123
    A->>D: Commit write first
    D-->>A: OK
    A->>C: DELETE product:123
    A-->>U: Updated response
    Note over C,D: Next read reloads source of truth

The diagram shows one key. In practice the same product appears in several keys (see What actually happens above), so a single update has to find all of them. Good key naming, surrogate tags (a shared label on every key that belongs to one entity, so they can be purged together), version numbers in keys, and a short TTL as a safety net keep one update from turning into a manual purge incident.

Multi-layer cache hierarchy

The flow above treated "the cache" as one box. Real systems stack several, and the table below lists them from closest to the user (fast, small) to the durable source of truth (slow, complete).

When to add each tier

Do not build the whole hierarchy on day one. Add a tier only when the previous one still leaves measurable database or origin pressure.

1. No cache: every read hits the database; simplest to run, and the first to fail under read-heavy traffic.
2. Distributed cache: shared hot reads and sessions served once for all app replicas.
3. L1 plus distributed: in-process memory for tiny hot keys, removing network hops on celebrity objects.
4. CDN / edge cache: public GET responses stop before they reach the app tier at all.

Read and write patterns

Cache-aside leaves the application responsible for misses and invalidation. Write-through updates a backing store as part of the write path, while write-behind accepts writes into a buffer and persists them later. The last option changes the durability boundary: if the buffer is lost before persistence, acknowledged writes may disappear. It is appropriate only when that buffer has the required durability or the product explicitly tolerates the loss.

Whichever tiers you pick, each one needs a rule for how data gets in and how writes are handled. Cache-aside is the pattern from the flow above; two alternatives move the write into the cache itself.

def get_user(user_id):
    data = cache.get(f"user:{user_id}")
    if data:
        return json.loads(data)  # hit
    data = db.query("SELECT * FROM users WHERE id=%s", user_id)
    cache.setex(f"user:{user_id}", 300, json.dumps(data))
    return data  # miss filled

def update_user(user_id, updates):
    db.execute("UPDATE users SET ...", user_id)
    cache.delete(f"user:{user_id}")  # invalidate

# WRITE-THROUGH (sync cache + DB)
def update_user(user_id, updates):
    data = db.execute("UPDATE ... RETURNING *", user_id)
    cache.setex(f"user:{user_id}", 300, json.dumps(data))
    return data

# WRITE-BEHIND (cache now, DB async)
def increment_views(video_id):
    cache.incr(f"views:{video_id}")
    # worker batch-flushes to DB

The table adds read-through, where the cache itself (not your app code) knows how to load a missing key from the origin.

Stampede and hot keys

A stampede is duplicated refill work; a hot key is concentrated demand even when the key is present. For a stampede, let one caller refresh while others wait briefly or receive an allowed stale value, and stagger expirations. For a hot key, distribute read copies or add a small local cache because adding unrelated shards does not divide one key’s traffic. Track refill concurrency and per-key demand separately to distinguish the two problems.

The patterns above assume misses arrive one at a time. The two problems in this section are what happens when they do not: many replicas missing the same key together, or one key drawing far more traffic than its neighbours.

lock = cache.set(
    f"lock:{key}", "1",
    nx=True, ex=10)
if lock:
    data = db.query(...)
    cache.set(key, data)
else:
    sleep(0.05)
    return cache.get(key)

Consistency patterns

Stampedes and hot keys are about load. The other half of caching is freshness: how wrong is a cached value allowed to be, and how does it get corrected after a write?

Numbers to know: sizing and hit ratio

Patterns tell you what to build; numbers tell you how big. This box collects every figure on the page, so later sections point back here instead of repeating them.

cache_bytes ≈ working_set_objects × avg_value_bytes × overhead_factor

Hit ratio hygiene

If the ratio from Numbers to know comes in lower than planned, one of three causes is almost always responsible.

• Working set larger than RAM: add capacity or split into tiers so the hot set fits in memory.
• Aggressive TTL: relax it once the business confirms how much staleness is acceptable.
• Deploy cold starts: warm keys with dual-write or shadow traffic before a pod takes live reads.

Distributed cache chooser

Sizing assumed a distributed pool; the remaining choice is which kind. In the table, AOF is an append-only file that logs every write so the store can rebuild after a restart, and cluster slots are the store's built-in way of assigning keys to shards.

Case study pattern: graph-shaped caching

At the very largest scale, the distributed pool stops being one cluster and becomes an architecture of its own. This is the vocabulary from the Facebook memcache paper in the references.

Leases are the same idea as the single-flight lock in the stampede section, applied fleet-wide: the cache hands one token to the first client that misses, and only that client may repopulate the key. A routing proxy sits between the apps and the cache nodes to multiplex connections, fail over shards, and centralise policy so every app does not need its own client logic. Regional caches serve reads from the nearest data centre and propagate invalidations asynchronously, accepting a short window of staleness across regions. Gutter pools are small, short-lived overflow caches that take the traffic of a node that has disappeared, so the database never sees the full miss load while the fleet heals.

Failure timeline, outage drill, and the invariant

Everything above describes the cache working. This section is about the cache not working: a stampede second by second, what to do when the whole pool is gone, and the one rule that must survive both.

t=0      Celebrity profile: 10K QPS hits; cache hit ratio ~95%; DB calm
t=60s    TTL expires on the same second across many replicas (no jitter)
t=61s    Every replica misses; 10K identical SELECTs hit the primary
t=62s    DB p99 collapses; app timeouts; clients retry → more misses
t=70s    With single-flight + jitter + soft TTL: one refresher, peers wait/stale
Outcome  Stampede is a coordination failure, not a "need bigger DB" moment

How to explain this in an interview

With the mechanics and failures covered, this is the shape of a strong caching answer and the questions that show you understand the trade-offs.

Wrap-up