System Design Patterns
Distributed locking
Flash-sale checkout reaches two API pods at once. Each reads stock=10, sells three units, and writes stock=7. Six units were sold but the database records a reduction of only three; the correct remaining count is four. This is a lost update, which can lead to later overselling. The first question is whether an atomic conditional decrement can enforce the stock rule directly, before introducing a separate lock service.
A is how many servers agree who may enter the critical section. On one machine a mutex is enough; across a fleet you need a lock service, a , and often a so a paused old holder cannot corrupt storage.
Lock journey
One worker wins the resource key.
Do the critical work
Read, update, write under ownership.
Crashes must not block forever.
Stale holders are rejected.
What actually happens
Many replicas may run the same cron, update the same row, or call a rate-limited partner API. A lock service (often Redis, a database row, , or ) records the winner with a unique owner id and a / so a dead client cannot hold forever.
Before you pick a backend, decide style: tries the write and retries on version conflict; blocks first. Optimistic fits rare conflicts; pessimistic fits hot inventory and expensive retries.
Locks alone are not enough under GC pauses and partitions. That is risk. A / TTL stops dead holders from blocking forever, but lease expiry without fencing is how a paused old owner still writes. Martin Kleppmann's critique of is the reminder: the protected resource must enforce ordering via a or equivalent authoritative check.
Correctness invariant
After lease expiry or failover, a stale lock holder must not successfully mutate the protected resource. The lock service picking a winner is not enough: storage (or another authoritative checker) must reject writes whose is older than the last accepted token. Lease alone ≠ mutual-exclusion safety.
Quantitative worked example
- TTL = 10s, renew every 3s. GC pause of 12s → lease expires; another worker acquires. Without fencing, the paused holder writes after wake.
- 5k inventory updates/sec with a single global lock → serialization to ~1 critical section; shard locks by SKU so contention tracks hot items only.
- Consensus lock (etcd/ZK) at ~1–5ms vs Redis SET NX at sub-ms: pay latency when money-critical paths need fencing-friendly ownership.
Failure timeline: lease expiry without fencing
T0 Client A acquires lock (token=33), reads stock=10 T1 A pauses (GC/network) longer than TTL; lease expires T2 Client B acquires lock (token=34), sets stock=7, releases T3 A wakes, still believes it holds the lock, writes stock=7 again (or overwrites B) T4 Without fencing at storage: silent corruption. With fencing: Write(token=33) REJECTED
Counterexample to "TTL makes the lock safe": expiry recovers liveness; fencing recovers safety.
How to choose a locking approach
1. Is duplicate execution dangerous? Billing crons and inventory decrements usually need mutual exclusion. Idempotent queue consumers often do not.
2. Can storage fence stale writers? If leases can expire, use a . Without it, a waking holder can overwrite newer work.
3. How strong must correctness be? Prefer , , or Consul for must-not-fail coordination. Use Redis when speed matters more and retries are safe.
Simple rule of thumb
Prefer database constraints or idempotent designs when they remove the need for a lock. When you do lock: keep critical sections short, choose lock granularity carefully, and treat lease expiry as expected, not exceptional.
In simple terms: a lock picks one writer; fencing (or another authoritative check at storage) stops the wrong writer after failover. Lease alone ≠ safety.
When you need a lock
Use a distributed lock when many replicas could run the same critical step and nothing else already guarantees a single writer.
Scheduled jobs
Every pod runs the same midnight cron. You only want one pod to run billing or cleanup.
Shared resource updates
Many workers touch the same row or file. One writer at a time stops lost updates.
Leader election
Choose one primary for replication or background work while other nodes stay standby.
Rate-limited APIs
Limit how many workers call a partner API at once so you stay within quota.
Two kinds of locking
Before picking Redis or , decide whether you want to block up front or detect conflicts later.
Try the work first. If someone else changed the data, detect that at commit time and retry.
Example: User profile update: read row with version = 5, change the name locally, then UPDATE … WHERE id = 123 AND version = 5. If zero rows updated, someone else won: re-read and retry.
Take the lock first, do the work, then release. Others wait until you are done.
Example: Inventory sale: in Redis (or acquire a znode), read stock, subtract, write, then release. Worker B blocks or gets “denied” until A finishes.
Core concepts
Any distributed lock you design or buy should satisfy these four properties.
One owner at a time
For a given resource, only one client may hold the lock.
Survives failures
If the holder crashes or loses network, the lock must expire or move on so others are not stuck forever.
Time limit ()
The lock does not last forever. A dead client cannot block the resource indefinitely.
Atomic acquire/release
Getting or releasing the lock must be one indivisible step so two clients cannot both think they won.
Parts of a lock
Most lock implementations boil down to these three pieces.
Lock name
The string every worker uses to refer to the same protected resource. If two services use different names for the same order, they are not coordinating: both can run at once.
Example: lock:order:123 and lock:product:sku-9 for different resources; never lock:orders and lock:order:123 for the same order. Match your partition key (one order, one product, one shard).
Time limit
How long the lock stays valid if the holder stops renewing it. Prevents a crashed worker from blocking everyone forever.
Example: Redis SET lock:job:billing NX EX 30: the key vanishes after 30 seconds if the holder never releases it.
Keep-alive
Renewals while work is still running so a long job does not lose the lock before it finishes.
Example: etcd keepAlive every 10s on a 30s lease; Kafka consumer heartbeats every ~3s on an ~18s session timeout.
Scenario: inventory sale on two workers
A concrete race, then the same flow with a lock: the pattern you will see in interviews and production.
Two API pods share one stock row. Both read stock = 10, each sells 3 units, and both write 7. The real answer should be 4. You oversold because both used the same stale number.
A lock on product-123 can serialize the read-modify-write sequence if every writer participates and storage rejects stale owners after lease expiry. But when one database owns the row, a conditional statement that decrements stock only when at least three units remain usually expresses the invariant more directly. Check the affected-row count to distinguish a successful reservation from insufficient stock, and use an operation identifier to make retries safe.
Read the diagram definition
sequenceDiagram
autonumber
participant A as Worker A
participant B as Worker B
participant L as Lock service
participant S as Shared storage
rect rgb(254, 226, 226)
Note over A, S: Without a lock: both think they are allowed to run
par Same moment
A->>S: read inventory = 10
B->>S: read inventory = 10
end
A->>S: sell 3 → write 7
B->>S: sell 3 → write 7
Note over S: Should be 4: lost update
end
rect rgb(209, 250, 229)
Note over A, L: With a distributed lock: one owner at a time
A->>L: acquire lock on resource X
L-->>A: lease granted
B->>L: acquire lock on resource X
L-->>B: wait / denied
A->>S: read 10, write 7
A->>L: release
B->>L: acquire after release
L-->>B: granted
B->>S: read 7, write 4
endKey considerations
Locks look simple until you hit deadlocks, wrong granularity, or coordination cost at scale.
Deadlocks
Two workers each wait for the other’s lock. Avoid taking multiple locks in inconsistent order.
How coarse is the lock?
Lock the whole table, one row, or one order? Finer locks allow more parallelism; coarser locks are simpler.
Coordination style
Row locks, version checks, or queue-based systems (like ) trade ease of use for safety under contention.
Cost of locking
Every lock is a network round trip. Sometimes partitioning or idempotent retries avoid a lock entirely.
If can expire, use at storage. For must-not-fail coordination, prefer , , or Consul. Use Redis when speed matters more than perfect safety and retries are safe.
Mutual exclusion across a global fleet
At large scale you do not have one cron server. You have hundreds of workers. Locks coordinate them, but they are easy to get wrong: a long GC pause, clock drift, or a network split can mean two workers both think they own the resource. Martin Kleppmann's critique of -style designs is a reminder that storage must enforce ordering too. That is where come in.
When the lock lies: why matter
Back to the stock example: Worker A holds the lock, then freezes (GC, network, etc.) past the . The lock service fairly gives the lock to Worker B, who updates storage. Worker A wakes up still thinking it owns the lock and tries to write old data.
The lock service cannot undo that write: the is already gone. let storage say no: each grant gets a higher number, and storage rejects anything older than what it has already accepted. Walk through the diagram below, then the sequence chart.
Interactive flow
Fencing tokens: blocking stale owners
Each time the lock is granted, the lock service issues a rising number (33, then 34, …). Storage only accepts writes that carry a number at least as new as the last one it has seen.
Top lane: clients talk to the lock service for tokens only. Bottom lane: writes go to storage: the lock is not on that path.
Client A
First owner. Becomes dangerous only if its lease expires and it wakes up late.
Lock service
Hands out token 33, then 34. Bigger number = newer owner.
Storage
Accepts 34, rejects 33 because 33 is older than what it already stored.
Client B
Newer owner. Writes to storage before Client A comes back.
Step 1: Client A gets the lock
The lock service gives Client A token 33. Storage has not accepted any write yet.
All good so far
Read the diagram definition
sequenceDiagram
participant A as Client A
participant Lock as Lock service
participant B as Client B
participant Store as Storage
A->>Lock: acquire resource
Lock-->>A: token 33
Note over A: long pause, lease expires
B->>Lock: acquire resource
Lock-->>B: token 34
B->>Store: write with token 34
Store-->>B: accepted
A->>Store: stale write with token 33
Store-->>A: rejected by fencingLock implementations
Same problem, many backends, from a single database row to a of Redis nodes. Expand each option for how it works in practice.
RDBMS row locks · SELECT … FOR UPDATESQL
Lock a row (or rows) inside a transaction. Other transactions block on the same row until you commit or roll back: no separate lock service.
Best for: Apps that already use one primary Postgres or MySQL and moderate contention.
BEGIN;
SELECT * FROM locks WHERE name = 'job_lock' FOR UPDATE;
-- critical section
COMMIT;Redis simple lock · SET NX + TTLRedis
Set a key only if it does not exist, with a TTL so a dead client cannot hold forever. Put a unique token in the value so only the real owner can delete the key (Lua on release).
Best for: Fast coordination when retries are safe and strict fencing is handled elsewhere.
SET lock:resource_123 token_abc NX EX 30
# NX: only set if not exists (atomic acquire)
# EX 30: auto-expire in 30 seconds (prevent deadlock)
# token_abc: unique value for safe release
# Release (Lua for atomicity):
if redis.call("get", KEYS[1]) == ARGV[1] then
return redis.call("del", KEYS[1])
end
return 0Redlock · quorum on N Redis nodesRedis
Try to acquire on several independent Redis masters; you need a majority before the TTL runs out. Improves availability vs one node, but correctness is debated.
Best for: Higher Redis availability, not for money-critical paths without fencing at storage.
Interactive walkthrough
: acquire in motion
Step through like a short video, or press Play to auto-advance. Watch the client, five masters, timer, and validity update together.
Quorum
0/5
Need ≥ 3 successes
Validity
Granted only if quorum + elapsed < TTL
Step 1: Five independent masters
Redlock talks to separate Redis masters, not replicas of one primary.
Need 3 of 5 for a quorum
ZooKeeper · ephemeral sequential znodesZK
Consensus-backed locks with a fair queue and natural ordering via sequence numbers.
- 1Each client creates an ephemeral sequential znode; lowest sequence number wins.
- 2Losers watch the znode in front of them and wait.
- 3Znode disappears when the session dies: no stuck lock forever.
Strong ordering, fair queue, and the zxid can act as a natural fencing token.
etcd · lease + concurrency mutexK8s
Raft-backed locks with leases: the pattern behind many Kubernetes control-plane coordinators.
Best for: Clusters that already run etcd (Kubernetes, Nomad-style setups).
mutex := concurrency.NewMutex(session, "/my-lock/")
mutex.Lock(ctx)
mutex.Unlock(ctx)Consul sessionsConsul
consul lock -name="mylock" ./my_script.sh
A session ties your lock to a health check. If the agent stops heartbeating, the session ends and the lock is released automatically.
Best for: HashiCorp stacks with Consul agents on each node.
Martin Kleppmann's critiqueRead this
Why Redis-only locks (especially Redlock) are risky for correctness-critical mutual exclusion.
Clock assumptions
Redlock assumes clocks stay roughly in sync. A forward jump on one Redis node can expire a lock early: two clients may both think they won.
Long pauses
Client A holds the lock; a GC or network stall lasts longer than the TTL; Client B acquires; A wakes up and writes anyway, unless storage enforces fencing.
What to do instead
Use monotonic fencing tokens at storage, or ZooKeeper / etcd / Consul where consensus backs the lock.
Putting to work
How to wire fencing into your stack: flow, code patterns, and what both sides must guarantee.
The lock service hands out a rising number with each grant. Storage checks that number on every write and rejects anything older than what it has already accepted.
Fencing flow · stale owner vs storageWalkthrough
The lock service is not on the write path. It only hands out rising token numbers; storage decides whether each write is still valid.
- 1Client A acquires the lock and receives token 33.
- 2Client A pauses (GC or network) longer than the lease: the lock expires.
- 3Client B acquires the lock and receives token 34, then writes successfully.
- 4Client A wakes up and tries to write with stale token 33: storage rejects it because 34 was already accepted.
The lock cannot retroactively stop A’s write; fencing at storage is what blocks it.
Client A acquires lock → token 33
Client A pauses (GC)
Lock expires
Client B acquires lock → token 34
Client A resumes with token 33
Storage layer enforces fencing:
Write(data, fencing_token=33) → REJECTED (last seen token is 34)
Write(data, fencing_token=34) → ACCEPTEDImplementation · lock service + storageCode
Acquire from ZooKeeper, etcd, or a custom lock API, then pass the token on every write. Storage must compare the incoming token to the highest it has already accepted.
SQL and DynamoDB can both enforce “only if this token is newer” with a conditional write.
# Lock service (ZooKeeper, etcd, or custom)
lock = acquireLock("resource_123")
token = lock.fencing_token # monotonically increasing
# Storage accepts writes with valid token
UPDATE resources SET value = $1
WHERE id = 'resource_123'
AND fencing_token <= $2 # reject stale tokens
RETURNING *;
# Or DynamoDB:
# ConditionExpression: "fencing_token < :new_token"Requirements · what must be trueChecklist
Fencing needs two cooperating parts. Skipping either side leaves the stale-writer problem unsolved.
- Lock service issues strictly increasing tokens
- Storage rejects writes with an older token than it has already seen
- ZooKeeper: sequential zxid can act as a natural fencing token
- Redis alone does not fence writes: enforce ordering at storage
Ways to avoid a distributed lock
Locks add latency and failure modes. Often you can get mutual exclusion with version checks or database features instead.
Optimistic concurrency reads a row and its version, computes a change, then updates only when the stored version still equals the observed version. Suppose two callers both read version 7: the first writes version 8, and the second conditional update affects zero rows. The second caller must reread and recompute rather than replay an update based on stale stock. This performs well when conflicts are rare; high contention can produce repeated wasted work and require backoff or serialization.
# No lock needed! Use version checks:
1. Read: SELECT *, version FROM accounts WHERE id = 123
→ {balance: 100, version: 5}
2. Modify locally: new_balance = 100 - 30 = 70
3. Write with version check:
UPDATE accounts SET balance = 70, version = 6
WHERE id = 123 AND version = 5
4. If rows_affected = 0 → conflict! → Re-read and retry
# conditional writes:
PutItem({...}, ConditionExpression: "version = :v")
# Pros: No locks, no deadlocks, high throughput
# Cons: Retries under contention, not suitable for high-conflict resourcesPostgres advisory locks
Use the database as a lightweight coordinator: good when you already have Postgres and moderate scale.
# relational database advisory locks:
# Application-level locks using DB as coordinator
# Session-level:
SELECT pg_advisory_lock(12345);
-- do exclusive work --
SELECT pg_advisory_unlock(12345);
-- (auto-released on session disconnect)
# Transaction-level:
SELECT pg_advisory_xact_lock(12345);
-- auto-released at COMMIT/ROLLBACK
# Try (non-blocking):
SELECT pg_try_advisory_lock(12345);
-- → true if acquired, false if held
# Pros: No external system needed
# Cons: Limited to DB connection scope, DB becomes lock bottleneck at scaleDynamoDB conditional writes as locks
On AWS, a conditional PutItem can act like acquire, only succeed if the row is missing or the previous lease expired.
# conditional writes provide lock-like behavior:
# Acquire lock:
PutItem(
TableName: "locks",
Item: { resource_id: "job_123", owner: "worker_A", ttl: now+30 },
ConditionExpression: "attribute_not_exists(resource_id) OR ttl < :now"
)
# → Succeeds only if no lock exists or lock expired
# Release lock:
DeleteItem(
TableName: "locks",
Key: { resource_id: "job_123" },
ConditionExpression: "owner = :my_id"
)
# → Only owner can releaseWhy teams pick DynamoDB for locks
- Strongly consistent reads when you need them
- Durable across availability zones
- TTL can expire stale lock rows automatically
- Conditional writes are atomic per item
A managed database with conditional writes can store lock ownership and a record version, but the safe protocol must handle lease renewal, owner identity, and stale holders. Capacity and cost depend on item size, consistency, retry rate, and the chosen billing mode, so use current service pricing and measured contention rather than a universal operations-per-table claim. A hot lock key can serialize the workload even when the table has abundant aggregate capacity.
Lease-based locking
A lease grants authority for a bounded interval and may be renewed before it expires. A client pause, lost renewal response, or partition can make ownership uncertain even while the client process remains alive. Stop work when authority cannot be established and attach a fencing generation that the protected resource checks atomically with the mutation. Consumer group generations and storage versions can serve related purposes, but an ordinary log offset is not by itself a general distributed-lock fencing protocol.
# etcd lease example:
# Create 30-second lease
lease = client.grant(30) # returns lease_id
# Associate key with lease
client.put("lock/resource_123", "worker_A", lease_id)
# Keep alive (renew every 10s)
client.keepAlive(lease_id)
# On client crash: no keepAlive → lease expires → key deleted
# Other clients watching the key get notifiedZooKeeper ephemeral nodes: The znode vanishes when the session ends, like a lease without managing TTL yourself. Default session heartbeat is about every 2 seconds.
A fixed TTL expires after a chosen interval; a renewable lease extends that interval when its owner successfully renews. Both depend on a specified time authority and both allow an old holder to resume after expiration. Renewal improves liveness for long tasks but adds messages and ambiguous outcomes. Choose the duration from observed task and pause behavior, then test stale-owner rejection; a larger TTL changes recovery delay rather than proving mutual exclusion.
Lock Safety Checklist and Lease TTL Math
Safety checklist
- Unique owner id on acquire and compare-owner on release.
- Lease TTL plus renewal before expiry; no infinite locks.
- Monotonic fencing token issued on every successful acquire.
- Protected resource rejects stale fencing tokens, not only the lock service.
- Retry path is idempotent and can resume after ambiguous acquire/release responses.
Lease TTL sizing
For example, a task usually runs for two seconds and sometimes pauses for one second. A five-second renewable lease might cover the ordinary path, but an unbounded process pause can still exceed it. Increasing the lease to a minute reduces accidental expiry while delaying recovery after a real crash. The resource-side fence is what rejects a resumed old generation; timing estimates only choose an operational trade-off.
lease_ttl >=
p99 work time
+ p99 process pause
+ max network delay
+ renewal interval
+ safety margin
Example:
p99 work 8s + pause 2s + network 500ms
+ renewal every 3s + 2s margin
=> choose at least 15.5s, round to 20sFailure drill
Pause the lock holder longer than the TTL, let another worker acquire the lock, then resume the first worker. The test passes only if the protected store rejects the stale holder by fencing token.
How wide should the lock be?
Lock the whole system, one shard, or one order? Wider locks are simpler; narrower locks let more work run in parallel.
Granularity is how much you put behind one lock name. Too wide and you serialize everything; too narrow and you pay coordination cost on every entity.
Coarse · one lock for everythingWidest
A single lock name guards an entire job type or pipeline. Only one worker runs that work anywhere in the fleet.
Best for: Midnight cron jobs, one-off migrations, or “only one billing run” rules with low throughput needs.
Trade-off: Zero parallelism on that work: throughput is capped by one worker.
lock("process_payments")
# Only 1 worker processes ALL payments
# Simple but zero parallelism
# Throughput: limited by single worker speedMedium · lock per shard or regionBalanced
Split the lock key by a natural boundary: region, tenant tier, or category. So independent shards can run in parallel.
Best for: Regional payment processors, per-tenant batch jobs, or sharded cron where each shard has its own queue.
More throughput than coarse, with still-manageable lock key count and ops overhead.
lock("process_payments:US")
lock("process_payments:EU")
# 1 worker per region
# Some parallelismFine · lock per entityNarrowest
One lock per order, user, or resource id. Many workers can each own a different entity at the same time.
Best for: High-throughput order processing, per-user workflows, or any hot path keyed by a stable id.
Trade-off: More lock keys and Redis/etcd traffic; watch for hotspots on celebrity keys.
lock("payment:order_12345")
# 1 worker per order, thousands in parallel
# Maximum throughput
# More lock overhead, potential hotspotsDesign guidelines · rules of thumbChecklist
Start simple, measure contention, then narrow the lock only when you have evidence, not upfront.
- 1Start coarse; narrow the lock only when you measure contention.
- 2Name locks after your partition key (order_id, user_id): same idea as lock name.
- 3Never take lock A then B in one service and B then A in another: classic deadlock.
- 4Set lock TTL shorter than your worst-case job timeout so crashes recover quickly.
Choosing an implementation
Trade speed against how strongly you need mutual exclusion. Numbers are rough guides, not guarantees.
| Approach | Latency | Correctness | Throughput | Best for |
|---|---|---|---|---|
| Redis ( + ) | <1 ms | Weak without | 100K+ ops/s | Fast paths where retries are safe |
| <5 ms | Debated | 50K+ ops/s | Higher Redis availability, not money-critical | |
| ~10 ms | Strong | ~10K ops/s | Correctness-critical coordination | |
| ~10 ms | Strong | ~10K ops/s | Kubernetes-style clusters | |
| Postgres advisory | ~1 ms | Strong on one primary DB | 50K+ ops/s | You already run Postgres |
| DynamoDB conditional write | ~5 ms | Strong | 100K+ ops/s | AWS-native, serverless fleets |
| (version check) | <1 ms | Strong when conflict is rare | Very high | Low-contention row updates |
Choosing and explaining distributed locking
A complete explanation connects the mechanism to a requirement. Describe what improves when you introduce it, which new failure cases become possible, and how you would know that the trade-off is worthwhile.
Explain the core idea
Ask first: do you need a lock at all? Partitioning or version checks may be enough.
A solid lock has a TTL, an owner token, safe release, and fencing enforced by storage.
Redis is fine for best-effort coordination; use ZooKeeper or etcd when correctness is non-negotiable.
When it belongs
Only one worker may touch a resource at a time.
Cron jobs, migrations, or payment flows could run twice without coordination.
Optimistic retries are burning CPU under heavy contention.
How it fails
A long pause lets a stale holder write after the lease moved on.
Release deletes another worker’s lock because the owner token was not checked.
Clock skew makes TTL-based locks expire too early or too late.
Trade-offs to weigh
Fast locks are often weaker; strict locks pay consensus latency.
Finer locks mean more parallelism but more lock traffic.
Longer leases mean fewer renewals but slower recovery after a crash.
Common misunderstandings
Claiming SET NX EX alone is safe for money-critical mutual exclusion.
Enforcing the lock only at the lock service, not at storage (no fencing).
Taking locks in different orders across services: classic deadlock.
Try it: fencing tokens after lock handoff
Simulate lock acquisition with monotonic tokens and a storage layer that rejects writes whose token is not strictly greater than the highest token already applied.
Lock service
Holder: none
Latest issued token: 0
Storage (fenced)
Max token applied: 0
Value: initial
- Lock holder
- —
- Fencing token
- —
- Storage max token
- 0
Acquire the lock as Client A, then as Client B. Writes include a fencing token; stale holders cannot overwrite newer work.
Going deeper
Distributed locking
The mechanism and its boundary
A distributed lock is a lease on exclusive work. It is safer to think in leases than forever-locks: the holder must renew, the protected resource must reject stale owners, and the system must define what happens after pauses.
Lease plus fencing token
Acquire lease
Write token
Renew
Reject stale token
Work through the sizing assumptions
When it belongs
Only one worker may run a critical section at a time. Duplicate work is expensive or unsafe. There is a shared resource that can enforce ownership tokens.
Assumptions that matter
Critical sections are short and bounded. The protected resource can compare monotonically increasing tokens. Retries are idempotent because lock outcomes can be ambiguous.
Easy mistakes
Acquiring a lock does not stop a paused process from later sending a database write or file update after its ownership has expired. The protected resource must compare fencing tokens and reject older ownership epochs; checking ownership only in the worker leaves a pause between that check and the write. A lease shorter than ordinary pauses or renewal delays causes unnecessary takeovers, but choosing a longer TTL still cannot make an arbitrarily paused old holder safe.
How it fails
Holder pauses past TTL and continues writing after another holder starts. Lock service partition grants inconsistent ownership. Renewal succeeds locally but protected resource never observes the new token.
Trade-offs to name
Exclusive ownership serializes conflicting work and can simplify correctness, but limits throughput for that resource, so use the smallest critical section that preserves the invariant. A short lease lets another worker take over sooner after a crash, while increasing the chance a slow but live holder expires; fencing at the protected resource is what rejects that stale holder’s later writes. A shared lock service centralizes ownership decisions but becomes a required coordination dependency, so its outage behavior and the resource’s token checks belong in the design together.
Wrap-up
You have seen fencing, leases, and when not to take a lock. Use these exercises before reading the classic correctness debate.
Exercises
- Fence the write path. Sketch the SQL/conditional write that rejects fencing_token ≤ last_seen. Where does the token come from on acquire, and what happens if the lock service and storage disagree?
- Do you need a lock? A daily billing cron must not run twice. Compare distributed lock vs lease on a "run_id" row vs idempotent invoice keys. Pick one and name the failure mode you still monitor.
- Redlock review. A PR uses Redlock for payment settlement. Cite Kleppmann's fencing argument and state the minimum change you require before merge (or the alternative coordination store).
Topic-specific references
- Kleppmann, "How to do distributed locking" : why lease-based Redis locks need fencing at the resource.
- Redis distributed locks / Redlock notes : algorithm shape and the correctness debate.
- etcd: why consensus-backed coordination : when you need stronger lock semantics than a cache TTL.
Key takeaway
Distributed locks look easy and bite hard. Redis-style locks are fast but unsafe for critical paths unless storage enforces . , , and Consul trade latency for stronger guarantees. Before adding a lock, try , conditional writes, or partitioning so only one worker owns each shard. When you do lock, keep keys fine-grained and pair with at the resource that actually stores data: the lock service is not the last line of defense.
Practice and further reading
Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.
Focus for this lesson: Use distributed locks only with leases, unique ownership, fencing-token enforcement, idempotency, and tests for pause, partition, expiry, and drift.
Explain the decision
Why is SET NX with expiry insufficient for protecting storage?
Read the reasoning
A holder can pause past expiry and resume; storage must compare a monotonic fencing token, not trust the process’s lock belief.
Apply it to a production scenario
Worker A pauses, lease expires, B writes, then A resumes. What happens?
Read one approach
A carries an older token and storage rejects it; A stops and reloads rather than overwriting B.
Work through the numbers
Lease TTL is 10 s, renewal every 3 s, p99 pause 4 s, network 1 s. What margin remains?
Read the worked calculation
10 − 3 − 4 − 1 = 2 s, likely too small after jitter/clock budget; increase TTL or reduce pauses while preserving recovery goals.
Reason through a failure
What must the protected resource enforce?
Read the recovery reasoning
Accepted fencing epochs are monotonically non-decreasing; any operation with a lower epoch is rejected regardless of lock-service state.
Sources and further study
Continue with Idempotency.