System Design Patterns

Leader election

You run the same service on so one failure does not take you down. But some tasks must happen only once: send one nightly report, update one shared config row, claim one “I own this job” lock.

is how the group picks the machine that is allowed to do that work. When the leader goes quiet, another takes over. Without , a frozen old leader can wake up and write again. That mess is .

What happens when the leader dies

Standbys notice the leader is gone.

A majority votes them in.

Authority expires unless renewed.

Old leaders are rejected.

What is a leader, and why have one?

A is simply the one machine in a group that is currently allowed to do a special job. The other machines (followers) keep a copy of the service running so they can take over, but they do not perform that special job while someone else is leader.

Distributed systems add machines for reliability and scale. That creates a new problem: many machines may try the same action at once. A leader gives the group a single decision-maker for work that must stay consistent.

Hot standby mental model

Leader does the special work; followers stay silent on that work until the leader is gone

While the leader is healthy

The runs the once-only / coordination job.
do not do that same job. For that role they stay quiet.
Followers are still “awake”: they listen to , stay caught up, and are ready to vote.

When the leader is down (or frozen)

Heartbeats stop. A follower campaigns and may become the new leader.
Only then does that machine start the special work.
That handoff is . Until then, followers are hot standbys for that role, not co-workers on it.

Do not confuse this with database read replicas. In many databases, replicas actively serve reads while the primary handles writes. In leader election for singleton / coordination work, followers usually do not share that special job; they wait to take it over.

Leader doing special work while two followers stay on standby, then one follower takes over after leader failure — Hot standby: followers are silent on the special role until failover, not co-workers on it.

What kinds of systems need this? Systems where several copies of a service are running for reliability, but one shared role must stay unique. Think control planes, once-a-day jobs, and “primary writer” groups. Followers look idle for that role on purpose; that standby capacity is the point.

Systems that need a leader (hot standby)

Kubernetes control plane: several kube-scheduler or controller replicas may run; only the elected leader schedules pods. Others stay idle on that duty until failover.
Kafka / Pulsar controller (or old ZooKeeper epoch): one controller assigns partitions and watches membership; standbys wait.
MongoDB replica set / MySQL Group Replication primary: one primary accepts writes for the set; secondaries apply the log and take over if the primary dies (secondaries may also serve reads; that is optional and separate from “who is primary”).
Payments / fintech nightly settlement: three app pods are up; only the leader should close the day’s books or fire one reconciliation email.
etcd / ZooKeeper themselves: the store elects a Raft/Zab leader so metadata writes have one sequencer.

Systems that usually stay leader-free (everyone works)

Stateless API pods behind a load balancer: every instance handles HTTP; no single “boss” for each request.
CDN / edge caches: many nodes serve the same static or cached objects in parallel.
Queue consumers (order emails, image resize): many workers pull jobs; work is or claimed per message, so idle hot-standby election is overkill.
Map-reduce / Spark executors: workers process different slices of data; coordination is elsewhere, not “only one machine may work.”

What a leader does

Owns work (one report, one schema change).
Writes shared (“who owns this job”).
Often acts as the primary writer for a data range in replicated stores.

Why we need one

Avoid two machines deciding different “truths.”
Avoid duplicate side effects (two emails, two payments).
Give clients one place to send writes that must stay ordered.

What a leader is not

Not “the most powerful server forever.”
Not required for every request (reads and many workers can stay parallel).
Not safe by title alone: the database must still reject old leaders.

Quick checklist: elect a leader?

Doing the action twice would be wrong or expensive.
Someone must serialize updates to shared control data.
A replica group needs one primary so writes stay ordered.
You want automatic for that unique role.

Quick checklist: skip election?

Work is and many workers can process the same queue safely.
Each request is independent (stateless APIs, CDN).
You can partition work so each machine owns a different piece.
A short “claim this job row” lock is enough.

Bottom line: use a leader when the system needs one temporary boss for consistency or once-only side effects, and you are okay with standbys staying idle on that role until . Use so the group can choose that boss, and replace it safely when it fails.

The problem in one story

Imagine three cashiers and one shared cash drawer. Any cashier can ring customers up. Only one should update the drawer totals at closing, or the books will disagree.

In software it is the same idea. Three healthy run your service. Some work is : only one machine should send tonight’s report or update the shared that says “who is in charge.” If two do it, you get duplicate emails or corrupted control data.

Naive idea: “lowest hostname wins,” or “whoever grabs a Redis key first is leader.” That fails when the leader freezes past key expiry, a new leader is elected, then the old one wakes and writes again.

That double-writer failure is . Election without does not stop it, because the shared database still accepts the stale writer unless it checks an .

What must stay true

At most one approved writer for a protected piece of data

Leadership is not “who feels special.” It is a rising / plus a that the database checks on every important write. A helps you notice death quickly. Lease expiry alone is not enough if a process can pause and wake up late.

What actually happens

Picture three replicas. One is the . The other two are . The leader keeps saying “I’m alive” with .

When those heartbeats stop, a follower waits a little longer (the election timeout), then becomes a . It bumps a round number called a , votes for itself, and asks the others for votes. If a (a majority) says yes, it is the new leader for that term. One-line rule: at most one leader per term.

What a term looks like

A term is an integer label for an election round, not a length of time

Think of it as a version stamp stored on every node: term = 41, then later term = 42. It can be one digit early on (3) or a large number after many elections (1087). Same idea either way: higher wins.

Not a duration. Election timeout (e.g. 5 seconds) is how long you wait. The term is which round you are in.
Bumps on campaign. When a follower becomes a candidate, it does roughly term = term + 1.
One leader per number. Term 42 may have at most one leader. If you later see term 43, that is a newer round.
Often pairs with fencing. The same rising idea shows up as an / when writing to shared storage.

term 41 → Leader Athenterm 42 → Leader B(A must step down)

Can many nodes become candidates at once?

Yes. They can all bump the term and ask for votes.

Raft does not magically stop two followers from timing out together. Example with 3 nodes: A and B both miss heartbeats, both set term = 42, both vote for themselves, both ask C for a vote. That is allowed. What Raft prevents is two winners for term 42.

One vote per term. Each node grants its vote to at most one candidate for a given term number. C can help only one of A or B.
Majority required. You need a (2 of 3, or 3 of 5). Two candidates splitting votes often means : nobody becomes leader for that term.
Randomized election timeouts. Followers wait a random window (e.g. 150–300 ms) before campaigning, so usually one starts first, wins, and sends heartbeats before others time out.
Retry on a higher term. If term 42 elects nobody, someone later bumps to 43 and tries again. Safety stays: still at most one leader per term.

So: multiple candidates asking for votes can happen. Multiple elected leaders for the same term cannot, if majority voting is enforced.

How terms are shared

If A and B both start from term 31, both become 32 — not 32 and 33

Each node only bumps its own local counter. There is no shared “next term” ticket machine. So both can campaign for the same round: term 32. That is the case. Only later, if 32 elects nobody, someone bumps again (32 → 33) and retries.

How the term is shared: every important Raft message carries the sender’s term (vote requests, heartbeats / AppendEntries). When a node hears a higher term, it updates its local currentTerm. A leader that sees a higher term steps down. Stale lower-term messages lose.

Tiny story

Both at 31 → both try 32 → split vote → A later tries 33 and wins → B hears RequestVote(term=33) and adopts 33.

Before it changes shared control data (“who owns this job,” “what is the current config”), the new leader also gets a and a from the store that accepts writes. If the old leader wakes later with an older token, the store rejects the write. Election without that check is not enough.

In production, prefer a mature coordination service (etcd, ZooKeeper, a managed consensus API). Hand-rolling is a long project unless you are specifically studying the internals.

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

sequenceDiagram
    participant F1 as Follower A
    participant C as Candidate B
    participant F2 as Follower C
    participant S as Shared store

    Note over F1,F2: Leader heartbeat expires
    C->>C: increment term and vote for self
    C->>F1: RequestVote(term)
    C->>F2: RequestVote(term)
    F1-->>C: vote granted if log is current
    F2-->>C: vote granted
    C->>S: acquire lease with fencing token
    S-->>C: token 42
    C->>F1: heartbeats(term, token 42)
    C->>F2: heartbeats(term, token 42)

Heartbeats stop → campaign → majority votes → lease + fencing token → heartbeats resume. The database is part of the story, not an afterthought.

Try it: run a leader ballot

Three nodes, majority quorum. Campaigning bumps the term, nodes grant at most one vote per term, and two affirmative ballots elect a leader. This model omits log replication and fencing tokens, but the ballot rules match the safety story on the page.

Current term: 41
Leader: A
Quorum needed: 2 of 3
Ballot events: 0

Node A

leader

Term 41 · role leader · voted for A

Node B

follower

Term 41 · role follower · voted for A

Node C

follower

Term 41 · role follower · voted for A

Ballot log

recent

No events yet.

Node A is leader for term 41. Stop heartbeats, then let a follower campaign and collect ballots. Majority wins; split votes leave no leader for that term.

Split votes. Use the split-vote demo, then grant C’s ballot to one candidate to see how majority breaks the tie. Randomized election timeouts reduce simultaneous campaigns in production.

Not enough alone. A new leader still needs a lease and fencing token before mutating shared storage, or a stale leader can write after waking from a long pause.

How to choose whether to elect

1. Is doing the work twice dangerous? Sending one report, updating one shared lock row, or changing the database schema once: yes. Processing queue jobs: often no.

2. Can the database reject an old ticket? If storage cannot check a , election will not protect you when an old leader wakes up.

3. How fast must be? Short fail over quicker but flap more; long leases leave a longer stale window.

Simple rule of thumb

Elect a leader when one authority must serialize protected work. A strict majority provides intersecting votes; an odd-sized group often uses voters efficiently. Size leases from measured pauses and recovery goals, jitter renewals, and enforce fencing at the protected write boundary. Practice failover during deploys: leader bugs show up in freezes and network cuts, not calm demos.

Three ideas you must keep separate

Most confusion comes from mixing these. They solve different problems.

1. Terms / epochs

A rising integer: 41, 42, … not “how many seconds.” Candidates bump the term; you vote at most once per term number; a higher term forces an old leader to step down. This stops two leaders in the same round.

2. Leases

A short timer the live leader must renew. When renewals stop, others can campaign sooner. A lease answers “do we think the leader is dead?” It does not by itself stop a paused ex-leader’s late write.

3. Fencing

A number that only goes up. The database (or lock table) checks it on every write and rejects stale numbers. Without this, election is advice; the store can still accept a zombie writer.

Safety (must never break)

For a protected resource R, at most one fenced authority may commit a write at a time.

Liveness (best effort)

If a majority of replicas is up and can talk, some leader should eventually be elected so work can continue.

The pause story (why fencing exists)

Leader A holds a lease, then freezes in a long .
The lease expires. Leader B is elected and gets fencing token 43.
A wakes up and still believes it is leader. It tries a write with token 42.
The store runs something like: UPDATE … SET … WHERE accepted_epoch <= :epoch. Zero rows updated → reject. A is fenced out.

Lease alone detected death. Fence at storage blocked the late write.

Intent and when it helps

Goal: exactly one live machine does the special work; the others stay ready to take over (hot standby: idle on that role until the leader fails).

When it helps

Kubernetes schedulers, Kafka controllers, DB primary election, fintech settlement jobs, etcd/ZooKeeper metadata leaders: anywhere duplicate writers would send duplicates or corrupt .

When it is overkill

Stateless APIs, CDN edges, and queue consumers where every instance should work in parallel.

Raft in three beats

Elections are labeled by a rising number (41, 42, …), not by a stopwatch. At most one leader per term. Randomized timeouts reduce everyone campaigning at once; if a still happens, the next term retries. A log entry is durable only after a majority has it on disk.

Election: follower misses heartbeats → becomes candidate, bumps term, requests votes.
Leadership: majority grants votes → leader appends entries and sends heartbeats / replication RPCs.
Safety: a peer reports a higher term → old leader steps down immediately. No two leaders in the same term.

Follower→Candidate→Leader

Sizing the clocks (timeout math)

You are balancing two bad outcomes: failing over too slowly (users wait) versus electing too eagerly (healthy leaders look dead under lag). Put numbers on the pieces, then check the inequalities in plain English.

Heartbeat interval

The heartbeat interval determines how often a leader communicates with followers or renews its coordination record. Choose it relative to measured network latency, process pauses, and the desired detection time; one second is an example, not a universal default. Sending more heartbeats increases overhead, while an election timeout too close to the interval can mistake ordinary scheduling delays for leader failure.

Election timeout

How long a follower waits before campaigning. It must clear several missed heartbeats and normal network/pause jitter, or healthy clusters thrash.

Lease TTL

How long leadership stays valid without a renew. Must cover renew interval plus pause, network tail latency, and clock-skew budget.

Rules in words

Election timeout should be larger than (missed heartbeats × heartbeat interval), and also larger than network p99 + pause budget + scheduling jitter.
Lease TTL should be larger than renew interval + network p99 + pause budget + clock-skew budget.
Failover time is roughly detection + campaign + warmup(catch-up, cache fill, connection reopen).

Worked example

Heartbeat every 1 s. Declare the leader missing after 5 missed beats → about 5 s to detect. Campaign takes 300 ms. Warmup takes 2 s.

Failover ≈ 5 s + 0.3 s + 2 s = 7.3 s before the new leader is useful again.

If your product cannot tolerate that gap, shorten detection carefully (and watch false elections), or warm standbys ahead of time.

Diagram text: the exact Mermaid definition below preserves the nodes, labels, and connections. Enable JavaScript to view the rendered diagram.

Read the diagram definition

sequenceDiagram
    participant Old as Old leader
    participant F as Followers
    participant New as New leader
    participant Store as Fenced store

    Old--xF: heartbeat lost
    Note over F: wait election timeout
    F->>New: vote majority
    New->>Store: acquire lease and fencing token 43
    Old->>Store: stale write with token 42
    Store-->>Old: reject stale token
    New->>F: heartbeats resume
    Note over New,F: steady state

Timeline: heartbeats stop → wait → vote → new lease/token → old write with stale token rejected → steady state.

When not to elect a leader

Simpler alternatives

Use idempotent jobs, competing queue consumers, or partitioned work with natural ownership when duplicate execution is harmless or easy to dedupe.

Added failure modes

Election adds stale leaders, duplicate “once-only” jobs, majority loss, lease renewal stalls, and storage bugs if fencing tokens are not enforced on every write.

Log replication

Write path

Client commandLeader log appendReplicate to followers

An entry is committed once a quorum of followers has made it durable on disk (majority rule). Only then may leaders tell state machines to apply it. This ordering keeps replicas convergent.

Bully algorithm (classroom only)

A simple teaching algorithm: the live node with the highest ID is the coordinator. It assumes a mostly reliable LAN. It is not a WAN production recipe by itself.

Suspect the coordinator died → ask every peer with a higher ID “are you up?”
No higher-ID reply → declare yourself coordinator and tell everyone.
A higher-ID peer answers → back off; that peer drives the next round.

Under partitions, two sides can each see a different “highest live ID.” Real systems add quorums, leases, and fencing; they do not stop at highest-ID wins.

Coordination recipe (ZooKeeper-style)

Many lock services use the same idea: create a numbered child under an election path; the smallest number wins; each waiter watches only the node just ahead of it so one failure wakes one standby, not a stampede.

How it looks

Create an ephemeral sequential child under /election/, e.g. n_0000000123.
If your sequence is the lowest, you are the leader.
Otherwise watch only your immediate predecessor. When it disappears, re-check whether you are now lowest.

Still pair this with fencing on the store that accepts writes. Winning the znode is not the same as being safe to mutate shared state.

Leases in practice

Tie leadership to a short . The live leader must renew before the TTL ends. When renewals stop, the key frees and a standby can campaign. Prefer store-side expiry over trusting the leader’s wall clock.

Typical knobs

TTL ≈ 30s (example)
Renew every ≈ 10s, with jitter so all nodes do not renew at once
Campaign only while holding a valid lease id

Remember

A lease grants authority for a bounded period unless it is renewed under the coordination system’s rules. A paused client may fail to notice that its lease expired, so its own belief is not sufficient evidence of authority. Before accepting a protected write, the resource must enforce the chosen authority rule, such as checking a current epoch or rejecting stale fencing tokens. Lease expiry helps recovery; storage enforcement supplies the protection.

Stopping split brain

Two writers that both believe they are leader will corrupt shared state. You usually need more than one of these layers.

Majority quorums

A strict majority is more than half of the configured voters. Any two majorities intersect, whether membership is odd or even, which prevents two candidates from winning the same term when each voter grants at most one vote. Odd-sized groups are common because adding a fourth voter to three does not increase the number of failures tolerated. Membership changes also need a safe protocol so old and new groups cannot form disjoint decision-making sets.

Fencing tokens

A fencing token is a monotonically increasing authority number attached to protected operations. Once the store has accepted token 42, it rejects a write carrying 41, even if that request was sent earlier and arrived late. The check and mutation must be atomic. A token only orders authority: the store may also need operation IDs or sequence checks to prevent duplicate or reordered writes within the same leadership epoch.

Time-bounded leases

Leadership expires unless renewed. That speeds detection; fencing still handles late writes after a pause.

Storms, recovery, and what to watch

Avoid election storms

Randomize timeouts, watch only the predecessor in sequential recipes, add campaign backoff, and avoid waking the whole cluster when one candidate disappears.

Recover from split brain

Stop stale actors, confirm fenced writes were rejected, reconcile any duplicated side effects, and add a drill that pauses a leader mid-renewal.

Observability checklist

Monitor both the election mechanism and the work it protects. Leader identity and term changes show who currently holds authority; renewal latency and lease age show whether that authority is becoming fragile. Fenced-write rejections reveal stale actors that were safely blocked. Also track useful work completion and failover duration: a cluster can report a stable leader while its job processing is stuck.

Graceful transfer

For a planned upgrade, select a healthy follower whose replicated state is sufficiently current, stop or drain new leader-only work, and initiate the coordination system’s supported transfer. The replacement must obtain authority before taking over protected writes. In-flight requests from the old leader still need fencing or deduplication. A graceful transfer reduces the outage caused by waiting for failure detection; it does not remove the need for failure-safe handoff.

Where it appears

Kubernetes scheduler / controllers

Several replicas run; only the elected leader places pods or reconciles a controller. Standbys stay idle on that duty.

Kafka controller / cluster metadata

One controller owns partition assignment and membership changes; others wait to take over.

MongoDB / MySQL primary

One primary accepts writes for the replica set; others apply the log and promote on failure.

etcd, ZooKeeper, settlement jobs

Coordination stores elect a log leader; apps use leases so only one pod runs a nightly close or schema migrate.

How to choose and explain

When to elect

Use a leader when two writers would hurt (duplicate report, corrupt shared lock row). Prefer a mature coordination store; explain terms, majorities, leases, and fencing.

Trade-offs

Short leases fail over faster but flap more. Long leases reduce flapping but leave a longer stale-leader window.

What still breaks

, , clock assumptions, majority loss, stores that accept stale tokens, and election storms if timeouts are too tight.

What to watch in production

Current leader, term, lease renew latency, step-down count, election rate, and fenced-write rejections. Drill during deploys, not only in steady state.

Deeper: Raft safety and fencing at the store

Why a new leader is “caught up”

A node votes at most once per term, and only for a candidate whose log is at least as up to date. So a winner already holds every committed entry. If it later sees a higher term, it steps down.

Changing membership safely

Jumping straight from old members to new ones can create two disjoint majorities. Raft’s joint consensus briefly requires agreement from both sets, then commits the final membership.

Partition story (term 41 → 42)

Leader A (term 41) loses the majority and cannot commit new entries.
Followers time out; B starts term 42 and wins after a majority checks its log.
Before singleton work, B gets fencing epoch 42 from the protected store.
A wakes and writes with epoch 41; the store rejects it even if A’s local code still feels like leader.
A sees the higher term, steps down, catches up, and only then may campaign again.

The consensus service elects; the data store must still enforce the epoch on every protected write:

sql

UPDATE shard_metadata
SET owner = :candidate,
    value = :new_value,
    accepted_epoch = :epoch
WHERE shard_id = :shard
  AND accepted_epoch <= :epoch;

-- 0 rows updated → reject as stale leader
-- Never reset accepted_epoch on process restart

Clocks and leases help you detect death quickly. They do not replace quorum evidence or store-side fencing when the write hits shared state.

Where the design still breaks

Quorums, leases, and fencing only help if every layer actually enforces them.

Stale leader resumes

GC-paused A wakes after B won. If the store does not check the token, A’s late write still corrupts the singleton.

Clock skew

Client wall clocks drift under NTP jumps. Prefer store-side expiry and monotonic tokens for safety decisions.

Majority loss

If a consensus group loses a majority, it normally stops committing new decisions because no remaining minority can safely distinguish failure from a network partition. Existing followers may still answer explicitly permitted stale reads, but that is a separate contract. Restore quorum using the system’s recovery procedure; casually shrinking membership to whichever machines are reachable can create conflicting authorities and discard committed history.

Election thrash

Many candidates at once split the vote; terms bump; no stable leader. Randomized timeouts and odd cluster sizes help.

GC pause timeline

T+0Leader A (term 41) freezes in GC. Locally it still thinks the lease is fine.
T+8sMajority elects B (term 42). B fences the store at epoch 42.
T+14sA wakes and tries a write with epoch 41.
NextStore rejects (0 rows) if fencing is wired; otherwise it accepts and you have split brain.

Safety holds only when the store checks the token on every protected write.

Exercises

Heartbeat 1s, election timeout 5s, campaign 300ms, warmup 2s. Estimate failover time. Is an 8s lease TTL safe if GC pauses reach 6s?
Old leader A (epoch 41) is paused; B wins term 42 and fences at 42. A wakes and writes. Where must the reject happen: election service or data store? Why?
You need one nightly report vs many workers safely processing the same queue jobs. When is election overkill?

Primary references

Topic sources for consensus, leases, and fencing, not generic system-design books.

Raft paper (Ongaro & Ousterhout)etcd: Lease API Martin Kleppmann: Fencing tokens

Takeaway

Election picks who is in charge; terms bound who may lead; leases help notice death; fencing stops late writes at the database. Prefer odd majorities, jittered renewals, and failover drills: leader bugs show up in freezes and network cuts, not calm demos.

Practice and further reading

Use these exercises to connect the lesson to a design decision. State your assumptions, trace what happens to one request, and explain the rule that must remain true even during a failure. That rule is the design's correctness invariant.

Focus for this lesson: Use terms, majority voting, membership rules, leases, and storage-enforced fencing so only a current leader can perform singleton writes.

Explain the decision

Why does a lease alone not make an old leader harmless?

Read the reasoning

A paused process can resume after expiry without knowing it is stale; every protected write needs a monotonically increasing fencing token enforced downstream.

Apply it to a production scenario

A five-node group changes membership from five to three while a partition occurs. What is dangerous?

Read one approach

Independent old and new majorities may elect leaders; use joint consensus so decisions overlap both configurations until the transition commits.

Work through the numbers

Heartbeat is 500 ms, five missed heartbeats trigger election, campaign takes 400 ms, warm-up 1.2 s. Estimate failover.

Read the worked calculation

About 2.5 + 0.4 + 1.2 = 4.1 seconds, assuming no split vote or additional pause.

Reason through a failure

What is the leader-safety invariant?

Read the recovery reasoning

At most one leader may commit for a term/configuration, and protected storage rejects any operation carrying an epoch older than the greatest accepted epoch.

Sources and further study

Continue with Strangler Fig Pattern.