Leader Election and Lease Safety
Consensus election invariants, lease timing risks, stale leaders, and fencing tokens for side-effect safety.
Leader election chooses one coordinator. It does not automatically make every side effect safe. The hard case is a stale leader that still believes it owns authority after the rest of the system moved on.
Election safety
Raft divides time into monotonically increasing terms. At most one leader can be elected in a term. A candidate wins by receiving votes from a majority, and a server grants at most one vote per term.
Majorities intersect. For a cluster of $N$ nodes, quorum size is:
\[q = \left\lfloor \frac{N}{2} \right\rfloor + 1\]Any two majorities share at least one voter. That intersection is the core reason election safety works under crash faults.
Timing and availability
Raft safety does not depend on timing. Timing affects availability. The paper states the practical timing requirement as:
\[broadcastTime \ll electionTimeout \ll MTBF\]If election timeouts are too low relative to broadcast time and storage latency, healthy leaders can be replaced unnecessarily. If they are too high, failover is slow.
Leases are different
A lease grants authority for a time interval. Lease safety depends on clock assumptions. If a process pauses, a VM stalls, or a clock jumps, a leader can act after its lease should have expired.
A lease is safe only if every participant agrees on bounds for clock drift, message delay, and pause behavior, or if external side effects are protected by fencing.
Fencing tokens
A fencing token is a monotonically increasing number attached to leader operations. Storage or downstream services reject stale tokens:
\[token_{new} > token_{last\_accepted}\]This protects the resource even if an old leader wakes up and sends delayed writes. The lock service or consensus group issues authority. The resource enforces it.
Failure modes
- A leader writes to an external database that does not check fencing tokens.
- A process pause exceeds the lease but the process continues after waking.
- Split brain is hidden by logs because only one side accepts client traffic.
- Read-only operations bypass the log and observe stale state.
- Election parameters pass unit tests but fail under real disk stalls or packet loss.
Practical checks
- Write down whether safety depends on consensus or on timing.
- Use quorum-backed terms or epochs for leadership decisions.
- Attach fencing tokens to external side effects.
- Test process pauses longer than the lease interval.
- Test packet loss, delayed messages, and disk fsync latency.
- Reject reads from stale leaders unless the protocol proves lease validity.
Design guidance
Use consensus for authority and fencing for side effects. Do not rely on a local boolean named is_leader to protect external resources. If a stale leader can still mutate state, the election protocol is not the full safety story.