Distributed Locks: Redlock, Fencing Tokens & Advisory

Q: What is a distributed lock and how is it different from a normal lock?

A distributed lock coordinates mutual exclusion between processes on different machines — only one worker runs the cron job, only one service instance processes a given order — where an in-process mutex cannot reach. The difference that changes everything is failure handling. A local mutex holder that dies takes the process down with it, releasing the lock; a distributed lock holder that dies silently would deadlock the system forever, so every practical distributed lock has an expiry (TTL or lease). That expiry is the poisoned gift: it converts the lock into a lease that can legitimately be granted to a second holder while the first still believes it owns it. The overlap scenario is mundane, not exotic — a stop-the-world garbage-collection pause of a few seconds, a network stall, a container scheduled out under CPU pressure, a frozen serverless function — any of these between acquiring the lock and writing the result means the TTL expires mid-section, a second worker acquires legitimately, and the first wakes up and keeps writing with stale confidence. No lock service, however well engineered, can prevent this from the server side, because the failure lives in the client's own execution timing. That is why the practical taxonomy splits locks by consequence: efficiency locks (overlap wastes some compute — acceptable) versus correctness locks (overlap corrupts data or moves money — needs fencing tokens at the resource, not just a lock).

Q: Is Redlock safe to use?

Safe for what it actually provides: a distributed lock that survives the failure of individual Redis nodes, suitable for efficiency purposes — deduplicating work, avoiding redundant computation, reducing thundering herds. The famous 2016 exchange between Martin Kleppmann and Redis's creator settled into a practical consensus that both sides essentially state: Redlock acquires the lock on a majority of five independent Redis nodes, which genuinely fixes the single-node failover hole (where a replica promoted before replicating the lock key would grant the lock twice). What no quorum can fix is the paused-client problem: a worker that acquires the lock, stalls past the TTL (GC, network, scheduling), and resumes stale now overlaps with the legitimate new holder — and neither of them did anything wrong. Kleppmann's analysis also flagged Redlock's sensitivity to clock jumps on the Redis nodes themselves. The operational conclusion is not "Redlock is broken" but "Redlock is an efficiency lock with better availability": if a rare overlap merely wastes a computation, single-node SET NX PX is usually already enough and Redlock is a hardening option; if overlap corrupts state or double-moves money, no lock-only design of any brand suffices — you need the protected resource to reject stale writers via fencing tokens, which Redis-based designs do not provide natively and consensus systems like ZooKeeper and etcd do.

Q: What is a fencing token and how does it prevent double-processing?

A fencing token is a number that the lock service increments on every acquisition and that every write to the protected resource must carry — with the resource itself rejecting any write bearing a token older than the newest it has accepted. It closes the gap that TTLs open. Walk the failure: Worker 1 acquires the lock with token 33, stalls past its TTL; Worker 2 acquires legitimately with token 34 and writes; Worker 1 wakes and attempts its stale write with token 33 — the storage compares 33 against the high-water mark of 34 and refuses. The incident that would have been a double-charge becomes a rejected request and a log line. Two requirements make fencing real rather than decorative. First, monotonicity at the source: the token must strictly increase across acquisitions — ZooKeeper's zxid and etcd's lease revisions provide this natively; a single Postgres sequence works fine at moderate scale. Second, and this is where designs quietly fail: conditional enforcement at the resource. The storage layer must be able to check the token atomically with the write — an UPDATE ... WHERE token < $new in SQL, compare-and-set in a KV store, or precondition headers on object storage. If the resource cannot check conditions (a plain filesystem, a third-party API with no idempotency mechanism), fencing is impossible at that boundary, and the correct move is relocating the invariant to a layer that can enforce it rather than pretending the lock alone protects it.

Q: Are PostgreSQL advisory locks good for distributed locking?

Genuinely good in a specific envelope, with one modern landmine. The case for them: if your fleet already shares a Postgres instance, pg_advisory_lock gives you mutual exclusion with no new infrastructure, and the session scoping cuts both ways — there is no TTL to tune because the lock lives exactly as long as the connection, and if the holder crashes, the connection drops and the lock releases automatically. Session-scoped locks plus a small token table for fencing make a respectable correctness setup for moderate scale, and pg_try_advisory_lock supports non-blocking attempts for job-runner patterns. The landmine is connection pooling: advisory locks belong to the server session, and a transaction-mode pooler (PgBouncer, RDS Proxy in transaction mode) hands consecutive transactions different server sessions. Acquire an advisory lock in one transaction and the pooler may return that session to the pool — your next statement runs elsewhere, the lock you believe you hold is attached to a connection you will never touch again, and it releases whenever that pooled session recycles. Transaction-scoped advisory locks (pg_advisory_xact_lock) are pooler-compatible since they release at commit, but they cannot span transactions. The rules: session advisory locks require a direct connection or session-mode pooling; audit this before adopting either the locks or the pooler; and remember the ceiling — every waiting lock is a held connection, so heavy lock contention and small pools compose badly.

Q: When should I use etcd or ZooKeeper instead of Redis for locking?

When the lock is load-bearing for correctness — leader election, singleton schedulers, coordination where split-brain corrupts state — and you need the guarantees a consensus protocol provides. The difference is architectural, not incremental. etcd (Raft) and ZooKeeper (ZAB) replicate every lock operation through a quorum before acknowledging: a lock granted is a lock the cluster agreed on, sessions and leases detect dead holders via heartbeats rather than guessed TTLs, and both expose natively monotonic counters (etcd's lease revisions, ZooKeeper's zxid) that serve as fencing tokens without extra machinery — which matters because the token, not the lock, is what actually protects the resource. The cost is the PACELC bill: consensus systems choose consistency over availability, so during a network partition the minority side cannot grant or renew locks at all. For a correctness lock this is exactly what you want — unavailable beats wrong — but it means your locking layer inherits consensus operational discipline (odd-sized clusters, quorum health, leader stability). Redis-based locks invert every one of those properties: available and fast, no consensus, no native fencing — the right shape for efficiency locks where overlap is cheap. The selection rule in one line: if a lock failure appears in a postmortem as "wasted compute," Redis; if it appears as "data corruption" or "double payment," consensus plus fencing — and if you are on Kubernetes, the built-in Lease object gives you the consensus-backed primitive without running your own cluster.

Q: How long should a distributed lock TTL be?

Long enough that a healthy holder never expires mid-work, short enough that a dead holder does not stall the system — and the honest answer is derived, not guessed: measure the critical section's p99 duration and set the TTL at 3 to 10x that, then re-derive when the work changes. A 200 ms section wants a TTL of a second or two; if your section legitimately runs 30 seconds, the design is wrong before the TTL question arises — do the slow work outside the lock and lock only the short commit step, because long TTLs widen every failure window (a crashed holder blocks successors for the full duration) while long sections invite pause-overlap (the GC stop that outlives your lease). For work of genuinely variable length, use a modest TTL plus a heartbeat: a background thread extends the lease periodically — always carrying the same token it acquired with, never blindly re-acquiring, so a lost lock is detected rather than papered over. A heartbeat failure must abort the work, not shrug. Pair the TTL with two more clock-adjacent disciplines: release-by-token (compare-and-delete, so a slow worker cannot delete the lock its successor now holds) and the standing assumption that expiry WILL happen mid-section someday regardless of tuning — which is the entire argument for fencing tokens on any path where that day costs more than a retry.

Q: Can I avoid distributed locks entirely?

More often than not, and the designs that avoid them are usually simpler and safer than the locks they replace. The substitutions, by problem shape. Duplicate execution (retries, at-least-once delivery, double-clicks): idempotency keys — a unique constraint on an operation ID makes the second execution a no-op, with no TTL to tune, no pause windows, and coverage for retry storms that locks never provided. One-worker-per-entity: partition instead of lock — a Kafka topic keyed by entity ID makes single-writer-per-key a structural property of the log; or, database-backed, SELECT ... FOR UPDATE SKIP LOCKED lets a fleet of workers claim rows exclusively with zero additional infrastructure, which is the foundation of every solid Postgres job queue. Invariants over rows you own: a transaction with FOR UPDATE at the appropriate isolation level is a real lock with automatic release and no clock problem — the row is simultaneously the resource and the fence. Cross-service invariants: no lock spans two services' databases anyway; that is saga-and-outbox territory, compensation rather than mutual exclusion. What legitimately remains after the substitutions is the true singleton: leader election, a migration runner, a scheduler that must be unique fleet-wide. Those deserve a consensus-backed lease (etcd, ZooKeeper, Kubernetes Lease) — and still write their outputs with fencing tokens, because deposed leaders reliably get one final write in before they notice the world moved on.

Q: What happens to distributed locks during a network partition?

The lock system reveals which side of the CAP trade it was built on, and you inherit the consequences. Consensus-backed systems (etcd, ZooKeeper) choose consistency: a node partitioned from the quorum cannot grant, renew, or confirm locks — clients on the minority side get errors, their leases expire, and work halts on that side while the majority side proceeds with a guarantee that no second holder exists anywhere. For correctness locks this is the desired behavior (unavailable beats wrong), but your application must handle "cannot acquire or renew" as a first-class state: pause the work, do not limp onward assuming the last-known lock still stands. Availability-leaning setups (single Redis, Redlock across nodes) keep answering during partitions, which is precisely when their guarantees thin out: with Redlock, a client that can reach three of five nodes can acquire while a paused or partitioned previous holder still believes it owns the lock, and nothing at the resource arbitrates — unless you added fencing, which is the recurring moral of every partition story. Two disciplines make partitions survivable regardless of flavor: heartbeats that treat renewal failure as a stop-work signal (a holder that cannot renew must assume it has lost the lock before its local timer says so), and fencing tokens at the resource so that whatever confusion the partition creates among clients, stale writers lose deterministically when the network heals. If you have never run a partition drill against your locking layer, the first real partition runs it for you — against production.