Siddhant Deval
Siddhant DevalAuthor
Senior Full-Stack Engineer·Oct 4, 2026·24 min read

Distributed Concurrency & Locking with Redis: SETNX, Redlock, Fencing Tokens & Leases

Distributed locks in Redis provide efficiency, not absolute safety, unless guarded against process pauses, network partitions, and clock drift. This article deconstructs atomic SETNX PX acquisition, Lua release scripts, watchdog heartbeat lease extensions, the Redlock quorum algorithm, Martin Kleppmann's critique, and Monotonic Fencing Tokens.

Technical Series

Caching & Distributed Concurrency

Part 4 of 4

Distributed Concurrency & Locking with Redis: SETNX, Redlock, Fencing Tokens & Leases

A cache is not an optimization layer you sprinkle over slow queries — it is a volatile, distributed state machine with independent failure modes and eventual consistency tradeoffs. When you repurpose that volatile state machine for distributed concurrency, every guarantee holds only until the clock drifts, the runtime pauses, or the network partitions. In microservice architectures, coordinating execution across multiple independent server pods is a fundamental requirement: processing a payment exactly once, generating a monthly payroll report without duplicate runs, or leasing access to an external third-party API. Developers frequently reach for Redis, implement a quick SETNX lock 1, and assume they have mutual exclusion. In reality, naive distributed locks in Redis create silent deadlocks, stolen locks, and catastrophic data corruption. This article deconstructs single-instance lock mechanics, analyzes the Redlock quorum algorithm, explores Martin Kleppmann's famous critique, and derives the production standard: Monotonic Fencing Tokens.

Architectural Note

This is Part 4 (the final capstone) of the Caching & Distributed Concurrency series. It follows Part 1 — Caching Topologies, Part 2 — Cache Invalidation Strategies, and Part 3 — High-Concurrency Cache Hazards.


1. The Anatomy of a Single-Instance Distributed Lock

A distributed lock must satisfy three non-negotiable invariants:

  1. Safety (Mutual Exclusion): At most one worker can hold the lock at any given time.
  2. Liveness A (Deadlock Free): Even if a worker holding the lock crashes or gets partitioned, the lock must eventually become available again.
  3. Liveness B (Fault Tolerance): A client cannot release a lock that belongs to another client.
TYPESCRIPT
// ❌ Broken pattern 1: The Non-Atomic Deadlock Trap
export async function naiveLock1(resource: string) {
  // If the process crashes between SETNX and EXPIRE, the lock NEVER expires!
  // Every subsequent worker is deadlocked forever.
  const acquired = await redis.setnx(`lock:${resource}`, '1');
  if (acquired) {
    await redis.expire(`lock:${resource}`, 30);
  }
}

// ❌ Broken pattern 2: The Silent Lock-Stealing Release Trap
export async function naiveLock2(resource: string) {
  const key = `lock:${resource}`;
  // Correct atomic acquisition...
  await redis.set(key, '1', 'NX', 'PX', 10000);

  // Do work that unexpectedly takes 12 seconds (GC pause, slow DB query)...
  await executeLongRunningJob();

  // 💥 DISASTER: Lock expired at 10s. Worker B acquired the lock at 10.1s.
  // Worker A finishes at 12s and calls DEL, DELETING WORKER B'S LOCK!
  // Worker C now acquires the lock, resulting in concurrent execution between B and C!
  await redis.del(key);
}
TYPESCRIPT
// ✅ Production Single-Instance Distributed Lock with Watchdog & Atomic Lua Release
import Redis from 'ioredis';
import crypto from 'crypto';

export class RedisLock {
  private redis: Redis;
  private key: string;
  private token: string;
  private ttlMs: number;
  private watchdogTimer: NodeJS.Timeout | null = null;

  constructor(redisClient: Redis, resourceKey: string, ttlMs = 10000) {
    this.redis = redisClient;
    this.key = `lock:${resourceKey}`;
    this.token = crypto.randomUUID(); // Unique ownership token
    this.ttlMs = ttlMs;
  }

  // 1. Atomic acquisition with unique token and millisecond lease (PX)
  async acquire(): Promise<boolean> {
    const result = await this.redis.set(this.key, this.token, 'NX', 'PX', this.ttlMs);
    if (result === 'OK') {
      this.startWatchdog();
      return true;
    }
    return false;
  }

  // 2. Background lease renewal (Heartbeat Watchdog)
  private startWatchdog() {
    const renewalInterval = Math.floor(this.ttlMs / 3);
    this.watchdogTimer = setInterval(async () => {
      // Atomic Lua script to renew TTL ONLY if we still own the token
      const renewLua = `
        if redis.call("get", KEYS[1]) == ARGV[1] then
          return redis.call("pexpire", KEYS[1], ARGV[2])
        else
          return 0
        end
      `;
      const renewed = await this.redis.eval(renewLua, 1, this.key, this.token, this.ttlMs);
      if (!renewed) {
        this.stopWatchdog();
      }
    }, renewalInterval);
  }

  private stopWatchdog() {
    if (this.watchdogTimer) {
      clearInterval(this.watchdogTimer);
      this.watchdogTimer = null;
    }
  }

  // 3. Atomic Lua release to guarantee we only release our own lock
  async release(): Promise<boolean> {
    this.stopWatchdog();
    const releaseLua = `
      if redis.call("get", KEYS[1]) == ARGV[1] then
        return redis.call("del", KEYS[1])
      else
        return 0
      end
    `;
    const result = await this.redis.eval(releaseLua, 1, this.key, this.token);
    return result === 1;
  }
}
Single-instance lock lifecycle showing atomic acquisition, watchdog heartbeat lease extension, and atomic Lua release.
Single-instance lock lifecycle showing atomic acquisition, watchdog heartbeat lease extension, and atomic Lua release.

2. Multi-Instance Locking: The Redlock Algorithm

Single-instance locks fail if the Redis master crashes before replicating the lock key to its asynchronous replica. If the replica is promoted to master, a second worker acquires the same lock.

To overcome single-node failure, Salvatore Sanfilippo designed the Redlock Algorithm, which operates across $N$ completely independent Redis master nodes (typically $N=5$, running on separate physical machines or availability zones).

2.1 The Redlock Protocol Steps

  1. Timestamp: The client records the current time ($T_1$).
  2. Sequential Acquisition: The client attempts to acquire the lock on all $N$ instances sequentially, using the same key name and unique random value, with a small timeout on each instance (e.g., 5–50ms) to prevent hanging on a dead node.
  3. Quorum Verification: The client calculates the total elapsed time: $$\text{elapsed_time} = T_2 - T_1$$ The lock is successfully acquired if and only if:
    • The client acquired the lock on a majority of nodes ($M \ge \lfloor N/2 \rfloor + 1$, e.g., $\ge 3$ of 5).
    • The total elapsed time is strictly less than the lock validity time: $$\text{validity_time} = \text{TTL} - \text{elapsed_time} - \text{clock_drift_allowance}$$
  4. Rollback on Failure: If the client fails to acquire a quorum or if the validity time is zero/negative, it immediately attempts to unlock all $N$ instances (even nodes it failed to lock).

3. The Distributed Systems Critique: Kleppmann vs. Sanfilippo

In 2016, distributed systems researcher Martin Kleppmann published a comprehensive critique of Redlock ("How to do distributed locking"), proving that in asynchronous distributed systems with unsynchronized clocks, no pure lock-based algorithm can guarantee safety on its own.

3.1 The Stop-the-World GC Pause & Hypervisor Stall Hazard

Consider what happens when Client 1 acquires a Redlock across a majority of nodes with a 10-second TTL:

Because Client 1 experiences a runtime pause that exceeds the lock TTL, the lock expires silently. Client 2 legitimately acquires the lock and commits a write. When Client 1 resumes, its execution pointer is inside the critical section; it issues its write to the storage layer, overwriting and corrupting Client 2's data.

3.2 Efficiency Locks vs. Correctness Locks

Kleppmann establishes a fundamental distinction:

  • Locks for Efficiency: You want to avoid duplicate expensive computations (e.g., sending an email twice or resizing an image). Redlock or single-instance Redis is suitable here because an occasional duplicate execution is benign.
  • Locks for Correctness: You must guarantee data integrity under all failure conditions (e.g., transferring financial funds or updating inventory). Redlock alone is insufficient.

3.3 Monotonic Fencing Tokens

To guarantee safety for correctness locks, the storage engine (PostgreSQL, MySQL, S3) must act as the ultimate concurrency validator using Monotonic Fencing Tokens:

  1. When a client acquires a distributed lock, the lock service returns a strictly monotonically increasing integer token ($1, 2, 3 \dots$).
  2. Every time a client writes to the storage layer, it passes its fencing token along with the payload.
  3. The storage layer checks: $$\text{incoming_token} > \text{highest_committed_token}$$
  4. If a client attempts to commit with an older token (e.g., Client 1 waking from a GC pause with token=34 after Client 2 committed token=35), the storage engine rejects the write with a concurrency violation.
SQL
-- ✅ PostgreSQL Schema enforcing Monotonic Fencing Tokens
CREATE TABLE inventory_allocations (
  product_id      BIGINT PRIMARY KEY,
  reserved_units  INT NOT NULL,
  fencing_token   BIGINT NOT NULL  -- Monotonically increasing revision counter
);

-- Atomically reserve inventory ONLY if the worker's fencing token is strictly newer
UPDATE inventory_allocations
SET 
  reserved_units = reserved_units + 5,
  fencing_token = :worker_token
WHERE 
  product_id = :product_id 
  AND fencing_token < :worker_token;  -- Rejects stale zombie writes!
The GC pause lock steal hazard contrasted with Monotonic Fencing Token storage rejection.
The GC pause lock steal hazard contrasted with Monotonic Fencing Token storage rejection.

4. Distributed Coordination Primitives in Redis

Beyond mutual exclusion locks, Redis powers critical distributed coordination primitives:

4.1 Distributed Rate Limiting: Sliding Window Counter via Atomic Lua

High-scale APIs require precise rate limiting across hundreds of container pods. While Fixed Window counters suffer from 2× burst spikes at window boundaries, a Sliding Window Counter with Dual Buckets provides 99% accuracy with $O(1)$ memory.

LUA
-- ✅ Atomic Sliding Window Counter Rate Limiter (Lua Script)
-- KEYS[1]: Current window key (e.g. rate:user_100:1700000000)
-- KEYS[2]: Previous window key (e.g. rate:user_100:1699999940)
-- ARGV[1]: Window size in seconds (e.g. 60)
-- ARGV[2]: Current timestamp in seconds
-- ARGV[3]: Max allowed requests per window (e.g. 100)

local current_key = KEYS[1]
local prev_key = KEYS[2]
local window_size = tonumber(ARGV[1])
local now = tonumber(ARGV[2])
local max_limit = tonumber(ARGV[3])

local current_count = tonumber(redis.call('get', current_key) or "0")
local prev_count = tonumber(redis.call('get', prev_key) or "0")

-- Calculate weight of previous window based on elapsed time within current window
local time_into_current_window = now % window_size
local prev_weight = (window_size - time_into_current_window) / window_size
local estimated_count = math.floor(prev_count * prev_weight + current_count)

if estimated_count < max_limit then
  redis.call('incr', current_key)
  if current_count == 0 then
    redis.call('expire', current_key, window_size * 2)
  end
  return 1 -- Allowed
else
  return 0 -- Rate Limited (HTTP 429)
end

4.2 Idempotency Key Stores + Distributed Locking

For financial transactions, combining distributed locks with an Idempotency Store guarantees both mutual exclusion during execution and cached responses for duplicate retries:

TYPESCRIPT
// ✅ Production Two-Phase Idempotent Payment Processor
export async function processPaymentIdempotent(idempotencyKey: string, paymentFn: () => Promise<PaymentResult>) {
  const lock = new RedisLock(redisClient, `idempotency:lock:${idempotencyKey}`, 15000);
  const resultKey = `idempotency:result:${idempotencyKey}`;

  // Step 1: Check if this idempotency key was already completed in the past
  const existingResult = await redisClient.get(resultKey);
  if (existingResult) {
    return JSON.parse(existingResult);
  }

  // Step 2: Acquire distributed lock to prevent concurrent in-flight duplicates
  const acquired = await lock.acquire();
  if (!acquired) {
    throw new Error('Concurrent request in progress. Please retry shortly.');
  }

  try {
    // Double-check result key inside lock
    const cachedInside = await redisClient.get(resultKey);
    if (cachedInside) return JSON.parse(cachedInside);

    // Step 3: Execute the authoritative payment operation
    const result = await paymentFn();

    // Step 4: Persist result with 24-hour retention
    await redisClient.set(resultKey, JSON.stringify(result), 'EX', 86400);
    return result;
  } finally {
    await lock.release();
  }
}

Summary

Concurrency Pattern Production Invariant
Atomic Acquisition Always use SET key token NX PX ttl in a single command; non-atomic SETNX + EXPIRE risks permanent deadlock on crash.
Atomic Ownership Release Always release locks using an atomic Lua script that compares the caller's unique UUID token before deleting to prevent silent lock stealing.
Lease Watchdogs Long-running tasks require background watchdog timers to renew leases while execution is healthy.
Redlock Quorum Redlock provides fault-tolerant mutual exclusion across $N$ independent masters, but relies on synchronized physical clocks and bounded network delays.
Monotonic Fencing Tokens Distributed locks alone cannot guarantee safety against stop-the-world GC pauses; storage layers must enforce strictly increasing fencing tokens to reject stale zombie writes.
Sliding Window Rate Limiting Dual-bucket sliding window counters implemented in atomic Lua scripts provide $O(1)$ memory efficiency and eliminate boundary burst vulnerabilities.

Series Conclusion: The Resilient Backend Curriculum

Across this 4-part series, we have bridged the gap between intermediate implementation and senior-level distributed systems design:

  1. Part 1: Caching Topologies established the multi-tier hierarchy spanning edge CDNs, in-process L1 memory with RESP3 tracking, and Redis Cluster sharding.
  2. Part 2: Cache Invalidation Strategies proved why Write-Invalidate beats Write-Update, solved Replica-Lag poisoning, and implemented CDC WAL tailing.
  3. Part 3: High-Concurrency Cache Hazards derived the XFetch probabilistic early expiration formula and built Cuckoo anti-penetration filters.
  4. Part 4: Distributed Concurrency & Locking deconstructed atomic lock lifecycles, watchdog heartbeats, Redlock trade-offs, and Monotonic Fencing Tokens.

Mastering these systems transforms caching from a fragile performance band-aid into a resilient, highly scalable distributed architecture.

Research & Synthesis Note

This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.

#Distributed Locking#Redis#Concurrency#Redlock#Distributed Systems
Siddhant Deval

Written by Siddhant Deval

Senior Full-Stack Engineer building high-scale architectures, browser performance engineering systems, and SaaS platforms.