Redis Cluster: The Operational Reality Beyond Hash Slots

Redis is not a cache you bolt onto a slow database — it is a data structure server with a precisely bounded contract: sub-millisecond latency, in-memory semantics, and optional persistence. When your dataset outgrows the RAM of a single node, or when write throughput saturates the single-threaded event loop, that contract must be honoured across multiple nodes simultaneously. Redis Cluster is the mechanism for this — but its operational reality is substantially more complex than its theory. The hash slot formula and hash tag syntax you may have encountered in introductory documentation describe the what of Redis Cluster. This article is about the why it fails in production: MOVED vs ASK redirect semantics that crash naive clients, the cluster-require-full-coverage yes default that silently takes the entire cluster offline when one shard fails, and a resharding process that pauses access to migrating keys during live traffic.

1. Cluster Topology: Shards, Gossip & the 16,384-Slot Count

1.1 Shard Assignment

A Redis Cluster consists of N primary nodes, each owning a contiguous range of the 16,384 hash slots. Each primary has one or more replicas for fault tolerance.

1.2 Why 16,384 Slots?

The slot count of 16,384 is deliberately chosen to fit a slot coverage bitmap in a single gossip message without fragmentation. Each gossip heartbeat (PING/PONG) carries a bitmap of all slots a node owns. At 16,384 slots:

$$\frac{16384 \text{ bits}}{8 \text{ bits/byte}} = 2048 \text{ bytes} = 2\text{KB}$$

A 2KB bitmap fits comfortably within a single TCP segment alongside other gossip metadata. Doubling the slots to 32,768 would double gossip message size and increase network overhead across all nodes for every heartbeat.

1.3 Gossip Protocol: MEET, PING, PONG, FAIL

Cluster nodes discover each other and share topology changes through a gossip protocol — not through a central coordinator:

# View cluster topology (run from any node)
redis-cli -c CLUSTER NODES
# a3b4c5... 10.0.0.1:6379@16379 master - 0 1725350400 1 connected 0-5460
# d6e7f8... 10.0.0.2:6379@16379 slave a3b4c5... 0 1725350401 1 connected
# g9h0i1... 10.0.0.3:6379@16379 master - 0 1725350402 2 connected 5461-10922
# j2k3l4... 10.0.0.4:6379@16379 slave g9h0i1... 0 1725350403 2 connected
# m5n6o7... 10.0.0.5:6379@16379 master - 0 1725350404 3 connected 10923-16383
# p8q9r0... 10.0.0.6:6379@16379 slave m5n6o7... 0 1725350405 3 connected

# Cluster health summary
redis-cli -c CLUSTER INFO
# cluster_enabled: 1
# cluster_state: ok
# cluster_slots_assigned: 16384
# cluster_known_nodes: 6
# cluster_size: 3          ← 3 primary shards
# cluster_stats_messages_sent: 8234751

2. MOVED vs. ASK: Client Redirect Semantics

When a client sends a command for a key that does not belong to the connected node, the node does not proxy the request — it redirects the client. There are two distinct redirect types with entirely different semantics.

2.1 MOVED: Permanent Redirect

MOVED means the key's hash slot permanently lives on another node. The client should update its slot map and send all future commands for this slot to the indicated node.

Client → Node A:  GET user:100
Node A:           CRC16("user:100") % 16384 = slot 5789  (belongs to Node B)
Node A → Client:  -MOVED 5789 10.0.0.3:6379
Client:           Update local slot map: slot 5789 → Node B
Client → Node B:  GET user:100  (correct node)
Node B → Client:  "alice"

When does MOVED happen?

• During normal cluster operation when the client's cached slot map is stale (e.g., after a shard was added and slots were migrated).
• On the first connection before the client has built a complete slot map.

2.2 ASK: Temporary Redirect

ASK means the key's slot is currently being migrated to another node. The key may or may not have already moved. The client must send ASKING before the actual command on the destination node — this is a one-time override that allows the destination to serve the key even though it doesn't officially own the slot yet.

Critical distinction:

2.3 Smart Client Slot Map Caching

import { Cluster } from 'ioredis'

// ✅ Smart cluster client — handles MOVED, ASK, slot map refreshes automatically
const cluster = new Cluster(
  [
    { host: '10.0.0.1', port: 6379 },
    { host: '10.0.0.3', port: 6379 },
    { host: '10.0.0.5', port: 6379 },
  ],
  {
    redisOptions: { password: process.env.REDIS_PASSWORD },
    clusterRetryStrategy: (times) => Math.min(times * 100, 3000),
    // On MOVED: automatically refresh full slot map from CLUSTER SLOTS
    enableReadyCheck: true,
    // Route read commands to replicas (reduces primary load)
    scaleReads: 'slave',
  }
)

// ioredis Cluster mode:
// - Maintains an internal slot→node map
// - On MOVED: refreshes the full map via CLUSTER SLOTS and retries
// - On ASK: sends ASKING + retries on redirected node without map update
// - On connection failure: retries on a different node in the same shard

3. cluster-require-full-coverage: The Default That Kills Clusters

# redis.conf (every cluster node)
cluster-require-full-coverage yes   # Default — and deeply dangerous

With cluster-require-full-coverage yes, if any shard's primary fails and no replica is available to take over (or the replica has not yet been promoted), the entire cluster stops accepting reads and writes — not just the failed shard, but all 16,384 slots across all nodes.

# ✅ Production configuration — always set this on all cluster nodes
cluster-require-full-coverage no
# When a shard fails, only that shard's keys become unavailable.
# All other shards continue serving their slot ranges normally.
# Clients hitting the failed shard's keys receive CLUSTERDOWN for those keys only.

4. Resharding: Slot Migration in Live Traffic

Adding or removing shards from a Redis Cluster requires migrating hash slots from their current owners to their new owners. This process is called resharding and it happens against live traffic.

4.1 The Slot Migration State Machine

# Step 1: Set slot as MIGRATING on the source node
redis-cli -h 10.0.0.1 CLUSTER SETSLOT 5000 MIGRATING <node-B-id>

# Step 2: Set slot as IMPORTING on the destination node
redis-cli -h 10.0.0.3 CLUSTER SETSLOT 5000 IMPORTING <node-A-id>

# Step 3: Move keys from source to destination
redis-cli -h 10.0.0.1 CLUSTER GETKEYSINSLOT 5000 100 | \
  xargs -I{} redis-cli -h 10.0.0.1 MIGRATE 10.0.0.3 6379 {} 0 5000 REPLACE

# Step 4: Complete the migration (gossip broadcasts the ownership change)
redis-cli -h 10.0.0.1 CLUSTER SETSLOT 5000 NODE <node-B-id>
redis-cli -h 10.0.0.3 CLUSTER SETSLOT 5000 NODE <node-B-id>

4.2 The Migration Pause

During the MIGRATING / IMPORTING state, each MIGRATE command for a key is atomic but blocking:

• The key is serialized, sent to the destination, and deleted from the source in a single operation.
• While the MIGRATE command executes, the source node's event loop is blocked for the duration of the network round-trip to the destination.

For large values (a Sorted Set with 100,000 members, a List with 50,000 entries), a single MIGRATE can block the event loop for tens of milliseconds. Clients hitting that node during MIGRATE experience latency spikes.

# Use the built-in cluster resharding tool (handles MIGRATE sequencing automatically)
redis-cli --cluster reshard 10.0.0.1:6379 \
  --cluster-from <source-node-id> \
  --cluster-to <dest-node-id> \
  --cluster-slots 500 \           # Migrate 500 slots
  --cluster-pipeline 20 \         # Batch 20 keys per MIGRATE call
  --cluster-yes                   # Skip interactive confirmation

4.3 Multi-Key Operations During Migration

During slot migration, the CROSSSLOT restriction still applies — but with an additional wrinkle. A Lua script or MULTI/EXEC block that references keys in a slot currently being migrated may receive ASK redirects mid-execution, which breaks atomicity.

5. Cluster Failure Modes

5.1 Primary Failure and Replica Promotion

When a primary fails in a cluster, its replicas wait for cluster-node-timeout (default: 15 seconds) before initiating an election:

redis-cli CLUSTER INFO | grep cluster_state
# cluster_state: fail   ← Shard primary down, replica not yet promoted

# Check specific node state
redis-cli CLUSTER NODES | grep fail
# a3b4c5... 10.0.0.1:6379 master,fail - ...  ← Failed primary flagged

The replica promotion in Cluster is automatic — no Sentinel required. Replicas vote among themselves using the Raft-adjacent cluster election protocol. The first replica to receive votes from a majority of primaries in the cluster is promoted.

5.2 CLUSTER RESET for Node Recovery

After replacing a failed node:

# Hard reset: removes all cluster configuration from node (use on a fresh replacement node)
redis-cli -h 10.0.0.1 CLUSTER RESET HARD

# Add the fresh node to the cluster
redis-cli --cluster add-node 10.0.0.7:6379 10.0.0.3:6379

# Make it a replica of the primary that lost its replica
redis-cli -h 10.0.0.7 CLUSTER REPLICATE <primary-node-id>

Summary

What's Next

In Part 5: Redis Pub/Sub vs. Streams — Choosing the Right Messaging Primitive, we examine Redis as a messaging system. Pub/Sub is fire-and-forget — a disconnected subscriber loses every message published during its absence, permanently. Streams provide persistence, consumer groups, and delivery acknowledgment. The distinction determines whether your notification system silently drops messages in production.