Siddhant Deval
Siddhant Deval
backend21 min read

Redis Replication & High Availability: Sentinel, Failover & Split-Brain

Redis replication is asynchronous by default — a primary can acknowledge a write, crash, and lose that write permanently before any replica receives it. Sentinel provides automated failover but introduces a split-brain window where two primaries briefly accept writes simultaneously. This article deconstructs PSYNC mechanics, replication backlog sizing, WAIT for synchronous durability, and the min-replicas-to-write + min-replicas-max-lag split-brain prevention pairing.

Redis Replication & High Availability: Sentinel, Failover & Split-Brain

Redis is not a cache you bolt onto a slow database — it is a data structure server with a precisely bounded contract: sub-millisecond latency, in-memory semantics, and optional persistence. Part 2 established that persistence is optional and must be explicitly configured. This article confronts the next layer of the same contract: even with appendonly yes and appendfsync everysec enabled, Redis replication is asynchronous by default — the primary acknowledges OK before any replica has received the write. If the primary crashes between the acknowledgment and the replication, that write is gone. Adding a replica does not eliminate data loss; it changes the failure mode from "process crash" to "primary crash before replication." Understanding this distinction is what separates an engineer who added a replica from an engineer who has high availability.

Architectural Note

This is Part 3 of the Redis Mastery series. It builds on Part 2 — Persistence (specifically the RDB full-sync mechanism used in replication) and introduces the HA topology that Part 4 (Cluster) extends further.


1. Replication Mechanics: PSYNC and the Replication Backlog

1.1 Full Sync vs. Partial Resync

When a replica connects to a primary for the first time — or after a disconnection — Redis must synchronize state. The protocol that manages this is PSYNC.

Full sync is expensive: it requires a BGSAVE on the primary (fork latency spike, COW memory doubling) plus a full RDB transfer over the network. For a 20GB dataset this can take minutes and puts pressure on both nodes.

Partial resync is cheap: the primary keeps a circular in-memory buffer (the replication backlog) of the most recent write commands. If the replica reconnects before the backlog overwrites the missed commands, only those missed bytes are resent.

1.2 The Replication Backlog: Size It or Pay Full-Sync Cost

BASH
# redis.conf
repl-backlog-size 1mb      # Default: 1MB — dangerously small for most production workloads
repl-backlog-ttl  3600     # How long to keep the backlog after last replica disconnects

The backlog is a circular buffer. If a replica is disconnected for longer than the buffer can hold (write throughput × disconnect duration > backlog size), partial resync is impossible and a full sync is triggered.

Sizing formula:

$$\text{repl-backlog-size} \geq \text{write throughput (bytes/sec)} \times \text{max expected disconnect (sec)}$$

Write throughput Expected network blip Required backlog
10 MB/s 10 seconds 100 MB
50 MB/s 30 seconds 1.5 GB
100 MB/s 60 seconds 6 GB
Performance / Safety Warning

The default repl-backlog-size 1mb is catastrophically small for write-heavy workloads. A 1MB backlog at 10MB/s write throughput is exhausted in 100 milliseconds — meaning any network hiccup longer than 100ms triggers an expensive full sync. Set the backlog to at least 10× your per-second write volume as a starting point.

BASH
# Check current backlog configuration and usage
redis-cli INFO replication | grep -E "repl_backlog|master_repl"
# master_repl_offset:    18446744073    ← bytes primary has sent since start
# repl_backlog_active:   1
# repl_backlog_size:     1048576        ← 1MB (default — too small)
# repl_backlog_first_byte_offset: 18445695497
# repl_backlog_histlen:  1048576        ← backlog is full (at capacity)

2. Asynchronous Replication: The Durability Gap

2.1 The Exact Failure Scenario

This is the canonical async replication data loss scenario. The write was acknowledged. The data is gone.

2.2 WAIT: Synchronous Acknowledgment Per Command

Redis provides the WAIT command for per-command synchronous durability:

BASH
# WAIT numreplicas timeout_ms
# Blocks until numreplicas replicas have acknowledged the current replication offset,
# or until timeout_ms milliseconds elapse — whichever comes first.

# After a critical write:
redis-cli SET payment:idempotency:abc "processed"
# → OK

redis-cli WAIT 1 100
# → 1  (1 replica acknowledged within 100ms)
# If 0 returned: no replica acknowledged within 100ms — treat write as unconfirmed
TYPESCRIPT
// ✅ Production pattern: WAIT after durability-critical writes
async function processPaymentIdempotent(idempotencyKey: string, result: string) {
  await redis.set(`payment:idempotency:${idempotencyKey}`, result, 'EX', 86400)

  // Require at least 1 replica to acknowledge before returning
  const ackedReplicas = await redis.wait(1, 100) // 1 replica, 100ms timeout

  if (ackedReplicas < 1) {
    // Replica did not acknowledge — the write may be lost on primary failure.
    // Options: retry, alert, or accept the risk based on business requirements.
    throw new Error(`Replication acknowledgment timeout for key ${idempotencyKey}`)
  }
}
Crucial Requirement

WAIT does not provide a hard durability guarantee. If the replica acknowledges and then the primary crashes before sending the WAIT response to the client, the write is still on the replica — but the client receives a timeout error. WAIT reduces the data-loss window; it does not eliminate it. For true zero-loss durability, use appendfsync always (Part 2) alongside WAIT 1 0 (infinite timeout).

2.3 replica-serve-stale-data: The Stale-Read Footgun

INI
# redis.conf (replica)
replica-serve-stale-data yes   # Default

With the default yes, a replica that loses contact with its primary continues serving its last known data to clients. This is correct for cache use cases. For application data it is a silent stale-read source: the replica's data may be seconds or minutes behind the primary's committed state.

BASH
# ❌ Scenario with replica-serve-stale-data yes:
# Primary: SET account:42:balance 1000
# Network partition: replica loses primary for 30 seconds
# Application reads from replica: GET account:42:balance → "1000"
# Primary (still alive on other side of partition): balance updated to 900 after debit
# Replica returns stale "1000" for 30 seconds — application makes decisions on wrong balance

# Check replication lag before trusting replica reads
redis-cli -p 6380 INFO replication | grep master_last_io_seconds_ago
# → master_last_io_seconds_ago: 31   ← 31 seconds since last contact with primary
INI
# ✅ For application data (not cache): reject stale reads on disconnected replicas
replica-serve-stale-data no
# Replicas return SYNC error for all read commands when disconnected from primary
# Client receives: LOADING Redis is loading the dataset in memory
# Better than silently serving stale data

3. Measuring and Monitoring Replication Lag

BASH
# Full replication status
redis-cli INFO replication
# role: master
# connected_slaves: 2
# slave0: ip=10.0.0.2,port=6380,state=online,offset=18446744000,lag=0
# slave1: ip=10.0.0.3,port=6380,state=online,offset=18446743950,lag=1
# master_repl_offset: 18446744073
# repl_backlog_size: 104857600

# lag=0: replica is within 1 replication cycle (typically <10ms) of primary
# lag=1: replica is 1 replication cycle behind — acceptable under load
# lag>5: significant lag — investigate disk I/O, network bandwidth, or slow replica

Lag in INFO replication is in seconds, reported by the replica's heartbeat. It is a coarse metric. For precise byte-level lag:

BASH
# Byte-level replication lag
master_offset=$(redis-cli INFO replication | grep master_repl_offset | cut -d: -f2 | tr -d '\r ')
slave_offset=$(redis-cli -p 6380 INFO replication | grep master_repl_offset | cut -d: -f2 | tr -d '\r ')
echo "Lag bytes: $((master_offset - slave_offset))"

4. Redis Sentinel: Automated Failover

4.1 Sentinel Architecture

Redis Sentinel is a distributed supervision system for Redis primary/replica topologies. It is a separate process (not built into the Redis server binary) that monitors your Redis instances and performs automated failover.

4.2 SDOWN vs. ODOWN: Subjective and Objective Down

Sentinel uses a two-stage failure detection to prevent split-brain scenarios where a network partition makes one Sentinel think the primary is down while others disagree:

State Meaning Trigger
SDOWN (Subjective Down) This Sentinel believes the primary is unreachable PING response timeout exceeds down-after-milliseconds
ODOWN (Objective Down) A quorum of Sentinels agree the primary is unreachable quorum count of Sentinels have declared SDOWN
INI
# sentinel.conf
sentinel monitor mymaster 10.0.0.1 6379 2
# Monitor "mymaster" at 10.0.0.1:6379
# Quorum: 2 — requires 2 Sentinels to agree on ODOWN before failover

sentinel down-after-milliseconds mymaster 5000
# Mark primary as SDOWN if no response for 5000ms

sentinel failover-timeout mymaster 60000
# Failover must complete within 60 seconds or it is aborted

sentinel parallel-syncs mymaster 1
# Number of replicas that can sync simultaneously after failover
# 1 = replicas sync one at a time (conservative — avoids overwhelming new primary)
Crucial Requirement

Minimum viable Sentinel cluster is 3 nodes. With 2 Sentinels and quorum=1, a single Sentinel declares ODOWN and initiates failover — but cannot achieve majority agreement for the Sentinel-leader election required to execute the failover. The practical result: failover is triggered but never completes. With 2 Sentinels and quorum=2, both Sentinels must agree, but if one dies, failover is impossible. Three Sentinels with quorum=2 is the minimum configuration that provides real fault tolerance.

4.3 Failover Sequence

4.4 Client-Side Sentinel Integration

Performance / Safety Warning

A sentinel-aware client connects to Sentinel addresses, not to the primary address directly. Connecting directly to the primary address and reusing it after failover points to the failed node indefinitely.

TYPESCRIPT
import Redis from 'ioredis'

// ❌ Broken: hardcoded primary address — fails permanently after failover
const redis = new Redis({ host: '10.0.0.1', port: 6379 })

// ✅ Correct: sentinel-aware client — automatically follows failover
const redis = new Redis({
  sentinels: [
    { host: '10.0.0.10', port: 26379 },
    { host: '10.0.0.11', port: 26379 },
    { host: '10.0.0.12', port: 26379 },
  ],
  name: 'mymaster',            // Matches sentinel.conf "sentinel monitor mymaster ..."
  sentinelRetryStrategy: (times) => Math.min(times * 100, 3000),
  connectTimeout: 10000,
})

// ioredis automatically:
// 1. Connects to any available Sentinel
// 2. Asks for current primary address (SENTINEL get-master-addr-by-name mymaster)
// 3. Connects to primary
// 4. Subscribes to Sentinel pub/sub for +switch-master events
// 5. Reconnects to new primary on failover notification

5. Split-Brain Prevention

The split-brain scenario occurs when a network partition causes some clients to reach the old primary (now isolated) while Sentinel promotes a new primary. Both primaries accept writes — and those writes diverge permanently.

5.1 min-replicas-to-write + min-replicas-max-lag

INI
# redis.conf (primary)

# Refuse to accept writes if fewer than 1 replica is connected and caught up
min-replicas-to-write 1

# A replica is considered "caught up" only if its replication lag is ≤ 10 seconds
min-replicas-max-lag 10

With this configuration, if the primary is partitioned and loses contact with all replicas for more than 10 seconds, it stops accepting writes — returning NOREPLICAS errors. This is deliberate write rejection to prevent split-brain divergence.

Configuration Behaviour Use case
min-replicas-to-write 0 (default) Primary always accepts writes regardless of replica state Pure cache — data loss acceptable
min-replicas-to-write 1 + min-replicas-max-lag 10 Primary refuses writes when isolated from replicas Application data — split-brain prevention
min-replicas-to-write 2 + min-replicas-max-lag 5 Strict — requires majority replica acknowledgment Financial / audit data
Pro Tip & Optimization

Set min-replicas-to-write 1 and min-replicas-max-lag 10 as the baseline for any Redis primary storing application state. This is the split-brain prevention default. Applications must handle NOREPLICAS errors gracefully (retry with backoff, fallback to database) — treat it the same as a connection error.


6. Sentinel vs. Cluster: The Decision

A common source of confusion is when to use Sentinel vs. Redis Cluster. They are not interchangeable:

Concern Redis Sentinel Redis Cluster
Primary purpose High availability (automated failover) Horizontal scaling + HA
Sharding ❌ No — entire dataset on one primary ✅ Yes — dataset split across shards
Failover ✅ Yes — Sentinel promotes a replica ✅ Yes — built into cluster gossip
Multi-key commands ✅ Yes — no slot constraints ⚠️ Only within same hash slot
Dataset size Bounded by single-node RAM Scales horizontally
Client complexity Low — sentinel-aware client Higher — cluster-aware client
When to use Dataset fits in RAM of one node Dataset exceeds single-node RAM, or write throughput saturates one node
Mental Model Check

Sentinel answers: "Who is the primary right now?" and automates the answer changing on failure. Cluster answers: "Which node owns this key?" and shards the dataset across nodes. They solve different problems. A Sentinel-configured client pointed at a Cluster node will break; a Cluster client pointed at a Sentinel topology will break. They have different connection models, different command constraints, and different client library APIs.


Summary

Concept Rule
Async replication gap OK does not mean the replica has the write — use WAIT 1 100 after durability-critical writes to require replica acknowledgment.
Replication backlog Size to at least write_throughput_bytes_per_sec × max_expected_disconnect_sec. The 1MB default is exhausted in milliseconds at production write rates.
replica-serve-stale-data Keep yes for cache workloads; set no for application data — stale reads are worse than LOADING errors.
Sentinel quorum Minimum 3 Sentinels with quorum=2. Two Sentinels cannot complete a failover under a single Sentinel failure.
SDOWN vs ODOWN SDOWN is one Sentinel's opinion. ODOWN is the quorum verdict that triggers failover.
Split-brain prevention min-replicas-to-write 1 + min-replicas-max-lag 10 causes the primary to reject writes when isolated from replicas — deliberate write refusal is safer than silent divergence.
Sentinel vs Cluster Sentinel = HA for a single-primary topology. Cluster = HA + sharding. Never mix client types.

What's Next

In Part 4: Redis Cluster — The Operational Reality Beyond Hash Slots, we move from single-primary high availability to horizontal sharding. The hash slot formula and hash tag syntax are just the theory — the operational reality is MOVED vs ASK redirect semantics, the cluster-require-full-coverage yes default that silently takes down your entire cluster when one shard fails, and a resharding process that pauses access to migrating keys in live production traffic.

Research & Synthesis Note

This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.

#Redis#Replication#High Availability#Sentinel#Distributed Systems
Siddhant Deval

Written by Siddhant Deval

Senior Full-Stack Engineer building high-scale architectures, browser performance engineering systems, and SaaS platforms.