Redis Replication & High Availability: Sentinel, Failover & Split-Brain
Redis replication is asynchronous by default — a primary can acknowledge a write, crash, and lose that write permanently before any replica receives it. Sentinel provides automated failover but introduces a split-brain window where two primaries briefly accept writes simultaneously. This article deconstructs PSYNC mechanics, replication backlog sizing, WAIT for synchronous durability, and the min-replicas-to-write + min-replicas-max-lag split-brain prevention pairing.
Redis Replication & High Availability: Sentinel, Failover & Split-Brain
Redis is not a cache you bolt onto a slow database — it is a data structure server with a precisely bounded contract: sub-millisecond latency, in-memory semantics, and optional persistence. Part 2 established that persistence is optional and must be explicitly configured. This article confronts the next layer of the same contract: even with appendonly yes and appendfsync everysec enabled, Redis replication is asynchronous by default — the primary acknowledges OK before any replica has received the write. If the primary crashes between the acknowledgment and the replication, that write is gone. Adding a replica does not eliminate data loss; it changes the failure mode from "process crash" to "primary crash before replication." Understanding this distinction is what separates an engineer who added a replica from an engineer who has high availability.
This is Part 3 of the Redis Mastery series. It builds on Part 2 — Persistence (specifically the RDB full-sync mechanism used in replication) and introduces the HA topology that Part 4 (Cluster) extends further.
1. Replication Mechanics: PSYNC and the Replication Backlog
1.1 Full Sync vs. Partial Resync
When a replica connects to a primary for the first time — or after a disconnection — Redis must synchronize state. The protocol that manages this is PSYNC.
Full sync is expensive: it requires a BGSAVE on the primary (fork latency spike, COW memory doubling) plus a full RDB transfer over the network. For a 20GB dataset this can take minutes and puts pressure on both nodes.
Partial resync is cheap: the primary keeps a circular in-memory buffer (the replication backlog) of the most recent write commands. If the replica reconnects before the backlog overwrites the missed commands, only those missed bytes are resent.
1.2 The Replication Backlog: Size It or Pay Full-Sync Cost
The backlog is a circular buffer. If a replica is disconnected for longer than the buffer can hold (write throughput × disconnect duration > backlog size), partial resync is impossible and a full sync is triggered.
Sizing formula:
$$\text{repl-backlog-size} \geq \text{write throughput (bytes/sec)} \times \text{max expected disconnect (sec)}$$
| Write throughput | Expected network blip | Required backlog |
|---|---|---|
| 10 MB/s | 10 seconds | 100 MB |
| 50 MB/s | 30 seconds | 1.5 GB |
| 100 MB/s | 60 seconds | 6 GB |
The default repl-backlog-size 1mb is catastrophically small for write-heavy workloads. A 1MB backlog at 10MB/s write throughput is exhausted in 100 milliseconds — meaning any network hiccup longer than 100ms triggers an expensive full sync. Set the backlog to at least 10× your per-second write volume as a starting point.
2. Asynchronous Replication: The Durability Gap
2.1 The Exact Failure Scenario
This is the canonical async replication data loss scenario. The write was acknowledged. The data is gone.
2.2 WAIT: Synchronous Acknowledgment Per Command
Redis provides the WAIT command for per-command synchronous durability:
WAIT does not provide a hard durability guarantee. If the replica acknowledges and then the primary crashes before sending the WAIT response to the client, the write is still on the replica — but the client receives a timeout error. WAIT reduces the data-loss window; it does not eliminate it. For true zero-loss durability, use appendfsync always (Part 2) alongside WAIT 1 0 (infinite timeout).
2.3 replica-serve-stale-data: The Stale-Read Footgun
With the default yes, a replica that loses contact with its primary continues serving its last known data to clients. This is correct for cache use cases. For application data it is a silent stale-read source: the replica's data may be seconds or minutes behind the primary's committed state.
3. Measuring and Monitoring Replication Lag
Lag in INFO replication is in seconds, reported by the replica's heartbeat. It is a coarse metric. For precise byte-level lag:
4. Redis Sentinel: Automated Failover
4.1 Sentinel Architecture
Redis Sentinel is a distributed supervision system for Redis primary/replica topologies. It is a separate process (not built into the Redis server binary) that monitors your Redis instances and performs automated failover.
4.2 SDOWN vs. ODOWN: Subjective and Objective Down
Sentinel uses a two-stage failure detection to prevent split-brain scenarios where a network partition makes one Sentinel think the primary is down while others disagree:
| State | Meaning | Trigger |
|---|---|---|
| SDOWN (Subjective Down) | This Sentinel believes the primary is unreachable | PING response timeout exceeds down-after-milliseconds |
| ODOWN (Objective Down) | A quorum of Sentinels agree the primary is unreachable | quorum count of Sentinels have declared SDOWN |
Minimum viable Sentinel cluster is 3 nodes. With 2 Sentinels and quorum=1, a single Sentinel declares ODOWN and initiates failover — but cannot achieve majority agreement for the Sentinel-leader election required to execute the failover. The practical result: failover is triggered but never completes. With 2 Sentinels and quorum=2, both Sentinels must agree, but if one dies, failover is impossible. Three Sentinels with quorum=2 is the minimum configuration that provides real fault tolerance.
4.3 Failover Sequence
4.4 Client-Side Sentinel Integration
A sentinel-aware client connects to Sentinel addresses, not to the primary address directly. Connecting directly to the primary address and reusing it after failover points to the failed node indefinitely.
5. Split-Brain Prevention
The split-brain scenario occurs when a network partition causes some clients to reach the old primary (now isolated) while Sentinel promotes a new primary. Both primaries accept writes — and those writes diverge permanently.
5.1 min-replicas-to-write + min-replicas-max-lag
With this configuration, if the primary is partitioned and loses contact with all replicas for more than 10 seconds, it stops accepting writes — returning NOREPLICAS errors. This is deliberate write rejection to prevent split-brain divergence.
| Configuration | Behaviour | Use case |
|---|---|---|
min-replicas-to-write 0 (default) |
Primary always accepts writes regardless of replica state | Pure cache — data loss acceptable |
min-replicas-to-write 1 + min-replicas-max-lag 10 |
Primary refuses writes when isolated from replicas | Application data — split-brain prevention |
min-replicas-to-write 2 + min-replicas-max-lag 5 |
Strict — requires majority replica acknowledgment | Financial / audit data |
Set min-replicas-to-write 1 and min-replicas-max-lag 10 as the baseline for any Redis primary storing application state. This is the split-brain prevention default. Applications must handle NOREPLICAS errors gracefully (retry with backoff, fallback to database) — treat it the same as a connection error.
6. Sentinel vs. Cluster: The Decision
A common source of confusion is when to use Sentinel vs. Redis Cluster. They are not interchangeable:
| Concern | Redis Sentinel | Redis Cluster |
|---|---|---|
| Primary purpose | High availability (automated failover) | Horizontal scaling + HA |
| Sharding | ❌ No — entire dataset on one primary | ✅ Yes — dataset split across shards |
| Failover | ✅ Yes — Sentinel promotes a replica | ✅ Yes — built into cluster gossip |
| Multi-key commands | ✅ Yes — no slot constraints | ⚠️ Only within same hash slot |
| Dataset size | Bounded by single-node RAM | Scales horizontally |
| Client complexity | Low — sentinel-aware client | Higher — cluster-aware client |
| When to use | Dataset fits in RAM of one node | Dataset exceeds single-node RAM, or write throughput saturates one node |
Sentinel answers: "Who is the primary right now?" and automates the answer changing on failure. Cluster answers: "Which node owns this key?" and shards the dataset across nodes. They solve different problems. A Sentinel-configured client pointed at a Cluster node will break; a Cluster client pointed at a Sentinel topology will break. They have different connection models, different command constraints, and different client library APIs.
Summary
| Concept | Rule |
|---|---|
| Async replication gap | OK does not mean the replica has the write — use WAIT 1 100 after durability-critical writes to require replica acknowledgment. |
| Replication backlog | Size to at least write_throughput_bytes_per_sec × max_expected_disconnect_sec. The 1MB default is exhausted in milliseconds at production write rates. |
| replica-serve-stale-data | Keep yes for cache workloads; set no for application data — stale reads are worse than LOADING errors. |
| Sentinel quorum | Minimum 3 Sentinels with quorum=2. Two Sentinels cannot complete a failover under a single Sentinel failure. |
| SDOWN vs ODOWN | SDOWN is one Sentinel's opinion. ODOWN is the quorum verdict that triggers failover. |
| Split-brain prevention | min-replicas-to-write 1 + min-replicas-max-lag 10 causes the primary to reject writes when isolated from replicas — deliberate write refusal is safer than silent divergence. |
| Sentinel vs Cluster | Sentinel = HA for a single-primary topology. Cluster = HA + sharding. Never mix client types. |
What's Next
In Part 4: Redis Cluster — The Operational Reality Beyond Hash Slots, we move from single-primary high availability to horizontal sharding. The hash slot formula and hash tag syntax are just the theory — the operational reality is
MOVEDvsASKredirect semantics, thecluster-require-full-coverage yesdefault that silently takes down your entire cluster when one shard fails, and a resharding process that pauses access to migrating keys in live production traffic.
This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.