Siddhant Deval
Siddhant Deval
backend21 min read

Lua Scripting, MULTI/EXEC & True Atomicity in Redis

MULTI/EXEC provides false atomicity — runtime errors inside a transaction do not abort the batch, silently leaving the dataset partially mutated. Lua scripts provide true all-or-nothing atomicity but block the entire server event loop for their duration. This article deconstructs the MULTI/EXEC partial-failure trap, WATCH optimistic locking, Lua server-side atomicity guarantees, EVALSHA script caching, and the Redis 7.0 FUNCTION LOAD upgrade path.

Lua Scripting, MULTI/EXEC & True Atomicity in Redis

Redis is not a cache you bolt onto a slow database — it is a data structure server with a precisely bounded contract: sub-millisecond latency, in-memory semantics, and optional persistence. Atomicity is the most misunderstood dimension of that contract. Engineers reach for MULTI/EXEC expecting database transaction semantics: if anything fails, the entire batch rolls back. This expectation is wrong. MULTI/EXEC executes a queued sequence of commands and returns a response array — but runtime errors in individual commands do not abort the batch. The preceding commands have already applied. The dataset is in a partially mutated state. There is no rollback, no compensation, and no log entry. This article deconstructs both Redis atomicity mechanisms — MULTI/EXEC and Lua scripts — and explains exactly what guarantee each one makes.

Architectural Note

This is Part 6 of the Redis Mastery series. It can be read after Part 4 (Cluster) or Part 5 (Streams).

On-ramp: The Distributed Locking article already contains production-complete Lua patterns: atomic lock release, watchdog lease renewal, and a full sliding-window rate limiter. This article does not repeat those patterns. It explains the execution engine they run on — the guarantees and non-guarantees of the Lua runtime and the MULTI/EXEC transaction model — so you can write new atomic operations with confidence.


1. MULTI/EXEC: The Partial-Failure Trap

1.1 How MULTI/EXEC Works

MULTI begins a transaction block. Commands sent after MULTI are not executed immediately — they are queued in the client's transaction buffer. EXEC sends all queued commands to the server in a single network write and receives an array of results.

TYPESCRIPT
// MULTI/EXEC transaction: debit balance + append ledger entry
const pipeline = redis.multi()
pipeline.decrby('account:42:balance', 50)     // Command 1: debit $50
pipeline.rpush('account:42:ledger', JSON.stringify({
  type: 'debit', amount: 50, ts: Date.now()
}))                                             // Command 2: ledger entry
const results = await pipeline.exec()
// results: [[null, 950], [null, 1]]
// [error, value] pairs — null error = success

The server processes both commands atomically in the sense that no other client command interleaves between Command 1 and Command 2. This is the atomicity guarantee MULTI/EXEC provides.

1.2 What MULTI/EXEC Does NOT Provide

MULTI/EXEC provides isolation (no interleaving), not atomicity in the ACID sense (all-or-nothing). The distinction:

TYPESCRIPT
// ❌ Broken pattern: assuming EXEC failure means nothing applied
const results = await redis.multi()
  .decrby('account:42:balance', 50)
  .rpush('account:42:ledger', JSON.stringify({ type: 'debit', amount: 50 }))
  .exec()

// results might be: [[null, 950], [WRONGTYPEError, null]]
// The debit applied (950) even though the ledger write failed
// Checking only the outer exec() result misses the per-command errors

// ✅ Correct: check each result pair individually
if (results === null) {
  // EXEC returned null — WATCH detected a conflict (see §2)
  throw new Error('Transaction aborted due to concurrent modification')
}
for (const [err, value] of results) {
  if (err) {
    // A command failed — handle compensation logic
    throw new Error(`Transaction partially applied: ${err.message}`)
  }
}

1.3 Two Types of Errors in MULTI/EXEC

Error type When detected Effect on EXEC
Syntax error (wrong arg count, unknown command) At queue time (before EXEC) EXEC is aborted entirely — no commands run
Runtime error (WRONGTYPE, out-of-range, etc.) During EXEC execution ❌ Preceding commands already applied; EXEC continues
BASH
# Syntax error — detected at queue time, EXEC aborts
redis-cli MULTI
redis-cli SET foo bar baz extra-arg-causes-syntax-error
# → ERR wrong number of arguments
redis-cli EXEC
# → EXECABORT Transaction discarded because of previous errors.
# No commands ran — safe.

# Runtime error — detected during execution, no rollback
redis-cli MULTI
redis-cli INCR mystring        # mystring holds "hello" — WRONGTYPE at runtime
redis-cli SET mykey myvalue    # This runs even though INCR failed
redis-cli EXEC
# → 1) ERR value is not an integer or out of range  ← INCR failed
# → 2) OK                                           ← SET succeeded
# SET ran. The dataset was mutated. No rollback.
Performance / Safety Warning

The most dangerous scenario: a type error on the second or third command in a long transaction. Commands before the error have applied. Commands after the error also apply (EXEC continues past runtime errors). Only the specific failing command is skipped. Engineers who test with consistent data never hit this path — it surfaces months later when data corruption causes a business logic error.


2. WATCH / MULTI / EXEC: Optimistic Locking (CAS)

WATCH implements Compare-and-Swap (CAS) optimistic locking. If any watched key is modified between WATCH and EXEC by any client, EXEC returns null (nil) — aborting the entire transaction without applying any commands.

TYPESCRIPT
// ✅ Optimistic locking: WATCH → MULTI → EXEC with retry
async function transferFunds(fromKey: string, toKey: string, amount: number): Promise<void> {
  const MAX_RETRIES = 5

  for (let attempt = 0; attempt < MAX_RETRIES; attempt++) {
    // Watch both keys — any external modification aborts the transaction
    await redis.watch(fromKey, toKey)

    const balance = parseInt(await redis.get(fromKey) ?? '0', 10)
    if (balance < amount) {
      await redis.unwatch()
      throw new Error('Insufficient funds')
    }

    const result = await redis
      .multi()
      .decrby(fromKey, amount)
      .incrby(toKey, amount)
      .exec()

    if (result !== null) {
      return // Transaction succeeded
    }
    // result === null: a concurrent client modified fromKey or toKey
    // Retry — re-read the current balance and attempt again
    await new Promise((r) => setTimeout(r, Math.random() * 50))
  }

  throw new Error('Transaction failed after maximum retries — high contention detected')
}

2.1 The ABA Problem with WATCH

WATCH detects any modification to the watched key — including a modification that sets the key back to the value it had when WATCH was called.

Timeline:
  Client A: WATCH account:42:balance   → current value: 1000
  Client B: SET account:42:balance 500
  Client B: SET account:42:balance 1000  (restored to original)
  Client A: MULTI / EXEC
  → Result: null (aborted) — WATCH detected B's modifications even though the final value is unchanged

For most use cases this is acceptable — Client A's retry will read the correct current value. For cases where the value identity matters (not just the current value), combine WATCH with a monotonic version counter.


3. Lua Scripts: True All-or-Nothing Atomicity

Lua scripts provide genuine atomic execution. A Lua script runs on the Redis server inside the event loop — no other command executes while the script is running. If the script encounters an error and calls error(), it aborts without applying any preceding redis.call() — true rollback.

3.1 EVAL Syntax and Argument Passing

LUA
-- EVAL script numkeys [key [key ...]] [arg [arg ...]]
-- KEYS[1..numkeys]: key names (used for cluster slot routing validation)
-- ARGV[1..N]: arbitrary arguments

-- Example: atomic check-and-set (CAS) — set only if current value matches expected
local current = redis.call('GET', KEYS[1])
if current == ARGV[1] then
  redis.call('SET', KEYS[1], ARGV[2])
  return 1  -- Success
else
  return 0  -- No-op: value changed
end
TYPESCRIPT
// Execute from TypeScript (ioredis)
const script = `
  local current = redis.call('GET', KEYS[1])
  if current == ARGV[1] then
    redis.call('SET', KEYS[1], ARGV[2])
    return 1
  else
    return 0
  end
`
const result = await redis.eval(
  script,
  1,                   // numkeys
  'account:42:status', // KEYS[1]
  'pending',           // ARGV[1] — expected current value
  'confirmed'          // ARGV[2] — new value to set
)
// result: 1 (applied) or 0 (value did not match expected — no-op)

3.2 redis.call() vs redis.pcall()

Function Error handling When to use
redis.call(cmd, ...) Propagates errors to the caller — aborts the script and returns an error reply Default — fail loudly on unexpected errors
redis.pcall(cmd, ...) Captures errors as return values — script continues executing When a specific command failure is an expected, handleable condition
LUA
-- ❌ Using pcall where call should be used — swallows unexpected errors silently
local ok = redis.pcall('INCR', KEYS[1])
-- If INCR fails with WRONGTYPE, ok = {err = "WRONGTYPE..."} and script continues
-- The caller receives the script's return value, not the error — may be misinterpreted

-- ✅ Correct: use call() for commands that must succeed; pcall() for optional side-effects
redis.call('DECRBY', KEYS[1], ARGV[1])        -- Must succeed — fail loudly if not
local logged = redis.pcall('RPUSH', KEYS[2], ARGV[2])  -- Logging is best-effort
if logged['err'] then
  -- Log the error but don't abort the critical path
end

3.3 The Event Loop Blocking Constraint

Performance / Safety Warning

A Lua script blocks the Redis event loop for its entire duration. While a script runs, no other command — GET, SET, PING, nothing — is processed. All connected clients queue up waiting for the script to complete.

Client A: EVAL complex_script 1 mykey → starts executing (30ms Lua computation)
Client B: GET other:key               → queued — blocked for 30ms
Client C: PING                        → queued — blocked for 30ms
Client D: SET session:99 "active"     → queued — blocked for 30ms
...30ms later...
All queued clients receive their responses simultaneously

For a Redis instance serving 10,000 requests/second, a 30ms script stalls every request for 30ms — a 300,000ms cumulative client wait for a single script invocation.

Script duration targets:

Target Acceptable use case
< 1ms Most atomic operations (CAS, counter increments, conditional sets)
1–5ms Small loops over bounded key sets
5–50ms Acceptable maximum — monitor with SLOWLOG
> 50ms lua-time-limit watchdog fires — SCRIPT KILL available
> lua-time-limit (default 5000ms) Server returns BUSY errors to all clients; only SCRIPT KILL or SHUTDOWN NOSAVE unblocks
INI
# redis.conf
lua-time-limit 5000   # Milliseconds before server starts accepting SCRIPT KILL
# After lua-time-limit, clients receive: BUSY Redis is busy running a script
# The script continues running until: it finishes, SCRIPT KILL is issued, or server shuts down
BASH
# Kill a runaway script (only works if the script hasn't written yet — scripts with writes cannot be killed)
redis-cli SCRIPT KILL
# → OK (script killed) or ERR NOTBUSY (no script running)

# Note: if the script has performed at least one write, SCRIPT KILL returns:
# ERR UNKILLABLE Sorry the script already executed write commands
# against the dataset. You can either wait the script terminates
# or kill the server in a savage way (SHUTDOWN NOSAVE).

4. EVALSHA: Script Caching in Production

EVAL transmits the full script body with every invocation. For scripts of 200+ lines invoked thousands of times per second, this is significant bandwidth overhead. EVALSHA decouples script loading from execution.

BASH
# Step 1: Load the script and receive its SHA1 hash
redis-cli SCRIPT LOAD "return redis.call('GET', KEYS[1])"
# → "e0e1f9fabfa9d353a0970a28c4ff7bcf81d6c9f1"

# Step 2: Execute by SHA1 — no script body transmitted
redis-cli EVALSHA e0e1f9fabfa9d353a0970a28c4ff7bcf81d6c9f1 1 mykey
# → "myvalue"

# Check if a script is loaded
redis-cli SCRIPT EXISTS e0e1f9fabfa9d353a0970a28c4ff7bcf81d6c9f1
# → 1 (loaded) or 0 (not loaded)

# Scripts are cleared by:
redis-cli SCRIPT FLUSH          # Explicitly (all scripts)
redis-cli DEBUG RELOAD          # Server config reload
# Note: RESTART also clears scripts — they are NOT persistent
TYPESCRIPT
// ✅ Production pattern: load once at startup, execute by SHA1 thereafter
class RedisScriptRunner {
  private sha: string | null = null
  private readonly script: string

  constructor(script: string) {
    this.script = script
  }

  async load(redis: Redis): Promise<void> {
    this.sha = await redis.script('LOAD', this.script) as string
  }

  async run(redis: Redis, numkeys: number, ...args: (string | number)[]): Promise<unknown> {
    if (!this.sha) throw new Error('Script not loaded — call load() first')
    try {
      return await redis.evalsha(this.sha, numkeys, ...args)
    } catch (err: any) {
      if (err.message.includes('NOSCRIPT')) {
        // Script was flushed (SCRIPT FLUSH or server restart) — reload and retry
        await this.load(redis)
        return await redis.evalsha(this.sha!, numkeys, ...args)
      }
      throw err
    }
  }
}

5. FUNCTION LOAD: Persistent Server-Side Logic (Redis 7.0+)

SCRIPT LOAD / EVALSHA has a critical limitation: loaded scripts are ephemeral. They are cleared by SCRIPT FLUSH, DEBUG RELOAD, and (in some configurations) server restart. Function Libraries fix this.

5.1 Creating a Function Library

LUA
-- mylib.lua — define a named library with one function
#!lua name=mylib

local function atomic_transfer(keys, args)
  local from    = keys[1]
  local to      = keys[2]
  local amount  = tonumber(args[1])

  local balance = tonumber(redis.call('GET', from) or '0')
  if balance < amount then
    return redis.error_reply('ERR Insufficient funds')
  end

  redis.call('DECRBY', from, amount)
  redis.call('INCRBY', to,   amount)
  return redis.status_reply('OK')
end

redis.register_function('atomic_transfer', atomic_transfer)
BASH
# Load the library (survives SCRIPT FLUSH and DEBUG RELOAD)
redis-cli FUNCTION LOAD "#!lua name=mylib\n$(cat mylib.lua)"

# Execute a registered function
redis-cli FCALL atomic_transfer 2 account:42:balance account:99:balance 100

# List all loaded libraries
redis-cli FUNCTION LIST

# Dump libraries for backup/restore
redis-cli FUNCTION DUMP

5.2 FUNCTION LOAD vs SCRIPT LOAD

Concern SCRIPT LOAD + EVALSHA FUNCTION LOAD + FCALL
Persistence ❌ Cleared by SCRIPT FLUSH, DEBUG RELOAD ✅ Persists across SCRIPT FLUSH; included in RDB/AOF
Naming SHA1 hash only — opaque Named function — human-readable
Library scoping No namespacing Library → function hierarchy
Cluster Must load on each node individually Replicates to replicas automatically
Redis version Available since Redis 2.6 Available since Redis 7.0
Pro Tip & Optimization

For new server-side logic: use FUNCTION LOAD. It survives restarts (because Function Libraries are included in RDB snapshots and AOF), replicates automatically to replicas, and is named instead of SHA1-addressed. SCRIPT LOAD is appropriate only for compatibility with Redis < 7.0.


6. Cluster Compatibility: Lua and Hash Slots

In a Redis Cluster, Lua scripts and MULTI/EXEC blocks have a critical constraint: all keys accessed must reside in the same hash slot. The server cannot proxy commands to other nodes mid-script.

LUA
-- ❌ Broken: keys in different hash slots in a Cluster
local a = redis.call('GET', 'user:42:profile')  -- slot 4041
local b = redis.call('GET', 'user:42:orders')   -- slot 5128 (different slot!)
-- CROSSSLOT error: Keys in request don't hash to the same slot
LUA
-- ✅ Correct: hash tags force all keys to the same slot
local a = redis.call('GET', '{user:42}:profile')  -- CRC16("user:42") = slot X
local b = redis.call('GET', '{user:42}:orders')   -- CRC16("user:42") = slot X (same)
-- Both keys co-located — script executes on one node
Crucial Requirement

All key names passed in KEYS[] to a Lua script must hash to the same slot. Redis validates this at script execution time in cluster mode. Design your key schemas with hash tags before writing multi-key Lua scripts — retrofitting hash tags requires renaming every affected key.


Summary

Concept Rule
MULTI/EXEC atomicity Provides isolation (no interleaving), not rollback. Runtime errors partially apply the batch — preceding commands are committed. Always check per-command result pairs in the EXEC response array.
Syntax vs runtime errors Syntax errors in queued commands abort EXEC entirely. Runtime errors (WRONGTYPE, out-of-range) do not — preceding commands already applied.
WATCH CAS EXEC returns null if any watched key is modified between WATCH and EXEC. Retry with fresh reads. Handles ABA: any modification triggers abort, even if value is restored.
Lua atomicity True all-or-nothing — no other command runs during a script; redis.call() errors abort the script without applying preceding writes.
Lua event loop cost A script blocks all clients for its duration. Target < 5ms. Monitor with SLOWLOG. Scripts with writes cannot be killed — only server shutdown unblocks them.
EVALSHA in production Load once at startup with SCRIPT LOAD; execute by SHA1 with EVALSHA. Handle NOSCRIPT errors by reloading.
FUNCTION LOAD (Redis 7.0+) Persistent, named, cluster-replicating alternative to SCRIPT LOAD. Use for all new server-side logic.
Cluster slot constraint All keys in a Lua script or MULTI/EXEC must hash to the same slot. Use hash tags ({tag}) to enforce co-location.

What's Next

In Part 7: Production Redis — Observability, Security, Connection Management & the Valkey Decision, we close the series with the operational layer: ACL-based security, TLS hardening, SLOWLOG and LATENCY HISTORY for latency forensics, hot/big key detection tools, connection pool sizing, and the 2025 Valkey vs. Redis licensing decision that every team running managed cloud Redis needs to make.

Research & Synthesis Note

This article was developed with AI-assisted deep search, specification cross-referencing, and technical research synthesis.

#Redis#Lua Scripting#Transactions#Atomicity#Backend Engineering
Siddhant Deval

Written by Siddhant Deval

Senior Full-Stack Engineer building high-scale architectures, browser performance engineering systems, and SaaS platforms.