# Safely reshard a Redis Cluster Resharding moves hash slots from one master to another while traffic is live. Redis Cluster supports this as a first-class operation, but the cluster is in a transient state for the duration, and a mistake can leave it that way. This runbook is the safe path. ## The mental model A Redis Cluster owns 16384 hash slots, partitioned across masters. `CLUSTER SLOTS` shows the current ownership. A reshard moves a range of slots from one master to one or more masters. While a slot is migrating, the source still serves it; the target redirects clients with `-ASK` until the move completes; then the cluster gossip flips ownership. The two cases this runbook covers: - **Add a master.** New shard joins, takes a share of the existing slots, and the cluster widens. - **Drain a master.** A retiring shard exports its slots to the remaining masters, then leaves the cluster. Both use the same `redis-cli --cluster reshard` machinery. ## 0. Pre-checks (do these before you touch anything) ```sh redis-cli -h $SEED -p $PORT CLUSTER INFO redis-cli -h $SEED -p $PORT CLUSTER NODES redis-cli -h $SEED -p $PORT --cluster check $SEED:$PORT ``` You want all of these to be true before resharding: - `cluster_state:ok` (not `fail`) - `cluster_slots_ok:16384`, `cluster_slots_pfail:0`, `cluster_slots_fail:0` - Every master has a replica (`--cluster check` says `OK`). Resharding without replicas means a host loss during the move can take a slot range out of service. - No `?` or `disconnected` lines in `CLUSTER NODES`. Also check business state: ongoing migrations, big `BGREWRITEAOF` in flight, or a deploy window that includes Redis are reasons to wait. ## 1. Snapshot the topology before the change ```sh redis-cli -h $SEED -p $PORT CLUSTER NODES > nodes.before.txt redis-cli -h $SEED -p $PORT CLUSTER SLOTS > slots.before.txt ``` A diff against the same commands after the reshard is the cheapest evidence that the move did what you intended. ## 2. Capacity check (will the target survive the slots?) A slot's size is `used_memory / number_of_slots_owned` only as an average. To be safer, look at the larger keys on the source: ```sh redis-cli -h $SRC --bigkeys redis-cli -h $SRC INFO memory | grep -E 'used_memory:|maxmemory:' ``` If the target master is already near its `maxmemory`, the move can push it over the limit and trigger eviction or OOM on the receiving side. Raise the target's `maxmemory` (or `CONFIG SET` it temporarily) before the move if so. ## 3. Adding a master Bring the new master up first as an isolated node, on the same network, with the same `cluster-enabled yes` config. Then join it to the cluster: ```sh # from a node already in the cluster redis-cli -h $SEED -p $PORT --cluster add-node $NEW_HOST:$NEW_PORT $SEED:$PORT ``` The new node is now a master with zero slots. Pair it with at least one replica before resharding: ```sh # bring up a new replica node (cluster-enabled), then attach it: redis-cli -h $SEED -p $PORT --cluster add-node $NEW_REP:$REP_PORT $SEED:$PORT --cluster-slave --cluster-master-id $NEW_MASTER_ID ``` Move slots into it. `--cluster reshard` is interactive by default; passing the flags up front makes the operation auditable: ```sh redis-cli -h $SEED -p $PORT --cluster reshard $SEED:$PORT \ --cluster-from ,, \ --cluster-to \ --cluster-slots \ --cluster-yes ``` For an even split across N masters, `` is `16384 / N` minus the slots already on the new node (zero on a fresh add). The from-list pulls a slice from each existing master so no single master is hit hard. ## 4. Draining a master The reverse: move every slot off the master, then remove it. ```sh # how many slots does it currently own? redis-cli -h $SEED -p $PORT CLUSTER COUNTKEYSINSLOT redis-cli -h $SEED -p $PORT --cluster check $SEED:$PORT # shows the master's slot range ``` Reshard them out, spread across the remaining masters: ```sh redis-cli -h $SEED -p $PORT --cluster reshard $SEED:$PORT \ --cluster-from \ --cluster-to \ --cluster-slots \ --cluster-yes ``` Repeat once per remaining master, splitting the slots. When the draining master owns zero slots, remove it (and its replica) from the cluster: ```sh redis-cli -h $SEED -p $PORT --cluster del-node $SEED:$PORT redis-cli -h $SEED -p $PORT --cluster del-node $SEED:$PORT ``` `del-node` refuses to remove a master that still owns slots, which is a useful guardrail. ## 5. While the reshard is running The slot range being moved is in a transient state. Things that can go wrong and how to handle them: - **Source dies mid-move.** The replica fails over and the cluster continues. Stop the reshard, wait for `cluster_state:ok`, and resume against the new master id. - **`--cluster check` reports `slots in importing state` or `migrating state` after the move ends.** A `redis-cli` SIGTERM or network blip can leave a slot half-migrated. Repair it: ```sh redis-cli -h $SEED -p $PORT --cluster fix $SEED:$PORT ``` This walks every node, finds slots in `IMPORTING` or `MIGRATING` state, and finishes or rolls them back according to where the keys actually live. - **Client errors (-MOVED / -ASK).** Smart clients handle these on their own. Older or hand-rolled clients sometimes do not. If you see -MOVED in app logs during the reshard, the client library is not cluster-aware and you have a separate problem. - **Memory ballooned on the target.** Pause the reshard (Ctrl-C is safe on the CLI tool; the in-progress slot finishes cleanly). Raise the target's `maxmemory` or move fewer slots per pass. ## 6. After ```sh redis-cli -h $SEED -p $PORT CLUSTER NODES > nodes.after.txt redis-cli -h $SEED -p $PORT CLUSTER SLOTS > slots.after.txt redis-cli -h $SEED -p $PORT --cluster check $SEED:$PORT redis-cli -h $SEED -p $PORT --cluster info $SEED:$PORT ``` `--cluster check` should report `OK` with `16384 slots covered`. `cluster_state` in `CLUSTER INFO` should be `ok`. Diff `nodes.before.txt` and `nodes.after.txt` to confirm the cluster looks the way you intended. Force one `BGREWRITEAOF` on each affected master if AOF is on, so the file is compact after the migration: ```sh for h in $MASTERS; do redis-cli -h $h BGREWRITEAOF; done ``` ## What the RKB tool gives you for free here - The Topology card visualizes the cluster shape (masters, replicas, slot ranges) and shows it after the live pull, so before and after look the same way. - The Nodes table flags any node with `link != connected` or `flags` other than master/replica, which is the same signal `--cluster check` gives. - The Health card flags `cluster_state:fail` and a non-ok replication link, so a stuck migration is visible immediately. - The Capacity card recomputes the per-shard memory from the plan, so a reshard that changes the master count produces a new sized `maxmemory` for every remaining master.