---
id: 99cebde8-b56d-4700-b303-aa2f92ce320f
---

# Redis cluster flapping incident.
---

A Redis cluster flapping incident occurs when changes are detected in the Redis replica connection. This can happen when replica nodes lose connection to the master and reconnect repeatedly. This can lead to unstable performance and can impact the stability of the Redis cluster.

### Parameters
```shell
# Environment Variables

export REDIS_HOST="PLACEHOLDER"

export REDIS_PORT="PLACEHOLDER"

export REDIS_LOG_FILE="PLACEHOLDER"

export REPLICA_NODE="PLACEHOLDER"

export MASTER_NODE="PLACEHOLDER"

```

## Debug

### Get the Redis cluster status
```shell
redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} cluster nodes
```

### Get the Redis cluster configuration
```shell
redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} cluster info
```

### Check if all Redis nodes are connected
```shell
redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} cluster check
```

### Check the Redis replication status
```shell
redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} info replication
```

### Check the Redis logs for any errors
```shell
tail -f ${REDIS_LOG_FILE}
```

### Check the Redis memory usage
```shell
redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} info memory
```

### Check the Redis CPU usage
```shell
redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} info cpu
```

### Check the Redis keyspace stats
```shell
redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} info keyspace
```

### Check the Redis client connections
```shell
redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} client list
```

### Check the network connectivity between Redis nodes
```shell
ping ${REDIS_HOST}
```

### Check the network latency between Redis nodes
```shell
redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} --latency
```

### Check the network bandwidth between Redis nodes
```shell
iperf -c ${REDIS_HOST} -p ${REDIS_PORT} -t 60 -i 10
```
## Repair

### Restart the Redis nodes one by one, starting with the replica nodes, to see if it resolves the issue.
```shell
#!/bin/bash

# Define the Redis nodes

REDIS_REPLICA_NODE=${REPLICA_NODE}

REDIS_MASTER_NODE=${MASTER_NODE}

# Restart the Redis replica nodes

for node in $REDIS_REPLICA_NODE; do

  ssh $node "sudo systemctl restart redis"

done

# Restart the Redis master node

ssh $REDIS_MASTER_NODE "sudo systemctl restart redis"

```

A Redis cluster flapping incident occurs when changes are detected in the Redis replica connection. This can happen when replica nodes lose connection to the master and reconnect repeatedly. This can lead to unstable performance and can impact the stability of the Redis cluster.


The Redis too many masters incident occurs when there are too many master nodes in a Redis cluster, leading to connection issues and potential data loss. This can happen due to misconfiguration, network issues, or other factors, and requires immediate attention to prevent further damage.


Redis too many masters incident

This incident type refers to an issue with Redis replication, which means that there is a problem with the synchronization of data between Redis instances. This issue could impact the availability and performance of the system and may require immediate attention to restore the replication and ensure data consistency. The incident could be caused by various factors, such as network problems, hardware failures, or configuration issues. The incident must be investigated and resolved as soon as possible to avoid any data loss or downtime.


Redis replication broken incident.

This incident type refers to a situation where Redis, a popular in-memory data structure store, has rejected connections. This could happen due to various reasons such as server overload, network issues, or misconfiguration. When this occurs, it can cause disruptions in the performance of applications that rely on Redis for data storage or caching. Thus, it requires immediate attention and resolution to minimize the impact on the applications and users.


Redis Rejected Connections Incident

The Redis missing master incident occurs when the Redis cluster has no node marked as master. This can cause problems with the Redis service and may result in service disruptions or failures. It is important to address this issue promptly to ensure the smooth operation of the Redis service.


Redis missing master incident

The Redis missing backup incident refers to a situation where Redis, a popular in-memory data structure store, has not been backed up for a certain period of time. This could lead to data loss or corruption in the event of a system failure or other issues. The incident typically requires immediate attention to ensure that the data is properly backed up and that any potential issues are resolved quickly to avoid further complications.


Redis missing backup incident.

```shell
# Environment Variables

export REDIS_HOST="PLACEHOLDER"

export REDIS_PORT="PLACEHOLDER"

export REDIS_LOG_FILE="PLACEHOLDER"

export REPLICA_NODE="PLACEHOLDER"

export MASTER_NODE="PLACEHOLDER"

```


### Get the Redis cluster status

```shell
redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} cluster nodes
```

### Get the Redis cluster configuration

```shell
redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} cluster info
```

### Check if all Redis nodes are connected

```shell
redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} cluster check
```

### Check the Redis replication status

```shell
redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} info replication
```

### Check the Redis logs for any errors

```shell
tail -f ${REDIS_LOG_FILE}
```

### Check the Redis memory usage

```shell
redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} info memory
```

### Check the Redis CPU usage

```shell
redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} info cpu
```

### Check the Redis keyspace stats

```shell
redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} info keyspace
```

### Check the Redis client connections

```shell
redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} client list
```

### Check the network connectivity between Redis nodes

```shell
ping ${REDIS_HOST}
```

### Check the network latency between Redis nodes

```shell
redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} --latency
```

### Check the network bandwidth between Redis nodes

```shell
iperf -c ${REDIS_HOST} -p ${REDIS_PORT} -t 60 -i 10
```


### Restart the Redis nodes one by one, starting with the replica nodes, to see if it resolves the issue.

```shell
#!/bin/bash

# Define the Redis nodes

REDIS_REPLICA_NODE=${REPLICA_NODE}

REDIS_MASTER_NODE=${MASTER_NODE}

# Restart the Redis replica nodes

for node in $REDIS_REPLICA_NODE; do

  ssh $node "sudo systemctl restart redis"

done

# Restart the Redis master node

ssh $REDIS_MASTER_NODE "sudo systemctl restart redis"

```


Redis cluster flapping incident.

Overview

Parameters

Debug

Get the Redis cluster status

Get the Redis cluster configuration

Check if all Redis nodes are connected

Check the Redis replication status

Check the Redis logs for any errors

Check the Redis memory usage

Check the Redis CPU usage

Check the Redis keyspace stats

Check the Redis client connections

Check the network connectivity between Redis nodes

Check the network latency between Redis nodes

Check the network bandwidth between Redis nodes

Repair

Restart the Redis nodes one by one, starting with the replica nodes, to see if it resolves the issue.

Learn more

Related Runbooks

Redis too many masters incident

Redis replication broken incident.

Redis Rejected Connections Incident

Redis missing master incident

Support