---
id: 275db20f-d744-47d5-a9aa-22a3ce986228
---


# Redis disconnected slaves incident
---

This incident type refers to an issue with Redis where one or more slave instances have become disconnected, resulting in replication failure. This can cause data inconsistencies and may require immediate attention to restore normal functioning. The incident may be caused by a variety of factors, such as network issues, server failures, or misconfiguration.

### Parameters
```shell
# Environment Variables

export REDIS_HOST="PLACEHOLDER"

export REDIS_PORT="PLACEHOLDER"

export REDIS_INSTANCE_NAME="PLACEHOLDER"

export MASTER_HOSTNAME="PLACEHOLDER"

export MASTER_PORT="PLACEHOLDER"


```

## Debug

### Check if Redis is running
```shell
systemctl status redis
```

### Check Redis logs for any errors
```shell
tail -n 100 /var/log/redis/redis.log
```

### Check Redis replication status
```shell
redis-cli info replication
```

### Check number of connected slaves
```shell
redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} info clients | grep connected_slaves
```

### Check Redis latency
```shell
redis-cli --latency
```

### Check network connectivity to Redis instance
```shell
ping ${REDIS_HOST}
```

### Check if Redis instance is listening on the correct port
```shell
netstat -tuln | grep ${REDIS_PORT}
```

### Check Redis configuration file for any issues
```shell
cat /etc/redis/redis.conf
```

### Check resource utilization of Redis instance
```shell
top -c -p $(pidof redis-server)
```

## Repair

### Get the Redis instance name
```shell
redis_instance=${REDIS_INSTANCE_NAME}
```

### Check the replication status
```shell
redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} info replication | grep -E '^(role|connected_slaves|slave[[:digit:]]*):'
```

### Identify the disconnected slaves
```shell
redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} info replication | grep -E '^(role|connected_slaves|slave[[:digit:]]*):' | grep -v 'role:master' | grep -v 'connected_slaves:0' | grep -v 'slave[[:digit:]]*:ip' | grep -v 'slave[[:digit:]]*:state=online'
```
### Restart the Redis server and monitor the replication status.
```shell
#!/bin/bash

# Stop the Redis server

sudo systemctl stop redis

# Wait for the server to stop

sleep 5

# Start the Redis server

sudo systemctl start redis

# Wait for the server to start

sleep 5

# Monitor the replication status

redis-cli info replication

```

### Manually replicate the data from the master to the disconnected slaves.
```shell
#!/bin/bash

# Set the Redis master hostname and port

REDIS_MASTER=${MASTER_HOSTNAME}:${MASTER_PORT}

# Set the list of Redis slave instances to replicate to

REDIS_SLAVES="PLACEHOLDER"

# Loop through the list of Redis slaves and replicate data from the master

for redis_slave in "${REDIS_SLAVES[@]}"

do

  # Extract the hostname and port from the Redis slave string

  IFS=':' read -ra redis_slave_parts <<< "$redis_slave"

  redis_slave_hostname="${redis_slave_parts[0]}"

  redis_slave_port="${redis_slave_parts[1]}"

  # Replicate data from the Redis master to the slave

  redis-cli -h $redis_slave_hostname -p $redis_slave_port SLAVEOF $REDIS_MASTER

done

echo "Data replication from Redis master to all disconnected slaves completed."

```


This incident type refers to an issue with Redis where one or more slave instances have become disconnected, resulting in replication failure. This can cause data inconsistencies and may require immediate attention to restore normal functioning. The incident may be caused by a variety of factors, such as network issues, server failures, or misconfiguration.


The Redis too many masters incident occurs when there are too many master nodes in a Redis cluster, leading to connection issues and potential data loss. This can happen due to misconfiguration, network issues, or other factors, and requires immediate attention to prevent further damage.


Redis too many masters incident

This incident type refers to an issue with Redis replication, which means that there is a problem with the synchronization of data between Redis instances. This issue could impact the availability and performance of the system and may require immediate attention to restore the replication and ensure data consistency. The incident could be caused by various factors, such as network problems, hardware failures, or configuration issues. The incident must be investigated and resolved as soon as possible to avoid any data loss or downtime.


Redis replication broken incident.

This incident type refers to a situation where Redis, a popular in-memory data structure store, has rejected connections. This could happen due to various reasons such as server overload, network issues, or misconfiguration. When this occurs, it can cause disruptions in the performance of applications that rely on Redis for data storage or caching. Thus, it requires immediate attention and resolution to minimize the impact on the applications and users.


Redis Rejected Connections Incident

The Redis missing master incident occurs when the Redis cluster has no node marked as master. This can cause problems with the Redis service and may result in service disruptions or failures. It is important to address this issue promptly to ensure the smooth operation of the Redis service.


Redis missing master incident

The Redis missing backup incident refers to a situation where Redis, a popular in-memory data structure store, has not been backed up for a certain period of time. This could lead to data loss or corruption in the event of a system failure or other issues. The incident typically requires immediate attention to ensure that the data is properly backed up and that any potential issues are resolved quickly to avoid further complications.


Redis missing backup incident.

```shell
# Environment Variables

export REDIS_HOST="PLACEHOLDER"

export REDIS_PORT="PLACEHOLDER"

export REDIS_INSTANCE_NAME="PLACEHOLDER"

export MASTER_HOSTNAME="PLACEHOLDER"

export MASTER_PORT="PLACEHOLDER"


```


### Check if Redis is running

```shell
systemctl status redis
```

### Check Redis logs for any errors

```shell
tail -n 100 /var/log/redis/redis.log
```

### Check Redis replication status

```shell
redis-cli info replication
```

### Check number of connected slaves

```shell
redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} info clients | grep connected_slaves
```

### Check Redis latency

```shell
redis-cli --latency
```

### Check network connectivity to Redis instance

```shell
ping ${REDIS_HOST}
```

### Check if Redis instance is listening on the correct port

```shell
netstat -tuln | grep ${REDIS_PORT}
```

### Check Redis configuration file for any issues

```shell
cat /etc/redis/redis.conf
```

### Check resource utilization of Redis instance

```shell
top -c -p $(pidof redis-server)
```


### Get the Redis instance name

```shell
redis_instance=${REDIS_INSTANCE_NAME}
```

### Check the replication status

```shell
redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} info replication | grep -E '^(role|connected_slaves|slave[[:digit:]]*):'
```

### Identify the disconnected slaves

```shell
redis-cli -h ${REDIS_HOST} -p ${REDIS_PORT} info replication | grep -E '^(role|connected_slaves|slave[[:digit:]]*):' | grep -v 'role:master' | grep -v 'connected_slaves:0' | grep -v 'slave[[:digit:]]*:ip' | grep -v 'slave[[:digit:]]*:state=online'
```

### Restart the Redis server and monitor the replication status.

```shell
#!/bin/bash

# Stop the Redis server

sudo systemctl stop redis

# Wait for the server to stop

sleep 5

# Start the Redis server

sudo systemctl start redis

# Wait for the server to start

sleep 5

# Monitor the replication status

redis-cli info replication

```

### Manually replicate the data from the master to the disconnected slaves.

```shell
#!/bin/bash

# Set the Redis master hostname and port

REDIS_MASTER=${MASTER_HOSTNAME}:${MASTER_PORT}

# Set the list of Redis slave instances to replicate to

REDIS_SLAVES="PLACEHOLDER"

# Loop through the list of Redis slaves and replicate data from the master

for redis_slave in "${REDIS_SLAVES[@]}"

do

  # Extract the hostname and port from the Redis slave string

  IFS=':' read -ra redis_slave_parts <<< "$redis_slave"

  redis_slave_hostname="${redis_slave_parts[0]}"

  redis_slave_port="${redis_slave_parts[1]}"

  # Replicate data from the Redis master to the slave

  redis-cli -h $redis_slave_hostname -p $redis_slave_port SLAVEOF $REDIS_MASTER

done

echo "Data replication from Redis master to all disconnected slaves completed."

```


Redis disconnected slaves incident

Overview

Parameters

Debug

Check if Redis is running

Check Redis logs for any errors

Check Redis replication status

Check number of connected slaves

Check Redis latency

Check network connectivity to Redis instance

Check if Redis instance is listening on the correct port

Check Redis configuration file for any issues

Check resource utilization of Redis instance

Repair

Get the Redis instance name

Check the replication status

Identify the disconnected slaves

Restart the Redis server and monitor the replication status.

Manually replicate the data from the master to the disconnected slaves.

Learn more

Related Runbooks

Redis too many masters incident

Redis replication broken incident.

Redis Rejected Connections Incident

Redis missing master incident

Support