---
id: ca7578b1-2d53-43f4-9666-bb68db5acc75
---

# Etcd member communication slow incident.
---

This incident type refers to an issue where communication between Etcd members is slowing down, resulting in a decrease in the performance of the Etcd system. The incident is triggered when the 99th percentile of communication time exceeds 0.15 seconds. This type of incident can impact the functionality and stability of the Etcd system, and requires immediate attention to restore normal operation.

### Parameters
```shell
export ETCD_ENDPOINTS="PLACEHOLDER"

export NETWORK_INTERFACE="PLACEHOLDER"

export ETCD_MEMBER_IP="PLACEHOLDER"

export ETCD_MEMBERS_LIST="PLACEHOLDER"

export ETCD_PORT="PLACEHOLDER"

export NEW_NODE_MEMORY="PLACEHOLDER"

export NEW_NODE_COUNT="PLACEHOLDER"

export NEW_NODE_CPU="PLACEHOLDER"
```

## Debug

### Check if the Etcd service is running
```shell
systemctl status etcd
```

### Check the logs for the Etcd service
```shell
journalctl -u etcd
```

### Check Etcd cluster health
```shell
etcdctl cluster-health
```

### Check the health of each Etcd member
```shell
etcdctl endpoint health
```

### Check the network latency between Etcd members
```shell
etcdctl --endpoints=${ETCD_ENDPOINTS} endpoint status --write-out=table
```

### Check the CPU and memory usage of Etcd processes
```shell
ps aux | grep etcd
```

### Check the network traffic between Etcd members
```shell
tcpdump -i ${NETWORK_INTERFACE} port 2379
```

### Check the network bandwidth between the Etcd members
```shell
iperf -c ${ETCD_MEMBER_IP}
```

### Check the firewall rules for Etcd ports
```shell
iptables -L | grep 2379
```

### Check the configuration file for Etcd
```shell
cat /etc/etcd/etcd.conf
```

### High network traffic between etcd cluster members.
```shell


#!/bin/bash



# Define variables

ETCD_MEMBERS=${ETCD_MEMBERS_LIST}

ETCD_PORT=${ETCD_PORT}



# Check network traffic

for member in ${ETCD_MEMBERS}; do

    traffic=$(netstat -an | grep ${member}:${ETCD_PORT} | awk '{print $2}')



    if [ -n "${traffic}" ]; then

        echo "High network traffic detected on ${member}:${ETCD_PORT} (${traffic} connections)"

    else

        echo "No network traffic detected on ${member}:${ETCD_PORT}"

    fi

done


```

## Repair

### Increase the resources allocated to the Etcd cluster by adding more nodes or increasing the CPU and memory on the existing nodes.
```shell
#!/bin/bash



# Define variables

NEW_NODE_COUNT=${NEW_NODE_COUNT}

NEW_NODE_CPU=${NEW_NODE_CPU}

NEW_NODE_MEMORY=${NEW_NODE_MEMORY}



# Add new nodes

for ((i=1; i<=$NEW_NODE_COUNT; i++))

do

  # Run etcdctl command to add new node

  etcdctl member add node$i --peer-urls=http://node$i:2380

done



# Update CPU and memory on existing nodes

for NODE in $(etcdctl member list | awk '{print $2}')

do

  # Run etcdctl command to update CPU and memory for node

  etcdctl member update $NODE --peer-urls=http://$NODE:2380 --client-urls=http://$NODE:2379 --node-cpu=$NEW_NODE_CPU --node-memory=$NEW_NODE_MEMORY

done



echo "Etcd cluster resources have been increased."


```

This incident type refers to an issue where communication between Etcd members is slowing down, resulting in a decrease in the performance of the Etcd system. The incident is triggered when the 99th percentile of communication time exceeds 0.15 seconds. This type of incident can impact the functionality and stability of the Etcd system, and requires immediate attention to restore normal operation.


The Redis too many masters incident occurs when there are too many master nodes in a Redis cluster, leading to connection issues and potential data loss. This can happen due to misconfiguration, network issues, or other factors, and requires immediate attention to prevent further damage.


Redis too many masters incident

This incident type refers to an issue with Redis replication, which means that there is a problem with the synchronization of data between Redis instances. This issue could impact the availability and performance of the system and may require immediate attention to restore the replication and ensure data consistency. The incident could be caused by various factors, such as network problems, hardware failures, or configuration issues. The incident must be investigated and resolved as soon as possible to avoid any data loss or downtime.


Redis replication broken incident.

This incident type refers to a situation where Redis, a popular in-memory data structure store, has rejected connections. This could happen due to various reasons such as server overload, network issues, or misconfiguration. When this occurs, it can cause disruptions in the performance of applications that rely on Redis for data storage or caching. Thus, it requires immediate attention and resolution to minimize the impact on the applications and users.


Redis Rejected Connections Incident

A Redis instance down incident refers to a situation where the Redis server, which is a popular open-source in-memory data structure store, is not functioning or accessible. This can cause interruption in the service and impact the performance of applications that rely on Redis for caching or data storage. Typically, this type of incident is considered high urgency and requires immediate attention from the responsible team to investigate, diagnose, and resolve the issue.


Redis instance down incident

This incident type refers to an issue with Redis where one or more slave instances have become disconnected, resulting in replication failure. This can cause data inconsistencies and may require immediate attention to restore normal functioning. The incident may be caused by a variety of factors, such as network issues, server failures, or misconfiguration.


Redis disconnected slaves incident

```shell
export ETCD_ENDPOINTS="PLACEHOLDER"

export NETWORK_INTERFACE="PLACEHOLDER"

export ETCD_MEMBER_IP="PLACEHOLDER"

export ETCD_MEMBERS_LIST="PLACEHOLDER"

export ETCD_PORT="PLACEHOLDER"

export NEW_NODE_MEMORY="PLACEHOLDER"

export NEW_NODE_COUNT="PLACEHOLDER"

export NEW_NODE_CPU="PLACEHOLDER"
```


### Check if the Etcd service is running

```shell
systemctl status etcd
```

### Check the logs for the Etcd service

```shell
journalctl -u etcd
```

### Check Etcd cluster health

```shell
etcdctl cluster-health
```

### Check the health of each Etcd member

```shell
etcdctl endpoint health
```

### Check the network latency between Etcd members

```shell
etcdctl --endpoints=${ETCD_ENDPOINTS} endpoint status --write-out=table
```

### Check the CPU and memory usage of Etcd processes

```shell
ps aux | grep etcd
```

### Check the network traffic between Etcd members

```shell
tcpdump -i ${NETWORK_INTERFACE} port 2379
```

### Check the network bandwidth between the Etcd members

```shell
iperf -c ${ETCD_MEMBER_IP}
```

### Check the firewall rules for Etcd ports

```shell
iptables -L | grep 2379
```

### Check the configuration file for Etcd

```shell
cat /etc/etcd/etcd.conf
```

### High network traffic between etcd cluster members.

```shell


#!/bin/bash



# Define variables

ETCD_MEMBERS=${ETCD_MEMBERS_LIST}

ETCD_PORT=${ETCD_PORT}



# Check network traffic

for member in ${ETCD_MEMBERS}; do

    traffic=$(netstat -an | grep ${member}:${ETCD_PORT} | awk '{print $2}')



    if [ -n "${traffic}" ]; then

        echo "High network traffic detected on ${member}:${ETCD_PORT} (${traffic} connections)"

    else

        echo "No network traffic detected on ${member}:${ETCD_PORT}"

    fi

done


```


### Increase the resources allocated to the Etcd cluster by adding more nodes or increasing the CPU and memory on the existing nodes.

```shell
#!/bin/bash



# Define variables

NEW_NODE_COUNT=${NEW_NODE_COUNT}

NEW_NODE_CPU=${NEW_NODE_CPU}

NEW_NODE_MEMORY=${NEW_NODE_MEMORY}



# Add new nodes

for ((i=1; i<=$NEW_NODE_COUNT; i++))

do

  # Run etcdctl command to add new node

  etcdctl member add node$i --peer-urls=http://node$i:2380

done



# Update CPU and memory on existing nodes

for NODE in $(etcdctl member list | awk '{print $2}')

do

  # Run etcdctl command to update CPU and memory for node

  etcdctl member update $NODE --peer-urls=http://$NODE:2380 --client-urls=http://$NODE:2379 --node-cpu=$NEW_NODE_CPU --node-memory=$NEW_NODE_MEMORY

done



echo "Etcd cluster resources have been increased."


```


Etcd member communication slow incident.

Overview

Parameters

Debug

Check if the Etcd service is running

Check the logs for the Etcd service

Check Etcd cluster health

Check the health of each Etcd member

Check the network latency between Etcd members

Check the CPU and memory usage of Etcd processes

Check the network traffic between Etcd members

Check the network bandwidth between the Etcd members

Check the firewall rules for Etcd ports

Check the configuration file for Etcd

High network traffic between etcd cluster members.

Repair

Increase the resources allocated to the Etcd cluster by adding more nodes or increasing the CPU and memory on the existing nodes.

Learn more

Related Runbooks

Redis too many masters incident

Redis replication broken incident.

Redis Rejected Connections Incident

Redis instance down incident

Support