---
id: 82c14073-0b28-4d75-887e-bd86e852b9f8
---

# Etcd insufficient Members incident.
---

The Etcd insufficient Members incident type refers to an issue where the Etcd cluster has an insufficient number of members. Etcd is a distributed key-value store used for shared configuration and service discovery. In order to maintain high availability and fault tolerance, the cluster should have an odd number of members. When the number of members falls below the minimum required, it can result in service outages and other disruptions. This incident type requires immediate attention to restore the service to normal operation.

### Parameters
```shell
export ETCD_ENDPOINTS="PLACEHOLDER"

export NEW_MEMBER_IP="PLACEHOLDER"

export CPU_THRESHOLD="PLACEHOLDER"

export MEMORY_THRESHOLD="PLACEHOLDER"

export MEMBER_NAME="PLACEHOLDER"

export DISK_THRESHOLD="PLACEHOLDER"

export NEW_MEMBER_NAME="PLACEHOLDER"
```

## Debug

### Check the status of the Etcd cluster
```shell
systemctl status etcd
```

### View the Etcd log to check for any errors or warnings
```shell
journalctl -u etcd
```

### Check the number of Etcd members
```shell
etcdctl member list
```

### Check the health of the Etcd cluster
```shell
etcdctl cluster-health
```

### Check the amount of available disk space on the Etcd nodes
```shell
df -h / | tail -n 1 | awk '{print $5}' | sed 's/%//'
```

### Check the memory usage on the Etcd nodes
```shell
free -m
```

### Check the CPU usage on the Etcd nodes
```shell
top
```

### Verify that the Etcd configuration has an odd number of members
```shell
cat /etc/etcd/etcd.conf | grep ETCD_INITIAL_CLUSTER
```

### Insufficient resources like CPU, memory, or disk space on one or more Etcd members can cause them to become unresponsive or crash, leading to a reduced cluster size.
```shell
bash

#!/bin/bash



# Define variables

ETCD_ENDPOINT=${ETCD_ENDPOINTS}

MEMBER_NAME=${MEMBER_NAME}

CPU_THRESHOLD=${CPU_THRESHOLD}

MEMORY_THRESHOLD=${MEMORY_THRESHOLD}

DISK_THRESHOLD=${DISK_THRESHOLD}



# Get CPU usage

CPU_USAGE=$(ssh $ETCD_ENDPOINT "top -bn1 | grep $MEMBER_NAME | awk '{print $9}'")



# Get memory usage

MEMORY_USAGE=$(ssh $ETCD_ENDPOINT "free -m | grep Mem | awk '{print $3/$2 * 100.0}'")



# Get disk usage

DISK_USAGE=$(ssh $ETCD_ENDPOINT "df -h / | tail -n 1 | awk '{print $5}' | sed 's/%//'")



# Check if any usage is above the threshold

if (( $(echo "$CPU_USAGE > $CPU_THRESHOLD" | bc -l) ))

then

    echo "CPU usage for $MEMBER_NAME on $ETCD_ENDPOINT is above the threshold of $CPU_THRESHOLD%: $CPU_USAGE%"

fi



if (( $(echo "$MEMORY_USAGE > $MEMORY_THRESHOLD" | bc -l) ))

then

    echo "Memory usage for $MEMBER_NAME on $ETCD_ENDPOINT is above the threshold of $MEMORY_THRESHOLD%: $MEMORY_USAGE%"

fi



if (( $(echo "$DISK_USAGE > $DISK_THRESHOLD" | bc -l) ))

then

    echo "Disk usage for $MEMBER_NAME on $ETCD_ENDPOINT is above the threshold of $DISK_THRESHOLD%: $DISK_USAGE%"

fi


```

## Repair

### Increase the number of Etcd members: In order to maintain the required odd number of members, additional Etcd members can be added to the cluster. This can be done manually or through automation.
```shell


#!/bin/bash



# Set variables
MEMBER_NAME=${NEW_MEMBER_NAME}

NEW_MEMBER_IP=${NEW_MEMBER_IP}

ETCD_ENDPOINTS=${ETCD_ENDPOINTS}



# Add new member to cluster

etcdctl --endpoints=$ETCD_ENDPOINTS member add $MEMBER_NAME --peer-urls=http://$NEW_MEMBER_IP:2380



# Verify cluster health

etcdctl --endpoints=$ETCD_ENDPOINTS cluster-health


```

The Etcd insufficient Members incident type refers to an issue where the Etcd cluster has an insufficient number of members. Etcd is a distributed key-value store used for shared configuration and service discovery. In order to maintain high availability and fault tolerance, the cluster should have an odd number of members. When the number of members falls below the minimum required, it can result in service outages and other disruptions. This incident type requires immediate attention to restore the service to normal operation.


The Redis too many masters incident occurs when there are too many master nodes in a Redis cluster, leading to connection issues and potential data loss. This can happen due to misconfiguration, network issues, or other factors, and requires immediate attention to prevent further damage.


Redis too many masters incident

This incident type refers to an issue with Redis replication, which means that there is a problem with the synchronization of data between Redis instances. This issue could impact the availability and performance of the system and may require immediate attention to restore the replication and ensure data consistency. The incident could be caused by various factors, such as network problems, hardware failures, or configuration issues. The incident must be investigated and resolved as soon as possible to avoid any data loss or downtime.


Redis replication broken incident.

The Redis missing master incident occurs when the Redis cluster has no node marked as master. This can cause problems with the Redis service and may result in service disruptions or failures. It is important to address this issue promptly to ensure the smooth operation of the Redis service.


Redis missing master incident

This incident type refers to an issue with Redis where one or more slave instances have become disconnected, resulting in replication failure. This can cause data inconsistencies and may require immediate attention to restore normal functioning. The incident may be caused by a variety of factors, such as network issues, server failures, or misconfiguration.


Redis disconnected slaves incident

This incident type is related to an issue with a Redis instance being down. Redis is an in-memory data structure store used as a database, cache, and message broker. This incident could potentially cause disruption to applications and services that rely on Redis for data storage or caching.


Redis Instance Down

```shell
export ETCD_ENDPOINTS="PLACEHOLDER"

export NEW_MEMBER_IP="PLACEHOLDER"

export CPU_THRESHOLD="PLACEHOLDER"

export MEMORY_THRESHOLD="PLACEHOLDER"

export MEMBER_NAME="PLACEHOLDER"

export DISK_THRESHOLD="PLACEHOLDER"

export NEW_MEMBER_NAME="PLACEHOLDER"
```


### Check the status of the Etcd cluster

```shell
systemctl status etcd
```

### View the Etcd log to check for any errors or warnings

```shell
journalctl -u etcd
```

### Check the number of Etcd members

```shell
etcdctl member list
```

### Check the health of the Etcd cluster

```shell
etcdctl cluster-health
```

### Check the amount of available disk space on the Etcd nodes

```shell
df -h / | tail -n 1 | awk '{print $5}' | sed 's/%//'
```

### Check the memory usage on the Etcd nodes

```shell
free -m
```

### Check the CPU usage on the Etcd nodes

```shell
top
```

### Verify that the Etcd configuration has an odd number of members

```shell
cat /etc/etcd/etcd.conf | grep ETCD_INITIAL_CLUSTER
```

### Insufficient resources like CPU, memory, or disk space on one or more Etcd members can cause them to become unresponsive or crash, leading to a reduced cluster size.

```shell
bash

#!/bin/bash



# Define variables

ETCD_ENDPOINT=${ETCD_ENDPOINTS}

MEMBER_NAME=${MEMBER_NAME}

CPU_THRESHOLD=${CPU_THRESHOLD}

MEMORY_THRESHOLD=${MEMORY_THRESHOLD}

DISK_THRESHOLD=${DISK_THRESHOLD}



# Get CPU usage

CPU_USAGE=$(ssh $ETCD_ENDPOINT "top -bn1 | grep $MEMBER_NAME | awk '{print $9}'")



# Get memory usage

MEMORY_USAGE=$(ssh $ETCD_ENDPOINT "free -m | grep Mem | awk '{print $3/$2 * 100.0}'")



# Get disk usage

DISK_USAGE=$(ssh $ETCD_ENDPOINT "df -h / | tail -n 1 | awk '{print $5}' | sed 's/%//'")



# Check if any usage is above the threshold

if (( $(echo "$CPU_USAGE > $CPU_THRESHOLD" | bc -l) ))

then

    echo "CPU usage for $MEMBER_NAME on $ETCD_ENDPOINT is above the threshold of $CPU_THRESHOLD%: $CPU_USAGE%"

fi



if (( $(echo "$MEMORY_USAGE > $MEMORY_THRESHOLD" | bc -l) ))

then

    echo "Memory usage for $MEMBER_NAME on $ETCD_ENDPOINT is above the threshold of $MEMORY_THRESHOLD%: $MEMORY_USAGE%"

fi



if (( $(echo "$DISK_USAGE > $DISK_THRESHOLD" | bc -l) ))

then

    echo "Disk usage for $MEMBER_NAME on $ETCD_ENDPOINT is above the threshold of $DISK_THRESHOLD%: $DISK_USAGE%"

fi


```


### Increase the number of Etcd members: In order to maintain the required odd number of members, additional Etcd members can be added to the cluster. This can be done manually or through automation.

```shell


#!/bin/bash



# Set variables
MEMBER_NAME=${NEW_MEMBER_NAME}

NEW_MEMBER_IP=${NEW_MEMBER_IP}

ETCD_ENDPOINTS=${ETCD_ENDPOINTS}



# Add new member to cluster

etcdctl --endpoints=$ETCD_ENDPOINTS member add $MEMBER_NAME --peer-urls=http://$NEW_MEMBER_IP:2380



# Verify cluster health

etcdctl --endpoints=$ETCD_ENDPOINTS cluster-health


```


Etcd insufficient Members incident.

Overview

Parameters

Debug

Check the status of the Etcd cluster

View the Etcd log to check for any errors or warnings

Check the number of Etcd members

Check the health of the Etcd cluster

Check the amount of available disk space on the Etcd nodes

Check the memory usage on the Etcd nodes

Check the CPU usage on the Etcd nodes

Verify that the Etcd configuration has an odd number of members

Insufficient resources like CPU, memory, or disk space on one or more Etcd members can cause them to become unresponsive or crash, leading to a reduced cluster size.

Repair

Increase the number of Etcd members: In order to maintain the required odd number of members, additional Etcd members can be added to the cluster. This can be done manually or through automation.

Learn more

Related Runbooks

Redis too many masters incident

Redis replication broken incident.

Redis missing master incident

Redis disconnected slaves incident

Support