---
id: 7cfeb622-dfd6-4106-b0d6-3197782ec484
---

# Etcd cluster no leader incident
---

The Etcd cluster no leader incident refers to an event where the Etcd cluster has no leader, which can cause issues with the system's performance and functionality. This incident type requires immediate attention and resolution to avoid any further impact on the system.

### Parameters
```shell
export ETCD_CERT_FILE="PLACEHOLDER"

export ETCD_ENDPOINT="PLACEHOLDER"

export ETCD_CA_FILE="PLACEHOLDER"

export ETCD_KEY_FILE="PLACEHOLDER"

export NODE_NAME="PLACEHOLDER"
```

## Debug

### Check if the Etcd service is running
```shell
systemctl status etcd.service
```

### Check the logs of the Etcd service
```shell
journalctl -u etcd.service
```

### Check the status of the Etcd cluster
```shell
etcdctl cluster-health
```

### Check the status of the Etcd members
```shell
etcdctl member list
```

### Check the leader of the Etcd cluster
```shell
etcdctl endpoint status --cluster | grep leader
```

### Check the connectivity between the Etcd members
```shell
etcdctl endpoint health --cluster
```

### Check the version of the Etcd cluster
```shell
etcdctl version
```

## Repair

### Check the Etcd cluster for any system failures or network connectivity issues that may have caused the cluster to lose its leader.
```shell


#!/bin/bash



# Check the Etcd cluster for any system failures or network connectivity issues that may have caused the cluster to lose its leader



# Replace ${ETCD_ENDPOINT} with the endpoint of the Etcd cluster

ETCD_ENDPOINT=${ETCD_ENDPOINT}



# Replace ${ETCD_CERT_FILE} with the path to the certificate file for Etcd

ETCD_CERT_FILE=${ETCD_CERT_FILE}



# Replace ${ETCD_KEY_FILE} with the path to the key file for Etcd

ETCD_KEY_FILE=${ETCD_KEY_FILE}



# Replace ${ETCD_CA_FILE} with the path to the CA file for Etcd

ETCD_CA_FILE=${ETCD_CA_FILE}



# Run the Etcd health check command

etcdctl --endpoints=$ETCD_ENDPOINT --cert=$ETCD_CERT_FILE --key=$ETCD_KEY_FILE --cacert=$ETCD_CA_FILE endpoint health


```

### Restart the Etcd service on the affected node to allow the system to elect a new leader.
```shell


#!/bin/bash



# Define the affected node

NODE=${NODE_NAME}



# Restart the Etcd service

systemctl restart etcd



# Wait for the service to restart

sleep 5



# Check the status of the service and log the result

STATUS=$(systemctl status etcd)

echo "$STATUS" >> etcd_restart.log


```

The Etcd cluster no leader incident refers to an event where the Etcd cluster has no leader, which can cause issues with the system's performance and functionality. This incident type requires immediate attention and resolution to avoid any further impact on the system.


The Redis too many masters incident occurs when there are too many master nodes in a Redis cluster, leading to connection issues and potential data loss. This can happen due to misconfiguration, network issues, or other factors, and requires immediate attention to prevent further damage.


Redis too many masters incident

This incident type refers to an issue with Redis replication, which means that there is a problem with the synchronization of data between Redis instances. This issue could impact the availability and performance of the system and may require immediate attention to restore the replication and ensure data consistency. The incident could be caused by various factors, such as network problems, hardware failures, or configuration issues. The incident must be investigated and resolved as soon as possible to avoid any data loss or downtime.


Redis replication broken incident.

The Redis missing master incident occurs when the Redis cluster has no node marked as master. This can cause problems with the Redis service and may result in service disruptions or failures. It is important to address this issue promptly to ensure the smooth operation of the Redis service.


Redis missing master incident

The Redis missing backup incident refers to a situation where Redis, a popular in-memory data structure store, has not been backed up for a certain period of time. This could lead to data loss or corruption in the event of a system failure or other issues. The incident typically requires immediate attention to ensure that the data is properly backed up and that any potential issues are resolved quickly to avoid further complications.


Redis missing backup incident.

A Redis instance down incident refers to a situation where the Redis server, which is a popular open-source in-memory data structure store, is not functioning or accessible. This can cause interruption in the service and impact the performance of applications that rely on Redis for caching or data storage. Typically, this type of incident is considered high urgency and requires immediate attention from the responsible team to investigate, diagnose, and resolve the issue.


Redis instance down incident

```shell
export ETCD_CERT_FILE="PLACEHOLDER"

export ETCD_ENDPOINT="PLACEHOLDER"

export ETCD_CA_FILE="PLACEHOLDER"

export ETCD_KEY_FILE="PLACEHOLDER"

export NODE_NAME="PLACEHOLDER"
```


### Check if the Etcd service is running

```shell
systemctl status etcd.service
```

### Check the logs of the Etcd service

```shell
journalctl -u etcd.service
```

### Check the status of the Etcd cluster

```shell
etcdctl cluster-health
```

### Check the status of the Etcd members

```shell
etcdctl member list
```

### Check the leader of the Etcd cluster

```shell
etcdctl endpoint status --cluster | grep leader
```

### Check the connectivity between the Etcd members

```shell
etcdctl endpoint health --cluster
```

### Check the version of the Etcd cluster

```shell
etcdctl version
```


### Check the Etcd cluster for any system failures or network connectivity issues that may have caused the cluster to lose its leader.

```shell


#!/bin/bash



# Check the Etcd cluster for any system failures or network connectivity issues that may have caused the cluster to lose its leader



# Replace ${ETCD_ENDPOINT} with the endpoint of the Etcd cluster

ETCD_ENDPOINT=${ETCD_ENDPOINT}



# Replace ${ETCD_CERT_FILE} with the path to the certificate file for Etcd

ETCD_CERT_FILE=${ETCD_CERT_FILE}



# Replace ${ETCD_KEY_FILE} with the path to the key file for Etcd

ETCD_KEY_FILE=${ETCD_KEY_FILE}



# Replace ${ETCD_CA_FILE} with the path to the CA file for Etcd

ETCD_CA_FILE=${ETCD_CA_FILE}



# Run the Etcd health check command

etcdctl --endpoints=$ETCD_ENDPOINT --cert=$ETCD_CERT_FILE --key=$ETCD_KEY_FILE --cacert=$ETCD_CA_FILE endpoint health


```

### Restart the Etcd service on the affected node to allow the system to elect a new leader.

```shell


#!/bin/bash



# Define the affected node

NODE=${NODE_NAME}



# Restart the Etcd service

systemctl restart etcd



# Wait for the service to restart

sleep 5



# Check the status of the service and log the result

STATUS=$(systemctl status etcd)

echo "$STATUS" >> etcd_restart.log


```


Etcd cluster no leader incident

Overview

Parameters

Debug

Check if the Etcd service is running

Check the logs of the Etcd service

Check the status of the Etcd cluster

Check the status of the Etcd members

Check the leader of the Etcd cluster

Check the connectivity between the Etcd members

Check the version of the Etcd cluster

Repair

Check the Etcd cluster for any system failures or network connectivity issues that may have caused the cluster to lose its leader.

Restart the Etcd service on the affected node to allow the system to elect a new leader.

Learn more

Related Runbooks

Redis too many masters incident

Redis replication broken incident.

Redis missing master incident

Redis missing backup incident.

Support