---
id: 015d8bf8-6da1-4547-8b40-39b0b5e5ebaf
---

# Elasticsearch Cluster Red Incident on Kubernetes.
---

The Elasticsearch Cluster Red incident occurs when the Elasticsearch cluster is experiencing issues and is in a critical state. This can cause interruptions to the functionality of various systems that rely on Elasticsearch. Immediate attention is required to resolve the issue and restore the Elasticsearch cluster to a healthy state.

### Parameters
```shell
export CLUSTER_NAME="PLACEHOLDER"

export ELASTICSEARCH_POD_NAME="PLACEHOLDER"

export ELASTICSEARCH_DEPLOYMENT_NAME="PLACEHOLDER"

export ELASTICSEARCH_NAMESPACE="PLACEHOLDER"

export ELASTICSEARCH_POD_IP="PLACEHOLDER"
```

## Debug

### Get the name of the Elasticsearch cluster
```shell
kubectl get elasticsearch
```

### Describe the Elasticsearch cluster to get more details
```shell
kubectl describe elasticsearch ${CLUSTER_NAME}
```

### Check the Elasticsearch cluster health
```shell
kubectl exec ${ELASTICSEARCH_POD_NAME} --namespace=${ELASTICSEARCH_NAMESPACE} -- curl -s -X GET http://localhost:9200/_cluster/health
```

### Check the Elasticsearch cluster status
```shell
kubectl exec ${ELASTICSEARCH_POD_NAME} --namespace=${ELASTICSEARCH_NAMESPACE} -- curl -s -X GET http://localhost:9200/_cat/health
```

### Check the Elasticsearch cluster logs
```shell
kubectl logs ${ELASTICSEARCH_POD_NAME} --namespace=${ELASTICSEARCH_NAMESPACE}
```

### Check the Elasticsearch cluster configuration for any errors
```shell
kubectl exec ${ELASTICSEARCH_POD_NAME} --namespace=${ELASTICSEARCH_NAMESPACE} -- cat /usr/share/elasticsearch/config/elasticsearch.yml
```

### Check the Elasticsearch cluster nodes
```shell
kubectl exec ${ELASTICSEARCH_POD_NAME} --namespace=${ELASTICSEARCH_NAMESPACE} -- curl -s -X GET http://localhost:9200/_cat/nodes?v
```

### Network issues: Elasticsearch cluster red status can also be caused by network issues. This can occur when there are network connectivity issues between the nodes in the cluster or when the network is not properly configured.
```shell
bash

# Check if Elasticsearch pods are running

es_pod=$(kubectl get pods -l app=elasticsearch -o jsonpath='{.items[0].metadata.name}')

if [ -z "$es_pod" ]; then

  echo "Error: Elasticsearch pods not found"

  exit 1

fi



# Check network connectivity between Elasticsearch pods

if ! kubectl exec "$es_pod" -- curl -sS -o /dev/null -w "%{http_code}" http://${ELASTICSEARCH_POD_IP}:9200/_cat/health; then

  echo "Error: Elasticsearch network connectivity issues"

  exit 1

fi



echo "Network connectivity between Elasticsearch nodes is healthy"


```

## Repair

### Restart Elasticsearch nodes: If the issue is minor and isolated to a few nodes, restarting the nodes can help resolve the issue. However, if the issue is more severe, restarting the entire cluster may be necessary.
```shell


#!/bin/bash



# Set the namespace and Elasticsearch deployment name

NAMESPACE=${ELASTICSEARCH_NAMESPACE}

DEPLOYMENT=${ELASTICSEARCH_DEPLOYMENT_NAME}



# Use rolling restart to restart pods one by one
kubectl rollout restart deployment/${DEPLOYMENT} -n ${NAMESPACE}
```

The Elasticsearch Cluster Red incident occurs when the Elasticsearch cluster is experiencing issues and is in a critical state. This can cause interruptions to the functionality of various systems that rely on Elasticsearch. Immediate attention is required to resolve the issue and restore the Elasticsearch cluster to a healthy state.


The Vault cluster health incident is related to the health of a Vault cluster instance. This incident type is triggered when the cluster instance is not healthy and requires attention to ensure it is functioning properly. The incident typically involves evaluating the current state of the cluster instance, diagnosing the issue, and taking corrective action to restore the health of the instance.


Vault cluster health incident on kubernetes

The incident type of "Kubernetes deployment with multiple restarts" indicates that a Kubernetes deployment has experienced multiple restarts within a certain timeframe, which is usually indicative of a problem. Kubernetes is a popular container orchestration platform that automates the deployment, scaling, and management of containerized applications. When a deployment experiences multiple restarts, it can impact the availability and performance of the application, and can be a sign of underlying issues that need to be addressed. This incident type is typically monitored and managed by DevOps teams responsible for ensuring the health and reliability of Kubernetes-based applications.


Kubernetes deployment with multiple restarts

This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

A Kubernetes Replicaset Incomplete incident typically occurs when a specific number of pods that should be running are not, due to reasons such as failed pod initialization, unavailability of resources in the cluster, or inability to pull the image. This incident is usually triggered when the difference between desired and running pods is greater than zero, and it can be detected through monitoring tools like Datadog.


Kubernetes Replicaset Incomplete

This incident type occurs when the Kubernetes node status is not OK. It means that the scheduler cannot place pods on the node due to some underlying issue with the node's health. This incident can impact the availability and performance of the applications running on the Kubernetes cluster. Immediate attention is required to resolve this incident to restore the normal functioning of the Kubernetes cluster.


Kubernetes Node Status Not OK

```shell
export CLUSTER_NAME="PLACEHOLDER"

export ELASTICSEARCH_POD_NAME="PLACEHOLDER"

export ELASTICSEARCH_DEPLOYMENT_NAME="PLACEHOLDER"

export ELASTICSEARCH_NAMESPACE="PLACEHOLDER"

export ELASTICSEARCH_POD_IP="PLACEHOLDER"
```


### Get the name of the Elasticsearch cluster

```shell
kubectl get elasticsearch
```

### Describe the Elasticsearch cluster to get more details

```shell
kubectl describe elasticsearch ${CLUSTER_NAME}
```

### Check the Elasticsearch cluster health

```shell
kubectl exec ${ELASTICSEARCH_POD_NAME} --namespace=${ELASTICSEARCH_NAMESPACE} -- curl -s -X GET http://localhost:9200/_cluster/health
```

### Check the Elasticsearch cluster status

```shell
kubectl exec ${ELASTICSEARCH_POD_NAME} --namespace=${ELASTICSEARCH_NAMESPACE} -- curl -s -X GET http://localhost:9200/_cat/health
```

### Check the Elasticsearch cluster logs

```shell
kubectl logs ${ELASTICSEARCH_POD_NAME} --namespace=${ELASTICSEARCH_NAMESPACE}
```

### Check the Elasticsearch cluster configuration for any errors

```shell
kubectl exec ${ELASTICSEARCH_POD_NAME} --namespace=${ELASTICSEARCH_NAMESPACE} -- cat /usr/share/elasticsearch/config/elasticsearch.yml
```

### Check the Elasticsearch cluster nodes

```shell
kubectl exec ${ELASTICSEARCH_POD_NAME} --namespace=${ELASTICSEARCH_NAMESPACE} -- curl -s -X GET http://localhost:9200/_cat/nodes?v
```

### Network issues: Elasticsearch cluster red status can also be caused by network issues. This can occur when there are network connectivity issues between the nodes in the cluster or when the network is not properly configured.

```shell
bash

# Check if Elasticsearch pods are running

es_pod=$(kubectl get pods -l app=elasticsearch -o jsonpath='{.items[0].metadata.name}')

if [ -z "$es_pod" ]; then

  echo "Error: Elasticsearch pods not found"

  exit 1

fi



# Check network connectivity between Elasticsearch pods

if ! kubectl exec "$es_pod" -- curl -sS -o /dev/null -w "%{http_code}" http://${ELASTICSEARCH_POD_IP}:9200/_cat/health; then

  echo "Error: Elasticsearch network connectivity issues"

  exit 1

fi



echo "Network connectivity between Elasticsearch nodes is healthy"


```


### Restart Elasticsearch nodes: If the issue is minor and isolated to a few nodes, restarting the nodes can help resolve the issue. However, if the issue is more severe, restarting the entire cluster may be necessary.

```shell


#!/bin/bash



# Set the namespace and Elasticsearch deployment name

NAMESPACE=${ELASTICSEARCH_NAMESPACE}

DEPLOYMENT=${ELASTICSEARCH_DEPLOYMENT_NAME}



# Use rolling restart to restart pods one by one
kubectl rollout restart deployment/${DEPLOYMENT} -n ${NAMESPACE}
```


Elasticsearch Cluster Red Incident on Kubernetes.

Overview

Parameters

Debug

Get the name of the Elasticsearch cluster

Describe the Elasticsearch cluster to get more details

Check the Elasticsearch cluster health

Check the Elasticsearch cluster status

Check the Elasticsearch cluster logs

Check the Elasticsearch cluster configuration for any errors

Check the Elasticsearch cluster nodes

Network issues: Elasticsearch cluster red status can also be caused by network issues. This can occur when there are network connectivity issues between the nodes in the cluster or when the network is not properly configured.

Repair

Restart Elasticsearch nodes: If the issue is minor and isolated to a few nodes, restarting the nodes can help resolve the issue. However, if the issue is more severe, restarting the entire cluster may be necessary.

Learn more

Related Runbooks

Vault cluster health incident on kubernetes

Kubernetes deployment with multiple restarts

Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Replicaset Incomplete

Support