---
id: 6108442e-4401-4ca3-85ec-ff8f6014803b
---

# Elasticsearch unassigned shards incident on Kubernetes
---

This incident type refers to a situation where Elasticsearch has unassigned shards. Shards are the basic building blocks of Elasticsearch indexes, and they contain the actual data that is stored in Elasticsearch. When a shard is unassigned, it means that Elasticsearch cannot find a node in the cluster that can hold the shard replicas. This can cause data loss and reduce the performance of the Elasticsearch cluster. Therefore, it is important to address this incident as soon as possible to prevent any potential data loss and maintain the optimal performance of the Elasticsearch cluster.

### Parameters
```shell
export ELASTICSEARCH_NAMESPACE="PLACEHOLDER"

export ELASTICSEARCH_APP="PLACEHOLDER"

export ELASTICSEARCH_SERVICE="PLACEHOLDER"

export ELASTICSEARCH_DEPLOYMENT="PLACEHOLDER"

export ELASTICSEARCH_POD_NAME="PLACEHOLDER"

export NUMBER_OF_REPLICAS="PLACEHOLDER"
```

## Debug

### Check if Elasticsearch pods are running
```shell
kubectl get pods -n ${ELASTICSEARCH_NAMESPACE} -l app=${ELASTICSEARCH_APP}
```

### Check if Elasticsearch service is running
```shell
kubectl get svc -n ${ELASTICSEARCH_NAMESPACE} ${ELASTICSEARCH_SERVICE}
```

### Check if Elasticsearch deployment is running
```shell
kubectl get deployment -n ${ELASTICSEARCH_NAMESPACE} ${ELASTICSEARCH_DEPLOYMENT}
```

### Check if Elasticsearch nodes are healthy
```shell
kubectl exec -it ${ELASTICSEARCH_POD_NAME} -n ${ELASTICSEARCH_NAMESPACE} curl -XGET 'http://localhost:9200/_cat/nodes?v'
```

### Check if Elasticsearch shards are assigned
```shell
kubectl exec -it ${ELASTICSEARCH_POD_NAME} -n ${ELASTICSEARCH_NAMESPACE} curl -XGET 'http://localhost:9200/_cat/shards?v'
```

### Check if Kubernetes nodes have enough resources
```shell
kubectl get nodes --show-labels
```

### Check if there are any pod evictions or disruptions
```shell
kubectl get events -n ${ELASTICSEARCH_NAMESPACE}
```

### Check if there are any Elasticsearch logs related to unassigned shards
```shell
kubectl logs -f ${ELASTICSEARCH_POD_NAME} -n ${ELASTICSEARCH_NAMESPACE} | grep "unassigned"
```

### The Elasticsearch cluster may have insufficient resources such as CPU, memory, or storage space, leading to unassigned shards.
```shell


#!/bin/bash



# Set the Elasticsearch namespace and pod name

NAMESPACE="${ELASTICSEARCH_NAMESPACE}"

POD_NAME="${ELASTICSEARCH_POD_NAME}"



# Check the CPU usage of the Elasticsearch pod

CPU_USAGE=$(kubectl exec -n $NAMESPACE $POD_NAME -- sh -c "ps -p 1 -o %cpu | tail -1")

if (( $(echo "$CPU_USAGE > 80.0" | bc -l) )); then

  echo "WARNING: CPU usage of Elasticsearch pod is high: $CPU_USAGE%"

fi



# Check the memory usage of the Elasticsearch pod

MEMORY_USAGE=$(kubectl exec -n $NAMESPACE $POD_NAME -- sh -c "free -m | awk 'NR==2{printf \"%s/%sMB (%.2f%%)\", $3,$2,$3*100/$2 }'")

if (( $(echo "$MEMORY_USAGE" | awk -F'[(%)]' '{print $2}' | awk '{print int($1)}') > 80 )); then

  echo "WARNING: Memory usage of Elasticsearch pod is high: $MEMORY_USAGE"

fi



# Check the storage space usage of the Elasticsearch pod

STORAGE_USAGE=$(kubectl exec -n $NAMESPACE $POD_NAME -- sh -c "df -h /usr/share/elasticsearch/data | awk '{print $5}' | tail -1 | sed 's/%//g'")

if (( $STORAGE_USAGE > 80 )); then

  echo "WARNING: Storage space usage of Elasticsearch pod is high: $STORAGE_USAGE%"

fi


```

## Repair

### Scale the Elasticsearch deployment to add more pods
```shell


#!/bin/bash



# Set the namespace and deployment name

NAMESPACE=${ELACTICSEARCH_NAMESPACE}

DEPLOYMENT_NAME=${ELASTICSEARCH_DEPLOYMENT}



# Set the desired number of replicas

REPLICAS=${NUMBER_OF_REPLICAS}



# Scale the deployment

kubectl scale deployment -n $NAMESPACE $DEPLOYMENT_NAME --replicas=$REPLICAS


```

This incident type refers to a situation where Elasticsearch has unassigned shards. Shards are the basic building blocks of Elasticsearch indexes, and they contain the actual data that is stored in Elasticsearch. When a shard is unassigned, it means that Elasticsearch cannot find a node in the cluster that can hold the shard replicas. This can cause data loss and reduce the performance of the Elasticsearch cluster. Therefore, it is important to address this incident as soon as possible to prevent any potential data loss and maintain the optimal performance of the Elasticsearch cluster.


This incident type is characterized by the detection of unauthorized access to the Kubernetes API server. This unauthorized access potentially enables attackers to manipulate cluster resources. Such incidents can lead to serious consequences, as unauthorized access allows attackers to make changes to the Kubernetes cluster, which may compromise the security and integrity of the entire system. Therefore, it is crucial to promptly detect and address such incidents to ensure the security and proper functioning of the Kubernetes cluster.


Unauthorized Access to Kubernetes API Server Detected

This incident type involves an issue with Kubernetes deployments where the expected number of pods to run is not matching the actual number of pods running. This can lead to alerts being triggered and potential disruptions in the system.


Kubernetes pods not starting - Deployment issue

This incident type involves nodes in a Kubernetes cluster that are experiencing network unavailability, meaning they are not accessible. This could be due to a misconfiguration, route exhaustion, or a physical problem with the network connection to the hardware. It is a high urgency incident that requires immediate attention to restore network connectivity to the affected nodes.


Kubernetes Nodes with Network Unavailable

This incident type relates to the monitoring of Kubernetes deployments replica pods. It implies that there is an issue with the number of replica pods available as compared to the desired number. The incident might be triggered by a query alert monitor and might require immediate action to resolve the issue. The incident could impact the deployment of applications hosted on Kubernetes and might require troubleshooting and fixing the underlying issue.


Kubernetes Deployments Replica Pods Monitoring Incident

This incident type occurs when the pods in a Kubernetes cluster fail to schedule, which prevents the application from scaling. Kubernetes is a container orchestration platform that automates the deployment, scaling, and management of containerized applications. Pods in Kubernetes are the smallest deployable units that can be created and managed. When pods do not schedule, it means that the Kubernetes scheduler is unable to find a node with enough resources to run the pod. This can cause issues with application performance and scalability.


Kubernetes - Pods not scheduled preventing application scaling.

```shell
export ELASTICSEARCH_NAMESPACE="PLACEHOLDER"

export ELASTICSEARCH_APP="PLACEHOLDER"

export ELASTICSEARCH_SERVICE="PLACEHOLDER"

export ELASTICSEARCH_DEPLOYMENT="PLACEHOLDER"

export ELASTICSEARCH_POD_NAME="PLACEHOLDER"

export NUMBER_OF_REPLICAS="PLACEHOLDER"
```


### Check if Elasticsearch pods are running

```shell
kubectl get pods -n ${ELASTICSEARCH_NAMESPACE} -l app=${ELASTICSEARCH_APP}
```

### Check if Elasticsearch service is running

```shell
kubectl get svc -n ${ELASTICSEARCH_NAMESPACE} ${ELASTICSEARCH_SERVICE}
```

### Check if Elasticsearch deployment is running

```shell
kubectl get deployment -n ${ELASTICSEARCH_NAMESPACE} ${ELASTICSEARCH_DEPLOYMENT}
```

### Check if Elasticsearch nodes are healthy

```shell
kubectl exec -it ${ELASTICSEARCH_POD_NAME} -n ${ELASTICSEARCH_NAMESPACE} curl -XGET 'http://localhost:9200/_cat/nodes?v'
```

### Check if Elasticsearch shards are assigned

```shell
kubectl exec -it ${ELASTICSEARCH_POD_NAME} -n ${ELASTICSEARCH_NAMESPACE} curl -XGET 'http://localhost:9200/_cat/shards?v'
```

### Check if Kubernetes nodes have enough resources

```shell
kubectl get nodes --show-labels
```

### Check if there are any pod evictions or disruptions

```shell
kubectl get events -n ${ELASTICSEARCH_NAMESPACE}
```

### Check if there are any Elasticsearch logs related to unassigned shards

```shell
kubectl logs -f ${ELASTICSEARCH_POD_NAME} -n ${ELASTICSEARCH_NAMESPACE} | grep "unassigned"
```

### The Elasticsearch cluster may have insufficient resources such as CPU, memory, or storage space, leading to unassigned shards.

```shell


#!/bin/bash



# Set the Elasticsearch namespace and pod name

NAMESPACE="${ELASTICSEARCH_NAMESPACE}"

POD_NAME="${ELASTICSEARCH_POD_NAME}"



# Check the CPU usage of the Elasticsearch pod

CPU_USAGE=$(kubectl exec -n $NAMESPACE $POD_NAME -- sh -c "ps -p 1 -o %cpu | tail -1")

if (( $(echo "$CPU_USAGE > 80.0" | bc -l) )); then

  echo "WARNING: CPU usage of Elasticsearch pod is high: $CPU_USAGE%"

fi



# Check the memory usage of the Elasticsearch pod

MEMORY_USAGE=$(kubectl exec -n $NAMESPACE $POD_NAME -- sh -c "free -m | awk 'NR==2{printf \"%s/%sMB (%.2f%%)\", $3,$2,$3*100/$2 }'")

if (( $(echo "$MEMORY_USAGE" | awk -F'[(%)]' '{print $2}' | awk '{print int($1)}') > 80 )); then

  echo "WARNING: Memory usage of Elasticsearch pod is high: $MEMORY_USAGE"

fi



# Check the storage space usage of the Elasticsearch pod

STORAGE_USAGE=$(kubectl exec -n $NAMESPACE $POD_NAME -- sh -c "df -h /usr/share/elasticsearch/data | awk '{print $5}' | tail -1 | sed 's/%//g'")

if (( $STORAGE_USAGE > 80 )); then

  echo "WARNING: Storage space usage of Elasticsearch pod is high: $STORAGE_USAGE%"

fi


```


### Scale the Elasticsearch deployment to add more pods

```shell


#!/bin/bash



# Set the namespace and deployment name

NAMESPACE=${ELACTICSEARCH_NAMESPACE}

DEPLOYMENT_NAME=${ELASTICSEARCH_DEPLOYMENT}



# Set the desired number of replicas

REPLICAS=${NUMBER_OF_REPLICAS}



# Scale the deployment

kubectl scale deployment -n $NAMESPACE $DEPLOYMENT_NAME --replicas=$REPLICAS


```


Elasticsearch unassigned shards incident on Kubernetes

Overview

Parameters

Debug

Check if Elasticsearch pods are running

Check if Elasticsearch service is running

Check if Elasticsearch deployment is running

Check if Elasticsearch nodes are healthy

Check if Elasticsearch shards are assigned

Check if Kubernetes nodes have enough resources

Check if there are any pod evictions or disruptions

The Elasticsearch cluster may have insufficient resources such as CPU, memory, or storage space, leading to unassigned shards.

Repair

Scale the Elasticsearch deployment to add more pods

Learn more

Related Runbooks

Unauthorized Access to Kubernetes API Server Detected

Kubernetes pods not starting - Deployment issue

Kubernetes Nodes with Network Unavailable

Kubernetes Deployments Replica Pods Monitoring Incident

Support