---
id: 69568196-4a65-4126-9130-a7adde0d057f
---

# Elasticsearch shards relocation Incident or Kubernetes.
---

Elasticsearch shards relocation is an incident type that occurs when Elasticsearch is moving its data shards from one node to another, which could cause temporary unavailability of data or increase in query latency. This may happen due to various reasons like node failure, hardware maintenance, or rebalancing of data.

### Parameters
```shell
export LABEL_SELECTOR="PLACEHOLDER"

export ELASTICSEARCH_POD_NAME="PLACEHOLDER"

export ELASTICSEARCH_ENDPOINT="PLACEHOLDER"

export ELASTICSEARCH_NAMESPACE="PLACEHOLDER"

export ELASTICSEARCH_DEPLOYMENT_NAME="PLACEHOLDER"
```

## Debug

### Check if the Elasticsearch pod is running
```shell
kubectl get pods -n ${ELASTICSEARCH_NAMESPACE} -l ${LABEL_SELECTOR}
```

### Check the logs of the Elasticsearch pod to see if there are any error messages related to shards relocation
```shell
kubectl logs ${ELASTICSEARCH_POD_NAME} -n ${ELASTICSEARCH_NAMESPACE}
```

### Check if the Elasticsearch cluster is healthy
```shell
kubectl exec -it ${ELASTICSEARCH_POD_NAME} -n ${ELASTICSEARCH_NAMESPACE} curl -X GET "http://localhost:9200/_cluster/health"
```

### Check the status of all Elasticsearch indices
```shell
kubectl exec -it ${ELASTICSEARCH_POD_NAME} -n ${ELASTICSEARCH_NAMESPACE} curl -X GET "http://localhost:9200/_cat/indices"
```

### Check the status of all Elasticsearch shards
```shell
kubectl exec -it ${ELASTICSEARCH_POD_NAME} -n ${ELASTICSEARCH_NAMESPACE} curl -X GET "http://localhost:9200/_cat/shards"
```

### Check the status of the Elasticsearch nodes
```shell
kubectl exec -it ${ELASTICSEARCH_POD_NAME} -n ${ELASTICSEARCH_NAMESPACE} curl -X GET "http://localhost:9200/_cat/nodes"
```

### A node failure or shutdown can trigger Elasticsearch to relocate shards from the affected node to other nodes in the cluster.
```shell


#!/bin/bash



# Set the namespace and deployment name for Elasticsearch

NAMESPACE=${ELASTICSEARCH_NAMESPACE}

DEPLOYMENT=${ELASTICSEARCH_DEPLOYMENT_NAME}



# Get the Elasticsearch pods

PODS=$(kubectl get pods -n $NAMESPACE -l app=$DEPLOYMENT -o jsonpath='{.items[*].metadata.name}')



# Check if any of the pods are not ready

for POD in $PODS; do

  if [[ $(kubectl get pod $POD -n $NAMESPACE -o jsonpath='{.status.phase}') != "Running" ]]; then

    echo "Pod $POD is not running. Elasticsearch may be relocating shards."

    exit 1

  fi

done



# Check if Elasticsearch is relocating shards

if [[ $(kubectl exec $POD -n $NAMESPACE -- curl -s localhost:9200/_cluster/health | jq -r '.relocating_shards') -gt 0 ]]; then

  echo "Elasticsearch is relocating shards. Please check the cluster status for more information."

  exit 1

else

  echo "No issues found with Elasticsearch shards relocation."

fi


```

### Rebalancing of data across the cluster due to node addition or removal can trigger Elasticsearch to relocate shards from one node to another.
```shell
bash

#!/bin/bash



# set the namespace and deployment name

NAMESPACE=${ELASTICSEARCH_NAMESPACE}

DEPLOYMENT=${ELASTICSEARCH_DEPLOYMENT_NAME}



# get the number of replicas in the deployment

REPLICAS=$(kubectl get deployment $DEPLOYMENT -n $NAMESPACE -o jsonpath='{.spec.replicas}')



# get the number of ready replicas in the deployment

READY_REPLICAS=$(kubectl get deployment $DEPLOYMENT -n $NAMESPACE -o jsonpath='{.status.readyReplicas}')



# check if the deployment is in the process of scaling up or down

if [ $READY_REPLICAS -lt $REPLICAS ]; then

    echo "Deployment is scaling up, Elasticsearch shards relocation may occur."

elif [ $READY_REPLICAS -gt $REPLICAS ]; then

    echo "Deployment is scaling down, Elasticsearch shards relocation may occur."

else

    echo "Deployment is not scaling, Elasticsearch shards relocation is unlikely to occur."

fi


```

### Check the Elasticsearch logs to identify the reason for shard relocation and the status of the relocation process.
```shell


#!/bin/bash



# Get the Pod name of the Elasticsearch instance 

POD_NAME=$(kubectl get pods -l ${LABEL_SELECTOR} -o jsonpath="{.items[0].metadata.name}")



# Check the Elasticsearch logs for shard relocation

kubectl logs $POD_NAME | grep "shard relocation"



# Check the status of the shard relocation process

kubectl exec -it $POD_NAME -- curl -XGET ${ELASTICSEARCH_ENDPOINT}/_cat/recovery


```

## Repair

### Rebalance the shards manually or use Elasticsearch's auto-rebalancing feature to redistribute the shards evenly across the cluster nodes.
```shell


#!/bin/bash



# Set the Elasticsearch namespace and deployment name

NAMESPACE=${ELASTICSEARCH_NAMESPACE}

DEPLOYMENT=${ELASTICSEARCH_DEPLOYMENT_NAME}



# Get the Elasticsearch pod names

PODS=$(kubectl get pods -n $NAMESPACE -l app=$DEPLOYMENT -o jsonpath='{.items[*].metadata.name}')



# Loop through the pods and rebalance the shards

for POD in $PODS; do

  kubectl exec -n $NAMESPACE $POD -- curl -X POST "http://localhost:9200/_cluster/reroute?retry_failed=true"

done


```

Elasticsearch shards relocation is an incident type that occurs when Elasticsearch is moving its data shards from one node to another, which could cause temporary unavailability of data or increase in query latency. This may happen due to various reasons like node failure, hardware maintenance, or rebalancing of data.


The incident type of "Kubernetes deployment with multiple restarts" indicates that a Kubernetes deployment has experienced multiple restarts within a certain timeframe, which is usually indicative of a problem. Kubernetes is a popular container orchestration platform that automates the deployment, scaling, and management of containerized applications. When a deployment experiences multiple restarts, it can impact the availability and performance of the application, and can be a sign of underlying issues that need to be addressed. This incident type is typically monitored and managed by DevOps teams responsible for ensuring the health and reliability of Kubernetes-based applications.


Kubernetes deployment with multiple restarts

This incident type involves nodes in a Kubernetes cluster that are experiencing network unavailability, meaning they are not accessible. This could be due to a misconfiguration, route exhaustion, or a physical problem with the network connection to the hardware. It is a high urgency incident that requires immediate attention to restore network connectivity to the affected nodes.


Kubernetes Nodes with Network Unavailable

This incident type relates to the monitoring of Kubernetes deployments replica pods. It implies that there is an issue with the number of replica pods available as compared to the desired number. The incident might be triggered by a query alert monitor and might require immediate action to resolve the issue. The incident could impact the deployment of applications hosted on Kubernetes and might require troubleshooting and fixing the underlying issue.


Kubernetes Deployments Replica Pods Monitoring Incident

This incident type occurs when the pods in a Kubernetes cluster fail to schedule, which prevents the application from scaling. Kubernetes is a container orchestration platform that automates the deployment, scaling, and management of containerized applications. Pods in Kubernetes are the smallest deployable units that can be created and managed. When pods do not schedule, it means that the Kubernetes scheduler is unable to find a node with enough resources to run the pod. This can cause issues with application performance and scalability.


Kubernetes - Pods not scheduled preventing application scaling.

This incident type occurs when a Kubernetes pod is not ready to accept traffic due to a configuration problem or a change in the pod specification. It can cause spikes in the number of unavailable pods and impact the performance of the system.


Kubernetes - Pod status not ready

```shell
export LABEL_SELECTOR="PLACEHOLDER"

export ELASTICSEARCH_POD_NAME="PLACEHOLDER"

export ELASTICSEARCH_ENDPOINT="PLACEHOLDER"

export ELASTICSEARCH_NAMESPACE="PLACEHOLDER"

export ELASTICSEARCH_DEPLOYMENT_NAME="PLACEHOLDER"
```


### Check if the Elasticsearch pod is running

```shell
kubectl get pods -n ${ELASTICSEARCH_NAMESPACE} -l ${LABEL_SELECTOR}
```

### Check the logs of the Elasticsearch pod to see if there are any error messages related to shards relocation

```shell
kubectl logs ${ELASTICSEARCH_POD_NAME} -n ${ELASTICSEARCH_NAMESPACE}
```

### Check if the Elasticsearch cluster is healthy

```shell
kubectl exec -it ${ELASTICSEARCH_POD_NAME} -n ${ELASTICSEARCH_NAMESPACE} curl -X GET "http://localhost:9200/_cluster/health"
```

### Check the status of all Elasticsearch indices

```shell
kubectl exec -it ${ELASTICSEARCH_POD_NAME} -n ${ELASTICSEARCH_NAMESPACE} curl -X GET "http://localhost:9200/_cat/indices"
```

### Check the status of all Elasticsearch shards

```shell
kubectl exec -it ${ELASTICSEARCH_POD_NAME} -n ${ELASTICSEARCH_NAMESPACE} curl -X GET "http://localhost:9200/_cat/shards"
```

### Check the status of the Elasticsearch nodes

```shell
kubectl exec -it ${ELASTICSEARCH_POD_NAME} -n ${ELASTICSEARCH_NAMESPACE} curl -X GET "http://localhost:9200/_cat/nodes"
```

### A node failure or shutdown can trigger Elasticsearch to relocate shards from the affected node to other nodes in the cluster.

```shell


#!/bin/bash



# Set the namespace and deployment name for Elasticsearch

NAMESPACE=${ELASTICSEARCH_NAMESPACE}

DEPLOYMENT=${ELASTICSEARCH_DEPLOYMENT_NAME}



# Get the Elasticsearch pods

PODS=$(kubectl get pods -n $NAMESPACE -l app=$DEPLOYMENT -o jsonpath='{.items[*].metadata.name}')



# Check if any of the pods are not ready

for POD in $PODS; do

  if [[ $(kubectl get pod $POD -n $NAMESPACE -o jsonpath='{.status.phase}') != "Running" ]]; then

    echo "Pod $POD is not running. Elasticsearch may be relocating shards."

    exit 1

  fi

done



# Check if Elasticsearch is relocating shards

if [[ $(kubectl exec $POD -n $NAMESPACE -- curl -s localhost:9200/_cluster/health | jq -r '.relocating_shards') -gt 0 ]]; then

  echo "Elasticsearch is relocating shards. Please check the cluster status for more information."

  exit 1

else

  echo "No issues found with Elasticsearch shards relocation."

fi


```

### Rebalancing of data across the cluster due to node addition or removal can trigger Elasticsearch to relocate shards from one node to another.

```shell
bash

#!/bin/bash



# set the namespace and deployment name

NAMESPACE=${ELASTICSEARCH_NAMESPACE}

DEPLOYMENT=${ELASTICSEARCH_DEPLOYMENT_NAME}



# get the number of replicas in the deployment

REPLICAS=$(kubectl get deployment $DEPLOYMENT -n $NAMESPACE -o jsonpath='{.spec.replicas}')



# get the number of ready replicas in the deployment

READY_REPLICAS=$(kubectl get deployment $DEPLOYMENT -n $NAMESPACE -o jsonpath='{.status.readyReplicas}')



# check if the deployment is in the process of scaling up or down

if [ $READY_REPLICAS -lt $REPLICAS ]; then

    echo "Deployment is scaling up, Elasticsearch shards relocation may occur."

elif [ $READY_REPLICAS -gt $REPLICAS ]; then

    echo "Deployment is scaling down, Elasticsearch shards relocation may occur."

else

    echo "Deployment is not scaling, Elasticsearch shards relocation is unlikely to occur."

fi


```

### Check the Elasticsearch logs to identify the reason for shard relocation and the status of the relocation process.

```shell


#!/bin/bash



# Get the Pod name of the Elasticsearch instance 

POD_NAME=$(kubectl get pods -l ${LABEL_SELECTOR} -o jsonpath="{.items[0].metadata.name}")



# Check the Elasticsearch logs for shard relocation

kubectl logs $POD_NAME | grep "shard relocation"



# Check the status of the shard relocation process

kubectl exec -it $POD_NAME -- curl -XGET ${ELASTICSEARCH_ENDPOINT}/_cat/recovery


```


### Rebalance the shards manually or use Elasticsearch's auto-rebalancing feature to redistribute the shards evenly across the cluster nodes.

```shell


#!/bin/bash



# Set the Elasticsearch namespace and deployment name

NAMESPACE=${ELASTICSEARCH_NAMESPACE}

DEPLOYMENT=${ELASTICSEARCH_DEPLOYMENT_NAME}



# Get the Elasticsearch pod names

PODS=$(kubectl get pods -n $NAMESPACE -l app=$DEPLOYMENT -o jsonpath='{.items[*].metadata.name}')



# Loop through the pods and rebalance the shards

for POD in $PODS; do

  kubectl exec -n $NAMESPACE $POD -- curl -X POST "http://localhost:9200/_cluster/reroute?retry_failed=true"

done


```


Elasticsearch shards relocation Incident or Kubernetes.

Overview

Parameters

Debug

Check if the Elasticsearch pod is running

Check if the Elasticsearch cluster is healthy

Check the status of all Elasticsearch indices

Check the status of all Elasticsearch shards

Check the status of the Elasticsearch nodes

A node failure or shutdown can trigger Elasticsearch to relocate shards from the affected node to other nodes in the cluster.

Rebalancing of data across the cluster due to node addition or removal can trigger Elasticsearch to relocate shards from one node to another.

Check the Elasticsearch logs to identify the reason for shard relocation and the status of the relocation process.

Repair

Rebalance the shards manually or use Elasticsearch's auto-rebalancing feature to redistribute the shards evenly across the cluster nodes.

Learn more

Related Runbooks

Kubernetes deployment with multiple restarts

Kubernetes Nodes with Network Unavailable

Kubernetes Deployments Replica Pods Monitoring Incident

Kubernetes - Pods not scheduled preventing application scaling.

Support