---
id: f0bfc6c3-4b42-4918-a4d5-9c2945c37b93
---
# Elasticsearch Heap Usage Too High Incident
---

An Elasticsearch Heap Usage Too High Incident is an issue where the Elasticsearch instance experiences high heap usage, which can lead to performance degradation or system failure. This incident is triggered when the heap usage crosses a predefined threshold, usually set at 80% of the allocated heap size. The incident requires immediate attention from the responsible team to investigate and resolve the root cause of the issue, which could be related to memory leaks, indexing issues, or insufficient heap size allocation.

### Parameters
```shell
export ELASTICSEARCH_INSTANCE_URL="PLACEHOLDER"

export ELASTICSEARCH_HOST="PLACEHOLDER"

export ELASTICSEARCH_PORT="PLACEHOLDER"

export EXPECTED_HEAP_SIZE="PLACEHOLDER"

export NEW_HEAP_SIZE_IN_MB="PLACEHOLDER"

export PATH_TO_ELASTICSEARCH_CONFIG_FILE="PLACEHOLDER"
```

## Debug

### Check Elasticsearch service status
```shell
systemctl status elasticsearch
```

### Check Elasticsearch logs for any errors
```shell
journalctl -u elasticsearch | tail
```

### Check Elasticsearch heap usage
```shell
curl -X GET "localhost:9200/_nodes/stats/jvm?pretty"
```

### Check Elasticsearch cluster health
```shell
curl -X GET "localhost:9200/_cluster/health?pretty"
```

### Check system memory usage
```shell
free -m
```

### Check disk usage
```shell
df -h
```

### Check CPU usage
```shell
top
```

### Large volume of data being indexed at a high rate causing heap memory usage to increase beyond the limit.
```shell


#!/bin/bash



# Set the Elasticsearch instance URL

ELASTICSEARCH_URL=${ELASTICSEARCH_INSTANCE_URL}



# Get the heap usage percentage from Elasticsearch

HEAP_USAGE=$(curl -s "${ELASTICSEARCH_URL}/_cat/nodes?h=heap.percent")



# Check if the heap usage is above the limit (80%)

if (( $(echo "${HEAP_USAGE} > 80" | bc -l) )); then

  echo "Heap usage is above the limit. Checking indexing rate..."



  # Get the indexing rate from Elasticsearch

  INDEXING_RATE=$(curl -s "${ELASTICSEARCH_URL}/_cat/indices?h=i,ss,si&bytes=b")



  # Check if the indexing rate is high

  if (( $(echo "${INDEXING_RATE}" | awk '{ sum += $3 } END { print sum }') )); then

    echo "Indexing rate is high. It could be causing the high heap usage."

  else

    echo "Indexing rate is normal. The high heap usage could be due to other reasons."

  fi

else

  echo "Heap usage is normal. No action necessary."

fi


```

### Misconfigured Elasticsearch JVM heap size, leading to insufficient memory allocation.
```shell


#!/bin/bash



# Set the Elasticsearch instance details

ELASTICSEARCH_HOST=${ELASTICSEARCH_HOST}

ELASTICSEARCH_PORT=${ELASTICSEARCH_PORT}



# Set the expected heap size in GB

EXPECTED_HEAP_SIZE=${EXPECTED_HEAP_SIZE}



# Get the current heap size of the Elasticsearch instance

CURRENT_HEAP_SIZE=$(curl -s -XGET "http://$ELASTICSEARCH_HOST:$ELASTICSEARCH_PORT/_nodes/_local/stats/jvm" | jq -r '.nodes[].jvm.mem.heap_max_in_bytes')



# Convert the heap size to GB

CURRENT_HEAP_SIZE_GB=$(echo "scale=2; $CURRENT_HEAP_SIZE/1024/1024/1024" | bc)



# Check if the current heap size is less than the expected heap size

if (( $(echo "$CURRENT_HEAP_SIZE_GB < $EXPECTED_HEAP_SIZE" | bc -l) )); then

    echo "Elasticsearch JVM heap size is misconfigured. Current heap size is $CURRENT_HEAP_SIZE_GB GB, but the expected heap size is $EXPECTED_HEAP_SIZE GB."

    # Add any additional diagnostic steps here

else

    echo "Elasticsearch JVM heap size is configured correctly. Current heap size is $CURRENT_HEAP_SIZE_GB GB."

fi


```

## Repair

### Increase the heap size of Elasticsearch to accommodate the increased usage.
```shell


#!/bin/bash



# Set the new heap size in MB

NEW_HEAP_SIZE=${NEW_HEAP_SIZE_IN_MB}



# Set the path to the Elasticsearch configuration file

ELASTICSEARCH_CONFIG=${PATH_TO_ELASTICSEARCH_CONFIG_FILE}



# Modify the Elasticsearch configuration file to update the heap size

sed -i "s/-Xmx[0-9]*m/-Xmx${NEW_HEAP_SIZE}m/g" ${ELASTICSEARCH_CONFIG}



# Restart Elasticsearch to apply the new heap size

systemctl restart elasticsearch


```



An Elasticsearch Heap Usage Too High Incident is an issue where the Elasticsearch instance experiences high heap usage, which can lead to performance degradation or system failure. This incident is triggered when the heap usage crosses a predefined threshold, usually set at 80% of the allocated heap size. The incident requires immediate attention from the responsible team to investigate and resolve the root cause of the issue, which could be related to memory leaks, indexing issues, or insufficient heap size allocation.


This incident type refers to a situation where the Tomcat server's JDBC connection pool usage surpasses its configured limits. This can lead to a shortage of available connections for incoming requests, resulting in slow response times or even server crashes. It is important to monitor and manage the JDBC connection pool usage to prevent such incidents from occurring.


Tomcat High JDBC Connection Pool Usage Incident.

This incident type refers to the situation where the Tomcat server is utilizing more memory than it should, causing system instability and performance issues. This can happen due to a variety of factors such as memory leaks, improper configuration, or application code issues. It is important to address this incident promptly to prevent system failures and ensure optimal application performance.


Excessive memory usage by Tomcat.

Apache Spark Out of Memory Error is a type of incident that occurs when the Apache Spark application running on a system is unable to allocate enough memory to perform its tasks. This can happen when the system's memory is insufficient or when there is a memory leak in the application's code. When this incident occurs, the application may crash, become unresponsive, or produce incorrect results. It is important to diagnose and resolve this issue as soon as possible to ensure the smooth functioning of the application.


Apache Spark Out of Memory Error incident.

A high CPU load incident occurs when the CPU usage of a host or server exceeds a certain threshold, typically 80%. This can cause slowdowns, unresponsiveness, or even crashes in the affected system. It is critical to address this issue promptly to minimize impact on users and prevent further damage to the system.


High CPU Load Incident

This incident type typically occurs when the unsuccessful requests rate in ElasticSearch is higher than the expected threshold. This may indicate that there are issues with the ElasticSearch cluster or that it is not able to handle the requests in a timely manner. To prevent further issues, it is important to investigate the root cause and address any underlying issues that may be causing the high unsuccessful requests rate. This will help ensure that the ElasticSearch cluster is able to handle requests efficiently and effectively.


High Unsuccessful Requests Rate in ElasticSearch

```shell
export ELASTICSEARCH_INSTANCE_URL="PLACEHOLDER"

export ELASTICSEARCH_HOST="PLACEHOLDER"

export ELASTICSEARCH_PORT="PLACEHOLDER"

export EXPECTED_HEAP_SIZE="PLACEHOLDER"

export NEW_HEAP_SIZE_IN_MB="PLACEHOLDER"

export PATH_TO_ELASTICSEARCH_CONFIG_FILE="PLACEHOLDER"
```


### Check Elasticsearch service status

```shell
systemctl status elasticsearch
```

### Check Elasticsearch logs for any errors

```shell
journalctl -u elasticsearch | tail
```

### Check Elasticsearch heap usage

```shell
curl -X GET "localhost:9200/_nodes/stats/jvm?pretty"
```

### Check Elasticsearch cluster health

```shell
curl -X GET "localhost:9200/_cluster/health?pretty"
```

### Check system memory usage

```shell
free -m
```

### Check disk usage

```shell
df -h
```

### Check CPU usage

```shell
top
```

### Large volume of data being indexed at a high rate causing heap memory usage to increase beyond the limit.

```shell


#!/bin/bash



# Set the Elasticsearch instance URL

ELASTICSEARCH_URL=${ELASTICSEARCH_INSTANCE_URL}



# Get the heap usage percentage from Elasticsearch

HEAP_USAGE=$(curl -s "${ELASTICSEARCH_URL}/_cat/nodes?h=heap.percent")



# Check if the heap usage is above the limit (80%)

if (( $(echo "${HEAP_USAGE} > 80" | bc -l) )); then

  echo "Heap usage is above the limit. Checking indexing rate..."



  # Get the indexing rate from Elasticsearch

  INDEXING_RATE=$(curl -s "${ELASTICSEARCH_URL}/_cat/indices?h=i,ss,si&bytes=b")



  # Check if the indexing rate is high

  if (( $(echo "${INDEXING_RATE}" | awk '{ sum += $3 } END { print sum }') )); then

    echo "Indexing rate is high. It could be causing the high heap usage."

  else

    echo "Indexing rate is normal. The high heap usage could be due to other reasons."

  fi

else

  echo "Heap usage is normal. No action necessary."

fi


```

### Misconfigured Elasticsearch JVM heap size, leading to insufficient memory allocation.

```shell


#!/bin/bash



# Set the Elasticsearch instance details

ELASTICSEARCH_HOST=${ELASTICSEARCH_HOST}

ELASTICSEARCH_PORT=${ELASTICSEARCH_PORT}



# Set the expected heap size in GB

EXPECTED_HEAP_SIZE=${EXPECTED_HEAP_SIZE}



# Get the current heap size of the Elasticsearch instance

CURRENT_HEAP_SIZE=$(curl -s -XGET "http://$ELASTICSEARCH_HOST:$ELASTICSEARCH_PORT/_nodes/_local/stats/jvm" | jq -r '.nodes[].jvm.mem.heap_max_in_bytes')



# Convert the heap size to GB

CURRENT_HEAP_SIZE_GB=$(echo "scale=2; $CURRENT_HEAP_SIZE/1024/1024/1024" | bc)



# Check if the current heap size is less than the expected heap size

if (( $(echo "$CURRENT_HEAP_SIZE_GB < $EXPECTED_HEAP_SIZE" | bc -l) )); then

    echo "Elasticsearch JVM heap size is misconfigured. Current heap size is $CURRENT_HEAP_SIZE_GB GB, but the expected heap size is $EXPECTED_HEAP_SIZE GB."

    # Add any additional diagnostic steps here

else

    echo "Elasticsearch JVM heap size is configured correctly. Current heap size is $CURRENT_HEAP_SIZE_GB GB."

fi


```


### Increase the heap size of Elasticsearch to accommodate the increased usage.

```shell


#!/bin/bash



# Set the new heap size in MB

NEW_HEAP_SIZE=${NEW_HEAP_SIZE_IN_MB}



# Set the path to the Elasticsearch configuration file

ELASTICSEARCH_CONFIG=${PATH_TO_ELASTICSEARCH_CONFIG_FILE}



# Modify the Elasticsearch configuration file to update the heap size

sed -i "s/-Xmx[0-9]*m/-Xmx${NEW_HEAP_SIZE}m/g" ${ELASTICSEARCH_CONFIG}



# Restart Elasticsearch to apply the new heap size

systemctl restart elasticsearch


```


Elasticsearch Heap Usage Too High Incident

Overview

Parameters

Debug

Check Elasticsearch service status

Check Elasticsearch logs for any errors

Check Elasticsearch heap usage

Check Elasticsearch cluster health

Check system memory usage

Check disk usage

Check CPU usage

Large volume of data being indexed at a high rate causing heap memory usage to increase beyond the limit.

Misconfigured Elasticsearch JVM heap size, leading to insufficient memory allocation.

Repair

Increase the heap size of Elasticsearch to accommodate the increased usage.

Learn more

Related Runbooks

Tomcat High JDBC Connection Pool Usage Incident.

Excessive memory usage by Tomcat.

Apache Spark Out of Memory Error incident.

High CPU Load Incident

Support