---
id: 7e0d897c-0d9a-4e4e-9227-865d592763bc
---
# High query load on ElasticSearch service
---

This incident type typically occurs when the number of queries currently in progress on an ElasticSearch service is high, leading to an increased query load. This can result in slower query response times and reduced system performance. It is important to investigate and resolve this issue to ensure optimal performance of the ElasticSearch service.

### Parameters
```shell
# Environment Variables

export NODE_NAME="PLACEHOLDER"

export INDEX_NAME="PLACEHOLDER"
```

## Debug

### Check if ElasticSearch service is running
```shell
systemctl status elasticsearch.service
```

### Check if ElasticSearch is listening on the correct port
```shell
netstat -tuln | grep 9200
```

### Check ElasticSearch query load
```shell
curl -X GET "http://localhost:9200}/_cat/thread_pool?v"
```

### Check ElasticSearch query performance
```shell
curl -X GET "http://localhost:9200/_cat/indices?v"
```

### Check ElasticSearch query logs
```shell
tail -n 100 /var/log/elasticsearch/${NODE_NAME}.log
```

### Check ElasticSearch cluster state
```shell
curl -X GET "http://localhost:9200/_cluster/state?pretty"
```

## Repair

### Stop ElasticSearch service
```shell
sudo systemctl stop elasticsearch
```

### Edit ElasticSearch configuration file to update number of shards and replicas
```shell
sed -i "s/number_of_shards: .*/number_of_shards: $NUM_SHARDS/g" $ES_CONFIG_FILE

sed -i "s/number_of_replicas: .*/number_of_replicas: $NUM_REPLICAS/g" $ES_CONFIG_FILE
```

### Start ElasticSearch service
```shell
sudo systemctl start elasticsearch
```

### Optimize the index
```shell
curl -XPOST 'http://localhost:9200/${INDEX_NAME}/_optimize?max_num_segments=1'
```


This incident type typically occurs when the number of queries currently in progress on an ElasticSearch service is high, leading to an increased query load. This can result in slower query response times and reduced system performance. It is important to investigate and resolve this issue to ensure optimal performance of the ElasticSearch service.


This incident type refers to a situation where a Spark cluster experiences performance bottlenecks when it is subjected to peak loads. In other words, the Spark cluster struggles to handle the high volume of requests it receives during times of heavy traffic or increased demand. This can lead to slower processing times, delays, or even system crashes. Identifying and resolving the root cause of the bottlenecks is crucial to ensure the smooth functioning of the Spark cluster during peak loads.


Spark cluster bottlenecks during peak loads.

This incident type refers to a situation where the MySQL database is experiencing a high level of threads running, which could indicate a performance issue. This may cause issues with the application that relies on the database, such as slow response times or even downtime. Resolving this incident typically involves investigating the root cause of the high thread count and optimizing the database configuration or query performance to reduce the load on the database.


MySQL high threads running incident.

This incident type is related to a high request rate on a Kafka producer. This could be caused by a sudden increase in traffic or a malfunction in the producer system. This incident can cause delays in message processing and may lead to data loss or system failure if not addressed promptly. It is important to identify the root cause of this incident and take appropriate measures to prevent similar incidents in the future.


High request rate on Kafka producer.

This incident type typically occurs when the unsuccessful requests rate in ElasticSearch is higher than the expected threshold. This may indicate that there are issues with the ElasticSearch cluster or that it is not able to handle the requests in a timely manner. To prevent further issues, it is important to investigate the root cause and address any underlying issues that may be causing the high unsuccessful requests rate. This will help ensure that the ElasticSearch cluster is able to handle requests efficiently and effectively.


High Unsuccessful Requests Rate in ElasticSearch

This incident type refers to an elevated time spent on processing queries in an Elasticsearch cluster. It could be an indication of potential resource bottlenecks or a need to optimize the queries. This incident requires investigation and resolution to prevent further issues and ensure a smooth functioning Elasticsearch cluster.


High time spent on Elasticsearch queries

```shell
# Environment Variables

export NODE_NAME="PLACEHOLDER"

export INDEX_NAME="PLACEHOLDER"
```


### Check if ElasticSearch service is running

```shell
systemctl status elasticsearch.service
```

### Check if ElasticSearch is listening on the correct port

```shell
netstat -tuln | grep 9200
```

### Check ElasticSearch query load

```shell
curl -X GET "http://localhost:9200}/_cat/thread_pool?v"
```

### Check ElasticSearch query performance

```shell
curl -X GET "http://localhost:9200/_cat/indices?v"
```

### Check ElasticSearch query logs

```shell
tail -n 100 /var/log/elasticsearch/${NODE_NAME}.log
```

### Check ElasticSearch cluster state

```shell
curl -X GET "http://localhost:9200/_cluster/state?pretty"
```


### Stop ElasticSearch service

```shell
sudo systemctl stop elasticsearch
```

### Edit ElasticSearch configuration file to update number of shards and replicas

```shell
sed -i "s/number_of_shards: .*/number_of_shards: $NUM_SHARDS/g" $ES_CONFIG_FILE

sed -i "s/number_of_replicas: .*/number_of_replicas: $NUM_REPLICAS/g" $ES_CONFIG_FILE
```

### Start ElasticSearch service

```shell
sudo systemctl start elasticsearch
```

### Optimize the index

```shell
curl -XPOST 'http://localhost:9200/${INDEX_NAME}/_optimize?max_num_segments=1'
```


High query load on ElasticSearch service

Overview

Parameters

Debug

Check if ElasticSearch service is running

Check if ElasticSearch is listening on the correct port

Check ElasticSearch query load

Check ElasticSearch query performance

Check ElasticSearch query logs

Check ElasticSearch cluster state

Repair

Stop ElasticSearch service

Edit ElasticSearch configuration file to update number of shards and replicas

Start ElasticSearch service

Optimize the index

Learn more

Related Runbooks

Spark cluster bottlenecks during peak loads.

MySQL high threads running incident.

High request rate on Kafka producer.

High Unsuccessful Requests Rate in ElasticSearch

Support