---
id: 2882e276-616e-11ee-8c99-0242ac120002
---

# High Average Queue Size on Cassandra
---

This incident type refers to a situation where the average queue size on a Cassandra database exceeds a certain threshold. A queue is a data structure that holds incoming requests until they can be processed. When the average queue size is too high, it can indicate that the database is struggling to keep up with incoming requests, which can result in slower response times and potential service disruptions. This type of incident requires investigation and resolution to ensure the database can handle the incoming workload efficiently.

### Parameters
```shell
export NODETOOL_PARAMETERS="PLACEHOLDER"

export NETSTAT_PARAMETERS="PLACEHOLDER"

export CASSANDRA_PORT="PLACEHOLDER"

export STATUS="PLACEHOLDER"

export NUMBER_OF_NODES="PLACEHOLDER"

export PATH_TO_CASSANDRA_CONFIG_FILE="PLACEHOLDER"
```

## Debug

### Check the current queue size
```shell
nodetool -- ${NODETOOL_PARAMETERS} tpstats | grep -A1 "ReadStage" | tail -n1
```

### Check the number of requests per second
```shell
nodetool -- ${NODETOOL_PARAMETERS} tpstats | grep -A1 "ReadStage" | tail -n1 | awk '{print $2}'
```

### Check the number of active connections
```shell
netstat -- ${NETSTAT_PARAMETERS} | grep "${CASSANDRA_PORT}" | wc -l
```

### Check the number of pending connections
```shell
netstat -- ${NETSTAT_PARAMETERS} | grep "${CASSANDRA_PORT}" | grep "${STATUS}" | wc -l
```

### Check the system load average
```shell
uptime
```

### Check the disk usage
```shell
df -h
```

## Repair

### Increase the number of nodes in the Cassandra cluster to distribute the workload and reduce the load on individual nodes.
```shell


#!/bin/bash



# Set variables

NODES=${NUMBER_OF_NODES}

CASSANDRA_CONFIG=${PATH_TO_CASSANDRA_CONFIG_FILE}



# Update the Cassandra configuration file

sed -i "s/.*num_tokens.*/num_tokens: $NODES/g" $CASSANDRA_CONFIG



# Restart the Cassandra service

sudo systemctl restart cassandra.service


```


This incident type refers to a situation where the average queue size on a Cassandra database exceeds a certain threshold. A queue is a data structure that holds incoming requests until they can be processed. When the average queue size is too high, it can indicate that the database is struggling to keep up with incoming requests, which can result in slower response times and potential service disruptions. This type of incident requires investigation and resolution to ensure the database can handle the incoming workload efficiently.


A Rabbitmq memory high incident occurs when a node uses more than 90% of the allocated RAM. This can cause performance issues and impact the overall functionality of the system. It is important to promptly address this issue to prevent any further damage.


Rabbitmq memory high incident.

This incident type refers to a situation where the Kafka broker cluster has exhausted its resources. This could happen due to several reasons such as an increase in traffic, inadequate resources, or inefficient configurations. As a result, the Kafka brokers may stop responding or experience performance issues, leading to a disruption in the overall Kafka cluster.


Kafka Resource Exhaustion in Kafka Broker Cluster.

This incident type refers to a situation where there is a significant delay in the execution of queries on a Cassandra cluster. This delay can cause the system to become unresponsive and result in slower performance. It may be caused by a variety of factors such as an increase in traffic, inefficient queries, or hardware issues. The issue can impact the functionality of the system and requires immediate attention to prevent further disruption.


Slow Query Performance on Cassandra Cluster.

This incident type refers to a performance degradation issue in a Cassandra database caused by the new generation garbage collection (GC) process. Garbage collection is a process by which the unused memory occupied by objects in a program is cleared. In some cases, the new generation GC process may take longer than usual, causing performance issues in the Cassandra database. This could lead to slowness in queries or even crashing of the database.


Performance Degradation Due to New Generation Garbage Collection on Cassandra

This incident type refers to a situation where a delay or slowness occurs in a system that uses Cassandra database due to the shared storage. Shared storage means multiple servers are accessing the same storage unit, and this can cause latency issues. This type of incident can lead to performance degradation, and it needs to be addressed promptly to ensure optimal system performance.


Latency Caused by Shared Storage in Cassandra

```shell
export NODETOOL_PARAMETERS="PLACEHOLDER"

export NETSTAT_PARAMETERS="PLACEHOLDER"

export CASSANDRA_PORT="PLACEHOLDER"

export STATUS="PLACEHOLDER"

export NUMBER_OF_NODES="PLACEHOLDER"

export PATH_TO_CASSANDRA_CONFIG_FILE="PLACEHOLDER"
```


### Check the current queue size

```shell
nodetool -- ${NODETOOL_PARAMETERS} tpstats | grep -A1 "ReadStage" | tail -n1
```

### Check the number of requests per second

```shell
nodetool -- ${NODETOOL_PARAMETERS} tpstats | grep -A1 "ReadStage" | tail -n1 | awk '{print $2}'
```

### Check the number of active connections

```shell
netstat -- ${NETSTAT_PARAMETERS} | grep "${CASSANDRA_PORT}" | wc -l
```

### Check the number of pending connections

```shell
netstat -- ${NETSTAT_PARAMETERS} | grep "${CASSANDRA_PORT}" | grep "${STATUS}" | wc -l
```

### Check the system load average

```shell
uptime
```

### Check the disk usage

```shell
df -h
```


### Increase the number of nodes in the Cassandra cluster to distribute the workload and reduce the load on individual nodes.

```shell


#!/bin/bash



# Set variables

NODES=${NUMBER_OF_NODES}

CASSANDRA_CONFIG=${PATH_TO_CASSANDRA_CONFIG_FILE}



# Update the Cassandra configuration file

sed -i "s/.*num_tokens.*/num_tokens: $NODES/g" $CASSANDRA_CONFIG



# Restart the Cassandra service

sudo systemctl restart cassandra.service


```


High Average Queue Size on Cassandra

Overview

Parameters

Debug

Check the current queue size

Check the number of requests per second

Check the number of active connections

Check the number of pending connections

Check the system load average

Check the disk usage

Repair

Increase the number of nodes in the Cassandra cluster to distribute the workload and reduce the load on individual nodes.

Learn more

Related Runbooks

Rabbitmq memory high incident.

Kafka Resource Exhaustion in Kafka Broker Cluster.

Slow Query Performance on Cassandra Cluster.

Performance Degradation Due to New Generation Garbage Collection on Cassandra

Support