---
id: a69bc3da-616e-11ee-8c99-0242ac120002
---

# Excessive JVM GC on Cassandra Cluster
---

This incident type refers to a problem where the Java Virtual Machine (JVM) running on a Cassandra cluster is experiencing an excessive amount of garbage collection (GC) events. GC is a process where the JVM frees up memory by removing objects that are no longer in use. Excessive GC can lead to performance degradation and even system crashes.

### Parameters
```shell
export PID="PLACEHOLDER"

export PATH_TO_CASSANDRA_LOG_FILE="PLACEHOLDER"

export PATH_TO_CASSANDRA_INSTALLATION_DIRECTORY="PLACEHOLDER"

export NEW_HEAP_SIZE_IN_GB="PLACEHOLDER"

export NEW_HEAP_SIZE="PLACEHOLDER"
```

## Debug

### Check if Cassandra is running
```shell
sudo service cassandra status
```

### Check Cassandra's logs for GC events
```shell
sudo tail -f /var/log/cassandra/system.log | grep GC
```

### Check the heap size and usage of the JVM
```shell
sudo jstat -gcutil ${PID}
```

### Check the overall CPU and memory usage of the server
```shell
top
```

### Check the disk usage and I/O operations of the server
```shell
iostat
```

### Check the network traffic of the server
```shell
iftop
```

### The Cassandra cluster may be running on hardware that does not meet the minimum requirements for the workload, resulting in insufficient memory resources and increased GC frequency.
```shell


#!/bin/bash



# Check the available memory on the system

total_memory=$(free -m | awk 'NR==2{print $2}')

used_memory=$(free -m | awk 'NR==2{print $3}')

free_memory=$(free -m | awk 'NR==2{print $4}')



echo "Total Memory: $total_memory MB"

echo "Used Memory: $used_memory MB"

echo "Free Memory: $free_memory MB"



# Check the Cassandra process memory usage

cassandra_pid=$(pgrep -f "org.apache.cassandra.service.CassandraDaemon")

cassandra_mem=$(pmap -x $cassandra_pid | tail -n 1 | awk '{print $3}')



echo "Cassandra Memory Usage: $cassandra_mem KB"



# Check the Cassandra system logs for any out-of-memory errors

log_file=${PATH_TO_CASSANDRA_LOG_FILE}

if grep -q "Out of memory" $log_file; then

  echo "Cassandra Out of Memory Error Detected!"

fi


```

## Repair

### Adjust JVM heap size: Increasing the heap size of the JVM running on the Cassandra cluster can help reduce the frequency of GC events. This can be done by configuring the JVM parameters.
```shell


#!/bin/bash



# Set the path to the Cassandra installation directory

CASSANDRA_HOME=${PATH_TO_CASSANDRA_INSTALLATION_DIRECTORY}



# Set the new heap size value (in GB)

NEW_HEAP_SIZE=${NEW_HEAP_SIZE_IN_GB}



# Set the path to the Cassandra configuration file

CASSANDRA_CONFIG_FILE=$CASSANDRA_HOME/conf/cassandra-env.sh



# Update the JVM heap size in the Cassandra configuration file

sed -i "s/-Xmx[0-9]*G/-Xmx${NEW_HEAP_SIZE}G/" $CASSANDRA_CONFIG_FILE



# Restart the Cassandra service to apply the new JVM heap size

sudo service cassandra restart


```


This incident type refers to a problem where the Java Virtual Machine (JVM) running on a Cassandra cluster is experiencing an excessive amount of garbage collection (GC) events. GC is a process where the JVM frees up memory by removing objects that are no longer in use. Excessive GC can lead to performance degradation and even system crashes.


When too many requests are sent to a Tomcat server, the thread pools that handle these requests can run out of available threads. This means that incoming requests will not be processed, leading to service degradation or even a complete outage. This incident type is critical for web applications that rely on Tomcat as their application server.


Tomcat thread pool exhaustion.

Tomcat JVM OutOfMemory incident occurs when the Java Virtual Machine (JVM) running the Apache Tomcat server runs out of memory, which causes the server to crash or become unresponsive. This type of incident can happen due to a variety of reasons such as incorrect configuration settings, memory leaks, or insufficient memory allocation. When this incident occurs, it can impact the availability and performance of the web application running on the Tomcat server.


Tomcat JVM OutOfMemory Incident

Tomcat is a widely used open source web server and servlet container software. Full GC (Garbage Collection) is a process that runs in Java Virtual Machine (JVM) to reclaim memory occupied by objects that are no longer in use. When Full GC events happen too frequently, it can cause performance issues and lead to application downtime. This incident type refers to the occurrence of frequent Full GC events in Tomcat.


Tomcat Frequent Full GC Events

This incident type refers to the situation where the Tomcat server is utilizing more memory than it should, causing system instability and performance issues. This can happen due to a variety of factors such as memory leaks, improper configuration, or application code issues. It is important to address this incident promptly to prevent system failures and ensure optimal application performance.


Excessive memory usage by Tomcat.

This incident type refers to a situation where there is a significant delay in the execution of queries on a Cassandra cluster. This delay can cause the system to become unresponsive and result in slower performance. It may be caused by a variety of factors such as an increase in traffic, inefficient queries, or hardware issues. The issue can impact the functionality of the system and requires immediate attention to prevent further disruption.


Slow Query Performance on Cassandra Cluster.

```shell
export PID="PLACEHOLDER"

export PATH_TO_CASSANDRA_LOG_FILE="PLACEHOLDER"

export PATH_TO_CASSANDRA_INSTALLATION_DIRECTORY="PLACEHOLDER"

export NEW_HEAP_SIZE_IN_GB="PLACEHOLDER"

export NEW_HEAP_SIZE="PLACEHOLDER"
```


### Check if Cassandra is running

```shell
sudo service cassandra status
```

### Check Cassandra's logs for GC events

```shell
sudo tail -f /var/log/cassandra/system.log | grep GC
```

### Check the heap size and usage of the JVM

```shell
sudo jstat -gcutil ${PID}
```

### Check the overall CPU and memory usage of the server

```shell
top
```

### Check the disk usage and I/O operations of the server

```shell
iostat
```

### Check the network traffic of the server

```shell
iftop
```

### The Cassandra cluster may be running on hardware that does not meet the minimum requirements for the workload, resulting in insufficient memory resources and increased GC frequency.

```shell


#!/bin/bash



# Check the available memory on the system

total_memory=$(free -m | awk 'NR==2{print $2}')

used_memory=$(free -m | awk 'NR==2{print $3}')

free_memory=$(free -m | awk 'NR==2{print $4}')



echo "Total Memory: $total_memory MB"

echo "Used Memory: $used_memory MB"

echo "Free Memory: $free_memory MB"



# Check the Cassandra process memory usage

cassandra_pid=$(pgrep -f "org.apache.cassandra.service.CassandraDaemon")

cassandra_mem=$(pmap -x $cassandra_pid | tail -n 1 | awk '{print $3}')



echo "Cassandra Memory Usage: $cassandra_mem KB"



# Check the Cassandra system logs for any out-of-memory errors

log_file=${PATH_TO_CASSANDRA_LOG_FILE}

if grep -q "Out of memory" $log_file; then

  echo "Cassandra Out of Memory Error Detected!"

fi


```


### Adjust JVM heap size: Increasing the heap size of the JVM running on the Cassandra cluster can help reduce the frequency of GC events. This can be done by configuring the JVM parameters.

```shell


#!/bin/bash



# Set the path to the Cassandra installation directory

CASSANDRA_HOME=${PATH_TO_CASSANDRA_INSTALLATION_DIRECTORY}



# Set the new heap size value (in GB)

NEW_HEAP_SIZE=${NEW_HEAP_SIZE_IN_GB}



# Set the path to the Cassandra configuration file

CASSANDRA_CONFIG_FILE=$CASSANDRA_HOME/conf/cassandra-env.sh



# Update the JVM heap size in the Cassandra configuration file

sed -i "s/-Xmx[0-9]*G/-Xmx${NEW_HEAP_SIZE}G/" $CASSANDRA_CONFIG_FILE



# Restart the Cassandra service to apply the new JVM heap size

sudo service cassandra restart


```


Excessive JVM GC on Cassandra Cluster

Overview

Parameters

Debug

Check if Cassandra is running

Check Cassandra's logs for GC events

Check the heap size and usage of the JVM

Check the overall CPU and memory usage of the server

Check the disk usage and I/O operations of the server

Check the network traffic of the server

The Cassandra cluster may be running on hardware that does not meet the minimum requirements for the workload, resulting in insufficient memory resources and increased GC frequency.

Repair

Adjust JVM heap size: Increasing the heap size of the JVM running on the Cassandra cluster can help reduce the frequency of GC events. This can be done by configuring the JVM parameters.

Learn more

Related Runbooks

Tomcat thread pool exhaustion.

Tomcat JVM OutOfMemory Incident

Tomcat Frequent Full GC Events

Excessive memory usage by Tomcat.

Support