---
id: 50ae58a2-616e-11ee-8c99-0242ac120002
---

# Excessive Memory Swap on Cassandra Cluster
---

Excessive Memory Swap on Cassandra Cluster refers to an incident where the Cassandra database cluster is experiencing high levels of swapping between physical memory and disk space due to the unavailability of enough physical memory. This can lead to degraded performance and unresponsiveness of the database cluster, causing disruptions to the application or service that relies on it. This type of incident requires immediate attention from the operations team to identify the root cause and take necessary measures to mitigate the issue.

### Parameters
```shell
export PID="PLACEHOLDER"

export SERVER_IP="PLACEHOLDER"

export MEMORY_SIZE="PLACEHOLDER"

export NEW_MEMORY="PLACEHOLDER"

export MAX_QUERIES="PLACEHOLDER"

export MAX_MEMORY="PLACEHOLDER"

export DESIRED_SWAPPINESS_VALUE="PLACEHOLDER"
```

## Debug

### Check total available memory
```shell
free -m
```

### Check the swap space utilization
```shell
swapon -s
```

### Display the process that consumes the most memory
```shell
ps -eo %mem,%cpu,cmd --sort=-%mem | head -n 10
```

### Display the processes that are using swap space
```shell
for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n | less
```

### Check the amount of swap space used by each process
```shell
pmap -x ${PID} | grep -i swap
```

### Check the amount of memory used by each process
```shell
pmap -x ${PID} | grep total
```

### Identify the processes that are causing excessive memory swapping
```shell
sudo atop -M | grep SWP | sort -k 8nr | head -n 10
```

### Check if the swappiness value is set correctly
```shell
cat /proc/sys/vm/swappiness
```

### Check the value of dirty_ratio and dirty_background_ratio
```shell
sudo cat /proc/sys/vm/dirty_ratio

sudo cat /proc/sys/vm/dirty_background_ratio
```

### Check the current value of vm.dirty_bytes
```shell
sudo cat /proc/sys/vm/dirty_bytes
```

### Check if Transparent Huge Pages (THP) is enabled
```shell
cat /sys/kernel/mm/transparent_hugepage/enabled
```

### Check the status of the Cassandra service
```shell
sudo systemctl status cassandra.service
```

## Repair

### Upgrade the physical memory of the servers in the Cassandra cluster to provide more space for data to reside in memory rather than needing to swap to disk.
```shell
bash

#!/bin/bash



# Set variables

SERVER_IP=${SERVER_IP}

MEMORY_SIZE=${MEMORY_SIZE}



# Upgrade memory size of the server

ssh user@$SERVER_IP "sudo shutdown -h now"

# Once the server is down, install the new memory

ssh user@$SERVER_IP "sudo apt-get install ${NEW_MEMORY}"

# Turn on the server

ssh user@$SERVER_IP "sudo reboot"



# Verify the new memory size

ssh user@$SERVER_IP "sudo grep MemTotal /proc/meminfo"


```

### Tune the Cassandra configuration to reduce memory consumption, for example, by limiting the number of concurrent queries or reducing the size of data kept in memory.
```shell


#!/bin/bash



# set the maximum number of concurrent queries

MAX_QUERIES=${MAX_QUERIES}



# set the maximum size of data kept in memory

MAX_MEMORY=${MAX_MEMORY}



# update the Cassandra configuration file to include the new settings

echo "max_concurrent_queries = $MAX_QUERIES" >> /etc/cassandra/cassandra.yaml

echo "max_memory_size = $MAX_MEMORY" >> /etc/cassandra/cassandra.yaml



# restart the Cassandra service to apply the changes

service cassandra restart



# verify that the changes were applied successfully

cqlsh -e "DESCRIBE CONFIG;"


```

### Adjust the swapping configuration of the operating system to prevent it from swapping too aggressively.
```shell


#!/bin/bash



# Set the desired swappiness value

SWAPPINESS=${DESIRED_SWAPPINESS_VALUE}



# Check if the current swappiness value is different from the desired value

if [ $(cat /proc/sys/vm/swappiness) -ne $SWAPPINESS ]; then

  # Update the swappiness value

  echo $SWAPPINESS | sudo tee /proc/sys/vm/swappiness



  # Update the sysctl.conf file to make the changes persistent across reboots

  echo "vm.swappiness = $SWAPPINESS" | sudo tee -a /etc/sysctl.conf



  # Apply the new swappiness value

  sudo sysctl -p

fi


```


Excessive Memory Swap on Cassandra Cluster refers to an incident where the Cassandra database cluster is experiencing high levels of swapping between physical memory and disk space due to the unavailability of enough physical memory. This can lead to degraded performance and unresponsiveness of the database cluster, causing disruptions to the application or service that relies on it. This type of incident requires immediate attention from the operations team to identify the root cause and take necessary measures to mitigate the issue.


Tomcat High Swap Usage Incident refers to an event where Tomcat, a popular open-source web server, is experiencing high swap usage. Swap usage is a memory management technique used by the operating system to temporarily move data from RAM to the hard disk. When Tomcat is consuming high swap memory, it can cause the server to slow down or even crash, resulting in downtime for the website or application hosted on the server. This incident type requires immediate attention to identify the root cause and resolve the issue to prevent further downtime.


Tomcat High Swap Usage Incident

Tomcat is a widely used open source web server and servlet container software. Full GC (Garbage Collection) is a process that runs in Java Virtual Machine (JVM) to reclaim memory occupied by objects that are no longer in use. When Full GC events happen too frequently, it can cause performance issues and lead to application downtime. This incident type refers to the occurrence of frequent Full GC events in Tomcat.


Tomcat Frequent Full GC Events

A swap space issue in Linux occurs when the system's usage of swap space becomes problematic or inefficient. while swap space is valuable for handling occasional memory demands, over-reliance on it can indicate underlying performance issues. Understanding swap usage patterns and tuning the system accordingly can help avoid swap space problems and improve overall system performance.


Swap Space filling Up

A Host Out of Memory(OOM) Incident occurs when a server or system runs out of memory, causing it to crash or become unresponsive. This can be caused by a variety of factors, such as an unexpected surge in traffic or insufficient resources allocated to the system. Resolving this type of incident requires identifying the root cause of the memory issue and taking appropriate measures such as optimizing system resources or increasing memory capacity.


Host Out of Memory (OOM) Incident

This incident type refers to a situation where the host memory of a system is not being fully utilized and may be causing performance issues. It may trigger when the memory load is below the expected level for a certain period of time. The incident may require investigation to determine the root cause and may involve reducing the amount of memory or other memory management strategies to optimize system performance.


"Host Memory is underutilized incident"

```shell
export PID="PLACEHOLDER"

export SERVER_IP="PLACEHOLDER"

export MEMORY_SIZE="PLACEHOLDER"

export NEW_MEMORY="PLACEHOLDER"

export MAX_QUERIES="PLACEHOLDER"

export MAX_MEMORY="PLACEHOLDER"

export DESIRED_SWAPPINESS_VALUE="PLACEHOLDER"
```


### Check total available memory

```shell
free -m
```

### Check the swap space utilization

```shell
swapon -s
```

### Display the process that consumes the most memory

```shell
ps -eo %mem,%cpu,cmd --sort=-%mem | head -n 10
```

### Display the processes that are using swap space

```shell
for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n | less
```

### Check the amount of swap space used by each process

```shell
pmap -x ${PID} | grep -i swap
```

### Check the amount of memory used by each process

```shell
pmap -x ${PID} | grep total
```

### Identify the processes that are causing excessive memory swapping

```shell
sudo atop -M | grep SWP | sort -k 8nr | head -n 10
```

### Check if the swappiness value is set correctly

```shell
cat /proc/sys/vm/swappiness
```

### Check the value of dirty\_ratio and dirty\_background\_ratio

```shell
sudo cat /proc/sys/vm/dirty_ratio

sudo cat /proc/sys/vm/dirty_background_ratio
```

### Check the current value of vm.dirty\_bytes

```shell
sudo cat /proc/sys/vm/dirty_bytes
```

### Check if Transparent Huge Pages (THP) is enabled

```shell
cat /sys/kernel/mm/transparent_hugepage/enabled
```

### Check the status of the Cassandra service

```shell
sudo systemctl status cassandra.service
```


### Upgrade the physical memory of the servers in the Cassandra cluster to provide more space for data to reside in memory rather than needing to swap to disk.

```shell
bash

#!/bin/bash



# Set variables

SERVER_IP=${SERVER_IP}

MEMORY_SIZE=${MEMORY_SIZE}



# Upgrade memory size of the server

ssh user@$SERVER_IP "sudo shutdown -h now"

# Once the server is down, install the new memory

ssh user@$SERVER_IP "sudo apt-get install ${NEW_MEMORY}"

# Turn on the server

ssh user@$SERVER_IP "sudo reboot"



# Verify the new memory size

ssh user@$SERVER_IP "sudo grep MemTotal /proc/meminfo"


```

### Tune the Cassandra configuration to reduce memory consumption, for example, by limiting the number of concurrent queries or reducing the size of data kept in memory.

```shell


#!/bin/bash



# set the maximum number of concurrent queries

MAX_QUERIES=${MAX_QUERIES}



# set the maximum size of data kept in memory

MAX_MEMORY=${MAX_MEMORY}



# update the Cassandra configuration file to include the new settings

echo "max_concurrent_queries = $MAX_QUERIES" >> /etc/cassandra/cassandra.yaml

echo "max_memory_size = $MAX_MEMORY" >> /etc/cassandra/cassandra.yaml



# restart the Cassandra service to apply the changes

service cassandra restart



# verify that the changes were applied successfully

cqlsh -e "DESCRIBE CONFIG;"


```

### Adjust the swapping configuration of the operating system to prevent it from swapping too aggressively.

```shell


#!/bin/bash



# Set the desired swappiness value

SWAPPINESS=${DESIRED_SWAPPINESS_VALUE}



# Check if the current swappiness value is different from the desired value

if [ $(cat /proc/sys/vm/swappiness) -ne $SWAPPINESS ]; then

  # Update the swappiness value

  echo $SWAPPINESS | sudo tee /proc/sys/vm/swappiness



  # Update the sysctl.conf file to make the changes persistent across reboots

  echo "vm.swappiness = $SWAPPINESS" | sudo tee -a /etc/sysctl.conf



  # Apply the new swappiness value

  sudo sysctl -p

fi


```


Excessive Memory Swap on Cassandra Cluster

Overview

Parameters

Debug

Check total available memory

Check the swap space utilization

Display the process that consumes the most memory

Display the processes that are using swap space

Check the amount of swap space used by each process

Check the amount of memory used by each process

Identify the processes that are causing excessive memory swapping

Check if the swappiness value is set correctly

Check the value of dirty_ratio and dirty_background_ratio

Check the current value of vm.dirty_bytes

Check if Transparent Huge Pages (THP) is enabled

Check the status of the Cassandra service

Repair

Upgrade the physical memory of the servers in the Cassandra cluster to provide more space for data to reside in memory rather than needing to swap to disk.

Tune the Cassandra configuration to reduce memory consumption, for example, by limiting the number of concurrent queries or reducing the size of data kept in memory.

Adjust the swapping configuration of the operating system to prevent it from swapping too aggressively.

Learn more

Related Runbooks

Tomcat High Swap Usage Incident

Tomcat Frequent Full GC Events

Swap Space filling Up

Host Out of Memory (OOM) Incident

Support