---
id: 3d66bb30-092c-4897-81ed-33418bb84251
---

# Cassandra many compaction tasks are pending.
---

This incident type occurs when there are many pending compaction tasks in a Cassandra cluster. Compaction is the process of merging and removing data from SSTables (sorted string tables) in Cassandra. When compaction tasks are pending, it means that the process is not completing in a timely manner and is causing performance issues in the cluster. This incident requires investigation and resolution to ensure the cluster's health and performance.

### Parameters
```shell
# Environment Variables

export CASSANDRA_NODE_IP="PLACEHOLDER"

export PATH_TO_CASSANDRA_YAML="PLACEHOLDER"

export NUMBER_OF_COMPACTION_THREADS="PLACEHOLDER"

export CASSANDRA_DATA_DIRECTORY="PLACEHOLDER"

export MEMORY_THRESHOLD="PLACEHOLDER"

export NEW_COMPACTION_THROUGHPUT="PLACEHOLDER"
```

## Debug

### Check the status of the Cassandra service
```shell
systemctl status cassandra
```

### Check the number of pending compaction tasks in Cassandra
```shell
nodetool compactionstats | grep pending_tasks
```

### Check the health of the Cassandra cluster
```shell
nodetool status
```

### Check the Cassandra logs for errors or warnings
```shell
tail -n 100 /var/log/cassandra/system.log
```

### Check the system load and CPU usage
```shell
top
```

### Check the disk usage and available space
```shell
df -h
```

### Check the network connectivity and latency
```shell
ping ${CASSANDRA_NODE_IP}
```

### Check the firewall rules and open ports
```shell
iptables -L -n
```

### Insufficient resources such as CPU, memory, or disk space, causing compaction to slow down or stop.
```shell


#!/bin/bash


 CPU_THRESHOLD="PLACEHOLDER"

 DISK_THRESHOLD="PLACEHOLDER"
# Check CPU usage

cpu_usage=$(top -b -n1 | grep "Cpu(s)" | awk '{print $2 + $4}')

if (( $(echo "$cpu_usage > ${CPU_THRESHOLD}" | bc -l) )); then

  echo "CPU usage is high ($cpu_usage%), which may be causing compaction to slow down or stop."

fi



# Check memory usage

mem_usage=$(free | awk '/Mem/{printf("%.2f"), $3/$2*100}')

if (( $(echo "$mem_usage > ${MEMORY_THRESHOLD}" | bc -l) )); then

  echo "Memory usage is high ($mem_usage%), which may be causing compaction to slow down or stop."

fi



# Check disk space

disk_usage=$(df -h ${CASSANDRA_DATA_DIRECTORY} | tail -1 | awk '{print $5}' | tr -d '%')

if (( $disk_usage > ${DISK_THRESHOLD} )); then

  echo "Disk space usage is high ($disk_usage%), which may be causing compaction to slow down or stop."

fi


```

## Repair

### Set variables
```shell
CASSANDRA_CONFIG_FILE=${PATH_TO_CASSANDRA_YAML}

CONCURRENT_COMPACTORS=${NUMBER_OF_COMPACTION_THREADS}
```

### Check if cassandra config file exists
```shell
if [ ! -f $CASSANDRA_CONFIG_FILE ]; then

  echo "Cassandra config file not found: $CASSANDRA_CONFIG_FILE"

  exit 1

fi
```

### Backup cassandra config file
```shell
cp $CASSANDRA_CONFIG_FILE $CASSANDRA_CONFIG_FILE.bak
```

### Update concurrent_compactors property
```shell
sed -i "s/^concurrent_compactors:.*$/concurrent_compactors: $CONCURRENT_COMPACTORS/" $CASSANDRA_CONFIG_FILE
```

### Restart cassandra service
```shell
systemctl restart cassandra.service
```

### Next Step
```shell
echo "Compaction threads increased to $CONCURRENT_COMPACTORS."
```

### Reduce the size of SSTables in the cluster by increasing the frequency of compaction or by manually triggering it. This will help reduce the number of pending tasks and improve performance.
```shell
bash

#!/bin/bash

# Set the Cassandra home directory

CASSANDRA_HOME="PLACEHOLDER"

# Set the maximum number of pending compaction tasks allowed before triggering manual compaction

MAX_PENDING_COMPACTION_TASKS="PLACEHOLDER"

# Get the current number of pending compaction tasks

PENDING_COMPACTION_TASKS=$(nodetool compactionstats | grep pending | awk '{print $NF}')

# If the number of pending compaction tasks is greater than the maximum allowed, trigger manual compaction

if [ "$PENDING_COMPACTION_TASKS" -gt "$MAX_PENDING_COMPACTION_TASKS" ]; then

    echo "Too many pending compaction tasks. Triggering manual compaction..."

    nodetool compact

fi

# If the number of pending compaction tasks is within the allowed limit, increase the frequency of compaction

echo "Increasing the frequency of compaction..."

sed -i 's/^# auto_snapshot: .*$/auto_snapshot: true/' $CASSANDRA_HOME/conf/cassandra.yaml

sed -i 's/^# compaction_throughput_mb_per_sec: .*$/compaction_throughput_mb_per_sec: ${NEW_COMPACTION_THROUGHPUT}/' $CASSANDRA_HOME/conf/cassandra.yaml

# Restart Cassandra to apply the changes

echo "Restarting Cassandra..."

sudo service cassandra restart

echo "Compaction remediation complete."

```

This incident type occurs when there are many pending compaction tasks in a Cassandra cluster. Compaction is the process of merging and removing data from SSTables (sorted string tables) in Cassandra. When compaction tasks are pending, it means that the process is not completing in a timely manner and is causing performance issues in the cluster. This incident requires investigation and resolution to ensure the cluster's health and performance.


This incident type refers to an alert triggered by a monitoring system indicating that the number of pending tasks in ElasticSearch is high. This can be an issue because it may indicate that the system is overloaded and unable to process all the incoming tasks, which can result in performance degradation or even downtime. The incident needs to be investigated and resolved as soon as possible to ensure the system is functioning properly.


High number of pending tasks in ElasticSearch.

Compaction is a process in Cassandra that merges multiple SSTables (Sorted String Tables) into a single SSTable, eliminating any redundant data and improving read performance. However, sometimes compaction can fail due to various reasons such as insufficient disk space or corrupted data, resulting in degraded performance or even complete failure of the database. Troubleshooting compaction merging SSTables involves identifying and resolving the root cause of such failures to ensure the smooth functioning of the Cassandra database.


Troubleshooting Compaction Merging SSTables in Cassandra

This incident type refers to a situation where there is a significant delay in the execution of queries on a Cassandra cluster. This delay can cause the system to become unresponsive and result in slower performance. It may be caused by a variety of factors such as an increase in traffic, inefficient queries, or hardware issues. The issue can impact the functionality of the system and requires immediate attention to prevent further disruption.


Slow Query Performance on Cassandra Cluster.

In this incident type, there is an issue with a Cassandra cluster where one or more disks are running slow. This can cause performance issues and potentially lead to data loss or downtime. The goal is to identify and address the specific disk(s) causing the problem in order to restore normal cluster operations.


Slow Disk in Cassandra Cluster

Misconfigured compaction strategy is an incident type that occurs when the way data is compacted within a database cluster is not properly configured. This can lead to excessive compaction activity that can negatively impact the cluster's performance and stability. The incident can cause slow query response times, increased disk usage, and high CPU utilization, among other issues. It is crucial to identify and fix the misconfiguration promptly to ensure the cluster's smooth operation.


Misconfigured Compaction Strategy.

```shell
# Environment Variables

export CASSANDRA_NODE_IP="PLACEHOLDER"

export PATH_TO_CASSANDRA_YAML="PLACEHOLDER"

export NUMBER_OF_COMPACTION_THREADS="PLACEHOLDER"

export CASSANDRA_DATA_DIRECTORY="PLACEHOLDER"

export MEMORY_THRESHOLD="PLACEHOLDER"

export NEW_COMPACTION_THROUGHPUT="PLACEHOLDER"
```


### Check the status of the Cassandra service

```shell
systemctl status cassandra
```

### Check the number of pending compaction tasks in Cassandra

```shell
nodetool compactionstats | grep pending_tasks
```

### Check the health of the Cassandra cluster

```shell
nodetool status
```

### Check the Cassandra logs for errors or warnings

```shell
tail -n 100 /var/log/cassandra/system.log
```

### Check the system load and CPU usage

```shell
top
```

### Check the disk usage and available space

```shell
df -h
```

### Check the network connectivity and latency

```shell
ping ${CASSANDRA_NODE_IP}
```

### Check the firewall rules and open ports

```shell
iptables -L -n
```

### Insufficient resources such as CPU, memory, or disk space, causing compaction to slow down or stop.

```shell


#!/bin/bash


 CPU_THRESHOLD="PLACEHOLDER"

 DISK_THRESHOLD="PLACEHOLDER"
# Check CPU usage

cpu_usage=$(top -b -n1 | grep "Cpu(s)" | awk '{print $2 + $4}')

if (( $(echo "$cpu_usage > ${CPU_THRESHOLD}" | bc -l) )); then

  echo "CPU usage is high ($cpu_usage%), which may be causing compaction to slow down or stop."

fi



# Check memory usage

mem_usage=$(free | awk '/Mem/{printf("%.2f"), $3/$2*100}')

if (( $(echo "$mem_usage > ${MEMORY_THRESHOLD}" | bc -l) )); then

  echo "Memory usage is high ($mem_usage%), which may be causing compaction to slow down or stop."

fi



# Check disk space

disk_usage=$(df -h ${CASSANDRA_DATA_DIRECTORY} | tail -1 | awk '{print $5}' | tr -d '%')

if (( $disk_usage > ${DISK_THRESHOLD} )); then

  echo "Disk space usage is high ($disk_usage%), which may be causing compaction to slow down or stop."

fi


```


### Set variables

```shell
CASSANDRA_CONFIG_FILE=${PATH_TO_CASSANDRA_YAML}

CONCURRENT_COMPACTORS=${NUMBER_OF_COMPACTION_THREADS}
```

### Check if cassandra config file exists

```shell
if [ ! -f $CASSANDRA_CONFIG_FILE ]; then

  echo "Cassandra config file not found: $CASSANDRA_CONFIG_FILE"

  exit 1

fi
```

### Backup cassandra config file

```shell
cp $CASSANDRA_CONFIG_FILE $CASSANDRA_CONFIG_FILE.bak
```

### Update concurrent\_compactors property

```shell
sed -i "s/^concurrent_compactors:.*$/concurrent_compactors: $CONCURRENT_COMPACTORS/" $CASSANDRA_CONFIG_FILE
```

### Restart cassandra service

```shell
systemctl restart cassandra.service
```

### Next Step

```shell
echo "Compaction threads increased to $CONCURRENT_COMPACTORS."
```

### Reduce the size of SSTables in the cluster by increasing the frequency of compaction or by manually triggering it. This will help reduce the number of pending tasks and improve performance.

```shell
bash

#!/bin/bash

# Set the Cassandra home directory

CASSANDRA_HOME="PLACEHOLDER"

# Set the maximum number of pending compaction tasks allowed before triggering manual compaction

MAX_PENDING_COMPACTION_TASKS="PLACEHOLDER"

# Get the current number of pending compaction tasks

PENDING_COMPACTION_TASKS=$(nodetool compactionstats | grep pending | awk '{print $NF}')

# If the number of pending compaction tasks is greater than the maximum allowed, trigger manual compaction

if [ "$PENDING_COMPACTION_TASKS" -gt "$MAX_PENDING_COMPACTION_TASKS" ]; then

    echo "Too many pending compaction tasks. Triggering manual compaction..."

    nodetool compact

fi

# If the number of pending compaction tasks is within the allowed limit, increase the frequency of compaction

echo "Increasing the frequency of compaction..."

sed -i 's/^# auto_snapshot: .*$/auto_snapshot: true/' $CASSANDRA_HOME/conf/cassandra.yaml

sed -i 's/^# compaction_throughput_mb_per_sec: .*$/compaction_throughput_mb_per_sec: ${NEW_COMPACTION_THROUGHPUT}/' $CASSANDRA_HOME/conf/cassandra.yaml

# Restart Cassandra to apply the changes

echo "Restarting Cassandra..."

sudo service cassandra restart

echo "Compaction remediation complete."

```


Cassandra many compaction tasks are pending.

Overview

Parameters

Debug

Check the status of the Cassandra service

Check the number of pending compaction tasks in Cassandra

Check the health of the Cassandra cluster

Check the Cassandra logs for errors or warnings

Check the system load and CPU usage

Check the disk usage and available space

Check the network connectivity and latency

Check the firewall rules and open ports

Insufficient resources such as CPU, memory, or disk space, causing compaction to slow down or stop.

Repair

Set variables

Check if cassandra config file exists

Backup cassandra config file

Update concurrent_compactors property

Restart cassandra service

Next Step

Reduce the size of SSTables in the cluster by increasing the frequency of compaction or by manually triggering it. This will help reduce the number of pending tasks and improve performance.

Learn more

Related Runbooks

High number of pending tasks in ElasticSearch.

Troubleshooting Compaction Merging SSTables in Cassandra

Slow Query Performance on Cassandra Cluster.

Slow Disk in Cassandra Cluster

Support