---
id: 553ca622-616d-11ee-8c99-0242ac120002
---

# Excessive Load on Cassandra
---

Excessive load on Cassandra incident refers to a situation where there is an overload of requests or data processing on the Cassandra database management system. This can cause the system to slow down or even crash, resulting in service disruptions and downtime. This incident can occur due to various factors such as increased user traffic, inefficient queries, or hardware issues.

### Parameters
```shell
export CASSANDRA_NODE="PLACEHOLDER"

export KEYSPACE="PLACEHOLDER"

export KEYSPACE_NAME="PLACEHOLDER"

export TABLE_NAME="PLACEHOLDER"

export NEW_NODE_IP="PLACEHOLDER"

export NEW_NODE_NAME="PLACEHOLDER"

export CASSANDRA_NODE_IP="PLACEHOLDER"

export DATACENTER_NAME="PLACEHOLDER"
```

## Debug

### Check the number of connections to Cassandra
```shell
ssh ${CASSANDRA_NODE} "nodetool info | grep 'Native Transport'"
```

### Check the disk usage on the Cassandra nodes
```shell
ssh ${CASSANDRA_NODE} "df -h"
```

### Check the CPU usage on the Cassandra nodes
```shell
ssh ${CASSANDRA_NODE} "top -b -n 1 | grep Cpu"
```

### Check the read/write latencies on Cassandra
```shell
ssh ${CASSANDRA_NODE} "nodetool cfstats | grep -A 10 'Keyspace: ${KEYSPACE}'"
```

### Check the Cassandra system log
```shell
ssh ${CASSANDRA_NODE} "tail -n 100 /var/log/cassandra/system.log"
```

### Increased traffic on the application leading to a higher number of requests to the Cassandra database.
```shell


#!/bin/bash



# Set variables

KEYSPACE=${KEYSPACE_NAME}

TABLE=${TABLE_NAME}

DATE=$(date +"%Y-%m-%d_%H-%M-%S")

LOG_FILE=/var/log/cassandra/diagnostic-$DATE.log

THRESHOLD=100



# Log start of diagnostic process

echo "Starting diagnostic process at $(date)" >> $LOG_FILE



# Check for increased traffic on the application

REQUEST_COUNT=$(nodetool cfstats $KEYSPACE.$TABLE | grep "Read Count" | awk '{print $3}')

if [ "$REQUEST_COUNT" -gt "$THRESHOLD" ]; then

    echo "Increased traffic detected. Request count: $REQUEST_COUNT" >> $LOG_FILE



    # Check for slow queries

    SLOW_QUERIES=$(nodetool cfstats $KEYSPACE.$TABLE | grep "Mean Row" | awk '{print $4}')

    if [ "$SLOW_QUERIES" -gt "$THRESHOLD" ]; then

        echo "Slow queries detected. Mean row size: $SLOW_QUERIES" >> $LOG_FILE

    fi



    # Check for node health

    NODE_STATUS=$(nodetool status | grep "UN")

    if [ -z "$NODE_STATUS" ]; then

        echo "Node(s) down or not responding." >> $LOG_FILE

    else

        echo "All nodes healthy." >> $LOG_FILE

    fi

else

    echo "No increased traffic detected. Request count: $REQUEST_COUNT" >> $LOG_FILE

fi



# Log end of diagnostic process

echo "Diagnostic process completed at $(date)" >> $LOG_FILE


```

## Repair

### Scale up the Cassandra cluster by adding more nodes to handle the increased load.
```shell


#!/bin/bash



# Set variables for the new node to be added

NEW_NODE_IP=${NEW_NODE_IP}

NEW_NODE_NAME=${NEW_NODE_NAME}



# SSH into the existing Cassandra nodes to add the new node

ssh ${CASSANDRA_NODE_IP} "nodetool add ${NEW_NODE_IP}; echo '${NEW_NODE_NAME}' >> /etc/hosts"


# Wait for the new node to join the cluster

sleep 30



# Check the status of the new node

nodetool status | grep ${NEW_NODE_NAME}



# Verify that the cluster is balanced

nodetool status | grep -A 1 "Datacenter: ${DATACENTER_NAME}" | tail -n 1 | awk '{print $NF}'


```


Excessive load on Cassandra incident refers to a situation where there is an overload of requests or data processing on the Cassandra database management system. This can cause the system to slow down or even crash, resulting in service disruptions and downtime. This incident can occur due to various factors such as increased user traffic, inefficient queries, or hardware issues.


This incident type refers to a situation where there is a significant delay in the execution of queries on a Cassandra cluster. This delay can cause the system to become unresponsive and result in slower performance. It may be caused by a variety of factors such as an increase in traffic, inefficient queries, or hardware issues. The issue can impact the functionality of the system and requires immediate attention to prevent further disruption.


Slow Query Performance on Cassandra Cluster.

This incident type refers to a performance degradation issue in a Cassandra database caused by the new generation garbage collection (GC) process. Garbage collection is a process by which the unused memory occupied by objects in a program is cleared. In some cases, the new generation GC process may take longer than usual, causing performance issues in the Cassandra database. This could lead to slowness in queries or even crashing of the database.


Performance Degradation Due to New Generation Garbage Collection on Cassandra

This incident type refers to the report of overheating by Prometheus exporter at a specific instance for the Cassandra service. It could be caused by a high volume of traffic or some issues with the server, and it requires immediate attention to avoid any potential downtime or data loss. The incident is assigned to an engineer who will investigate and resolve the issue as quickly as possible.


Overheating reported by Prometheus exporter at {{ $labels.instance }} for Cassandra service.

This incident type refers to a situation where a high number of mutations are being dropped on a Cassandra database. Mutations are changes made to the database, such as inserting new data or updating existing data. When mutations are dropped, it means that they were not successfully recorded in the database. This can be caused by a variety of factors, such as hardware or network issues, configuration problems, or bugs in the software. When this occurs, it can result in data inconsistencies or loss, and can impact the performance and reliability of the application that relies on the database.


High Number of Dropped Mutations on Cassandra Database

This incident type refers to issues that occur in Cassandra due to a high replication factor. Replication factor is a setting that determines how many copies of data are stored in a cluster. A high replication factor means that more copies of data are stored in the cluster, which can cause performance issues and potentially lead to data inconsistencies. When this incident occurs, it may affect the availability and stability of the Cassandra cluster, impacting the ability to read and write data.


High Replication Factor Issues in Cassandra

```shell
export CASSANDRA_NODE="PLACEHOLDER"

export KEYSPACE="PLACEHOLDER"

export KEYSPACE_NAME="PLACEHOLDER"

export TABLE_NAME="PLACEHOLDER"

export NEW_NODE_IP="PLACEHOLDER"

export NEW_NODE_NAME="PLACEHOLDER"

export CASSANDRA_NODE_IP="PLACEHOLDER"

export DATACENTER_NAME="PLACEHOLDER"
```


### Check the number of connections to Cassandra

```shell
ssh ${CASSANDRA_NODE} "nodetool info | grep 'Native Transport'"
```

### Check the disk usage on the Cassandra nodes

```shell
ssh ${CASSANDRA_NODE} "df -h"
```

### Check the CPU usage on the Cassandra nodes

```shell
ssh ${CASSANDRA_NODE} "top -b -n 1 | grep Cpu"
```

### Check the read/write latencies on Cassandra

```shell
ssh ${CASSANDRA_NODE} "nodetool cfstats | grep -A 10 'Keyspace: ${KEYSPACE}'"
```

### Check the Cassandra system log

```shell
ssh ${CASSANDRA_NODE} "tail -n 100 /var/log/cassandra/system.log"
```

### Increased traffic on the application leading to a higher number of requests to the Cassandra database.

```shell


#!/bin/bash



# Set variables

KEYSPACE=${KEYSPACE_NAME}

TABLE=${TABLE_NAME}

DATE=$(date +"%Y-%m-%d_%H-%M-%S")

LOG_FILE=/var/log/cassandra/diagnostic-$DATE.log

THRESHOLD=100



# Log start of diagnostic process

echo "Starting diagnostic process at $(date)" >> $LOG_FILE



# Check for increased traffic on the application

REQUEST_COUNT=$(nodetool cfstats $KEYSPACE.$TABLE | grep "Read Count" | awk '{print $3}')

if [ "$REQUEST_COUNT" -gt "$THRESHOLD" ]; then

    echo "Increased traffic detected. Request count: $REQUEST_COUNT" >> $LOG_FILE



    # Check for slow queries

    SLOW_QUERIES=$(nodetool cfstats $KEYSPACE.$TABLE | grep "Mean Row" | awk '{print $4}')

    if [ "$SLOW_QUERIES" -gt "$THRESHOLD" ]; then

        echo "Slow queries detected. Mean row size: $SLOW_QUERIES" >> $LOG_FILE

    fi



    # Check for node health

    NODE_STATUS=$(nodetool status | grep "UN")

    if [ -z "$NODE_STATUS" ]; then

        echo "Node(s) down or not responding." >> $LOG_FILE

    else

        echo "All nodes healthy." >> $LOG_FILE

    fi

else

    echo "No increased traffic detected. Request count: $REQUEST_COUNT" >> $LOG_FILE

fi



# Log end of diagnostic process

echo "Diagnostic process completed at $(date)" >> $LOG_FILE


```


### Scale up the Cassandra cluster by adding more nodes to handle the increased load.

```shell


#!/bin/bash



# Set variables for the new node to be added

NEW_NODE_IP=${NEW_NODE_IP}

NEW_NODE_NAME=${NEW_NODE_NAME}



# SSH into the existing Cassandra nodes to add the new node

ssh ${CASSANDRA_NODE_IP} "nodetool add ${NEW_NODE_IP}; echo '${NEW_NODE_NAME}' >> /etc/hosts"


# Wait for the new node to join the cluster

sleep 30



# Check the status of the new node

nodetool status | grep ${NEW_NODE_NAME}



# Verify that the cluster is balanced

nodetool status | grep -A 1 "Datacenter: ${DATACENTER_NAME}" | tail -n 1 | awk '{print $NF}'


```


Excessive Load on Cassandra

Overview

Parameters

Debug

Check the number of connections to Cassandra

Check the disk usage on the Cassandra nodes

Check the CPU usage on the Cassandra nodes

Check the read/write latencies on Cassandra

Check the Cassandra system log

Increased traffic on the application leading to a higher number of requests to the Cassandra database.

Repair

Scale up the Cassandra cluster by adding more nodes to handle the increased load.

Learn more

Related Runbooks

Slow Query Performance on Cassandra Cluster.

Performance Degradation Due to New Generation Garbage Collection on Cassandra

Overheating reported by Prometheus exporter at {{ $labels.instance }} for Cassandra service.

High Number of Dropped Mutations on Cassandra Database

Support