---
id: a56b2d35-bd0f-46ae-8882-6dceb29a0ce7
---

# Cassandra storage exceptions incident.
---

This incident type refers to an issue related to the Cassandra storage system. It indicates that there are exceptions occurring in the Cassandra storage system, which is causing problems with the functioning of the system. This can have a significant impact on the performance of the system and may require urgent attention to resolve the issue.

### Parameters
```shell
# Environment Variables

export KEYSPACE="PLACEHOLDER"

export TABLE="PLACEHOLDER"

export THREAD_POOL_NAME="PLACEHOLDER"

export CASSANDRA_NODE_IP="PLACEHOLDER"

export APPLICATION_LOG_FILE="PLACEHOLDER"

```

## Debug

### Check if Cassandra is running
```shell
systemctl status cassandra
```

### Check the Cassandra logs for any errors
```shell
tail -f /var/log/cassandra/system.log
```

### Check the status of the Cassandra cluster
```shell
nodetool status
```

### Check the disk space usage on each node in the cluster
```shell
nodetool status | awk '{print $2}' | xargs -I {} ssh {} df -h
```

### Check the status of the Cassandra storage on each node in the cluster
```shell
nodetool status | awk '{print $2}' | xargs -I {} ssh {} nodetool cfstats | grep -A 3 "Table: ${KEYSPACE}.${TABLE}"
```

### Check the metrics for the Cassandra cluster
```shell
nodetool cfstats -H
```

### Check the load on each node in the cluster
```shell
nodetool tpstats | grep "${THREAD_POOL_NAME}"
```

### Check the replication factor of the keyspace
```shell
cqlsh -e "describe keyspace ${KEYSPACE}"
```

### Check the schema of the table
```shell
cqlsh -e "describe table ${KEYSPACE}.${TABLE}"
```

## Repair

### Step 1: Analyze system logs for errors related to Cassandra storage
```shell
grep "Cassandra storage" /var/log/syslog > cassandra_errors.txt
```

### Step 2: Review Cassandra configuration settings
```shell
cat /etc/cassandra/cassandra.yaml > cassandra_config.txt
```

### Step 3: Verify network connectivity for Cassandra nodes
```shell
ping ${CASSANDRA_NODE_IP}
```

### Step 4: Check hardware resources of each Cassandra node
```shell
ssh ${CASSANDRA_NODE_IP} "free -h && cat /proc/cpuinfo"
```

### Step 5: Review query patterns and data access patterns of the application
```shell
cat ${APPLICATION_LOG_FILE} | grep "Cassandra" > cassandra_queries.txt
```

### Try restarting the Cassandra service to see if the issue resolves itself. If restarting the service does not work, consider performing a rolling restart of Cassandra across the cluster.
```shell


#!/bin/bash



# Stop Cassandra service

sudo service cassandra stop



# Wait for 30 seconds to allow Cassandra to fully stop

sleep 30



# Start Cassandra service again

sudo service cassandra start



# Wait for 30 seconds to allow Cassandra to fully start

sleep 30



# Check if Cassandra service is running

if sudo service cassandra status | grep -q "is running"; then

  echo "Cassandra service has been restarted successfully"

else

  # If Cassandra service is not running, perform a rolling restart across the cluster

  echo "Cassandra service failed to restart. Performing a rolling restart across the cluster..."



  # Get list of nodes in the cluster

  nodes="PLACEHOLDER"



  # Loop through each node and restart Cassandra service

  for node in $nodes; do

    # Stop Cassandra service on the node

    ssh $node "sudo service cassandra stop"



    # Wait for 30 seconds to allow Cassandra to fully stop

    sleep 30



    # Start Cassandra service on the node

    ssh $node "sudo service cassandra start"



    # Wait for 30 seconds to allow Cassandra to fully start

    sleep 30



    # Check if Cassandra service is running on the node

    if ssh $node "sudo service cassandra status" | grep -q "is running"; then

      echo "Cassandra service has been restarted successfully on $node"

    else

      echo "Failed to restart Cassandra service on $node"

    fi

  done



  echo "Rolling restart of Cassandra across the cluster has been completed"

fi


```

This incident type refers to an issue related to the Cassandra storage system. It indicates that there are exceptions occurring in the Cassandra storage system, which is causing problems with the functioning of the system. This can have a significant impact on the performance of the system and may require urgent attention to resolve the issue.


This incident type refers to a situation where there is a significant delay in the execution of queries on a Cassandra cluster. This delay can cause the system to become unresponsive and result in slower performance. It may be caused by a variety of factors such as an increase in traffic, inefficient queries, or hardware issues. The issue can impact the functionality of the system and requires immediate attention to prevent further disruption.


Slow Query Performance on Cassandra Cluster.

In this incident type, there is an issue with a Cassandra cluster where one or more disks are running slow. This can cause performance issues and potentially lead to data loss or downtime. The goal is to identify and address the specific disk(s) causing the problem in order to restore normal cluster operations.


Slow Disk in Cassandra Cluster

This incident type refers to a performance degradation issue in a Cassandra database caused by the new generation garbage collection (GC) process. Garbage collection is a process by which the unused memory occupied by objects in a program is cleared. In some cases, the new generation GC process may take longer than usual, causing performance issues in the Cassandra database. This could lead to slowness in queries or even crashing of the database.


Performance Degradation Due to New Generation Garbage Collection on Cassandra

This incident type refers to a situation where a delay or slowness occurs in a system that uses Cassandra database due to the shared storage. Shared storage means multiple servers are accessing the same storage unit, and this can cause latency issues. This type of incident can lead to performance degradation, and it needs to be addressed promptly to ensure optimal system performance.


Latency Caused by Shared Storage in Cassandra

This incident type refers to a situation where a high number of mutations are being dropped on a Cassandra database. Mutations are changes made to the database, such as inserting new data or updating existing data. When mutations are dropped, it means that they were not successfully recorded in the database. This can be caused by a variety of factors, such as hardware or network issues, configuration problems, or bugs in the software. When this occurs, it can result in data inconsistencies or loss, and can impact the performance and reliability of the application that relies on the database.


High Number of Dropped Mutations on Cassandra Database

```shell
# Environment Variables

export KEYSPACE="PLACEHOLDER"

export TABLE="PLACEHOLDER"

export THREAD_POOL_NAME="PLACEHOLDER"

export CASSANDRA_NODE_IP="PLACEHOLDER"

export APPLICATION_LOG_FILE="PLACEHOLDER"

```


### Check if Cassandra is running

```shell
systemctl status cassandra
```

### Check the Cassandra logs for any errors

```shell
tail -f /var/log/cassandra/system.log
```

### Check the status of the Cassandra cluster

```shell
nodetool status
```

### Check the disk space usage on each node in the cluster

```shell
nodetool status | awk '{print $2}' | xargs -I {} ssh {} df -h
```

### Check the status of the Cassandra storage on each node in the cluster

```shell
nodetool status | awk '{print $2}' | xargs -I {} ssh {} nodetool cfstats | grep -A 3 "Table: ${KEYSPACE}.${TABLE}"
```

### Check the metrics for the Cassandra cluster

```shell
nodetool cfstats -H
```

### Check the load on each node in the cluster

```shell
nodetool tpstats | grep "${THREAD_POOL_NAME}"
```

### Check the replication factor of the keyspace

```shell
cqlsh -e "describe keyspace ${KEYSPACE}"
```

### Check the schema of the table

```shell
cqlsh -e "describe table ${KEYSPACE}.${TABLE}"
```


### Step 1: Analyze system logs for errors related to Cassandra storage

```shell
grep "Cassandra storage" /var/log/syslog > cassandra_errors.txt
```

### Step 2: Review Cassandra configuration settings

```shell
cat /etc/cassandra/cassandra.yaml > cassandra_config.txt
```

### Step 3: Verify network connectivity for Cassandra nodes

```shell
ping ${CASSANDRA_NODE_IP}
```

### Step 4: Check hardware resources of each Cassandra node

```shell
ssh ${CASSANDRA_NODE_IP} "free -h && cat /proc/cpuinfo"
```

### Step 5: Review query patterns and data access patterns of the application

```shell
cat ${APPLICATION_LOG_FILE} | grep "Cassandra" > cassandra_queries.txt
```

### Try restarting the Cassandra service to see if the issue resolves itself. If restarting the service does not work, consider performing a rolling restart of Cassandra across the cluster.

```shell


#!/bin/bash



# Stop Cassandra service

sudo service cassandra stop



# Wait for 30 seconds to allow Cassandra to fully stop

sleep 30



# Start Cassandra service again

sudo service cassandra start



# Wait for 30 seconds to allow Cassandra to fully start

sleep 30



# Check if Cassandra service is running

if sudo service cassandra status | grep -q "is running"; then

  echo "Cassandra service has been restarted successfully"

else

  # If Cassandra service is not running, perform a rolling restart across the cluster

  echo "Cassandra service failed to restart. Performing a rolling restart across the cluster..."



  # Get list of nodes in the cluster

  nodes="PLACEHOLDER"



  # Loop through each node and restart Cassandra service

  for node in $nodes; do

    # Stop Cassandra service on the node

    ssh $node "sudo service cassandra stop"



    # Wait for 30 seconds to allow Cassandra to fully stop

    sleep 30



    # Start Cassandra service on the node

    ssh $node "sudo service cassandra start"



    # Wait for 30 seconds to allow Cassandra to fully start

    sleep 30



    # Check if Cassandra service is running on the node

    if ssh $node "sudo service cassandra status" | grep -q "is running"; then

      echo "Cassandra service has been restarted successfully on $node"

    else

      echo "Failed to restart Cassandra service on $node"

    fi

  done



  echo "Rolling restart of Cassandra across the cluster has been completed"

fi


```


Cassandra storage exceptions incident.

Overview

Parameters

Debug

Check if Cassandra is running

Check the Cassandra logs for any errors

Check the status of the Cassandra cluster

Check the disk space usage on each node in the cluster

Check the status of the Cassandra storage on each node in the cluster

Check the metrics for the Cassandra cluster

Check the load on each node in the cluster

Check the replication factor of the keyspace

Check the schema of the table

Repair

Step 2: Review Cassandra configuration settings

Step 3: Verify network connectivity for Cassandra nodes

Step 4: Check hardware resources of each Cassandra node

Step 5: Review query patterns and data access patterns of the application

Try restarting the Cassandra service to see if the issue resolves itself. If restarting the service does not work, consider performing a rolling restart of Cassandra across the cluster.

Learn more

Related Runbooks

Slow Query Performance on Cassandra Cluster.

Slow Disk in Cassandra Cluster

Performance Degradation Due to New Generation Garbage Collection on Cassandra

Latency Caused by Shared Storage in Cassandra

Support