---
id: fe4a0d30-616e-11ee-8c99-0242ac120002
---

# Excessive Mutations in Cassandra Due to Full Reads and Writes
---

This incident type describes a problem where the number of mutations in a Cassandra database has reached an excessive level. This is usually caused by full reads and writes, which can lead to an increased number of changes within the database. Excessive mutations can cause performance issues and impact the overall stability of the system. Therefore, it is important to resolve this issue as soon as possible to ensure the system is running smoothly.

### Parameters
```shell
export KEYSPACE_TABLE="PLACEHOLDER"

export DATABASE_NAME="PLACEHOLDER"

export SCHEMA_NAME="PLACEHOLDER"

export CACHE_NAME="PLACEHOLDER"

export STORAGE_NAME="PLACEHOLDER"
```

## Debug

### Check if Cassandra is running
```shell
systemctl status cassandra.service
```

### Check Cassandra logs for errors
```shell
journalctl -u cassandra.service | grep ERROR
```

### Check Cassandra compaction history
```shell
nodetool compactionhistory
```

### Check Cassandra table statistics
```shell
nodetool tablestats ${KEYSPACE_TABLE}
```

### Check Cassandra node information
```shell
nodetool info
```

### Check Cassandra system statistics
```shell
nodetool cfstats
```

### Check for high I/O wait time
```shell
iostat -x 1
```

### Check for high CPU usage
```shell
top
```

## Repair

### Optimize Cassandra query performance to reduce the number of full reads and writes, and minimize the number of mutations caused by these operations. This can involve restructuring the database schema, introducing caching layers, or using alternative data storage solutions.
```shell


#!/bin/bash



# Set variables

database=${DATABASE_NAME}

schema=${SCHEMA_NAME}

cache=${CACHE_NAME}

storage=${STORAGE_NAME}



# Optimize schema

cqlsh -e "ALTER TABLE $database.$schema WITH compaction = {'class': 'LeveledCompactionStrategy'} AND compression = {'sstable_compression': 'LZ4Compressor'};"



# Introduce caching layer

cqlsh -e "CREATE $cache;"



# Use alternative data storage

cqlsh -e "CREATE $storage;"



# Restart Cassandra service

sudo service cassandra restart


```


This incident type describes a problem where the number of mutations in a Cassandra database has reached an excessive level. This is usually caused by full reads and writes, which can lead to an increased number of changes within the database. Excessive mutations can cause performance issues and impact the overall stability of the system. Therefore, it is important to resolve this issue as soon as possible to ensure the system is running smoothly.


This incident type refers to a problem in a Cassandra cluster where the token range imbalances cause uneven distribution of data across the cluster. This can result in slower read and write performance that can impact the overall functionality of the system. Token range imbalances occur when the distribution of the tokens that define the ranges of data each node is responsible for is not evenly spread across the cluster. As a result, certain nodes may be responsible for a disproportionate amount of data, leading to performance issues and potential failure of the system.


Token Range Imbalances Causing Uneven Data Distribution and Performance Issues in Cassandra Cluster

This incident type refers to a situation where there is a significant delay in the execution of queries on a Cassandra cluster. This delay can cause the system to become unresponsive and result in slower performance. It may be caused by a variety of factors such as an increase in traffic, inefficient queries, or hardware issues. The issue can impact the functionality of the system and requires immediate attention to prevent further disruption.


Slow Query Performance on Cassandra Cluster.

In this incident type, there is an issue with a Cassandra cluster where one or more disks are running slow. This can cause performance issues and potentially lead to data loss or downtime. The goal is to identify and address the specific disk(s) causing the problem in order to restore normal cluster operations.


Slow Disk in Cassandra Cluster

This incident type refers to a performance degradation issue in a Cassandra database caused by the new generation garbage collection (GC) process. Garbage collection is a process by which the unused memory occupied by objects in a program is cleared. In some cases, the new generation GC process may take longer than usual, causing performance issues in the Cassandra database. This could lead to slowness in queries or even crashing of the database.


Performance Degradation Due to New Generation Garbage Collection on Cassandra

This incident type refers to a situation where a delay or slowness occurs in a system that uses Cassandra database due to the shared storage. Shared storage means multiple servers are accessing the same storage unit, and this can cause latency issues. This type of incident can lead to performance degradation, and it needs to be addressed promptly to ensure optimal system performance.


Latency Caused by Shared Storage in Cassandra

```shell
export KEYSPACE_TABLE="PLACEHOLDER"

export DATABASE_NAME="PLACEHOLDER"

export SCHEMA_NAME="PLACEHOLDER"

export CACHE_NAME="PLACEHOLDER"

export STORAGE_NAME="PLACEHOLDER"
```


### Check if Cassandra is running

```shell
systemctl status cassandra.service
```

### Check Cassandra logs for errors

```shell
journalctl -u cassandra.service | grep ERROR
```

### Check Cassandra compaction history

```shell
nodetool compactionhistory
```

### Check Cassandra table statistics

```shell
nodetool tablestats ${KEYSPACE_TABLE}
```

### Check Cassandra node information

```shell
nodetool info
```

### Check Cassandra system statistics

```shell
nodetool cfstats
```

### Check for high I/O wait time

```shell
iostat -x 1
```

### Check for high CPU usage

```shell
top
```


### Optimize Cassandra query performance to reduce the number of full reads and writes, and minimize the number of mutations caused by these operations. This can involve restructuring the database schema, introducing caching layers, or using alternative data storage solutions.

```shell


#!/bin/bash



# Set variables

database=${DATABASE_NAME}

schema=${SCHEMA_NAME}

cache=${CACHE_NAME}

storage=${STORAGE_NAME}



# Optimize schema

cqlsh -e "ALTER TABLE $database.$schema WITH compaction = {'class': 'LeveledCompactionStrategy'} AND compression = {'sstable_compression': 'LZ4Compressor'};"



# Introduce caching layer

cqlsh -e "CREATE $cache;"



# Use alternative data storage

cqlsh -e "CREATE $storage;"



# Restart Cassandra service

sudo service cassandra restart


```


Excessive Mutations in Cassandra Due to Full Reads and Writes

Overview

Parameters

Debug

Check if Cassandra is running

Check Cassandra logs for errors

Check Cassandra compaction history

Check Cassandra table statistics

Check Cassandra node information

Check Cassandra system statistics

Check for high I/O wait time

Check for high CPU usage

Repair

Optimize Cassandra query performance to reduce the number of full reads and writes, and minimize the number of mutations caused by these operations. This can involve restructuring the database schema, introducing caching layers, or using alternative data storage solutions.

Learn more

Related Runbooks

Token Range Imbalances Causing Uneven Data Distribution and Performance Issues in Cassandra Cluster

Slow Query Performance on Cassandra Cluster.

Slow Disk in Cassandra Cluster

Performance Degradation Due to New Generation Garbage Collection on Cassandra

Support