---
id: 35e916b0-6150-11ee-8c99-0242ac120002
---


# High Number of Dropped Mutations on Cassandra Database
---

This incident type refers to a situation where a high number of mutations are being dropped on a Cassandra database. Mutations are changes made to the database, such as inserting new data or updating existing data. When mutations are dropped, it means that they were not successfully recorded in the database. This can be caused by a variety of factors, such as hardware or network issues, configuration problems, or bugs in the software. When this occurs, it can result in data inconsistencies or loss, and can impact the performance and reliability of the application that relies on the database.

### Parameters
```shell
export KEYSPACE_NAME="PLACEHOLDER"

export TABLE_NAME="PLACEHOLDER"

export COLUMN_NAME="PLACEHOLDER"

export VALUE="PLACEHOLDER"

export CONSISTENCY_LEVEL="PLACEHOLDER"

export NEW_NODE_COUNT="PLACEHOLDER"

export NEW_REPLICATION_FACTOR="PLACEHOLDER"

export CASSANDRA_CLUSTER="PLACEHOLDER"
```

## Debug

### Check the status of Cassandra service
```shell
systemctl status cassandra
```

### Check the Cassandra log for any errors or warnings
```shell
tail -n 100 /var/log/cassandra/system.log
```

### Check the number of dropped mutations
```shell
nodetool tpstats | grep Dropped
```

### Check the status of the Cassandra node
```shell
nodetool status
```

### Check the load on the Cassandra node
```shell
nodetool info | grep Load
```

### Check the disk space usage on the Cassandra node
```shell
df -h
```

### Check the network connectivity between the Cassandra nodes
```shell
nodetool describecluster
```

### Check the replication factor for the affected keyspace
```shell
cqlsh -e "describe keyspace ${KEYSPACE_NAME};"
```

### Check the consistency level of the affected queries
```shell
cqlsh -e "select * from ${TABLE_NAME} where ${COLUMN_NAME} = ${VALUE} consistency ${CONSISTENCY_LEVEL};"
```

## Repair

### Increase the number of Cassandra nodes to handle the increased load and ensure that there is proper replication factor to avoid any data loss.
```shell


#!/bin/bash



# Define the number of nodes and replication factor

CASSANDRA_NODES=${NUMBER_OF_NODES}

REPLICATION_FACTOR=${REPLICATION_FACTOR}



# Stop Cassandra service

sudo systemctl stop cassandra.service



# Update Cassandra configuration to increase the number of nodes

sudo sed -i "s/num_tokens: 256/num_tokens: ${CASSANDRA_NODES}/g" /etc/cassandra/cassandra.yaml



# Update Cassandra configuration to set the replication factor

sudo sed -i "s/replication_factor: 1/replication_factor: ${REPLICATION_FACTOR}/g" /etc/cassandra/cassandra.yaml



# Start Cassandra service

sudo systemctl start cassandra.service


```


This incident type refers to a situation where a high number of mutations are being dropped on a Cassandra database. Mutations are changes made to the database, such as inserting new data or updating existing data. When mutations are dropped, it means that they were not successfully recorded in the database. This can be caused by a variety of factors, such as hardware or network issues, configuration problems, or bugs in the software. When this occurs, it can result in data inconsistencies or loss, and can impact the performance and reliability of the application that relies on the database.


The Kafka Data Loss Incident refers to an incident where there is a misconfiguration of the data retention policy in Kafka, which can result in data loss. Kafka is a distributed streaming platform that is commonly used in big data environments. When data retention policies are misconfigured, data that should be retained is deleted, or data that should be deleted is retained. This can lead to significant data loss and can have serious consequences for businesses that rely on the data stored in Kafka.


Kafka Data Loss Incident

This incident type refers to a problem in a Cassandra cluster where the token range imbalances cause uneven distribution of data across the cluster. This can result in slower read and write performance that can impact the overall functionality of the system. Token range imbalances occur when the distribution of the tokens that define the ranges of data each node is responsible for is not evenly spread across the cluster. As a result, certain nodes may be responsible for a disproportionate amount of data, leading to performance issues and potential failure of the system.


Token Range Imbalances Causing Uneven Data Distribution and Performance Issues in Cassandra Cluster

This incident type refers to a situation where there is a significant delay in the execution of queries on a Cassandra cluster. This delay can cause the system to become unresponsive and result in slower performance. It may be caused by a variety of factors such as an increase in traffic, inefficient queries, or hardware issues. The issue can impact the functionality of the system and requires immediate attention to prevent further disruption.


Slow Query Performance on Cassandra Cluster.

In this incident type, there is an issue with a Cassandra cluster where one or more disks are running slow. This can cause performance issues and potentially lead to data loss or downtime. The goal is to identify and address the specific disk(s) causing the problem in order to restore normal cluster operations.


Slow Disk in Cassandra Cluster

This incident type refers to a performance degradation issue in a Cassandra database caused by the new generation garbage collection (GC) process. Garbage collection is a process by which the unused memory occupied by objects in a program is cleared. In some cases, the new generation GC process may take longer than usual, causing performance issues in the Cassandra database. This could lead to slowness in queries or even crashing of the database.


Performance Degradation Due to New Generation Garbage Collection on Cassandra

```shell
export KEYSPACE_NAME="PLACEHOLDER"

export TABLE_NAME="PLACEHOLDER"

export COLUMN_NAME="PLACEHOLDER"

export VALUE="PLACEHOLDER"

export CONSISTENCY_LEVEL="PLACEHOLDER"

export NEW_NODE_COUNT="PLACEHOLDER"

export NEW_REPLICATION_FACTOR="PLACEHOLDER"

export CASSANDRA_CLUSTER="PLACEHOLDER"
```


### Check the status of Cassandra service

```shell
systemctl status cassandra
```

### Check the Cassandra log for any errors or warnings

```shell
tail -n 100 /var/log/cassandra/system.log
```

### Check the number of dropped mutations

```shell
nodetool tpstats | grep Dropped
```

### Check the status of the Cassandra node

```shell
nodetool status
```

### Check the load on the Cassandra node

```shell
nodetool info | grep Load
```

### Check the disk space usage on the Cassandra node

```shell
df -h
```

### Check the network connectivity between the Cassandra nodes

```shell
nodetool describecluster
```

### Check the replication factor for the affected keyspace

```shell
cqlsh -e "describe keyspace ${KEYSPACE_NAME};"
```

### Check the consistency level of the affected queries

```shell
cqlsh -e "select * from ${TABLE_NAME} where ${COLUMN_NAME} = ${VALUE} consistency ${CONSISTENCY_LEVEL};"
```


### Increase the number of Cassandra nodes to handle the increased load and ensure that there is proper replication factor to avoid any data loss.

```shell


#!/bin/bash



# Define the number of nodes and replication factor

CASSANDRA_NODES=${NUMBER_OF_NODES}

REPLICATION_FACTOR=${REPLICATION_FACTOR}



# Stop Cassandra service

sudo systemctl stop cassandra.service



# Update Cassandra configuration to increase the number of nodes

sudo sed -i "s/num_tokens: 256/num_tokens: ${CASSANDRA_NODES}/g" /etc/cassandra/cassandra.yaml



# Update Cassandra configuration to set the replication factor

sudo sed -i "s/replication_factor: 1/replication_factor: ${REPLICATION_FACTOR}/g" /etc/cassandra/cassandra.yaml



# Start Cassandra service

sudo systemctl start cassandra.service


```


High Number of Dropped Mutations on Cassandra Database

Overview

Parameters

Debug

Check the status of Cassandra service

Check the Cassandra log for any errors or warnings

Check the number of dropped mutations

Check the status of the Cassandra node

Check the load on the Cassandra node

Check the disk space usage on the Cassandra node

Check the network connectivity between the Cassandra nodes

Check the replication factor for the affected keyspace

Check the consistency level of the affected queries

Repair

Increase the number of Cassandra nodes to handle the increased load and ensure that there is proper replication factor to avoid any data loss.

Learn more

Related Runbooks

Kafka Data Loss Incident

Token Range Imbalances Causing Uneven Data Distribution and Performance Issues in Cassandra Cluster

Slow Query Performance on Cassandra Cluster.

Slow Disk in Cassandra Cluster

Support