---
id: e9fd7dc6-616e-11ee-8c99-0242ac120002
---

# Cassandra Performance Issues Due to Uneven Partitions
---

This incident type pertains to identifying uneven partitions in a Cassandra database, which can cause performance issues related to read/write operations and query execution. Cassandra is a distributed database that stores data across multiple nodes, and partitions are used to distribute the data evenly among the nodes. However, if the partition sizes are uneven, some nodes may become overloaded while others remain underutilized, leading to performance bottlenecks. Identifying these uneven partitions is crucial in ensuring optimal performance of the Cassandra database.

### Parameters
```shell
export CASSANDRA_NODE="PLACEHOLDER"

export KEYSPACE="PLACEHOLDER"

export TABLE="PLACEHOLDER"

export PATH_TO_CASSANDRA_DIR="PLACEHOLDER"

export PATH_TO_NODETOOL_DIR="PLACEHOLDER"
```

## Debug

### Connect to Cassandra cluster
```shell
cqlsh ${CASSANDRA_NODE}
```

### Check if nodetool is installed
```shell
nodetool status
```

### Check the token ranges for each node
```shell
nodetool ring
```

### Identify the highest and lowest token values
```shell
nodetool ring | awk '{print $NF, $8}' | sort | awk 'NR==1{print "Lowest Token:", $1}; END{print "Highest Token:", $1}'
```

### Check the size of each partition
```shell
nodetool cfstats ${KEYSPACE}.${TABLE} | grep "Partition Size"
```

### Identify the largest and smallest partitions
```shell
nodetool cfstats ${KEYSPACE}.${TABLE} | grep "Partition Size" | awk '{print $3, $4}' | sort -k2 -n | awk 'NR==1{print "Smallest Partition:", $1, $2}; END{print "Largest Partition:", $1, $2}'
```

## Repair

### Rebalance the partitions: One way to remediate this incident is to rebalance the partitions in the Cassandra database. This involves redistributing the data across the nodes in a way that ensures even partition sizes. This can be done by running the nodetool repair command or by using a third-party tool such as Cassandra Reaper.
```shell


#!/bin/bash



# set variables

cassandra_dir=${PATH_TO_CASSANDRA_DIR}

nodetool_dir=${PATH_TO_NODETOOL_DIR}



# stop Cassandra service

sudo service cassandra stop



# run nodetool repair command

sudo $cassandra_dir/$nodetool_dir/nodetool repair -pr



# start Cassandra service

sudo service cassandra start


```


This incident type pertains to identifying uneven partitions in a Cassandra database, which can cause performance issues related to read/write operations and query execution. Cassandra is a distributed database that stores data across multiple nodes, and partitions are used to distribute the data evenly among the nodes. However, if the partition sizes are uneven, some nodes may become overloaded while others remain underutilized, leading to performance bottlenecks. Identifying these uneven partitions is crucial in ensuring optimal performance of the Cassandra database.


This incident type refers to a situation where the number of partitions for a Kafka topic is insufficient to handle the incoming data stream. This can lead to low throughput and/or increased write latency. In order to resolve this issue, it may be necessary to increase the number of Kafka topic partitions. Failure to address this issue can lead to degraded performance and potential data loss.


Inadequate Kafka Topic Partitions Incident

This incident type refers to an issue with a Kafka cluster where one or more partitions are experiencing high lag. Lag is the difference between the latest message produced to a partition and the latest message consumed from that partition. When a partition lags behind, it means that messages are not being consumed as quickly as they are being produced. This can lead to a backlog of messages and potential data loss if not addressed. The incident description suggests checking for hot partitions, which are partitions that receive a disproportionate amount of traffic compared to others. Identifying and resolving high partition lag is critical to ensure the stability and reliability of a Kafka cluster.


High Partition Lag on Kafka Cluster

This incident type refers to a problem in a Cassandra cluster where the token range imbalances cause uneven distribution of data across the cluster. This can result in slower read and write performance that can impact the overall functionality of the system. Token range imbalances occur when the distribution of the tokens that define the ranges of data each node is responsible for is not evenly spread across the cluster. As a result, certain nodes may be responsible for a disproportionate amount of data, leading to performance issues and potential failure of the system.


Token Range Imbalances Causing Uneven Data Distribution and Performance Issues in Cassandra Cluster

This incident type relates to identifying slow running queries on the Cassandra database and determining the users responsible for running them. Slow queries can cause performance issues and impact the overall efficiency of the system. Identifying and troubleshooting slow queries is crucial for maintaining optimal performance and ensuring smooth operations of the database. The incident may require investigating the root cause of the slow queries, optimizing the database configuration and queries, and providing recommendations to mitigate future incidents.


Slow Running Queries on Cassandra

This incident type refers to a situation where there is a significant delay in the execution of queries on a Cassandra cluster. This delay can cause the system to become unresponsive and result in slower performance. It may be caused by a variety of factors such as an increase in traffic, inefficient queries, or hardware issues. The issue can impact the functionality of the system and requires immediate attention to prevent further disruption.


Slow Query Performance on Cassandra Cluster.

```shell
export CASSANDRA_NODE="PLACEHOLDER"

export KEYSPACE="PLACEHOLDER"

export TABLE="PLACEHOLDER"

export PATH_TO_CASSANDRA_DIR="PLACEHOLDER"

export PATH_TO_NODETOOL_DIR="PLACEHOLDER"
```


### Connect to Cassandra cluster

```shell
cqlsh ${CASSANDRA_NODE}
```

### Check if nodetool is installed

```shell
nodetool status
```

### Check the token ranges for each node

```shell
nodetool ring
```

### Identify the highest and lowest token values

```shell
nodetool ring | awk '{print $NF, $8}' | sort | awk 'NR==1{print "Lowest Token:", $1}; END{print "Highest Token:", $1}'
```

### Check the size of each partition

```shell
nodetool cfstats ${KEYSPACE}.${TABLE} | grep "Partition Size"
```

### Identify the largest and smallest partitions

```shell
nodetool cfstats ${KEYSPACE}.${TABLE} | grep "Partition Size" | awk '{print $3, $4}' | sort -k2 -n | awk 'NR==1{print "Smallest Partition:", $1, $2}; END{print "Largest Partition:", $1, $2}'
```


### Rebalance the partitions: One way to remediate this incident is to rebalance the partitions in the Cassandra database. This involves redistributing the data across the nodes in a way that ensures even partition sizes. This can be done by running the nodetool repair command or by using a third-party tool such as Cassandra Reaper.

```shell


#!/bin/bash



# set variables

cassandra_dir=${PATH_TO_CASSANDRA_DIR}

nodetool_dir=${PATH_TO_NODETOOL_DIR}



# stop Cassandra service

sudo service cassandra stop



# run nodetool repair command

sudo $cassandra_dir/$nodetool_dir/nodetool repair -pr



# start Cassandra service

sudo service cassandra start


```


Cassandra Performance Issues Due to Uneven Partitions

Overview

Parameters

Debug

Connect to Cassandra cluster

Check if nodetool is installed

Check the token ranges for each node

Identify the highest and lowest token values

Check the size of each partition

Identify the largest and smallest partitions

Repair

Learn more

Related Runbooks

Inadequate Kafka Topic Partitions Incident

High Partition Lag on Kafka Cluster

Token Range Imbalances Causing Uneven Data Distribution and Performance Issues in Cassandra Cluster

Slow Running Queries on Cassandra

Support