---
id: 0f4e478c-616e-11ee-8c99-0242ac120002
---

# Latency Caused by Shared Storage in Cassandra
---

This incident type refers to a situation where a delay or slowness occurs in a system that uses Cassandra database due to the shared storage. Shared storage means multiple servers are accessing the same storage unit, and this can cause latency issues. This type of incident can lead to performance degradation, and it needs to be addressed promptly to ensure optimal system performance.

### Parameters
```shell
export SHARED_STORAGE="PLACEHOLDER"

export CASSANDRA_NODE1="PLACEHOLDER"

export CASSANDRA_NODE2="PLACEHOLDER"

export CASSANDRA_NODE="PLACEHOLDER"

export TABLE_NAME="PLACEHOLDER"

export SHARED_STORAGE_PATH="PLACEHOLDER"

export NEW_STORAGE_PATH="PLACEHOLDER"
```

## Debug

### Check if Cassandra is running
```shell
systemctl status cassandra
```

### Check for any errors in Cassandra logs
```shell
tail -f /var/log/cassandra/system.log
```

### Check the Cassandra cluster status
```shell
nodetool status
```

### Check the status of the shared storage
```shell
df -h ${SHARED_STORAGE}
```

### Check the network latency between Cassandra nodes
```shell
ping ${CASSANDRA_NODE1}

ping ${CASSANDRA_NODE2}
```

### Check the read/write latency of Cassandra using cqlsh
```shell
cqlsh ${CASSANDRA_NODE} -e "SELECT * FROM ${TABLE_NAME} LIMIT 10;"
```

## Repair

### Split the shared storage into separate storage units to reduce contention for resources. It is recommended to avoid shared storage like EBS, SAN, NAS. Shared storage degrades performance and creates a single point of failure.
```shell
bash

#!/bin/bash



# Define the variables

${SHARED_STORAGE_PATH}="/path/to/shared/storage"

${NEW_STORAGE_PATH}="/path/to/new/storage"



# Stop the Cassandra service

sudo systemctl stop cassandra



# Copy the shared storage to the new storage location

sudo cp -r $SHARED_STORAGE_PATH/* $NEW_STORAGE_PATH/



# Modify the Cassandra configuration to use the new storage location

sudo sed -i "s|$SHARED_STORAGE_PATH|$NEW_STORAGE_PATH|g" /etc/cassandra/cassandra.yaml



# Start the Cassandra service

sudo systemctl start cassandra


```


This incident type refers to a situation where a delay or slowness occurs in a system that uses Cassandra database due to the shared storage. Shared storage means multiple servers are accessing the same storage unit, and this can cause latency issues. This type of incident can lead to performance degradation, and it needs to be addressed promptly to ensure optimal system performance.


This incident type refers to a situation where there is a significant delay in the execution of queries on a Cassandra cluster. This delay can cause the system to become unresponsive and result in slower performance. It may be caused by a variety of factors such as an increase in traffic, inefficient queries, or hardware issues. The issue can impact the functionality of the system and requires immediate attention to prevent further disruption.


Slow Query Performance on Cassandra Cluster.

In this incident type, there is an issue with a Cassandra cluster where one or more disks are running slow. This can cause performance issues and potentially lead to data loss or downtime. The goal is to identify and address the specific disk(s) causing the problem in order to restore normal cluster operations.


Slow Disk in Cassandra Cluster

This incident type refers to a performance degradation issue in a Cassandra database caused by the new generation garbage collection (GC) process. Garbage collection is a process by which the unused memory occupied by objects in a program is cleared. In some cases, the new generation GC process may take longer than usual, causing performance issues in the Cassandra database. This could lead to slowness in queries or even crashing of the database.


Performance Degradation Due to New Generation Garbage Collection on Cassandra

This incident type refers to a situation where a high number of mutations are being dropped on a Cassandra database. Mutations are changes made to the database, such as inserting new data or updating existing data. When mutations are dropped, it means that they were not successfully recorded in the database. This can be caused by a variety of factors, such as hardware or network issues, configuration problems, or bugs in the software. When this occurs, it can result in data inconsistencies or loss, and can impact the performance and reliability of the application that relies on the database.


High Number of Dropped Mutations on Cassandra Database

This incident type refers to issues that occur in Cassandra due to a high replication factor. Replication factor is a setting that determines how many copies of data are stored in a cluster. A high replication factor means that more copies of data are stored in the cluster, which can cause performance issues and potentially lead to data inconsistencies. When this incident occurs, it may affect the availability and stability of the Cassandra cluster, impacting the ability to read and write data.


High Replication Factor Issues in Cassandra

```shell
export SHARED_STORAGE="PLACEHOLDER"

export CASSANDRA_NODE1="PLACEHOLDER"

export CASSANDRA_NODE2="PLACEHOLDER"

export CASSANDRA_NODE="PLACEHOLDER"

export TABLE_NAME="PLACEHOLDER"

export SHARED_STORAGE_PATH="PLACEHOLDER"

export NEW_STORAGE_PATH="PLACEHOLDER"
```


### Check if Cassandra is running

```shell
systemctl status cassandra
```

### Check for any errors in Cassandra logs

```shell
tail -f /var/log/cassandra/system.log
```

### Check the Cassandra cluster status

```shell
nodetool status
```

### Check the status of the shared storage

```shell
df -h ${SHARED_STORAGE}
```

### Check the network latency between Cassandra nodes

```shell
ping ${CASSANDRA_NODE1}

ping ${CASSANDRA_NODE2}
```

### Check the read/write latency of Cassandra using cqlsh

```shell
cqlsh ${CASSANDRA_NODE} -e "SELECT * FROM ${TABLE_NAME} LIMIT 10;"
```


### Split the shared storage into separate storage units to reduce contention for resources. It is recommended to avoid shared storage like EBS, SAN, NAS. Shared storage degrades performance and creates a single point of failure.

```shell
bash

#!/bin/bash



# Define the variables

${SHARED_STORAGE_PATH}="/path/to/shared/storage"

${NEW_STORAGE_PATH}="/path/to/new/storage"



# Stop the Cassandra service

sudo systemctl stop cassandra



# Copy the shared storage to the new storage location

sudo cp -r $SHARED_STORAGE_PATH/* $NEW_STORAGE_PATH/



# Modify the Cassandra configuration to use the new storage location

sudo sed -i "s|$SHARED_STORAGE_PATH|$NEW_STORAGE_PATH|g" /etc/cassandra/cassandra.yaml



# Start the Cassandra service

sudo systemctl start cassandra


```


Latency Caused by Shared Storage in Cassandra

Overview

Parameters

Debug

Check if Cassandra is running

Check for any errors in Cassandra logs

Check the Cassandra cluster status

Check the status of the shared storage

Check the network latency between Cassandra nodes

Check the read/write latency of Cassandra using cqlsh

Repair

Split the shared storage into separate storage units to reduce contention for resources. It is recommended to avoid shared storage like EBS, SAN, NAS. Shared storage degrades performance and creates a single point of failure.

Learn more

Related Runbooks

Slow Query Performance on Cassandra Cluster.

Slow Disk in Cassandra Cluster

Performance Degradation Due to New Generation Garbage Collection on Cassandra

High Number of Dropped Mutations on Cassandra Database

Support