---
id: d91cd6e7-cd80-47c2-b4d7-20b6a9bfd5fa
---

# Cassandra client request unavailable write incident.
---

The Cassandra client request unavailable write incident occurs when the software's client application is unable to write data to the Cassandra database. This incident can occur due to various reasons such as network issues, configuration issues, or availability problems with the Cassandra nodes. This incident can cause significant disruptions in the software's functionality and must be resolved quickly to avoid any data loss or downtime.

### Parameters
```shell
# Environment Variables

export KEYSPACE="PLACEHOLDER"

export TABLE="PLACEHOLDER"

export CASSANDRA_CLUSTER_IP_ADDRESS="PLACEHOLDER"

```

## Debug

### Check if Cassandra is running
```shell
systemctl status cassandra
```

### Check if there are any nodes unavailable
```shell
nodetool status
```

### Check if there are any read failures
```shell
nodetool cfstats
```

### Check the disk usage of the nodes
```shell
nodetool tablestats ${KEYSPACE}.${TABLE}
```

### Check if the Cassandra client is installed and running
```shell
which cassandra-cli && cassandra-cli
```

### Check the Cassandra client logs for any errors
```shell
tail -f /var/log/cassandra/system.log
```

### Check the network connection between the client and the Cassandra cluster
```shell
ping ${CASSANDRA_CLUSTER_IP_ADDRESS}
```

### Check if the client is able to connect to the Cassandra cluster
```shell
cqlsh ${CASSANDRA_CLUSTER_IP_ADDRESS}
```

### Check if the client is able to write to the Cassandra cluster
```shell
cqlsh ${CASSANDRA_CLUSTER_IP_ADDRESS} -e "INSERT INTO ${KEYSPACE}.${TABLE} (column1, column2) VALUES ('value1', 'value2')"
```
## Repair

### Check the Cassandra cluster for any nodes that are unavailable or experiencing high load, and take appropriate action to restore them or migrate their workload to other nodes.
```shell

#!/bin/bash

# Define the Cassandra cluster endpoint

CASSANDRA_ENDPOINT="PLACEHOLDER"

# Check the status of all nodes in the cluster

nodetool status $CASSANDRA_ENDPOINT

# Identify any nodes that are experiencing high load or are unavailable

unavailable_nodes=$(nodetool status $CASSANDRA_ENDPOINT | grep -E "DN|^-|^ ")

echo "Unavailable nodes: $unavailable_nodes"

# If there are unavailable nodes, restore them or migrate their workload to other nodes

if [ -n "$unavailable_nodes" ]; then

    # Restore the nodes

    nodetool repair $CASSANDRA_ENDPOINT

    # Migrate the workload to other nodes

    nodetool cleanup $CASSANDRA_ENDPOINT

fi

# Verify that the cluster is fully functional

nodetool describecluster $CASSANDRA_ENDPOINT


```


The Cassandra client request unavailable write incident occurs when the software's client application is unable to write data to the Cassandra database. This incident can occur due to various reasons such as network issues, configuration issues, or availability problems with the Cassandra nodes. This incident can cause significant disruptions in the software's functionality and must be resolved quickly to avoid any data loss or downtime.


This incident type refers to a situation where the Kafka broker, which is responsible for managing and storing messages in a Kafka cluster, has failed. This failure results in the unavailability of one or more partitions, which are used to distribute messages across the cluster. As a result, messages cannot be sent or received, leading to disruptions in the system's operations. This type of incident requires immediate attention to restore the Kafka broker and ensure that messages can be processed as expected.


Kafka Broker Failure Causing Partition Unavailability

This incident type refers to a situation where there is a significant delay in the execution of queries on a Cassandra cluster. This delay can cause the system to become unresponsive and result in slower performance. It may be caused by a variety of factors such as an increase in traffic, inefficient queries, or hardware issues. The issue can impact the functionality of the system and requires immediate attention to prevent further disruption.


Slow Query Performance on Cassandra Cluster.

This incident type refers to a situation where a delay or slowness occurs in a system that uses Cassandra database due to the shared storage. Shared storage means multiple servers are accessing the same storage unit, and this can cause latency issues. This type of incident can lead to performance degradation, and it needs to be addressed promptly to ensure optimal system performance.


Latency Caused by Shared Storage in Cassandra

This incident type refers to a situation where a high number of mutations are being dropped on a Cassandra database. Mutations are changes made to the database, such as inserting new data or updating existing data. When mutations are dropped, it means that they were not successfully recorded in the database. This can be caused by a variety of factors, such as hardware or network issues, configuration problems, or bugs in the software. When this occurs, it can result in data inconsistencies or loss, and can impact the performance and reliability of the application that relies on the database.


High Number of Dropped Mutations on Cassandra Database

This incident type refers to issues that occur in Cassandra due to a high replication factor. Replication factor is a setting that determines how many copies of data are stored in a cluster. A high replication factor means that more copies of data are stored in the cluster, which can cause performance issues and potentially lead to data inconsistencies. When this incident occurs, it may affect the availability and stability of the Cassandra cluster, impacting the ability to read and write data.


High Replication Factor Issues in Cassandra

```shell
# Environment Variables

export KEYSPACE="PLACEHOLDER"

export TABLE="PLACEHOLDER"

export CASSANDRA_CLUSTER_IP_ADDRESS="PLACEHOLDER"

```


### Check if Cassandra is running

```shell
systemctl status cassandra
```

### Check if there are any nodes unavailable

```shell
nodetool status
```

### Check if there are any read failures

```shell
nodetool cfstats
```

### Check the disk usage of the nodes

```shell
nodetool tablestats ${KEYSPACE}.${TABLE}
```

### Check if the Cassandra client is installed and running

```shell
which cassandra-cli && cassandra-cli
```

### Check the Cassandra client logs for any errors

```shell
tail -f /var/log/cassandra/system.log
```

### Check the network connection between the client and the Cassandra cluster

```shell
ping ${CASSANDRA_CLUSTER_IP_ADDRESS}
```

### Check if the client is able to connect to the Cassandra cluster

```shell
cqlsh ${CASSANDRA_CLUSTER_IP_ADDRESS}
```

### Check if the client is able to write to the Cassandra cluster

```shell
cqlsh ${CASSANDRA_CLUSTER_IP_ADDRESS} -e "INSERT INTO ${KEYSPACE}.${TABLE} (column1, column2) VALUES ('value1', 'value2')"
```


### Check the Cassandra cluster for any nodes that are unavailable or experiencing high load, and take appropriate action to restore them or migrate their workload to other nodes.

```shell

#!/bin/bash

# Define the Cassandra cluster endpoint

CASSANDRA_ENDPOINT="PLACEHOLDER"

# Check the status of all nodes in the cluster

nodetool status $CASSANDRA_ENDPOINT

# Identify any nodes that are experiencing high load or are unavailable

unavailable_nodes=$(nodetool status $CASSANDRA_ENDPOINT | grep -E "DN|^-|^ ")

echo "Unavailable nodes: $unavailable_nodes"

# If there are unavailable nodes, restore them or migrate their workload to other nodes

if [ -n "$unavailable_nodes" ]; then

    # Restore the nodes

    nodetool repair $CASSANDRA_ENDPOINT

    # Migrate the workload to other nodes

    nodetool cleanup $CASSANDRA_ENDPOINT

fi

# Verify that the cluster is fully functional

nodetool describecluster $CASSANDRA_ENDPOINT


```


Cassandra client request unavailable write incident.

Overview

Parameters

Debug

Check if Cassandra is running

Check if there are any nodes unavailable

Check if there are any read failures

Check the disk usage of the nodes

Check if the Cassandra client is installed and running

Check the Cassandra client logs for any errors

Check the network connection between the client and the Cassandra cluster

Check if the client is able to connect to the Cassandra cluster

Check if the client is able to write to the Cassandra cluster

Repair

Check the Cassandra cluster for any nodes that are unavailable or experiencing high load, and take appropriate action to restore them or migrate their workload to other nodes.

Learn more

Related Runbooks

Kafka Broker Failure Causing Partition Unavailability

Slow Query Performance on Cassandra Cluster.

Latency Caused by Shared Storage in Cassandra

High Number of Dropped Mutations on Cassandra Database

Support