---
id: 7a05407e-10b3-479c-bbd1-51119c19ff77
---

# Cassandra Node Unavailability
---

This incident type refers to a situation where a Cassandra node becomes unavailable. This can cause interruptions to the expected functionality of the system and can lead to data loss or corruption. The cause of this incident can be due to a variety of reasons such as hardware failure, network issues, or software bugs. It is critical to resolve the issue as soon as possible to minimize the impact on the system and ensure the smooth functioning of the application.

### Parameters
```shell
# Environment Variables

export CASSANDRA_NODE_IP_ADDRESS="PLACEHOLDER"

export PORT_NUMBER="PLACEHOLDER"

export BACKUP_FILE_NAME="PLACEHOLDER"

export RESTORE_DIRECTORY="PLACEHOLDER"

export CASSANDRA_DATA_LOCATION="PLACEHOLDER"

export BACKUP_LOCATION="PLACEHOLDER"
```

## Debug

### Check Cassandra service status
```shell
sudo service cassandra status
```

### Check Cassandra process status
```shell
ps aux | grep cassandra
```

### Check Cassandra system logs
```shell
tail -f /var/log/cassandra/system.log
```

### Check Cassandra node health
```shell
nodetool status
```

### Check Cassandra node ring information
```shell
nodetool ring
```

### Check Cassandra node gossip information
```shell
nodetool gossipinfo
```

### Check Cassandra node log for errors
```shell
tail -f /var/log/cassandra/system.log | grep ERROR
```

### Check Cassandra node log for warnings
```shell
tail -f /var/log/cassandra/system.log | grep WARN
```

### Check for any network connectivity issues
```shell
ping ${CASSANDRA_NODE_IP_ADDRESS}
```

### Check for any firewall issues
```shell
sudo iptables -L
```

## Repair

### Attempt to repair the Cassandra installation or reinstall it if necessary.
```shell


#!/bin/bash

# Stop Cassandra service

sudo service cassandra stop

# Remove Cassandra package

sudo apt-get remove cassandra

# Clean up any remaining files

sudo rm -rf /var/lib/cassandra

# Reinstall Cassandra

sudo apt-get install cassandra

# Start Cassandra service

sudo service cassandra start

# Check Cassandra status

sudo nodetool status

```

### Restore from a recent backup if data loss or corruption has occurred.
```shell

#!/bin/bash

# Specify the backup filename

BACKUP_FILE=${BACKUP_FILE_NAME}

# Specify the directory to restore the backup to

RESTORE_DIR=${RESTORE_DIRECTORY}


# Stop the Cassandra service

sudo systemctl stop cassandra

# Restore the backup

sudo tar -xzf $BACKUP_FILE -C $RESTORE_DIR


# start the Cassandra service

sudo systemctl start cassandra


# check the status of the Cassandra service

sudo systemctl status cassandra

```

### Reboot the node to attempt to clear any software issues that may be causing the unavailability.
```shell

#!/bin/bash

NODE_IP=${NODE_IP_ADDRESS}

# Check if node is pingable

if ping -c1 $NODE_IP &> /dev/null

then

    echo "Node is pingable. Proceeding with reboot."

else

    echo "Node is not pingable. Aborting reboot."

    exit 1

fi

# Reboot node

ssh root@$NODE_IP 'reboot'

```

### Restore from a cassandra backup if data loss or corruption has occurred.
```shell

#!/bin/bash

# Set the variables

BACKUP_LOCATION=${BACKUP_LOCATION}

CASSANDRA_DATA_LOCATION=${CASSANDRA_DATA_LOCATION}

# Stop the Cassandra service

service cassandra stop

# Restore the backup

cp -r $BACKUP_LOCATION/* $CASSANDRA_DATA_LOCATION/

# Start the Cassandra service

service cassandra start

# Verify that Cassandra is running normally

nodetool status

```

This incident type refers to a situation where a Cassandra node becomes unavailable. This can cause interruptions to the expected functionality of the system and can lead to data loss or corruption. The cause of this incident can be due to a variety of reasons such as hardware failure, network issues, or software bugs. It is critical to resolve the issue as soon as possible to minimize the impact on the system and ensure the smooth functioning of the application.


This incident type refers to the failure of a replica node in a PostgreSQL database system that is running on a Linux-based operating system. A replica node is a copy of the primary database node that is used to provide high availability and fault tolerance. When a replica node fails, it can result in data loss, decreased system performance, and potential downtime for users. This type of incident requires immediate attention from a software engineer to diagnose and resolve the issue as quickly as possible.


PostgreSQL Replica Node Failure on Linux.

This incident type refers to a failure in the primary PostgreSQL node on a Linux system. PostgreSQL is a popular open source relational database management system used by many organizations. The primary node is responsible for handling the majority of the traffic and data replication in a PostgreSQL cluster. When the primary node fails, it can result in data loss, slow performance, and potential downtime for users. This type of incident requires immediate attention and resolution to minimize the impact on users and ensure data integrity.


Primary PostgreSQL Node Failure on Linux

This incident type involves nodes in a Kubernetes cluster that are experiencing network unavailability, meaning they are not accessible. This could be due to a misconfiguration, route exhaustion, or a physical problem with the network connection to the hardware. It is a high urgency incident that requires immediate attention to restore network connectivity to the affected nodes.


Kubernetes Nodes with Network Unavailable

Node Not Ready in Kubernetes Cluster is an incident type that occurs when a node in a Kubernetes cluster fails to respond, is unresponsive, or is not ready to take on workloads. This can cause disruptions in service and lead to downtime, as the cluster is unable to allocate resources effectively. This incident type can be caused by a range of factors, including hardware issues, network problems, and configuration errors. Swift resolution of this incident is essential to ensure that the Kubernetes cluster is able to function correctly and provide uninterrupted service.


Node Not Ready in Kubernetes Cluster

This incident type refers to a situation where the Kafka broker, which is responsible for managing and storing messages in a Kafka cluster, has failed. This failure results in the unavailability of one or more partitions, which are used to distribute messages across the cluster. As a result, messages cannot be sent or received, leading to disruptions in the system's operations. This type of incident requires immediate attention to restore the Kafka broker and ensure that messages can be processed as expected.


Kafka Broker Failure Causing Partition Unavailability

```shell
# Environment Variables

export CASSANDRA_NODE_IP_ADDRESS="PLACEHOLDER"

export PORT_NUMBER="PLACEHOLDER"

export BACKUP_FILE_NAME="PLACEHOLDER"

export RESTORE_DIRECTORY="PLACEHOLDER"

export CASSANDRA_DATA_LOCATION="PLACEHOLDER"

export BACKUP_LOCATION="PLACEHOLDER"
```


### Check Cassandra service status

```shell
sudo service cassandra status
```

### Check Cassandra process status

```shell
ps aux | grep cassandra
```

### Check Cassandra system logs

```shell
tail -f /var/log/cassandra/system.log
```

### Check Cassandra node health

```shell
nodetool status
```

### Check Cassandra node ring information

```shell
nodetool ring
```

### Check Cassandra node gossip information

```shell
nodetool gossipinfo
```

### Check Cassandra node log for errors

```shell
tail -f /var/log/cassandra/system.log | grep ERROR
```

### Check Cassandra node log for warnings

```shell
tail -f /var/log/cassandra/system.log | grep WARN
```

### Check for any network connectivity issues

```shell
ping ${CASSANDRA_NODE_IP_ADDRESS}
```

### Check for any firewall issues

```shell
sudo iptables -L
```


### Attempt to repair the Cassandra installation or reinstall it if necessary.

```shell


#!/bin/bash

# Stop Cassandra service

sudo service cassandra stop

# Remove Cassandra package

sudo apt-get remove cassandra

# Clean up any remaining files

sudo rm -rf /var/lib/cassandra

# Reinstall Cassandra

sudo apt-get install cassandra

# Start Cassandra service

sudo service cassandra start

# Check Cassandra status

sudo nodetool status

```

### Restore from a recent backup if data loss or corruption has occurred.

```shell

#!/bin/bash

# Specify the backup filename

BACKUP_FILE=${BACKUP_FILE_NAME}

# Specify the directory to restore the backup to

RESTORE_DIR=${RESTORE_DIRECTORY}


# Stop the Cassandra service

sudo systemctl stop cassandra

# Restore the backup

sudo tar -xzf $BACKUP_FILE -C $RESTORE_DIR


# start the Cassandra service

sudo systemctl start cassandra


# check the status of the Cassandra service

sudo systemctl status cassandra

```

### Reboot the node to attempt to clear any software issues that may be causing the unavailability.

```shell

#!/bin/bash

NODE_IP=${NODE_IP_ADDRESS}

# Check if node is pingable

if ping -c1 $NODE_IP &> /dev/null

then

    echo "Node is pingable. Proceeding with reboot."

else

    echo "Node is not pingable. Aborting reboot."

    exit 1

fi

# Reboot node

ssh root@$NODE_IP 'reboot'

```

### Restore from a cassandra backup if data loss or corruption has occurred.

```shell

#!/bin/bash

# Set the variables

BACKUP_LOCATION=${BACKUP_LOCATION}

CASSANDRA_DATA_LOCATION=${CASSANDRA_DATA_LOCATION}

# Stop the Cassandra service

service cassandra stop

# Restore the backup

cp -r $BACKUP_LOCATION/* $CASSANDRA_DATA_LOCATION/

# Start the Cassandra service

service cassandra start

# Verify that Cassandra is running normally

nodetool status

```


Cassandra Node Unavailability

Overview

Parameters

Debug

Check Cassandra service status

Check Cassandra process status

Check Cassandra system logs

Check Cassandra node health

Check Cassandra node ring information

Check Cassandra node gossip information

Check Cassandra node log for errors

Check Cassandra node log for warnings

Check for any network connectivity issues

Check for any firewall issues

Repair

Attempt to repair the Cassandra installation or reinstall it if necessary.

Restore from a recent backup if data loss or corruption has occurred.

Reboot the node to attempt to clear any software issues that may be causing the unavailability.

Restore from a cassandra backup if data loss or corruption has occurred.

Learn more

Related Runbooks

PostgreSQL Replica Node Failure on Linux.

Primary PostgreSQL Node Failure on Linux

Kubernetes Nodes with Network Unavailable

Node Not Ready in Kubernetes Cluster

Support