---
id: 98287d12-616d-11ee-8c99-0242ac120002
---

# Cassandra Tombstone Cleanup
---

In a Cassandra database, tombstones are markers that indicate a deleted record. If tombstones are not properly cleaned up, they can cause performance issues and potentially lead to data inconsistency. A Cassandra tombstone cleanup incident is an event where there is a need to identify and remove tombstones from the database in order to maintain optimal performance and data consistency. This incident type typically involves database administrators and software engineers working together to identify the source of the tombstones and implement a solution to remove them.

### Parameters
```shell
export KEYSPACE_NAME="PLACEHOLDER"

export TABLE_NAME="PLACEHOLDER"

export DATABASE_NAME="PLACEHOLDER"

export PATH_TO_LOG_FILE="PLACEHOLDER"

export PATH_TO_METRICS_FILE="PLACEHOLDER"
```

## Debug

### Check Cassandra process status
```shell
systemctl status cassandra
```

### Check Cassandra logs for tombstone warnings/errors
```shell
grep -i tombstone /var/log/cassandra/system.log
```

### List Cassandra keyspaces
```shell
cqlsh -e "DESC KEYSPACES;"
```

### List tables in a keyspace
```shell
cqlsh -e "DESC ${KEYSPACE_NAME};"
```

### Count the number of tombstones in a table
```shell
nodetool tablestats ${KEYSPACE_NAME}.${TABLE_NAME} | grep "Number of tombstones"
```

### Force a cleanup of tombstones in a table
```shell
nodetool cleanup ${KEYSPACE_NAME} ${TABLE_NAME}
```

### Check for tombstones in a table after cleanup
```shell
nodetool tablestats ${KEYSPACE_NAME}.${TABLE_NAME} | grep "Number of tombstones"
```

## Repair

### Remove tombstones: This involves running a compaction process on the database to remove the tombstones or manually deleting them.
```shell
bash

#!/bin/bash



# Set the necessary parameters

KEYSPACE=${KEYSPACE_NAME}

TABLE=${TABLE_NAME}



# Run the compaction process to remove tombstones

nodetool compact $KEYSPACE $TABLE



# Alternatively, manually delete the tombstones

nodetool flush $KEYSPACE $TABLE

nodetool repair $KEYSPACE $TABLE


```


In a Cassandra database, tombstones are markers that indicate a deleted record. If tombstones are not properly cleaned up, they can cause performance issues and potentially lead to data inconsistency. A Cassandra tombstone cleanup incident is an event where there is a need to identify and remove tombstones from the database in order to maintain optimal performance and data consistency. This incident type typically involves database administrators and software engineers working together to identify the source of the tombstones and implement a solution to remove them.


Compaction is a process in Cassandra that merges multiple SSTables (Sorted String Tables) into a single SSTable, eliminating any redundant data and improving read performance. However, sometimes compaction can fail due to various reasons such as insufficient disk space or corrupted data, resulting in degraded performance or even complete failure of the database. Troubleshooting compaction merging SSTables involves identifying and resolving the root cause of such failures to ensure the smooth functioning of the Cassandra database.


Troubleshooting Compaction Merging SSTables in Cassandra

This incident type refers to a situation where there is a significant delay in the execution of queries on a Cassandra cluster. This delay can cause the system to become unresponsive and result in slower performance. It may be caused by a variety of factors such as an increase in traffic, inefficient queries, or hardware issues. The issue can impact the functionality of the system and requires immediate attention to prevent further disruption.


Slow Query Performance on Cassandra Cluster.

In this incident type, there is an issue with a Cassandra cluster where one or more disks are running slow. This can cause performance issues and potentially lead to data loss or downtime. The goal is to identify and address the specific disk(s) causing the problem in order to restore normal cluster operations.


Slow Disk in Cassandra Cluster

This incident type refers to a performance degradation issue in a Cassandra database caused by the new generation garbage collection (GC) process. Garbage collection is a process by which the unused memory occupied by objects in a program is cleared. In some cases, the new generation GC process may take longer than usual, causing performance issues in the Cassandra database. This could lead to slowness in queries or even crashing of the database.


Performance Degradation Due to New Generation Garbage Collection on Cassandra

Misconfigured compaction strategy is an incident type that occurs when the way data is compacted within a database cluster is not properly configured. This can lead to excessive compaction activity that can negatively impact the cluster's performance and stability. The incident can cause slow query response times, increased disk usage, and high CPU utilization, among other issues. It is crucial to identify and fix the misconfiguration promptly to ensure the cluster's smooth operation.


Misconfigured Compaction Strategy.

```shell
export KEYSPACE_NAME="PLACEHOLDER"

export TABLE_NAME="PLACEHOLDER"

export DATABASE_NAME="PLACEHOLDER"

export PATH_TO_LOG_FILE="PLACEHOLDER"

export PATH_TO_METRICS_FILE="PLACEHOLDER"
```


### Check Cassandra process status

```shell
systemctl status cassandra
```

### Check Cassandra logs for tombstone warnings/errors

```shell
grep -i tombstone /var/log/cassandra/system.log
```

### List Cassandra keyspaces

```shell
cqlsh -e "DESC KEYSPACES;"
```

### List tables in a keyspace

```shell
cqlsh -e "DESC ${KEYSPACE_NAME};"
```

### Count the number of tombstones in a table

```shell
nodetool tablestats ${KEYSPACE_NAME}.${TABLE_NAME} | grep "Number of tombstones"
```

### Force a cleanup of tombstones in a table

```shell
nodetool cleanup ${KEYSPACE_NAME} ${TABLE_NAME}
```

### Check for tombstones in a table after cleanup

```shell
nodetool tablestats ${KEYSPACE_NAME}.${TABLE_NAME} | grep "Number of tombstones"
```


### Remove tombstones: This involves running a compaction process on the database to remove the tombstones or manually deleting them.

```shell
bash

#!/bin/bash



# Set the necessary parameters

KEYSPACE=${KEYSPACE_NAME}

TABLE=${TABLE_NAME}



# Run the compaction process to remove tombstones

nodetool compact $KEYSPACE $TABLE



# Alternatively, manually delete the tombstones

nodetool flush $KEYSPACE $TABLE

nodetool repair $KEYSPACE $TABLE


```


Cassandra Tombstone Cleanup

Overview

Parameters

Debug

Check Cassandra process status

Check Cassandra logs for tombstone warnings/errors

List Cassandra keyspaces

List tables in a keyspace

Count the number of tombstones in a table

Force a cleanup of tombstones in a table

Check for tombstones in a table after cleanup

Repair

Remove tombstones: This involves running a compaction process on the database to remove the tombstones or manually deleting them.

Learn more

Related Runbooks

Troubleshooting Compaction Merging SSTables in Cassandra

Slow Query Performance on Cassandra Cluster.

Slow Disk in Cassandra Cluster

Performance Degradation Due to New Generation Garbage Collection on Cassandra

Support