---
id: 13483da3-12b1-43d0-bba3-2f4c2c2bab4b
---

# Deadlock caused by concurrent transactions in Cassandra.
---

Deadlock is a type of incident that can occur when multiple transactions are executed concurrently in a distributed database management system such as Cassandra. In this scenario, two or more transactions might try to acquire the same exclusive lock on a resource, and neither transaction can proceed until the other releases its lock. This can result in a situation where all the transactions are blocked, and no progress can be made. In the context of Cassandra, this can lead to blocked queries and overall decreased performance of the system.

### Parameters
```shell
export CASSANDRA_JMX_PASSWORD="PLACEHOLDER"

export CASSANDRA_NODE_IP="PLACEHOLDER"

export CASSANDRA_JMX_USERNAME="PLACEHOLDER"

export CASSANDRA_JMX_PORT="PLACEHOLDER"

```

## Debug

### Check the load on the system
```shell
top
```

### Check the load on the CPU
```shell
mpstat
```

### Check the CPU usage per process
```shell
ps -eo %cpu,pid,cmd | sort -k 1 -r | head -10
```

### Check the memory usage per process
```shell
ps -eo %mem,pid,cmd | sort -k 1 -r | head -10
```

### Check the number of connections to Cassandra
```shell
nodetool tpstats
```

### Check the number of running threads in Cassandra
```shell
nodetool -h ${CASSANDRA_NODE_IP} -p ${CASSANDRA_JMX_PORT} -u ${CASSANDRA_JMX_USERNAME} -pw ${CASSANDRA_JMX_PASSWORD} -ssl enable get /metrics/thread-pool/RequestResponseStage/ActiveTasks
```

### Check the Cassandra system log for errors
```shell
tail -f /var/log/cassandra/system.log
```

### Check the Cassandra slow query log for queries taking too long
```shell
tail -f /var/log/cassandra/slowquery.log
```

### Multiple queries trying to access the same Cassandra table simultaneously, leading to contention and deadlock.
```shell


#!/bin/bash



# Set the table name

table=${TABLE_NAME}



# Get the process IDs of all Cassandra instances

cassandra_pids=$(pgrep -f "cassandra")



# Loop through each process ID and check for contention on the specified table

for pid in $cassandra_pids; do

    # Use nodetool to get the thread dump for the process

    thread_dump=$(nodetool --pid $pid --threaddump)



    # Check if the table name appears in the thread dump

    if echo "$thread_dump" | grep -q "$table"; then

        # If it does, print a warning message

        echo "WARNING: Contention detected on table $table in process $pid"

    fi

done


```

## Repair

### Tune the Cassandra cluster settings to optimize performance and minimize the risk of deadlocks.
```shell

#!/bin/bash

CLUSTER_NAME=${CLUSTER_NAME}

SEED_NODE_IP=${SEED_NODE_IP}

NUM_TOKENS=${NUM_TOKENS}

CONCURRENT_READS=${CONCURRENT_READS}

CONCURRENT_WRITES=${CONCURRENT_WRITES}

COMMITLOG_SYNC=${COMMITLOG_SYNC}

sed -i "s/cluster_name: .*/cluster_name: $CLUSTER_NAME/g" /etc/cassandra/cassandra.yaml

sed -i "s/seeds: .*/seeds: \"$SEED_NODE_IP\"/g" /etc/cassandra/cassandra.yaml

sed -i "s/num_tokens: .*/num_tokens: $NUM_TOKENS/g" /etc/cassandra/cassandra.yaml

sed -i "s/concurrent_reads: .*/concurrent_reads: $CONCURRENT_READS/g" /etc/cassandra/cassandra.yaml

sed -i "s/concurrent_writes: .*/concurrent_writes: $CONCURRENT_WRITES/g" /etc/cassandra/cassandra.yaml

sed -i "s/commitlog_sync: .*/commitlog_sync: $COMMITLOG_SYNC/g" /etc/cassandra/cassandra.yaml


```

Deadlock is a type of incident that can occur when multiple transactions are executed concurrently in a distributed database management system such as Cassandra. In this scenario, two or more transactions might try to acquire the same exclusive lock on a resource, and neither transaction can proceed until the other releases its lock. This can result in a situation where all the transactions are blocked, and no progress can be made. In the context of Cassandra, this can lead to blocked queries and overall decreased performance of the system.


Thread deadlock conditions occur when two or more threads are blocked and waiting for each other to release a resource, causing a system to hang or become unresponsive. In the context of a Tomcat server, this could cause the server to stop responding to incoming requests. Detection of thread deadlock conditions on Tomcat is an indication of a critical issue that requires immediate attention to restore normal server operation.


Tomcat Thread Deadlock Conditions Detected

A SQL Server deadlock incident occurs when two or more processes are blocked and waiting on each other to release a resource. This results in a situation where none of the processes can continue executing, causing a system slowdown or complete system failure. The incident requires immediate attention from a software engineer to identify and resolve the issue.


SQL Server Deadlock Incident

This incident type refers to a situation where the Postgresql database has a high rollback rate, which means that a high percentage of transactions are being aborted compared to the committed ones. This can cause issues with data consistency and performance, and may require investigation and resolution by the responsible team. The incident details may include information about the affected database instance, the service or system impacted, the urgency level, and any related alerts or escalations.


Postgresql high rollback rate incident

This incident type refers to the occurrence of deadlocks in a PostgreSQL database. A deadlock happens when two or more transactions are waiting for each other to release locks on resources, resulting in a situation where none of the transactions can proceed. This can cause significant disruption to the database and the applications that rely on it, as it prevents any further progress until the deadlock is resolved.


Postgresql Deadlocks Incident

This incident type refers to a situation where there is a significant delay in the execution of queries on a Cassandra cluster. This delay can cause the system to become unresponsive and result in slower performance. It may be caused by a variety of factors such as an increase in traffic, inefficient queries, or hardware issues. The issue can impact the functionality of the system and requires immediate attention to prevent further disruption.


Slow Query Performance on Cassandra Cluster.

```shell
export CASSANDRA_JMX_PASSWORD="PLACEHOLDER"

export CASSANDRA_NODE_IP="PLACEHOLDER"

export CASSANDRA_JMX_USERNAME="PLACEHOLDER"

export CASSANDRA_JMX_PORT="PLACEHOLDER"

```


### Check the load on the system

```shell
top
```

### Check the load on the CPU

```shell
mpstat
```

### Check the CPU usage per process

```shell
ps -eo %cpu,pid,cmd | sort -k 1 -r | head -10
```

### Check the memory usage per process

```shell
ps -eo %mem,pid,cmd | sort -k 1 -r | head -10
```

### Check the number of connections to Cassandra

```shell
nodetool tpstats
```

### Check the number of running threads in Cassandra

```shell
nodetool -h ${CASSANDRA_NODE_IP} -p ${CASSANDRA_JMX_PORT} -u ${CASSANDRA_JMX_USERNAME} -pw ${CASSANDRA_JMX_PASSWORD} -ssl enable get /metrics/thread-pool/RequestResponseStage/ActiveTasks
```

### Check the Cassandra system log for errors

```shell
tail -f /var/log/cassandra/system.log
```

### Check the Cassandra slow query log for queries taking too long

```shell
tail -f /var/log/cassandra/slowquery.log
```

### Multiple queries trying to access the same Cassandra table simultaneously, leading to contention and deadlock.

```shell


#!/bin/bash



# Set the table name

table=${TABLE_NAME}



# Get the process IDs of all Cassandra instances

cassandra_pids=$(pgrep -f "cassandra")



# Loop through each process ID and check for contention on the specified table

for pid in $cassandra_pids; do

    # Use nodetool to get the thread dump for the process

    thread_dump=$(nodetool --pid $pid --threaddump)



    # Check if the table name appears in the thread dump

    if echo "$thread_dump" | grep -q "$table"; then

        # If it does, print a warning message

        echo "WARNING: Contention detected on table $table in process $pid"

    fi

done


```


### Tune the Cassandra cluster settings to optimize performance and minimize the risk of deadlocks.

```shell

#!/bin/bash

CLUSTER_NAME=${CLUSTER_NAME}

SEED_NODE_IP=${SEED_NODE_IP}

NUM_TOKENS=${NUM_TOKENS}

CONCURRENT_READS=${CONCURRENT_READS}

CONCURRENT_WRITES=${CONCURRENT_WRITES}

COMMITLOG_SYNC=${COMMITLOG_SYNC}

sed -i "s/cluster_name: .*/cluster_name: $CLUSTER_NAME/g" /etc/cassandra/cassandra.yaml

sed -i "s/seeds: .*/seeds: \"$SEED_NODE_IP\"/g" /etc/cassandra/cassandra.yaml

sed -i "s/num_tokens: .*/num_tokens: $NUM_TOKENS/g" /etc/cassandra/cassandra.yaml

sed -i "s/concurrent_reads: .*/concurrent_reads: $CONCURRENT_READS/g" /etc/cassandra/cassandra.yaml

sed -i "s/concurrent_writes: .*/concurrent_writes: $CONCURRENT_WRITES/g" /etc/cassandra/cassandra.yaml

sed -i "s/commitlog_sync: .*/commitlog_sync: $COMMITLOG_SYNC/g" /etc/cassandra/cassandra.yaml


```


Deadlock caused by concurrent transactions in Cassandra.

Overview

Parameters

Debug

Check the load on the system

Check the load on the CPU

Check the CPU usage per process

Check the memory usage per process

Check the number of connections to Cassandra

Check the number of running threads in Cassandra

Check the Cassandra system log for errors

Check the Cassandra slow query log for queries taking too long

Multiple queries trying to access the same Cassandra table simultaneously, leading to contention and deadlock.

Repair

Tune the Cassandra cluster settings to optimize performance and minimize the risk of deadlocks.

Learn more

Related Runbooks

Tomcat Thread Deadlock Conditions Detected

SQL Server Deadlock Incident

Postgresql high rollback rate incident

Postgresql Deadlocks Incident

Support