---
id: bf5c571d-45b2-455e-a00b-df972943c99a
---

#  Tomcat Thread Deadlock Conditions Detected
---

Thread deadlock conditions occur when two or more threads are blocked and waiting for each other to release a resource, causing a system to hang or become unresponsive. In the context of a Tomcat server, this could cause the server to stop responding to incoming requests. Detection of thread deadlock conditions on Tomcat is an indication of a critical issue that requires immediate attention to restore normal server operation.

### Parameters
```shell
export PID_OF_TOMCAT="PLACEHOLDER"

export PATH_TO_TOMCAT="PLACEHOLDER"

export PATH_TO_SERVER_XML="PLACEHOLDER"

export PATH_TO_TOMCAT_HOME="PLACEHOLDER"

export NEW_THREAD_POOL_SIZE="PLACEHOLDER"
```

## Debug

### Check the number of threads that Tomcat is currently running
```shell
ps aux | grep tomcat
```

### Check the number of idle threads
```shell
${PATH_TO_TOMCAT}/bin/jstack ${PID_OF_TOMCAT} | grep "java.lang.Thread.State: WAITING (parking)"

# If the output shows a high number of WAITING threads, it could indicate a deadlock
```

### Check the status of the Tomcat server
```shell
sudo systemctl status tomcat
```

### Check the server logs for errors related to thread deadlock
```shell
cat ${PATH_TO_TOMCAT}/logs/catalina.out | grep "deadlock"
```

### Check the current memory usage of Tomcat
```shell
jmap -heap ${PID_OF_TOMCAT}
```

### Check which threads are holding onto locks
```shell
${PATH_TO_TOMCAT}/bin/jstack ${PID_OF_TOMCAT} | grep -E "^\".*\".*$|^.*waiting on.*$"
```

### Incorrect synchronization of threads leading to deadlock.
```shell


#!/bin/bash



# Check if the JVM is running

if ! pgrep java >/dev/null 2>&1; then

    echo "JVM not running. Exiting."

    exit 1

fi



# Get the process ID of the Tomcat instance

tomcat_pid=$(ps aux | grep tomcat | grep -v grep | awk '{print $2}')



# Check if there are any deadlocked threads

deadlocked_threads=$(jstack $tomcat_pid | grep "java.lang.Thread.State: BLOCKED" | wc -l)



if [[ $deadlocked_threads -eq 0 ]]; then

    echo "No deadlocked threads found."

else

    echo "$deadlocked_threads deadlocked threads found."

    

    # Print information about the deadlocked threads

    jstack $tomcat_pid | grep "java.lang.Thread.State: BLOCKED" -A 1

fi


```

## Repair

### Restart Tomcat: This is the most common solution to fix the thread deadlock conditions. Restarting Tomcat forces all threads to stop and start fresh.
```shell
bash

#!/bin/bash



# Stop Tomcat

sudo systemctl stop tomcat.service



# Wait for Tomcat to stop

while [ "$(sudo systemctl is-active tomcat.service)" == "active" ]

do

  sleep 5

done



# Start Tomcat

sudo systemctl start tomcat.service


```

### Increase Thread Pool Size: A deadlock can occur when there are not enough threads available to execute requests. Increasing the size of the thread pool can help avoid deadlock conditions.
```shell


#!/bin/bash



# Define variables

TOMCAT_HOME=${PATH_TO_TOMCAT_HOME}

SERVER_XML=${PATH_TO_SERVER_XML}

THREAD_POOL_SIZE=${NEW_THREAD_POOL_SIZE}



# Stop Tomcat

${TOMCAT_HOME}/bin/shutdown.sh



# Update server.xml with new thread pool size

sed -i "s/maxThreads=\"[0-9]*\"/maxThreads=\"$THREAD_POOL_SIZE\"/" $SERVER_XML


# Start Tomcat

sudo systemctl start tomcat.service

```

Thread deadlock conditions occur when two or more threads are blocked and waiting for each other to release a resource, causing a system to hang or become unresponsive. In the context of a Tomcat server, this could cause the server to stop responding to incoming requests. Detection of thread deadlock conditions on Tomcat is an indication of a critical issue that requires immediate attention to restore normal server operation.


When too many requests are sent to a Tomcat server, the thread pools that handle these requests can run out of available threads. This means that incoming requests will not be processed, leading to service degradation or even a complete outage. This incident type is critical for web applications that rely on Tomcat as their application server.


Tomcat thread pool exhaustion.

Tomcat Server Crash incident type refers to a situation where the Tomcat Server, an open-source web server and servlet container, becomes unresponsive and stops functioning. This can result in a complete or partial system outage, leading to a disruption in the delivery of web applications and services. The root cause of the crash can be attributed to a variety of factors such as misconfiguration, network or hardware issues, software bugs, or excessive server load. It is critical to address this incident quickly to minimize the impact on users and restore normal system functionality.


Tomcat Server Crash

This incident type is related to the Tomcat server, which is a popular open-source Java application server. It occurs when the available threads in the connection pool are running low, which means that there are not enough resources to serve incoming requests. This can cause delays or even failures in the application's performance, impacting the end-user experience. It is important to monitor and address this issue promptly to ensure that the application runs smoothly and efficiently.


Tomcat Low Available Threads in Connection Pool

In software engineering, a large number of suspended threads in Tomcat is an incident type that occurs when a high number of threads are created but are blocked from executing. This can happen due to a variety of reasons, such as slow database queries, network latency, or resource contention. When this occurs, the system may become unresponsive or slow, causing a degradation in performance or even a complete system failure. It is important to investigate and resolve this issue as quickly as possible to ensure the system can continue to function properly.


Tomcat Large Number of Suspended Threads Incident.

Tomcat JVM OutOfMemory incident occurs when the Java Virtual Machine (JVM) running the Apache Tomcat server runs out of memory, which causes the server to crash or become unresponsive. This type of incident can happen due to a variety of reasons such as incorrect configuration settings, memory leaks, or insufficient memory allocation. When this incident occurs, it can impact the availability and performance of the web application running on the Tomcat server.


Tomcat JVM OutOfMemory Incident

```shell
export PID_OF_TOMCAT="PLACEHOLDER"

export PATH_TO_TOMCAT="PLACEHOLDER"

export PATH_TO_SERVER_XML="PLACEHOLDER"

export PATH_TO_TOMCAT_HOME="PLACEHOLDER"

export NEW_THREAD_POOL_SIZE="PLACEHOLDER"
```


### Check the number of threads that Tomcat is currently running

```shell
ps aux | grep tomcat
```

### Check the number of idle threads

```shell
${PATH_TO_TOMCAT}/bin/jstack ${PID_OF_TOMCAT} | grep "java.lang.Thread.State: WAITING (parking)"

# If the output shows a high number of WAITING threads, it could indicate a deadlock
```

### Check the status of the Tomcat server

```shell
sudo systemctl status tomcat
```

### Check the server logs for errors related to thread deadlock

```shell
cat ${PATH_TO_TOMCAT}/logs/catalina.out | grep "deadlock"
```

### Check the current memory usage of Tomcat

```shell
jmap -heap ${PID_OF_TOMCAT}
```

### Check which threads are holding onto locks

```shell
${PATH_TO_TOMCAT}/bin/jstack ${PID_OF_TOMCAT} | grep -E "^\".*\".*$|^.*waiting on.*$"
```

### Incorrect synchronization of threads leading to deadlock.

```shell


#!/bin/bash



# Check if the JVM is running

if ! pgrep java >/dev/null 2>&1; then

    echo "JVM not running. Exiting."

    exit 1

fi



# Get the process ID of the Tomcat instance

tomcat_pid=$(ps aux | grep tomcat | grep -v grep | awk '{print $2}')



# Check if there are any deadlocked threads

deadlocked_threads=$(jstack $tomcat_pid | grep "java.lang.Thread.State: BLOCKED" | wc -l)



if [[ $deadlocked_threads -eq 0 ]]; then

    echo "No deadlocked threads found."

else

    echo "$deadlocked_threads deadlocked threads found."

    

    # Print information about the deadlocked threads

    jstack $tomcat_pid | grep "java.lang.Thread.State: BLOCKED" -A 1

fi


```


### Restart Tomcat: This is the most common solution to fix the thread deadlock conditions. Restarting Tomcat forces all threads to stop and start fresh.

```shell
bash

#!/bin/bash



# Stop Tomcat

sudo systemctl stop tomcat.service



# Wait for Tomcat to stop

while [ "$(sudo systemctl is-active tomcat.service)" == "active" ]

do

  sleep 5

done



# Start Tomcat

sudo systemctl start tomcat.service


```

### Increase Thread Pool Size: A deadlock can occur when there are not enough threads available to execute requests. Increasing the size of the thread pool can help avoid deadlock conditions.

```shell


#!/bin/bash



# Define variables

TOMCAT_HOME=${PATH_TO_TOMCAT_HOME}

SERVER_XML=${PATH_TO_SERVER_XML}

THREAD_POOL_SIZE=${NEW_THREAD_POOL_SIZE}



# Stop Tomcat

${TOMCAT_HOME}/bin/shutdown.sh



# Update server.xml with new thread pool size

sed -i "s/maxThreads=\"[0-9]*\"/maxThreads=\"$THREAD_POOL_SIZE\"/" $SERVER_XML


# Start Tomcat

sudo systemctl start tomcat.service

```


Tomcat Thread Deadlock Conditions Detected

Overview

Parameters

Debug

Check the number of threads that Tomcat is currently running

Check the number of idle threads

Check the status of the Tomcat server

Check the current memory usage of Tomcat

Check which threads are holding onto locks

Incorrect synchronization of threads leading to deadlock.

Repair

Restart Tomcat: This is the most common solution to fix the thread deadlock conditions. Restarting Tomcat forces all threads to stop and start fresh.

Increase Thread Pool Size: A deadlock can occur when there are not enough threads available to execute requests. Increasing the size of the thread pool can help avoid deadlock conditions.

Learn more

Related Runbooks

Tomcat thread pool exhaustion.

Tomcat Server Crash

Tomcat Low Available Threads in Connection Pool

Tomcat Large Number of Suspended Threads Incident.

Support