---
id: 4c019ab2-baf1-4feb-b34b-dd8600df6ea1
---

# High percentage of blocked items in Jenkins queue.
---

This incident type refers to a situation where there is a high percentage of blocked items in the Jenkins queue, which may be caused by various issues such as slow build or test execution times, network or server issues, or other performance-related problems. This can lead to delays in software development and deployment processes, resulting in longer wait times for developers and ultimately, slower time-to-market for software products.

### Parameters
```shell
# Environment Variables

export JENKINS_URL="PLACEHOLDER"

export JENKINS_NODE="PLACEHOLDER"

export NAME_OF_BUILD_AGENT="PLACEHOLDER"

export BUILD_AGENT_URL="PLACEHOLDER"

export MAX_LOAD="PLACEHOLDER"

export JENKINS_API_TOKEN="PLACEHOLDER"

export NEW_EXECUTORS="PLACEHOLDER"
```

## Debug

### Check Jenkins queue size
```shell
curl -s -X GET ${JENKINS_URL}/queue/api/json | jq '.items | length'
```

### Check if Jenkins queue is blocked
```shell
curl -s -X GET ${JENKINS_URL}/queue/api/json | jq '.items[].blocked'
```

### Check if Jenkins server is running
```shell
systemctl status jenkins
```

### Check if Jenkins is overloaded
```shell
top -bn1 | grep -Ei 'jenkins|java'
```

### Check if Jenkins is running out of memory
```shell
curl -s -X GET ${JENKINS_URL}/computer/api/json | jq '.computer[].executors[].currentExecutable.url' | xargs curl -s -X GET | jq '.actions[].causes[].shortDescription' | grep 'OutOfMemoryError'
```

### Check Jenkins logs for errors
```shell
tail -n 50 /var/log/jenkins/jenkins.log | grep -Ei 'error|exception'
```

### Check if there are any issues with the Jenkins node
```shell
ssh ${JENKINS_NODE} 'uptime; free -m; df -h'
```

### An issue with the build agents, such as agents being offline or misconfigured, causing the high percentage of blocked items in the queue.
```shell
bash

#!/bin/bash



# Define variables

BUILD_AGENT_NAME="${NAME_OF_BUILD_AGENT}"

BUILD_AGENT_STATUS="$(curl -s -o /dev/null -w '%{http_code}' http://${BUILD_AGENT_URL}/status)"



# Check if build agent is offline

if [[ "$BUILD_AGENT_STATUS" -ne 200 ]]; then

    echo "ERROR: $BUILD_AGENT_NAME is offline or misconfigured."

    echo "Please check the status of $BUILD_AGENT_NAME and ensure it is properly configured."

    exit 1

else

    echo "$BUILD_AGENT_NAME is online and properly configured."

    exit 0

fi


```

### An issue with the Jenkins server, such as a bug in the Jenkins software or an overload of the server, causing the high percentage of blocked items in the queue.
```shell


#!/bin/bash



# Check if Jenkins is running

jenkins_status=$(systemctl status jenkins | grep "Active" | awk '{print $2}')



if [ "$jenkins_status" == "inactive" ]; then

  echo "Jenkins is not running"

  exit 1

fi



# Check if Jenkins is overloaded

queue_size=$(curl -s -X GET http://${JENKINS_URL}/queue/api/json | jq '.items | length')

blocked_items=$(curl -s -X GET http://${JENKINS_URL}/queue/api/json | jq '.items[].blocked')



if [ "$queue_size" -eq 0 ]; then

  echo "Jenkins queue is empty"

  exit 1

fi



if [ "$blocked_items" -gt "$queue_size" ]; then

  echo "Jenkins queue is overloaded with blocked items"

  exit 1

fi



echo "Jenkins is running and queue is processing normally"


```

## Repair

### Increase the number of executors on the Jenkins server to improve job processing and reduce queue blockages.
```shell
#!/bin/bash

# Set the Jenkins URL and API token

JENKINS_URL=${JENKINS_URL}

JENKINS_API_TOKEN=${JENKINS_API_TOKEN}

# Set the number of executors to increase to

NEW_EXECUTORS=${NEW_EXECUTORS}

# Get the current number of executors

CURRENT_EXECUTORS=$(curl --silent "${JENKINS_URL}/computer/api/json" | jq '.computer | .[].numExecutors')


# Check if the number of executors needs to be increased

if [ "$CURRENT_EXECUTORS" -lt "$NEW_EXECUTORS" ]; then

  # Increase the number of executors

  curl --silent --user "${JENKINS_API_TOKEN}:" -X POST "${JENKINS_URL}/computer/newConfig" \

       --data-urlencode "numExecutors=${NEW_EXECUTORS}"

  echo "Successfully increased the number of executors to ${NEW_EXECUTORS}."

else

  echo "The number of executors is already ${CURRENT_EXECUTORS}, no action needed."

fi

```

This incident type refers to a situation where there is a high percentage of blocked items in the Jenkins queue, which may be caused by various issues such as slow build or test execution times, network or server issues, or other performance-related problems. This can lead to delays in software development and deployment processes, resulting in longer wait times for developers and ultimately, slower time-to-market for software products.


In software engineering, a large number of suspended threads in Tomcat is an incident type that occurs when a high number of threads are created but are blocked from executing. This can happen due to a variety of reasons, such as slow database queries, network latency, or resource contention. When this occurs, the system may become unresponsive or slow, causing a degradation in performance or even a complete system failure. It is important to investigate and resolve this issue as quickly as possible to ensure the system can continue to function properly.


Tomcat Large Number of Suspended Threads Incident.

This incident type refers to a situation where the Tomcat server's JDBC connection pool usage surpasses its configured limits. This can lead to a shortage of available connections for incoming requests, resulting in slow response times or even server crashes. It is important to monitor and manage the JDBC connection pool usage to prevent such incidents from occurring.


Tomcat High JDBC Connection Pool Usage Incident.

This incident type refers to an increase in the number of errors per second on a Tomcat server, which could indicate an issue with the server itself, the host, a deployed application, or an application servlet. This could include errors generated when the Tomcat server runs out of memory, can't find a requested file or servlet, or is unable to serve a JSP due to syntax errors in the servlet codebase. This incident type requires immediate attention to diagnose and address the underlying issue.


Increase of the errors/second rate for Tomcat server

This incident type refers to a situation where Spark jobs are failing due to resource contentions in the cluster. When multiple Spark jobs are trying to access the same resources or data at the same time, it can cause a bottleneck that leads to job failures. This can happen when the resources in the cluster are not properly allocated or when the number of jobs running simultaneously exceeds the cluster's capacity to handle them. The result is that Spark jobs fail, leading to disruptions in data processing and analysis.


Spark job failures due to cluster resource contentions.

This incident type refers to a failure in one or more Spark executors during the execution of a job. Spark executors are worker processes that run computations and store data in memory or on disk. When an executor fails, it can cause the entire job to fail or result in degraded performance. This type of incident can occur for a variety of reasons, such as hardware or network issues, memory errors, or software bugs.


Spark executor failure during job execution.

```shell
# Environment Variables

export JENKINS_URL="PLACEHOLDER"

export JENKINS_NODE="PLACEHOLDER"

export NAME_OF_BUILD_AGENT="PLACEHOLDER"

export BUILD_AGENT_URL="PLACEHOLDER"

export MAX_LOAD="PLACEHOLDER"

export JENKINS_API_TOKEN="PLACEHOLDER"

export NEW_EXECUTORS="PLACEHOLDER"
```


### Check Jenkins queue size

```shell
curl -s -X GET ${JENKINS_URL}/queue/api/json | jq '.items | length'
```

### Check if Jenkins queue is blocked

```shell
curl -s -X GET ${JENKINS_URL}/queue/api/json | jq '.items[].blocked'
```

### Check if Jenkins server is running

```shell
systemctl status jenkins
```

### Check if Jenkins is overloaded

```shell
top -bn1 | grep -Ei 'jenkins|java'
```

### Check if Jenkins is running out of memory

```shell
curl -s -X GET ${JENKINS_URL}/computer/api/json | jq '.computer[].executors[].currentExecutable.url' | xargs curl -s -X GET | jq '.actions[].causes[].shortDescription' | grep 'OutOfMemoryError'
```

### Check Jenkins logs for errors

```shell
tail -n 50 /var/log/jenkins/jenkins.log | grep -Ei 'error|exception'
```

### Check if there are any issues with the Jenkins node

```shell
ssh ${JENKINS_NODE} 'uptime; free -m; df -h'
```

### An issue with the build agents, such as agents being offline or misconfigured, causing the high percentage of blocked items in the queue.

```shell
bash

#!/bin/bash



# Define variables

BUILD_AGENT_NAME="${NAME_OF_BUILD_AGENT}"

BUILD_AGENT_STATUS="$(curl -s -o /dev/null -w '%{http_code}' http://${BUILD_AGENT_URL}/status)"



# Check if build agent is offline

if [[ "$BUILD_AGENT_STATUS" -ne 200 ]]; then

    echo "ERROR: $BUILD_AGENT_NAME is offline or misconfigured."

    echo "Please check the status of $BUILD_AGENT_NAME and ensure it is properly configured."

    exit 1

else

    echo "$BUILD_AGENT_NAME is online and properly configured."

    exit 0

fi


```

### An issue with the Jenkins server, such as a bug in the Jenkins software or an overload of the server, causing the high percentage of blocked items in the queue.

```shell


#!/bin/bash



# Check if Jenkins is running

jenkins_status=$(systemctl status jenkins | grep "Active" | awk '{print $2}')



if [ "$jenkins_status" == "inactive" ]; then

  echo "Jenkins is not running"

  exit 1

fi



# Check if Jenkins is overloaded

queue_size=$(curl -s -X GET http://${JENKINS_URL}/queue/api/json | jq '.items | length')

blocked_items=$(curl -s -X GET http://${JENKINS_URL}/queue/api/json | jq '.items[].blocked')



if [ "$queue_size" -eq 0 ]; then

  echo "Jenkins queue is empty"

  exit 1

fi



if [ "$blocked_items" -gt "$queue_size" ]; then

  echo "Jenkins queue is overloaded with blocked items"

  exit 1

fi



echo "Jenkins is running and queue is processing normally"


```


### Increase the number of executors on the Jenkins server to improve job processing and reduce queue blockages.

```shell
#!/bin/bash

# Set the Jenkins URL and API token

JENKINS_URL=${JENKINS_URL}

JENKINS_API_TOKEN=${JENKINS_API_TOKEN}

# Set the number of executors to increase to

NEW_EXECUTORS=${NEW_EXECUTORS}

# Get the current number of executors

CURRENT_EXECUTORS=$(curl --silent "${JENKINS_URL}/computer/api/json" | jq '.computer | .[].numExecutors')


# Check if the number of executors needs to be increased

if [ "$CURRENT_EXECUTORS" -lt "$NEW_EXECUTORS" ]; then

  # Increase the number of executors

  curl --silent --user "${JENKINS_API_TOKEN}:" -X POST "${JENKINS_URL}/computer/newConfig" \

       --data-urlencode "numExecutors=${NEW_EXECUTORS}"

  echo "Successfully increased the number of executors to ${NEW_EXECUTORS}."

else

  echo "The number of executors is already ${CURRENT_EXECUTORS}, no action needed."

fi

```


High percentage of blocked items in Jenkins queue.

Overview

Parameters

Debug

Check Jenkins queue size

Check if Jenkins queue is blocked

Check if Jenkins server is running

Check if Jenkins is overloaded

Check if Jenkins is running out of memory

Check Jenkins logs for errors

Check if there are any issues with the Jenkins node

An issue with the build agents, such as agents being offline or misconfigured, causing the high percentage of blocked items in the queue.

An issue with the Jenkins server, such as a bug in the Jenkins software or an overload of the server, causing the high percentage of blocked items in the queue.

Repair

Increase the number of executors on the Jenkins server to improve job processing and reduce queue blockages.

Learn more

Related Runbooks

Tomcat Large Number of Suspended Threads Incident.

Tomcat High JDBC Connection Pool Usage Incident.

Increase of the errors/second rate for Tomcat server

Spark job failures due to cluster resource contentions.

Support