---
id: c40d1dc8-f8e2-47bf-82d9-7f3f1c2c5ea1
---

# Jenkins builds health score incident.
---

This incident type relates to issues with the health score of Jenkins builds. It could indicate that the builds are failing or not performing optimally. The incident could be triggered by failures in the health check process for a specific instance of Jenkins, and it may require investigation and remediation to restore normal operation.

### Parameters
```shell
export JENKINS_HOSTNAME="PLACEHOLDER"

export JENKINS_USERNAME="PLACEHOLDER"

export JENKINS_API_TOKEN="PLACEHOLDER"

export JOB_NAME="PLACEHOLDER"

export INSTANCE_NAME="PLACEHOLDER"

export METRICS_ENDPOINT="PLACEHOLDER"

export PROJECT_DIRECTORY="PLACEHOLDER"
```

## Debug

### Check if Jenkins is running
```shell
systemctl status jenkins
```

### Check if Jenkins is reachable from the affected instance
```shell
ping ${JENKINS_HOSTNAME}
```

### Check if the Jenkins plugin for the affected instance is installed and updated
```shell
java -jar jenkins-cli.jar -s https://${JENKINS_HOSTNAME}/ list-plugins
```

### Check if the Jenkins job is configured correctly
```shell
curl -X GET -u ${JENKINS_USERNAME}:${JENKINS_API_TOKEN} https://${JENKINS_HOSTNAME}/job/${JOB_NAME}/config.xml
```

### Check the Jenkins logs for errors
```shell
tail -f /var/log/jenkins/jenkins.log
```

### Check the health score metrics for the affected instance
```shell
curl -X GET https://${METRICS_ENDPOINT}/${INSTANCE_NAME}/health
```

### Changes made to the codebase that caused compatibility issues with the Jenkins build server.
```shell


#!/bin/bash



# Set variables

PROJECT_DIRECTORY=${PROJECT_DIRECTORY}

JENKINS_URL=${JENKINS_HOSTNAME}

JENKINS_USERNAME=${JENKINS_USERNAME}

JENKINS_PASSWORD=${JENKINS_API_TOKEN}



# Step 1: Verify code changes

cd $PROJECT_DIRECTORY

git pull origin master # or whichever branch is being used

if [ $? -ne 0 ]; then

  echo "Error: Failed to pull latest changes from git"

  exit 1

fi



# Step 2: Build code locally

./build.sh # or any other build script used

if [ $? -ne 0 ]; then

  echo "Error: Failed to build code locally"

  exit 1

fi



# Step 3: Verify Jenkins build server configuration

curl -sSL -u $JENKINS_USERNAME:$JENKINS_PASSWORD https://$JENKINS_URL/api/json | jq '.jobs[] | select(.name == "${JOB_NAME}") | .color' # replace ${JOB_NAME} with the name of the Jenkins job

if [ $? -ne 0 ]; then

  echo "Error: Failed to verify Jenkins build server configuration"

  exit 1

fi



# Step 4: Trigger Jenkins build

curl -sSL -u $JENKINS_USERNAME:$JENKINS_PASSWORD -X POST https://$JENKINS_URL/job/${JOB_NAME}/build # replace ${JOB_NAME} with the name of the Jenkins job

if [ $? -ne 0 ]; then

  echo "Error: Failed to trigger Jenkins build"

  exit 1

fi



# Step 5: Monitor Jenkins build logs

curl -sSL -u $JENKINS_USERNAME:$JENKINS_PASSWORD https://$JENKINS_URL/job/${JOB_NAME}/lastBuild/consoleText # replace ${JOB_NAME} with the name of the Jenkins job

if [ $? -ne 0 ]; then

  echo "Error: Failed to monitor Jenkins build logs"

  exit 1

fi



echo "Success: Diagnosis complete"

exit 0


```

## Repair

### Restart the Jenkins server to clear any cache or memory-related issues.
```shell


#!/bin/bash

sudo systemctl restart jenkins.service


```

This incident type relates to issues with the health score of Jenkins builds. It could indicate that the builds are failing or not performing optimally. The incident could be triggered by failures in the health check process for a specific instance of Jenkins, and it may require investigation and remediation to restore normal operation.


The Vault cluster health incident is related to the health of a Vault cluster instance. This incident type is triggered when the cluster instance is not healthy and requires attention to ensure it is functioning properly. The incident typically involves evaluating the current state of the cluster instance, diagnosing the issue, and taking corrective action to restore the health of the instance.


Vault cluster health incident on kubernetes

This incident type refers to a situation where the Tomcat server's JDBC connection pool usage surpasses its configured limits. This can lead to a shortage of available connections for incoming requests, resulting in slow response times or even server crashes. It is important to monitor and manage the JDBC connection pool usage to prevent such incidents from occurring.


Tomcat High JDBC Connection Pool Usage Incident.

This incident type refers to an increase in the number of errors per second on a Tomcat server, which could indicate an issue with the server itself, the host, a deployed application, or an application servlet. This could include errors generated when the Tomcat server runs out of memory, can't find a requested file or servlet, or is unable to serve a JSP due to syntax errors in the servlet codebase. This incident type requires immediate attention to diagnose and address the underlying issue.


Increase of the errors/second rate for Tomcat server

This incident type refers to a situation where Spark jobs are failing due to resource contentions in the cluster. When multiple Spark jobs are trying to access the same resources or data at the same time, it can cause a bottleneck that leads to job failures. This can happen when the resources in the cluster are not properly allocated or when the number of jobs running simultaneously exceeds the cluster's capacity to handle them. The result is that Spark jobs fail, leading to disruptions in data processing and analysis.


Spark job failures due to cluster resource contentions.

This incident type refers to a failure in one or more Spark executors during the execution of a job. Spark executors are worker processes that run computations and store data in memory or on disk. When an executor fails, it can cause the entire job to fail or result in degraded performance. This type of incident can occur for a variety of reasons, such as hardware or network issues, memory errors, or software bugs.


Spark executor failure during job execution.

```shell
export JENKINS_HOSTNAME="PLACEHOLDER"

export JENKINS_USERNAME="PLACEHOLDER"

export JENKINS_API_TOKEN="PLACEHOLDER"

export JOB_NAME="PLACEHOLDER"

export INSTANCE_NAME="PLACEHOLDER"

export METRICS_ENDPOINT="PLACEHOLDER"

export PROJECT_DIRECTORY="PLACEHOLDER"
```


### Check if Jenkins is running

```shell
systemctl status jenkins
```

### Check if Jenkins is reachable from the affected instance

```shell
ping ${JENKINS_HOSTNAME}
```

### Check if the Jenkins plugin for the affected instance is installed and updated

```shell
java -jar jenkins-cli.jar -s https://${JENKINS_HOSTNAME}/ list-plugins
```

### Check if the Jenkins job is configured correctly

```shell
curl -X GET -u ${JENKINS_USERNAME}:${JENKINS_API_TOKEN} https://${JENKINS_HOSTNAME}/job/${JOB_NAME}/config.xml
```

### Check the Jenkins logs for errors

```shell
tail -f /var/log/jenkins/jenkins.log
```

### Check the health score metrics for the affected instance

```shell
curl -X GET https://${METRICS_ENDPOINT}/${INSTANCE_NAME}/health
```

### Changes made to the codebase that caused compatibility issues with the Jenkins build server.

```shell


#!/bin/bash



# Set variables

PROJECT_DIRECTORY=${PROJECT_DIRECTORY}

JENKINS_URL=${JENKINS_HOSTNAME}

JENKINS_USERNAME=${JENKINS_USERNAME}

JENKINS_PASSWORD=${JENKINS_API_TOKEN}



# Step 1: Verify code changes

cd $PROJECT_DIRECTORY

git pull origin master # or whichever branch is being used

if [ $? -ne 0 ]; then

  echo "Error: Failed to pull latest changes from git"

  exit 1

fi



# Step 2: Build code locally

./build.sh # or any other build script used

if [ $? -ne 0 ]; then

  echo "Error: Failed to build code locally"

  exit 1

fi



# Step 3: Verify Jenkins build server configuration

curl -sSL -u $JENKINS_USERNAME:$JENKINS_PASSWORD https://$JENKINS_URL/api/json | jq '.jobs[] | select(.name == "${JOB_NAME}") | .color' # replace ${JOB_NAME} with the name of the Jenkins job

if [ $? -ne 0 ]; then

  echo "Error: Failed to verify Jenkins build server configuration"

  exit 1

fi



# Step 4: Trigger Jenkins build

curl -sSL -u $JENKINS_USERNAME:$JENKINS_PASSWORD -X POST https://$JENKINS_URL/job/${JOB_NAME}/build # replace ${JOB_NAME} with the name of the Jenkins job

if [ $? -ne 0 ]; then

  echo "Error: Failed to trigger Jenkins build"

  exit 1

fi



# Step 5: Monitor Jenkins build logs

curl -sSL -u $JENKINS_USERNAME:$JENKINS_PASSWORD https://$JENKINS_URL/job/${JOB_NAME}/lastBuild/consoleText # replace ${JOB_NAME} with the name of the Jenkins job

if [ $? -ne 0 ]; then

  echo "Error: Failed to monitor Jenkins build logs"

  exit 1

fi



echo "Success: Diagnosis complete"

exit 0


```


### Restart the Jenkins server to clear any cache or memory-related issues.

```shell


#!/bin/bash

sudo systemctl restart jenkins.service


```


Jenkins builds health score incident.

Overview

Parameters

Debug

Check if Jenkins is running

Check if Jenkins is reachable from the affected instance

Check if the Jenkins plugin for the affected instance is installed and updated

Check if the Jenkins job is configured correctly

Check the Jenkins logs for errors

Check the health score metrics for the affected instance

Changes made to the codebase that caused compatibility issues with the Jenkins build server.

Repair

Learn more

Related Runbooks

Vault cluster health incident on kubernetes

Tomcat High JDBC Connection Pool Usage Incident.

Increase of the errors/second rate for Tomcat server

Spark job failures due to cluster resource contentions.

Support