---
id: 8be6ab8d-76e3-40d1-bb67-dcfae44558ad
---

# Airflow worker resource exhaustion incident.
---

This incident type refers to situations where Apache Airflow workers have exhausted their resources. Apache Airflow is an open-source platform to programmatically author, schedule, and monitor workflows. Workers in Apache Airflow are responsible for executing tasks and are essential to the platform's functionality. Resource exhaustion in workers can cause a significant impact on the performance of the platform and can result in workflow failures. This incident type requires immediate attention to ensure the workers have sufficient resources to execute tasks, and the platform is functioning correctly.

### Parameters
```shell
export DESIRED_NUMBER_OF_WORKERS="PLACEHOLDER"

export AIRFLOW_LOG_FILE="PLACEHOLDER"

export WORKER_PID="PLACEHOLDER"

export WORKER_CONFIG_FILE="PLACEHOLDER"
```

## Debug

### Check the CPU usage of the affected worker node
```shell
top
```

### Check the disk usage of the affected worker node
```shell
df -h
```

### Check the memory usage of the affected worker node
```shell
free -m
```

### Check the currently running processes on the affected worker node
```shell
ps aux
```

### Identify any specific Airflow tasks that may be causing the resource exhaustion
```shell
grep "airflow" /var/log/syslog
```

### Check the Airflow logs for any errors or warnings that may be related to the resource exhaustion
```shell
tail -f /var/log/airflow/worker.log
```

### Restart the Airflow worker process on the affected node
```shell
sudo systemctl restart airflow-worker.service
```

### The worker may not have been configured with enough resources to handle the workload it was given.
```shell
bash

#!/bin/bash



# Displaying a message to inform the user that the script is starting.

echo "Starting diagnosis..."



# Checking the worker's resource usage.

# Replace ${WORKER_PID} with the PID of the worker process.

worker_pid=${WORKER_PID}

ps aux | grep $worker_pid | awk '{print "Worker CPU usage: " $3 "%, Memory usage: " $4 "%"}'



# Checking the worker's configuration.

# Replace ${WORKER_CONFIG_FILE} with the path to the worker's configuration file.

worker_config_file=${WORKER_CONFIG_FILE}

cat $worker_config_file



# Checking the workload being given to the worker.

# Replace ${AIRFLOW_LOG_FILE} with the path to the Apache Airflow log file.

airflow_log_file=${AIRFLOW_LOG_FILE}

grep "Task instance" $airflow_log_file | tail -n 10



# Displaying a message to inform the user that the diagnosis is complete.

echo "Diagnosis complete."


```

## Repair

### Scale up the number of airflow workers to ensure sufficient resources are available to handle the workload.
```shell


#!/bin/bash



# Get the desired number of workers from a configuration file or environment variable

NUM_WORKERS=${DESIRED_NUMBER_OF_WORKERS}



# Get the current number of workers

CURRENT_WORKERS=$(sudo systemctl status airflow-worker | grep "Active:" | grep -o "[0-9]*")



# If the current number of workers is less than the desired number, scale up

if [ "$CURRENT_WORKERS" -lt "$NUM_WORKERS" ]; then

    sudo systemctl stop airflow-worker

    sudo sed -i "s/num_processes=[0-9]*/num_processes=$NUM_WORKERS/" /etc/airflow/airflow.cfg

    sudo systemctl start airflow-worker

    echo "Scaled up the number of airflow workers to $NUM_WORKERS."

else

    echo "The current number of airflow workers is already sufficient."

fi


```


This incident type refers to situations where Apache Airflow workers have exhausted their resources. Apache Airflow is an open-source platform to programmatically author, schedule, and monitor workflows. Workers in Apache Airflow are responsible for executing tasks and are essential to the platform's functionality. Resource exhaustion in workers can cause a significant impact on the performance of the platform and can result in workflow failures. This incident type requires immediate attention to ensure the workers have sufficient resources to execute tasks, and the platform is functioning correctly.


Tomcat Connection Pool Exhaustion Incident occurs when the maximum number of connections that a Tomcat server can handle has been reached. This can happen due to an increase in traffic or a sudden surge in user activity. When the connection pool is exhausted, new requests cannot be handled, and the application may become unavailable. This type of incident requires immediate attention to ensure that the application is restored to its normal functioning state.


Tomcat Connection Pool Exhaustion Incident

This incident type refers to a situation where Spark jobs are failing due to resource contentions in the cluster. When multiple Spark jobs are trying to access the same resources or data at the same time, it can cause a bottleneck that leads to job failures. This can happen when the resources in the cluster are not properly allocated or when the number of jobs running simultaneously exceeds the cluster's capacity to handle them. The result is that Spark jobs fail, leading to disruptions in data processing and analysis.


Spark job failures due to cluster resource contentions.

This incident type refers to a failure in one or more Spark executors during the execution of a job. Spark executors are worker processes that run computations and store data in memory or on disk. When an executor fails, it can cause the entire job to fail or result in degraded performance. This type of incident can occur for a variety of reasons, such as hardware or network issues, memory errors, or software bugs.


Spark executor failure during job execution.

Apache Spark Out of Memory Error is a type of incident that occurs when the Apache Spark application running on a system is unable to allocate enough memory to perform its tasks. This can happen when the system's memory is insufficient or when there is a memory leak in the application's code. When this incident occurs, the application may crash, become unresponsive, or produce incorrect results. It is important to diagnose and resolve this issue as soon as possible to ensure the smooth functioning of the application.


Apache Spark Out of Memory Error incident.

This incident occurs when there is a high number of idle workers in the Apache server. Idle workers are connections that have been established by clients but are not currently processing any requests. When the number of idle workers becomes too high, it can lead to performance issues and potentially impact the server's ability to handle incoming requests. This can lead to slow response times or even server crashes.


High Number of Idle Workers in Apache Server

```shell
export DESIRED_NUMBER_OF_WORKERS="PLACEHOLDER"

export AIRFLOW_LOG_FILE="PLACEHOLDER"

export WORKER_PID="PLACEHOLDER"

export WORKER_CONFIG_FILE="PLACEHOLDER"
```


### Check the CPU usage of the affected worker node

```shell
top
```

### Check the disk usage of the affected worker node

```shell
df -h
```

### Check the memory usage of the affected worker node

```shell
free -m
```

### Check the currently running processes on the affected worker node

```shell
ps aux
```

### Identify any specific Airflow tasks that may be causing the resource exhaustion

```shell
grep "airflow" /var/log/syslog
```

### Check the Airflow logs for any errors or warnings that may be related to the resource exhaustion

```shell
tail -f /var/log/airflow/worker.log
```

### Restart the Airflow worker process on the affected node

```shell
sudo systemctl restart airflow-worker.service
```

### The worker may not have been configured with enough resources to handle the workload it was given.

```shell
bash

#!/bin/bash



# Displaying a message to inform the user that the script is starting.

echo "Starting diagnosis..."



# Checking the worker's resource usage.

# Replace ${WORKER_PID} with the PID of the worker process.

worker_pid=${WORKER_PID}

ps aux | grep $worker_pid | awk '{print "Worker CPU usage: " $3 "%, Memory usage: " $4 "%"}'



# Checking the worker's configuration.

# Replace ${WORKER_CONFIG_FILE} with the path to the worker's configuration file.

worker_config_file=${WORKER_CONFIG_FILE}

cat $worker_config_file



# Checking the workload being given to the worker.

# Replace ${AIRFLOW_LOG_FILE} with the path to the Apache Airflow log file.

airflow_log_file=${AIRFLOW_LOG_FILE}

grep "Task instance" $airflow_log_file | tail -n 10



# Displaying a message to inform the user that the diagnosis is complete.

echo "Diagnosis complete."


```


### Scale up the number of airflow workers to ensure sufficient resources are available to handle the workload.

```shell


#!/bin/bash



# Get the desired number of workers from a configuration file or environment variable

NUM_WORKERS=${DESIRED_NUMBER_OF_WORKERS}



# Get the current number of workers

CURRENT_WORKERS=$(sudo systemctl status airflow-worker | grep "Active:" | grep -o "[0-9]*")



# If the current number of workers is less than the desired number, scale up

if [ "$CURRENT_WORKERS" -lt "$NUM_WORKERS" ]; then

    sudo systemctl stop airflow-worker

    sudo sed -i "s/num_processes=[0-9]*/num_processes=$NUM_WORKERS/" /etc/airflow/airflow.cfg

    sudo systemctl start airflow-worker

    echo "Scaled up the number of airflow workers to $NUM_WORKERS."

else

    echo "The current number of airflow workers is already sufficient."

fi


```


Airflow worker resource exhaustion incident.

Overview

Parameters

Debug

Check the CPU usage of the affected worker node

Check the disk usage of the affected worker node

Check the memory usage of the affected worker node

Check the currently running processes on the affected worker node

Identify any specific Airflow tasks that may be causing the resource exhaustion

Restart the Airflow worker process on the affected node

The worker may not have been configured with enough resources to handle the workload it was given.

Repair

Scale up the number of airflow workers to ensure sufficient resources are available to handle the workload.

Learn more

Related Runbooks

Tomcat Connection Pool Exhaustion Incident

Spark job failures due to cluster resource contentions.

Spark executor failure during job execution.

Apache Spark Out of Memory Error incident.

Support