---
id: 63a4a865-0f39-494c-b3ff-e16340384885
---

# Airflow Worker Node Overload.
---

Apache Airflow is a platform used for creating, scheduling, and monitoring workflows. A worker node is a component of this platform that executes tasks and runs jobs in parallel. When a worker node is overloaded, it means that it is unable to handle the number of tasks assigned to it, causing failures in the execution of workflows. This incident type refers to the situation where an Apache Airflow worker node is overloaded and needs to be addressed to ensure that the platform can continue to function properly.

### Parameters
```shell
export AIRFLOW_HOME="PLACEHOLDER"

export NEW_CPU_VALUE="PLACEHOLDER"

export NEW_MEM_VALUE="PLACEHOLDER"

export WORKER_NODE_CONFIG="PLACEHOLDER"

export CPU_THRESHOLD="PLACEHOLDER"

export DISK_THRESHOLD="PLACEHOLDER"

export MEMORY_THRESHOLD="PLACEHOLDER"
```

## Debug

### Check CPU usage of worker nodes
```shell
top -bn1 | grep "Cpu(s)" | sed "s/.*, *$[0-9.]*$%* id.*/\1/" | awk '{print 100 - $1"% CPU usage"}'
```

### Check memory usage of worker nodes
```shell
free -m | grep -i mem | awk '{print $3/$2 * 100.0"% memory usage"}'
```

### Check disk usage of worker nodes
```shell
df -h /
```

### Check airflow worker logs for errors
```shell
tail -f ${AIRFLOW_HOME}/logs/worker.log
```

### Check the number of running tasks on the worker nodes
```shell
ps aux | grep -i "airflow worker" | wc -l
```

### Configuration issues such as improper settings for worker node resources like CPU, memory, or disk space that do not align with the workload requirements.
```shell


#!/bin/bash



# Check CPU usage

cpu_usage=$(top -b -n1 | grep "Cpu(s)" | awk '{print $2 + $4}')

if [[ $(echo "$cpu_usage > ${CPU_THRESHOLD}" | bc -l) -eq 1 ]]; then

    echo "CPU usage is above threshold. Current usage: $cpu_usage%"

fi



# Check memory usage

mem_usage=$(free | grep Mem | awk '{print $3/$2 * 100.0}')

if [[ $(echo "$mem_usage > ${MEMORY_THRESHOLD}" | bc -l) -eq 1 ]]; then

    echo "Memory usage is above threshold. Current usage: $mem_usage%"

fi



# Check disk space usage

disk_usage=$(df -h / | awk '{print $5}' | tail -n 1 | tr -d '%')

if [[ $disk_usage -ge ${DISK_THRESHOLD} ]]; then

    echo "Disk space usage is above threshold. Current usage: $disk_usage%"

fi


```

## Repair

### Increase the capacity of the worker node by adding more resources such as CPU, memory, or storage.
```shell


#!/bin/bash



# Set the new CPU and memory values

NEW_CPU=${NEW_CPU_VALUE}

NEW_MEM=${NEW_MEM_VALUE}



# Get the current CPU and memory values

CUR_CPU=$(grep -c ^processor /proc/cpuinfo)

CUR_MEM=$(free -m | awk '/^Mem:/{print $2}')



# Calculate the new CPU and memory values

NEW_CPU_TOTAL=$(($CUR_CPU + $NEW_CPU))

NEW_MEM_TOTAL=$(($CUR_MEM + $NEW_MEM))



# Update the CPU and memory values in the worker node

sudo sed -i "s/^resources:/resources:\n  limits:\n    cpu: $NEW_CPU_TOTAL\n    memory: ${NEW_MEM_TOTAL}Mi/" /path/to/worker/node.yaml



# Restart the worker node to apply the changes

sudo systemctl restart airflow-worker


```


Apache Airflow is a platform used for creating, scheduling, and monitoring workflows. A worker node is a component of this platform that executes tasks and runs jobs in parallel. When a worker node is overloaded, it means that it is unable to handle the number of tasks assigned to it, causing failures in the execution of workflows. This incident type refers to the situation where an Apache Airflow worker node is overloaded and needs to be addressed to ensure that the platform can continue to function properly.


Apache Spark driver failure refers to an incident where the driver program in an Apache Spark cluster fails to execute or crashes during runtime. This can happen due to a variety of reasons such as hardware failure, software bugs, resource constraints, or programming errors. As the driver program is responsible for coordinating the execution of tasks across the cluster, any failure in the driver can result in the entire Spark job failing. This can lead to data loss, processing delays, and impact the overall performance of the Spark cluster.


Spark driver failure incident.

This type of incident occurs when a node in the Kafka cluster is not being fully utilized. Kafka is a distributed streaming platform, and each node in the cluster is responsible for processing and storing a portion of the data. If a node is underutilized, it means that it is not processing as much data as it should be, which can lead to performance issues and potential data loss. Therefore, it is important to identify and address underutilized nodes in a Kafka cluster to ensure optimal performance and data reliability.


Kafka node underutilization check incident.

This incident type refers to a problem with the load balancing functionality of the Apache HTTP server when using the mod\_jk connector. This can cause issues with distributing incoming traffic across multiple servers, resulting in reduced performance or even downtime for the affected services. It is important to address and resolve these issues as quickly as possible to minimize impact on users and ensure the stability of the system.


Mod_jk Load Balancing Issues in Apache HTTP.

This incident occurs when there is a high number of idle workers in the Apache server. Idle workers are connections that have been established by clients but are not currently processing any requests. When the number of idle workers becomes too high, it can lead to performance issues and potentially impact the server's ability to handle incoming requests. This can lead to slow response times or even server crashes.


High Number of Idle Workers in Apache Server

This incident type refers to a situation where the number of child processes created by an Apache server reaches a high threshold, causing performance issues and potentially leading to server crashes. This can occur due to factors such as high traffic volume or misconfigured server settings. The incident requires investigation and mitigation to restore normal server functioning and prevent future occurrences.


High Number of Apache Child Processes Incident

```shell
export AIRFLOW_HOME="PLACEHOLDER"

export NEW_CPU_VALUE="PLACEHOLDER"

export NEW_MEM_VALUE="PLACEHOLDER"

export WORKER_NODE_CONFIG="PLACEHOLDER"

export CPU_THRESHOLD="PLACEHOLDER"

export DISK_THRESHOLD="PLACEHOLDER"

export MEMORY_THRESHOLD="PLACEHOLDER"
```


### Check CPU usage of worker nodes

```shell
top -bn1 | grep "Cpu(s)" | sed "s/.*, *$[0-9.]*$%* id.*/\1/" | awk '{print 100 - $1"% CPU usage"}'
```

### Check memory usage of worker nodes

```shell
free -m | grep -i mem | awk '{print $3/$2 * 100.0"% memory usage"}'
```

### Check disk usage of worker nodes

```shell
df -h /
```

### Check airflow worker logs for errors

```shell
tail -f ${AIRFLOW_HOME}/logs/worker.log
```

### Check the number of running tasks on the worker nodes

```shell
ps aux | grep -i "airflow worker" | wc -l
```

### Configuration issues such as improper settings for worker node resources like CPU, memory, or disk space that do not align with the workload requirements.

```shell


#!/bin/bash



# Check CPU usage

cpu_usage=$(top -b -n1 | grep "Cpu(s)" | awk '{print $2 + $4}')

if [[ $(echo "$cpu_usage > ${CPU_THRESHOLD}" | bc -l) -eq 1 ]]; then

    echo "CPU usage is above threshold. Current usage: $cpu_usage%"

fi



# Check memory usage

mem_usage=$(free | grep Mem | awk '{print $3/$2 * 100.0}')

if [[ $(echo "$mem_usage > ${MEMORY_THRESHOLD}" | bc -l) -eq 1 ]]; then

    echo "Memory usage is above threshold. Current usage: $mem_usage%"

fi



# Check disk space usage

disk_usage=$(df -h / | awk '{print $5}' | tail -n 1 | tr -d '%')

if [[ $disk_usage -ge ${DISK_THRESHOLD} ]]; then

    echo "Disk space usage is above threshold. Current usage: $disk_usage%"

fi


```


### Increase the capacity of the worker node by adding more resources such as CPU, memory, or storage.

```shell


#!/bin/bash



# Set the new CPU and memory values

NEW_CPU=${NEW_CPU_VALUE}

NEW_MEM=${NEW_MEM_VALUE}



# Get the current CPU and memory values

CUR_CPU=$(grep -c ^processor /proc/cpuinfo)

CUR_MEM=$(free -m | awk '/^Mem:/{print $2}')



# Calculate the new CPU and memory values

NEW_CPU_TOTAL=$(($CUR_CPU + $NEW_CPU))

NEW_MEM_TOTAL=$(($CUR_MEM + $NEW_MEM))



# Update the CPU and memory values in the worker node

sudo sed -i "s/^resources:/resources:\n  limits:\n    cpu: $NEW_CPU_TOTAL\n    memory: ${NEW_MEM_TOTAL}Mi/" /path/to/worker/node.yaml



# Restart the worker node to apply the changes

sudo systemctl restart airflow-worker


```


Airflow Worker Node Overload.

Overview

Parameters

Debug

Check CPU usage of worker nodes

Check memory usage of worker nodes

Check disk usage of worker nodes

Check airflow worker logs for errors

Check the number of running tasks on the worker nodes

Configuration issues such as improper settings for worker node resources like CPU, memory, or disk space that do not align with the workload requirements.

Repair

Increase the capacity of the worker node by adding more resources such as CPU, memory, or storage.

Learn more

Related Runbooks

Spark driver failure incident.

Kafka node underutilization check incident.

Mod_jk Load Balancing Issues in Apache HTTP.

High Number of Idle Workers in Apache Server

Support