---
id: c762fa52-d11e-4117-a2c3-a9f3849fffbb
---

# Airflow worker node outage.
---

An Airflow worker node is a component of the Apache Airflow system that performs scheduled tasks and manages workflow execution. When a worker node goes down, it means that the tasks and workflows assigned to that node cannot be executed until the issue is resolved. This can cause delays and disruptions to scheduled processes and may require urgent attention from the system administrators to restore functionality. The incident may occur due to hardware or software failures, connectivity issues, or other technical problems.

### Parameters
```shell
export WORKER_NODE_IP_ADDRESS="PLACEHOLDER"

export WORKER_NODE_PORT="PLACEHOLDER"

export MEMORY_THRESHOLD="PLACEHOLDER"

export CPU_THRESHOLD="PLACEHOLDER"

export NEW_CPU_COUNT="PLACEHOLDER"

export NEW_DISK_SIZE="PLACEHOLDER"

export NODE_ID="PLACEHOLDER"

export NEW_MEMORY_SIZE="PLACEHOLDER"
```

## Debug

### Check if the Airflow worker node is responding to ping requests
```shell
ping ${WORKER_NODE_IP_ADDRESS}
```

### Check if the Airflow worker node is listening on the expected port
```shell
nc -vz ${WORKER_NODE_IP_ADDRESS} ${WORKER_NODE_PORT}
```

### Check if there are any Airflow-related processes running on the worker node
```shell
ps aux | grep airflow
```

### Check the logs of the Airflow worker process
```shell
tail -n 100 /var/log/airflow/worker.log
```

### Check the status of the Airflow worker service
```shell
systemctl status airflow-worker.service
```

### Insufficient resources on the Airflow worker node, leading to memory or CPU exhaustion.
```shell


#!/bin/bash



# Check if the node is running low on memory

if [[ $(free -m | awk 'NR==2{printf "%.2f%%", $3*100/$2}') > ${MEMORY_THRESHOLD} ]]; then

    echo "Memory usage is high. Current usage: $(free -m | awk 'NR==2{print $3}') MB"

fi



# Check if the node is running low on CPU

if [[ $(top -bn1 | grep load | awk '{printf "%.2f%%", $(NF-2)}') > ${CPU_THRESHOLD} ]]; then

    echo "CPU usage is high. Current usage: $(top -bn1 | grep load | awk '{printf "%.2f%%", $(NF-2)}')"

fi


```

## Repair

### If the issue persists, consider increasing the resources allocated to the worker node, such as CPU, memory, or disk space.
```shell
bash

#!/bin/bash



# Set the node ID of the affected worker node

NODE_ID=${NODE_ID}



# Increase CPU allocation for the worker node

sudo virsh setvcpus $NODE_ID ${NEW_CPU_COUNT}



# Increase memory allocation for the worker node

sudo virsh setmem $NODE_ID ${NEW_MEMORY_SIZE}



# Increase disk space allocation for the worker node

sudo virsh blockresize $NODE_ID vda --size ${NEW_DISK_SIZE}



echo "Resources updated for worker node $NODE_ID."


```

An Airflow worker node is a component of the Apache Airflow system that performs scheduled tasks and manages workflow execution. When a worker node goes down, it means that the tasks and workflows assigned to that node cannot be executed until the issue is resolved. This can cause delays and disruptions to scheduled processes and may require urgent attention from the system administrators to restore functionality. The incident may occur due to hardware or software failures, connectivity issues, or other technical problems.


Apache Spark driver failure refers to an incident where the driver program in an Apache Spark cluster fails to execute or crashes during runtime. This can happen due to a variety of reasons such as hardware failure, software bugs, resource constraints, or programming errors. As the driver program is responsible for coordinating the execution of tasks across the cluster, any failure in the driver can result in the entire Spark job failing. This can lead to data loss, processing delays, and impact the overall performance of the Spark cluster.


Spark driver failure incident.

This incident type refers to a failure in the primary PostgreSQL node on a Linux system. PostgreSQL is a popular open source relational database management system used by many organizations. The primary node is responsible for handling the majority of the traffic and data replication in a PostgreSQL cluster. When the primary node fails, it can result in data loss, slow performance, and potential downtime for users. This type of incident requires immediate attention and resolution to minimize the impact on users and ensure data integrity.


Primary PostgreSQL Node Failure on Linux

This type of incident occurs when a node in the Kafka cluster is not being fully utilized. Kafka is a distributed streaming platform, and each node in the cluster is responsible for processing and storing a portion of the data. If a node is underutilized, it means that it is not processing as much data as it should be, which can lead to performance issues and potential data loss. Therefore, it is important to identify and address underutilized nodes in a Kafka cluster to ensure optimal performance and data reliability.


Kafka node underutilization check incident.

This incident type involves the detection of an unresponsive node in a Cassandra cluster, which can result in data unavailability and potential disruptions to services. The cause of the issue may vary, but it can be related to factors such as hardware failure or network problems. It is important to address such incidents quickly to minimize any negative impact on the affected services and ensure data availability.


Cassandra cluster Node unresponsive and resulting data unavailability.

This incident type refers to a situation where a Cassandra node becomes unavailable. This can cause interruptions to the expected functionality of the system and can lead to data loss or corruption. The cause of this incident can be due to a variety of reasons such as hardware failure, network issues, or software bugs. It is critical to resolve the issue as soon as possible to minimize the impact on the system and ensure the smooth functioning of the application.


Cassandra Node Unavailability

```shell
export WORKER_NODE_IP_ADDRESS="PLACEHOLDER"

export WORKER_NODE_PORT="PLACEHOLDER"

export MEMORY_THRESHOLD="PLACEHOLDER"

export CPU_THRESHOLD="PLACEHOLDER"

export NEW_CPU_COUNT="PLACEHOLDER"

export NEW_DISK_SIZE="PLACEHOLDER"

export NODE_ID="PLACEHOLDER"

export NEW_MEMORY_SIZE="PLACEHOLDER"
```


### Check if the Airflow worker node is responding to ping requests

```shell
ping ${WORKER_NODE_IP_ADDRESS}
```

### Check if the Airflow worker node is listening on the expected port

```shell
nc -vz ${WORKER_NODE_IP_ADDRESS} ${WORKER_NODE_PORT}
```

### Check if there are any Airflow-related processes running on the worker node

```shell
ps aux | grep airflow
```

### Check the logs of the Airflow worker process

```shell
tail -n 100 /var/log/airflow/worker.log
```

### Check the status of the Airflow worker service

```shell
systemctl status airflow-worker.service
```

### Insufficient resources on the Airflow worker node, leading to memory or CPU exhaustion.

```shell


#!/bin/bash



# Check if the node is running low on memory

if [[ $(free -m | awk 'NR==2{printf "%.2f%%", $3*100/$2}') > ${MEMORY_THRESHOLD} ]]; then

    echo "Memory usage is high. Current usage: $(free -m | awk 'NR==2{print $3}') MB"

fi



# Check if the node is running low on CPU

if [[ $(top -bn1 | grep load | awk '{printf "%.2f%%", $(NF-2)}') > ${CPU_THRESHOLD} ]]; then

    echo "CPU usage is high. Current usage: $(top -bn1 | grep load | awk '{printf "%.2f%%", $(NF-2)}')"

fi


```


### If the issue persists, consider increasing the resources allocated to the worker node, such as CPU, memory, or disk space.

```shell
bash

#!/bin/bash



# Set the node ID of the affected worker node

NODE_ID=${NODE_ID}



# Increase CPU allocation for the worker node

sudo virsh setvcpus $NODE_ID ${NEW_CPU_COUNT}



# Increase memory allocation for the worker node

sudo virsh setmem $NODE_ID ${NEW_MEMORY_SIZE}



# Increase disk space allocation for the worker node

sudo virsh blockresize $NODE_ID vda --size ${NEW_DISK_SIZE}



echo "Resources updated for worker node $NODE_ID."


```


Airflow worker node outage.

Overview

Parameters

Debug

Check if the Airflow worker node is responding to ping requests

Check if the Airflow worker node is listening on the expected port

Check the logs of the Airflow worker process

Check the status of the Airflow worker service

Insufficient resources on the Airflow worker node, leading to memory or CPU exhaustion.

Repair

If the issue persists, consider increasing the resources allocated to the worker node, such as CPU, memory, or disk space.

Learn more

Related Runbooks

Spark driver failure incident.

Primary PostgreSQL Node Failure on Linux

Kafka node underutilization check incident.

Cassandra cluster Node unresponsive and resulting data unavailability.

Support