---
id: 2613aa8a-87a4-44ff-9200-0b7e5d254b8f
---

# Airflow server resource exhaustion during peak workflow executions.
---

This incident type describes an issue where the Airflow server, which is used to manage and schedule workflows, is running out of resources, such as CPU, memory, or disk space, during periods of high workflow executions. This can cause delays in workflow execution or even complete failures. It is important to monitor the server's resource usage and allocate sufficient resources to ensure smooth and uninterrupted workflow execution.

### Parameters
```shell
export RESOURCE_LIMIT="PLACEHOLDER"

export THRESHOLD="PLACEHOLDER"

export CLOUD_PROVIDER="PLACEHOLDER"

export INSTANCE_NAME="PLACEHOLDER"

export INSTANCE_IP="PLACEHOLDER"

export REGION_RESOURCE_GROUP_PROJECT_NAME="PLACEHOLDER"

export ZONE_NAME="PLACEHOLDER" #GCP ONLY

export NEW_INSTANCE_TYPE="PLACEHOLDER"
```

## Debug

### Check if there are any zombie processes
```shell
ps aux | grep Z
```

### Check CPU usage
```shell
top -b -n 1 | head -n 20
```

### Check memory usage
```shell
free -h
```

### Check disk space usage
```shell
df -h
```

### Check Airflow logs for errors or warnings
```shell
tail -n 100 /var/log/airflow/*.log
```

### Check Airflow configuration for resource limits
```shell
grep -R "${RESOURCE_LIMIT}" /etc/airflow/
```

### Check if Airflow workers are consuming too many resources
```shell
ps -aux | grep airflow-worker | awk '{print $2, $4, $11}' | sort -k2 -r
```

### Check if there are any blocked I/O operations
```shell
iotop -obn2
```

### The Airflow server is running on a machine with insufficient resources, such as low memory or CPU capacity, to handle peak workflow loads.
```shell


#!/bin/bash



# Check available memory

total_mem=$(free -m | grep Mem | awk '{print $2}')

used_mem=$(free -m | grep Mem | awk '{print $3}')

free_mem=$(($total_mem - $used_mem))



if [ $free_mem -lt ${THRESHOLD} ]; then

    echo "Insufficient memory available ($free_mem MB free)."

fi



# Check CPU usage

cpu_usage=$(top -bn1 | grep load | awk '{printf "%.2f\n", $(NF-2)}')



if (( $(echo "$cpu_usage > ${THRESHOLD}" | bc -l) )); then

    echo "High CPU usage detected ($cpu_usage%)."

fi


```

## Repair

### Scaling up server resources: One possible remediation strategy is to increase the resources available to the Airflow server during peak execution periods. This can be done by adding more CPU, memory, or disk space to the server or by moving to a more powerful server altogether.
```shell


#!/bin/bash



# Set variables
SERVER_NAME=${INSTANCE_NAME}
SERVER_IP=${INSTANCE_IP}
RGP_NAME=${REGION_RESOURCE_GROUP_PROJECT_NAME}
NEW_INSTANCE_TYPE=${NEW_INSTANCE_TYPE}
ZONE_NAME=${ZONE_NAME}


# Check if server is online

ping -c 1 $SERVER_IP > /dev/null

if [ $? -eq 0 ]; then

    echo "Server is online"

else

    echo "Server is offline"

    exit 1

fi

# Change instance types
case ${CLOUD_PROVIDER} in
    "aws")
        # Stop the instance before modifying
        aws ec2 stop-instances --region $RGP_NAME --instance-ids $SERVER_NAME

        # Modify the instance type
        aws ec2 modify-instance-attribute --region $RGP_NAME --instance-id $SERVER_NAME --instance-type $NEW_INSTANCE_TYPE

        # Start the instance again
        aws ec2 start-instances --region $RGP_NAME --instance-ids $SERVER_NAME
        ;;
    "azure")
        az vm resize --resource-group $RGP_NAME --name $SERVER_NAME --size $NEW_INSTANCE_TYPE
        ;;
    "gcp")
        gcloud compute instances set-machine-type $SERVER_NAME --machine-type $NEW_INSTANCE_TYPE --project $RGP_NAME --zone=$ZONE_NAME
        ;;
    *)
        echo "Error unsupported cloud provider! Supported cloud providers [aws, azure, gcp]."
        exit 1
esac


```

This incident type describes an issue where the Airflow server, which is used to manage and schedule workflows, is running out of resources, such as CPU, memory, or disk space, during periods of high workflow executions. This can cause delays in workflow execution or even complete failures. It is important to monitor the server's resource usage and allocate sufficient resources to ensure smooth and uninterrupted workflow execution.


This incident type refers to a situation where Spark tasks are failing due to out of memory errors. Spark is a distributed computing system used for big data processing. When the data volume exceeds the allocated memory, the Spark tasks fail, and the system generates an out of memory error. This type of incident can cause data processing delays or even system downtime, which can impact the overall performance of the application.


Spark tasks failing due to out of memory errors.

This incident type refers to a situation where Spark jobs are failing due to resource contentions in the cluster. When multiple Spark jobs are trying to access the same resources or data at the same time, it can cause a bottleneck that leads to job failures. This can happen when the resources in the cluster are not properly allocated or when the number of jobs running simultaneously exceeds the cluster's capacity to handle them. The result is that Spark jobs fail, leading to disruptions in data processing and analysis.


Spark job failures due to cluster resource contentions.

This incident type refers to a failure in one or more Spark executors during the execution of a job. Spark executors are worker processes that run computations and store data in memory or on disk. When an executor fails, it can cause the entire job to fail or result in degraded performance. This type of incident can occur for a variety of reasons, such as hardware or network issues, memory errors, or software bugs.


Spark executor failure during job execution.

This incident type refers to a situation where a Spark cluster experiences performance bottlenecks when it is subjected to peak loads. In other words, the Spark cluster struggles to handle the high volume of requests it receives during times of heavy traffic or increased demand. This can lead to slower processing times, delays, or even system crashes. Identifying and resolving the root cause of the bottlenecks is crucial to ensure the smooth functioning of the Spark cluster during peak loads.


Spark cluster bottlenecks during peak loads.

Apache Spark Out of Memory Error is a type of incident that occurs when the Apache Spark application running on a system is unable to allocate enough memory to perform its tasks. This can happen when the system's memory is insufficient or when there is a memory leak in the application's code. When this incident occurs, the application may crash, become unresponsive, or produce incorrect results. It is important to diagnose and resolve this issue as soon as possible to ensure the smooth functioning of the application.


Apache Spark Out of Memory Error incident.

```shell
export RESOURCE_LIMIT="PLACEHOLDER"

export THRESHOLD="PLACEHOLDER"

export CLOUD_PROVIDER="PLACEHOLDER"

export INSTANCE_NAME="PLACEHOLDER"

export INSTANCE_IP="PLACEHOLDER"

export REGION_RESOURCE_GROUP_PROJECT_NAME="PLACEHOLDER"

export ZONE_NAME="PLACEHOLDER" #GCP ONLY

export NEW_INSTANCE_TYPE="PLACEHOLDER"
```


### Check if there are any zombie processes

```shell
ps aux | grep Z
```

### Check CPU usage

```shell
top -b -n 1 | head -n 20
```

### Check memory usage

```shell
free -h
```

### Check disk space usage

```shell
df -h
```

### Check Airflow logs for errors or warnings

```shell
tail -n 100 /var/log/airflow/*.log
```

### Check Airflow configuration for resource limits

```shell
grep -R "${RESOURCE_LIMIT}" /etc/airflow/
```

### Check if Airflow workers are consuming too many resources

```shell
ps -aux | grep airflow-worker | awk '{print $2, $4, $11}' | sort -k2 -r
```

### Check if there are any blocked I/O operations

```shell
iotop -obn2
```

### The Airflow server is running on a machine with insufficient resources, such as low memory or CPU capacity, to handle peak workflow loads.

```shell


#!/bin/bash



# Check available memory

total_mem=$(free -m | grep Mem | awk '{print $2}')

used_mem=$(free -m | grep Mem | awk '{print $3}')

free_mem=$(($total_mem - $used_mem))



if [ $free_mem -lt ${THRESHOLD} ]; then

    echo "Insufficient memory available ($free_mem MB free)."

fi



# Check CPU usage

cpu_usage=$(top -bn1 | grep load | awk '{printf "%.2f\n", $(NF-2)}')



if (( $(echo "$cpu_usage > ${THRESHOLD}" | bc -l) )); then

    echo "High CPU usage detected ($cpu_usage%)."

fi


```


### Scaling up server resources: One possible remediation strategy is to increase the resources available to the Airflow server during peak execution periods. This can be done by adding more CPU, memory, or disk space to the server or by moving to a more powerful server altogether.

```shell


#!/bin/bash



# Set variables
SERVER_NAME=${INSTANCE_NAME}
SERVER_IP=${INSTANCE_IP}
RGP_NAME=${REGION_RESOURCE_GROUP_PROJECT_NAME}
NEW_INSTANCE_TYPE=${NEW_INSTANCE_TYPE}
ZONE_NAME=${ZONE_NAME}


# Check if server is online

ping -c 1 $SERVER_IP > /dev/null

if [ $? -eq 0 ]; then

    echo "Server is online"

else

    echo "Server is offline"

    exit 1

fi

# Change instance types
case ${CLOUD_PROVIDER} in
    "aws")
        # Stop the instance before modifying
        aws ec2 stop-instances --region $RGP_NAME --instance-ids $SERVER_NAME

        # Modify the instance type
        aws ec2 modify-instance-attribute --region $RGP_NAME --instance-id $SERVER_NAME --instance-type $NEW_INSTANCE_TYPE

        # Start the instance again
        aws ec2 start-instances --region $RGP_NAME --instance-ids $SERVER_NAME
        ;;
    "azure")
        az vm resize --resource-group $RGP_NAME --name $SERVER_NAME --size $NEW_INSTANCE_TYPE
        ;;
    "gcp")
        gcloud compute instances set-machine-type $SERVER_NAME --machine-type $NEW_INSTANCE_TYPE --project $RGP_NAME --zone=$ZONE_NAME
        ;;
    *)
        echo "Error unsupported cloud provider! Supported cloud providers [aws, azure, gcp]."
        exit 1
esac


```


Airflow server resource exhaustion during peak workflow executions.

Overview

Parameters

Debug

Check if there are any zombie processes

Check CPU usage

Check memory usage

Check disk space usage

Check Airflow logs for errors or warnings

Check Airflow configuration for resource limits

Check if Airflow workers are consuming too many resources

Check if there are any blocked I/O operations

The Airflow server is running on a machine with insufficient resources, such as low memory or CPU capacity, to handle peak workflow loads.

Repair

Scaling up server resources: One possible remediation strategy is to increase the resources available to the Airflow server during peak execution periods. This can be done by adding more CPU, memory, or disk space to the server or by moving to a more powerful server altogether.

Learn more

Related Runbooks

Spark tasks failing due to out of memory errors.

Spark job failures due to cluster resource contentions.

Spark executor failure during job execution.

Spark cluster bottlenecks during peak loads.

Support