---
id: f0445989-7202-4511-a608-ad66f1c44f4e
---

# Airflow DAGs Failed to Execute Incident
---

Airflow DAGs Failed to Execute Incident refers to an issue where the Directed Acyclic Graphs (DAGs) in Apache Airflow, a popular open-source platform to programmatically author, schedule, and monitor workflows, have failed to execute as expected. This can result in delays or failures in executing tasks that are integral to the workflow, leading to potential disruptions in the entire system.

### Parameters
```shell
export AIRFLOW_DAG_FOLDER="PLACEHOLDER"

export DAG_ID="PLACEHOLDER"

export TASK_ID="PLACEHOLDER"

export MEMORY="PLACEHOLDER"

export CPU_CORES="PLACEHOLDER"

export NUM_WORKERS="PLACEHOLDER"
```

## Debug

### Check if Airflow webserver is running
```shell
sudo systemctl status airflow-webserver.service
```

### Check if Airflow scheduler is running
```shell
sudo systemctl status airflow-scheduler.service
```

### Check if there are any errors in Airflow logs
```shell
sudo tail -f /var/log/airflow/*.log
```

### Check if DAGs are present in the Airflow dag folder
```shell
ls ${AIRFLOW_DAG_FOLDER}
```

### Check if DAGs are parsing correctly
```shell
airflow list_dags -sd ${AIRFLOW_DAG_FOLDER}
```

### Check if tasks are listed correctly in the DAG
```shell
airflow list_tasks ${DAG_ID}
```

### Check if task instances are running and not in a failed state
```shell
airflow task_state ${DAG_ID} ${TASK_ID}
```

### Check if task logs are showing any errors
```shell
airflow logs ${DAG_ID} ${TASK_ID}
```

### Check if dependencies for a DAG are met
```shell
airflow test ${DAG_ID} ${TASK_ID}
```

### Check if there are any issues with database connections
```shell
airflow connections
```

### Check if there are any issues with variables
```shell
airflow variables
```

### Check if there are any issues with plugins
```shell
airflow plugins
```

### Resource constraints: The Airflow server may not have enough resources to execute the DAGs. This could include issues with memory, CPU, or other resources that prevent the DAGs from being executed.
```shell


#!/bin/bash



# Check CPU usage

CPU_USAGE=$(top -bn1 | grep load | awk '{printf "%.2f\n", $(NF-2)}')

CPU_THRESHOLD=80



if (( $(echo "$CPU_USAGE > $CPU_THRESHOLD" | bc -l) )); then

    echo "CPU usage is high"

fi



# Check memory usage

MEM_USAGE=$(free | grep Mem | awk '{printf "%.2f\n", $3/$2 * 100.0}')

MEM_THRESHOLD=80



if (( $(echo "$MEM_USAGE > $MEM_THRESHOLD" | bc -l) )); then

    echo "Memory usage is high"

fi



# Check disk space

DISK_USAGE=$(df -h | awk '{if ($NF == "/") {print $5}}' | sed 's/%//g')

DISK_THRESHOLD=80



if (( $(echo "$DISK_USAGE > $DISK_THRESHOLD" | bc -l) )); then

    echo "Disk space is running low"

fi


```

## Repair

### Ensure that the Airflow environment is configured correctly, with the correct number of workers and resources allocated.
```shell


#!/bin/bash



# Set the desired number of workers and resources

num_workers=${NUM_WORKERS}

cpu_cores=${CPU_CORES}

memory=${MEMORY}



# Stop the Airflow scheduler and workers

systemctl stop airflow-scheduler

systemctl stop airflow-worker



# Update the Airflow configuration file with the desired settings

sed -i "s/executor = LocalExecutor/executor = CeleryExecutor/" /etc/airflow/airflow.cfg

sed -i "s/parallelism = 32/parallelism = $num_workers/" /etc/airflow/airflow.cfg

sed -i "s/dags_are_paused_at_creation = True/dags_are_paused_at_creation = False/" /etc/airflow/airflow.cfg

sed -i "s/worker_concurrency = 16/worker_concurrency = $num_workers/" /etc/airflow/airflow.cfg

sed -i "s/cpu_cores = 4/cpu_cores = $cpu_cores/" /etc/airflow/airflow.cfg

sed -i "s/memory = 8GB/memory = $memory/" /etc/airflow/airflow.cfg



# Start the Airflow scheduler and workers

systemctl start airflow-scheduler

systemctl start airflow-worker



echo "Airflow environment has been configured with $num_workers workers, $cpu_cores CPU cores, and $memory memory."


```

Airflow DAGs Failed to Execute Incident refers to an issue where the Directed Acyclic Graphs (DAGs) in Apache Airflow, a popular open-source platform to programmatically author, schedule, and monitor workflows, have failed to execute as expected. This can result in delays or failures in executing tasks that are integral to the workflow, leading to potential disruptions in the entire system.


This incident type refers to a failure in one or more Spark executors during the execution of a job. Spark executors are worker processes that run computations and store data in memory or on disk. When an executor fails, it can cause the entire job to fail or result in degraded performance. This type of incident can occur for a variety of reasons, such as hardware or network issues, memory errors, or software bugs.


Spark executor failure during job execution.

Apache Spark driver failure refers to an incident where the driver program in an Apache Spark cluster fails to execute or crashes during runtime. This can happen due to a variety of reasons such as hardware failure, software bugs, resource constraints, or programming errors. As the driver program is responsible for coordinating the execution of tasks across the cluster, any failure in the driver can result in the entire Spark job failing. This can lead to data loss, processing delays, and impact the overall performance of the Spark cluster.


Spark driver failure incident.

This incident type refers to a problem in a Spark application where it fails during the checkpointing process. Checkpointing is an important feature of Spark applications that allows for fault tolerance and recovery. When a checkpoint fails, it can cause data loss and potentially lead to application failure. This type of incident requires investigation to determine the root cause and implement a solution to prevent it from happening again.


Spark application failure during checkpointing.

Apache Spark Out of Memory Error is a type of incident that occurs when the Apache Spark application running on a system is unable to allocate enough memory to perform its tasks. This can happen when the system's memory is insufficient or when there is a memory leak in the application's code. When this incident occurs, the application may crash, become unresponsive, or produce incorrect results. It is important to diagnose and resolve this issue as soon as possible to ensure the smooth functioning of the application.


Apache Spark Out of Memory Error incident.

The Spark HDFS Connectivity Issue Incident refers to a situation where there is a problem with the connectivity between Apache Spark and Hadoop Distributed File System (HDFS). This can result in Spark being unable to access data stored in HDFS or facing performance issues due to slow connectivity. This type of incident can be caused by various factors such as network issues, misconfiguration, or software bugs. It can impact the overall functioning of Spark-based applications and require immediate attention to ensure smooth operations.


Spark HDFS Connectivity Issue Incident

```shell
export AIRFLOW_DAG_FOLDER="PLACEHOLDER"

export DAG_ID="PLACEHOLDER"

export TASK_ID="PLACEHOLDER"

export MEMORY="PLACEHOLDER"

export CPU_CORES="PLACEHOLDER"

export NUM_WORKERS="PLACEHOLDER"
```


### Check if Airflow webserver is running

```shell
sudo systemctl status airflow-webserver.service
```

### Check if Airflow scheduler is running

```shell
sudo systemctl status airflow-scheduler.service
```

### Check if there are any errors in Airflow logs

```shell
sudo tail -f /var/log/airflow/*.log
```

### Check if DAGs are present in the Airflow dag folder

```shell
ls ${AIRFLOW_DAG_FOLDER}
```

### Check if DAGs are parsing correctly

```shell
airflow list_dags -sd ${AIRFLOW_DAG_FOLDER}
```

### Check if tasks are listed correctly in the DAG

```shell
airflow list_tasks ${DAG_ID}
```

### Check if task instances are running and not in a failed state

```shell
airflow task_state ${DAG_ID} ${TASK_ID}
```

### Check if task logs are showing any errors

```shell
airflow logs ${DAG_ID} ${TASK_ID}
```

### Check if dependencies for a DAG are met

```shell
airflow test ${DAG_ID} ${TASK_ID}
```

### Check if there are any issues with database connections

```shell
airflow connections
```

### Check if there are any issues with variables

```shell
airflow variables
```

### Check if there are any issues with plugins

```shell
airflow plugins
```

### Resource constraints: The Airflow server may not have enough resources to execute the DAGs. This could include issues with memory, CPU, or other resources that prevent the DAGs from being executed.

```shell


#!/bin/bash



# Check CPU usage

CPU_USAGE=$(top -bn1 | grep load | awk '{printf "%.2f\n", $(NF-2)}')

CPU_THRESHOLD=80



if (( $(echo "$CPU_USAGE > $CPU_THRESHOLD" | bc -l) )); then

    echo "CPU usage is high"

fi



# Check memory usage

MEM_USAGE=$(free | grep Mem | awk '{printf "%.2f\n", $3/$2 * 100.0}')

MEM_THRESHOLD=80



if (( $(echo "$MEM_USAGE > $MEM_THRESHOLD" | bc -l) )); then

    echo "Memory usage is high"

fi



# Check disk space

DISK_USAGE=$(df -h | awk '{if ($NF == "/") {print $5}}' | sed 's/%//g')

DISK_THRESHOLD=80



if (( $(echo "$DISK_USAGE > $DISK_THRESHOLD" | bc -l) )); then

    echo "Disk space is running low"

fi


```


### Ensure that the Airflow environment is configured correctly, with the correct number of workers and resources allocated.

```shell


#!/bin/bash



# Set the desired number of workers and resources

num_workers=${NUM_WORKERS}

cpu_cores=${CPU_CORES}

memory=${MEMORY}



# Stop the Airflow scheduler and workers

systemctl stop airflow-scheduler

systemctl stop airflow-worker



# Update the Airflow configuration file with the desired settings

sed -i "s/executor = LocalExecutor/executor = CeleryExecutor/" /etc/airflow/airflow.cfg

sed -i "s/parallelism = 32/parallelism = $num_workers/" /etc/airflow/airflow.cfg

sed -i "s/dags_are_paused_at_creation = True/dags_are_paused_at_creation = False/" /etc/airflow/airflow.cfg

sed -i "s/worker_concurrency = 16/worker_concurrency = $num_workers/" /etc/airflow/airflow.cfg

sed -i "s/cpu_cores = 4/cpu_cores = $cpu_cores/" /etc/airflow/airflow.cfg

sed -i "s/memory = 8GB/memory = $memory/" /etc/airflow/airflow.cfg



# Start the Airflow scheduler and workers

systemctl start airflow-scheduler

systemctl start airflow-worker



echo "Airflow environment has been configured with $num_workers workers, $cpu_cores CPU cores, and $memory memory."


```


Airflow DAGs Failed to Execute Incident

Overview

Parameters

Debug

Check if Airflow webserver is running

Check if Airflow scheduler is running

Check if there are any errors in Airflow logs

Check if DAGs are present in the Airflow dag folder

Check if DAGs are parsing correctly

Check if tasks are listed correctly in the DAG

Check if task instances are running and not in a failed state

Check if task logs are showing any errors

Check if dependencies for a DAG are met

Check if there are any issues with database connections

Check if there are any issues with variables

Check if there are any issues with plugins

Resource constraints: The Airflow server may not have enough resources to execute the DAGs. This could include issues with memory, CPU, or other resources that prevent the DAGs from being executed.

Repair

Ensure that the Airflow environment is configured correctly, with the correct number of workers and resources allocated.

Learn more

Related Runbooks

Spark executor failure during job execution.

Spark driver failure incident.

Spark application failure during checkpointing.

Apache Spark Out of Memory Error incident.

Support