---
id: f479968d-222b-4b31-be71-8fb8b0845aca
---

# Airflow scheduler outage.
---

An Airflow scheduler outage occurs when the Airflow scheduler, which is a platform used for programmatically authoring, scheduling, and monitoring workflows, stops working or becomes unresponsive. This can cause delays or failures in the execution of workflows, resulting in disruptions to business operations.

### Parameters
```shell
export PATH_TO_AIRFLOW_SCHEDULER_LOGS="PLACEHOLDER"

export TASK_ID="PLACEHOLDER"

export DAG_ID="PLACEHOLDER"

export EXECUTION_DATE="PLACEHOLDER"
```

## Debug

### Check if the Airflow scheduler process is running
```shell
ps aux | grep airflow | grep scheduler
```

### Check the scheduler logs for errors
```shell
tail -n 100 ${PATH_TO_AIRFLOW_SCHEDULER_LOGS}
```

### Check if the database connection is working
```shell
airflow db check
```

### Check if there are any DAG parsing errors
```shell
airflow list_dags
```

### Check if the scheduler is able to schedule tasks
```shell
airflow test ${DAG_ID} ${TASK_ID} ${EXECUTION_DATE}
```

### Check if there are any issues with the worker processes
```shell
airflow worker
```

### Resource constraints, such as insufficient memory or processing power for the Airflow scheduler to function properly.
```shell
bash

#!/bin/bash



# Set the ${MEMORY_THRESHOLD} and ${CPU_THRESHOLD} values to monitor resource usage

memory_threshold=80

cpu_threshold=80



# Get the current memory and CPU usage for the Airflow scheduler process

memory_usage=$(ps -p $(pgrep -f "airflow scheduler") -o %mem | awk 'NR==2')

cpu_usage=$(ps -p $(pgrep -f "airflow scheduler") -o %cpu | awk 'NR==2')



# Check if the memory or CPU usage has exceeded the threshold

if (( $(echo "$memory_usage > $memory_threshold" | bc -l) )) || (( $(echo "$cpu_usage > $cpu_threshold" | bc -l) )); then

    echo "Resource constraints detected. Memory usage: $memory_usage%, CPU usage: $cpu_usage%"

    # Add any additional commands to diagnose and fix the issue here

else

    echo "No resource constraints detected. Memory usage: $memory_usage%, CPU usage: $cpu_usage%"

fi


```

## Repair

### Restart the Airflow scheduler to see if it resolves the issue.
```shell


#!/bin/bash



# Stop the Airflow scheduler

sudo systemctl restart airflow-scheduler.service



# Wait for a few seconds to ensure the process has restarted

sleep 5



# Check the status of the Airflow scheduler to ensure it's running

sudo systemctl status airflow-scheduler.service


```

An Airflow scheduler outage occurs when the Airflow scheduler, which is a platform used for programmatically authoring, scheduling, and monitoring workflows, stops working or becomes unresponsive. This can cause delays or failures in the execution of workflows, resulting in disruptions to business operations.


This incident type refers to a failure in one or more Spark executors during the execution of a job. Spark executors are worker processes that run computations and store data in memory or on disk. When an executor fails, it can cause the entire job to fail or result in degraded performance. This type of incident can occur for a variety of reasons, such as hardware or network issues, memory errors, or software bugs.


Spark executor failure during job execution.

This incident type refers to an unexpected termination of the Spark driver program during the runtime of a job. The driver program is responsible for coordinating the execution of a Spark job and if it crashes, the entire job is affected. This can result in data loss and downtime, and requires investigation and troubleshooting to identify the root cause of the issue.


Spark driver program crash during job runtime.

Apache Spark driver failure refers to an incident where the driver program in an Apache Spark cluster fails to execute or crashes during runtime. This can happen due to a variety of reasons such as hardware failure, software bugs, resource constraints, or programming errors. As the driver program is responsible for coordinating the execution of tasks across the cluster, any failure in the driver can result in the entire Spark job failing. This can lead to data loss, processing delays, and impact the overall performance of the Spark cluster.


Spark driver failure incident.

Spark YARN container failure is an incident type that occurs when a container running on the YARN (Yet Another Resource Negotiator) resource manager in a Spark cluster fails to complete its assigned task. This can be caused by a variety of factors such as resource constraints, hardware failures, or software bugs. When a container fails, it can cause delays or failures in the overall Spark job, leading to reduced performance or data loss.


Spark YARN Container Failure

Apache Spark Out of Memory Error is a type of incident that occurs when the Apache Spark application running on a system is unable to allocate enough memory to perform its tasks. This can happen when the system's memory is insufficient or when there is a memory leak in the application's code. When this incident occurs, the application may crash, become unresponsive, or produce incorrect results. It is important to diagnose and resolve this issue as soon as possible to ensure the smooth functioning of the application.


Apache Spark Out of Memory Error incident.

```shell
export PATH_TO_AIRFLOW_SCHEDULER_LOGS="PLACEHOLDER"

export TASK_ID="PLACEHOLDER"

export DAG_ID="PLACEHOLDER"

export EXECUTION_DATE="PLACEHOLDER"
```


### Check if the Airflow scheduler process is running

```shell
ps aux | grep airflow | grep scheduler
```

### Check the scheduler logs for errors

```shell
tail -n 100 ${PATH_TO_AIRFLOW_SCHEDULER_LOGS}
```

### Check if the database connection is working

```shell
airflow db check
```

### Check if there are any DAG parsing errors

```shell
airflow list_dags
```

### Check if the scheduler is able to schedule tasks

```shell
airflow test ${DAG_ID} ${TASK_ID} ${EXECUTION_DATE}
```

### Check if there are any issues with the worker processes

```shell
airflow worker
```

### Resource constraints, such as insufficient memory or processing power for the Airflow scheduler to function properly.

```shell
bash

#!/bin/bash



# Set the ${MEMORY_THRESHOLD} and ${CPU_THRESHOLD} values to monitor resource usage

memory_threshold=80

cpu_threshold=80



# Get the current memory and CPU usage for the Airflow scheduler process

memory_usage=$(ps -p $(pgrep -f "airflow scheduler") -o %mem | awk 'NR==2')

cpu_usage=$(ps -p $(pgrep -f "airflow scheduler") -o %cpu | awk 'NR==2')



# Check if the memory or CPU usage has exceeded the threshold

if (( $(echo "$memory_usage > $memory_threshold" | bc -l) )) || (( $(echo "$cpu_usage > $cpu_threshold" | bc -l) )); then

    echo "Resource constraints detected. Memory usage: $memory_usage%, CPU usage: $cpu_usage%"

    # Add any additional commands to diagnose and fix the issue here

else

    echo "No resource constraints detected. Memory usage: $memory_usage%, CPU usage: $cpu_usage%"

fi


```


### Restart the Airflow scheduler to see if it resolves the issue.

```shell


#!/bin/bash



# Stop the Airflow scheduler

sudo systemctl restart airflow-scheduler.service



# Wait for a few seconds to ensure the process has restarted

sleep 5



# Check the status of the Airflow scheduler to ensure it's running

sudo systemctl status airflow-scheduler.service


```


Airflow scheduler outage.

Overview

Parameters

Debug

Check if the Airflow scheduler process is running

Check the scheduler logs for errors

Check if the database connection is working

Check if there are any DAG parsing errors

Check if the scheduler is able to schedule tasks

Check if there are any issues with the worker processes

Resource constraints, such as insufficient memory or processing power for the Airflow scheduler to function properly.

Repair

Restart the Airflow scheduler to see if it resolves the issue.

Learn more

Related Runbooks

Spark executor failure during job execution.

Spark driver program crash during job runtime.

Spark driver failure incident.

Spark YARN Container Failure

Support