---
id: 2b175798-7f4b-4989-abe2-187169c49261
---

# Airflow Scheduler Crash on Linux Server
---

This incident type involves the crash of the Apache Airflow scheduler on a Linux server. Apache Airflow is an open-source platform to programmatically author, schedule, and monitor workflows. The scheduler manages the execution of workflows and their tasks. When the scheduler crashes, it can cause workflow failures and disruptions in the overall system. This type of incident requires investigation and troubleshooting to identify the root cause of the crash and implement a solution to prevent it from recurring.

### Parameters
```shell
export AIRFLOW_SCHEDULER="PLACEHOLDER"

export AIRFLOW_SCHEDULER_LOG="PLACEHOLDER"

export SYSLOG="PLACEHOLDER"
```

## Debug

### Check if Apache Airflow scheduler is currently running
```shell
ps aux | grep airflow
```

### Check the status of the Apache Airflow scheduler service
```shell
systemctl status ${AIRFLOW_SCHEDULER}
```

### Check if the Airflow configuration file is valid
```shell
airflow check
```

### Check the scheduler logs for any errors
```shell
tail -n 100 ${AIRFLOW_SCHEDULER_LOG}
```

### Check the system logs for any relevant errors
```shell
tail -n 100 ${SYSLOG}
```

### Check the disk space usage on the server
```shell
df -h
```

### Check the memory usage on the server
```shell
free -m
```

### Check the CPU usage on the server
```shell
top
```

### Check if there are any other processes consuming too much resources
```shell
ps aux | sort -nrk 3,3 | head -n 5
```

### Insufficient server resources (memory, disk space, CPU) causing the scheduler to crash.
```shell
bash

#!/bin/bash



# Check memory usage

free_output=$(free -m)

total_mem=$(echo "$free_output" | awk '/Mem:/ {print $2}')

used_mem=$(echo "$free_output" | awk '/Mem:/ {print $3}')

free_mem=$(echo "$free_output" | awk '/Mem:/ {print $4}')

mem_usage=$(echo "scale=2; $used_mem/$total_mem*100" | bc)

if (( $(echo "$mem_usage > 90" | bc -l) )); then

  echo "Memory usage is high ($mem_usage%)."

fi



# Check disk usage

df_output=$(df -h)

disk_usage=$(echo "$df_output" | awk '/\/$/ {print $5}' | sed 's/%//')

if (( disk_usage > 90 )); then

  echo "Disk usage is high ($disk_usage%)."

fi



# Check CPU usage

top_output=$(top -bn1)

cpu_usage=$(echo "$top_output" | awk '/^%Cpu/ {print $2}')

if (( $(echo "$cpu_usage > 90" | bc -l) )); then

  echo "CPU usage is high ($cpu_usage%)."

fi


```

## Repair

### Restart the Apache airflow scheduler service to see if it resolves the issue.
```shell


#!/bin/bash

sudo systemctl restart ${AIRFLOW_SCHEDULER}


```


This incident type involves the crash of the Apache Airflow scheduler on a Linux server. Apache Airflow is an open-source platform to programmatically author, schedule, and monitor workflows. The scheduler manages the execution of workflows and their tasks. When the scheduler crashes, it can cause workflow failures and disruptions in the overall system. This type of incident requires investigation and troubleshooting to identify the root cause of the crash and implement a solution to prevent it from recurring.


Tomcat Server Crash incident type refers to a situation where the Tomcat Server, an open-source web server and servlet container, becomes unresponsive and stops functioning. This can result in a complete or partial system outage, leading to a disruption in the delivery of web applications and services. The root cause of the crash can be attributed to a variety of factors such as misconfiguration, network or hardware issues, software bugs, or excessive server load. It is critical to address this incident quickly to minimize the impact on users and restore normal system functionality.


Tomcat Server Crash

This incident type refers to an unexpected termination of the Spark driver program during the runtime of a job. The driver program is responsible for coordinating the execution of a Spark job and if it crashes, the entire job is affected. This can result in data loss and downtime, and requires investigation and troubleshooting to identify the root cause of the issue.


Spark driver program crash during job runtime.

Apache Spark driver failure refers to an incident where the driver program in an Apache Spark cluster fails to execute or crashes during runtime. This can happen due to a variety of reasons such as hardware failure, software bugs, resource constraints, or programming errors. As the driver program is responsible for coordinating the execution of tasks across the cluster, any failure in the driver can result in the entire Spark job failing. This can lead to data loss, processing delays, and impact the overall performance of the Spark cluster.


Spark driver failure incident.

Apache Spark Out of Memory Error is a type of incident that occurs when the Apache Spark application running on a system is unable to allocate enough memory to perform its tasks. This can happen when the system's memory is insufficient or when there is a memory leak in the application's code. When this incident occurs, the application may crash, become unresponsive, or produce incorrect results. It is important to diagnose and resolve this issue as soon as possible to ensure the smooth functioning of the application.


Apache Spark Out of Memory Error incident.

This incident occurs when there is a high number of idle workers in the Apache server. Idle workers are connections that have been established by clients but are not currently processing any requests. When the number of idle workers becomes too high, it can lead to performance issues and potentially impact the server's ability to handle incoming requests. This can lead to slow response times or even server crashes.


High Number of Idle Workers in Apache Server

```shell
export AIRFLOW_SCHEDULER="PLACEHOLDER"

export AIRFLOW_SCHEDULER_LOG="PLACEHOLDER"

export SYSLOG="PLACEHOLDER"
```


### Check if Apache Airflow scheduler is currently running

```shell
ps aux | grep airflow
```

### Check the status of the Apache Airflow scheduler service

```shell
systemctl status ${AIRFLOW_SCHEDULER}
```

### Check if the Airflow configuration file is valid

```shell
airflow check
```

### Check the scheduler logs for any errors

```shell
tail -n 100 ${AIRFLOW_SCHEDULER_LOG}
```

### Check the system logs for any relevant errors

```shell
tail -n 100 ${SYSLOG}
```

### Check the disk space usage on the server

```shell
df -h
```

### Check the memory usage on the server

```shell
free -m
```

### Check the CPU usage on the server

```shell
top
```

### Check if there are any other processes consuming too much resources

```shell
ps aux | sort -nrk 3,3 | head -n 5
```

### Insufficient server resources (memory, disk space, CPU) causing the scheduler to crash.

```shell
bash

#!/bin/bash



# Check memory usage

free_output=$(free -m)

total_mem=$(echo "$free_output" | awk '/Mem:/ {print $2}')

used_mem=$(echo "$free_output" | awk '/Mem:/ {print $3}')

free_mem=$(echo "$free_output" | awk '/Mem:/ {print $4}')

mem_usage=$(echo "scale=2; $used_mem/$total_mem*100" | bc)

if (( $(echo "$mem_usage > 90" | bc -l) )); then

  echo "Memory usage is high ($mem_usage%)."

fi



# Check disk usage

df_output=$(df -h)

disk_usage=$(echo "$df_output" | awk '/\/$/ {print $5}' | sed 's/%//')

if (( disk_usage > 90 )); then

  echo "Disk usage is high ($disk_usage%)."

fi



# Check CPU usage

top_output=$(top -bn1)

cpu_usage=$(echo "$top_output" | awk '/^%Cpu/ {print $2}')

if (( $(echo "$cpu_usage > 90" | bc -l) )); then

  echo "CPU usage is high ($cpu_usage%)."

fi


```


### Restart the Apache airflow scheduler service to see if it resolves the issue.

```shell


#!/bin/bash

sudo systemctl restart ${AIRFLOW_SCHEDULER}


```


Airflow Scheduler Crash on Linux Server

Overview

Parameters

Debug

Check if Apache Airflow scheduler is currently running

Check the status of the Apache Airflow scheduler service

Check if the Airflow configuration file is valid

Check the scheduler logs for any errors

Check the system logs for any relevant errors

Check the disk space usage on the server

Check the memory usage on the server

Check the CPU usage on the server

Check if there are any other processes consuming too much resources

Insufficient server resources (memory, disk space, CPU) causing the scheduler to crash.

Repair

Restart the Apache airflow scheduler service to see if it resolves the issue.

Learn more

Related Runbooks

Tomcat Server Crash

Spark driver program crash during job runtime.

Spark driver failure incident.

Apache Spark Out of Memory Error incident.

Support