---
id: 418d9dd0-cce2-484d-9fec-428a2da66bc0
---

# Database connection errors in Airflow tasks.
---

This incident type refers to situations where Airflow tasks are unable to connect to the database due to errors. These errors may result from a variety of causes such as network issues, database configuration issues, or system overload. When such errors occur, the affected tasks may fail to complete successfully, leading to disruptions in the workflow.

### Parameters
```shell
export DATABASE_SERVICE_NAME="PLACEHOLDER"

export AIRFLOW_CONFIG_FILE="PLACEHOLDER"

export DATABASE_PORT="PLACEHOLDER"

export DATABASE_HOST="PLACEHOLDER"

export DATABASE_USER="PLACEHOLDER"

export AIRFLOW_TASK_FILE="PLACEHOLDER"
```

## Debug

### Check if the database is running
```shell
systemctl status ${DATABASE_SERVICE_NAME}
```

### Check if the database credentials are correct
```shell
cat ${AIRFLOW_CONFIG_FILE} | grep -E "sql_alchemy_conn"
```

### Check if the database is accepting connections
```shell
nc -zv ${DATABASE_HOST} ${DATABASE_PORT}
```

### Check if the database user has sufficient privileges
```shell
psql -h ${DATABASE_HOST} -p ${DATABASE_PORT} -U ${DATABASE_USER} -c "SELECT 1"
```

### Check if the Airflow task is using the correct database connection
```shell
cat ${AIRFLOW_TASK_FILE} | grep -E "sql_alchemy_conn"
```

### Concurrent database connections limit exceeded.
```shell
bash

#!/bin/bash



# Get the maximum number of concurrent database connections allowed

max_connections=$(sudo su - postgres -c "psql -t -c 'SHOW max_connections;'" | tr -d '[:space:]')

echo "Maximum number of concurrent database connections allowed: $max_connections"



# Get the current number of connections to the database

current_connections=$(sudo su - postgres -c "psql -t -c 'SELECT count(*) FROM pg_stat_activity;'" | tr -d '[:space:]')

echo "Current number of connections to the database: $current_connections"



# Calculate the percentage of connections in use

percent_in_use=$(awk "BEGIN {printf \"%.1f\", ($current_connections / $max_connections) * 100}")

echo "Percentage of connections in use: $percent_in_use%"



# Check if the percentage of connections in use exceeds a certain threshold (e.g. 80%)

threshold=80

if (( $(echo "$percent_in_use >= $threshold" | bc -l) )); then

    echo "Concurrent database connections limit exceeded."

else

    echo "Concurrent database connections limit not exceeded."

fi


```

## Repair

### Check if the database server is up and running. If the server is down, bring it back online as soon as possible.
```shell


#!/bin/bash



# Check if the database server is up and running

if ! systemctl status ${DATABASE_SERVICE_NAME} | grep -q "active (running)"; then

    # Bring the database server back online

    systemctl start ${DATABASE_SERVICE_NAME}

    echo "Database server has been restarted."

fi


```

This incident type refers to situations where Airflow tasks are unable to connect to the database due to errors. These errors may result from a variety of causes such as network issues, database configuration issues, or system overload. When such errors occur, the affected tasks may fail to complete successfully, leading to disruptions in the workflow.


This incident type refers to a situation where the Tomcat server's JDBC connection pool usage surpasses its configured limits. This can lead to a shortage of available connections for incoming requests, resulting in slow response times or even server crashes. It is important to monitor and manage the JDBC connection pool usage to prevent such incidents from occurring.


Tomcat High JDBC Connection Pool Usage Incident.

This incident type refers to a failure in one or more Spark executors during the execution of a job. Spark executors are worker processes that run computations and store data in memory or on disk. When an executor fails, it can cause the entire job to fail or result in degraded performance. This type of incident can occur for a variety of reasons, such as hardware or network issues, memory errors, or software bugs.


Spark executor failure during job execution.

Apache Spark driver failure refers to an incident where the driver program in an Apache Spark cluster fails to execute or crashes during runtime. This can happen due to a variety of reasons such as hardware failure, software bugs, resource constraints, or programming errors. As the driver program is responsible for coordinating the execution of tasks across the cluster, any failure in the driver can result in the entire Spark job failing. This can lead to data loss, processing delays, and impact the overall performance of the Spark cluster.


Spark driver failure incident.

This incident type refers to a situation where a SQLServer database goes offline, which can potentially cause disruptions in software or applications that rely on it. The incident is triggered by a monitoring system that detects the database's state is not available. A prompt response is necessary to resolve the issue and prevent any adverse impact on the system.


Database Offline (SQLServer)

This incident type refers to a scenario where there is a high rate of statement timeouts in a Postgresql database instance. This can lead to degraded performance and potentially impact the availability of the database. It is important to quickly identify and address the underlying cause of the timeouts to ensure the stability of the system.


Postgresql high rate statement timeout incident.

```shell
export DATABASE_SERVICE_NAME="PLACEHOLDER"

export AIRFLOW_CONFIG_FILE="PLACEHOLDER"

export DATABASE_PORT="PLACEHOLDER"

export DATABASE_HOST="PLACEHOLDER"

export DATABASE_USER="PLACEHOLDER"

export AIRFLOW_TASK_FILE="PLACEHOLDER"
```


### Check if the database is running

```shell
systemctl status ${DATABASE_SERVICE_NAME}
```

### Check if the database credentials are correct

```shell
cat ${AIRFLOW_CONFIG_FILE} | grep -E "sql_alchemy_conn"
```

### Check if the database is accepting connections

```shell
nc -zv ${DATABASE_HOST} ${DATABASE_PORT}
```

### Check if the database user has sufficient privileges

```shell
psql -h ${DATABASE_HOST} -p ${DATABASE_PORT} -U ${DATABASE_USER} -c "SELECT 1"
```

### Check if the Airflow task is using the correct database connection

```shell
cat ${AIRFLOW_TASK_FILE} | grep -E "sql_alchemy_conn"
```

### Concurrent database connections limit exceeded.

```shell
bash

#!/bin/bash



# Get the maximum number of concurrent database connections allowed

max_connections=$(sudo su - postgres -c "psql -t -c 'SHOW max_connections;'" | tr -d '[:space:]')

echo "Maximum number of concurrent database connections allowed: $max_connections"



# Get the current number of connections to the database

current_connections=$(sudo su - postgres -c "psql -t -c 'SELECT count(*) FROM pg_stat_activity;'" | tr -d '[:space:]')

echo "Current number of connections to the database: $current_connections"



# Calculate the percentage of connections in use

percent_in_use=$(awk "BEGIN {printf \"%.1f\", ($current_connections / $max_connections) * 100}")

echo "Percentage of connections in use: $percent_in_use%"



# Check if the percentage of connections in use exceeds a certain threshold (e.g. 80%)

threshold=80

if (( $(echo "$percent_in_use >= $threshold" | bc -l) )); then

    echo "Concurrent database connections limit exceeded."

else

    echo "Concurrent database connections limit not exceeded."

fi


```


### Check if the database server is up and running. If the server is down, bring it back online as soon as possible.

```shell


#!/bin/bash



# Check if the database server is up and running

if ! systemctl status ${DATABASE_SERVICE_NAME} | grep -q "active (running)"; then

    # Bring the database server back online

    systemctl start ${DATABASE_SERVICE_NAME}

    echo "Database server has been restarted."

fi


```


Database connection errors in Airflow tasks.

Overview

Parameters

Debug

Check if the database is running

Check if the database credentials are correct

Check if the database is accepting connections

Check if the database user has sufficient privileges

Check if the Airflow task is using the correct database connection

Concurrent database connections limit exceeded.

Repair

Check if the database server is up and running. If the server is down, bring it back online as soon as possible.

Learn more

Related Runbooks

Tomcat High JDBC Connection Pool Usage Incident.

Spark executor failure during job execution.

Spark driver failure incident.

Database Offline (SQLServer)

Support