---
id: 163b0ee4-38d8-4321-ab82-e8642a93971c
---

# Lost connection to MySQL server while querying a database incident.
---

This incident type occurs when there is a loss of connection to the MySQL server during the process of querying a database. This can happen due to various reasons such as network issues, server overload, or software errors. As a result, the query execution fails and users may experience disruptions in accessing data from the database. It is critical to resolve this issue promptly to avoid data loss or corruption.

### Parameters
```shell
export APPLICATION_SERVER_IP="PLACEHOLDER"
export MYSQL_PASSWORD="PLACEHOLDER"
export MYSQL_DATABASE_NAME="PLACEHOLDER"
export MYSQL_USERNAME="PLACEHOLDER"
export MYSQL_SERVER_IP_ADDRESS="PLACEHOLDER"
export DB_PORT="PLACEHOLDER"
export VALUE="PLACEHOLDER"
export PREVIOUS_VALUE="PLACEHOLDER"
```

## Debug

### Check if MySQL server is running
```shell
systemctl status mysql
```

### Check if MySQL server is listening on the correct port
```shell
sudo netstat -tulpn | grep 3306
```

### Check MySQL server logs for errors
```shell
sudo tail -f /var/log/mysql/error.log
```

### Check MySQL client logs for errors
```shell
sudo tail -f /var/log/mysql/mysql.log
```

### Check if there are any network issues
```shell
ping ${MYSQL_SERVER_IP_ADDRESS}
```

### Check if there is enough disk space available
```shell
df -h
```

### Check if there are any issues with the database tables
```shell
mysql -u ${MYSQL_USERNAME} -p ${MYSQL_PASSWORD} -e "USE ${MYSQL_DATABASE_NAME}; SHOW TABLES;"
```

### Check if there are any locked tables
```shell
mysql -u ${MYSQL_USERNAME} -p ${MYSQL_PASSWORD} -e "SHOW OPEN TABLES WHERE In_use > 0;"
```

### Check if there are any long-running queries
```shell
mysql -u ${MYSQL_USERNAME} -p ${MYSQL_PASSWORD} -e "SHOW FULL PROCESSLIST;"
```

### Check the network connectivity between the application server and the database server. Make sure there are no firewall rules or network issues blocking the connection.
```shell

#!/bin/bash
APP_SERVER=${APPLICATION_SERVER_IP}
DB_SERVER=${MYSQL_SERVER_IP_ADDRESS}

ping -c 4 $APP_SERVER > /dev/null 2>&1
if [ $? -eq 0 ]; then
  echo "Application server is reachable."
else
  echo "Application server is not reachable."
fi

ping -c 4 $DB_SERVER > /dev/null 2>&1
if [ $? -eq 0 ]; then
  echo "Database server is reachable."
else
  echo "Database server is not reachable."
fi

nc -zv $DB_SERVER 3306 > /dev/null 2>&1
if [ $? -eq 0 ]; then
  echo "Connection to MySQL server is successful."
else
  echo "Connection to MySQL server failed."
fi

```

### The network connection to the MySQL server may have been disrupted or lost.
```shell
#!/bin/bash
# Define variables
MYSQL_HOST=${MYSQL_SERVER_IP_ADDRESS}
MYSQL_USER=${MYSQL_USERNAME}
MYSQL_PASSWORD=${MYSQL_PASSWORD}
MYSQL_DATABASE=${MYSQL_DATABASE_NAME}

# Test MySQL connection
echo "Testing MySQL connection to $MYSQL_HOST..."
if mysql -h $MYSQL_HOST -u $MYSQL_USER -p $MYSQL_PASSWORD -e "use $MYSQL_DATABASE;"; then
    echo "MySQL connection successful."
else
    echo "MySQL connection failed - network connection may have been disrupted or lost."
fi

```

## Repair
### Increase the timeout value for the connection to the MySQL server to allow for longer queries to complete. This can be done in the application code or in the MySQL server configuration.
```shell
#!/bin/bash

# Set the timeout value for the MySQL server connection
timeout_value=${VALUE}

# Update the MySQL server configuration file to include the new timeout value
sudo sed -i "s/timeout=${PREVIOUS_VALUE}/timeout=$timeout_value/g" /etc/mysql/my.cnf

# Restart the MySQL server to apply the new configuration
sudo service mysql restart

echo "Timeout value updated to $timeout_value seconds."
```


This incident type occurs when there is a loss of connection to the MySQL server during the process of querying a database. This can happen due to various reasons such as network issues, server overload, or software errors. As a result, the query execution fails and users may experience disruptions in accessing data from the database. It is critical to resolve this issue promptly to avoid data loss or corruption.


This incident type refers to a situation where the Tomcat server's JDBC connection pool usage surpasses its configured limits. This can lead to a shortage of available connections for incoming requests, resulting in slow response times or even server crashes. It is important to monitor and manage the JDBC connection pool usage to prevent such incidents from occurring.


Tomcat High JDBC Connection Pool Usage Incident.

This incident type refers to a situation where a SQLServer database goes offline, which can potentially cause disruptions in software or applications that rely on it. The incident is triggered by a monitoring system that detects the database's state is not available. A prompt response is necessary to resolve the issue and prevent any adverse impact on the system.


Database Offline (SQLServer)

This incident type refers to a situation where Redis, a popular in-memory data structure store, has rejected connections. This could happen due to various reasons such as server overload, network issues, or misconfiguration. When this occurs, it can cause disruptions in the performance of applications that rely on Redis for data storage or caching. Thus, it requires immediate attention and resolution to minimize the impact on the applications and users.


Redis Rejected Connections Incident

This incident type refers to an issue with Redis where one or more slave instances have become disconnected, resulting in replication failure. This can cause data inconsistencies and may require immediate attention to restore normal functioning. The incident may be caused by a variety of factors, such as network issues, server failures, or misconfiguration.


Redis disconnected slaves incident

This incident type refers to a scenario where there is a high rate of statement timeouts in a Postgresql database instance. This can lead to degraded performance and potentially impact the availability of the database. It is important to quickly identify and address the underlying cause of the timeouts to ensure the stability of the system.


Postgresql high rate statement timeout incident.

```shell
export APPLICATION_SERVER_IP="PLACEHOLDER"
export MYSQL_PASSWORD="PLACEHOLDER"
export MYSQL_DATABASE_NAME="PLACEHOLDER"
export MYSQL_USERNAME="PLACEHOLDER"
export MYSQL_SERVER_IP_ADDRESS="PLACEHOLDER"
export DB_PORT="PLACEHOLDER"
export VALUE="PLACEHOLDER"
export PREVIOUS_VALUE="PLACEHOLDER"
```


### Check if MySQL server is running

```shell
systemctl status mysql
```

### Check if MySQL server is listening on the correct port

```shell
sudo netstat -tulpn | grep 3306
```

### Check MySQL server logs for errors

```shell
sudo tail -f /var/log/mysql/error.log
```

### Check MySQL client logs for errors

```shell
sudo tail -f /var/log/mysql/mysql.log
```

### Check if there are any network issues

```shell
ping ${MYSQL_SERVER_IP_ADDRESS}
```

### Check if there is enough disk space available

```shell
df -h
```

### Check if there are any issues with the database tables

```shell
mysql -u ${MYSQL_USERNAME} -p ${MYSQL_PASSWORD} -e "USE ${MYSQL_DATABASE_NAME}; SHOW TABLES;"
```

### Check if there are any locked tables

```shell
mysql -u ${MYSQL_USERNAME} -p ${MYSQL_PASSWORD} -e "SHOW OPEN TABLES WHERE In_use > 0;"
```

### Check if there are any long-running queries

```shell
mysql -u ${MYSQL_USERNAME} -p ${MYSQL_PASSWORD} -e "SHOW FULL PROCESSLIST;"
```

### Check the network connectivity between the application server and the database server. Make sure there are no firewall rules or network issues blocking the connection.

```shell

#!/bin/bash
APP_SERVER=${APPLICATION_SERVER_IP}
DB_SERVER=${MYSQL_SERVER_IP_ADDRESS}

ping -c 4 $APP_SERVER > /dev/null 2>&1
if [ $? -eq 0 ]; then
  echo "Application server is reachable."
else
  echo "Application server is not reachable."
fi

ping -c 4 $DB_SERVER > /dev/null 2>&1
if [ $? -eq 0 ]; then
  echo "Database server is reachable."
else
  echo "Database server is not reachable."
fi

nc -zv $DB_SERVER 3306 > /dev/null 2>&1
if [ $? -eq 0 ]; then
  echo "Connection to MySQL server is successful."
else
  echo "Connection to MySQL server failed."
fi

```

### The network connection to the MySQL server may have been disrupted or lost.

```shell
#!/bin/bash
# Define variables
MYSQL_HOST=${MYSQL_SERVER_IP_ADDRESS}
MYSQL_USER=${MYSQL_USERNAME}
MYSQL_PASSWORD=${MYSQL_PASSWORD}
MYSQL_DATABASE=${MYSQL_DATABASE_NAME}

# Test MySQL connection
echo "Testing MySQL connection to $MYSQL_HOST..."
if mysql -h $MYSQL_HOST -u $MYSQL_USER -p $MYSQL_PASSWORD -e "use $MYSQL_DATABASE;"; then
    echo "MySQL connection successful."
else
    echo "MySQL connection failed - network connection may have been disrupted or lost."
fi

```


### Increase the timeout value for the connection to the MySQL server to allow for longer queries to complete. This can be done in the application code or in the MySQL server configuration.

```shell
#!/bin/bash

# Set the timeout value for the MySQL server connection
timeout_value=${VALUE}

# Update the MySQL server configuration file to include the new timeout value
sudo sed -i "s/timeout=${PREVIOUS_VALUE}/timeout=$timeout_value/g" /etc/mysql/my.cnf

# Restart the MySQL server to apply the new configuration
sudo service mysql restart

echo "Timeout value updated to $timeout_value seconds."
```


Lost connection to MySQL server while querying a database incident.

Overview

Parameters

Debug

Check if MySQL server is running

Check if MySQL server is listening on the correct port

Check MySQL server logs for errors

Check MySQL client logs for errors

Check if there are any network issues

Check if there is enough disk space available

Check if there are any issues with the database tables

Check if there are any locked tables

Check if there are any long-running queries

Check the network connectivity between the application server and the database server. Make sure there are no firewall rules or network issues blocking the connection.

The network connection to the MySQL server may have been disrupted or lost.

Repair

Increase the timeout value for the connection to the MySQL server to allow for longer queries to complete. This can be done in the application code or in the MySQL server configuration.

Learn more

Related Runbooks

Tomcat High JDBC Connection Pool Usage Incident.

Database Offline (SQLServer)

Redis Rejected Connections Incident

Redis disconnected slaves incident

Support