---
id: 80132be0-90ad-48db-bc5b-20bc280a892b
---

# High backend session usage in HAProxy
---

High backend session usage in HAProxy is an incident that occurs when the amount of backend sessions in use for a host reaches a detection threshold of 80%. When this happens, HAProxy denies additional clients until resource consumption drops, which could result in service disruptions. This incident may require modifying HAProxy's configuration to allow more sessions or migrating the HAProxy server to a bigger box.

### Parameters
```shell
# Environment Variables

export HAPROXY_PROCESS_ID="PLACEHOLDER"

export PATH_TO_HAPROXY_CONFIG_FILE="PLACEHOLDER"

export NEW_VALUE_FOR_MAXCONN_PARAMETER="PLACEHOLDER"

export NAME_OF_THE_BACKEND="PLACEHOLDER"

export PATH_TO_HAPROXY_LOG_FILE="PLACEHOLDER"

export PATH_TO_HAPROXY_CFG="PLACEHOLDER"
```

## Debug

### Check the status of HAProxy
```shell
systemctl status haproxy
```

### Check the current backend session usage for a specific host
```shell
echo "show stat" | socat /run/haproxy/admin.sock stdio | awk -F',' '{print $1,$2,$5}'
```

### Check the HAProxy configuration file
```shell
less /etc/haproxy/haproxy.cfg
```

### Check the current resource consumption of the HAProxy server
```shell
top
```

### Check the current network connections
```shell
netstat -anp | grep ${HAPROXY_PROCESS_ID}
```

### Multiple misbehaving clients that are not releasing backend sessions after use, causing a backlog of sessions.
```shell
bash

#!/bin/bash



# Set the necessary parameters

haproxy_config=${PATH_TO_HAPROXY_CFG}

backend_name=${NAME_OF_THE_BACKEND}

log_file=${PATH_TO_HAPROXY_LOG_FILE}



# Check the backend limit and current usage

backend_limit=$(grep -oP "$backend_name\s+maxconn\s+\K\d+" $haproxy_config)

backend_current=$(tail -n 1000 $log_file | grep "$backend_name" | grep "backend_http-in" | grep -c "queued")



if [ "$backend_current" -gt "$backend_limit" ]; then

    echo "The backend session usage is currently at $backend_current, which is greater than the limit of $backend_limit."

    echo "Please investigate the misbehaving clients and determine why they are not releasing backend sessions after use."

else

    echo "The backend session usage is currently at $backend_current, which is within the limit of $backend_limit."

fi


```

## Repair

### Set the path to the HAProxy configuration file
```shell
CONFIG_FILE=${PATH_TO_HAPROXY_CONFIG_FILE}
```

### Set the new value for maxconn parameter
```shell
NEW_MAXCONN=${NEW_VALUE_FOR_MAXCONN_PARAMETER}
```

### Replace the old maxconn parameter with the new value in the configuration file
```shell
sed -i "s/^maxconn.*$/maxconn $NEW_MAXCONN/" $CONFIG_FILE
```

### Restart the HAProxy service to apply the changes
```shell
systemctl restart haproxy.service
```

### Check if the service is running
```shell
if [ "$(systemctl is-active haproxy.service)" = "active" ]; then

    echo "HAProxy service restarted successfully"

else

    echo "Failed to restart HAProxy service"

fi
```


High backend session usage in HAProxy is an incident that occurs when the amount of backend sessions in use for a host reaches a detection threshold of 80%. When this happens, HAProxy denies additional clients until resource consumption drops, which could result in service disruptions. This incident may require modifying HAProxy's configuration to allow more sessions or migrating the HAProxy server to a bigger box.


This incident type occurs when the number of 5xx errors on Traffic Server is higher than usual. This can indicate issues with server performance or connectivity problems. It requires investigation to identify the root cause and resolve the issue.


High 5xx Errors on Traffic Server

This incident occurs when the Tomcat server is experiencing an unusually high number of active sessions. Active sessions refer to the number of users currently accessing a web application hosted on the Tomcat server. When the number of active sessions exceeds the server's capacity, it can lead to slow response times, unresponsive applications, or even server crashes. This incident requires immediate attention to resolve the issue and prevent any further impact on the web application's performance.


Tomcat High Number of Active Sessions.

This incident type refers to a situation where the number of 5xx errors on NGINX upstreams is higher than usual. It is an indication that there could be an issue with the NGINX server or the upstream application server. This type of incident requires immediate attention as it can impact the availability and performance of the application.


Higher than usual 5xx Errors for NGINX

The "High error rate on NGINX" incident type refers to a situation where the error rate on the NGINX server is above 1% for the last 5 minutes. This can result in degraded performance or downtime of the affected service, impacting user experience and potentially leading to lost revenue. The incident requires immediate attention and resolution to minimize the impact on users and prevent further damage.


High error rate on NGINX incident

A Host CPU high iowait incident occurs when the CPU iowait is greater than 10%, indicating that the system is disk or network bound. This can cause performance issues and may require investigation to identify the root cause and resolve the issue.


Host CPU high iowait incident.

```shell
# Environment Variables

export HAPROXY_PROCESS_ID="PLACEHOLDER"

export PATH_TO_HAPROXY_CONFIG_FILE="PLACEHOLDER"

export NEW_VALUE_FOR_MAXCONN_PARAMETER="PLACEHOLDER"

export NAME_OF_THE_BACKEND="PLACEHOLDER"

export PATH_TO_HAPROXY_LOG_FILE="PLACEHOLDER"

export PATH_TO_HAPROXY_CFG="PLACEHOLDER"
```


### Check the status of HAProxy

```shell
systemctl status haproxy
```

### Check the current backend session usage for a specific host

```shell
echo "show stat" | socat /run/haproxy/admin.sock stdio | awk -F',' '{print $1,$2,$5}'
```

### Check the HAProxy configuration file

```shell
less /etc/haproxy/haproxy.cfg
```

### Check the current resource consumption of the HAProxy server

```shell
top
```

### Check the current network connections

```shell
netstat -anp | grep ${HAPROXY_PROCESS_ID}
```

### Multiple misbehaving clients that are not releasing backend sessions after use, causing a backlog of sessions.

```shell
bash

#!/bin/bash



# Set the necessary parameters

haproxy_config=${PATH_TO_HAPROXY_CFG}

backend_name=${NAME_OF_THE_BACKEND}

log_file=${PATH_TO_HAPROXY_LOG_FILE}



# Check the backend limit and current usage

backend_limit=$(grep -oP "$backend_name\s+maxconn\s+\K\d+" $haproxy_config)

backend_current=$(tail -n 1000 $log_file | grep "$backend_name" | grep "backend_http-in" | grep -c "queued")



if [ "$backend_current" -gt "$backend_limit" ]; then

    echo "The backend session usage is currently at $backend_current, which is greater than the limit of $backend_limit."

    echo "Please investigate the misbehaving clients and determine why they are not releasing backend sessions after use."

else

    echo "The backend session usage is currently at $backend_current, which is within the limit of $backend_limit."

fi


```


### Set the path to the HAProxy configuration file

```shell
CONFIG_FILE=${PATH_TO_HAPROXY_CONFIG_FILE}
```

### Set the new value for maxconn parameter

```shell
NEW_MAXCONN=${NEW_VALUE_FOR_MAXCONN_PARAMETER}
```

### Replace the old maxconn parameter with the new value in the configuration file

```shell
sed -i "s/^maxconn.*$/maxconn $NEW_MAXCONN/" $CONFIG_FILE
```

### Restart the HAProxy service to apply the changes

```shell
systemctl restart haproxy.service
```

### Check if the service is running

```shell
if [ "$(systemctl is-active haproxy.service)" = "active" ]; then

    echo "HAProxy service restarted successfully"

else

    echo "Failed to restart HAProxy service"

fi
```


High backend session usage in HAProxy

Overview

Parameters

Debug

Check the status of HAProxy

Check the current backend session usage for a specific host

Check the HAProxy configuration file

Check the current resource consumption of the HAProxy server

Check the current network connections

Multiple misbehaving clients that are not releasing backend sessions after use, causing a backlog of sessions.

Repair

Set the path to the HAProxy configuration file

Set the new value for maxconn parameter

Replace the old maxconn parameter with the new value in the configuration file

Restart the HAProxy service to apply the changes

Check if the service is running

Learn more

Related Runbooks

High 5xx Errors on Traffic Server

Tomcat High Number of Active Sessions.

Higher than usual 5xx Errors for NGINX

High error rate on NGINX incident

Support