---
id: 7a91a70b-2ac0-44c2-815c-c5be87a16ac4
---

# HAProxy Backend response time is above 500ms for host.
---

This incident type occurs when the response time of the backend for a specific host in the HAProxy load balancing software exceeds 500 milliseconds. This can be an indication of a performance issue or bottleneck in the system that requires investigation and resolution to prevent further impact.

### Parameters
```shell
# Environment Variables

export HAPROXY_SERVER="PLACEHOLDER"

export HOST="PLACEHOLDER"

export BACKEND_SERVER_IP_ADDRESS="PLACEHOLDER"

export VALUE="PLACEHOLDER"
```

## Debug

### Check HAProxy status
```shell
systemctl status haproxy
```

### SSH into the server where HAProxy is running
```shell
ssh ${HAPROXY_SERVER}
```

### Check HAProxy logs for errors
```shell
tail -f /var/log/haproxy.log
```

### Check backend response time for a specific host
```shell
echo "show stat" | sudo socat /var/run/haproxy.sock stdio | grep ${HOST} | cut -d ',' -f 35
```

### Check backend response time for all hosts
```shell
echo "show stat" | sudo socat /var/run/haproxy.sock stdio | grep -E '^backend' | cut -d ',' -f 1,35
```
### Check CPU usage
```shell
CPU_USAGE=$(ssh $BACKEND_SERVER "top -b -n1 | grep 'Cpu(s)' | awk '{print $2+$4}'")

if [[ $(echo "$CPU_USAGE > 80" | bc -l) -eq 1 ]]; then

  echo "High CPU usage detected on $BACKEND_SERVER. Please investigate."

fi
```

### Check memory usage
```shell
MEMORY_USAGE=$(ssh $BACKEND_SERVER "free | awk '/Mem/{print $3/$2 * 100.0}'")

if [[ $(echo "$MEMORY_USAGE > 80" | bc -l) -eq 1 ]]; then

  echo "High memory usage detected on $BACKEND_SERVER. Please investigate."

fi
```

### Check network connectivity
```shell
ping -c 3 $BACKEND_SERVER > /dev/null

if [ $? -ne 0 ]; then

  echo "Unable to ping $BACKEND_SERVER. Please investigate network connectivity."

fi
```

## Repair

### Tune the server's operating system parameters to improve performance 
```shell
bash

#!/bin/bash



# Set the sysctl parameters for network performance

sudo sysctl -w net.core.somaxconn=${VALUE}

sudo sysctl -w net.ipv4.tcp_max_syn_backlog=${VALUE}

sudo sysctl -w net.ipv4.tcp_fin_timeout=${VALUE}



# Set the sysctl parameters for kernel performance

sudo sysctl -w kernel.pid_max=${VALUE}

sudo sysctl -w kernel.threads-max=${VALUE}

sudo sysctl -w kernel.sem=${VALUE}



# Set the sysctl parameters for file system performance

sudo sysctl -w fs.file-max=${VALUE}

sudo sysctl -w fs.nr_open=${VALUE}

sudo sysctl -w vm.max_map_count=${VALUE}



# Restart HAProxy to apply the changes

sudo systemctl restart haproxy.service


```


This incident type occurs when the response time of the backend for a specific host in the HAProxy load balancing software exceeds 500 milliseconds. This can be an indication of a performance issue or bottleneck in the system that requires investigation and resolution to prevent further impact.


This incident type occurs when the number of 5xx errors on Traffic Server is higher than usual. This can indicate issues with server performance or connectivity problems. It requires investigation to identify the root cause and resolve the issue.


High 5xx Errors on Traffic Server

This incident type occurs when the size of HTTP request headers or payload exceeds the limits set by Tomcat server. When this happens, the server may reject the request or return an error message, causing disruption to the normal functioning of the web application. This can be caused by a variety of factors, including large file uploads, excessive use of cookies or query parameters, or attacks that intentionally send oversized requests.


Tomcat HTTP Request Headers or Payload Size Exceeds Limits.

This incident type refers to an increase in the number of errors per second on a Tomcat server, which could indicate an issue with the server itself, the host, a deployed application, or an application servlet. This could include errors generated when the Tomcat server runs out of memory, can't find a requested file or servlet, or is unable to serve a JSP due to syntax errors in the servlet codebase. This incident type requires immediate attention to diagnose and address the underlying issue.


Increase of the errors/second rate for Tomcat server

This incident type typically refers to a situation where there is an anomaly in the Tomcat max processing time on a host. The maximum amount of time it takes for the server to process one request, from the time an available thread starts processing the request to the time it returns a response, has exceeded the expected threshold. This could indicate that a JSP page isn’t loading or an associated process (such as a database query) is taking too long to complete.


Anomalous max processing time for Tomcat host

This incident type refers to a situation where the number of 5xx errors on NGINX upstreams is higher than usual. It is an indication that there could be an issue with the NGINX server or the upstream application server. This type of incident requires immediate attention as it can impact the availability and performance of the application.


Higher than usual 5xx Errors for NGINX

```shell
# Environment Variables

export HAPROXY_SERVER="PLACEHOLDER"

export HOST="PLACEHOLDER"

export BACKEND_SERVER_IP_ADDRESS="PLACEHOLDER"

export VALUE="PLACEHOLDER"
```


### Check HAProxy status

```shell
systemctl status haproxy
```

### SSH into the server where HAProxy is running

```shell
ssh ${HAPROXY_SERVER}
```

### Check HAProxy logs for errors

```shell
tail -f /var/log/haproxy.log
```

### Check backend response time for a specific host

```shell
echo "show stat" | sudo socat /var/run/haproxy.sock stdio | grep ${HOST} | cut -d ',' -f 35
```

### Check backend response time for all hosts

```shell
echo "show stat" | sudo socat /var/run/haproxy.sock stdio | grep -E '^backend' | cut -d ',' -f 1,35
```

### Check CPU usage

```shell
CPU_USAGE=$(ssh $BACKEND_SERVER "top -b -n1 | grep 'Cpu(s)' | awk '{print $2+$4}'")

if [[ $(echo "$CPU_USAGE > 80" | bc -l) -eq 1 ]]; then

  echo "High CPU usage detected on $BACKEND_SERVER. Please investigate."

fi
```

### Check memory usage

```shell
MEMORY_USAGE=$(ssh $BACKEND_SERVER "free | awk '/Mem/{print $3/$2 * 100.0}'")

if [[ $(echo "$MEMORY_USAGE > 80" | bc -l) -eq 1 ]]; then

  echo "High memory usage detected on $BACKEND_SERVER. Please investigate."

fi
```

### Check network connectivity

```shell
ping -c 3 $BACKEND_SERVER > /dev/null

if [ $? -ne 0 ]; then

  echo "Unable to ping $BACKEND_SERVER. Please investigate network connectivity."

fi
```


### Tune the server's operating system parameters to improve performance

```shell
bash

#!/bin/bash



# Set the sysctl parameters for network performance

sudo sysctl -w net.core.somaxconn=${VALUE}

sudo sysctl -w net.ipv4.tcp_max_syn_backlog=${VALUE}

sudo sysctl -w net.ipv4.tcp_fin_timeout=${VALUE}



# Set the sysctl parameters for kernel performance

sudo sysctl -w kernel.pid_max=${VALUE}

sudo sysctl -w kernel.threads-max=${VALUE}

sudo sysctl -w kernel.sem=${VALUE}



# Set the sysctl parameters for file system performance

sudo sysctl -w fs.file-max=${VALUE}

sudo sysctl -w fs.nr_open=${VALUE}

sudo sysctl -w vm.max_map_count=${VALUE}



# Restart HAProxy to apply the changes

sudo systemctl restart haproxy.service


```


HAProxy Backend response time is above 500ms for host.

Overview

Parameters

Debug

Check HAProxy status

SSH into the server where HAProxy is running

Check HAProxy logs for errors

Check backend response time for a specific host

Check backend response time for all hosts

Check CPU usage

Check memory usage

Check network connectivity

Repair

Tune the server's operating system parameters to improve performance

Learn more

Related Runbooks

High 5xx Errors on Traffic Server

Tomcat HTTP Request Headers or Payload Size Exceeds Limits.

Increase of the errors/second rate for Tomcat server

Anomalous max processing time for Tomcat host

Support