---
id: 70db55b3-083d-45d0-87b0-62b71b5716d5
---
# Higher than usual 5xx Errors for NGINX 
---

This incident type refers to a situation where the number of 5xx errors on NGINX upstreams is higher than usual. It is an indication that there could be an issue with the NGINX server or the upstream application server. This type of incident requires immediate attention as it can impact the availability and performance of the application.

### Parameters
```shell
# Environment Variables

export UPSTREAM_SERVER_URL="PLACEHOLDER"

export SERVER_IP="PLACEHOLDER"

export SERVICE_NAME="PLACEHOLDER"

export UPSTREAM_URL="PLACEHOLDER"

export NUM_WORKERS="PLACEHOLDER"
```

## Debug

### 1. Check server status
```shell
systemctl status nginx
```

### 2. Check if the server is running
```shell
ps aux | grep nginx
```

### 3. Check the error log for any relevant information
```shell
tail -f /var/log/nginx/error.log
```

### 4. Check the access log to see if there are any suspiciously high traffic spikes
```shell
tail -f /var/log/nginx/access.log
```

### 5. Check the server configuration for any errors
```shell
nginx -t
```

### 6. Check if there are any issues with upstream servers
```shell
curl -I ${UPSTREAM_SERVER_URL}
```

### 7. Check the network connectivity to the server
```shell
ping ${SERVER_IP}
```

### 8. Check if there are any firewall rules blocking traffic
```shell
iptables -L
```

### 9. Check if there are any other services running on the same host that could be interfering with NGINX
```shell
ps aux | grep ${SERVICE_NAME}
```

### 10. Check if there are any resource constraints on the server
```shell
top
```

### 11. Check if there are any other system-level issues that may be impacting NGINX
```shell
dmesg
```
## Check if the upstream server is responding with a HTTP 200 status code
```shell
if curl -s -o /dev/null -w "%{http_code}" $UPSTREAM_URL | grep -q "200"; then

  echo "Upstream server is responding with HTTP 200"

else

  echo "Upstream server is not responding with HTTP 200"

fi
```
## Repair

### Increase the number of NGINX worker processes to handle the increased traffic and prevent the server from getting overloaded.
```shell

#!/bin/bash

# Set the number of NGINX worker processes to ${NUM_WORKERS}

NUM_WORKERS=${NUM_WORKERS}

sed -i "s/worker_processes [0-9]\+;/worker_processes $NUM_WORKERS;/" /etc/nginx/nginx.conf

# Reload the NGINX configuration

nginx -t && service nginx reload

echo "NGINX worker processes increased to $NUM_WORKERS"


```


This incident type refers to a situation where the number of 5xx errors on NGINX upstreams is higher than usual. It is an indication that there could be an issue with the NGINX server or the upstream application server. This type of incident requires immediate attention as it can impact the availability and performance of the application.


This incident type occurs when the number of 5xx errors on Traffic Server is higher than usual. This can indicate issues with server performance or connectivity problems. It requires investigation to identify the root cause and resolve the issue.


High 5xx Errors on Traffic Server

This incident type occurs when the number of 4xx errors on Traffic Server is at an anomalous level, higher than usual. It could be an indicator of an issue with the server or an increase in traffic that is causing errors. It requires investigation and resolution to ensure the server is functioning correctly and not impacting users.


High 4xx Errors on Traffic Server

This incident type occurs when the size of HTTP request headers or payload exceeds the limits set by Tomcat server. When this happens, the server may reject the request or return an error message, causing disruption to the normal functioning of the web application. This can be caused by a variety of factors, including large file uploads, excessive use of cookies or query parameters, or attacks that intentionally send oversized requests.


Tomcat HTTP Request Headers or Payload Size Exceeds Limits.

This incident type refers to an increase in the number of errors per second on a Tomcat server, which could indicate an issue with the server itself, the host, a deployed application, or an application servlet. This could include errors generated when the Tomcat server runs out of memory, can't find a requested file or servlet, or is unable to serve a JSP due to syntax errors in the servlet codebase. This incident type requires immediate attention to diagnose and address the underlying issue.


Increase of the errors/second rate for Tomcat server

This incident type refers to a scenario where there is a high rate of statement timeouts in a Postgresql database instance. This can lead to degraded performance and potentially impact the availability of the database. It is important to quickly identify and address the underlying cause of the timeouts to ensure the stability of the system.


Postgresql high rate statement timeout incident.

```shell
# Environment Variables

export UPSTREAM_SERVER_URL="PLACEHOLDER"

export SERVER_IP="PLACEHOLDER"

export SERVICE_NAME="PLACEHOLDER"

export UPSTREAM_URL="PLACEHOLDER"

export NUM_WORKERS="PLACEHOLDER"
```


### 1. Check server status

```shell
systemctl status nginx
```

### 2. Check if the server is running

```shell
ps aux | grep nginx
```

### 3. Check the error log for any relevant information

```shell
tail -f /var/log/nginx/error.log
```

### 4. Check the access log to see if there are any suspiciously high traffic spikes

```shell
tail -f /var/log/nginx/access.log
```

### 5. Check the server configuration for any errors

```shell
nginx -t
```

### 6. Check if there are any issues with upstream servers

```shell
curl -I ${UPSTREAM_SERVER_URL}
```

### 7. Check the network connectivity to the server

```shell
ping ${SERVER_IP}
```

### 8. Check if there are any firewall rules blocking traffic

```shell
iptables -L
```

### 9. Check if there are any other services running on the same host that could be interfering with NGINX

```shell
ps aux | grep ${SERVICE_NAME}
```

### 10. Check if there are any resource constraints on the server

```shell
top
```

### 11. Check if there are any other system-level issues that may be impacting NGINX

```shell
dmesg
```

## Check if the upstream server is responding with a HTTP 200 status code

```shell
if curl -s -o /dev/null -w "%{http_code}" $UPSTREAM_URL | grep -q "200"; then

  echo "Upstream server is responding with HTTP 200"

else

  echo "Upstream server is not responding with HTTP 200"

fi
```


### Increase the number of NGINX worker processes to handle the increased traffic and prevent the server from getting overloaded.

```shell

#!/bin/bash

# Set the number of NGINX worker processes to ${NUM_WORKERS}

NUM_WORKERS=${NUM_WORKERS}

sed -i "s/worker_processes [0-9]\+;/worker_processes $NUM_WORKERS;/" /etc/nginx/nginx.conf

# Reload the NGINX configuration

nginx -t && service nginx reload

echo "NGINX worker processes increased to $NUM_WORKERS"


```


Higher than usual 5xx Errors for NGINX

Overview

Parameters

Debug

1. Check server status

2. Check if the server is running

3. Check the error log for any relevant information

4. Check the access log to see if there are any suspiciously high traffic spikes

5. Check the server configuration for any errors

6. Check if there are any issues with upstream servers

7. Check the network connectivity to the server

8. Check if there are any firewall rules blocking traffic

9. Check if there are any other services running on the same host that could be interfering with NGINX

10. Check if there are any resource constraints on the server

11. Check if there are any other system-level issues that may be impacting NGINX

Check if the upstream server is responding with a HTTP 200 status code

Repair

Increase the number of NGINX worker processes to handle the increased traffic and prevent the server from getting overloaded.

Learn more

Related Runbooks

High 5xx Errors on Traffic Server

High 4xx Errors on Traffic Server

Tomcat HTTP Request Headers or Payload Size Exceeds Limits.

Increase of the errors/second rate for Tomcat server

Support