---
id: 644d0ad8-fad1-416c-a3f0-f5043a1fdcfc
---
# Nginx Upstream peers fails
---

This incident type involves an issue with NGINX upstream peers failing, which has triggered an alert. The details of the alert include information about the percentage of failures and how they compare to predicted values over a certain time period. The incident may be resolved or ongoing, and it may have impacted a specific service or team.

### Parameters
```shell
# Environment Variables

export PATH_TO_NGINX_CONFIG="PLACEHOLDER"

export UPSTREAM_PEER_URL="PLACEHOLDER"

export PATH_TO_NGINX_LOG="PLACEHOLDER"

export COMMAND_TO_CHECK_FOR_ANOMALIES="PLACEHOLDER"

export UPSTREAM_PEERS_URL="PLACEHOLDER"
```

## Debug

### Check NGINX status
```shell
systemctl status nginx
```

### Check current upstream peers configuration
```shell
cat ${PATH_TO_NGINX_CONFIG}/nginx.conf | grep upstream
```

### Check upstream peers health
```shell
curl -I ${UPSTREAM_PEER_URL}
```

### Check the network connections to upstream servers
```shell
netstat -anp | grep ${UPSTREAM_SERVER_IP}
```

### Check NGINX log for errors
```shell
tail -n 50 ${PATH_TO_NGINX_LOG}/error.log
```

### Check CPU and memory usage
```shell
top
```

### Check network usage
```shell
iftop
```

### Check system load
```shell
top -n 1 -b`
```


## Repair

### Optimize the NGINX configuration to ensure optimal performance and reduce the likelihood of upstream peer failures.
```shell
#!/bin/bash

# Backup the original NGINX configuration file

cp ${NGINX_CONFIG_FILE} ${NGINX_CONFIG_FILE}.bak


# Set the worker_connections value to 1024

sed -i 's/worker_connections [0-9]\+/worker_connections 1024/' ${NGINX_CONFIG_FILE}

# Set the worker_processes value to the number of available CPU cores

worker_processes=$(nproc)

sed -i 's/worker_processes [0-9]\+/worker_processes '$worker_processes'/' ${NGINX_CONFIG_FILE}

# Increase the keepalive_timeout value to 60 seconds

sed -i 's/keepalive_timeout [0-9]\+/keepalive_timeout 60/' ${NGINX_CONFIG_FILE}

# Reload NGINX to apply the new configuration

service nginx reload

```


This incident type involves an issue with NGINX upstream peers failing, which has triggered an alert. The details of the alert include information about the percentage of failures and how they compare to predicted values over a certain time period. The incident may be resolved or ongoing, and it may have impacted a specific service or team.


This incident type refers to a situation where Redis, a popular in-memory data structure store, has rejected connections. This could happen due to various reasons such as server overload, network issues, or misconfiguration. When this occurs, it can cause disruptions in the performance of applications that rely on Redis for data storage or caching. Thus, it requires immediate attention and resolution to minimize the impact on the applications and users.


Redis Rejected Connections Incident

This incident type is related to an issue with a Redis instance being down. Redis is an in-memory data structure store used as a database, cache, and message broker. This incident could potentially cause disruption to applications and services that rely on Redis for data storage or caching.


Redis Instance Down

This incident type refers to the failure of a replica node in a PostgreSQL database system that is running on a Linux-based operating system. A replica node is a copy of the primary database node that is used to provide high availability and fault tolerance. When a replica node fails, it can result in data loss, decreased system performance, and potential downtime for users. This type of incident requires immediate attention from a software engineer to diagnose and resolve the issue as quickly as possible.


PostgreSQL Replica Node Failure on Linux.

This incident type refers to a failure in the primary PostgreSQL node on a Linux system. PostgreSQL is a popular open source relational database management system used by many organizations. The primary node is responsible for handling the majority of the traffic and data replication in a PostgreSQL cluster. When the primary node fails, it can result in data loss, slow performance, and potential downtime for users. This type of incident requires immediate attention and resolution to minimize the impact on users and ensure data integrity.


Primary PostgreSQL Node Failure on Linux

High Nginx latency typically refers to a situation where the response time of Nginx servers is significantly slower than expected. This may occur due to a variety of reasons, such as increased traffic or a misconfiguration in the server settings. High Nginx latency can cause website downtime, slow page load times, and poor user experience. It is important to address this incident type promptly to prevent any negative impact on users or business operations.


High Nginx Latency

```shell
# Environment Variables

export PATH_TO_NGINX_CONFIG="PLACEHOLDER"

export UPSTREAM_PEER_URL="PLACEHOLDER"

export PATH_TO_NGINX_LOG="PLACEHOLDER"

export COMMAND_TO_CHECK_FOR_ANOMALIES="PLACEHOLDER"

export UPSTREAM_PEERS_URL="PLACEHOLDER"
```


### Check NGINX status

```shell
systemctl status nginx
```

### Check current upstream peers configuration

```shell
cat ${PATH_TO_NGINX_CONFIG}/nginx.conf | grep upstream
```

### Check upstream peers health

```shell
curl -I ${UPSTREAM_PEER_URL}
```

### Check the network connections to upstream servers

```shell
netstat -anp | grep ${UPSTREAM_SERVER_IP}
```

### Check NGINX log for errors

```shell
tail -n 50 ${PATH_TO_NGINX_LOG}/error.log
```

### Check CPU and memory usage

```shell
top
```

### Check network usage

```shell
iftop
```

### Check system load

```shell
top -n 1 -b`
```


### Optimize the NGINX configuration to ensure optimal performance and reduce the likelihood of upstream peer failures.

```shell
#!/bin/bash

# Backup the original NGINX configuration file

cp ${NGINX_CONFIG_FILE} ${NGINX_CONFIG_FILE}.bak


# Set the worker_connections value to 1024

sed -i 's/worker_connections [0-9]\+/worker_connections 1024/' ${NGINX_CONFIG_FILE}

# Set the worker_processes value to the number of available CPU cores

worker_processes=$(nproc)

sed -i 's/worker_processes [0-9]\+/worker_processes '$worker_processes'/' ${NGINX_CONFIG_FILE}

# Increase the keepalive_timeout value to 60 seconds

sed -i 's/keepalive_timeout [0-9]\+/keepalive_timeout 60/' ${NGINX_CONFIG_FILE}

# Reload NGINX to apply the new configuration

service nginx reload

```


Nginx Upstream peers fails

Overview

Parameters

Debug

Check NGINX status

Check current upstream peers configuration

Check upstream peers health

Check the network connections to upstream servers

Check NGINX log for errors

Check CPU and memory usage

Check network usage

Check system load

Repair

Optimize the NGINX configuration to ensure optimal performance and reduce the likelihood of upstream peer failures.

Learn more

Related Runbooks

Redis Rejected Connections Incident

Redis Instance Down

PostgreSQL Replica Node Failure on Linux.

Primary PostgreSQL Node Failure on Linux

Support