---
id: 02b01dd1-a1ab-4b61-9774-b9b46031b04d
---

# Host conntrack limit incident
---

A host conntrack limit incident occurs when the number of conntrack is approaching the limit of a particular host. This can cause network connectivity issues and can potentially affect the performance of the host. The incident may need to be investigated and resolved by a software engineer or network administrator.

### Parameters
```shell
export IP_ADDRESS="PLACEHOLDER"

export PORT_NUMBER="PLACEHOLDER"

export PROTOCOL_NAME="PLACEHOLDER"

export INTERFACE="PLACEHOLDER"

export FILE_NAME="PLACEHOLDER"

export CONNTRACK_THRESHOLD="PLACEHOLDER"

export NEW_LIMIT="PLACEHOLDER"
```

## Debug

### Check the current conntrack limit
```shell
sudo sysctl net.netfilter.nf_conntrack_max
```

### Check the current number of conntracked connections
```shell
sudo cat /proc/sys/net/netfilter/nf_conntrack_count
```

### Check the current number of connections for a specific IP address
```shell
sudo grep ${IP_ADDRESS} /proc/net/nf_conntrack
```

### Check the current number of connections for a specific port
```shell
sudo grep ${PORT_NUMBER} /proc/net/nf_conntrack | wc -l
```

### Check the current number of connections for a specific protocol
```shell
sudo cat /proc/net/nf_conntrack | grep ${PROTOCOL_NAME} | wc -l
```

### Check the list of established connections for a specific IP address
```shell
sudo netstat -nputw | grep ${IP_ADDRESS} | grep ESTABLISHED
```

### Check the list of established connections for a specific port
```shell
sudo netstat -nputw | grep ${PORT_NUMBER} | grep ESTABLISHED
```

### Check the list of listening ports
```shell
sudo netstat -nputw | grep LISTEN
```

### Check the network traffic using tcpdump
```shell
sudo tcpdump -i ${INTERFACE} -nn -s0 -vvv -w ${FILE_NAME}.pcap
```

### Increase in traffic causing the conntrack limit to be reached on the host.
```shell
bash

#!/bin/bash



# Set the threshold for the conntrack limit

CONNTRACK_LIMIT=${CONNTRACK_THRESHOLD}



# Get the current number of conntrack entries

CURRENT_CONNTRACK=$(cat /proc/sys/net/netfilter/nf_conntrack_count)



# Check if the current number of conntrack entries is approaching the limit

if [ $CURRENT_CONNTRACK -gt $((CONNTRACK_LIMIT*0.8)) ]

then

  echo "WARNING: Number of conntrack entries is approaching the limit. Current count: $CURRENT_CONNTRACK"

else

  echo "Conntrack count is within limit. Current count: $CURRENT_CONNTRACK"

fi


```

## Repair

### Increase the conntrack limit on the affected host.
```shell


#!/bin/bash



# Set the new conntrack limit

NEW_LIMIT=${NEW_LIMIT}



# Get the current conntrack limit

CURRENT_LIMIT=$(sysctl net.netfilter.nf_conntrack_max | awk '{print $3}')



# Check if the new limit is greater than the current limit

if [[ $NEW_LIMIT -gt $CURRENT_LIMIT ]]; then

  # Set the new conntrack limit

  sysctl net.netfilter.nf_conntrack_max=$NEW_LIMIT



  # Update the sysctl configuration file to make the change permanent

  echo "net.netfilter.nf_conntrack_max=$NEW_LIMIT" >> /etc/sysctl.conf



  # Reload the sysctl configuration

  sysctl -p

fi


```

A host conntrack limit incident occurs when the number of conntrack is approaching the limit of a particular host. This can cause network connectivity issues and can potentially affect the performance of the host. The incident may need to be investigated and resolved by a software engineer or network administrator.


This incident type occurs when the size of HTTP request headers or payload exceeds the limits set by Tomcat server. When this happens, the server may reject the request or return an error message, causing disruption to the normal functioning of the web application. This can be caused by a variety of factors, including large file uploads, excessive use of cookies or query parameters, or attacks that intentionally send oversized requests.


Tomcat HTTP Request Headers or Payload Size Exceeds Limits.

This incident type refers to an increase in the number of errors per second on a Tomcat server, which could indicate an issue with the server itself, the host, a deployed application, or an application servlet. This could include errors generated when the Tomcat server runs out of memory, can't find a requested file or servlet, or is unable to serve a JSP due to syntax errors in the servlet codebase. This incident type requires immediate attention to diagnose and address the underlying issue.


Increase of the errors/second rate for Tomcat server

This incident type occurs when the maximum number of connections allowed to a Postgres database has been reached. This prevents new connections from being established and can result in errors or downtime. It is important to monitor connection usage and adjust the maximum connection limit as needed to prevent this issue from occurring.


Postgres Connection Limit Reached

This incident type refers to a situation where the MySQL database is experiencing a high level of threads running, which could indicate a performance issue. This may cause issues with the application that relies on the database, such as slow response times or even downtime. Resolving this incident typically involves investigating the root cause of the high thread count and optimizing the database configuration or query performance to reduce the load on the database.


MySQL high threads running incident.

If there are too many processes running on hosts, it could lead to performance issues and resource constraints. You may want to review and prioritize the running processes, terminating unnecessary ones, or optimizing resource allocation to improve system performance. Monitoring tools can help identify resource-intensive processes and their impact on the system.


Too many processes running on the host

```shell
export IP_ADDRESS="PLACEHOLDER"

export PORT_NUMBER="PLACEHOLDER"

export PROTOCOL_NAME="PLACEHOLDER"

export INTERFACE="PLACEHOLDER"

export FILE_NAME="PLACEHOLDER"

export CONNTRACK_THRESHOLD="PLACEHOLDER"

export NEW_LIMIT="PLACEHOLDER"
```


### Check the current conntrack limit

```shell
sudo sysctl net.netfilter.nf_conntrack_max
```

### Check the current number of conntracked connections

```shell
sudo cat /proc/sys/net/netfilter/nf_conntrack_count
```

### Check the current number of connections for a specific IP address

```shell
sudo grep ${IP_ADDRESS} /proc/net/nf_conntrack
```

### Check the current number of connections for a specific port

```shell
sudo grep ${PORT_NUMBER} /proc/net/nf_conntrack | wc -l
```

### Check the current number of connections for a specific protocol

```shell
sudo cat /proc/net/nf_conntrack | grep ${PROTOCOL_NAME} | wc -l
```

### Check the list of established connections for a specific IP address

```shell
sudo netstat -nputw | grep ${IP_ADDRESS} | grep ESTABLISHED
```

### Check the list of established connections for a specific port

```shell
sudo netstat -nputw | grep ${PORT_NUMBER} | grep ESTABLISHED
```

### Check the list of listening ports

```shell
sudo netstat -nputw | grep LISTEN
```

### Check the network traffic using tcpdump

```shell
sudo tcpdump -i ${INTERFACE} -nn -s0 -vvv -w ${FILE_NAME}.pcap
```

### Increase in traffic causing the conntrack limit to be reached on the host.

```shell
bash

#!/bin/bash



# Set the threshold for the conntrack limit

CONNTRACK_LIMIT=${CONNTRACK_THRESHOLD}



# Get the current number of conntrack entries

CURRENT_CONNTRACK=$(cat /proc/sys/net/netfilter/nf_conntrack_count)



# Check if the current number of conntrack entries is approaching the limit

if [ $CURRENT_CONNTRACK -gt $((CONNTRACK_LIMIT*0.8)) ]

then

  echo "WARNING: Number of conntrack entries is approaching the limit. Current count: $CURRENT_CONNTRACK"

else

  echo "Conntrack count is within limit. Current count: $CURRENT_CONNTRACK"

fi


```


### Increase the conntrack limit on the affected host.

```shell


#!/bin/bash



# Set the new conntrack limit

NEW_LIMIT=${NEW_LIMIT}



# Get the current conntrack limit

CURRENT_LIMIT=$(sysctl net.netfilter.nf_conntrack_max | awk '{print $3}')



# Check if the new limit is greater than the current limit

if [[ $NEW_LIMIT -gt $CURRENT_LIMIT ]]; then

  # Set the new conntrack limit

  sysctl net.netfilter.nf_conntrack_max=$NEW_LIMIT



  # Update the sysctl configuration file to make the change permanent

  echo "net.netfilter.nf_conntrack_max=$NEW_LIMIT" >> /etc/sysctl.conf



  # Reload the sysctl configuration

  sysctl -p

fi


```


Host conntrack limit incident

Overview

Parameters

Debug

Check the current conntrack limit

Check the current number of conntracked connections

Check the current number of connections for a specific IP address

Check the current number of connections for a specific port

Check the current number of connections for a specific protocol

Check the list of established connections for a specific IP address

Check the list of established connections for a specific port

Check the list of listening ports

Check the network traffic using tcpdump

Increase in traffic causing the conntrack limit to be reached on the host.

Repair

Increase the conntrack limit on the affected host.

Learn more

Related Runbooks

Tomcat HTTP Request Headers or Payload Size Exceeds Limits.

Increase of the errors/second rate for Tomcat server

Postgres Connection Limit Reached

MySQL high threads running incident.

Support