---
id: a13ea313-9c64-48cc-ba8b-c57b41d37f21
---

# Nodes with PID Pressure in Kubernetes
---

Nodes with PID Pressure in Kubernetes is an incident type that occurs when a Kubernetes cluster node experiences PID pressure, meaning that it may not be able to start more containers. This is a rare condition where a pod or container spawns too many processes and starves the node of available process IDs. Each node has a limited number of process IDs to distribute amongst running processes; and if it runs out of IDs, no other processes can be started. Kubernetes lets you set PID thresholds for pods to limit their ability to perform runaway process-spawning, and a PID pressure condition means that one or more pods are using up their allocated PIDs and need to be examined.

### Parameters
```shell
# Environment Variables
export NODE_NAME="PLACEHOLDER"
export PID="PLACEHOLDER"
export NODE_IP="PLACEHOLDER"
export NODE_USERNAME="PLACEHOLDER"
export NEW_PID_THRESHOLD="PLACEHOLDER"

```

## Debug

### Get the list of nodes that have PID pressure
```shell
kubectl get node --selector=kubernetes.io/pid-pressure=true
```

### Get the list of pods that are using too many PIDs
```shell
kubectl top pods --all-namespaces | sort --reverse --key 3 | head
```

### Get the list of containers running on a node
```shell
kubectl describe node ${NODE_NAME}
```

### Get the PID usage of a process
```shell
ssh $NODE_USERNAME@$NODE_IP 
ps -o pid,ppid,%cpu,%mem,cmd ax | grep ${PID}
```

### Get the number of PIDs in use by a process
```shell
ssh $NODE_USERNAME@$NODE_IP 
cat /proc/${PID}/status | grep "Pid"
```

### Get the maximum number of PIDs available to a process
```shell
ssh $NODE_USERNAME@$NODE_IP 
cat /proc/sys/kernel/pid_max
```


## Repair
---

### Check if there are any misbehaving or stuck processes in the node and kill them to free up PIDs.
```shell

#!/bin/bash

# Set the node name
NODE=${NODE_NAME}

# Get the PID of the misbehaving process, if any
PID=$(ssh ${NODE_USERNAME}@${NODE_IP} "ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%cpu | head -n 2 | tail -n 1 | awk '{print $1}'")

# Check if a PID was returned
if [[ -n "$PID" ]]; then
  echo "Misbehaving process found with PID $PID."

  # Kill the process
  ssh ${NODE_USERNAME}@${NODE_IP} "kill $PID"

  # Check if the process was successfully killed
  if [[ $? -eq 0 ]]; then
    echo "Process with PID $PID has been killed successfully."
  else
    echo "Failed to kill process with PID $PID."
  fi
else
  echo "No misbehaving processes found."
fi

```

### Monitor the Kubernetes cluster carefully and identify which pods are using up the most PIDs. Once identified, adjust the PID thresholds for those pods to limit their ability to perform runaway process-spawning.
```shell

#!/bin/bash

# Get the list of pods running in the cluster
pods=$(kubectl get pods -o=name)

# Loop through each pod to calculate the number of PIDs it is using
for pod in $pods; do
  container_id=$(kubectl get $pod -o=jsonpath='{.status.containerStatuses[0].containerID}' | cut -d/ -f3)
  pid_count=$(docker exec $container_id ps -eLf | wc -l)
  echo "Pod $pod is using $pid_count PIDs"
done

# Identify the pods using up the most PIDs
most_pids_pod=$(kubectl get pods -o=jsonpath='{range .items[*]}{.metadata.name} {.status.containerStatuses[*].containerID} {.spec.containers[*].name} {.status.containerStatuses[*].state.running.startedAt} {print "\n"}{end}' | sort -nrk 2 | head -n 1 | awk '{print $1}')

# Adjust the PID thresholds for the most PIDs pod
kubectl patch pod $most_pids_pod -p '{"spec":{"securityContext":{"sysctls":[{"name":"kernel.pid_max","value":"${NEW_PID_THRESHOLD}"}]}}}}'

```

Nodes with PID Pressure in Kubernetes is an incident type that occurs when a Kubernetes cluster node experiences PID pressure, meaning that it may not be able to start more containers. This is a rare condition where a pod or container spawns too many processes and starves the node of available process IDs. Each node has a limited number of process IDs to distribute amongst running processes; and if it runs out of IDs, no other processes can be started. Kubernetes lets you set PID thresholds for pods to limit their ability to perform runaway process-spawning, and a PID pressure condition means that one or more pods are using up their allocated PIDs and need to be examined.


This incident type involves an issue with Kubernetes deployments where the expected number of pods to run is not matching the actual number of pods running. This can lead to alerts being triggered and potential disruptions in the system.


Kubernetes pods not starting - Deployment issue

This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

A Kubernetes Replicaset Incomplete incident typically occurs when a specific number of pods that should be running are not, due to reasons such as failed pod initialization, unavailability of resources in the cluster, or inability to pull the image. This incident is usually triggered when the difference between desired and running pods is greater than zero, and it can be detected through monitoring tools like Datadog.


Kubernetes Replicaset Incomplete

A Kubernetes Pod Restarting Monitoring incident is triggered when a pod running on a Kubernetes cluster restarts multiple times within a certain time frame. This incident type is usually used to detect issues with the application or infrastructure running on the cluster, and can be caused by various factors such as resource constraints, misconfigurations, or bugs in the application code. The incident is typically resolved by identifying and addressing the underlying cause of the pod restarts.


Kubernetes Pod Restarting Monitoring

The Kubernetes Nodes with Memorypressure incident type occurs when a Kubernetes cluster node is running low on memory, which can be caused by a memory leak in an application. This incident type requires immediate attention to prevent any downtime and ensure the proper functioning of the Kubernetes cluster. Typically, this incident type is monitored by DevOps teams using various monitoring tools, including PagerDuty, to identify and address memory pressure issues quickly.


Kubernetes Nodes with Memorypressure incident

```shell
# Environment Variables
export NODE_NAME="PLACEHOLDER"
export PID="PLACEHOLDER"
export NODE_IP="PLACEHOLDER"
export NODE_USERNAME="PLACEHOLDER"
export NEW_PID_THRESHOLD="PLACEHOLDER"

```


### Get the list of nodes that have PID pressure

```shell
kubectl get node --selector=kubernetes.io/pid-pressure=true
```

### Get the list of pods that are using too many PIDs

```shell
kubectl top pods --all-namespaces | sort --reverse --key 3 | head
```

### Get the list of containers running on a node

```shell
kubectl describe node ${NODE_NAME}
```

### Get the PID usage of a process

```shell
ssh $NODE_USERNAME@$NODE_IP 
ps -o pid,ppid,%cpu,%mem,cmd ax | grep ${PID}
```

### Get the number of PIDs in use by a process

```shell
ssh $NODE_USERNAME@$NODE_IP 
cat /proc/${PID}/status | grep "Pid"
```

### Get the maximum number of PIDs available to a process

```shell
ssh $NODE_USERNAME@$NODE_IP 
cat /proc/sys/kernel/pid_max
```


### Check if there are any misbehaving or stuck processes in the node and kill them to free up PIDs.

```shell

#!/bin/bash

# Set the node name
NODE=${NODE_NAME}

# Get the PID of the misbehaving process, if any
PID=$(ssh ${NODE_USERNAME}@${NODE_IP} "ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%cpu | head -n 2 | tail -n 1 | awk '{print $1}'")

# Check if a PID was returned
if [[ -n "$PID" ]]; then
  echo "Misbehaving process found with PID $PID."

  # Kill the process
  ssh ${NODE_USERNAME}@${NODE_IP} "kill $PID"

  # Check if the process was successfully killed
  if [[ $? -eq 0 ]]; then
    echo "Process with PID $PID has been killed successfully."
  else
    echo "Failed to kill process with PID $PID."
  fi
else
  echo "No misbehaving processes found."
fi

```

### Monitor the Kubernetes cluster carefully and identify which pods are using up the most PIDs. Once identified, adjust the PID thresholds for those pods to limit their ability to perform runaway process-spawning.

```shell

#!/bin/bash

# Get the list of pods running in the cluster
pods=$(kubectl get pods -o=name)

# Loop through each pod to calculate the number of PIDs it is using
for pod in $pods; do
  container_id=$(kubectl get $pod -o=jsonpath='{.status.containerStatuses[0].containerID}' | cut -d/ -f3)
  pid_count=$(docker exec $container_id ps -eLf | wc -l)
  echo "Pod $pod is using $pid_count PIDs"
done

# Identify the pods using up the most PIDs
most_pids_pod=$(kubectl get pods -o=jsonpath='{range .items[*]}{.metadata.name} {.status.containerStatuses[*].containerID} {.spec.containers[*].name} {.status.containerStatuses[*].state.running.startedAt} {print "\n"}{end}' | sort -nrk 2 | head -n 1 | awk '{print $1}')

# Adjust the PID thresholds for the most PIDs pod
kubectl patch pod $most_pids_pod -p '{"spec":{"securityContext":{"sysctls":[{"name":"kernel.pid_max","value":"${NEW_PID_THRESHOLD}"}]}}}}'

```


Nodes with PID Pressure in Kubernetes

Overview

Parameters

Debug

Get the list of nodes that have PID pressure

Get the list of pods that are using too many PIDs

Get the list of containers running on a node

Get the PID usage of a process

Get the number of PIDs in use by a process

Get the maximum number of PIDs available to a process

Repair

Check if there are any misbehaving or stuck processes in the node and kill them to free up PIDs.

Monitor the Kubernetes cluster carefully and identify which pods are using up the most PIDs. Once identified, adjust the PID thresholds for those pods to limit their ability to perform runaway process-spawning.

Learn more

Related Runbooks

Kubernetes pods not starting - Deployment issue

Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Replicaset Incomplete

Kubernetes Pod Restarting Monitoring

Support