---
id: 374aaa3b-81d3-47c1-892a-87c1534f9fbe
---

# Kubernetes Nodes with Disk Pressure
---

Kubernetes is a popular container orchestration system used to manage and deploy containerized applications. Kubernetes nodes are the individual servers in a Kubernetes cluster that run the containers. Disk pressure is a condition where a node is using too much disk space or is using disk space too fast according to the thresholds set in the Kubernetes configuration. This condition can be caused by applications legitimately needing more space or an application misbehaving and filling up the disk prematurely in an unanticipated manner. It is important to monitor disk pressure as it can lead to performance issues, instability, or even downtime. When a Kubernetes node experiences disk pressure, it can trigger an incident that needs to be addressed to ensure the stability of the cluster.

### Parameters
```shell
# Environment Variables

export NODE_NAME="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"

export DISK_SPACE_LIMIT="PLACEHOLDER"

```

## Debug

### Get a list of nodes in the Kubernetes cluster
```shell
kubectl get nodes
```

### Check the disk usage on a specific node
```shell
kubectl ssh ${NODE_NAME} df -h
```

### Check if the Kubernetes nodes are running out of disk space
```shell
kubectl get nodes --field-selector spec.unschedulable=true
```

### Check the Kubernetes event logs for any disk-related errors
```shell
kubectl get events --field-selector involvedObject.kind=Node,reason=OutOfDisk
```

### Check if any pods are using a lot of disk space
```shell
kubectl get pods --all-namespaces -o json | jq '.items[].spec.containers[].resources.requests.storage' | grep -v null
```

### Check which namespaces are using the most disk space
```shell
kubectl get namespace --no-headers | xargs -I {} sh -c 'echo {}; kubectl get pods -n {} --no-headers | xargs -I {} sh -c "kubectl logs {} -n {} | wc -c"' | awk '{print $1" "($2/1024/1024)" MB"}' | sort -k2 -n -r | head
```
## Repair

### Cordon and then drain the node.
```shell
kubectl cordon $NODE_NAME && kubectl drain $NODE_NAME--ignore-daemonsets --delete-local-data

```

Kubernetes is a popular container orchestration system used to manage and deploy containerized applications. Kubernetes nodes are the individual servers in a Kubernetes cluster that run the containers. Disk pressure is a condition where a node is using too much disk space or is using disk space too fast according to the thresholds set in the Kubernetes configuration. This condition can be caused by applications legitimately needing more space or an application misbehaving and filling up the disk prematurely in an unanticipated manner. It is important to monitor disk pressure as it can lead to performance issues, instability, or even downtime. When a Kubernetes node experiences disk pressure, it can trigger an incident that needs to be addressed to ensure the stability of the cluster.


Nodes with PID Pressure in Kubernetes is an incident type that occurs when a Kubernetes cluster node experiences PID pressure, meaning that it may not be able to start more containers. This is a rare condition where a pod or container spawns too many processes and starves the node of available process IDs. Each node has a limited number of process IDs to distribute amongst running processes; and if it runs out of IDs, no other processes can be started. Kubernetes lets you set PID thresholds for pods to limit their ability to perform runaway process-spawning, and a PID pressure condition means that one or more pods are using up their allocated PIDs and need to be examined.


Nodes with PID Pressure in Kubernetes

The incident type of "Kubernetes deployment with multiple restarts" indicates that a Kubernetes deployment has experienced multiple restarts within a certain timeframe, which is usually indicative of a problem. Kubernetes is a popular container orchestration platform that automates the deployment, scaling, and management of containerized applications. When a deployment experiences multiple restarts, it can impact the availability and performance of the application, and can be a sign of underlying issues that need to be addressed. This incident type is typically monitored and managed by DevOps teams responsible for ensuring the health and reliability of Kubernetes-based applications.


Kubernetes deployment with multiple restarts

This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Pods Pending incident indicates that one or more pods in a Kubernetes cluster are not running as expected and are in a pending state. This can happen due to various reasons such as resource constraints, scheduling issues, or network problems. This incident can impact the availability and performance of the application running on the Kubernetes cluster. It requires immediate attention to diagnose and resolve the underlying issue to ensure the pods are running as expected.


Kubernetes Pods Pending

This incident type involves nodes in a Kubernetes cluster that are experiencing network unavailability, meaning they are not accessible. This could be due to a misconfiguration, route exhaustion, or a physical problem with the network connection to the hardware. It is a high urgency incident that requires immediate attention to restore network connectivity to the affected nodes.


Kubernetes Nodes with Network Unavailable

```shell
# Environment Variables

export NODE_NAME="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"

export DISK_SPACE_LIMIT="PLACEHOLDER"

```


### Get a list of nodes in the Kubernetes cluster

```shell
kubectl get nodes
```

### Check the disk usage on a specific node

```shell
kubectl ssh ${NODE_NAME} df -h
```

### Check if the Kubernetes nodes are running out of disk space

```shell
kubectl get nodes --field-selector spec.unschedulable=true
```

### Check the Kubernetes event logs for any disk-related errors

```shell
kubectl get events --field-selector involvedObject.kind=Node,reason=OutOfDisk
```

### Check if any pods are using a lot of disk space

```shell
kubectl get pods --all-namespaces -o json | jq '.items[].spec.containers[].resources.requests.storage' | grep -v null
```

### Check which namespaces are using the most disk space

```shell
kubectl get namespace --no-headers | xargs -I {} sh -c 'echo {}; kubectl get pods -n {} --no-headers | xargs -I {} sh -c "kubectl logs {} -n {} | wc -c"' | awk '{print $1" "($2/1024/1024)" MB"}' | sort -k2 -n -r | head
```


### Cordon and then drain the node.

```shell
kubectl cordon $NODE_NAME && kubectl drain $NODE_NAME--ignore-daemonsets --delete-local-data

```


Kubernetes Nodes with Disk Pressure

Overview

Parameters

Debug

Get a list of nodes in the Kubernetes cluster

Check the disk usage on a specific node

Check if the Kubernetes nodes are running out of disk space

Check if any pods are using a lot of disk space

Check which namespaces are using the most disk space

Repair

Cordon and then drain the node.

Learn more

Related Runbooks

Nodes with PID Pressure in Kubernetes

Kubernetes deployment with multiple restarts

Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Pods Pending

Support