---
id: e770ee28-8081-4234-ac9f-06840e40ff77
---

# Kubernetes - Troubleshooting Failed Persistent Volume Claims
---

This incident type involves issues with persistent volume claims in a Kubernetes cluster. This could be due to storage problems or other configuration issues. It is important to resolve these issues promptly to ensure that the Kubernetes cluster is functioning properly.

### Parameters
```shell
# Environment Variables
export NAMESPACE_NAME="PLACEHOLDER"
export POD_NAME="PLACEHOLDER"
export STORAGE_CLASS_NAME="PLACEHOLDER"
export NODE_NAME="PLACEHOLDER"
export VOLUME_SIZE="PLACEHOLDER"
export PERSISTENT_VOLUME_CLAIM_NAME="PLACEHOLDER"
```

## Debug

### Check the status of the persistent volume claims
```shell
kubectl get pvc
```

### Check the status of persistent volumes
```shell
kubectl get pv
```

### Check if there are any failed pods in the cluster
```shell
kubectl get pods --field-selector=status.phase=Failed
```

### Check the events for the namespace where the incident occurred
```shell
kubectl get events -n ${NAMESPACE_NAME}
```

### Describe the pod that is having issues with persistent volume claims
```shell
kubectl describe pod ${POD_NAME}
```

### Check the logs of the pod
```shell
kubectl logs ${POD_NAME}
```

### Check the storage class and volume configurations
```shell
kubectl get storageclass
kubectl describe storageclass ${STORAGE_CLASS_NAME}
```

### Check the state of the nodes in the cluster
```shell
kubectl get nodes
kubectl describe node ${NODE_NAME}
```

### Misconfigured storage classes, which can cause volumes to fail to mount properly.
```shell

#!/bin/bash

# Set the namespace for the Kubernetes resources we'll be checking
NAMESPACE=${NAMESPACE_NAME}

# Get a list of all storage classes in the namespace
STORAGE_CLASSES=$(kubectl get storageclasses -n $NAMESPACE -o=name)

# Loop through each storage class and check if it's set to the default mount options
for storage_class in $STORAGE_CLASSES; do
    MOUNT_OPTIONS=$(kubectl get $storage_class -n $NAMESPACE -o=jsonpath='{.parameters.mountOptions}')
    if [ "$MOUNT_OPTIONS" == "[]]" ]; then
        echo "Storage class $storage_class is misconfigured - mount options not set"
    fi
done

```

### Insufficient storage capacity in the cluster leading to failed persistent volume claims.
```shell

#!/bin/bash

# Set variables
NAMESPACE="${NAMESPACE_NAME}"
STORAGE_CLASS="${STORAGE_CLASS_NAME}"
VOLUME_SIZE="${VOLUME_SIZE}"

# Check available storage capacity
AVAILABLE_CAPACITY=$(kubectl get nodes -o json | jq -r '.items[].status.allocatable."ephemeral-storage"' | awk '{total += $1} END {print total}')
if [ $AVAILABLE_CAPACITY -lt $VOLUME_SIZE ]; then
    echo "Error: Insufficient storage capacity available in the cluster."
    exit 1
fi

# Check storage class
STORAGE_CLASS_EXISTS=$(kubectl get storageclass | grep $STORAGE_CLASS | wc -l)
if [ $STORAGE_CLASS_EXISTS -eq 0 ]; then
    echo "Error: Storage class $STORAGE_CLASS does not exist."
    exit 1
fi

# Check PVC status
PVC_STATUS=$(kubectl get pvc -n $NAMESPACE | awk '{print $2}' | tail -1)
if [ $PVC_STATUS != "Bound" ]; then
    echo "Error: Persistent Volume Claim in namespace $NAMESPACE is not bound."
    exit 1
fi

echo "No issues found."
exit 0

```

---

## Repair
---

### Check the storage configuration of the Kubernetes cluster to ensure that it is properly configured.
```shell

#!/bin/bash

# Ensure that the storage configuration is properly set by checking the StorageClass configuration
kubectl get storageclass

# Check if there are any errors in the StorageClass configuration
kubectl describe storageclass

# If there are any issues with the StorageClass, update the configuration
kubectl edit storageclass ${STORAGE_CLASS_NAME}

```

### Check the persistent volume claims to ensure that they are correctly configured and that the storage class is available.
```shell

#!/bin/bash

# Check if the kubectl command is installed
which kubectl > /dev/null 2>&1
if [ $? -ne 0 ]; then
  echo "Error: kubectl command not found"
  exit 1
fi

# Check the persistent volume claims
kubectl get pvc ${PERSISTENT_VOLUME_CLAIM_NAME} -n ${NAMESPACE} > /dev/null 2>&1
if [ $? -ne 0 ]; then
  echo "Error: Persistent volume claim ${PERSISTENT_VOLUME_CLAIM_NAME} not found in namespace ${NAMESPACE}"
  exit 1
else
  echo "Persistent volume claim ${PERSISTENT_VOLUME_CLAIM_NAME} found in namespace ${NAMESPACE}"
fi

# Check the storage class
kubectl get sc ${STORAGE_CLASS_NAME} > /dev/null 2>&1
if [ $? -ne 0 ]; then
  echo "Error: Storage class ${STORAGE_CLASS_NAME} not found"
  exit 1
else
  echo "Storage class ${STORAGE_CLASS_NAME} found"
fi

```

---


This incident type involves issues with persistent volume claims in a Kubernetes cluster. This could be due to storage problems or other configuration issues. It is important to resolve these issues promptly to ensure that the Kubernetes cluster is functioning properly.


The Vault cluster health incident is related to the health of a Vault cluster instance. This incident type is triggered when the cluster instance is not healthy and requires attention to ensure it is functioning properly. The incident typically involves evaluating the current state of the cluster instance, diagnosing the issue, and taking corrective action to restore the health of the instance.


Vault cluster health incident on kubernetes

This incident type involves an issue with Kubernetes deployments where the expected number of pods to run is not matching the actual number of pods running. This can lead to alerts being triggered and potential disruptions in the system.


Kubernetes pods not starting - Deployment issue

This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Pods Pending incident indicates that one or more pods in a Kubernetes cluster are not running as expected and are in a pending state. This can happen due to various reasons such as resource constraints, scheduling issues, or network problems. This incident can impact the availability and performance of the application running on the Kubernetes cluster. It requires immediate attention to diagnose and resolve the underlying issue to ensure the pods are running as expected.


Kubernetes Pods Pending

A Kubernetes Pod Restarting Monitoring incident is triggered when a pod running on a Kubernetes cluster restarts multiple times within a certain time frame. This incident type is usually used to detect issues with the application or infrastructure running on the cluster, and can be caused by various factors such as resource constraints, misconfigurations, or bugs in the application code. The incident is typically resolved by identifying and addressing the underlying cause of the pod restarts.


Kubernetes Pod Restarting Monitoring

```shell
# Environment Variables
export NAMESPACE_NAME="PLACEHOLDER"
export POD_NAME="PLACEHOLDER"
export STORAGE_CLASS_NAME="PLACEHOLDER"
export NODE_NAME="PLACEHOLDER"
export VOLUME_SIZE="PLACEHOLDER"
export PERSISTENT_VOLUME_CLAIM_NAME="PLACEHOLDER"
```


### Check the status of the persistent volume claims

```shell
kubectl get pvc
```

### Check the status of persistent volumes

```shell
kubectl get pv
```

### Check if there are any failed pods in the cluster

```shell
kubectl get pods --field-selector=status.phase=Failed
```

### Check the events for the namespace where the incident occurred

```shell
kubectl get events -n ${NAMESPACE_NAME}
```

### Describe the pod that is having issues with persistent volume claims

```shell
kubectl describe pod ${POD_NAME}
```

### Check the logs of the pod

```shell
kubectl logs ${POD_NAME}
```

### Check the storage class and volume configurations

```shell
kubectl get storageclass
kubectl describe storageclass ${STORAGE_CLASS_NAME}
```

### Check the state of the nodes in the cluster

```shell
kubectl get nodes
kubectl describe node ${NODE_NAME}
```

### Misconfigured storage classes, which can cause volumes to fail to mount properly.

```shell

#!/bin/bash

# Set the namespace for the Kubernetes resources we'll be checking
NAMESPACE=${NAMESPACE_NAME}

# Get a list of all storage classes in the namespace
STORAGE_CLASSES=$(kubectl get storageclasses -n $NAMESPACE -o=name)

# Loop through each storage class and check if it's set to the default mount options
for storage_class in $STORAGE_CLASSES; do
    MOUNT_OPTIONS=$(kubectl get $storage_class -n $NAMESPACE -o=jsonpath='{.parameters.mountOptions}')
    if [ "$MOUNT_OPTIONS" == "[]]" ]; then
        echo "Storage class $storage_class is misconfigured - mount options not set"
    fi
done

```

### Insufficient storage capacity in the cluster leading to failed persistent volume claims.

```shell

#!/bin/bash

# Set variables
NAMESPACE="${NAMESPACE_NAME}"
STORAGE_CLASS="${STORAGE_CLASS_NAME}"
VOLUME_SIZE="${VOLUME_SIZE}"

# Check available storage capacity
AVAILABLE_CAPACITY=$(kubectl get nodes -o json | jq -r '.items[].status.allocatable."ephemeral-storage"' | awk '{total += $1} END {print total}')
if [ $AVAILABLE_CAPACITY -lt $VOLUME_SIZE ]; then
    echo "Error: Insufficient storage capacity available in the cluster."
    exit 1
fi

# Check storage class
STORAGE_CLASS_EXISTS=$(kubectl get storageclass | grep $STORAGE_CLASS | wc -l)
if [ $STORAGE_CLASS_EXISTS -eq 0 ]; then
    echo "Error: Storage class $STORAGE_CLASS does not exist."
    exit 1
fi

# Check PVC status
PVC_STATUS=$(kubectl get pvc -n $NAMESPACE | awk '{print $2}' | tail -1)
if [ $PVC_STATUS != "Bound" ]; then
    echo "Error: Persistent Volume Claim in namespace $NAMESPACE is not bound."
    exit 1
fi

echo "No issues found."
exit 0

```


### Check the storage configuration of the Kubernetes cluster to ensure that it is properly configured.

```shell

#!/bin/bash

# Ensure that the storage configuration is properly set by checking the StorageClass configuration
kubectl get storageclass

# Check if there are any errors in the StorageClass configuration
kubectl describe storageclass

# If there are any issues with the StorageClass, update the configuration
kubectl edit storageclass ${STORAGE_CLASS_NAME}

```

### Check the persistent volume claims to ensure that they are correctly configured and that the storage class is available.

```shell

#!/bin/bash

# Check if the kubectl command is installed
which kubectl > /dev/null 2>&1
if [ $? -ne 0 ]; then
  echo "Error: kubectl command not found"
  exit 1
fi

# Check the persistent volume claims
kubectl get pvc ${PERSISTENT_VOLUME_CLAIM_NAME} -n ${NAMESPACE} > /dev/null 2>&1
if [ $? -ne 0 ]; then
  echo "Error: Persistent volume claim ${PERSISTENT_VOLUME_CLAIM_NAME} not found in namespace ${NAMESPACE}"
  exit 1
else
  echo "Persistent volume claim ${PERSISTENT_VOLUME_CLAIM_NAME} found in namespace ${NAMESPACE}"
fi

# Check the storage class
kubectl get sc ${STORAGE_CLASS_NAME} > /dev/null 2>&1
if [ $? -ne 0 ]; then
  echo "Error: Storage class ${STORAGE_CLASS_NAME} not found"
  exit 1
else
  echo "Storage class ${STORAGE_CLASS_NAME} found"
fi

```


Kubernetes - Troubleshooting Failed Persistent Volume Claims

Overview

Parameters

Debug

Check the status of the persistent volume claims

Check the status of persistent volumes

Check if there are any failed pods in the cluster

Check the events for the namespace where the incident occurred

Describe the pod that is having issues with persistent volume claims

Check the logs of the pod

Check the storage class and volume configurations

Check the state of the nodes in the cluster

Misconfigured storage classes, which can cause volumes to fail to mount properly.

Insufficient storage capacity in the cluster leading to failed persistent volume claims.

Repair

Check the storage configuration of the Kubernetes cluster to ensure that it is properly configured.

Check the persistent volume claims to ensure that they are correctly configured and that the storage class is available.

Learn more

Related Runbooks

Vault cluster health incident on kubernetes

Kubernetes pods not starting - Deployment issue

Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Pods Pending

Support