---
id: 3ca60976-47a1-4e0b-be21-dd30635daa74
---

# Kubernetes - Failed Persistent Volume Claims
---

This incident type involves issues with persistent volume claims in a Kubernetes cluster. This could be due to storage problems or other configuration issues. It is important to resolve these issues promptly to ensure that the Kubernetes cluster is functioning properly.

### Parameters
```shell
# Environment Variables

export PVC_NAME="PLACEHOLDER"

export PV_NAME="PLACEHOLDER"

export SC_NAME="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export STORAGE_CLASS_NAME="PLACEHOLDER"

export STORAGE_PROVIDER="PLACEHOLDER"

```

## Debug

### Get a list of all PersistentVolumes
```shell
kubectl get pv
```

### Get a list of all PersistentVolumeClaims
```shell
kubectl get pvc
```

### Check the status of a specific PersistentVolumeClaim
```shell
kubectl describe pvc ${PVC_NAME}
```

### Check the status of a specific PersistentVolume
```shell
kubectl describe pv ${PV_NAME}
```
### Check the status of a specific storage class
```shell
kubectl describe sc ${SC_NAME}
```
###  Check for misconfiguration in the Persistent Volume Claims (PVCs) setting
```shell
#!/bin/bash

# Set the namespace and PVC name

namespace=${NAMESPACE}

pvc_name=${PVC_NAME}

# Get the PVC details

pvc=$(kubectl get pvc $pvc_name -n $namespace -o json)

# Check if the PVC is in a failed state

if [[ $(echo $pvc | jq -r '.status.phase') == "Failed" ]]; then

  echo "The PVC '$pvc_name' in namespace '$namespace' is in a failed state."

  # Get the events for the PVC

  events=$(kubectl get events -n $namespace --field-selector involvedObject.kind=PersistentVolumeClaim,involvedObject.name=$pvc_name -o json)

  # Check if there are any events for the PVC

  if [[ $(echo $events | jq -r '.items | length') == 0 ]]; then

    echo "There are no events for the PVC '$pvc_name' in namespace '$namespace'."

  else

    # Print the events for the PVC

    echo "Events for the PVC '$pvc_name' in namespace '$namespace':"

    echo $events | jq -r '.items[] | .lastTimestamp + " " + .reason + " " + .message'

  fi

else

  echo "The PVC '$pvc_name' in namespace '$namespace' is not in a failed state."

fi

```
### Check the storage class and ensure that it is correctly configured and able to provide the requested storage.
```shell
bash

#!/bin/bash



# Get the storage class

storage_class=$(kubectl get storageclass ${STORAGE_CLASS_NAME} -o jsonpath='{.provisioner}')



# Check if the storage class is available

if [ "$storage_class" != "${STORAGE_PROVIDER}" ]; then

  # If the storage class is not available, exit with an error

  echo "Error: Storage class ${STORAGE_CLASS_NAME} is not available."

  exit 1

else

  # If the storage class is available, exit with a success message

  echo "Storage class ${STORAGE_CLASS_NAME} is available and correctly configured."

  exit 0

fi


```

## Repair

### Check the storage provisioner and ensure that it is correctly configured and running.
```shell

#!/bin/bash

STORAGE_PROVISIONER_NAMESPACE="PLACEHOLDER"

STORAGE_PROVISIONER_YAML_FILE="PLACEHOLDER"

STORAGE_PROVISIONER_POD_NAME="PLACEHOLDER"

STORAGE_PROVISIONER_YAML_FILE="PLACEHOLDER"



# Check if the storage provisioner is installed

if ! kubectl get storageclass ${STORAGE_CLASS_NAME} &> /dev/null; then

  echo "Storage provisioner ${STORAGE_CLASS_NAME} not found. Installing..."

  kubectl apply -f ${STORAGE_PROVISIONER_YAML_FILE}

fi



# Check if the storage provisioner is running

if ! kubectl get pods -n ${STORAGE_PROVISIONER_NAMESPACE} | grep ${STORAGE_PROVISIONER_POD_NAME} | grep Running &> /dev/null; then

  echo "Storage provisioner ${STORAGE_PROVISIONER_POD_NAME} not running. Restarting..."

  kubectl delete pod ${STORAGE_PROVISIONER_POD_NAME} -n ${STORAGE_PROVISIONER_NAMESPACE}

fi



echo "Storage provisioner ${STORAGE_CLASS_NAME} is correctly configured and running."


```
### Recreates Failed Persistent Volume Claims.
```shell

kubectl get pvc $PVC_NAME -o name | xargs kubectl delete

```


This incident type involves issues with persistent volume claims in a Kubernetes cluster. This could be due to storage problems or other configuration issues. It is important to resolve these issues promptly to ensure that the Kubernetes cluster is functioning properly.


The Vault cluster health incident is related to the health of a Vault cluster instance. This incident type is triggered when the cluster instance is not healthy and requires attention to ensure it is functioning properly. The incident typically involves evaluating the current state of the cluster instance, diagnosing the issue, and taking corrective action to restore the health of the instance.


Vault cluster health incident on kubernetes

The Vault Sealed incident type occurs when an instance of the Vault has been sealed due to an error or issue. This can cause disruption to the system and requires immediate attention to resolve the issue and unseal the Vault instance. It is important to identify the root cause of the issue in order to prevent it from happening again in the future.


Vault Sealed Incident

This incident type involves an issue with Kubernetes deployments where the expected number of pods to run is not matching the actual number of pods running. This can lead to alerts being triggered and potential disruptions in the system.


Kubernetes pods not starting - Deployment issue

This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Pods Pending incident indicates that one or more pods in a Kubernetes cluster are not running as expected and are in a pending state. This can happen due to various reasons such as resource constraints, scheduling issues, or network problems. This incident can impact the availability and performance of the application running on the Kubernetes cluster. It requires immediate attention to diagnose and resolve the underlying issue to ensure the pods are running as expected.


Kubernetes Pods Pending

```shell
# Environment Variables

export PVC_NAME="PLACEHOLDER"

export PV_NAME="PLACEHOLDER"

export SC_NAME="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export STORAGE_CLASS_NAME="PLACEHOLDER"

export STORAGE_PROVIDER="PLACEHOLDER"

```


### Get a list of all PersistentVolumes

```shell
kubectl get pv
```

### Get a list of all PersistentVolumeClaims

```shell
kubectl get pvc
```

### Check the status of a specific PersistentVolumeClaim

```shell
kubectl describe pvc ${PVC_NAME}
```

### Check the status of a specific PersistentVolume

```shell
kubectl describe pv ${PV_NAME}
```

### Check the status of a specific storage class

```shell
kubectl describe sc ${SC_NAME}
```

### Check for misconfiguration in the Persistent Volume Claims (PVCs) setting

```shell
#!/bin/bash

# Set the namespace and PVC name

namespace=${NAMESPACE}

pvc_name=${PVC_NAME}

# Get the PVC details

pvc=$(kubectl get pvc $pvc_name -n $namespace -o json)

# Check if the PVC is in a failed state

if [[ $(echo $pvc | jq -r '.status.phase') == "Failed" ]]; then

  echo "The PVC '$pvc_name' in namespace '$namespace' is in a failed state."

  # Get the events for the PVC

  events=$(kubectl get events -n $namespace --field-selector involvedObject.kind=PersistentVolumeClaim,involvedObject.name=$pvc_name -o json)

  # Check if there are any events for the PVC

  if [[ $(echo $events | jq -r '.items | length') == 0 ]]; then

    echo "There are no events for the PVC '$pvc_name' in namespace '$namespace'."

  else

    # Print the events for the PVC

    echo "Events for the PVC '$pvc_name' in namespace '$namespace':"

    echo $events | jq -r '.items[] | .lastTimestamp + " " + .reason + " " + .message'

  fi

else

  echo "The PVC '$pvc_name' in namespace '$namespace' is not in a failed state."

fi

```

### Check the storage class and ensure that it is correctly configured and able to provide the requested storage.

```shell
bash

#!/bin/bash



# Get the storage class

storage_class=$(kubectl get storageclass ${STORAGE_CLASS_NAME} -o jsonpath='{.provisioner}')



# Check if the storage class is available

if [ "$storage_class" != "${STORAGE_PROVIDER}" ]; then

  # If the storage class is not available, exit with an error

  echo "Error: Storage class ${STORAGE_CLASS_NAME} is not available."

  exit 1

else

  # If the storage class is available, exit with a success message

  echo "Storage class ${STORAGE_CLASS_NAME} is available and correctly configured."

  exit 0

fi


```


### Check the storage provisioner and ensure that it is correctly configured and running.

```shell

#!/bin/bash

STORAGE_PROVISIONER_NAMESPACE="PLACEHOLDER"

STORAGE_PROVISIONER_YAML_FILE="PLACEHOLDER"

STORAGE_PROVISIONER_POD_NAME="PLACEHOLDER"

STORAGE_PROVISIONER_YAML_FILE="PLACEHOLDER"



# Check if the storage provisioner is installed

if ! kubectl get storageclass ${STORAGE_CLASS_NAME} &> /dev/null; then

  echo "Storage provisioner ${STORAGE_CLASS_NAME} not found. Installing..."

  kubectl apply -f ${STORAGE_PROVISIONER_YAML_FILE}

fi



# Check if the storage provisioner is running

if ! kubectl get pods -n ${STORAGE_PROVISIONER_NAMESPACE} | grep ${STORAGE_PROVISIONER_POD_NAME} | grep Running &> /dev/null; then

  echo "Storage provisioner ${STORAGE_PROVISIONER_POD_NAME} not running. Restarting..."

  kubectl delete pod ${STORAGE_PROVISIONER_POD_NAME} -n ${STORAGE_PROVISIONER_NAMESPACE}

fi



echo "Storage provisioner ${STORAGE_CLASS_NAME} is correctly configured and running."


```

### Recreates Failed Persistent Volume Claims.

```shell

kubectl get pvc $PVC_NAME -o name | xargs kubectl delete

```


Kubernetes - Failed Persistent Volume Claims

Overview

Parameters

Debug

Get a list of all PersistentVolumes

Get a list of all PersistentVolumeClaims

Check the status of a specific PersistentVolumeClaim

Check the status of a specific PersistentVolume

Check the status of a specific storage class

Check for misconfiguration in the Persistent Volume Claims (PVCs) setting

Check the storage class and ensure that it is correctly configured and able to provide the requested storage.

Repair

Check the storage provisioner and ensure that it is correctly configured and running.

Recreates Failed Persistent Volume Claims.

Learn more

Related Runbooks

Vault cluster health incident on kubernetes

Vault Sealed Incident

Kubernetes pods not starting - Deployment issue

Kubernetes Statefulset Replicas Monitoring Incident

Support