---
id: 844675d9-39f9-4ed1-bb3a-b30cef06f728
---

# Kubernetes - Pods Failed
---

This incident type relates to a failure of Kubernetes pods, which are a fundamental unit of deployment for containerized applications in Kubernetes. The failure could be related to any number of issues, such as a misconfiguration, a software bug, or a resource constraint. The incident may require immediate attention from a software engineer to resolve the issue and prevent further disruption to the application.

### Parameters
```shell
# Environment Variables

export NAMESPACE="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

export DEPLOYMENT="PLACEHOLDER"

export MEMORY_THRESHOLD="PLACEHOLDER"

export CPU_THRESHOLD="PLACEHOLDER"

export STORAGE_THRESHOLD="PLACEHOLDER"
```

## Debug

### Get the list of pods in the cluster
```shell
kubectl get pods -n ${NAMESPACE}
```

### Check the logs of a pod
```shell
kubectl logs ${POD_NAME} -n ${NAMESPACE}
```

### Describe a pod to get more details about its state and configuration
```shell
kubectl describe pod ${POD_NAME} -n ${NAMESPACE}
```

### Check the status of the pod's containers
```shell
kubectl get pods ${POD_NAME} -n ${NAMESPACE} -o jsonpath='{.status.containerStatuses[*].state}'
```

### Get the events related to a pod
```shell
kubectl get events --field-selector involvedObject.name=${POD_NAME} -n ${NAMESPACE}
```

### Check the resource usage of a pod
```shell
kubectl top pod ${POD_NAME} -n ${NAMESPACE}
```
### Resource constraints: If a pod exceeds its resource limits, such as CPU or memory, it may be terminated by Kubernetes.
```shell
bash

#!/bin/bash

# Set the namespace and pod name

NAMESPACE=${NAMESPACE}

POD_NAME=${POD_NAME}

# Get the pod status

POD_STATUS=$(kubectl get pod $POD_NAME -n $NAMESPACE -o jsonpath='{.status.phase}')

# If the pod is terminated, check the reason for termination

if [[ "$POD_STATUS" == "Failed" ]]; then

  TERMINATION_REASON=$(kubectl get pod $POD_NAME -n $NAMESPACE -o jsonpath='{.status.containerStatuses[0].lastTerminationReason}')

    # If the reason is related to resource constraints, print the relevant information

  if [[ "$TERMINATION_REASON" == *"out of memory"* ]] || [[ "$TERMINATION_REASON" == *"out of cpu"* ]]; then

    echo "Pod $POD_NAME in namespace $NAMESPACE was terminated due to resource constraints."

    echo "Termination reason: $TERMINATION_REASON"

    # Describe the pod to get more information

    kubectl describe pod $POD_NAME -n $NAMESPACE

  else

    echo "Pod $POD_NAME in namespace $NAMESPACE was terminated, but not due to resource constraints."

    echo "Termination reason: $TERMINATION_REASON"

  fi

fi

```

## Repair
### Verify that the Kubernetes cluster has enough resources available for the pods to run. Check CPU, memory, and storage usage, and make sure that the pods are not being starved for resources.
```shell
bash

#!/bin/bash

# Define the Kubernetes namespace and deployment name

NAMESPACE=${NAMESPACE}

DEPLOYMENT=${DEPLOYMENT}

# Get the current resource usage for the Kubernetes deployment

CPU=$(kubectl top pods --namespace=$NAMESPACE | grep $DEPLOYMENT | awk '{print $2}')

MEMORY=$(kubectl top pods --namespace=$NAMESPACE | grep $DEPLOYMENT | awk '{print $3}')

STORAGE=$(kubectl describe pod $DEPLOYMENT --namespace=$NAMESPACE | grep -E "^Storage" | awk '{print $2}')

# Define the thresholds for resource usage

CPU_THRESHOLD=${CPU_THRESHOLD}

MEMORY_THRESHOLD=${MEMORY_THRESHOLD}

STORAGE_THRESHOLD=${STORAGE_THRESHOLD}

# Check if the resource usage is above the thresholds and remediate if necessary

if [[ $CPU -gt $CPU_THRESHOLD ]]; then

    # Scale up the deployment to increase CPU resources

    kubectl scale deployment $DEPLOYMENT --namespace=$NAMESPACE --replicas=2

elif [[ $MEMORY -gt $MEMORY_THRESHOLD ]]; then

    # Scale up the deployment to increase memory resources

    kubectl scale deployment $DEPLOYMENT --namespace=$NAMESPACE --replicas=2

elif [[ $STORAGE -gt $STORAGE_THRESHOLD ]]; then

    # Delete old pods to free up storage space

    kubectl delete pod $DEPLOYMENT-$(date +%Y%m%d%H%M%S) --namespace=$NAMESPACE

fi


```

This incident type relates to a failure of Kubernetes pods, which are a fundamental unit of deployment for containerized applications in Kubernetes. The failure could be related to any number of issues, such as a misconfiguration, a software bug, or a resource constraint. The incident may require immediate attention from a software engineer to resolve the issue and prevent further disruption to the application.


The Vault cluster health incident is related to the health of a Vault cluster instance. This incident type is triggered when the cluster instance is not healthy and requires attention to ensure it is functioning properly. The incident typically involves evaluating the current state of the cluster instance, diagnosing the issue, and taking corrective action to restore the health of the instance.


Vault cluster health incident on kubernetes

This incident type involves an issue with Kubernetes deployments where the expected number of pods to run is not matching the actual number of pods running. This can lead to alerts being triggered and potential disruptions in the system.


Kubernetes pods not starting - Deployment issue

This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

A Kubernetes Replicaset Incomplete incident typically occurs when a specific number of pods that should be running are not, due to reasons such as failed pod initialization, unavailability of resources in the cluster, or inability to pull the image. This incident is usually triggered when the difference between desired and running pods is greater than zero, and it can be detected through monitoring tools like Datadog.


Kubernetes Replicaset Incomplete

Kubernetes Pods Pending incident indicates that one or more pods in a Kubernetes cluster are not running as expected and are in a pending state. This can happen due to various reasons such as resource constraints, scheduling issues, or network problems. This incident can impact the availability and performance of the application running on the Kubernetes cluster. It requires immediate attention to diagnose and resolve the underlying issue to ensure the pods are running as expected.


Kubernetes Pods Pending

```shell
# Environment Variables

export NAMESPACE="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

export DEPLOYMENT="PLACEHOLDER"

export MEMORY_THRESHOLD="PLACEHOLDER"

export CPU_THRESHOLD="PLACEHOLDER"

export STORAGE_THRESHOLD="PLACEHOLDER"
```


### Get the list of pods in the cluster

```shell
kubectl get pods -n ${NAMESPACE}
```

### Check the logs of a pod

```shell
kubectl logs ${POD_NAME} -n ${NAMESPACE}
```

### Describe a pod to get more details about its state and configuration

```shell
kubectl describe pod ${POD_NAME} -n ${NAMESPACE}
```

### Check the status of the pod's containers

```shell
kubectl get pods ${POD_NAME} -n ${NAMESPACE} -o jsonpath='{.status.containerStatuses[*].state}'
```

### Get the events related to a pod

```shell
kubectl get events --field-selector involvedObject.name=${POD_NAME} -n ${NAMESPACE}
```

### Check the resource usage of a pod

```shell
kubectl top pod ${POD_NAME} -n ${NAMESPACE}
```

### Resource constraints: If a pod exceeds its resource limits, such as CPU or memory, it may be terminated by Kubernetes.

```shell
bash

#!/bin/bash

# Set the namespace and pod name

NAMESPACE=${NAMESPACE}

POD_NAME=${POD_NAME}

# Get the pod status

POD_STATUS=$(kubectl get pod $POD_NAME -n $NAMESPACE -o jsonpath='{.status.phase}')

# If the pod is terminated, check the reason for termination

if [[ "$POD_STATUS" == "Failed" ]]; then

  TERMINATION_REASON=$(kubectl get pod $POD_NAME -n $NAMESPACE -o jsonpath='{.status.containerStatuses[0].lastTerminationReason}')

    # If the reason is related to resource constraints, print the relevant information

  if [[ "$TERMINATION_REASON" == *"out of memory"* ]] || [[ "$TERMINATION_REASON" == *"out of cpu"* ]]; then

    echo "Pod $POD_NAME in namespace $NAMESPACE was terminated due to resource constraints."

    echo "Termination reason: $TERMINATION_REASON"

    # Describe the pod to get more information

    kubectl describe pod $POD_NAME -n $NAMESPACE

  else

    echo "Pod $POD_NAME in namespace $NAMESPACE was terminated, but not due to resource constraints."

    echo "Termination reason: $TERMINATION_REASON"

  fi

fi

```


### Verify that the Kubernetes cluster has enough resources available for the pods to run. Check CPU, memory, and storage usage, and make sure that the pods are not being starved for resources.

```shell
bash

#!/bin/bash

# Define the Kubernetes namespace and deployment name

NAMESPACE=${NAMESPACE}

DEPLOYMENT=${DEPLOYMENT}

# Get the current resource usage for the Kubernetes deployment

CPU=$(kubectl top pods --namespace=$NAMESPACE | grep $DEPLOYMENT | awk '{print $2}')

MEMORY=$(kubectl top pods --namespace=$NAMESPACE | grep $DEPLOYMENT | awk '{print $3}')

STORAGE=$(kubectl describe pod $DEPLOYMENT --namespace=$NAMESPACE | grep -E "^Storage" | awk '{print $2}')

# Define the thresholds for resource usage

CPU_THRESHOLD=${CPU_THRESHOLD}

MEMORY_THRESHOLD=${MEMORY_THRESHOLD}

STORAGE_THRESHOLD=${STORAGE_THRESHOLD}

# Check if the resource usage is above the thresholds and remediate if necessary

if [[ $CPU -gt $CPU_THRESHOLD ]]; then

    # Scale up the deployment to increase CPU resources

    kubectl scale deployment $DEPLOYMENT --namespace=$NAMESPACE --replicas=2

elif [[ $MEMORY -gt $MEMORY_THRESHOLD ]]; then

    # Scale up the deployment to increase memory resources

    kubectl scale deployment $DEPLOYMENT --namespace=$NAMESPACE --replicas=2

elif [[ $STORAGE -gt $STORAGE_THRESHOLD ]]; then

    # Delete old pods to free up storage space

    kubectl delete pod $DEPLOYMENT-$(date +%Y%m%d%H%M%S) --namespace=$NAMESPACE

fi


```


Kubernetes - Pods Failed

Overview

Parameters

Debug

Get the list of pods in the cluster

Check the logs of a pod

Describe a pod to get more details about its state and configuration

Check the status of the pod's containers

Check the resource usage of a pod

Resource constraints: If a pod exceeds its resource limits, such as CPU or memory, it may be terminated by Kubernetes.

Repair

Verify that the Kubernetes cluster has enough resources available for the pods to run. Check CPU, memory, and storage usage, and make sure that the pods are not being starved for resources.

Learn more

Related Runbooks

Vault cluster health incident on kubernetes

Kubernetes pods not starting - Deployment issue

Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Replicaset Incomplete

Support