---
id: 606739b8-e1ac-4991-8427-c72fbcde3a66
---

# CrashloopBackOff on Kubernetes Pod
---

CrashloopBackOff on a Kubernetes Pod is an incident that occurs when a container running in a Kubernetes Pod repeatedly crashes immediately after starting up. This could be caused by a variety of factors, such as a bad deployment, an issue with the Pod's configuration, or a problem with the underlying infrastructure. This incident can cause disruption to the application or service running in the Pod and needs to be resolved as quickly as possible.

### Parameters
```shell
# Environment Variables
export POD_NAME="PLACEHOLDER"
export NAMESPACE="PLACEHOLDER"
export CONTAINER_NAME="PLACEHOLDER"
export DEBUG_IMAGE="PLACEHOLDER"
export CONTAINER_IMAGE_URL="PLACEHOLDER"

```

## Debug

### Find the Pod that is in CrashloopBackOff state
```shell
kubectl get pods -n ${NAMESPACE} | grep ${POD_NAME}
```

### View the logs of the container that is crashing
```shell
kubectl logs ${POD_NAME} -c ${CONTAINER_NAME} -n ${NAMESPACE}
```

### Get a detailed description of the Pod to check for any issues
```shell
kubectl describe pod ${POD_NAME} -n ${NAMESPACE}
```

### Check the events related to the Pod to see if there are any warnings or errors
```shell
kubectl get events -n ${NAMESPACE} | grep ${POD_NAME}
```

### Check the resource usage of the Pod and its containers to ensure they are not exceeding limits
```shell
kubectl top pod ${POD_NAME} -n ${NAMESPACE}
kubectl top container ${CONTAINER_NAME} -p ${POD_NAME} -n ${NAMESPACE}
```
### Image Issues: The container image could be misconfigured or corrupted, causing the container to fail repeatedly immediately after starting up.
```shell

#!/bin/bash

# Set the namespace and pod name variables
NAMESPACE=${NAMESPACE}
POD_NAME=${POD_NAME}

# Get the logs for the container
kubectl logs $POD_NAME -n $NAMESPACE

# Check the status of the container
kubectl describe pod $POD_NAME -n $NAMESPACE | grep -A 2 "Containers\{1\}"

# Check the status of the image
kubectl describe pod $POD_NAME -n $NAMESPACE | grep -A 2 "Image\{1\}"

# Check the image pull policy
kubectl describe pod $POD_NAME -n $NAMESPACE | grep -A 2 "Image Pull Policy\{1\}"

# Check the container image for issues
kubectl run -it --rm debug --image=${DEBUG_IMAGE} --restart=Never -- /bin/sh -c "wget -O - ${CONTAINER_IMAGE_URL} | tar xzvf - -C /dev/null"

```

---

## Repair
---

### Verify Pod configuration: Verify the configuration of the Pod to ensure that it is correctly configured and that there are no errors in the configuration files.
```shell

#!/bin/bash

# Set the namespace and Pod name
NAMESPACE=${NAMESPACE}
POD=${POD_NAME}

# Get the Pod configuration
kubectl get pod $POD -n $NAMESPACE -o yaml > pod.yaml

# Check for errors in the configuration file
if ! kubectl apply -f pod.yaml --dry-run > /dev/null; then
    echo "Pod configuration file contains errors"
    exit 1
fi

echo "Pod configuration is valid"

```

### Restart the Pod: Restarting the Pod may help resolve the issue if the cause of the incident is a transient issue.
```shell

#!/bin/bash

# Set the name of the Kubernetes Pod
POD_NAME=${POD_NAME}

# Set the namespace of the Kubernetes Pod
NAMESPACE=${NAMESPACE}

# Restart the Kubernetes Pod
kubectl delete pod $POD_NAME -n $NAMESPACE

bash script.sh

```
---


CrashloopBackOff on a Kubernetes Pod is an incident that occurs when a container running in a Kubernetes Pod repeatedly crashes immediately after starting up. This could be caused by a variety of factors, such as a bad deployment, an issue with the Pod's configuration, or a problem with the underlying infrastructure. This incident can cause disruption to the application or service running in the Pod and needs to be resolved as quickly as possible.


This incident type occurs when the number of 4xx errors on Traffic Server is at an anomalous level, higher than usual. It could be an indicator of an issue with the server or an increase in traffic that is causing errors. It requires investigation and resolution to ensure the server is functioning correctly and not impacting users.


High 4xx Errors on Traffic Server

This incident type involves an issue with Kubernetes deployments where the expected number of pods to run is not matching the actual number of pods running. This can lead to alerts being triggered and potential disruptions in the system.


Kubernetes pods not starting - Deployment issue

This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

A Kubernetes Replicaset Incomplete incident typically occurs when a specific number of pods that should be running are not, due to reasons such as failed pod initialization, unavailability of resources in the cluster, or inability to pull the image. This incident is usually triggered when the difference between desired and running pods is greater than zero, and it can be detected through monitoring tools like Datadog.


Kubernetes Replicaset Incomplete

Kubernetes Pods Pending incident indicates that one or more pods in a Kubernetes cluster are not running as expected and are in a pending state. This can happen due to various reasons such as resource constraints, scheduling issues, or network problems. This incident can impact the availability and performance of the application running on the Kubernetes cluster. It requires immediate attention to diagnose and resolve the underlying issue to ensure the pods are running as expected.


Kubernetes Pods Pending

```shell
# Environment Variables
export POD_NAME="PLACEHOLDER"
export NAMESPACE="PLACEHOLDER"
export CONTAINER_NAME="PLACEHOLDER"
export DEBUG_IMAGE="PLACEHOLDER"
export CONTAINER_IMAGE_URL="PLACEHOLDER"

```


### Find the Pod that is in CrashloopBackOff state

```shell
kubectl get pods -n ${NAMESPACE} | grep ${POD_NAME}
```

### View the logs of the container that is crashing

```shell
kubectl logs ${POD_NAME} -c ${CONTAINER_NAME} -n ${NAMESPACE}
```

### Get a detailed description of the Pod to check for any issues

```shell
kubectl describe pod ${POD_NAME} -n ${NAMESPACE}
```

### Check the events related to the Pod to see if there are any warnings or errors

```shell
kubectl get events -n ${NAMESPACE} | grep ${POD_NAME}
```

### Check the resource usage of the Pod and its containers to ensure they are not exceeding limits

```shell
kubectl top pod ${POD_NAME} -n ${NAMESPACE}
kubectl top container ${CONTAINER_NAME} -p ${POD_NAME} -n ${NAMESPACE}
```

### Image Issues: The container image could be misconfigured or corrupted, causing the container to fail repeatedly immediately after starting up.

```shell

#!/bin/bash

# Set the namespace and pod name variables
NAMESPACE=${NAMESPACE}
POD_NAME=${POD_NAME}

# Get the logs for the container
kubectl logs $POD_NAME -n $NAMESPACE

# Check the status of the container
kubectl describe pod $POD_NAME -n $NAMESPACE | grep -A 2 "Containers\{1\}"

# Check the status of the image
kubectl describe pod $POD_NAME -n $NAMESPACE | grep -A 2 "Image\{1\}"

# Check the image pull policy
kubectl describe pod $POD_NAME -n $NAMESPACE | grep -A 2 "Image Pull Policy\{1\}"

# Check the container image for issues
kubectl run -it --rm debug --image=${DEBUG_IMAGE} --restart=Never -- /bin/sh -c "wget -O - ${CONTAINER_IMAGE_URL} | tar xzvf - -C /dev/null"

```


### Verify Pod configuration: Verify the configuration of the Pod to ensure that it is correctly configured and that there are no errors in the configuration files.

```shell

#!/bin/bash

# Set the namespace and Pod name
NAMESPACE=${NAMESPACE}
POD=${POD_NAME}

# Get the Pod configuration
kubectl get pod $POD -n $NAMESPACE -o yaml > pod.yaml

# Check for errors in the configuration file
if ! kubectl apply -f pod.yaml --dry-run > /dev/null; then
    echo "Pod configuration file contains errors"
    exit 1
fi

echo "Pod configuration is valid"

```

### Restart the Pod: Restarting the Pod may help resolve the issue if the cause of the incident is a transient issue.

```shell

#!/bin/bash

# Set the name of the Kubernetes Pod
POD_NAME=${POD_NAME}

# Set the namespace of the Kubernetes Pod
NAMESPACE=${NAMESPACE}

# Restart the Kubernetes Pod
kubectl delete pod $POD_NAME -n $NAMESPACE

bash script.sh

```


CrashloopBackOff on Kubernetes Pod

Overview

Parameters

Debug

Find the Pod that is in CrashloopBackOff state

View the logs of the container that is crashing

Get a detailed description of the Pod to check for any issues

Check the resource usage of the Pod and its containers to ensure they are not exceeding limits

Image Issues: The container image could be misconfigured or corrupted, causing the container to fail repeatedly immediately after starting up.

Repair

Verify Pod configuration: Verify the configuration of the Pod to ensure that it is correctly configured and that there are no errors in the configuration files.

Restart the Pod: Restarting the Pod may help resolve the issue if the cause of the incident is a transient issue.

Learn more

Related Runbooks

High 4xx Errors on Traffic Server

Kubernetes pods not starting - Deployment issue

Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Replicaset Incomplete

Support