---
id: 23f9abc3-87d6-4dcd-b334-9417c52479b9
---
# Container Absent Incident
---

When a container that should be running is not present or has suddenly ceased, it is referred to as a "container absent" issue in a containerized environment like Kubernetes or Docker. This may result in unexpected behavior, application unavailability, or service interruptions. Common causes include container crashes due to errors, resource constraints, dependencies failures, misconfigurations in deployment files, image pull failures, health check issues, and manual terminations.

### Parameters
```shell
export IP_ADDRESS="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"

export SERVICE_NAME="PLACEHOLDER"
```

## Debug

### 1. Check if the instance is reachable
```shell
ping ${IP_ADDRESS}
```

### 2. Check the status of the container
```shell
docker ps -a | grep ${CONTAINER_NAME}
```

### 3. Check the logs of the container to see if there are any errors or issues
```shell
docker logs ${CONTAINER_NAME}
```

### 4. Check the status of the service
```shell
systemctl status ${SERVICE_NAME}
```

### 5. Check the system logs for any relevant error messages
```shell
journalctl -u ${SERVICE_NAME}
```

### 7. Check the CPU and memory usage of the container
```shell
docker stats ${CONTAINER_NAME}
```

### 8. Check the container's configuration and environment variables
```shell
docker inspect ${CONTAINER_NAME}
```

### 9. Check the status of the host machine's resources
```shell
top
```

## Repair

### Restart the containers: Try to restart the absent containers using the system's management tools or command-line interface. This can help bring them back online and restore their normal functioning.
```shell


#!/bin/bash



# Replace ${CONTAINER_NAME} with the name of the container that needs to be restarted

CONTAINER_NAME="${CONTAINER_NAME}"



# Get the container ID of the specified container

CONTAINER_ID=$(docker ps -q -f name=$CONTAINER_NAME)



# Check if the container is already running

if [ ! -z "$CONTAINER_ID" ]; then

  echo "Container $CONTAINER_NAME is already running."

else

  # Start the container

  docker start $CONTAINER_NAME

  echo "Container $CONTAINER_NAME has been restarted."

fi


```



When a container that should be running is not present or has suddenly ceased, it is referred to as a "container absent" issue in a containerized environment like Kubernetes or Docker. This may result in unexpected behavior, application unavailability, or service interruptions. Common causes include container crashes due to errors, resource constraints, dependencies failures, misconfigurations in deployment files, image pull failures, health check issues, and manual terminations.


This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

A Kubernetes Replicaset Incomplete incident typically occurs when a specific number of pods that should be running are not, due to reasons such as failed pod initialization, unavailability of resources in the cluster, or inability to pull the image. This incident is usually triggered when the difference between desired and running pods is greater than zero, and it can be detected through monitoring tools like Datadog.


Kubernetes Replicaset Incomplete

Kubernetes Pods Pending incident indicates that one or more pods in a Kubernetes cluster are not running as expected and are in a pending state. This can happen due to various reasons such as resource constraints, scheduling issues, or network problems. This incident can impact the availability and performance of the application running on the Kubernetes cluster. It requires immediate attention to diagnose and resolve the underlying issue to ensure the pods are running as expected.


Kubernetes Pods Pending

A Kubernetes Pod Restarting Monitoring incident is triggered when a pod running on a Kubernetes cluster restarts multiple times within a certain time frame. This incident type is usually used to detect issues with the application or infrastructure running on the cluster, and can be caused by various factors such as resource constraints, misconfigurations, or bugs in the application code. The incident is typically resolved by identifying and addressing the underlying cause of the pod restarts.


Kubernetes Pod Restarting Monitoring

A Kubernetes Pod ImagePullBackOff incident occurs when a pod in a Kubernetes cluster is unable to pull its container image. This can happen due to various reasons, such as incorrect image path or tag, or misconfigured image pulling credentials. This incident can cause the pod to fail to start and impact the availability of the application running in the pod. It requires investigation and resolution to ensure the pod can pull its container image and restart successfully.


Kubernetes Pod ImagePullBackOff Incident

```shell
export IP_ADDRESS="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"

export SERVICE_NAME="PLACEHOLDER"
```


### 1. Check if the instance is reachable

```shell
ping ${IP_ADDRESS}
```

### 2. Check the status of the container

```shell
docker ps -a | grep ${CONTAINER_NAME}
```

### 3. Check the logs of the container to see if there are any errors or issues

```shell
docker logs ${CONTAINER_NAME}
```

### 4. Check the status of the service

```shell
systemctl status ${SERVICE_NAME}
```

### 5. Check the system logs for any relevant error messages

```shell
journalctl -u ${SERVICE_NAME}
```

### 7. Check the CPU and memory usage of the container

```shell
docker stats ${CONTAINER_NAME}
```

### 8. Check the container's configuration and environment variables

```shell
docker inspect ${CONTAINER_NAME}
```

### 9. Check the status of the host machine's resources

```shell
top
```


### Restart the containers: Try to restart the absent containers using the system's management tools or command-line interface. This can help bring them back online and restore their normal functioning.

```shell


#!/bin/bash



# Replace ${CONTAINER_NAME} with the name of the container that needs to be restarted

CONTAINER_NAME="${CONTAINER_NAME}"



# Get the container ID of the specified container

CONTAINER_ID=$(docker ps -q -f name=$CONTAINER_NAME)



# Check if the container is already running

if [ ! -z "$CONTAINER_ID" ]; then

  echo "Container $CONTAINER_NAME is already running."

else

  # Start the container

  docker start $CONTAINER_NAME

  echo "Container $CONTAINER_NAME has been restarted."

fi


```


Container Absent Incident

Overview

Parameters

Debug

1. Check if the instance is reachable

2. Check the status of the container

3. Check the logs of the container to see if there are any errors or issues

4. Check the status of the service

5. Check the system logs for any relevant error messages

7. Check the CPU and memory usage of the container

8. Check the container's configuration and environment variables

9. Check the status of the host machine's resources

Repair

Restart the containers: Try to restart the absent containers using the system's management tools or command-line interface. This can help bring them back online and restore their normal functioning.

Learn more

Related Runbooks

Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Replicaset Incomplete

Kubernetes Pods Pending

Kubernetes Pod Restarting Monitoring

Support