---
id: 74032705-f658-484e-b088-323640407246
---

# Kubernetes Daemonset Incomplete
---

This incident type occurs when a Kubernetes DaemonSet fails to run the same pod across all nodes due to reasons such as missing images, initialization failures, or a lack of resources in the cluster. The incident is triggered when the desired number of pods - the running pods is greater than zero. This incident can impact the availability of the services running on Kubernetes and requires immediate attention.

### Parameters
```shell
# Environment Variables

export NAMESPACE="PLACEHOLDER"

export DAEMONSET_NAME="PLACEHOLDER"

export APP_NAME="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

```

## Debug

### Get the list of DaemonSets in the cluster
```shell
kubectl get daemonsets --all-namespaces
```

### Describe a specific DaemonSet to check its status
```shell
kubectl describe daemonset ${DAEMONSET_NAME} --namespace ${NAMESPACE}
```

### Check the status of the pods created by the DaemonSet
```shell
kubectl get pods --selector=app=${APP_NAME} --namespace ${NAMESPACE}
```

### Check the logs of a specific pod to see if there are any errors
```shell
kubectl logs ${POD_NAME} --namespace ${NAMESPACE}
```

### Check the events related to the DaemonSet to see if there were any issues during the creation of the pods
```shell
kubectl get events --field-selector involvedObject.kind=DaemonSet --namespace ${NAMESPACE}
```

### Check the status of the nodes in the cluster to see if there are any issues
```shell
kubectl get nodes
```

### Check the resource usage of the nodes to see if there are any resource constraints
```shell
kubectl top nodes
```


### The pod spec in the DaemonSet configuration could be incorrect or incomplete.
```shell

#!/bin/bash
# Specify the DaemonSet name and namespace

DAEMONSET_NAME=${DAEMONSET_NAME}

NAMESPACE=${NAMESPACE}

# Get the pod template from the DaemonSet's YAML file

POD_TEMPLATE=$(kubectl get daemonset/${DAEMONSET_NAME} -n ${NAMESPACE} -o jsonpath='{.spec.template}')

# Check if the pod template is empty or incomplete

if [[ -z ${POD_TEMPLATE} ]]; then

    echo "ERROR: The pod template in the DaemonSet ${DAEMONSET_NAME} is empty."

    exit 1

fi

# Check if any required fields are missing from the pod template

if [[ -z $(echo ${POD_TEMPLATE} | jq '.spec.containers[].image') ]]; then

    echo "ERROR: The image field is missing from the pod template in the DaemonSet ${DAEMONSET_NAME}."

    exit 1

fi

# Check if any optional fields are missing from the pod template

if [[ -z $(echo ${POD_TEMPLATE} | jq '.spec.containers[].resources') ]]; then

    echo "WARNING: The resources field is missing from the pod template in the DaemonSet ${DAEMONSET_NAME}."

fi

echo "SUCCESS: The pod template in the DaemonSet ${DAEMONSET_NAME} is correct."

exit 0

```
## Repair
### Review the Kubernetes resource limits for the daemonset and adjust as necessary.
```shell
#!/bin/bash

# Set the namespace and daemonset name

NAMESPACE=${NAMESPACE}
DAEMONSET=${DAEMONSET_NAME}
CONTAINER_NAME="PLACEHOLDER"

# Check the current resource limits

kubectl describe daemonset $DAEMONSET -n $NAMESPACE | grep -A 3 Resources
 
# Set the desired CPU and memory limits
cpu_limit="PLACEHOLDER"
memory_limit="PLACEHOLDER"

# Adjust the resource limits
kubectl patch deployments/$deployment_name -p \
  '{"spec":{"template":{"spec":{"containers":[{"name":"'$CONTAINER_NAME'","resources":{"limits":{"cpu":"'$cpu_limit'","memory":"'$memory_limit'"}}}]}}}}'


# Check the new resource limits

kubectl describe daemonset $DAEMONSET -n $NAMESPACE | grep -A 3 Resources


```


This incident type occurs when a Kubernetes DaemonSet fails to run the same pod across all nodes due to reasons such as missing images, initialization failures, or a lack of resources in the cluster. The incident is triggered when the desired number of pods - the running pods is greater than zero. This incident can impact the availability of the services running on Kubernetes and requires immediate attention.


The Vault cluster health incident is related to the health of a Vault cluster instance. This incident type is triggered when the cluster instance is not healthy and requires attention to ensure it is functioning properly. The incident typically involves evaluating the current state of the cluster instance, diagnosing the issue, and taking corrective action to restore the health of the instance.


Vault cluster health incident on kubernetes

This incident type involves an issue with Kubernetes deployments where the expected number of pods to run is not matching the actual number of pods running. This can lead to alerts being triggered and potential disruptions in the system.


Kubernetes pods not starting - Deployment issue

This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

A Kubernetes Replicaset Incomplete incident typically occurs when a specific number of pods that should be running are not, due to reasons such as failed pod initialization, unavailability of resources in the cluster, or inability to pull the image. This incident is usually triggered when the difference between desired and running pods is greater than zero, and it can be detected through monitoring tools like Datadog.


Kubernetes Replicaset Incomplete

Kubernetes Pods Pending incident indicates that one or more pods in a Kubernetes cluster are not running as expected and are in a pending state. This can happen due to various reasons such as resource constraints, scheduling issues, or network problems. This incident can impact the availability and performance of the application running on the Kubernetes cluster. It requires immediate attention to diagnose and resolve the underlying issue to ensure the pods are running as expected.


Kubernetes Pods Pending

```shell
# Environment Variables

export NAMESPACE="PLACEHOLDER"

export DAEMONSET_NAME="PLACEHOLDER"

export APP_NAME="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

```


### Get the list of DaemonSets in the cluster

```shell
kubectl get daemonsets --all-namespaces
```

### Describe a specific DaemonSet to check its status

```shell
kubectl describe daemonset ${DAEMONSET_NAME} --namespace ${NAMESPACE}
```

### Check the status of the pods created by the DaemonSet

```shell
kubectl get pods --selector=app=${APP_NAME} --namespace ${NAMESPACE}
```

### Check the logs of a specific pod to see if there are any errors

```shell
kubectl logs ${POD_NAME} --namespace ${NAMESPACE}
```

### Check the events related to the DaemonSet to see if there were any issues during the creation of the pods

```shell
kubectl get events --field-selector involvedObject.kind=DaemonSet --namespace ${NAMESPACE}
```

### Check the status of the nodes in the cluster to see if there are any issues

```shell
kubectl get nodes
```

### Check the resource usage of the nodes to see if there are any resource constraints

```shell
kubectl top nodes
```

### The pod spec in the DaemonSet configuration could be incorrect or incomplete.

```shell

#!/bin/bash
# Specify the DaemonSet name and namespace

DAEMONSET_NAME=${DAEMONSET_NAME}

NAMESPACE=${NAMESPACE}

# Get the pod template from the DaemonSet's YAML file

POD_TEMPLATE=$(kubectl get daemonset/${DAEMONSET_NAME} -n ${NAMESPACE} -o jsonpath='{.spec.template}')

# Check if the pod template is empty or incomplete

if [[ -z ${POD_TEMPLATE} ]]; then

    echo "ERROR: The pod template in the DaemonSet ${DAEMONSET_NAME} is empty."

    exit 1

fi

# Check if any required fields are missing from the pod template

if [[ -z $(echo ${POD_TEMPLATE} | jq '.spec.containers[].image') ]]; then

    echo "ERROR: The image field is missing from the pod template in the DaemonSet ${DAEMONSET_NAME}."

    exit 1

fi

# Check if any optional fields are missing from the pod template

if [[ -z $(echo ${POD_TEMPLATE} | jq '.spec.containers[].resources') ]]; then

    echo "WARNING: The resources field is missing from the pod template in the DaemonSet ${DAEMONSET_NAME}."

fi

echo "SUCCESS: The pod template in the DaemonSet ${DAEMONSET_NAME} is correct."

exit 0

```


### Review the Kubernetes resource limits for the daemonset and adjust as necessary.

```shell
#!/bin/bash

# Set the namespace and daemonset name

NAMESPACE=${NAMESPACE}
DAEMONSET=${DAEMONSET_NAME}
CONTAINER_NAME="PLACEHOLDER"

# Check the current resource limits

kubectl describe daemonset $DAEMONSET -n $NAMESPACE | grep -A 3 Resources
 
# Set the desired CPU and memory limits
cpu_limit="PLACEHOLDER"
memory_limit="PLACEHOLDER"

# Adjust the resource limits
kubectl patch deployments/$deployment_name -p \
  '{"spec":{"template":{"spec":{"containers":[{"name":"'$CONTAINER_NAME'","resources":{"limits":{"cpu":"'$cpu_limit'","memory":"'$memory_limit'"}}}]}}}}'


# Check the new resource limits

kubectl describe daemonset $DAEMONSET -n $NAMESPACE | grep -A 3 Resources


```


Kubernetes Daemonset Incomplete

Overview

Parameters

Debug

Get the list of DaemonSets in the cluster

Describe a specific DaemonSet to check its status

Check the status of the pods created by the DaemonSet

Check the logs of a specific pod to see if there are any errors

Check the status of the nodes in the cluster to see if there are any issues

Check the resource usage of the nodes to see if there are any resource constraints

The pod spec in the DaemonSet configuration could be incorrect or incomplete.

Repair

Review the Kubernetes resource limits for the daemonset and adjust as necessary.

Learn more

Related Runbooks

Vault cluster health incident on kubernetes

Kubernetes pods not starting - Deployment issue

Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Replicaset Incomplete

Support