---
id: f989c9ba-1a8a-4816-a19a-c421d2aec2ec
---

# Datadog Agent Not Running in Kubernetes Cluster.
---

This incident type involves an alert triggered by the Datadog monitoring agent indicating that it has stopped running in a particular Kubernetes cluster. This can lead to a variety of issues such as loss of visibility into cluster performance, potential security risks, and other problems that can impact operations. The incident requires immediate attention and resolution to ensure the Datadog agent is running properly and that the cluster is operating as expected.

### Parameters
```shell
# Environment Variables

export DATADOG_AGENT_POD_NAME="PLACEHOLDER"

export DATADOG_AGENT_NAMESPACE="PLACEHOLDER"

export SELECTOR_TO_IDENTIFY_AGENT_POD="PLACEHOLDER"

export DATADOG_AGENT_CONTAINER="PLACEHOLDER"

export DATADOG_AGENT_PORT="PLACEHOLDER"

export DATADOG_AGENT_HOST="PLACEHOLDER"
```

## Debug

### Check if the Datadog agent is deployed in the Kubernetes cluster
```shell
kubectl get daemonset datadog-agent
```

### Check the logs of the Datadog agent pod
```shell
kubectl logs ${DATADOG_AGENT_POD_NAME}
```

### Check if the Datadog agent is running on all nodes in the Kubernetes cluster
```shell
kubectl exec ${DATADOG_AGENT_POD_NAME} -- agent status
```

### Check if the Datadog agent is able to connect to the Datadog backend
```shell
kubectl exec ${DATADOG_AGENT_POD_NAME} -- agent configcheck
```

### Check the status of the Kubernetes nodes in the cluster
```shell
kubectl get nodes
```

### Check the status of the Kubernetes pods in the cluster
```shell
kubectl get pods --all-namespaces
```

### Check the status of the Kubernetes services in the cluster
```shell
kubectl get services --all-namespaces
```
### There may have been an issue with the network or communication between the Datadog agent and the Kubernetes cluster.
```shell
bash

#!/bin/bash

# Set variables for Datadog agent and Kubernetes cluster

DATADOG_AGENT=${DATADOG_AGENT_POD_NAME}

# Check if Datadog agent is running in the Kubernetes cluster

if kubectl exec -it $(kubectl get pods -n ${DATADOG_AGENT_NAMESPACE} | grep $DATADOG_AGENT | awk '{print $1}') -n kube-system -- pgrep $DATADOG_AGENT >/dev/null; then

  echo "$DATADOG_AGENT is running in the current kubernetes cluster"

else

  echo "$DATADOG_AGENT is not running in the current kubernetes cluster"

fi
# Check if there are any network or communication issues between Kubernetes cluster and Datadog agent

if kubectl exec -it $(kubectl get pods -n ${DATADOG_AGENT_NAMESPACE} | grep $DATADOG_AGENT | awk '{print $1}') -n kube-system -- curl -fsSL -o /dev/null https://app.datadoghq.com/api/v1/validate >/dev/null; then

  echo "Communication between the current kubernetes cluster and Datadog agent is working"

else

  echo "There may be a network or communication issue between the current kubernetes cluster and Datadog agent"

fi


```

## Repair

### Verify the connection between the Kubernetes cluster and the Datadog agent.
```shell
bash

#!/bin/bash

# Set the host and port for the Datadog agent.

DD_HOST=${DATADOG_AGENT_HOST}

DD_PORT=${DATADOG_AGENT_PORT}

# Set the namespace and pod name for the Datadog agent.

DD_NAMESPACE=${DATADOG_AGENT_NAMESPACE}

DD_POD=${DATADOG_AGENT_POD_NAME}

# Verify that the connection between the Kubernetes cluster and the Datadog agent is working.

kubectl exec $DD_POD -n $DD_NAMESPACE -- sh -c "echo 'statsd.ping:true' | nc -w 1 $DD_HOST $DD_PORT"

# Check the exit code of the previous command to see if the connection was successful.

if [ $? -eq 0 ]; then

  echo "Connection to Datadog agent successful."

else

  echo "Connection to Datadog agent failed."

  exit 1

fi

```
### Perform a rolling restart of the Datadog agent DaemonSet.
```shell

#!/bin/bash

# rolling restart of the Datadog agent
kubectl rollout restart  daemonset/datadog-agent -n datadog

```

This incident type involves an alert triggered by the Datadog monitoring agent indicating that it has stopped running in a particular Kubernetes cluster. This can lead to a variety of issues such as loss of visibility into cluster performance, potential security risks, and other problems that can impact operations. The incident requires immediate attention and resolution to ensure the Datadog agent is running properly and that the cluster is operating as expected.


This incident type is characterized by the detection of unauthorized access to the Kubernetes API server. This unauthorized access potentially enables attackers to manipulate cluster resources. Such incidents can lead to serious consequences, as unauthorized access allows attackers to make changes to the Kubernetes cluster, which may compromise the security and integrity of the entire system. Therefore, it is crucial to promptly detect and address such incidents to ensure the security and proper functioning of the Kubernetes cluster.


Unauthorized Access to Kubernetes API Server Detected

This incident type involves an issue with Kubernetes deployments where the expected number of pods to run is not matching the actual number of pods running. This can lead to alerts being triggered and potential disruptions in the system.


Kubernetes pods not starting - Deployment issue

This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

A Kubernetes Pod Restarting Monitoring incident is triggered when a pod running on a Kubernetes cluster restarts multiple times within a certain time frame. This incident type is usually used to detect issues with the application or infrastructure running on the cluster, and can be caused by various factors such as resource constraints, misconfigurations, or bugs in the application code. The incident is typically resolved by identifying and addressing the underlying cause of the pod restarts.


Kubernetes Pod Restarting Monitoring

This incident type is related to an alert triggered when the available memory on a Kubernetes node drops below a certain threshold (in this case, 90%). The alert is designed to monitor the memory usage percentage and notify the relevant teams when the threshold is breached. This incident type is critical as it helps ensure that Kubernetes clusters are operating within acceptable memory usage levels and that potential issues are identified and resolved promptly.


Kubernetes Memory Usage Alert

```shell
# Environment Variables

export DATADOG_AGENT_POD_NAME="PLACEHOLDER"

export DATADOG_AGENT_NAMESPACE="PLACEHOLDER"

export SELECTOR_TO_IDENTIFY_AGENT_POD="PLACEHOLDER"

export DATADOG_AGENT_CONTAINER="PLACEHOLDER"

export DATADOG_AGENT_PORT="PLACEHOLDER"

export DATADOG_AGENT_HOST="PLACEHOLDER"
```


### Check if the Datadog agent is deployed in the Kubernetes cluster

```shell
kubectl get daemonset datadog-agent
```

### Check the logs of the Datadog agent pod

```shell
kubectl logs ${DATADOG_AGENT_POD_NAME}
```

### Check if the Datadog agent is running on all nodes in the Kubernetes cluster

```shell
kubectl exec ${DATADOG_AGENT_POD_NAME} -- agent status
```

### Check if the Datadog agent is able to connect to the Datadog backend

```shell
kubectl exec ${DATADOG_AGENT_POD_NAME} -- agent configcheck
```

### Check the status of the Kubernetes nodes in the cluster

```shell
kubectl get nodes
```

### Check the status of the Kubernetes pods in the cluster

```shell
kubectl get pods --all-namespaces
```

### Check the status of the Kubernetes services in the cluster

```shell
kubectl get services --all-namespaces
```

### There may have been an issue with the network or communication between the Datadog agent and the Kubernetes cluster.

```shell
bash

#!/bin/bash

# Set variables for Datadog agent and Kubernetes cluster

DATADOG_AGENT=${DATADOG_AGENT_POD_NAME}

# Check if Datadog agent is running in the Kubernetes cluster

if kubectl exec -it $(kubectl get pods -n ${DATADOG_AGENT_NAMESPACE} | grep $DATADOG_AGENT | awk '{print $1}') -n kube-system -- pgrep $DATADOG_AGENT >/dev/null; then

  echo "$DATADOG_AGENT is running in the current kubernetes cluster"

else

  echo "$DATADOG_AGENT is not running in the current kubernetes cluster"

fi
# Check if there are any network or communication issues between Kubernetes cluster and Datadog agent

if kubectl exec -it $(kubectl get pods -n ${DATADOG_AGENT_NAMESPACE} | grep $DATADOG_AGENT | awk '{print $1}') -n kube-system -- curl -fsSL -o /dev/null https://app.datadoghq.com/api/v1/validate >/dev/null; then

  echo "Communication between the current kubernetes cluster and Datadog agent is working"

else

  echo "There may be a network or communication issue between the current kubernetes cluster and Datadog agent"

fi


```


### Verify the connection between the Kubernetes cluster and the Datadog agent.

```shell
bash

#!/bin/bash

# Set the host and port for the Datadog agent.

DD_HOST=${DATADOG_AGENT_HOST}

DD_PORT=${DATADOG_AGENT_PORT}

# Set the namespace and pod name for the Datadog agent.

DD_NAMESPACE=${DATADOG_AGENT_NAMESPACE}

DD_POD=${DATADOG_AGENT_POD_NAME}

# Verify that the connection between the Kubernetes cluster and the Datadog agent is working.

kubectl exec $DD_POD -n $DD_NAMESPACE -- sh -c "echo 'statsd.ping:true' | nc -w 1 $DD_HOST $DD_PORT"

# Check the exit code of the previous command to see if the connection was successful.

if [ $? -eq 0 ]; then

  echo "Connection to Datadog agent successful."

else

  echo "Connection to Datadog agent failed."

  exit 1

fi

```

### Perform a rolling restart of the Datadog agent DaemonSet.

```shell

#!/bin/bash

# rolling restart of the Datadog agent
kubectl rollout restart  daemonset/datadog-agent -n datadog

```


Datadog Agent Not Running in Kubernetes Cluster.

Overview

Parameters

Debug

Check if the Datadog agent is deployed in the Kubernetes cluster

Check the logs of the Datadog agent pod

Check if the Datadog agent is running on all nodes in the Kubernetes cluster

Check if the Datadog agent is able to connect to the Datadog backend

Check the status of the Kubernetes nodes in the cluster

Check the status of the Kubernetes pods in the cluster

Check the status of the Kubernetes services in the cluster

There may have been an issue with the network or communication between the Datadog agent and the Kubernetes cluster.

Repair

Verify the connection between the Kubernetes cluster and the Datadog agent.

Perform a rolling restart of the Datadog agent DaemonSet.

Learn more

Related Runbooks

Unauthorized Access to Kubernetes API Server Detected

Kubernetes pods not starting - Deployment issue

Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Pod Restarting Monitoring

Support