---
id: 04f04906-4421-4948-aaae-2786973297bf
---
# Dead Kafka Node Detection.
---

Dead Kafka Node Detection incident is a type of incident that occurs when a node of the Kafka cluster goes down or fails. Kafka is an open-source distributed event streaming platform used by software engineers to build real-time streaming data pipelines and applications. When a node of the Kafka cluster goes down, it can cause data loss, message duplication, and various other issues. Therefore, it's essential to detect and resolve dead Kafka nodes as quickly as possible to minimize the impact on the system's performance and data integrity.

### Parameters
```shell
export KAFKA_NODE_IP="PLACEHOLDER"

export ZOOKEEPER_NODE_IP="PLACEHOLDER"

export KAFKA_DATA_DIRECTORY="PLACEHOLDER"

export KAFKA_LOG_FILE="PLACEHOLDER"

export HOSTNAME_OF_THE_NEW_KAFKA_NODE="PLACEHOLDER"

export HOSTNAME_OF_THE_FAILED_KAFKA_NODE="PLACEHOLDER"

export OLD_NODE_HOSTNAME="PLACEHOLDER"
```

## Debug

### Check if the Kafka service is running
```shell
systemctl status kafka.service
```

### Check if the Kafka process is running
```shell
ps aux | grep kafka
```

### Check if the Kafka node is reachable from other nodes in the cluster
```shell
ping ${KAFKA_NODE_IP}
```

### Check if the Kafka node is able to communicate with ZooKeeper
```shell
echo ruok | nc ${ZOOKEEPER_NODE_IP} 2181
```

### Check if the Kafka node is able to access the Kafka data directory
```shell
ls -l ${KAFKA_DATA_DIRECTORY}
```

### Check if there are any Kafka logs indicating a potential issue with the node
```shell
tail -n 100 ${KAFKA_LOG_FILE}
```

## Repair

### Replace the failed node: If the node is found to be defective, it should be replaced with a new one.
```shell


#!/bin/bash



# Set variables

OLD_NODE=${HOSTNAME_OF_THE_FAILED_KAFKA_NODE}

NEW_NODE=${HOSTNAME_OF_THE_NEW_KAFKA_NODE}



# Stop Kafka service on the failed node

ssh $OLD_NODE sudo service kafka stop



# Remove the old Kafka node from the cluster

ssh $OLD_NODE sudo kafka-server-stop.sh

ssh $OLD_NODE sudo rm -rf /tmp/kafka-logs

ssh $OLD_NODE sudo rm -rf /var/lib/kafka



# Install Kafka on the new node

ssh $NEW_NODE sudo apt-get update

ssh $NEW_NODE sudo apt-get install kafka



# Configure the new Kafka node

ssh $NEW_NODE sudo sed -i 's/${OLD_NODE_HOSTNAME}/$OLD_NODE/' /etc/kafka/server.properties

ssh $NEW_NODE sudo chown -R kafka:kafka /var/log/kafka

ssh $NEW_NODE sudo chown -R kafka:kafka /var/lib/kafka



# Start Kafka service on the new node

ssh $NEW_NODE sudo service kafka start


```


Dead Kafka Node Detection incident is a type of incident that occurs when a node of the Kafka cluster goes down or fails. Kafka is an open-source distributed event streaming platform used by software engineers to build real-time streaming data pipelines and applications. When a node of the Kafka cluster goes down, it can cause data loss, message duplication, and various other issues. Therefore, it's essential to detect and resolve dead Kafka nodes as quickly as possible to minimize the impact on the system's performance and data integrity.


This incident type involves an alert triggered by the Datadog monitoring agent indicating that it has stopped running in a particular Kubernetes cluster. This can lead to a variety of issues such as loss of visibility into cluster performance, potential security risks, and other problems that can impact operations. The incident requires immediate attention and resolution to ensure the Datadog agent is running properly and that the cluster is operating as expected.


Datadog Agent Not Running in Kubernetes Cluster.

This incident type refers to a situation where there is an issue with the communication between Kafka brokers due to network partitioning. A network partition is a situation where a network is split into two or more parts, and communication between them is disrupted. This issue can lead to data inconsistency, loss of messages, delayed message delivery, and other related problems. As a result, it is important to quickly detect and resolve these incidents to prevent any downstream impact on the system.


Network Partitioning Between Kafka Brokers.

This type of incident occurs when a node in the Kafka cluster is not being fully utilized. Kafka is a distributed streaming platform, and each node in the cluster is responsible for processing and storing a portion of the data. If a node is underutilized, it means that it is not processing as much data as it should be, which can lead to performance issues and potential data loss. Therefore, it is important to identify and address underutilized nodes in a Kafka cluster to ensure optimal performance and data reliability.


Kafka node underutilization check incident.

This incident type refers to an issue where the network bandwidth on one or more Kafka nodes becomes fully saturated, causing the node(s) to experience performance degradation or even failure. This can result in message delivery delays or loss, and can also impact other nodes that rely on the affected node(s) for message replication. The saturation can be caused by a variety of factors, such as increased message traffic, misconfiguration, or hardware issues.


Network Saturation Check for Kafka Nodes

This incident type refers to a situation where an unauthorized user gains access to a Kafka cluster due to misconfigured Access Control Lists (ACLs) in Kafka. This can result in potential data breaches or other security issues, as the unauthorized user may be able to read, write or modify data within the Kafka cluster. The incident typically requires immediate attention and remediation to prevent further unauthorized access.


Kafka Unauthorized Access

```shell
export KAFKA_NODE_IP="PLACEHOLDER"

export ZOOKEEPER_NODE_IP="PLACEHOLDER"

export KAFKA_DATA_DIRECTORY="PLACEHOLDER"

export KAFKA_LOG_FILE="PLACEHOLDER"

export HOSTNAME_OF_THE_NEW_KAFKA_NODE="PLACEHOLDER"

export HOSTNAME_OF_THE_FAILED_KAFKA_NODE="PLACEHOLDER"

export OLD_NODE_HOSTNAME="PLACEHOLDER"
```


### Check if the Kafka service is running

```shell
systemctl status kafka.service
```

### Check if the Kafka process is running

```shell
ps aux | grep kafka
```

### Check if the Kafka node is reachable from other nodes in the cluster

```shell
ping ${KAFKA_NODE_IP}
```

### Check if the Kafka node is able to communicate with ZooKeeper

```shell
echo ruok | nc ${ZOOKEEPER_NODE_IP} 2181
```

### Check if the Kafka node is able to access the Kafka data directory

```shell
ls -l ${KAFKA_DATA_DIRECTORY}
```

### Check if there are any Kafka logs indicating a potential issue with the node

```shell
tail -n 100 ${KAFKA_LOG_FILE}
```


### Replace the failed node: If the node is found to be defective, it should be replaced with a new one.

```shell


#!/bin/bash



# Set variables

OLD_NODE=${HOSTNAME_OF_THE_FAILED_KAFKA_NODE}

NEW_NODE=${HOSTNAME_OF_THE_NEW_KAFKA_NODE}



# Stop Kafka service on the failed node

ssh $OLD_NODE sudo service kafka stop



# Remove the old Kafka node from the cluster

ssh $OLD_NODE sudo kafka-server-stop.sh

ssh $OLD_NODE sudo rm -rf /tmp/kafka-logs

ssh $OLD_NODE sudo rm -rf /var/lib/kafka



# Install Kafka on the new node

ssh $NEW_NODE sudo apt-get update

ssh $NEW_NODE sudo apt-get install kafka



# Configure the new Kafka node

ssh $NEW_NODE sudo sed -i 's/${OLD_NODE_HOSTNAME}/$OLD_NODE/' /etc/kafka/server.properties

ssh $NEW_NODE sudo chown -R kafka:kafka /var/log/kafka

ssh $NEW_NODE sudo chown -R kafka:kafka /var/lib/kafka



# Start Kafka service on the new node

ssh $NEW_NODE sudo service kafka start


```


Dead Kafka Node Detection.

Overview

Parameters

Debug

Check if the Kafka service is running

Check if the Kafka process is running

Check if the Kafka node is reachable from other nodes in the cluster

Check if the Kafka node is able to communicate with ZooKeeper

Check if the Kafka node is able to access the Kafka data directory

Check if there are any Kafka logs indicating a potential issue with the node

Repair

Replace the failed node: If the node is found to be defective, it should be replaced with a new one.

Learn more

Related Runbooks

Datadog Agent Not Running in Kubernetes Cluster.

Network Partitioning Between Kafka Brokers.

Kafka node underutilization check incident.

Network Saturation Check for Kafka Nodes

Support