---
id: 0004059b-9930-4ed5-9949-8388fcf8fe38
---
# Kafka unclean leader election
---

A Kafka unclean leader election is a situation where multiple brokers in a Kafka cluster claim to be the leader for the same partition. This can occur due to misconfiguration or network issues. The result is that data may be lost or duplicated, which can cause downstream issues for consumers. It is important to quickly identify and resolve this type of incident to ensure the proper functioning of the Kafka cluster.

### Parameters
```shell
export KAFKA_BOOTSTRAP_SERVER="PLACEHOLDER"

export TOPIC_NAME="PLACEHOLDER"

export ZOOKEEPER_CONNECTION_STRING="PLACEHOLDER"

export LIST_OF_BROKER_IPS="PLACEHOLDER"

export KAFKA_CONFIGURATION_FILE="PLACEHOLDER"

export PARTITION_NUMBER="PLACEHOLDER"

export NEW_REPLICA_COUNT="PLACEHOLDER"

```

## Debug

### List all topics in the Kafka cluster
```shell
kafka-topics --bootstrap-server ${KAFKA_BOOTSTRAP_SERVER} --list
```

### Describe a specific topic, including the number of partitions and replica assignment
```shell
kafka-topics --bootstrap-server ${KAFKA_BOOTSTRAP_SERVER} --describe --topic ${TOPIC_NAME}
```

### Check the status of the Kafka brokers
```shell
kafka-broker-api-versions --bootstrap-server ${KAFKA_BOOTSTRAP_SERVER}
```

### Check the status of the ZooKeeper ensemble that is used by the Kafka cluster for coordination
```shell
zookeeper-shell ${ZOOKEEPER_CONNECTION_STRING} ls /brokers/ids
```

### Check the Kafka configuration file  to ensure that the `unclean.leader.election.enable`
```shell
grep "unclean.leader.election.enable" ${KAFKA_CONFIGURATION_FILE}
```

### Check the Kafka logs for any errors or warning messages related to leader elections
```shell
tail -f /var/log/kafka/server.log | grep "unclean leader election"
```

### Increase the number of replicas for partition.
```shell
bash

#!/bin/bash



# Define variables

topic=${TOPIC_NAME}

partition=${PARTITION_NUMBER}

replicas=${NEW_REPLICA_COUNT}



# Increase the number of replicas for the partition

echo "Increasing the number of replicas for partition $partition to $replicas..."

kafka-topics --zookeeper ${ZOOKEEPER_CONNECT_STRING} --alter --topic $topic --partitions $partition --replication-factor $replicas



# Verify that the replica count has been increased

echo "Verifying that the replica count has been increased..."

kafka-topics --zookeeper ${ZOOKEEPER_CONNECT_STRING} --describe --topic $topic | grep -w "Partition:$partition" | grep -w "Replicas:$replicas"

if [ $? -eq 0 ]

then

    echo "Replica count for partition $partition has been increased to $replicas."

else

    echo "Failed to increase replica count for partition $partition."

fi


```

## Repair

### Check the Kafka configuration file on each broker to ensure that the `unclean.leader.election.enable` property is set to `false`. If it is set to `true`, change it to `false` and restart the broker.
```shell


#!/bin/bash



# Set the path to the Kafka configuration file

KAFKA_CONFIG=${KAFKA_CONFIGURATION_FILE}



# Set the value that `unclean.leader.election.enable` should be set to

NEW_VALUE=false



# Loop through each broker and update its configuration file

for BROKER in ${LIST_OF_BROKER_IPS}

do

    ssh $BROKER "sed -i 's/unclean.leader.election.enable=.*/unclean.leader.election.enable=$NEW_VALUE/' $KAFKA_CONFIG"

    ssh $BROKER "systemctl restart kafka"

done


```


A Kafka unclean leader election is a situation where multiple brokers in a Kafka cluster claim to be the leader for the same partition. This can occur due to misconfiguration or network issues. The result is that data may be lost or duplicated, which can cause downstream issues for consumers. It is important to quickly identify and resolve this type of incident to ensure the proper functioning of the Kafka cluster.


This incident type refers to a situation where there is an issue with the communication between Kafka brokers due to network partitioning. A network partition is a situation where a network is split into two or more parts, and communication between them is disrupted. This issue can lead to data inconsistency, loss of messages, delayed message delivery, and other related problems. As a result, it is important to quickly detect and resolve these incidents to prevent any downstream impact on the system.


Network Partitioning Between Kafka Brokers.

This type of incident occurs when the offset of a Kafka topic decreases unexpectedly. In Kafka, an offset is a unique identifier that represents the position of a message within a partition. When the offset decreases, it means that some messages have been lost or deleted, which can cause data inconsistencies or other issues within the system. This incident type is critical for Kafka-based systems as it affects data integrity and can lead to data loss.


Kafka topic offset decreased incident.

This incident type refers to when a Kafka node runs out of storage space to store data. This can cause issues with data processing and storage, potentially leading to data loss and system downtime.


Kafka node insufficient storage check

This incident type refers to a situation where an unauthorized user gains access to a Kafka cluster due to misconfigured Access Control Lists (ACLs) in Kafka. This can result in potential data breaches or other security issues, as the unauthorized user may be able to read, write or modify data within the Kafka cluster. The incident typically requires immediate attention and remediation to prevent further unauthorized access.


Kafka Unauthorized Access

This incident type relates to an issue with time synchronization that affects the proper functioning of a Kafka cluster. As Kafka relies on accurate time synchronization to maintain data consistency, any failure in this process can cause serious issues. The incident might cause data loss or duplication, service disruptions, or other problems that can impact the overall performance of the system. It requires immediate attention from the engineering team to investigate and resolve the root cause of the issue.


Kafka Time synchronization issue impacting Kafka cluster.

```shell
export KAFKA_BOOTSTRAP_SERVER="PLACEHOLDER"

export TOPIC_NAME="PLACEHOLDER"

export ZOOKEEPER_CONNECTION_STRING="PLACEHOLDER"

export LIST_OF_BROKER_IPS="PLACEHOLDER"

export KAFKA_CONFIGURATION_FILE="PLACEHOLDER"

export PARTITION_NUMBER="PLACEHOLDER"

export NEW_REPLICA_COUNT="PLACEHOLDER"

```


### List all topics in the Kafka cluster

```shell
kafka-topics --bootstrap-server ${KAFKA_BOOTSTRAP_SERVER} --list
```

### Describe a specific topic, including the number of partitions and replica assignment

```shell
kafka-topics --bootstrap-server ${KAFKA_BOOTSTRAP_SERVER} --describe --topic ${TOPIC_NAME}
```

### Check the status of the Kafka brokers

```shell
kafka-broker-api-versions --bootstrap-server ${KAFKA_BOOTSTRAP_SERVER}
```

### Check the status of the ZooKeeper ensemble that is used by the Kafka cluster for coordination

```shell
zookeeper-shell ${ZOOKEEPER_CONNECTION_STRING} ls /brokers/ids
```

### Check the Kafka configuration file  to ensure that the `unclean.leader.election.enable`

```shell
grep "unclean.leader.election.enable" ${KAFKA_CONFIGURATION_FILE}
```

### Check the Kafka logs for any errors or warning messages related to leader elections

```shell
tail -f /var/log/kafka/server.log | grep "unclean leader election"
```

### Increase the number of replicas for partition.

```shell
bash

#!/bin/bash



# Define variables

topic=${TOPIC_NAME}

partition=${PARTITION_NUMBER}

replicas=${NEW_REPLICA_COUNT}



# Increase the number of replicas for the partition

echo "Increasing the number of replicas for partition $partition to $replicas..."

kafka-topics --zookeeper ${ZOOKEEPER_CONNECT_STRING} --alter --topic $topic --partitions $partition --replication-factor $replicas



# Verify that the replica count has been increased

echo "Verifying that the replica count has been increased..."

kafka-topics --zookeeper ${ZOOKEEPER_CONNECT_STRING} --describe --topic $topic | grep -w "Partition:$partition" | grep -w "Replicas:$replicas"

if [ $? -eq 0 ]

then

    echo "Replica count for partition $partition has been increased to $replicas."

else

    echo "Failed to increase replica count for partition $partition."

fi


```


### Check the Kafka configuration file on each broker to ensure that the `unclean.leader.election.enable` property is set to `false`. If it is set to `true`, change it to `false` and restart the broker.

```shell


#!/bin/bash



# Set the path to the Kafka configuration file

KAFKA_CONFIG=${KAFKA_CONFIGURATION_FILE}



# Set the value that `unclean.leader.election.enable` should be set to

NEW_VALUE=false



# Loop through each broker and update its configuration file

for BROKER in ${LIST_OF_BROKER_IPS}

do

    ssh $BROKER "sed -i 's/unclean.leader.election.enable=.*/unclean.leader.election.enable=$NEW_VALUE/' $KAFKA_CONFIG"

    ssh $BROKER "systemctl restart kafka"

done


```


Kafka unclean leader election

Overview

Parameters

Debug

List all topics in the Kafka cluster

Describe a specific topic, including the number of partitions and replica assignment

Check the status of the Kafka brokers

Check the status of the ZooKeeper ensemble that is used by the Kafka cluster for coordination

Check the Kafka configuration file to ensure that the `unclean.leader.election.enable`

Increase the number of replicas for partition.

Repair

Check the Kafka configuration file on each broker to ensure that the `unclean.leader.election.enable` property is set to `false`. If it is set to `true`, change it to `false` and restart the broker.

Learn more

Related Runbooks

Network Partitioning Between Kafka Brokers.

Kafka topic offset decreased incident.

Kafka node insufficient storage check

Kafka Unauthorized Access

Support

Kafka unclean leader election

Overview

Parameters

Debug

List all topics in the Kafka cluster

Describe a specific topic, including the number of partitions and replica assignment

Check the status of the Kafka brokers

Check the status of the ZooKeeper ensemble that is used by the Kafka cluster for coordination

Check the Kafka configuration file to ensure that the unclean.leader.election.enable

Check the Kafka logs for any errors or warning messages related to leader elections

Increase the number of replicas for partition.

Repair

Check the Kafka configuration file on each broker to ensure that the unclean.leader.election.enable property is set to false. If it is set to true, change it to false and restart the broker.

Learn more

Related Runbooks

Network Partitioning Between Kafka Brokers.

Kafka topic offset decreased incident.

Kafka node insufficient storage check

Kafka Unauthorized Access

Check the Kafka configuration file to ensure that the `unclean.leader.election.enable`

Check the Kafka configuration file on each broker to ensure that the `unclean.leader.election.enable` property is set to `false`. If it is set to `true`, change it to `false` and restart the broker.