---
id: 71d6dac2-61cc-4792-a249-9ae6fca174d1
---

# Unused topic partition in Confluent Platform
---

This incident type indicates that there is an unused topic partition in the Confluent Platform. This can lead to reduced efficiency and increased Broker overhead. It is recommended to delete unused partitions to avoid these issues. This incident may be triggered by a monitoring tool or an alert and can be resolved by verifying if the unused partition is intentional and deleting it if required.

### Parameters
```shell
# Environment Variables

export ZOOKEEPER_PORT="PLACEHOLDER"

export TOPIC_NAME="PLACEHOLDER"

export CONFLUENT_PATH="PLACEHOLDER"

export ZOOKEEPER_HOST="PLACEHOLDER"

export PARTITION_NUMBER="PLACEHOLDER"

export KAFKA_BROKER_ADDRESS="PLACEHOLDER"

export PATH_TO_KAFKA_FOLDER="PLACEHOLDER"

export PARTITION_NAME="PLACEHOLDER"
```

## Debug

### Check the status of the Kafka brokers
```shell
systemctl status kafka.service
```

### Check if the topic exists
```shell
${CONFLUENT_PATH}/bin/kafka-topics --zookeeper ${ZOOKEEPER_HOST}:${ZOOKEEPER_PORT} --describe --topic ${TOPIC_NAME}
```

### List all the Kafka topics
```shell
${CONFLUENT_PATH}/bin/kafka-topics --zookeeper ${ZOOKEEPER_HOST}:${ZOOKEEPER_PORT} --list
```

### Check the disk usage of the Kafka brokers
```shell
df -h
```

### Check the available memory
```shell
free -h
```

### Check the Kafka logs
```shell
tail -f ${CONFLUENT_PATH}/logs/kafka.log
```

### Check the Zookeeper logs
```shell
tail -f ${CONFLUENT_PATH}/logs/zookeeper.log
```

### A new topic partition was created and then left unused by mistake.
```shell


#!/bin/bash



# Parameters

KAFKA_BROKER=${KAFKA_BROKER_ADDRESS}

TOPIC_NAME=${TOPIC_NAME}

PARTITION_NUMBER=${PARTITION_NUMBER}



# Check if partition is empty

partition_size=$(kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list $KAFKA_BROKER --topic $TOPIC_NAME --partitions $PARTITION_NUMBER --time -1 | awk -F ":" '{sum += $3} END {print sum}')

if [ "$partition_size" -eq "0" ]; then

  echo "Partition $PARTITION_NUMBER in topic $TOPIC_NAME is empty."

else

  echo "Partition $PARTITION_NUMBER in topic $TOPIC_NAME is not empty."

fi



# Check if any messages have been written to the partition

latest_offset=$(kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list $KAFKA_BROKER --topic $TOPIC_NAME --partitions $PARTITION_NUMBER --time -1 | awk -F ":" '{print $3}')

earliest_offset=$(kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list $KAFKA_BROKER --topic $TOPIC_NAME --partitions $PARTITION_NUMBER --time -2 | awk -F ":" '{print $3}')

if [ "$latest_offset" -eq "$earliest_offset" ]; then

  echo "No messages have been written to partition $PARTITION_NUMBER in topic $TOPIC_NAME."

else

  echo "Messages have been written to partition $PARTITION_NUMBER in topic $TOPIC_NAME."

fi


```

### A topic partition was purposely left unused, but the overhead caused by it was not considered.
```shell


#!/bin/bash



# Set the topic name and partition number

topic=${TOPIC_NAME}

partition=${PARTITION_NUMBER}



# Get the byte in rate for the topic partition

bytes_in=$(curl -s -X GET "http://localhost:9090/api/v1/query?query=avg(last_5m):avg:confluent.kafka.server.topic.bytes_in_per_sec.rate{$topic-partition=$partition}" | jq -r '.data.result[].value[1]')



# Get the byte out rate for the topic partition

bytes_out=$(curl -s -X GET "http://localhost:9090/api/v1/query?query=avg(last_5m):avg:confluent.kafka.server.topic.bytes_out_per_sec.rate{$topic-partition=$partition}" | jq -r '.data.result[].value[1]')



# Check if both byte in and byte out rates are 0

if (( $(echo "$bytes_in + $bytes_out == 0" | bc -l) )); then

  echo "The partition appears to be unused."

else

  echo "The partition is being used."

fi


```

## Repair

### Verify the partition is actually unused.
```shell


#!/bin/bash



# Set the required environment variables

export KAFKA_HOME=${PATH_TO_KAFKA_FOLDER}

export TOPIC_NAME=${TOPIC_NAME}

export PARTITION_NUMBER=${PARTITION_NUMBER}



# Get the partition details using Kafka command-line tools

partition_details=$($KAFKA_HOME/bin/kafka-topics.sh --describe --topic $TOPIC_NAME --zookeeper localhost:2181 | grep -w "Partition:$PARTITION_NUMBER")



# Check if the partition is unused

if [[ $partition_details == *"leader: none"* ]] && [[ $partition_details == *"isr: []"* ]]; then

    echo "Partition $PARTITION_NUMBER is unused."

else

    echo "Partition $PARTITION_NUMBER is being used."

fi


```

### If the partition is unused, delete it to reduce Broker overhead.
```shell


#!/bin/bash



# Replace ${PARTITION_NAME} with the name of the unused partition

partition_name=${PARTITION_NAME}



# Check if the partition is unused

if kafka-topics.sh --zookeeper ${ZOOKEEPER_HOST}:${ZOOKEEPER_PORT} --describe --topic ${TOPIC_NAME} | grep -q "${PARTITION_NAME}\t[[:digit:]]\t"; then

    # Delete the partition

    kafka-topics.sh --zookeeper ${ZOOKEEPER_HOST}:${ZOOKEEPER_PORT} --delete --topic ${TOPIC_NAME} --partitions ${PARTITION_NAME}

    echo "Unused partition ${PARTITION_NAME} has been deleted."

else

    echo "Partition ${PARTITION_NAME} is not unused."

fi


```

### Verify the partition is actually unused.
```shell


#!/bin/bash



# Set the required environment variables

export KAFKA_HOME=${PATH_TO_KAFKA_FOLDER}

export TOPIC_NAME=${TOPIC_NAME}

export PARTITION_NUMBER=${PARTITION_NUMBER}



# Get the partition details using Kafka command-line tools

partition_details=$($KAFKA_HOME/bin/kafka-topics.sh --describe --topic $TOPIC_NAME --zookeeper localhost:2181 | grep -w "Partition:$PARTITION_NUMBER")



# Check if the partition is unused

if [[ $partition_details == *"leader: none"* ]] && [[ $partition_details == *"isr: []"* ]]; then

    echo "Partition $PARTITION_NUMBER is unused."

else

    echo "Partition $PARTITION_NUMBER is being used."

fi


```

This incident type indicates that there is an unused topic partition in the Confluent Platform. This can lead to reduced efficiency and increased Broker overhead. It is recommended to delete unused partitions to avoid these issues. This incident may be triggered by a monitoring tool or an alert and can be resolved by verifying if the unused partition is intentional and deleting it if required.


This incident type refers to the detection of an unused topic partition in Kafka Platform, a popular data streaming platform. Unused partitions can increase broker overhead and reduce efficiency, so it is recommended to delete them. The incident is triggered by a test notification or a monitoring query that checks the value of relevant metrics. The incident is resolved once the partition is deleted or confirmed to be intentional.


Unused Topic Partition in Kafka

This incident type refers to a situation where there is an issue with the communication between Kafka brokers due to network partitioning. A network partition is a situation where a network is split into two or more parts, and communication between them is disrupted. This issue can lead to data inconsistency, loss of messages, delayed message delivery, and other related problems. As a result, it is important to quickly detect and resolve these incidents to prevent any downstream impact on the system.


Parameters

This incident type refers to an issue where the retention size for a Kafka topic is set too low. The retention size is the amount of time that Kafka retains messages in a topic before they are deleted. When the retention size is set too low, messages may be deleted before they can be consumed or processed, which can result in data loss or other issues. This can be caused by misconfiguration or changes in the data ingestion rate. It is important to monitor retention sizes and adjust them as necessary to prevent data loss and ensure smooth data processing.


Low Retention Size for Kafka Topic.

This type of incident occurs when the offset of a Kafka topic decreases unexpectedly. In Kafka, an offset is a unique identifier that represents the position of a message within a partition. When the offset decreases, it means that some messages have been lost or deleted, which can cause data inconsistencies or other issues within the system. This incident type is critical for Kafka-based systems as it affects data integrity and can lead to data loss.


Kafka topic offset decreased incident.

```shell
# Environment Variables

export ZOOKEEPER_PORT="PLACEHOLDER"

export TOPIC_NAME="PLACEHOLDER"

export CONFLUENT_PATH="PLACEHOLDER"

export ZOOKEEPER_HOST="PLACEHOLDER"

export PARTITION_NUMBER="PLACEHOLDER"

export KAFKA_BROKER_ADDRESS="PLACEHOLDER"

export PATH_TO_KAFKA_FOLDER="PLACEHOLDER"

export PARTITION_NAME="PLACEHOLDER"
```


### Check the status of the Kafka brokers

```shell
systemctl status kafka.service
```

### Check if the topic exists

```shell
${CONFLUENT_PATH}/bin/kafka-topics --zookeeper ${ZOOKEEPER_HOST}:${ZOOKEEPER_PORT} --describe --topic ${TOPIC_NAME}
```

### List all the Kafka topics

```shell
${CONFLUENT_PATH}/bin/kafka-topics --zookeeper ${ZOOKEEPER_HOST}:${ZOOKEEPER_PORT} --list
```

### Check the disk usage of the Kafka brokers

```shell
df -h
```

### Check the available memory

```shell
free -h
```

### Check the Kafka logs

```shell
tail -f ${CONFLUENT_PATH}/logs/kafka.log
```

### Check the Zookeeper logs

```shell
tail -f ${CONFLUENT_PATH}/logs/zookeeper.log
```

### A new topic partition was created and then left unused by mistake.

```shell


#!/bin/bash



# Parameters

KAFKA_BROKER=${KAFKA_BROKER_ADDRESS}

TOPIC_NAME=${TOPIC_NAME}

PARTITION_NUMBER=${PARTITION_NUMBER}



# Check if partition is empty

partition_size=$(kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list $KAFKA_BROKER --topic $TOPIC_NAME --partitions $PARTITION_NUMBER --time -1 | awk -F ":" '{sum += $3} END {print sum}')

if [ "$partition_size" -eq "0" ]; then

  echo "Partition $PARTITION_NUMBER in topic $TOPIC_NAME is empty."

else

  echo "Partition $PARTITION_NUMBER in topic $TOPIC_NAME is not empty."

fi



# Check if any messages have been written to the partition

latest_offset=$(kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list $KAFKA_BROKER --topic $TOPIC_NAME --partitions $PARTITION_NUMBER --time -1 | awk -F ":" '{print $3}')

earliest_offset=$(kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list $KAFKA_BROKER --topic $TOPIC_NAME --partitions $PARTITION_NUMBER --time -2 | awk -F ":" '{print $3}')

if [ "$latest_offset" -eq "$earliest_offset" ]; then

  echo "No messages have been written to partition $PARTITION_NUMBER in topic $TOPIC_NAME."

else

  echo "Messages have been written to partition $PARTITION_NUMBER in topic $TOPIC_NAME."

fi


```

### A topic partition was purposely left unused, but the overhead caused by it was not considered.

```shell


#!/bin/bash



# Set the topic name and partition number

topic=${TOPIC_NAME}

partition=${PARTITION_NUMBER}



# Get the byte in rate for the topic partition

bytes_in=$(curl -s -X GET "http://localhost:9090/api/v1/query?query=avg(last_5m):avg:confluent.kafka.server.topic.bytes_in_per_sec.rate{$topic-partition=$partition}" | jq -r '.data.result[].value[1]')



# Get the byte out rate for the topic partition

bytes_out=$(curl -s -X GET "http://localhost:9090/api/v1/query?query=avg(last_5m):avg:confluent.kafka.server.topic.bytes_out_per_sec.rate{$topic-partition=$partition}" | jq -r '.data.result[].value[1]')



# Check if both byte in and byte out rates are 0

if (( $(echo "$bytes_in + $bytes_out == 0" | bc -l) )); then

  echo "The partition appears to be unused."

else

  echo "The partition is being used."

fi


```


### Verify the partition is actually unused.

```shell


#!/bin/bash



# Set the required environment variables

export KAFKA_HOME=${PATH_TO_KAFKA_FOLDER}

export TOPIC_NAME=${TOPIC_NAME}

export PARTITION_NUMBER=${PARTITION_NUMBER}



# Get the partition details using Kafka command-line tools

partition_details=$($KAFKA_HOME/bin/kafka-topics.sh --describe --topic $TOPIC_NAME --zookeeper localhost:2181 | grep -w "Partition:$PARTITION_NUMBER")



# Check if the partition is unused

if [[ $partition_details == *"leader: none"* ]] && [[ $partition_details == *"isr: []"* ]]; then

    echo "Partition $PARTITION_NUMBER is unused."

else

    echo "Partition $PARTITION_NUMBER is being used."

fi


```

### If the partition is unused, delete it to reduce Broker overhead.

```shell


#!/bin/bash



# Replace ${PARTITION_NAME} with the name of the unused partition

partition_name=${PARTITION_NAME}



# Check if the partition is unused

if kafka-topics.sh --zookeeper ${ZOOKEEPER_HOST}:${ZOOKEEPER_PORT} --describe --topic ${TOPIC_NAME} | grep -q "${PARTITION_NAME}\t[[:digit:]]\t"; then

    # Delete the partition

    kafka-topics.sh --zookeeper ${ZOOKEEPER_HOST}:${ZOOKEEPER_PORT} --delete --topic ${TOPIC_NAME} --partitions ${PARTITION_NAME}

    echo "Unused partition ${PARTITION_NAME} has been deleted."

else

    echo "Partition ${PARTITION_NAME} is not unused."

fi


```

### Verify the partition is actually unused.

```shell


#!/bin/bash



# Set the required environment variables

export KAFKA_HOME=${PATH_TO_KAFKA_FOLDER}

export TOPIC_NAME=${TOPIC_NAME}

export PARTITION_NUMBER=${PARTITION_NUMBER}



# Get the partition details using Kafka command-line tools

partition_details=$($KAFKA_HOME/bin/kafka-topics.sh --describe --topic $TOPIC_NAME --zookeeper localhost:2181 | grep -w "Partition:$PARTITION_NUMBER")



# Check if the partition is unused

if [[ $partition_details == *"leader: none"* ]] && [[ $partition_details == *"isr: []"* ]]; then

    echo "Partition $PARTITION_NUMBER is unused."

else

    echo "Partition $PARTITION_NUMBER is being used."

fi


```


Unused topic partition in Confluent Platform

Overview

Parameters

Debug

Check the status of the Kafka brokers

Check if the topic exists

List all the Kafka topics

Check the disk usage of the Kafka brokers

Check the available memory

Check the Kafka logs

Check the Zookeeper logs

A new topic partition was created and then left unused by mistake.

A topic partition was purposely left unused, but the overhead caused by it was not considered.

Repair

Verify the partition is actually unused.

If the partition is unused, delete it to reduce Broker overhead.

Verify the partition is actually unused.

Learn more

Related Runbooks

Unused Topic Partition in Kafka

Network Partitioning Between Kafka Brokers.

Parameters

Low Retention Size for Kafka Topic.

Support