---
id: 27ae8d5f-8364-44ef-9814-af7c44fb11d9
---
# High Partition Lag on Kafka Cluster
---

This incident type refers to an issue with a Kafka cluster where one or more partitions are experiencing high lag. Lag is the difference between the latest message produced to a partition and the latest message consumed from that partition. When a partition lags behind, it means that messages are not being consumed as quickly as they are being produced. This can lead to a backlog of messages and potential data loss if not addressed. The incident description suggests checking for hot partitions, which are partitions that receive a disproportionate amount of traffic compared to others. Identifying and resolving high partition lag is critical to ensure the stability and reliability of a Kafka cluster.

### Parameters
```shell
export ZOOKEEPER_PORT="PLACEHOLDER"

export ZOOKEEPER_HOSTNAME="PLACEHOLDER"

export CONSUMER_GROUP_NAME="PLACEHOLDER"

export KAFKA_BROKER_HOSTNAME="PLACEHOLDER"

export KAFKA_BROKER_PORT="PLACEHOLDER"

export TOPIC_NAME="PLACEHOLDER"

export PARTITION_NUMBER="PLACEHOLDER"

export KAFKA_BOOTSTRAP_SERVER="PLACEHOLDER"

export PATH_TO_KAFKA_HOME="PLACEHOLDER"
```

## Debug

### List all topics and their partition count
```shell
kafka-topics.sh --zookeeper ${ZOOKEEPER_HOSTNAME}:${ZOOKEEPER_PORT} --list | while read topic; do echo "$topic: $(kafka-topics.sh --zookeeper ${ZOOKEEPER_HOSTNAME}:${ZOOKEEPER_PORT} --describe --topic $topic | grep -c '^partition') partitions"; done
```

### List the partition lag for a specific consumer group
```shell
kafka-consumer-groups.sh --bootstrap-server ${KAFKA_BROKER_HOSTNAME}:${KAFKA_BROKER_PORT} --describe --group ${CONSUMER_GROUP_NAME}
```

### List the current offset for a specific partition
```shell
kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list ${KAFKA_BROKER_HOSTNAME}:${KAFKA_BROKER_PORT} --topic ${TOPIC_NAME} --partition ${PARTITION_NUMBER} --time -1
```

### List the end offset for a specific partition
```shell
kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list ${KAFKA_BROKER_HOSTNAME}:${KAFKA_BROKER_PORT} --topic ${TOPIC_NAME} --partition ${PARTITION_NUMBER} --time -2
```

### List the number of messages in a specific partition
```shell
kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list ${KAFKA_BROKER_HOSTNAME}:${KAFKA_BROKER_PORT} --topic ${TOPIC_NAME} --partition ${PARTITION_NUMBER} --time -1 | awk -F ":" '{print $3}' | tail -1
```

### List the number of messages consumed by a specific consumer group for a specific topic
```shell
kafka-consumer-groups.sh --bootstrap-server ${KAFKA_BROKER_HOSTNAME}:${KAFKA_BROKER_PORT} --describe --group ${CONSUMER_GROUP_NAME} | grep ${TOPIC_NAME} | awk '{print $5}'
```

## Repair

### Increase the number of consumer instances for the high-traffic partitions to reduce the lag. This can be done by adding more consumers to the consumer group or creating a new group.
```shell


#!/bin/bash



# Set variables

KAFKA_HOME=${PATH_TO_KAFKA_HOME}

TOPIC=${TOPIC_NAME}

GROUP=${CONSUMER_GROUP_NAME}



# Increase consumer instances

$KAFKA_HOME/bin/kafka-consumer-groups.sh --bootstrap-server ${KAFKA_BOOTSTRAP_SERVER} --topic $TOPIC --group $GROUP --describe | awk '{print $2}' | tail -n +3 | xargs -I {} $KAFKA_HOME/bin/kafka-consumer-groups.sh --bootstrap-server ${KAFKA_BOOTSTRAP_SERVER} --topic $TOPIC --group $GROUP --reset-offsets --to-latest --execute --partitions {}



echo "Increased consumer instances for $TOPIC in consumer group $GROUP"


```


This incident type refers to an issue with a Kafka cluster where one or more partitions are experiencing high lag. Lag is the difference between the latest message produced to a partition and the latest message consumed from that partition. When a partition lags behind, it means that messages are not being consumed as quickly as they are being produced. This can lead to a backlog of messages and potential data loss if not addressed. The incident description suggests checking for hot partitions, which are partitions that receive a disproportionate amount of traffic compared to others. Identifying and resolving high partition lag is critical to ensure the stability and reliability of a Kafka cluster.


This incident type refers to the detection of an unused topic partition in Kafka Platform, a popular data streaming platform. Unused partitions can increase broker overhead and reduce efficiency, so it is recommended to delete them. The incident is triggered by a test notification or a monitoring query that checks the value of relevant metrics. The incident is resolved once the partition is deleted or confirmed to be intentional.


Unused Topic Partition in Kafka

This incident type refers to a situation where there is an issue with the communication between Kafka brokers due to network partitioning. A network partition is a situation where a network is split into two or more parts, and communication between them is disrupted. This issue can lead to data inconsistency, loss of messages, delayed message delivery, and other related problems. As a result, it is important to quickly detect and resolve these incidents to prevent any downstream impact on the system.


Parameters

This incident type refers to an issue where the retention size for a Kafka topic is set too low. The retention size is the amount of time that Kafka retains messages in a topic before they are deleted. When the retention size is set too low, messages may be deleted before they can be consumed or processed, which can result in data loss or other issues. This can be caused by misconfiguration or changes in the data ingestion rate. It is important to monitor retention sizes and adjust them as necessary to prevent data loss and ensure smooth data processing.


Low Retention Size for Kafka Topic.

This type of incident occurs when the offset of a Kafka topic decreases unexpectedly. In Kafka, an offset is a unique identifier that represents the position of a message within a partition. When the offset decreases, it means that some messages have been lost or deleted, which can cause data inconsistencies or other issues within the system. This incident type is critical for Kafka-based systems as it affects data integrity and can lead to data loss.


Kafka topic offset decreased incident.

```shell
export ZOOKEEPER_PORT="PLACEHOLDER"

export ZOOKEEPER_HOSTNAME="PLACEHOLDER"

export CONSUMER_GROUP_NAME="PLACEHOLDER"

export KAFKA_BROKER_HOSTNAME="PLACEHOLDER"

export KAFKA_BROKER_PORT="PLACEHOLDER"

export TOPIC_NAME="PLACEHOLDER"

export PARTITION_NUMBER="PLACEHOLDER"

export KAFKA_BOOTSTRAP_SERVER="PLACEHOLDER"

export PATH_TO_KAFKA_HOME="PLACEHOLDER"
```


### List all topics and their partition count

```shell
kafka-topics.sh --zookeeper ${ZOOKEEPER_HOSTNAME}:${ZOOKEEPER_PORT} --list | while read topic; do echo "$topic: $(kafka-topics.sh --zookeeper ${ZOOKEEPER_HOSTNAME}:${ZOOKEEPER_PORT} --describe --topic $topic | grep -c '^partition') partitions"; done
```

### List the partition lag for a specific consumer group

```shell
kafka-consumer-groups.sh --bootstrap-server ${KAFKA_BROKER_HOSTNAME}:${KAFKA_BROKER_PORT} --describe --group ${CONSUMER_GROUP_NAME}
```

### List the current offset for a specific partition

```shell
kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list ${KAFKA_BROKER_HOSTNAME}:${KAFKA_BROKER_PORT} --topic ${TOPIC_NAME} --partition ${PARTITION_NUMBER} --time -1
```

### List the end offset for a specific partition

```shell
kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list ${KAFKA_BROKER_HOSTNAME}:${KAFKA_BROKER_PORT} --topic ${TOPIC_NAME} --partition ${PARTITION_NUMBER} --time -2
```

### List the number of messages in a specific partition

```shell
kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list ${KAFKA_BROKER_HOSTNAME}:${KAFKA_BROKER_PORT} --topic ${TOPIC_NAME} --partition ${PARTITION_NUMBER} --time -1 | awk -F ":" '{print $3}' | tail -1
```

### List the number of messages consumed by a specific consumer group for a specific topic

```shell
kafka-consumer-groups.sh --bootstrap-server ${KAFKA_BROKER_HOSTNAME}:${KAFKA_BROKER_PORT} --describe --group ${CONSUMER_GROUP_NAME} | grep ${TOPIC_NAME} | awk '{print $5}'
```


### Increase the number of consumer instances for the high-traffic partitions to reduce the lag. This can be done by adding more consumers to the consumer group or creating a new group.

```shell


#!/bin/bash



# Set variables

KAFKA_HOME=${PATH_TO_KAFKA_HOME}

TOPIC=${TOPIC_NAME}

GROUP=${CONSUMER_GROUP_NAME}



# Increase consumer instances

$KAFKA_HOME/bin/kafka-consumer-groups.sh --bootstrap-server ${KAFKA_BOOTSTRAP_SERVER} --topic $TOPIC --group $GROUP --describe | awk '{print $2}' | tail -n +3 | xargs -I {} $KAFKA_HOME/bin/kafka-consumer-groups.sh --bootstrap-server ${KAFKA_BOOTSTRAP_SERVER} --topic $TOPIC --group $GROUP --reset-offsets --to-latest --execute --partitions {}



echo "Increased consumer instances for $TOPIC in consumer group $GROUP"


```


High Partition Lag on Kafka Cluster

Overview

Parameters

Debug

List all topics and their partition count

List the partition lag for a specific consumer group

List the current offset for a specific partition

List the end offset for a specific partition

List the number of messages in a specific partition

List the number of messages consumed by a specific consumer group for a specific topic

Repair

Increase the number of consumer instances for the high-traffic partitions to reduce the lag. This can be done by adding more consumers to the consumer group or creating a new group.

Learn more

Related Runbooks

Unused Topic Partition in Kafka

Network Partitioning Between Kafka Brokers.

Parameters

Low Retention Size for Kafka Topic.

Support