---
id: 9d5a1b04-b658-4f4c-ba4f-3b5fa46f4f44
---

# Kafka high producer latency on broker.
---

This incident type typically involves a high producer latency on a broker in a Kafka environment. This can result in various potential impacts, such as client timeouts, delays in processing workload, or a lack of capacity leading to performance issues. Recommended actions typically include investigating the state of the broker, rebalancing topics, expanding capacity, or restarting/replacing the broker. Monitoring tools may trigger alerts to notify engineers of this potential issue.

### Parameters
```shell
# Environment Variables



export BROKER_IP="PLACEHOLDER"

export BROKER_PORT="PLACEHOLDER"

export TOPIC_NAME="PLACEHOLDER"

export ZOOKEEPER_URL="PLACEHOLDER"

export PATH_TO_KAFKA_CONFIG_FILE="PLACEHOLDER"

export NEW_BATCH_SIZE="PLACEHOLDER"

export NEW_NUMBER_OF_ACKNOWLEDGMENTS="PLACEHOLDER"

```

## Debug

### Check if Kafka is running on the broker
```shell
systemctl status kafka
```

### Check if there are any issues with the Kafka process
```shell
journalctl -u kafka  --since "2 hours ago"
```

### Check the disk usage on the broker
```shell
df -h
```

### Check the network connections on the broker
```shell
netstat -an | grep ${BROKER_IP}
```

### Check the number of connections to the broker
```shell
netstat -an | grep ${BROKER_IP} | wc -l
```

### Check the number of messages in the topic
```shell
kafka-run-class kafka.tools.GetOffsetShell --broker-list ${BROKER_IP}:${BROKER_PORT} --topic ${TOPIC_NAME}
```

### Check the number of partitions in the topic
```shell
kafka-topics --zookeeper ${ZOOKEEPER_IP}:${ZOOKEEPER_PORT} --describe --topic ${TOPIC_NAME}
```

### Check the resource utilization of the broker
```shell
top
```

### Check the Kafka log file for errors
```shell
tail -f /var/log/kafka/kafka.log
```

### High network latency can cause producer latency on Kafka brokers.
```shell
#!/bin/bash

# Get the IP address of the Kafka broker to test

KAFKA_BROKER_IP=${KAFKA_BROKER_IP}

# Ping the Kafka broker to check for network latency

ping -c 10 $KAFKA_BROKER_IP

# Check the producer latency of the Kafka broker

KAFKA_PRODUCER_LATENCY=$(kafka-producer-perf-test.sh --topic test --num-records 1000000 --record-size 100 --throughput -1 --producer-props bootstrap.servers=$KAFKA_BROKER_IP:9092 acks=1 | grep "Average" | awk '{print $NF}')

# Check if the producer latency is high

if (( $(echo "$KAFKA_PRODUCER_LATENCY > 50" | bc -l) )); then

  echo "High producer latency detected on Kafka broker."

else

  echo "No issues detected with Kafka broker producer latency."

fi

```

## Repair

### Tune Kafka configuration parameters, such as the producer batch size and the number of acknowledgments required, to optimize performance.
```shell

#!/bin/bash

# Set the Kafka configuration file path

KAFKA_CONFIG_FILE=${PATH_TO_KAFKA_CONFIG_FILE}

# Backup the original configuration file

cp $KAFKA_CONFIG_FILE $KAFKA_CONFIG_FILE.bak

# Update the producer batch size parameter

sed -i 's/producer.batch.size=.*/producer.batch.size=${NEW_BATCH_SIZE}/g' $KAFKA_CONFIG_FILE

# Update the number of acknowledgments required parameter

sed -i 's/acks=.*/acks=${NEW_NUMBER_OF_ACKNOWLEDGMENTS}/g' $KAFKA_CONFIG_FILE

# Restart the Kafka service

systemctl restart kafka.service

```



This incident type typically involves a high producer latency on a broker in a Kafka environment. This can result in various potential impacts, such as client timeouts, delays in processing workload, or a lack of capacity leading to performance issues. Recommended actions typically include investigating the state of the broker, rebalancing topics, expanding capacity, or restarting/replacing the broker. Monitoring tools may trigger alerts to notify engineers of this potential issue.


This incident type indicates that there is a high latency issue in the execution of a Spark job. Spark is a distributed computing framework that is used for processing large datasets. High latency in this context means that the time taken to execute the Spark job is significantly longer than expected or normal. This can result in delays in processing data and can impact the performance of the application or system that is utilizing Spark.


High Latency Incident for Spark Job Execution.

This incident type refers to a situation where there is an issue with the communication between Kafka brokers due to network partitioning. A network partition is a situation where a network is split into two or more parts, and communication between them is disrupted. This issue can lead to data inconsistency, loss of messages, delayed message delivery, and other related problems. As a result, it is important to quickly detect and resolve these incidents to prevent any downstream impact on the system.


Parameters

This type of incident occurs when the offset of a Kafka topic decreases unexpectedly. In Kafka, an offset is a unique identifier that represents the position of a message within a partition. When the offset decreases, it means that some messages have been lost or deleted, which can cause data inconsistencies or other issues within the system. This incident type is critical for Kafka-based systems as it affects data integrity and can lead to data loss.


Kafka topic offset decreased incident.

This incident type refers to an issue where the network bandwidth on one or more Kafka nodes becomes fully saturated, causing the node(s) to experience performance degradation or even failure. This can result in message delivery delays or loss, and can also impact other nodes that rely on the affected node(s) for message replication. The saturation can be caused by a variety of factors, such as increased message traffic, misconfiguration, or hardware issues.


### Check if Kafka is running on the broker

```shell
systemctl status kafka
```

### Check if there are any issues with the Kafka process

```shell
journalctl -u kafka  --since "2 hours ago"
```

### Check the disk usage on the broker

```shell
df -h
```

### Check the network connections on the broker

```shell
netstat -an | grep ${BROKER_IP}
```

### Check the number of connections to the broker

```shell
netstat -an | grep ${BROKER_IP} | wc -l
```

### Check the number of messages in the topic

```shell
kafka-run-class kafka.tools.GetOffsetShell --broker-list ${BROKER_IP}:${BROKER_PORT} --topic ${TOPIC_NAME}
```

### Check the number of partitions in the topic

```shell
kafka-topics --zookeeper ${ZOOKEEPER_IP}:${ZOOKEEPER_PORT} --describe --topic ${TOPIC_NAME}
```

### Check the resource utilization of the broker

```shell
top
```

### Check the Kafka log file for errors

```shell
tail -f /var/log/kafka/kafka.log
```

### High network latency can cause producer latency on Kafka brokers.

```shell
#!/bin/bash

# Get the IP address of the Kafka broker to test

KAFKA_BROKER_IP=${KAFKA_BROKER_IP}

# Ping the Kafka broker to check for network latency

ping -c 10 $KAFKA_BROKER_IP

# Check the producer latency of the Kafka broker

KAFKA_PRODUCER_LATENCY=$(kafka-producer-perf-test.sh --topic test --num-records 1000000 --record-size 100 --throughput -1 --producer-props bootstrap.servers=$KAFKA_BROKER_IP:9092 acks=1 | grep "Average" | awk '{print $NF}')

# Check if the producer latency is high

if (( $(echo "$KAFKA_PRODUCER_LATENCY > 50" | bc -l) )); then

  echo "High producer latency detected on Kafka broker."

else

  echo "No issues detected with Kafka broker producer latency."

fi

```


### Tune Kafka configuration parameters, such as the producer batch size and the number of acknowledgments required, to optimize performance.

```shell

#!/bin/bash

# Set the Kafka configuration file path

KAFKA_CONFIG_FILE=${PATH_TO_KAFKA_CONFIG_FILE}

# Backup the original configuration file

cp $KAFKA_CONFIG_FILE $KAFKA_CONFIG_FILE.bak

# Update the producer batch size parameter

sed -i 's/producer.batch.size=.*/producer.batch.size=${NEW_BATCH_SIZE}/g' $KAFKA_CONFIG_FILE

# Update the number of acknowledgments required parameter

sed -i 's/acks=.*/acks=${NEW_NUMBER_OF_ACKNOWLEDGMENTS}/g' $KAFKA_CONFIG_FILE

# Restart the Kafka service

systemctl restart kafka.service

```


Kafka high producer latency on broker.

Overview

Parameters

Debug

Check if Kafka is running on the broker

Check if there are any issues with the Kafka process

Check the disk usage on the broker

Check the network connections on the broker

Check the number of connections to the broker

Check the number of messages in the topic

Check the number of partitions in the topic

Check the resource utilization of the broker

Check the Kafka log file for errors

High network latency can cause producer latency on Kafka brokers.

Repair

Tune Kafka configuration parameters, such as the producer batch size and the number of acknowledgments required, to optimize performance.

Learn more

Related Runbooks

High Latency Incident for Spark Job Execution.

Network Partitioning Between Kafka Brokers.

Parameters

Kafka topic offset decreased incident.

Support