---
id: b78c3e39-4bfc-486e-b3f1-9174582aeb09
---
# High request rate on Kafka producer.
---

This incident type is related to a high request rate on a Kafka producer. This could be caused by a sudden increase in traffic or a malfunction in the producer system. This incident can cause delays in message processing and may lead to data loss or system failure if not addressed promptly. It is important to identify the root cause of this incident and take appropriate measures to prevent similar incidents in the future.

### Parameters
```shell
# Environment Variables

export KAFKA_PRODUCER="PLACEHOLDER"

export KAFKA_CONSUMER="PLACEHOLDER"

export KAFKA_BROKER="PLACEHOLDER"

export TOPIC_NAME="PLACEHOLDER"

export ZOOKEEPER_HOST_PORT="PLACEHOLDER"

```

## Debug

### Check the network connection between the Kafka producer and consumer
```shell
ping ${KAFKA_PRODUCER}

ping ${KAFKA_CONSUMER}
```

### Check if the Kafka producer is running
```shell
systemctl status ${KAFKA_PRODUCER}
```

### Check the Kafka producer logs for any errors or warnings
```shell
journalctl -u ${KAFKA_PRODUCER} -f
```

### Check the Kafka broker logs for any errors or warnings
```shell
journalctl -u ${KAFKA_BROKER} -f
```

### Check if the Kafka topic is created and has the expected number of partitions
```shell
kafka-topics --zookeeper ${ZOOKEEPER_HOST_PORT} --describe --topic ${TOPIC_NAME}
```

### Check if the Kafka consumer is running
```shell
systemctl status ${KAFKA_CONSUMER}
```

### Check the Kafka consumer logs for any errors or warnings
```shell
journalctl -u ${KAFKA_CONSUMER} -f
```

### Check the CPU and memory usage of the Kafka producer and consumer processes
```shell
top 
```

### Check the network traffic between the Kafka producer and consumer
```shell
tcpdump -i eth01 port 9092
```

### A sudden surge in demand for the service or application that is using the Kafka producer, leading to a high request rate.
```shell
#!/bin/bash

# Set parameters

REQUEST_THRESHOLD="PLACEHOLDER"

TIME_PERIOD="PLACEHOLDER"

# Get request rate

request_rate=$(kafka-console-consumer.sh --bootstrap-server ${KAFKA_BROKER} --topic $TOPIC_NAME --from-beginning --timeout-ms 1000 | wc -l)

# Check if request rate exceeds threshold

if [ $request_rate -gt $REQUEST_THRESHOLD ]

then

    echo "High request rate detected: $request_rate requests in the last $TIME_PERIOD seconds."

# Check for system overload

    cpu_usage=$(top -bn1 | awk '/Cpu/ {print $2}' | sed 's/%//')

    memory_usage=$(free | awk '/Mem/ {printf("%.2f"), $3/$2*100}')

    if [ $(echo "$cpu_usage > 80" | bc -l) -eq 1 ] || [ $(echo "$memory_usage > 80" | bc -l) -eq 1 ]

    then

        echo "System overload detected: CPU usage is at $cpu_usage% and memory usage is at $memory_usage%."

        # Take appropriate action here, such as scaling up the system or offloading traffic.

    else

        echo "System overload not detected. Check for other possible causes."

        # Check for other possible causes here, such as network issues or inefficient code.

    fi

else

    echo "Request rate within normal limits."

fi

```

## Repair

### Restart Kafka service to fix intermittent issues.
```shell
bash

#!/bin/bash

# stop the Kafka cluster

systemctl stop kafka

# start the Kafka cluster

systemctl start kafka
```


This incident type is related to a high request rate on a Kafka producer. This could be caused by a sudden increase in traffic or a malfunction in the producer system. This incident can cause delays in message processing and may lead to data loss or system failure if not addressed promptly. It is important to identify the root cause of this incident and take appropriate measures to prevent similar incidents in the future.


This incident type refers to an increase in the number of errors per second on a Tomcat server, which could indicate an issue with the server itself, the host, a deployed application, or an application servlet. This could include errors generated when the Tomcat server runs out of memory, can't find a requested file or servlet, or is unable to serve a JSP due to syntax errors in the servlet codebase. This incident type requires immediate attention to diagnose and address the underlying issue.


Increase of the errors/second rate for Tomcat server

This incident type refers to a situation where there is an issue with the communication between Kafka brokers due to network partitioning. A network partition is a situation where a network is split into two or more parts, and communication between them is disrupted. This issue can lead to data inconsistency, loss of messages, delayed message delivery, and other related problems. As a result, it is important to quickly detect and resolve these incidents to prevent any downstream impact on the system.


Parameters

This type of incident occurs when the offset of a Kafka topic decreases unexpectedly. In Kafka, an offset is a unique identifier that represents the position of a message within a partition. When the offset decreases, it means that some messages have been lost or deleted, which can cause data inconsistencies or other issues within the system. This incident type is critical for Kafka-based systems as it affects data integrity and can lead to data loss.


Kafka topic offset decreased incident.

This type of incident occurs when a node in the Kafka cluster is not being fully utilized. Kafka is a distributed streaming platform, and each node in the cluster is responsible for processing and storing a portion of the data. If a node is underutilized, it means that it is not processing as much data as it should be, which can lead to performance issues and potential data loss. Therefore, it is important to identify and address underutilized nodes in a Kafka cluster to ensure optimal performance and data reliability.


Kafka node underutilization check incident.

```shell
# Environment Variables

export KAFKA_PRODUCER="PLACEHOLDER"

export KAFKA_CONSUMER="PLACEHOLDER"

export KAFKA_BROKER="PLACEHOLDER"

export TOPIC_NAME="PLACEHOLDER"

export ZOOKEEPER_HOST_PORT="PLACEHOLDER"

```


### Check the network connection between the Kafka producer and consumer

```shell
ping ${KAFKA_PRODUCER}

ping ${KAFKA_CONSUMER}
```

### Check if the Kafka producer is running

```shell
systemctl status ${KAFKA_PRODUCER}
```

### Check the Kafka producer logs for any errors or warnings

```shell
journalctl -u ${KAFKA_PRODUCER} -f
```

### Check the Kafka broker logs for any errors or warnings

```shell
journalctl -u ${KAFKA_BROKER} -f
```

### Check if the Kafka topic is created and has the expected number of partitions

```shell
kafka-topics --zookeeper ${ZOOKEEPER_HOST_PORT} --describe --topic ${TOPIC_NAME}
```

### Check if the Kafka consumer is running

```shell
systemctl status ${KAFKA_CONSUMER}
```

### Check the Kafka consumer logs for any errors or warnings

```shell
journalctl -u ${KAFKA_CONSUMER} -f
```

### Check the CPU and memory usage of the Kafka producer and consumer processes

```shell
top 
```

### Check the network traffic between the Kafka producer and consumer

```shell
tcpdump -i eth01 port 9092
```

### A sudden surge in demand for the service or application that is using the Kafka producer, leading to a high request rate.

```shell
#!/bin/bash

# Set parameters

REQUEST_THRESHOLD="PLACEHOLDER"

TIME_PERIOD="PLACEHOLDER"

# Get request rate

request_rate=$(kafka-console-consumer.sh --bootstrap-server ${KAFKA_BROKER} --topic $TOPIC_NAME --from-beginning --timeout-ms 1000 | wc -l)

# Check if request rate exceeds threshold

if [ $request_rate -gt $REQUEST_THRESHOLD ]

then

    echo "High request rate detected: $request_rate requests in the last $TIME_PERIOD seconds."

# Check for system overload

    cpu_usage=$(top -bn1 | awk '/Cpu/ {print $2}' | sed 's/%//')

    memory_usage=$(free | awk '/Mem/ {printf("%.2f"), $3/$2*100}')

    if [ $(echo "$cpu_usage > 80" | bc -l) -eq 1 ] || [ $(echo "$memory_usage > 80" | bc -l) -eq 1 ]

    then

        echo "System overload detected: CPU usage is at $cpu_usage% and memory usage is at $memory_usage%."

        # Take appropriate action here, such as scaling up the system or offloading traffic.

    else

        echo "System overload not detected. Check for other possible causes."

        # Check for other possible causes here, such as network issues or inefficient code.

    fi

else

    echo "Request rate within normal limits."

fi

```


### Restart Kafka service to fix intermittent issues.

```shell
bash

#!/bin/bash

# stop the Kafka cluster

systemctl stop kafka

# start the Kafka cluster

systemctl start kafka
```


High request rate on Kafka producer.

Overview

Parameters

Debug

Check the network connection between the Kafka producer and consumer

Check if the Kafka producer is running

Check the Kafka producer logs for any errors or warnings

Check the Kafka broker logs for any errors or warnings

Check if the Kafka topic is created and has the expected number of partitions

Check if the Kafka consumer is running

Check the Kafka consumer logs for any errors or warnings

Check the CPU and memory usage of the Kafka producer and consumer processes

Check the network traffic between the Kafka producer and consumer

A sudden surge in demand for the service or application that is using the Kafka producer, leading to a high request rate.

Repair

Restart Kafka service to fix intermittent issues.

Learn more

Related Runbooks

Increase of the errors/second rate for Tomcat server

Network Partitioning Between Kafka Brokers.

Parameters

Kafka topic offset decreased incident.

Support