---
id: 49ce2690-d45a-4c9f-9539-a0d4dce64b34
---

# Kafka Disk IO Latency Spike Incident
---

This incident type refers to a sudden increase in disk input/output (I/O) latency in the Kafka message broker, which can lead to degraded performance, slow processing of messages, and potentially impact system availability. This may be caused by a variety of factors, such as hardware failure, network issues, or software bugs. It is important to quickly identify and resolve this issue to prevent disruption to the system and ensure smooth operation of the Kafka message broker.

### Parameters
```shell
export DISK_NAME="PLACEHOLDER"

export PATH_TO_KAFKA_LOGS="PLACEHOLDER"

export AMOUNT_OF_TIME_TO_RETAIN_MESSAGES_IN_MILLISECONDS="PLACEHOLDER"

export NAME_OF_KAFKA_TOPIC_TO_CLEAN_UP="PLACEHOLDER"

export PATH_TO_KAFKA_HOME_DIRECTORY="PLACEHOLDER"
```

## Debug

### Check the current CPU usage
```shell
top
```

### Check if Kafka is running
```shell
systemctl status kafka
```

### Check the current disk usage
```shell
df -h
```

### Check the disk I/O utilization
```shell
iostat -x ${DISK_NAME}
```

### Check the disk I/O wait time
```shell
iostat -x ${DISK_NAME} | grep -E 'avgqu-sz|await'
```

### Check the current network usage
```shell
nload
```

### Check the Kafka logs for any errors or warnings
```shell
tail -f ${PATH_TO_KAFKA_LOGS}
```

## Repair

### Implement data retention policies to limit the amount of data stored on the Kafka brokers, and periodically archive or delete older data to free up disk space and improve disk I/O performance.
```shell


#!/bin/bash



# Define variables

KAFKA_HOME=${PATH_TO_KAFKA_HOME_DIRECTORY}

TOPIC=${NAME_OF_KAFKA_TOPIC_TO_CLEAN_UP}

RETENTION_TIME=${AMOUNT_OF_TIME_TO_RETAIN_MESSAGES_IN_MILLISECONDS}



# Set retention policy for Kafka topic

$KAFKA_HOME/bin/kafka-topics.sh --alter --topic $TOPIC --config retention.ms=$RETENTION_TIME



# Clean up old messages from Kafka topic

$KAFKA_HOME/bin/kafka-topics.sh --alter --topic $TOPIC --config cleanup.policy=compact


```

This incident type refers to a sudden increase in disk input/output (I/O) latency in the Kafka message broker, which can lead to degraded performance, slow processing of messages, and potentially impact system availability. This may be caused by a variety of factors, such as hardware failure, network issues, or software bugs. It is important to quickly identify and resolve this issue to prevent disruption to the system and ensure smooth operation of the Kafka message broker.


```shell
export DISK_NAME="PLACEHOLDER"

export PATH_TO_KAFKA_LOGS="PLACEHOLDER"

export AMOUNT_OF_TIME_TO_RETAIN_MESSAGES_IN_MILLISECONDS="PLACEHOLDER"

export NAME_OF_KAFKA_TOPIC_TO_CLEAN_UP="PLACEHOLDER"

export PATH_TO_KAFKA_HOME_DIRECTORY="PLACEHOLDER"
```


### Check the current CPU usage

```shell
top
```

### Check if Kafka is running

```shell
systemctl status kafka
```

### Check the current disk usage

```shell
df -h
```

### Check the disk I/O utilization

```shell
iostat -x ${DISK_NAME}
```

### Check the disk I/O wait time

```shell
iostat -x ${DISK_NAME} | grep -E 'avgqu-sz|await'
```

### Check the current network usage

```shell
nload
```

### Check the Kafka logs for any errors or warnings

```shell
tail -f ${PATH_TO_KAFKA_LOGS}
```


### Implement data retention policies to limit the amount of data stored on the Kafka brokers, and periodically archive or delete older data to free up disk space and improve disk I/O performance.

```shell


#!/bin/bash



# Define variables

KAFKA_HOME=${PATH_TO_KAFKA_HOME_DIRECTORY}

TOPIC=${NAME_OF_KAFKA_TOPIC_TO_CLEAN_UP}

RETENTION_TIME=${AMOUNT_OF_TIME_TO_RETAIN_MESSAGES_IN_MILLISECONDS}



# Set retention policy for Kafka topic

$KAFKA_HOME/bin/kafka-topics.sh --alter --topic $TOPIC --config retention.ms=$RETENTION_TIME



# Clean up old messages from Kafka topic

$KAFKA_HOME/bin/kafka-topics.sh --alter --topic $TOPIC --config cleanup.policy=compact


```


Kafka Disk IO Latency Spike Incident

Overview

Parameters

Debug

Check the current CPU usage

Check if Kafka is running

Check the current disk usage

Check the disk I/O utilization

Check the disk I/O wait time

Check the current network usage

Check the Kafka logs for any errors or warnings

Repair

Implement data retention policies to limit the amount of data stored on the Kafka brokers, and periodically archive or delete older data to free up disk space and improve disk I/O performance.

Learn more

Related Runbooks

Support