---
id: 35efdaa6-37d1-46c2-9524-5593271467e7
---

# Kafka Time synchronization issue impacting Kafka cluster.
---

This incident type relates to an issue with time synchronization that affects the proper functioning of a Kafka cluster. As Kafka relies on accurate time synchronization to maintain data consistency, any failure in this process can cause serious issues. The incident might cause data loss or duplication, service disruptions, or other problems that can impact the overall performance of the system. It requires immediate attention from the engineering team to investigate and resolve the root cause of the issue.

### Parameters
```shell
export TIME_ZONE="PLACEHOLDER"

export TIME_SERVER="PLACEHOLDER"

export SERVER1="PLACEHOLDER"

export SERVER3="PLACEHOLDER"

export SERVER2="PLACEHOLDER"
```

## Debug

### Check system time and date
```shell
date
```

### Check if NTP is installed and running
```shell
ntpstat
```

### Check the status of the NTP service
```shell
systemctl status ntp
```

### Check the NTP configuration file for the correct NTP servers
```shell
cat /etc/ntp.conf
```

### Check the NTP query responses
```shell
ntpq -p
```

### Check the Kafka logs for any time synchronization errors
```shell
grep time /var/log/kafka/server.log
```

### Check the status of the Kafka service
```shell
systemctl status kafka
```

### Check the Kafka configuration file for the correct time zone
```shell
cat /etc/kafka/server.properties
```

### The time zone settings on the affected servers may have been incorrect or changed.
```shell


#!/bin/bash



# Set the list of affected servers

servers="${SERVER1} ${SERVER2} ${SERVER3}"



# Loop through each server and check the time zone setting

for server in $servers

do

    echo "Checking time zone setting on $server..."

    ssh $server "date"

    ssh $server "timedatectl status"

done


```

## Repair

### Check the time synchronization settings on all servers hosting the Kafka cluster and ensure that they are set to the correct time zone and are synced with a reliable time source.
```shell


#!/bin/bash



# Set the correct time zone

sudo timedatectl set-timezone ${TIME_ZONE}



# Install NTP if not already installed

sudo apt-get install ntp -y



# Stop NTP service

sudo systemctl stop ntp.service



# Sync time with a reliable time server

sudo ntpdate ${TIME_SERVER}



# Start NTP service

sudo systemctl start ntp.service


```

This incident type relates to an issue with time synchronization that affects the proper functioning of a Kafka cluster. As Kafka relies on accurate time synchronization to maintain data consistency, any failure in this process can cause serious issues. The incident might cause data loss or duplication, service disruptions, or other problems that can impact the overall performance of the system. It requires immediate attention from the engineering team to investigate and resolve the root cause of the issue.


A host clock skew incident occurs when there is a discrepancy between the system clock of a host and the actual time. This can lead to issues with timestamps and synchronization between different systems. This incident type often requires investigation and resolution of the underlying clock synchronization issue.


Host Clock Skew Incident

This incident type refers to a situation where there is an issue with the communication between Kafka brokers due to network partitioning. A network partition is a situation where a network is split into two or more parts, and communication between them is disrupted. This issue can lead to data inconsistency, loss of messages, delayed message delivery, and other related problems. As a result, it is important to quickly detect and resolve these incidents to prevent any downstream impact on the system.


Parameters

This type of incident occurs when the offset of a Kafka topic decreases unexpectedly. In Kafka, an offset is a unique identifier that represents the position of a message within a partition. When the offset decreases, it means that some messages have been lost or deleted, which can cause data inconsistencies or other issues within the system. This incident type is critical for Kafka-based systems as it affects data integrity and can lead to data loss.


Kafka topic offset decreased incident.

This incident type refers to when a Kafka node runs out of storage space to store data. This can cause issues with data processing and storage, potentially leading to data loss and system downtime.


Kafka node insufficient storage check

```shell
export TIME_ZONE="PLACEHOLDER"

export TIME_SERVER="PLACEHOLDER"

export SERVER1="PLACEHOLDER"

export SERVER3="PLACEHOLDER"

export SERVER2="PLACEHOLDER"
```


### Check system time and date

```shell
date
```

### Check if NTP is installed and running

```shell
ntpstat
```

### Check the status of the NTP service

```shell
systemctl status ntp
```

### Check the NTP configuration file for the correct NTP servers

```shell
cat /etc/ntp.conf
```

### Check the NTP query responses

```shell
ntpq -p
```

### Check the Kafka logs for any time synchronization errors

```shell
grep time /var/log/kafka/server.log
```

### Check the status of the Kafka service

```shell
systemctl status kafka
```

### Check the Kafka configuration file for the correct time zone

```shell
cat /etc/kafka/server.properties
```

### The time zone settings on the affected servers may have been incorrect or changed.

```shell


#!/bin/bash



# Set the list of affected servers

servers="${SERVER1} ${SERVER2} ${SERVER3}"



# Loop through each server and check the time zone setting

for server in $servers

do

    echo "Checking time zone setting on $server..."

    ssh $server "date"

    ssh $server "timedatectl status"

done


```


### Check the time synchronization settings on all servers hosting the Kafka cluster and ensure that they are set to the correct time zone and are synced with a reliable time source.

```shell


#!/bin/bash



# Set the correct time zone

sudo timedatectl set-timezone ${TIME_ZONE}



# Install NTP if not already installed

sudo apt-get install ntp -y



# Stop NTP service

sudo systemctl stop ntp.service



# Sync time with a reliable time server

sudo ntpdate ${TIME_SERVER}



# Start NTP service

sudo systemctl start ntp.service


```


Kafka Time synchronization issue impacting Kafka cluster.

Overview

Parameters

Debug

Check system time and date

Check if NTP is installed and running

Check the status of the NTP service

Check the NTP configuration file for the correct NTP servers

Check the NTP query responses

Check the Kafka logs for any time synchronization errors

Check the status of the Kafka service

Check the Kafka configuration file for the correct time zone

The time zone settings on the affected servers may have been incorrect or changed.

Repair

Check the time synchronization settings on all servers hosting the Kafka cluster and ensure that they are set to the correct time zone and are synced with a reliable time source.

Learn more

Related Runbooks

Host Clock Skew Incident

Network Partitioning Between Kafka Brokers.

Parameters

Kafka topic offset decreased incident.

Support