---
id: e0b0ad98-69d6-42e9-9d4e-afb6d24ec3ae
---

# Host Clock Skew Incident
---

A host clock skew incident occurs when there is a discrepancy between the system clock of a host and the actual time. This can lead to issues with timestamps and synchronization between different systems. This incident type often requires investigation and resolution of the underlying clock synchronization issue.

### Parameters
```shell
export NTP_SERVER_IP_ADDRESS="PLACEHOLDER"

export EXPECTED_TIMEZONE="PLACEHOLDER"

export NTP_SERVER_IP="PLACEHOLDER"
```

## Debug

### Check the current system time
```shell
date
```

### Check the time settings of the host
```shell
timedatectl
```

### Check the NTP status and configuration
```shell
systemctl status ntp

grep -v ^# /etc/ntp.conf
```

### Check the NTP peers and their synchronization status
```shell
ntpq -p
```

### Force NTP time synchronization
```shell
sudo systemctl stop ntp

sudo ntpd -gq

sudo systemctl start ntp
```

### Check the system logs for time-related issues
```shell
journalctl -u ntp
```

### Misconfigured Network Time Protocol (NTP) server on the host, leading to incorrect synchronization of time.
```shell
bash

#!/bin/bash



# Check if NTP service is running

if ! systemctl is-active ntpd > /dev/null; then

    echo "NTP service is not running."

    exit 1

fi



# Check NTP server configuration

if ! grep -q "^server ${NTP_SERVER_IP_ADDRESS}" /etc/ntp.conf; then

    echo "NTP server is not configured correctly."

    exit 1

fi



# Check if NTP server is reachable

if ! ntpdate -q ${NTP_SERVER_IP_ADDRESS} > /dev/null; then

    echo "Unable to reach NTP server."

    exit 1

fi



echo "NTP server is configured and reachable."

exit 0


```

### Incorrect system time zone settings on the host.
```shell
bash

#!/bin/bash



# Get the current system time

current_time=$(date +%Y-%m-%d\ %H:%M:%S)



# Get the current system time zone

current_timezone=$(date +%Z)



# Get the expected system time zone

expected_timezone=${EXPECTED_TIMEZONE}



# Compare the current and expected time zones

if [ "$current_timezone" != "$expected_timezone" ]; then

  echo "ERROR: System time zone is incorrect. Current time zone is $current_timezone, but expected time zone is $expected_timezone."

  echo "Current time is $current_time"

else

  echo "SUCCESS: System time zone is correct."

fi


```

## Repair

### Verify that the Network Time Protocol (NTP) service is running and properly configured on the affected host(s). If NTP is not installed, install and configure it.
```shell


#!/bin/bash



# Check if NTP service is running

systemctl is-active --quiet ntpd

if [ $? -eq 0 ]; then

    echo "NTP service is already running."

else

    # Install NTP if it is not installed

    if ! dpkg -s ntp > /dev/null 2>&1 ; then

        apt-get update

        apt-get install ntp -y

    fi



    # Configure NTP

    sed -i 's/^pool/#pool/' /etc/ntp.conf

    echo "server ${NTP_SERVER_IP}" >> /etc/ntp.conf



    # Restart NTP service

    systemctl restart ntpd

    echo "NTP service has been restarted."

fi


```

### Set server to the correct time zone
```shell


#!/bin/bash



# Set the time zone

sudo timedatectl set-timezone ${EXPECTED_TIME_ZONE}



# Verify the time zone has been set correctly

timedatectl


```

A host clock skew incident occurs when there is a discrepancy between the system clock of a host and the actual time. This can lead to issues with timestamps and synchronization between different systems. This incident type often requires investigation and resolution of the underlying clock synchronization issue.


The Host Out of Memory (OOM) Incident occurs when a server or system runs out of memory, causing it to crash or become unresponsive. This can be caused by various factors, such as an unexpected surge in traffic or insufficient resources allocated to the system. Resolving this type of incident requires identifying the root cause of the memory issue and taking appropriate measures such as optimizing system resources or increasing memory capacity.


Host Out of Memory (OOM) Incident

This incident type typically refers to a situation where there is an anomaly in the Tomcat max processing time on a host. The maximum amount of time it takes for the server to process one request, from the time an available thread starts processing the request to the time it returns a response, has exceeded the expected threshold. This could indicate that a JSP page isn’t loading or an associated process (such as a database query) is taking too long to complete.


Anomalous max processing time for Tomcat host

Data skew slowing down a Spark job is a common incident that occurs when there is an uneven distribution of data across partitions. This can lead to some partitions handling a much larger amount of data than others, causing delays and slowing down the entire job. The result is a significant decrease in performance and a longer processing time for the Spark job.


Data Skew Slowing Down Spark Job.

This incident type refers to the failure of a systemd service on a particular host instance. The incident could be triggered by various causes such as a software bug, hardware failure, or system overload. This type of incident can cause downtime or service disruption to the affected host instance, which may require immediate resolution to restore normal operations.


Host systemd service crashed (instance) incident.

This incident type occurs when a host requires a reboot. It could be due to various reasons like an update, configuration changes, or malfunctioning. If the host is not rebooted, it could cause performance issues or even lead to downtime. Therefore, immediate action is necessary to resolve the issue and restore the system to its optimal state. The incident could be triggered automatically by monitoring tools or reported by users.


Host requires reboot incident.

```shell
export NTP_SERVER_IP_ADDRESS="PLACEHOLDER"

export EXPECTED_TIMEZONE="PLACEHOLDER"

export NTP_SERVER_IP="PLACEHOLDER"
```


### Check the current system time

```shell
date
```

### Check the time settings of the host

```shell
timedatectl
```

### Check the NTP status and configuration

```shell
systemctl status ntp

grep -v ^# /etc/ntp.conf
```

### Check the NTP peers and their synchronization status

```shell
ntpq -p
```

### Force NTP time synchronization

```shell
sudo systemctl stop ntp

sudo ntpd -gq

sudo systemctl start ntp
```

### Check the system logs for time-related issues

```shell
journalctl -u ntp
```

### Misconfigured Network Time Protocol (NTP) server on the host, leading to incorrect synchronization of time.

```shell
bash

#!/bin/bash



# Check if NTP service is running

if ! systemctl is-active ntpd > /dev/null; then

    echo "NTP service is not running."

    exit 1

fi



# Check NTP server configuration

if ! grep -q "^server ${NTP_SERVER_IP_ADDRESS}" /etc/ntp.conf; then

    echo "NTP server is not configured correctly."

    exit 1

fi



# Check if NTP server is reachable

if ! ntpdate -q ${NTP_SERVER_IP_ADDRESS} > /dev/null; then

    echo "Unable to reach NTP server."

    exit 1

fi



echo "NTP server is configured and reachable."

exit 0


```

### Incorrect system time zone settings on the host.

```shell
bash

#!/bin/bash



# Get the current system time

current_time=$(date +%Y-%m-%d\ %H:%M:%S)



# Get the current system time zone

current_timezone=$(date +%Z)



# Get the expected system time zone

expected_timezone=${EXPECTED_TIMEZONE}



# Compare the current and expected time zones

if [ "$current_timezone" != "$expected_timezone" ]; then

  echo "ERROR: System time zone is incorrect. Current time zone is $current_timezone, but expected time zone is $expected_timezone."

  echo "Current time is $current_time"

else

  echo "SUCCESS: System time zone is correct."

fi


```


### Verify that the Network Time Protocol (NTP) service is running and properly configured on the affected host(s). If NTP is not installed, install and configure it.

```shell


#!/bin/bash



# Check if NTP service is running

systemctl is-active --quiet ntpd

if [ $? -eq 0 ]; then

    echo "NTP service is already running."

else

    # Install NTP if it is not installed

    if ! dpkg -s ntp > /dev/null 2>&1 ; then

        apt-get update

        apt-get install ntp -y

    fi



    # Configure NTP

    sed -i 's/^pool/#pool/' /etc/ntp.conf

    echo "server ${NTP_SERVER_IP}" >> /etc/ntp.conf



    # Restart NTP service

    systemctl restart ntpd

    echo "NTP service has been restarted."

fi


```

### Set server to the correct time zone

```shell


#!/bin/bash



# Set the time zone

sudo timedatectl set-timezone ${EXPECTED_TIME_ZONE}



# Verify the time zone has been set correctly

timedatectl


```


Host Clock Skew Incident

Overview

Parameters

Debug

Check the current system time

Check the time settings of the host

Check the NTP status and configuration

Check the NTP peers and their synchronization status

Force NTP time synchronization

Misconfigured Network Time Protocol (NTP) server on the host, leading to incorrect synchronization of time.

Incorrect system time zone settings on the host.

Repair

Verify that the Network Time Protocol (NTP) service is running and properly configured on the affected host(s). If NTP is not installed, install and configure it.

Set server to the correct time zone

Learn more

Related Runbooks

Host Out of Memory (OOM) Incident

Anomalous max processing time for Tomcat host

Data Skew Slowing Down Spark Job.

Host systemd service crashed (instance) incident.

Support