---
id: d0283612-2e10-48c9-9f94-66f3d5c1ede3
---

# Etcd high fsync durations incident.
---

The Etcd high fsync durations incident occurs when the fsync duration of the Etcd service exceeds a certain threshold. This can be caused by various factors, such as high load, network issues, or other system errors. When this incident occurs, it can impact the performance and availability of the Etcd service, and it requires immediate attention from the responsible team to diagnose and resolve the underlying issue.

### Parameters
```shell
export TIMEFRAME="PLACEHOLDER"

export ETCD_METRICS_ENDPOINT="PLACEHOLDER"

export ETCD_SERVER_IP="PLACEHOLDER"

export DISK_DEVICE="PLACEHOLDER"

export WAL_SYNC_INTERVAL="PLACEHOLDER"

export PATH_TO_ETCD_CONFIG_FILE="PLACEHOLDER"

export CONCURRENT_COMPACTING_PROCESSES="PLACEHOLDER"
```

## Debug

### Check if Etcd service is running
```shell
systemctl status etcd.service
```

### Check Etcd log for any errors or warnings
```shell
journalctl -u etcd.service --since "${TIMEFRAME}" | grep -iE 'error|warn'
```

### Check Etcd metrics for fsync duration
```shell
curl -s ${ETCD_METRICS_ENDPOINT} | grep etcd_disk_backend_commit_duration_seconds
```

### Check system load and resource usage
```shell
top -n 1
```

### Check network connectivity and latency to Etcd server
```shell
ping ${ETCD_SERVER_IP}

traceroute ${ETCD_SERVER_IP}
```

### Check disk I/O performance
```shell
iostat -xd ${DISK_DEVICE}
```

### High load: If the Etcd service is experiencing high traffic or a sudden spike in requests, it can cause the fsync duration to increase, leading to this incident.
```shell
bash

#!/bin/bash



# Set the threshold for fsync duration

THRESHOLD=0.25



# Fetch the current fsync duration of the Etcd service

ETCD_FSYNC=$(etcdctl check perf | awk '{print $7}')



# Check if the fsync duration is above the threshold

if (( $(echo "$ETCD_FSYNC > $THRESHOLD" | bc -l) )); then

  echo "Etcd high fsync durations detected"

  

  # Check the current load on the Etcd service

  ETCD_LOAD=$(etcdctl check perf | awk '{print $6}')

  

  # If the load is high, take action

  if (( $(echo "$ETCD_LOAD > 1" | bc -l) )); then

    echo "High traffic detected on the Etcd service"

    

    # Restart the Etcd service to reduce the load and fsync duration

    systemctl restart etcd

    

    echo "Etcd service restarted"

  else

    echo "No high traffic detected on the Etcd service"

  fi

else

  echo "Etcd fsync durations are normal"

fi


```

## Repair

### Optimize the Etcd configuration settings, such as the WAL sync interval and the number of concurrent compacting processes, to improve the system performance and reduce the fsync duration.
```shell


#!/bin/bash



# Define the Etcd configuration file path

ETCD_CONFIG_FILE=${PATH_TO_ETCD_CONFIG_FILE}



# Define the WAL sync interval and the number of concurrent compacting processes

WAL_SYNC_INTERVAL=${WAL_SYNC_INTERVAL}

CONCURRENT_COMPACTING_PROCESSES=${CONCURRENT_COMPACTING_PROCESSES}



# Update the Etcd configuration file with the new values

sed -i 's/^WAL_SYNC_INTERVAL=.*/WAL_SYNC_INTERVAL='${WAL_SYNC_INTERVAL}'/' $ETCD_CONFIG_FILE

sed -i 's/^CONCURRENT_COMPACTING_PROCESSES=.*/CONCURRENT_COMPACTING_PROCESSES='${CONCURRENT_COMPACTING_PROCESSES}'/' $ETCD_CONFIG_FILE



# Restart the Etcd service to apply the changes

systemctl restart etcd.service


```

The Etcd high fsync durations incident occurs when the fsync duration of the Etcd service exceeds a certain threshold. This can be caused by various factors, such as high load, network issues, or other system errors. When this incident occurs, it can impact the performance and availability of the Etcd service, and it requires immediate attention from the responsible team to diagnose and resolve the underlying issue.


This incident type indicates that there is a high latency issue in the execution of a Spark job. Spark is a distributed computing framework that is used for processing large datasets. High latency in this context means that the time taken to execute the Spark job is significantly longer than expected or normal. This can result in delays in processing data and can impact the performance of the application or system that is utilizing Spark.


High Latency Incident for Spark Job Execution.

This incident type refers to a scenario where there is a high rate of statement timeouts in a Postgresql database instance. This can lead to degraded performance and potentially impact the availability of the database. It is important to quickly identify and address the underlying cause of the timeouts to ensure the stability of the system.


Postgresql high rate statement timeout incident.

This incident type relates to a high urgency issue regarding host context switching. The incident is triggered when the context switching grows on the node beyond a certain level, typically over 10000 per second. This issue can cause performance degradation and impact the stability of the system. The incident requires immediate attention from a software engineer to identify the root cause and take the necessary steps to resolve the issue.


High urgency incident related to host context switching.

The Istio High Request Latency incident type refers to an issue where the average execution time for requests in Istio is longer than the expected threshold of 100ms. This may result in slow or unresponsive services, negatively impacting the user experience. It may require investigation and troubleshooting to identify the underlying cause and resolve the issue.


Istio High Request Latency

This incident type relates to an issue with time synchronization that affects the proper functioning of a Kafka cluster. As Kafka relies on accurate time synchronization to maintain data consistency, any failure in this process can cause serious issues. The incident might cause data loss or duplication, service disruptions, or other problems that can impact the overall performance of the system. It requires immediate attention from the engineering team to investigate and resolve the root cause of the issue.


Kafka Time synchronization issue impacting Kafka cluster.

```shell
export TIMEFRAME="PLACEHOLDER"

export ETCD_METRICS_ENDPOINT="PLACEHOLDER"

export ETCD_SERVER_IP="PLACEHOLDER"

export DISK_DEVICE="PLACEHOLDER"

export WAL_SYNC_INTERVAL="PLACEHOLDER"

export PATH_TO_ETCD_CONFIG_FILE="PLACEHOLDER"

export CONCURRENT_COMPACTING_PROCESSES="PLACEHOLDER"
```


### Check if Etcd service is running

```shell
systemctl status etcd.service
```

### Check Etcd log for any errors or warnings

```shell
journalctl -u etcd.service --since "${TIMEFRAME}" | grep -iE 'error|warn'
```

### Check Etcd metrics for fsync duration

```shell
curl -s ${ETCD_METRICS_ENDPOINT} | grep etcd_disk_backend_commit_duration_seconds
```

### Check system load and resource usage

```shell
top -n 1
```

### Check network connectivity and latency to Etcd server

```shell
ping ${ETCD_SERVER_IP}

traceroute ${ETCD_SERVER_IP}
```

### Check disk I/O performance

```shell
iostat -xd ${DISK_DEVICE}
```

### High load: If the Etcd service is experiencing high traffic or a sudden spike in requests, it can cause the fsync duration to increase, leading to this incident.

```shell
bash

#!/bin/bash



# Set the threshold for fsync duration

THRESHOLD=0.25



# Fetch the current fsync duration of the Etcd service

ETCD_FSYNC=$(etcdctl check perf | awk '{print $7}')



# Check if the fsync duration is above the threshold

if (( $(echo "$ETCD_FSYNC > $THRESHOLD" | bc -l) )); then

  echo "Etcd high fsync durations detected"

  

  # Check the current load on the Etcd service

  ETCD_LOAD=$(etcdctl check perf | awk '{print $6}')

  

  # If the load is high, take action

  if (( $(echo "$ETCD_LOAD > 1" | bc -l) )); then

    echo "High traffic detected on the Etcd service"

    

    # Restart the Etcd service to reduce the load and fsync duration

    systemctl restart etcd

    

    echo "Etcd service restarted"

  else

    echo "No high traffic detected on the Etcd service"

  fi

else

  echo "Etcd fsync durations are normal"

fi


```


### Optimize the Etcd configuration settings, such as the WAL sync interval and the number of concurrent compacting processes, to improve the system performance and reduce the fsync duration.

```shell


#!/bin/bash



# Define the Etcd configuration file path

ETCD_CONFIG_FILE=${PATH_TO_ETCD_CONFIG_FILE}



# Define the WAL sync interval and the number of concurrent compacting processes

WAL_SYNC_INTERVAL=${WAL_SYNC_INTERVAL}

CONCURRENT_COMPACTING_PROCESSES=${CONCURRENT_COMPACTING_PROCESSES}



# Update the Etcd configuration file with the new values

sed -i 's/^WAL_SYNC_INTERVAL=.*/WAL_SYNC_INTERVAL='${WAL_SYNC_INTERVAL}'/' $ETCD_CONFIG_FILE

sed -i 's/^CONCURRENT_COMPACTING_PROCESSES=.*/CONCURRENT_COMPACTING_PROCESSES='${CONCURRENT_COMPACTING_PROCESSES}'/' $ETCD_CONFIG_FILE



# Restart the Etcd service to apply the changes

systemctl restart etcd.service


```


Etcd high fsync durations incident.

Overview

Parameters

Debug

Check if Etcd service is running

Check Etcd log for any errors or warnings

Check Etcd metrics for fsync duration

Check system load and resource usage

Check network connectivity and latency to Etcd server

Check disk I/O performance

High load: If the Etcd service is experiencing high traffic or a sudden spike in requests, it can cause the fsync duration to increase, leading to this incident.

Repair

Optimize the Etcd configuration settings, such as the WAL sync interval and the number of concurrent compacting processes, to improve the system performance and reduce the fsync duration.

Learn more

Related Runbooks

High Latency Incident for Spark Job Execution.

Postgresql high rate statement timeout incident.

High urgency incident related to host context switching.

Istio High Request Latency

Support