---
id: a5863191-a10a-43d3-a0cc-bf8c1316acac
---

# Slow Etcd GRPC Requests
---

The incident type "Slow Etcd GRPC Requests" is triggered when the performance of Etcd GRPC requests slows down. This may be due to HTTP requests slowing down, and the 99th percentile exceeding 0.15s. This incident can affect the overall performance of the system, and needs to be resolved promptly.

### Parameters
```shell
export INTERFACE_NAME="PLACEHOLDER"

export NEW_NODE_COUNT="PLACEHOLDER"

export NEW_NODE_IP="PLACEHOLDER"

export CPU_LIMIT="PLACEHOLDER"

export NETWORK_LIMIT="PLACEHOLDER"

export DISK_LIMIT="PLACEHOLDER"

export MEMORY_LIMIT="PLACEHOLDER"
```

## Debug

### Check the current CPU usage
```shell
top -b -n 1 | head -n 20
```

### Check the current memory usage
```shell
free -m
```

### Check the network connections to the affected instance
```shell
ss -s
```

### Check the network traffic on the affected network interface
```shell
ifstat -i ${INTERFACE_NAME} 1
```

### Check the etcd logs for errors or warnings
```shell
journalctl -u etcd.service | tail -n 100
```

### Check the GRPC logs for errors or warnings
```shell
journalctl -u grpc.service | tail -n 100
```

### Check the Prometheus metrics for etcd and GRPC
```shell
curl -s http://localhost:9090/metrics | grep etcd | grep grpc
```

### Check the etcd cluster health
```shell
etcdctl cluster-health
```

### Check the etcd member list
```shell
etcdctl member list
```

### Check the etcd endpoint status
```shell
etcdctl endpoint status
```

### Check the etcd key space size
```shell
etcdctl endpoint fsync total
```

### Check the etcd watch counts
```shell
etcdctl watch count
```

### Check the etcd snapshot size
```shell
du -sh /var/lib/etcd/member/snap/db
```

### Resource contention on the etcd cluster.
```shell


#!/bin/bash



# Check for resource contention on the etcd cluster



# Check CPU usage

cpu_limit=${CPU_LIMIT}

cpu_usage=$(top -b -n 1 | grep etcd | awk '{print $9}')

if (( $(echo "$cpu_usage > $cpu_limit" | bc -l) )); then

    echo "CPU usage is high. Current usage is $cpu_usage%"

fi



# Check memory usage

mem_limit=${MEMORY_LIMIT}

mem_usage=$(top -b -n 1 | grep etcd | awk '{print $10}')

if (( $(echo "$mem_usage > $mem_limit" | bc -l) )); then

    echo "Memory usage is high. Current usage is $mem_usage%"

fi



# Check network throughput

net_limit=${NETWORK_LIMIT}

net_usage=$(netstat -an | grep -c etcd)

if (( $(echo "$net_usage > $net_limit" | bc -l) )); then

    echo "Network throughput is high. Current throughput is $net_usage connections"

fi



# Check disk usage

disk_limit=${DISK_LIMIT}

disk_usage=$(df -h | grep etcd | awk '{print $5}')

if (( $(echo "${disk_usage::-1} > $disk_limit" | bc -l) )); then

    echo "Disk usage is high. Current usage is $disk_usage"

fi



# Check for any locked resources

if [[ $(find /var/lib/etcd/member/wal -type f -name lock) ]]; then

    echo "Etcd cluster has locked resources"

fi




```

## Repair

### Increase the number of etcd nodes to improve the capacity of the cluster.
```shell


#!/bin/bash



# Set the number of nodes to increase the capacity of the etcd cluster.

NEW_NODE_COUNT=${NEW_NODE_COUNT}



# Get the current number of nodes in the etcd cluster.

CURRENT_NODE_COUNT=$(etcdctl member list | wc -l)



# Check if the current node count is less than the new node count.

if [ "$CURRENT_NODE_COUNT" -lt "$NEW_NODE_COUNT" ]; then

  # Calculate the number of new nodes to add.

  NODES_TO_ADD=$((NEW_NODE_COUNT - CURRENT_NODE_COUNT))



  # Loop through the number of new nodes to add.

  for i in $(seq 1 $NODES_TO_ADD); do

    # Generate a new unique name for the etcd node.

    NODE_NAME=$(uuidgen)



    # Add the new node to the etcd cluster.

    etcdctl member add "$NODE_NAME" --peer-urls="https://${NEW_NODE_IP}:2380"



    # Start the etcd service on the new node.

    systemctl start etcd

  done



  # Check the status of the etcd cluster.

  etcdctl cluster-health

fi


```

The incident type "Slow Etcd GRPC Requests" is triggered when the performance of Etcd GRPC requests slows down. This may be due to HTTP requests slowing down, and the 99th percentile exceeding 0.15s. This incident can affect the overall performance of the system, and needs to be resolved promptly.


This incident type refers to an increase in the number of errors per second on a Tomcat server, which could indicate an issue with the server itself, the host, a deployed application, or an application servlet. This could include errors generated when the Tomcat server runs out of memory, can't find a requested file or servlet, or is unable to serve a JSP due to syntax errors in the servlet codebase. This incident type requires immediate attention to diagnose and address the underlying issue.


Increase of the errors/second rate for Tomcat server

This incident type refers to an issue with Redis replication, which means that there is a problem with the synchronization of data between Redis instances. This issue could impact the availability and performance of the system and may require immediate attention to restore the replication and ensure data consistency. The incident could be caused by various factors, such as network problems, hardware failures, or configuration issues. The incident must be investigated and resolved as soon as possible to avoid any data loss or downtime.


Redis replication broken incident.

This incident type refers to an issue where Istio latency has exceeded the 99th percentile, indicating that the slowest 1% of requests are taking longer than 1 second to complete. This can cause performance issues and impact the user experience. It requires immediate attention and investigation to resolve the issue and prevent any further impact.


Istio Latency 99 Percentile Incident

The Istio High Request Latency incident type refers to an issue where the average execution time for requests in Istio is longer than the expected threshold of 100ms. This may result in slow or unresponsive services, negatively impacting the user experience. It may require investigation and troubleshooting to identify the underlying cause and resolve the issue.


Istio High Request Latency

This incident type is related to a high request rate on a Kafka producer. This could be caused by a sudden increase in traffic or a malfunction in the producer system. This incident can cause delays in message processing and may lead to data loss or system failure if not addressed promptly. It is important to identify the root cause of this incident and take appropriate measures to prevent similar incidents in the future.


High request rate on Kafka producer.

```shell
export INTERFACE_NAME="PLACEHOLDER"

export NEW_NODE_COUNT="PLACEHOLDER"

export NEW_NODE_IP="PLACEHOLDER"

export CPU_LIMIT="PLACEHOLDER"

export NETWORK_LIMIT="PLACEHOLDER"

export DISK_LIMIT="PLACEHOLDER"

export MEMORY_LIMIT="PLACEHOLDER"
```


### Check the current CPU usage

```shell
top -b -n 1 | head -n 20
```

### Check the current memory usage

```shell
free -m
```

### Check the network connections to the affected instance

```shell
ss -s
```

### Check the network traffic on the affected network interface

```shell
ifstat -i ${INTERFACE_NAME} 1
```

### Check the etcd logs for errors or warnings

```shell
journalctl -u etcd.service | tail -n 100
```

### Check the GRPC logs for errors or warnings

```shell
journalctl -u grpc.service | tail -n 100
```

### Check the Prometheus metrics for etcd and GRPC

```shell
curl -s http://localhost:9090/metrics | grep etcd | grep grpc
```

### Check the etcd cluster health

```shell
etcdctl cluster-health
```

### Check the etcd member list

```shell
etcdctl member list
```

### Check the etcd endpoint status

```shell
etcdctl endpoint status
```

### Check the etcd key space size

```shell
etcdctl endpoint fsync total
```

### Check the etcd watch counts

```shell
etcdctl watch count
```

### Check the etcd snapshot size

```shell
du -sh /var/lib/etcd/member/snap/db
```

### Resource contention on the etcd cluster.

```shell


#!/bin/bash



# Check for resource contention on the etcd cluster



# Check CPU usage

cpu_limit=${CPU_LIMIT}

cpu_usage=$(top -b -n 1 | grep etcd | awk '{print $9}')

if (( $(echo "$cpu_usage > $cpu_limit" | bc -l) )); then

    echo "CPU usage is high. Current usage is $cpu_usage%"

fi



# Check memory usage

mem_limit=${MEMORY_LIMIT}

mem_usage=$(top -b -n 1 | grep etcd | awk '{print $10}')

if (( $(echo "$mem_usage > $mem_limit" | bc -l) )); then

    echo "Memory usage is high. Current usage is $mem_usage%"

fi



# Check network throughput

net_limit=${NETWORK_LIMIT}

net_usage=$(netstat -an | grep -c etcd)

if (( $(echo "$net_usage > $net_limit" | bc -l) )); then

    echo "Network throughput is high. Current throughput is $net_usage connections"

fi



# Check disk usage

disk_limit=${DISK_LIMIT}

disk_usage=$(df -h | grep etcd | awk '{print $5}')

if (( $(echo "${disk_usage::-1} > $disk_limit" | bc -l) )); then

    echo "Disk usage is high. Current usage is $disk_usage"

fi



# Check for any locked resources

if [[ $(find /var/lib/etcd/member/wal -type f -name lock) ]]; then

    echo "Etcd cluster has locked resources"

fi




```


### Increase the number of etcd nodes to improve the capacity of the cluster.

```shell


#!/bin/bash



# Set the number of nodes to increase the capacity of the etcd cluster.

NEW_NODE_COUNT=${NEW_NODE_COUNT}



# Get the current number of nodes in the etcd cluster.

CURRENT_NODE_COUNT=$(etcdctl member list | wc -l)



# Check if the current node count is less than the new node count.

if [ "$CURRENT_NODE_COUNT" -lt "$NEW_NODE_COUNT" ]; then

  # Calculate the number of new nodes to add.

  NODES_TO_ADD=$((NEW_NODE_COUNT - CURRENT_NODE_COUNT))



  # Loop through the number of new nodes to add.

  for i in $(seq 1 $NODES_TO_ADD); do

    # Generate a new unique name for the etcd node.

    NODE_NAME=$(uuidgen)



    # Add the new node to the etcd cluster.

    etcdctl member add "$NODE_NAME" --peer-urls="https://${NEW_NODE_IP}:2380"



    # Start the etcd service on the new node.

    systemctl start etcd

  done



  # Check the status of the etcd cluster.

  etcdctl cluster-health

fi


```


Slow Etcd GRPC Requests

Overview

Parameters

Debug

Check the current CPU usage

Check the current memory usage

Check the network connections to the affected instance

Check the network traffic on the affected network interface

Check the etcd logs for errors or warnings

Check the GRPC logs for errors or warnings

Check the Prometheus metrics for etcd and GRPC

Check the etcd cluster health

Check the etcd member list

Check the etcd endpoint status

Check the etcd key space size

Check the etcd watch counts

Check the etcd snapshot size

Resource contention on the etcd cluster.

Repair

Increase the number of etcd nodes to improve the capacity of the cluster.

Learn more

Related Runbooks

Increase of the errors/second rate for Tomcat server

Redis replication broken incident.

Istio Latency 99 Percentile Incident

Istio High Request Latency

Support