---
id: 85bdb7e7-7150-49c1-815f-acaf9d22e3d6
---

# Istio High Request Latency
---

The Istio High Request Latency incident type refers to an issue where the average execution time for requests in Istio is longer than the expected threshold of 100ms. This may result in slow or unresponsive services, negatively impacting the user experience. It may require investigation and troubleshooting to identify the underlying cause and resolve the issue.

### Parameters
```shell
export POD_NAME="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"

export SERVICE_NAME="PLACEHOLDER"

export RESOURCE_NAME="PLACEHOLDER"

export RESOURCE_VALUE="PLACEHOLDER"

export DEPLOYMENT_NAME="PLACEHOLDER"
```

## Debug

### Check if Istio is installed in the cluster
```shell
kubectl get ns istio-system
```

### Check the status of the Istio pods and services
```shell
kubectl get pods -n istio-system

kubectl get svc -n istio-system
```

### Check the logs of the Istio sidecar proxy for the affected service
```shell
kubectl logs ${POD_NAME} -c istio-proxy -n ${NAMESPACE}
```

### Check the latency of the requests to the affected service
```shell
kubectl exec ${POD_NAME} -c ${CONTAINER_NAME} -n ${NAMESPACE} -- sh -c "curl http://localhost:8080/healthz"
```

### Check the Istio configuration for the affected service
```shell
kubectl get virtualservices -n ${NAMESPACE}

kubectl get destinationrules -n ${NAMESPACE}

kubectl get gateway -n ${NAMESPACE}
```

### Check if there are any network issues affecting the service
```shell
kubectl exec ${POD_NAME} -n ${NAMESPACE} -- ping ${SERVICE_NAME}
```

### Check if there are any CPU or memory issues affecting the service
```shell
kubectl top pods -n ${NAMESPACE}

kubectl describe pod ${POD_NAME} -n ${NAMESPACE}
```

### Resource constraints: The service may be under-resourced, leading to a slower response time for requests.
```shell


#!/bin/bash



# Check CPU and memory usage of the pod

CPU=$(kubectl top pod ${POD_NAME} | awk '{print $2}' | tail -n1)

MEMORY=$(kubectl top pod ${POD_NAME} | awk '{print $3}' | tail -n1)



# Check resource limits and requests set in the pod spec

CPU_LIMIT=$(kubectl describe pod ${POD_NAME} | grep "Limits" | awk '{print $4}')

MEMORY_LIMIT=$(kubectl describe pod ${POD_NAME} | grep "Limits" | awk '{print $6}')

CPU_REQUEST=$(kubectl describe pod ${POD_NAME} | grep "Requests" | awk '{print $4}')

MEMORY_REQUEST=$(kubectl describe pod ${POD_NAME} | grep "Requests" | awk '{print $6}')



# Check if CPU and memory usage are close to or exceeding the limits and requests

if (( $(echo "$CPU >= $CPU_LIMIT" |bc -l) )) || (( $(echo "$MEMORY >= $MEMORY_LIMIT" |bc -l) )) || (( $(echo "$CPU >= $CPU_REQUEST" |bc -l) )) || (( $(echo "$MEMORY >= $MEMORY_REQUEST" |bc -l) )); then

    echo "Pod ${POD_NAME} is using high CPU and/or memory resources, which may be causing the slow response time."

fi


```

## Repair

### If the issue is related to hardware resource constraints, consider increasing the resources allocated to the affected service or adding more nodes to the cluster.
```shell


#!/bin/bash



# Set variables

DEPLOYMENT_NAME=${DEPLOYMENT_NAME}

RESOURCE_NAME=${RESOURCE_NAME}

RESOURCE_VALUE=${RESOURCE_VALUE}

# Use kubectl patch to patch the deployment
kubectl patch deployment $DEPLOYMENT_NAME -p '{"spec":{"template":{"spec":{"containers":{"resources":[{"limits":{"'$RESOURCE_NAME'":"'$RESOURCE_VALUE'"}, "requests":{"'$RESOURCE_NAME'":"'$RESOURCE_VALUE'"}}]}}}}}'

```

The Istio High Request Latency incident type refers to an issue where the average execution time for requests in Istio is longer than the expected threshold of 100ms. This may result in slow or unresponsive services, negatively impacting the user experience. It may require investigation and troubleshooting to identify the underlying cause and resolve the issue.


This incident type occurs when the number of 4xx errors on Traffic Server is at an anomalous level, higher than usual. It could be an indicator of an issue with the server or an increase in traffic that is causing errors. It requires investigation and resolution to ensure the server is functioning correctly and not impacting users.


High 4xx Errors on Traffic Server

This incident type indicates that there is a high latency issue in the execution of a Spark job. Spark is a distributed computing framework that is used for processing large datasets. High latency in this context means that the time taken to execute the Spark job is significantly longer than expected or normal. This can result in delays in processing data and can impact the performance of the application or system that is utilizing Spark.


High Latency Incident for Spark Job Execution.

High Nginx latency typically refers to a situation where the response time of Nginx servers is significantly slower than expected. This may occur due to a variety of reasons, such as increased traffic or a misconfiguration in the server settings. High Nginx latency can cause website downtime, slow page load times, and poor user experience. It is important to address this incident type promptly to prevent any negative impact on users or business operations.


High Nginx Latency

The "High error rate on NGINX" incident type refers to a situation where the error rate on the NGINX server is above 1% for the last 5 minutes. This can result in degraded performance or downtime of the affected service, impacting user experience and potentially leading to lost revenue. The incident requires immediate attention and resolution to minimize the impact on users and prevent further damage.


High error rate on NGINX incident

This incident type involves an alert triggered when read latency is higher than usual for a MongoDB host. The alert is triggered when at least 100% of the average read latency values have been more than 2 deviations above the predicted values during the last 15 minutes. The incident requires monitoring and investigation to identify the root cause of the increased read latency and take appropriate actions to resolve it.


Read latency higher than average for MongoDB host.

```shell
export POD_NAME="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"

export SERVICE_NAME="PLACEHOLDER"

export RESOURCE_NAME="PLACEHOLDER"

export RESOURCE_VALUE="PLACEHOLDER"

export DEPLOYMENT_NAME="PLACEHOLDER"
```


### Check if Istio is installed in the cluster

```shell
kubectl get ns istio-system
```

### Check the status of the Istio pods and services

```shell
kubectl get pods -n istio-system

kubectl get svc -n istio-system
```

### Check the logs of the Istio sidecar proxy for the affected service

```shell
kubectl logs ${POD_NAME} -c istio-proxy -n ${NAMESPACE}
```

### Check the latency of the requests to the affected service

```shell
kubectl exec ${POD_NAME} -c ${CONTAINER_NAME} -n ${NAMESPACE} -- sh -c "curl http://localhost:8080/healthz"
```

### Check the Istio configuration for the affected service

```shell
kubectl get virtualservices -n ${NAMESPACE}

kubectl get destinationrules -n ${NAMESPACE}

kubectl get gateway -n ${NAMESPACE}
```

### Check if there are any network issues affecting the service

```shell
kubectl exec ${POD_NAME} -n ${NAMESPACE} -- ping ${SERVICE_NAME}
```

### Check if there are any CPU or memory issues affecting the service

```shell
kubectl top pods -n ${NAMESPACE}

kubectl describe pod ${POD_NAME} -n ${NAMESPACE}
```

### Resource constraints: The service may be under-resourced, leading to a slower response time for requests.

```shell


#!/bin/bash



# Check CPU and memory usage of the pod

CPU=$(kubectl top pod ${POD_NAME} | awk '{print $2}' | tail -n1)

MEMORY=$(kubectl top pod ${POD_NAME} | awk '{print $3}' | tail -n1)



# Check resource limits and requests set in the pod spec

CPU_LIMIT=$(kubectl describe pod ${POD_NAME} | grep "Limits" | awk '{print $4}')

MEMORY_LIMIT=$(kubectl describe pod ${POD_NAME} | grep "Limits" | awk '{print $6}')

CPU_REQUEST=$(kubectl describe pod ${POD_NAME} | grep "Requests" | awk '{print $4}')

MEMORY_REQUEST=$(kubectl describe pod ${POD_NAME} | grep "Requests" | awk '{print $6}')



# Check if CPU and memory usage are close to or exceeding the limits and requests

if (( $(echo "$CPU >= $CPU_LIMIT" |bc -l) )) || (( $(echo "$MEMORY >= $MEMORY_LIMIT" |bc -l) )) || (( $(echo "$CPU >= $CPU_REQUEST" |bc -l) )) || (( $(echo "$MEMORY >= $MEMORY_REQUEST" |bc -l) )); then

    echo "Pod ${POD_NAME} is using high CPU and/or memory resources, which may be causing the slow response time."

fi


```


### If the issue is related to hardware resource constraints, consider increasing the resources allocated to the affected service or adding more nodes to the cluster.

```shell


#!/bin/bash



# Set variables

DEPLOYMENT_NAME=${DEPLOYMENT_NAME}

RESOURCE_NAME=${RESOURCE_NAME}

RESOURCE_VALUE=${RESOURCE_VALUE}

# Use kubectl patch to patch the deployment
kubectl patch deployment $DEPLOYMENT_NAME -p '{"spec":{"template":{"spec":{"containers":{"resources":[{"limits":{"'$RESOURCE_NAME'":"'$RESOURCE_VALUE'"}, "requests":{"'$RESOURCE_NAME'":"'$RESOURCE_VALUE'"}}]}}}}}'

```


Istio High Request Latency

Overview

Parameters

Debug

Check if Istio is installed in the cluster

Check the status of the Istio pods and services

Check the logs of the Istio sidecar proxy for the affected service

Check the latency of the requests to the affected service

Check the Istio configuration for the affected service

Check if there are any network issues affecting the service

Check if there are any CPU or memory issues affecting the service

Resource constraints: The service may be under-resourced, leading to a slower response time for requests.

Repair

Learn more

Related Runbooks

High 4xx Errors on Traffic Server

High Latency Incident for Spark Job Execution.

High Nginx Latency

High error rate on NGINX incident

Support