---
id: d6e24d54-8bd8-44ac-8970-e1f8f20b65ea
---

# Istio Latency 99 Percentile Incident
---

This incident type refers to an issue where Istio latency has exceeded the 99th percentile, indicating that the slowest 1% of requests are taking longer than 1 second to complete. This can cause performance issues and impact the user experience. It requires immediate attention and investigation to resolve the issue and prevent any further impact.

### Parameters
```shell
export ISTIO_LABEL="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export ISTIO_MIXER_POD_NAME="PLACEHOLDER"

export ISTIO_TELEMETRY_POD_NAME="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

export DESIRED_REPLICAS="PLACEHOLDER"

export DEPLOYMENT_NAME="PLACEHOLDER"
```

## Debug

### 1. Check Istio version
```shell
istioctl version
```

### 2. Get Istio pod status
```shell
kubectl -n ${NAMESPACE} get pods -l ${ISTIO_LABEL}
```

### 3. Check Istio telemetry service status
```shell
kubectl -n ${NAMESPACE} get svc prometheus
```

### 4. Check Istio telemetry configuration
```shell
kubectl -n ${NAMESPACE} get svc istio-telemetry -o yaml
```

### 5. Check Istio Mixer config
```shell
kubectl -n ${NAMESPACE} get configmap istio -o yaml
```

### 6. Check Istio Mixer log
```shell
kubectl -n ${NAMESPACE} logs ${ISTIO_MIXER_POD_NAME}
```

### 7. Check Istio gateway configuration
```shell
kubectl -n ${NAMESPACE} get gateway
```

### 8. Check Istio virtual service configuration
```shell
kubectl -n ${NAMESPACE} get virtualservice
```

### 9. Check Istio destination rule configuration
```shell
kubectl -n ${NAMESPACE} get destinationrule
```

### 10. Check Istio metrics
```shell
kubectl -n ${NAMESPACE} exec -it ${ISTIO_TELEMETRY_POD_NAME} -c prometheus -- curl http://localhost:9090/metrics
```

### 11. Check Istio logs for specific pod
```shell
kubectl -n ${NAMESPACE} logs ${POD_NAME} -c istio-proxy
```

### Resource constraints: The Istio service may be experiencing resource constraints. For example, if there are not enough resources allocated to the service, it could result in increased latency.
```shell


#!/bin/bash



# Set the namespace and deployment name for Istio

NAMESPACE=${NAMESPACE}

DEPLOYMENT_NAME=${DEPLOYMENT_NAME}



# Check the resource limits and usage for the Istio deployment

echo "Checking resource limits and usage for Istio deployment..."

kubectl describe deployment $DEPLOYMENT_NAME -n $NAMESPACE | grep -E 'Limits|Requests|CPU|Memory'



# Check the resource requests and limits for the Istio containers

echo "Checking resource requests and limits for Istio containers..."

kubectl describe pods -l app=$DEPLOYMENT_NAME -n $NAMESPACE | grep -E 'Limits|Requests|CPU|Memory'



# Check the resource usage for the Istio containers

echo "Checking resource usage for Istio containers..."

kubectl top pods -n $NAMESPACE | grep $DEPLOYMENT_NAME


```

## Repair

### Scale up or down resources as necessary to ensure optimal system performance.
```shell


#!/bin/bash



# Set the deployment name

DEPLOYMENT=${DEPLOYMENT_NAME}



# Set the desired number of replicas

REPLICAS=${DESIRED_REPLICAS}



# Scale the deployment to the desired number of replicas

kubectl scale deployment $DEPLOYMENT --replicas=$REPLICAS


```

This incident type refers to an issue where Istio latency has exceeded the 99th percentile, indicating that the slowest 1% of requests are taking longer than 1 second to complete. This can cause performance issues and impact the user experience. It requires immediate attention and investigation to resolve the issue and prevent any further impact.


This incident type occurs when the number of 4xx errors on Traffic Server is at an anomalous level, higher than usual. It could be an indicator of an issue with the server or an increase in traffic that is causing errors. It requires investigation and resolution to ensure the server is functioning correctly and not impacting users.


High 4xx Errors on Traffic Server

This incident type refers to a situation where a Spark cluster experiences performance bottlenecks when it is subjected to peak loads. In other words, the Spark cluster struggles to handle the high volume of requests it receives during times of heavy traffic or increased demand. This can lead to slower processing times, delays, or even system crashes. Identifying and resolving the root cause of the bottlenecks is crucial to ensure the smooth functioning of the Spark cluster during peak loads.


Spark cluster bottlenecks during peak loads.

This incident type indicates that there is a high latency issue in the execution of a Spark job. Spark is a distributed computing framework that is used for processing large datasets. High latency in this context means that the time taken to execute the Spark job is significantly longer than expected or normal. This can result in delays in processing data and can impact the performance of the application or system that is utilizing Spark.


High Latency Incident for Spark Job Execution.

High Nginx latency typically refers to a situation where the response time of Nginx servers is significantly slower than expected. This may occur due to a variety of reasons, such as increased traffic or a misconfiguration in the server settings. High Nginx latency can cause website downtime, slow page load times, and poor user experience. It is important to address this incident type promptly to prevent any negative impact on users or business operations.


High Nginx Latency

This incident type refers to a situation where the write latency for a MongoDB database is higher than the average. This might cause delays in write operations and can negatively impact the performance of the system. The incident might be triggered by an alert that indicates that the write latency has exceeded a certain threshold, and it requires immediate attention from a software engineer to identify and resolve the underlying issue. The incident can be resolved by identifying and fixing the root cause of the latency issue, and ensuring that the system is functioning within acceptable performance parameters.


MongoDB Write Latency Higher Than Average

```shell
export ISTIO_LABEL="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export ISTIO_MIXER_POD_NAME="PLACEHOLDER"

export ISTIO_TELEMETRY_POD_NAME="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

export DESIRED_REPLICAS="PLACEHOLDER"

export DEPLOYMENT_NAME="PLACEHOLDER"
```


### 1. Check Istio version

```shell
istioctl version
```

### 2. Get Istio pod status

```shell
kubectl -n ${NAMESPACE} get pods -l ${ISTIO_LABEL}
```

### 3. Check Istio telemetry service status

```shell
kubectl -n ${NAMESPACE} get svc prometheus
```

### 4. Check Istio telemetry configuration

```shell
kubectl -n ${NAMESPACE} get svc istio-telemetry -o yaml
```

### 5. Check Istio Mixer config

```shell
kubectl -n ${NAMESPACE} get configmap istio -o yaml
```

### 6. Check Istio Mixer log

```shell
kubectl -n ${NAMESPACE} logs ${ISTIO_MIXER_POD_NAME}
```

### 7. Check Istio gateway configuration

```shell
kubectl -n ${NAMESPACE} get gateway
```

### 8. Check Istio virtual service configuration

```shell
kubectl -n ${NAMESPACE} get virtualservice
```

### 9. Check Istio destination rule configuration

```shell
kubectl -n ${NAMESPACE} get destinationrule
```

### 10. Check Istio metrics

```shell
kubectl -n ${NAMESPACE} exec -it ${ISTIO_TELEMETRY_POD_NAME} -c prometheus -- curl http://localhost:9090/metrics
```

### 11. Check Istio logs for specific pod

```shell
kubectl -n ${NAMESPACE} logs ${POD_NAME} -c istio-proxy
```

### Resource constraints: The Istio service may be experiencing resource constraints. For example, if there are not enough resources allocated to the service, it could result in increased latency.

```shell


#!/bin/bash



# Set the namespace and deployment name for Istio

NAMESPACE=${NAMESPACE}

DEPLOYMENT_NAME=${DEPLOYMENT_NAME}



# Check the resource limits and usage for the Istio deployment

echo "Checking resource limits and usage for Istio deployment..."

kubectl describe deployment $DEPLOYMENT_NAME -n $NAMESPACE | grep -E 'Limits|Requests|CPU|Memory'



# Check the resource requests and limits for the Istio containers

echo "Checking resource requests and limits for Istio containers..."

kubectl describe pods -l app=$DEPLOYMENT_NAME -n $NAMESPACE | grep -E 'Limits|Requests|CPU|Memory'



# Check the resource usage for the Istio containers

echo "Checking resource usage for Istio containers..."

kubectl top pods -n $NAMESPACE | grep $DEPLOYMENT_NAME


```


### Scale up or down resources as necessary to ensure optimal system performance.

```shell


#!/bin/bash



# Set the deployment name

DEPLOYMENT=${DEPLOYMENT_NAME}



# Set the desired number of replicas

REPLICAS=${DESIRED_REPLICAS}



# Scale the deployment to the desired number of replicas

kubectl scale deployment $DEPLOYMENT --replicas=$REPLICAS


```


Istio Latency 99 Percentile Incident

Overview

Parameters

Debug

1. Check Istio version

2. Get Istio pod status

3. Check Istio telemetry service status

4. Check Istio telemetry configuration

5. Check Istio Mixer config

6. Check Istio Mixer log

7. Check Istio gateway configuration

8. Check Istio virtual service configuration

9. Check Istio destination rule configuration

10. Check Istio metrics

11. Check Istio logs for specific pod

Resource constraints: The Istio service may be experiencing resource constraints. For example, if there are not enough resources allocated to the service, it could result in increased latency.

Repair

Scale up or down resources as necessary to ensure optimal system performance.

Learn more

Related Runbooks

High 4xx Errors on Traffic Server

Spark cluster bottlenecks during peak loads.

High Latency Incident for Spark Job Execution.

High Nginx Latency

Support