---
id: 06af2f32-e5ab-4933-8698-c6d29639dc8d
---
# High Request Duration on CoreDNS
---

This incident type refers to an alert triggered due to high request duration on CoreDNS, which is a DNS server that translates domain names to IP addresses. The alert indicates that the request duration process for CoreDNS is high, meaning that it is taking longer than expected to process DNS requests. This can cause delays or failures in the resolution of domain names, leading to potential service disruptions or outages.

### Parameters
```shell
# Environment Variables

export POD_NAME="PLACEHOLDER"

export SERVICE_NAME="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export DEPLOYMENT_NAME="PLACEHOLDER"

export CPU_LIMIT="PLACEHOLDER"

export MEMORY_LIMIT="PLACEHOLDER"

export CPU_REQUEST="PLACEHOLDER"

export MEMORY_REQUEST="PLACEHOLDER"
```

## Debug

### Get the name(s) of the pod(s) running CoreDNS
```shell
kubectl get pods -l k8s-app=kube-dns
```

### Check the logs of the CoreDNS pod(s) for errors
```shell
kubectl logs ${POD_NAME} -n kube-system
```

### Check the CPU and memory usage of the CoreDNS pod(s)
```shell
kubectl top pods -n kube-system | grep ${POD_NAME}
```

### Check the Kubernetes events related to the CoreDNS pod(s)
```shell
kubectl get events --sort-by=.metadata.creationTimestamp | grep ${POD_NAME}
```

### Check the status of the container(s) in the CoreDNS pod(s)
```shell
kubectl describe pod ${POD_NAME} -n kube-system | grep -A 2 -B 2 ContainerStatuses:
```

### Check the network latency between the CoreDNS pod(s) and other pods/services
```shell
kubectl exec ${POD_NAME} -n kube-system -- nslookup ${SERVICE_NAME}
```

### Check the Kubernetes services and endpoints related to CoreDNS
```shell
kubectl get svc,endpoints -n kube-system | grep kube-dns
```

## Repair

### Scale the CoreDNS deployment to handle the increased load.
```shell
kubectl scale deployment $DEPLOYMENT_NAME --replicas=$REPLICAS -n $NAMESPACE
```

### Update resources limits for CoreDNS deployment to handle the increased load.
```shell
kubectl patch deployment ${DEPLOYMENT_NAME} -n ${NAMESPACE} --type=json -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/resources", "value": {"limits": {"cpu": "${CPU_LIMIT}", "memory": "${MEMORY_LIMIT}"}, "requests": {"cpu": "${CPU_REQUEST}", "memory": "${MEMORY_REQUEST}"}}}]'
```

### Increase the resources allocated to the affected system to handle the increased load.
```shell
kubectl set resources deployment <deployment-name> --limits=<resource-limits>
```


This incident type refers to an alert triggered due to high request duration on CoreDNS, which is a DNS server that translates domain names to IP addresses. The alert indicates that the request duration process for CoreDNS is high, meaning that it is taking longer than expected to process DNS requests. This can cause delays or failures in the resolution of domain names, leading to potential service disruptions or outages.


```shell
# Environment Variables

export POD_NAME="PLACEHOLDER"

export SERVICE_NAME="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export DEPLOYMENT_NAME="PLACEHOLDER"

export CPU_LIMIT="PLACEHOLDER"

export MEMORY_LIMIT="PLACEHOLDER"

export CPU_REQUEST="PLACEHOLDER"

export MEMORY_REQUEST="PLACEHOLDER"
```


### Get the name(s) of the pod(s) running CoreDNS

```shell
kubectl get pods -l k8s-app=kube-dns
```

### Check the logs of the CoreDNS pod(s) for errors

```shell
kubectl logs ${POD_NAME} -n kube-system
```

### Check the CPU and memory usage of the CoreDNS pod(s)

```shell
kubectl top pods -n kube-system | grep ${POD_NAME}
```

### Check the Kubernetes events related to the CoreDNS pod(s)

```shell
kubectl get events --sort-by=.metadata.creationTimestamp | grep ${POD_NAME}
```

### Check the status of the container(s) in the CoreDNS pod(s)

```shell
kubectl describe pod ${POD_NAME} -n kube-system | grep -A 2 -B 2 ContainerStatuses:
```

### Check the network latency between the CoreDNS pod(s) and other pods/services

```shell
kubectl exec ${POD_NAME} -n kube-system -- nslookup ${SERVICE_NAME}
```

### Check the Kubernetes services and endpoints related to CoreDNS

```shell
kubectl get svc,endpoints -n kube-system | grep kube-dns
```


### Scale the CoreDNS deployment to handle the increased load.

```shell
kubectl scale deployment $DEPLOYMENT_NAME --replicas=$REPLICAS -n $NAMESPACE
```

### Update resources limits for CoreDNS deployment to handle the increased load.

```shell
kubectl patch deployment ${DEPLOYMENT_NAME} -n ${NAMESPACE} --type=json -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/resources", "value": {"limits": {"cpu": "${CPU_LIMIT}", "memory": "${MEMORY_LIMIT}"}, "requests": {"cpu": "${CPU_REQUEST}", "memory": "${MEMORY_REQUEST}"}}}]'
```

### Increase the resources allocated to the affected system to handle the increased load.

```shell
kubectl set resources deployment <deployment-name> --limits=<resource-limits>
```


High Request Duration on CoreDNS

Overview

Parameters

Debug

Get the name(s) of the pod(s) running CoreDNS

Check the logs of the CoreDNS pod(s) for errors

Check the CPU and memory usage of the CoreDNS pod(s)

Check the status of the container(s) in the CoreDNS pod(s)

Check the network latency between the CoreDNS pod(s) and other pods/services

Repair

Scale the CoreDNS deployment to handle the increased load.

Update resources limits for CoreDNS deployment to handle the increased load.

Increase the resources allocated to the affected system to handle the increased load.

Learn more

Related Runbooks

High 5xx Errors on Traffic Server

High 4xx Errors on Traffic Server

Increase of the errors/second rate for Tomcat server

Anomalous max processing time for Tomcat host

Product

Resources

Support

Company