---
id: 0b381e1b-5d6c-47d2-8b31-1e5d93bb1b60
---

# Kubernetes - DNS Resolution Failure
---

DNS resolution failure is a common incident type that occurs when services are unable to resolve domain names into IP addresses, leading to communication failure. DNS (Domain Name System) is responsible for translating human-readable domain names into machine-readable IP addresses. When this translation fails, services are unable to communicate with each other, resulting in outages or performance degradation. This incident type can be caused by various factors such as misconfigured DNS settings, DNS server failures, network connectivity issues, or cyber attacks.

### Parameters
```shell
export DNS_NAME="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

export DNS_SERVER_IP="PLACEHOLDER"

export EXAMPLE_COM="PLACEHOLDER"

export DNS_SERVER_DEPLOYMENT_NAME="PLACEHOLDER"
```

## Debug

### Check if the kube-dns service is running
```shell
kubectl get svc kube-dns -n kube-system
```

### View the logs of the kube-dns pod
```shell
kubectl logs ${POD_NAME} -n kube-system
```

### Check if the DNS service IP is correctly set
```shell
kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}{"\n"}'
```

### Check if the DNS pod is running and ready
```shell
kubectl get pods -n kube-system -l k8s-app=kube-dns
```

### Check if the DNS configuration is correct
```shell
kubectl exec ${POD_NAME} -n kube-system -- cat /etc/resolv.conf
```

### Check if the DNS server is reachable from the cluster
```shell
kubectl run -it --rm dnsutils --image=tutum/dnsutils --restart=Never -- nslookup ${DNS_NAME}
```

### Check if the DNS resolution is working for a specific pod
```shell
kubectl exec ${POD_NAME} -- nslookup ${DNS_NAME}
```

### Check the DNS server configuration and ensure that it is configured correctly. Verify that the DNS server is operational and responding to queries.
```shell


#!/bin/bash



# Set the DNS server IP address

DNS_SERVER=${DNS_SERVER_IP}



# Verify that the DNS server is operational and responding to queries

ping -c 5 $DNS_SERVER

if [ $? -ne 0 ]; then

    echo "DNS server is not responding"

    exit 1

fi



# Check the DNS server configuration

nslookup google.com $DNS_SERVER

if [ $? -ne 0 ]; then

    echo "DNS server configuration is incorrect"

    exit 1

fi



echo "DNS server configuration is correct"


```

## Repair

### Restart the DNS server to clear any cache or temporary issues.
```shell


#!/bin/bash



# Set the DNS server deployment name

deployment_name=${DNS_SERVER_DEPLOYMENT_NAME}



# Retrieve the current number of replicas

replicas=$(kubectl get deployment $deployment_name -o jsonpath='{.spec.replicas}')



# Scale the deployment down to 0 replicas

kubectl scale deployment $deployment_name --replicas=0



# Wait for the deployment to be fully scaled down

while [[ $(kubectl get deployment $deployment_name -o jsonpath='{.status.readyReplicas}') -ne 0 ]]; do

    sleep 5

done



# Scale the deployment back up to the original number of replicas

kubectl scale deployment $deployment_name --replicas=$replicas



# Wait for the deployment to be fully scaled up

while [[ $(kubectl get deployment $deployment_name -o jsonpath='{.status.readyReplicas}') -ne $replicas ]]; do

    sleep 5

done



echo "DNS server deployment has been restarted."


```


DNS resolution failure is a common incident type that occurs when services are unable to resolve domain names into IP addresses, leading to communication failure. DNS (Domain Name System) is responsible for translating human-readable domain names into machine-readable IP addresses. When this translation fails, services are unable to communicate with each other, resulting in outages or performance degradation. This incident type can be caused by various factors such as misconfigured DNS settings, DNS server failures, network connectivity issues, or cyber attacks.


This incident type occurs when a software system is unable to establish a connection to a PostgreSQL database due to the database server refusing the connection. This can be caused by a variety of factors such as network issues, incorrect credentials, or the database server being unavailable. This can lead to the system being unable to perform necessary database operations, which can impact the functionality of the software system.


PostgreSQL database connection refused.

This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

A Kubernetes Replicaset Incomplete incident typically occurs when a specific number of pods that should be running are not, due to reasons such as failed pod initialization, unavailability of resources in the cluster, or inability to pull the image. This incident is usually triggered when the difference between desired and running pods is greater than zero, and it can be detected through monitoring tools like Datadog.


Kubernetes Replicaset Incomplete

Kubernetes Pods Pending incident indicates that one or more pods in a Kubernetes cluster are not running as expected and are in a pending state. This can happen due to various reasons such as resource constraints, scheduling issues, or network problems. This incident can impact the availability and performance of the application running on the Kubernetes cluster. It requires immediate attention to diagnose and resolve the underlying issue to ensure the pods are running as expected.


Kubernetes Pods Pending

This incident type involves nodes in a Kubernetes cluster that are experiencing network unavailability, meaning they are not accessible. This could be due to a misconfiguration, route exhaustion, or a physical problem with the network connection to the hardware. It is a high urgency incident that requires immediate attention to restore network connectivity to the affected nodes.


Kubernetes Nodes with Network Unavailable

```shell
export DNS_NAME="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

export DNS_SERVER_IP="PLACEHOLDER"

export EXAMPLE_COM="PLACEHOLDER"

export DNS_SERVER_DEPLOYMENT_NAME="PLACEHOLDER"
```


### Check if the kube-dns service is running

```shell
kubectl get svc kube-dns -n kube-system
```

### View the logs of the kube-dns pod

```shell
kubectl logs ${POD_NAME} -n kube-system
```

### Check if the DNS service IP is correctly set

```shell
kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}{"\n"}'
```

### Check if the DNS pod is running and ready

```shell
kubectl get pods -n kube-system -l k8s-app=kube-dns
```

### Check if the DNS configuration is correct

```shell
kubectl exec ${POD_NAME} -n kube-system -- cat /etc/resolv.conf
```

### Check if the DNS server is reachable from the cluster

```shell
kubectl run -it --rm dnsutils --image=tutum/dnsutils --restart=Never -- nslookup ${DNS_NAME}
```

### Check if the DNS resolution is working for a specific pod

```shell
kubectl exec ${POD_NAME} -- nslookup ${DNS_NAME}
```

### Check the DNS server configuration and ensure that it is configured correctly. Verify that the DNS server is operational and responding to queries.

```shell


#!/bin/bash



# Set the DNS server IP address

DNS_SERVER=${DNS_SERVER_IP}



# Verify that the DNS server is operational and responding to queries

ping -c 5 $DNS_SERVER

if [ $? -ne 0 ]; then

    echo "DNS server is not responding"

    exit 1

fi



# Check the DNS server configuration

nslookup google.com $DNS_SERVER

if [ $? -ne 0 ]; then

    echo "DNS server configuration is incorrect"

    exit 1

fi



echo "DNS server configuration is correct"


```


### Restart the DNS server to clear any cache or temporary issues.

```shell


#!/bin/bash



# Set the DNS server deployment name

deployment_name=${DNS_SERVER_DEPLOYMENT_NAME}



# Retrieve the current number of replicas

replicas=$(kubectl get deployment $deployment_name -o jsonpath='{.spec.replicas}')



# Scale the deployment down to 0 replicas

kubectl scale deployment $deployment_name --replicas=0



# Wait for the deployment to be fully scaled down

while [[ $(kubectl get deployment $deployment_name -o jsonpath='{.status.readyReplicas}') -ne 0 ]]; do

    sleep 5

done



# Scale the deployment back up to the original number of replicas

kubectl scale deployment $deployment_name --replicas=$replicas



# Wait for the deployment to be fully scaled up

while [[ $(kubectl get deployment $deployment_name -o jsonpath='{.status.readyReplicas}') -ne $replicas ]]; do

    sleep 5

done



echo "DNS server deployment has been restarted."


```


Kubernetes - DNS Resolution Failure

Overview

Parameters

Debug

Check if the kube-dns service is running

View the logs of the kube-dns pod

Check if the DNS service IP is correctly set

Check if the DNS pod is running and ready

Check if the DNS configuration is correct

Check if the DNS server is reachable from the cluster

Check if the DNS resolution is working for a specific pod

Check the DNS server configuration and ensure that it is configured correctly. Verify that the DNS server is operational and responding to queries.

Repair

Restart the DNS server to clear any cache or temporary issues.

Learn more

Related Runbooks

PostgreSQL database connection refused.

Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Replicaset Incomplete

Kubernetes Pods Pending

Support