---
id: f4ec2610-bee3-4709-999d-ff1bc297adb1
---

# Kubernetes HPA Status Incident
---

A Kubernetes HPA (Horizontal Pod Autoscaler) Status Incident refers to an issue where the autoscaling feature of Kubernetes, which automatically scales the number of pods in a replication controller, deployment, replica set or stateful set based on observed CPU utilization, is not functioning as expected. This can result in insufficient resources being provisioned to handle incoming load and potentially lead to service disruptions.

### Parameters
```shell
# Environment Variables

export NAMESPACE="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

export DEPLOYMENT_NAME="PLACEHOLDER"

export HPA_NAME="PLACEHOLDER"

export NODE_NAME="PLACEHOLDER"
```

## Debug


### List all deployments in the cluster
```shell
kubectl get deployments -n ${NAMESPACE}
```

### Describe a specific deployment
```shell
kubectl describe deployment ${DEPLOYMENT_NAME} -n ${NAMESPACE}
```

### List all horizontal pod autoscalers in the cluster
```shell
kubectl get hpa -n ${NAMESPACE}
```

### Describe a specific horizontal pod autoscaler
```shell
kubectl describe hpa ${HPA_NAME} -n ${NAMESPACE}
```
### List all nodes in the cluster
```shell
kubectl get nodes
```

### Check the CPU and memory usage of a specific node
```shell
kubectl top node ${NODE_NAME}
```

### Check the CPU and memory usage of a specific pod
```shell
kubectl top pod ${POD_NAME} -n ${NAMESPACE}
```

## Repair

export POD_NAME="PLACEHOLDER"

export HPA_NAME="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export DEPLOYMENT_NAME="PLACEHOLDER"
```

## Debug

### Get the list of nodes in the Kubernetes cluster
```shell
kubectl get nodes
```

### Get the list of pods in the Kubernetes cluster
```shell
kubectl get pods
```

### Get the list of HPA resources in the Kubernetes cluster
```shell
kubectl get hpa
```

### Get the logs of a specific pod
```shell
kubectl logs ${POD_NAME}
```

### Get the events in the Kubernetes cluster
```shell
kubectl get events
```

### Check the status of the HPA resource
```shell
kubectl describe hpa ${HPA_NAME}
```

### Check the CPU and memory usage of a specific pod
```shell
kubectl top pod ${POD_NAME}
```

### Check the CPU and memory usage of all pods in a specific namespace
```shell
kubectl top pod -n ${NAMESPACE}
```

### Check the status of the Kubernetes API server
```shell
kubectl cluster-info
```

### Check the status of the Kubernetes controllers
```shell
kubectl get deployments
```

### Check the status of the Kubernetes scheduler
```shell
kubectl get pods -n kube-system -l component=kube-scheduler
```

### Check the status of the Kubernetes etcd
```shell
kubectl get pods -n kube-system -l component=etcd
```

## Repair
### Restart the affected pods and monitor the HPA status to ensure it stabilizes.
```shell
#!/bin/bash

# Define the namespace and deployment name

namespace=${NAMESPACE}

deployment=${DEPLOYMENT}

# Get the pods that are part of the deployment

pods=$(kubectl get pods -n $namespace -l app=$deployment -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}')

# Restart the pods one by one and wait for them to become ready

for pod in $pods; do

  kubectl delete pod -n $namespace $pod

  echo "Restarting pod $pod..."

  until kubectl get pod -n $namespace $pod | grep "Running" >/dev/null; do sleep 1; done

  echo "Pod $pod is ready"

done
# Monitor the HPA status until it stabilizes

echo "Monitoring HPA status..."

until kubectl describe hpa -n $namespace $deployment | grep -E "Min Replicas:|Max Replicas:" >/dev/null; do sleep 1; done

echo "HPA status stabilized"
```


A Kubernetes HPA (Horizontal Pod Autoscaler) Status Incident refers to an issue where the autoscaling feature of Kubernetes, which automatically scales the number of pods in a replication controller, deployment, replica set or stateful set based on observed CPU utilization, is not functioning as expected. This can result in insufficient resources being provisioned to handle incoming load and potentially lead to service disruptions.


This incident type occurs when the number of 5xx errors on Traffic Server is higher than usual. This can indicate issues with server performance or connectivity problems. It requires investigation to identify the root cause and resolve the issue.


High 5xx Errors on Traffic Server

This incident type involves an issue with Kubernetes deployments where the expected number of pods to run is not matching the actual number of pods running. This can lead to alerts being triggered and potential disruptions in the system.


Kubernetes pods not starting - Deployment issue

This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Pods Pending incident indicates that one or more pods in a Kubernetes cluster are not running as expected and are in a pending state. This can happen due to various reasons such as resource constraints, scheduling issues, or network problems. This incident can impact the availability and performance of the application running on the Kubernetes cluster. It requires immediate attention to diagnose and resolve the underlying issue to ensure the pods are running as expected.


Kubernetes Pods Pending

A Kubernetes Pod ImagePullBackOff incident occurs when a pod in a Kubernetes cluster is unable to pull its container image. This can happen due to various reasons, such as incorrect image path or tag, or misconfigured image pulling credentials. This incident can cause the pod to fail to start and impact the availability of the application running in the pod. It requires investigation and resolution to ensure the pod can pull its container image and restart successfully.


Kubernetes Pod ImagePullBackOff Incident

```shell
# Environment Variables

export NAMESPACE="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

export DEPLOYMENT_NAME="PLACEHOLDER"

export HPA_NAME="PLACEHOLDER"

export NODE_NAME="PLACEHOLDER"
```


### List all deployments in the cluster

```shell
kubectl get deployments -n ${NAMESPACE}
```

### Describe a specific deployment

```shell
kubectl describe deployment ${DEPLOYMENT_NAME} -n ${NAMESPACE}
```

### List all horizontal pod autoscalers in the cluster

```shell
kubectl get hpa -n ${NAMESPACE}
```

### Describe a specific horizontal pod autoscaler

```shell
kubectl describe hpa ${HPA_NAME} -n ${NAMESPACE}
```

### List all nodes in the cluster

```shell
kubectl get nodes
```

### Check the CPU and memory usage of a specific node

```shell
kubectl top node ${NODE_NAME}
```

### Check the CPU and memory usage of a specific pod

```shell
kubectl top pod ${POD_NAME} -n ${NAMESPACE}
```


### Restart the affected pods and monitor the HPA status to ensure it stabilizes.

```shell
#!/bin/bash

# Define the namespace and deployment name

namespace=${NAMESPACE}

deployment=${DEPLOYMENT}

# Get the pods that are part of the deployment

pods=$(kubectl get pods -n $namespace -l app=$deployment -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}')

# Restart the pods one by one and wait for them to become ready

for pod in $pods; do

  kubectl delete pod -n $namespace $pod

  echo "Restarting pod $pod..."

  until kubectl get pod -n $namespace $pod | grep "Running" >/dev/null; do sleep 1; done

  echo "Pod $pod is ready"

done
# Monitor the HPA status until it stabilizes

echo "Monitoring HPA status..."

until kubectl describe hpa -n $namespace $deployment | grep -E "Min Replicas:|Max Replicas:" >/dev/null; do sleep 1; done

echo "HPA status stabilized"
```


Kubernetes HPA Status Incident

Overview

Parameters

Debug

List all deployments in the cluster

Describe a specific deployment

List all horizontal pod autoscalers in the cluster

Describe a specific horizontal pod autoscaler

List all nodes in the cluster

Check the CPU and memory usage of a specific node

Check the CPU and memory usage of a specific pod

Repair

Restart the affected pods and monitor the HPA status to ensure it stabilizes.

Learn more

Related Runbooks

High 5xx Errors on Traffic Server

Kubernetes pods not starting - Deployment issue

Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Pods Pending

Support