---
id: c4f9c1f5-1639-4b06-8f0c-452bb2eb7e58
---

# Kubernetes HPA Status Incident
---

A Kubernetes HPA (Horizontal Pod Autoscaler) Status Incident refers to an issue where the autoscaling feature of Kubernetes, which automatically scales the number of pods in a replication controller, deployment, replica set or stateful set based on observed CPU utilization, is not functioning as expected. This can result in insufficient resources being provisioned to handle incoming load and potentially lead to service disruptions.

### Parameters
```shell
# Environment Variables
export NAMESPACE="PLACEHOLDER"
export HPA_NAME="PLACEHOLDER"
export POD_NAME="PLACEHOLDER"
export MAX_PODS="PLACEHOLDER"
export DEPLOYMENT_NAME="PLACEHOLDER"
export CLUSTER_NAME="PLACEHOLDER"
export RESOURCE_NAME="PLACEHOLDER"
export RESOURCE_TYPE="PLACEHOLDER"
export METRIC_THRESHOLD="PLACEHOLDER"
export METRIC_NAME="PLACEHOLDER"
```

## Debug

### Check the status of all HorizontalPodAutoscalers in the namespace
```shell
kubectl get hpa -n ${NAMESPACE}
```

### Check the status of a specific HorizontalPodAutoscaler in the namespace
```shell
kubectl describe hpa/${HPA_NAME} -n ${NAMESPACE}
```

### Check the status of all pods in the namespace
```shell
kubectl get pods -n ${NAMESPACE}
```

### Check the CPU and memory usage of a specific pod in the namespace
```shell
kubectl top pod ${POD_NAME} -n ${NAMESPACE}
```

### Check the logs of a specific pod in the namespace
```shell
kubectl logs ${POD_NAME} -n ${NAMESPACE}
```

### Check the status of the Kubernetes cluster metrics server
```shell
kubectl get deployment metrics-server -n kube-system
```

### Insufficient resources available in the cluster to support the desired number of pods.
```shell

#!/bin/bash

# Define variables
NAMESPACE=${NAMESPACE}
DEPLOYMENT=${DEPLOYMENT_NAME}
MAX_PODS=${MAX_PODS}

# Get the current number of replicas and pods
CURRENT_REPLICAS=$(kubectl get deployment $DEPLOYMENT -n $NAMESPACE -o=jsonpath='{.spec.replicas}')
CURRENT_PODS=$(kubectl get pods -n $NAMESPACE | grep $DEPLOYMENT | wc -l)

# Calculate the maximum number of pods that can be created based on available resources
MAX_AVAILABLE_PODS=$(kubectl describe nodes | grep "Allocatable pods" | awk '{print $3}')
MAX_AVAILABLE_PODS=$(($MAX_AVAILABLE_PODS/$MAX_PODS))

# Check if the current number of pods exceeds the maximum available pods
if [ $CURRENT_PODS -gt $MAX_AVAILABLE_PODS ]; then
    echo "Error: There are insufficient resources available in the Kubernetes cluster to support the desired number of pods."
    echo "Current replicas: $CURRENT_REPLICAS"
    echo "Current pods: $CURRENT_PODS"
    echo "Max available pods: $MAX_AVAILABLE_PODS"
else
    echo "Everything looks good! Current replicas: $CURRENT_REPLICAS, current pods: $CURRENT_PODS, max available pods: $MAX_AVAILABLE_PODS."
fi

```
## Repair
---
### Verify that the HPA is correctly configured for the deployment or stateful set in question, and that the minimum and maximum number of pods are set appropriately.
```shell
bash
#!/bin/bash

# Define variables
CLUSTER_NAME=${CLUSTER_NAME}
NAMESPACE=${NAMESPACE}
RESOURCE_TYPE=${RESOURCE_TYPE}
RESOURCE_NAME=${RESOURCE_NAME}

# Verify that the HPA is correctly configured for the deployment or stateful set
HPA_MIN=$(kubectl -n $NAMESPACE get hpa -l "app.kubernetes.io/name=$RESOURCE_NAME" -o jsonpath='{.items[0].spec.minReplicas}')
HPA_MAX=$(kubectl -n $NAMESPACE get hpa -l "app.kubernetes.io/name=$RESOURCE_NAME" -o jsonpath='{.items[0].spec.maxReplicas}')

if [[ -z $HPA_MIN || -z $HPA_MAX ]]; then
  echo "Error: HPA is not configured for $RESOURCE_TYPE $RESOURCE_NAME in namespace $NAMESPACE"
  exit 1
fi

if (( $HPA_MIN < 1 || $HPA_MAX < $HPA_MIN )); then
  echo "Error: Invalid HPA configuration for $RESOURCE_TYPE $RESOURCE_NAME in namespace $NAMESPACE"
  exit 1
fi

echo "HPA is correctly configured for $RESOURCE_TYPE $RESOURCE_NAME in namespace $NAMESPACE. Min replicas: $HPA_MIN, Max replicas: $HPA_MAX"
exit 0

```

### Check the metrics used by the HPA to determine whether to scale up or down the number of pods. Ensure that the metrics are correctly defined and that they reflect the actual resource utilization of the pods.
```shell
bash
#!/bin/bash

# Define the variables
NAMESPACE=${NAMESPACE}
HPA_NAME=${HPA_NAME}
METRIC_NAME=${METRIC_NAME}
METRIC_THRESHOLD=${METRIC_THRESHOLD}

# Get the current status of the HPA
HPA_STATUS=$(kubectl get hpa $HPA_NAME -n $NAMESPACE -o jsonpath='{.status.conditions[0].status}')

# Check if the HPA is scaled up or down
if [[ "$HPA_STATUS" == "True" ]]; then
  echo "The HPA is scaled up."
else
  echo "The HPA is scaled down."
fi

# Get the current metrics used by the HPA
METRIC_VALUE=$(kubectl get hpa $HPA_NAME -n $NAMESPACE -o jsonpath="{.status.currentMetrics[?(@.type=='Object')].currentValue}")

# Check if the metrics are above the threshold
if (( $(echo "$METRIC_VALUE > $METRIC_THRESHOLD" | bc -l) )); then
  echo "The $METRIC_NAME metric is above the threshold."
else
  echo "The $METRIC_NAME metric is below the threshold."
fi

```

### If the metrics are not available or not working as expected, consider using alternative metrics to determine scaling. For example, you can use custom metrics or metrics from external monitoring systems.
```shell

#!/bin/bash

# Define the necessary variables
NAMESPACE=${NAMESPACE}
DEPLOYMENT=${DEPLOYMENT_NAME}
METRIC=${METRIC_NAME}

# Check if the metrics are available
if kubectl top pods -n $NAMESPACE | grep $DEPLOYMENT | awk '{print $2}' | grep -qE '[0-9]'; then
  echo "Metrics for $DEPLOYMENT in namespace $NAMESPACE are available."
else
  echo "Metrics for $DEPLOYMENT in namespace $NAMESPACE are not available."
fi

# Check if the metrics are working as expected
if kubectl get hpa -n $NAMESPACE | grep $DEPLOYMENT | awk '{print $5}' | grep -qE '[0-9]'; then
  echo "Metrics for $DEPLOYMENT in namespace $NAMESPACE are working as expected."
else
  echo "Metrics for $DEPLOYMENT in namespace $NAMESPACE are not working as expected. Consider using alternative metrics."
  
  # Check if custom metrics are available
  if kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/ | grep -qE "$METRIC"; then
    echo "Custom metrics are available. Consider using these metrics for scaling."
  else
    echo "Custom metrics are not available. Consider using metrics from external monitoring systems."
  fi
fi

```

A Kubernetes HPA (Horizontal Pod Autoscaler) Status Incident refers to an issue where the autoscaling feature of Kubernetes, which automatically scales the number of pods in a replication controller, deployment, replica set or stateful set based on observed CPU utilization, is not functioning as expected. This can result in insufficient resources being provisioned to handle incoming load and potentially lead to service disruptions.


This incident type occurs when the number of 5xx errors on Traffic Server is higher than usual. This can indicate issues with server performance or connectivity problems. It requires investigation to identify the root cause and resolve the issue.


High 5xx Errors on Traffic Server

This incident type involves an issue with Kubernetes deployments where the expected number of pods to run is not matching the actual number of pods running. This can lead to alerts being triggered and potential disruptions in the system.


Kubernetes pods not starting - Deployment issue

This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Pods Pending incident indicates that one or more pods in a Kubernetes cluster are not running as expected and are in a pending state. This can happen due to various reasons such as resource constraints, scheduling issues, or network problems. This incident can impact the availability and performance of the application running on the Kubernetes cluster. It requires immediate attention to diagnose and resolve the underlying issue to ensure the pods are running as expected.


Kubernetes Pods Pending

A Kubernetes Pod ImagePullBackOff incident occurs when a pod in a Kubernetes cluster is unable to pull its container image. This can happen due to various reasons, such as incorrect image path or tag, or misconfigured image pulling credentials. This incident can cause the pod to fail to start and impact the availability of the application running in the pod. It requires investigation and resolution to ensure the pod can pull its container image and restart successfully.


Kubernetes Pod ImagePullBackOff Incident

```shell
# Environment Variables
export NAMESPACE="PLACEHOLDER"
export HPA_NAME="PLACEHOLDER"
export POD_NAME="PLACEHOLDER"
export MAX_PODS="PLACEHOLDER"
export DEPLOYMENT_NAME="PLACEHOLDER"
export CLUSTER_NAME="PLACEHOLDER"
export RESOURCE_NAME="PLACEHOLDER"
export RESOURCE_TYPE="PLACEHOLDER"
export METRIC_THRESHOLD="PLACEHOLDER"
export METRIC_NAME="PLACEHOLDER"
```


### Check the status of all HorizontalPodAutoscalers in the namespace

```shell
kubectl get hpa -n ${NAMESPACE}
```

### Check the status of a specific HorizontalPodAutoscaler in the namespace

```shell
kubectl describe hpa/${HPA_NAME} -n ${NAMESPACE}
```

### Check the status of all pods in the namespace

```shell
kubectl get pods -n ${NAMESPACE}
```

### Check the CPU and memory usage of a specific pod in the namespace

```shell
kubectl top pod ${POD_NAME} -n ${NAMESPACE}
```

### Check the logs of a specific pod in the namespace

```shell
kubectl logs ${POD_NAME} -n ${NAMESPACE}
```

### Check the status of the Kubernetes cluster metrics server

```shell
kubectl get deployment metrics-server -n kube-system
```

### Insufficient resources available in the cluster to support the desired number of pods.

```shell

#!/bin/bash

# Define variables
NAMESPACE=${NAMESPACE}
DEPLOYMENT=${DEPLOYMENT_NAME}
MAX_PODS=${MAX_PODS}

# Get the current number of replicas and pods
CURRENT_REPLICAS=$(kubectl get deployment $DEPLOYMENT -n $NAMESPACE -o=jsonpath='{.spec.replicas}')
CURRENT_PODS=$(kubectl get pods -n $NAMESPACE | grep $DEPLOYMENT | wc -l)

# Calculate the maximum number of pods that can be created based on available resources
MAX_AVAILABLE_PODS=$(kubectl describe nodes | grep "Allocatable pods" | awk '{print $3}')
MAX_AVAILABLE_PODS=$(($MAX_AVAILABLE_PODS/$MAX_PODS))

# Check if the current number of pods exceeds the maximum available pods
if [ $CURRENT_PODS -gt $MAX_AVAILABLE_PODS ]; then
    echo "Error: There are insufficient resources available in the Kubernetes cluster to support the desired number of pods."
    echo "Current replicas: $CURRENT_REPLICAS"
    echo "Current pods: $CURRENT_PODS"
    echo "Max available pods: $MAX_AVAILABLE_PODS"
else
    echo "Everything looks good! Current replicas: $CURRENT_REPLICAS, current pods: $CURRENT_PODS, max available pods: $MAX_AVAILABLE_PODS."
fi

```


### Verify that the HPA is correctly configured for the deployment or stateful set in question, and that the minimum and maximum number of pods are set appropriately.

```shell
bash
#!/bin/bash

# Define variables
CLUSTER_NAME=${CLUSTER_NAME}
NAMESPACE=${NAMESPACE}
RESOURCE_TYPE=${RESOURCE_TYPE}
RESOURCE_NAME=${RESOURCE_NAME}

# Verify that the HPA is correctly configured for the deployment or stateful set
HPA_MIN=$(kubectl -n $NAMESPACE get hpa -l "app.kubernetes.io/name=$RESOURCE_NAME" -o jsonpath='{.items[0].spec.minReplicas}')
HPA_MAX=$(kubectl -n $NAMESPACE get hpa -l "app.kubernetes.io/name=$RESOURCE_NAME" -o jsonpath='{.items[0].spec.maxReplicas}')

if [[ -z $HPA_MIN || -z $HPA_MAX ]]; then
  echo "Error: HPA is not configured for $RESOURCE_TYPE $RESOURCE_NAME in namespace $NAMESPACE"
  exit 1
fi

if (( $HPA_MIN < 1 || $HPA_MAX < $HPA_MIN )); then
  echo "Error: Invalid HPA configuration for $RESOURCE_TYPE $RESOURCE_NAME in namespace $NAMESPACE"
  exit 1
fi

echo "HPA is correctly configured for $RESOURCE_TYPE $RESOURCE_NAME in namespace $NAMESPACE. Min replicas: $HPA_MIN, Max replicas: $HPA_MAX"
exit 0

```

### Check the metrics used by the HPA to determine whether to scale up or down the number of pods. Ensure that the metrics are correctly defined and that they reflect the actual resource utilization of the pods.

```shell
bash
#!/bin/bash

# Define the variables
NAMESPACE=${NAMESPACE}
HPA_NAME=${HPA_NAME}
METRIC_NAME=${METRIC_NAME}
METRIC_THRESHOLD=${METRIC_THRESHOLD}

# Get the current status of the HPA
HPA_STATUS=$(kubectl get hpa $HPA_NAME -n $NAMESPACE -o jsonpath='{.status.conditions[0].status}')

# Check if the HPA is scaled up or down
if [[ "$HPA_STATUS" == "True" ]]; then
  echo "The HPA is scaled up."
else
  echo "The HPA is scaled down."
fi

# Get the current metrics used by the HPA
METRIC_VALUE=$(kubectl get hpa $HPA_NAME -n $NAMESPACE -o jsonpath="{.status.currentMetrics[?(@.type=='Object')].currentValue}")

# Check if the metrics are above the threshold
if (( $(echo "$METRIC_VALUE > $METRIC_THRESHOLD" | bc -l) )); then
  echo "The $METRIC_NAME metric is above the threshold."
else
  echo "The $METRIC_NAME metric is below the threshold."
fi

```

### If the metrics are not available or not working as expected, consider using alternative metrics to determine scaling. For example, you can use custom metrics or metrics from external monitoring systems.

```shell

#!/bin/bash

# Define the necessary variables
NAMESPACE=${NAMESPACE}
DEPLOYMENT=${DEPLOYMENT_NAME}
METRIC=${METRIC_NAME}

# Check if the metrics are available
if kubectl top pods -n $NAMESPACE | grep $DEPLOYMENT | awk '{print $2}' | grep -qE '[0-9]'; then
  echo "Metrics for $DEPLOYMENT in namespace $NAMESPACE are available."
else
  echo "Metrics for $DEPLOYMENT in namespace $NAMESPACE are not available."
fi

# Check if the metrics are working as expected
if kubectl get hpa -n $NAMESPACE | grep $DEPLOYMENT | awk '{print $5}' | grep -qE '[0-9]'; then
  echo "Metrics for $DEPLOYMENT in namespace $NAMESPACE are working as expected."
else
  echo "Metrics for $DEPLOYMENT in namespace $NAMESPACE are not working as expected. Consider using alternative metrics."
  
  # Check if custom metrics are available
  if kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/ | grep -qE "$METRIC"; then
    echo "Custom metrics are available. Consider using these metrics for scaling."
  else
    echo "Custom metrics are not available. Consider using metrics from external monitoring systems."
  fi
fi

```


Kubernetes HPA Status Incident

Overview

Parameters

Debug

Check the status of all HorizontalPodAutoscalers in the namespace

Check the status of a specific HorizontalPodAutoscaler in the namespace

Check the status of all pods in the namespace

Check the CPU and memory usage of a specific pod in the namespace

Check the logs of a specific pod in the namespace

Check the status of the Kubernetes cluster metrics server

Insufficient resources available in the cluster to support the desired number of pods.

Repair

Verify that the HPA is correctly configured for the deployment or stateful set in question, and that the minimum and maximum number of pods are set appropriately.

Check the metrics used by the HPA to determine whether to scale up or down the number of pods. Ensure that the metrics are correctly defined and that they reflect the actual resource utilization of the pods.

If the metrics are not available or not working as expected, consider using alternative metrics to determine scaling. For example, you can use custom metrics or metrics from external monitoring systems.

Learn more

Related Runbooks

High 5xx Errors on Traffic Server

Kubernetes pods not starting - Deployment issue

Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Pods Pending

Support