---
id: 8b408e01-f40a-4ecb-84e3-696c36092b60
---

# Kubernetes - Available Memory for Limits in percentage Low
---

This incident type refers to situations where the available memory for limits in percentage of a Kubernetes cluster is low. This can cause issues with the performance and stability of the cluster, potentially leading to downtime or other problems. The incident may be triggered by an automated query or monitoring tool that checks for certain metrics or thresholds. It is important to address and resolve this issue as soon as possible to ensure the continued health and reliability of the Kubernetes cluster.

### Parameters
```shell
# Environment Variables

export CONTAINER_NAME="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

export CONTEXT_NAME="PLACEHOLDER"

export POD_LABEL="PLACEHOLDER"

export MEMORY_THRESHOLD="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"
```

## Debug

### Check the available memory for limits in percentage for all nodes in the cluster
```shell
kubectl top nodes | awk '{print $1,$4}' | column -t
```

### Check the available memory for limits in percentage for all pods in the cluster
```shell
kubectl top pods --all-namespaces | awk '{print $1,$2,$4}' | column -t
```

### Check the resource requests and limits for all pods in the cluster
```shell
kubectl describe pods | grep -e 'Name:\|Limits\|Requests'
```

### Check the memory usage for specific container in a pod
```shell
kubectl exec -it ${POD_NAME} -c ${CONTAINER_NAME} -- sh -c "free -m"
```

### Check the logs for a specific pod to see if any errors or issues are occurring
```shell
kubectl logs ${POD_NAME}
```

### Check the event log for the cluster to see if any events are being generated related to memory usage
```shell
kubectl get events
```

### Check the cluster autoscaler to see if it is scaling up or down properly based on resource usage
```shell
kubectl get hpa
```

### Check the cluster pods and nodes status
```shell
kubectl get pods,nodes
```

## Repair

### Check the resource requests and limits for the pods running on the Kubernetes cluster to ensure they are properly configured. Adjust any values as necessary, keeping in mind the available resources on the cluster.
```shell
bash

#!/bin/bash

# Set the namespace where the resources are located

NAMESPACE=${NAMESPACE}

# Loop through all pods in the namespace and check the resource requests and limits

for pod in $(kubectl get pods -n $NAMESPACE -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do

    # Get the resource requests and limits for the pod

    requests=$(kubectl get pod $pod -n $NAMESPACE -o jsonpath='{.spec.containers[*].resources.requests.memory}')

    limits=$(kubectl get pod $pod -n $NAMESPACE -o jsonpath='{.spec.containers[*].resources.limits.memory}')

    # If the requests or limits are not set, set them to a default value

    if [ -z "$requests" ]; then

        kubectl patch pod $pod -n $NAMESPACE --patch '{"spec": {"containers": [{"name": "'"$pod"'", "resources": {"requests": {"memory": "256Mi"}}}]}}'

    fi

    if [ -z "$limits" ]; then

        kubectl patch pod $pod -n $NAMESPACE --patch '{"spec": {"containers": [{"name": "'"$pod"'", "resources": {"limits": {"memory": "512Mi"}}}]}}'

    fi

      # If the requests or limits are too low, increase them to a higher value

    if [ "$requests" -lt "256Mi" ]; then

        kubectl patch pod $pod -n $NAMESPACE --patch '{"spec": {"containers": [{"name": "'"$pod"'", "resources": {"requests": {"memory": "256Mi"}}}]}}'

    fi

    if [ "$limits" -lt "512Mi" ]; then

        kubectl patch pod $pod -n $NAMESPACE --patch '{"spec": {"containers": [{"name": "'"$pod"'", "resources": {"limits": {"memory": "512Mi"}}}]}}'

    fi

done

```
### Identify and terminate any pods or containers that are using excessive amounts of memory, either due to a memory leak or other issue. This can free up resources for other parts of the cluster.
```shell
#!/bin/bash

# Define the threshold for excessive memory usage

MEMORY_THRESHOLD=${MEMORY_THRESHOLD}

# Get a list of all pods running on the Kubernetes cluster

PODS=$(kubectl get pods -o=jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}')

# Loop through each pod and check if it is using excessive memory

for POD in $PODS

do

  MEMORY_USAGE=$(kubectl top pod $POD | awk '{print $2}' | grep -v MEMORY)

  if [ $MEMORY_USAGE -gt $MEMORY_THRESHOLD ]

  then

    # If the pod is using excessive memory, terminate it

    kubectl delete pod $POD

    echo "Terminated pod: $POD"

  fi

done

echo "Memory usage check complete."

```

This incident type refers to situations where the available memory for limits in percentage of a Kubernetes cluster is low. This can cause issues with the performance and stability of the cluster, potentially leading to downtime or other problems. The incident may be triggered by an automated query or monitoring tool that checks for certain metrics or thresholds. It is important to address and resolve this issue as soon as possible to ensure the continued health and reliability of the Kubernetes cluster.


A Host Out of Memory(OOM) Incident occurs when a server or system runs out of memory, causing it to crash or become unresponsive. This can be caused by a variety of factors, such as an unexpected surge in traffic or insufficient resources allocated to the system. Resolving this type of incident requires identifying the root cause of the memory issue and taking appropriate measures such as optimizing system resources or increasing memory capacity.


Host Out of Memory (OOM) Incident

This incident type refers to a situation where the host memory of a system is not being fully utilized and may be causing performance issues. It may trigger when the memory load is below the expected level for a certain period of time. The incident may require investigation to determine the root cause and may involve reducing the amount of memory or other memory management strategies to optimize system performance.


"Host Memory is underutilized incident"

This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

A Kubernetes Replicaset Incomplete incident typically occurs when a specific number of pods that should be running are not, due to reasons such as failed pod initialization, unavailability of resources in the cluster, or inability to pull the image. This incident is usually triggered when the difference between desired and running pods is greater than zero, and it can be detected through monitoring tools like Datadog.


Kubernetes Replicaset Incomplete

Kubernetes Pods Pending incident indicates that one or more pods in a Kubernetes cluster are not running as expected and are in a pending state. This can happen due to various reasons such as resource constraints, scheduling issues, or network problems. This incident can impact the availability and performance of the application running on the Kubernetes cluster. It requires immediate attention to diagnose and resolve the underlying issue to ensure the pods are running as expected.


Kubernetes Pods Pending

```shell
# Environment Variables

export CONTAINER_NAME="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

export CONTEXT_NAME="PLACEHOLDER"

export POD_LABEL="PLACEHOLDER"

export MEMORY_THRESHOLD="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"
```


### Check the available memory for limits in percentage for all nodes in the cluster

```shell
kubectl top nodes | awk '{print $1,$4}' | column -t
```

### Check the available memory for limits in percentage for all pods in the cluster

```shell
kubectl top pods --all-namespaces | awk '{print $1,$2,$4}' | column -t
```

### Check the resource requests and limits for all pods in the cluster

```shell
kubectl describe pods | grep -e 'Name:\|Limits\|Requests'
```

### Check the memory usage for specific container in a pod

```shell
kubectl exec -it ${POD_NAME} -c ${CONTAINER_NAME} -- sh -c "free -m"
```

### Check the logs for a specific pod to see if any errors or issues are occurring

```shell
kubectl logs ${POD_NAME}
```

### Check the event log for the cluster to see if any events are being generated related to memory usage

```shell
kubectl get events
```

### Check the cluster autoscaler to see if it is scaling up or down properly based on resource usage

```shell
kubectl get hpa
```

### Check the cluster pods and nodes status

```shell
kubectl get pods,nodes
```


### Check the resource requests and limits for the pods running on the Kubernetes cluster to ensure they are properly configured. Adjust any values as necessary, keeping in mind the available resources on the cluster.

```shell
bash

#!/bin/bash

# Set the namespace where the resources are located

NAMESPACE=${NAMESPACE}

# Loop through all pods in the namespace and check the resource requests and limits

for pod in $(kubectl get pods -n $NAMESPACE -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do

    # Get the resource requests and limits for the pod

    requests=$(kubectl get pod $pod -n $NAMESPACE -o jsonpath='{.spec.containers[*].resources.requests.memory}')

    limits=$(kubectl get pod $pod -n $NAMESPACE -o jsonpath='{.spec.containers[*].resources.limits.memory}')

    # If the requests or limits are not set, set them to a default value

    if [ -z "$requests" ]; then

        kubectl patch pod $pod -n $NAMESPACE --patch '{"spec": {"containers": [{"name": "'"$pod"'", "resources": {"requests": {"memory": "256Mi"}}}]}}'

    fi

    if [ -z "$limits" ]; then

        kubectl patch pod $pod -n $NAMESPACE --patch '{"spec": {"containers": [{"name": "'"$pod"'", "resources": {"limits": {"memory": "512Mi"}}}]}}'

    fi

      # If the requests or limits are too low, increase them to a higher value

    if [ "$requests" -lt "256Mi" ]; then

        kubectl patch pod $pod -n $NAMESPACE --patch '{"spec": {"containers": [{"name": "'"$pod"'", "resources": {"requests": {"memory": "256Mi"}}}]}}'

    fi

    if [ "$limits" -lt "512Mi" ]; then

        kubectl patch pod $pod -n $NAMESPACE --patch '{"spec": {"containers": [{"name": "'"$pod"'", "resources": {"limits": {"memory": "512Mi"}}}]}}'

    fi

done

```

### Identify and terminate any pods or containers that are using excessive amounts of memory, either due to a memory leak or other issue. This can free up resources for other parts of the cluster.

```shell
#!/bin/bash

# Define the threshold for excessive memory usage

MEMORY_THRESHOLD=${MEMORY_THRESHOLD}

# Get a list of all pods running on the Kubernetes cluster

PODS=$(kubectl get pods -o=jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}')

# Loop through each pod and check if it is using excessive memory

for POD in $PODS

do

  MEMORY_USAGE=$(kubectl top pod $POD | awk '{print $2}' | grep -v MEMORY)

  if [ $MEMORY_USAGE -gt $MEMORY_THRESHOLD ]

  then

    # If the pod is using excessive memory, terminate it

    kubectl delete pod $POD

    echo "Terminated pod: $POD"

  fi

done

echo "Memory usage check complete."

```


Kubernetes - Available Memory for Limits in percentage Low

Overview

Parameters

Debug

Check the available memory for limits in percentage for all nodes in the cluster

Check the available memory for limits in percentage for all pods in the cluster

Check the resource requests and limits for all pods in the cluster

Check the memory usage for specific container in a pod

Check the logs for a specific pod to see if any errors or issues are occurring

Check the cluster autoscaler to see if it is scaling up or down properly based on resource usage

Check the cluster pods and nodes status

Repair

Check the resource requests and limits for the pods running on the Kubernetes cluster to ensure they are properly configured. Adjust any values as necessary, keeping in mind the available resources on the cluster.

Identify and terminate any pods or containers that are using excessive amounts of memory, either due to a memory leak or other issue. This can free up resources for other parts of the cluster.

Learn more

Related Runbooks

Host Out of Memory (OOM) Incident

"Host Memory is underutilized incident"

Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Replicaset Incomplete

Support