---
id: 23d2c3d6-96e8-4bcd-aac0-3d6442a4f303
---

# Kubernetes - Available Memory for Requests in percentage Low
---

This incident type pertains to a situation where the available memory for requests in percentage is low in a Kubernetes cluster. It means that the resource limit for a container has been exceeded, and this can result in performance issues and potentially cause the application to crash. This incident requires immediate attention and resolution to ensure the stability and availability of the affected system.

### Parameters
```shell
# Environment Variables

export POD_NAME="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export DEPLOYMENT="PLACEHOLDER"

```

## Debug

### Check if any nodes are in NotReady state
```shell
kubectl get nodes -o wide | grep -v Ready
```

### Check if any pods are in the Pending state
```shell
kubectl get pods --all-namespaces | grep Pending
```

### Check the memory usage of all pods in the default namespace
```shell
kubectl top pods
```

### Check the memory usage of a specific pod in the default namespace
```shell
kubectl top pod ${POD_NAME}
```

### Check the memory usage of all containers in a specific pod in the default namespace
```shell
kubectl top pod ${POD_NAME} --containers
```

### Check the resource requests and limits for all pods in the default namespace
```shell
kubectl describe pods | grep -A 5 "Limits"
```

### Check the resource requests and limits for a specific pod in the default namespace
```shell
kubectl describe pod ${POD_NAME} | grep -A 5 "Limits"
```

### Check the current memory usage and available memory for each node in the cluster
```shell
kubectl describe nodes | grep -A 5 "Allocatable\|Capacity"
```

## Repair

### Optimize memory usage of applications running on the cluster by tuning their memory requests and limits.
```shell
bash

#!/bin/bash



# Define the namespace and deployment name

NAMESPACE=${NAMESPACE}

DEPLOYMENT=${DEPLOYMENT}



# Get the current memory requests and limits for the deployment

MEMORY_REQUEST=$(kubectl get deploy -n $NAMESPACE $DEPLOYMENT -o jsonpath='{.spec.template.spec.containers[0].resources.requests.memory}')

MEMORY_LIMIT=$(kubectl get deploy -n $NAMESPACE $DEPLOYMENT -o jsonpath='{.spec.template.spec.containers[0].resources.limits.memory}')



# Calculate the recommended memory requests and limits based on usage statistics

MEMORY_USAGE=$(kubectl top pod -n $NAMESPACE | grep $DEPLOYMENT | awk '{print $3}')

MEMORY_RECOMMENDED=$((MEMORY_USAGE * 2))Mi



# Update the deployment with the recommended memory requests and limits

kubectl patch deployment -n $NAMESPACE $DEPLOYMENT -p '{"spec":{"template":{"spec":{"containers":[{"name":"'$DEPLOYMENT'","resources":{"requests":{"memory":"'$MEMORY_RECOMMENDED'"},"limits":{"memory":"'$MEMORY_RECOMMENDED'"}}}]}}}}'


```

### Increase available memory on the Kubernetes cluster by adding more nodes or increasing the memory allocated to existing nodes.
```shell
#AWS

aws eks update-nodegroup-config --cluster-name <cluster-name> --nodegroup-name <nodegroup-name> --scaling-config minSize=<min-size>,maxSize=<max-size>,desiredSize=<desired-size>

#Azure

az aks nodepool scale --resource-group <resource-group-name> --cluster-name <cluster-name> --name <nodepool-name> --node-count <new-node-count>

# GCP 

gcloud container clusters resize <cluster-name>  --size=<new-node-count>  --zone=<zone> 

```

This incident type pertains to a situation where the available memory for requests in percentage is low in a Kubernetes cluster. It means that the resource limit for a container has been exceeded, and this can result in performance issues and potentially cause the application to crash. This incident requires immediate attention and resolution to ensure the stability and availability of the affected system.


A Host Out of Memory(OOM) Incident occurs when a server or system runs out of memory, causing it to crash or become unresponsive. This can be caused by a variety of factors, such as an unexpected surge in traffic or insufficient resources allocated to the system. Resolving this type of incident requires identifying the root cause of the memory issue and taking appropriate measures such as optimizing system resources or increasing memory capacity.


Host Out of Memory (OOM) Incident

This incident type refers to a situation where the host memory of a system is not being fully utilized and may be causing performance issues. It may trigger when the memory load is below the expected level for a certain period of time. The incident may require investigation to determine the root cause and may involve reducing the amount of memory or other memory management strategies to optimize system performance.


"Host Memory is underutilized incident"

This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

A Kubernetes Replicaset Incomplete incident typically occurs when a specific number of pods that should be running are not, due to reasons such as failed pod initialization, unavailability of resources in the cluster, or inability to pull the image. This incident is usually triggered when the difference between desired and running pods is greater than zero, and it can be detected through monitoring tools like Datadog.


Kubernetes Replicaset Incomplete

Kubernetes Pods Pending incident indicates that one or more pods in a Kubernetes cluster are not running as expected and are in a pending state. This can happen due to various reasons such as resource constraints, scheduling issues, or network problems. This incident can impact the availability and performance of the application running on the Kubernetes cluster. It requires immediate attention to diagnose and resolve the underlying issue to ensure the pods are running as expected.


Kubernetes Pods Pending

```shell
# Environment Variables

export POD_NAME="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export DEPLOYMENT="PLACEHOLDER"

```


### Check if any nodes are in NotReady state

```shell
kubectl get nodes -o wide | grep -v Ready
```

### Check if any pods are in the Pending state

```shell
kubectl get pods --all-namespaces | grep Pending
```

### Check the memory usage of all pods in the default namespace

```shell
kubectl top pods
```

### Check the memory usage of a specific pod in the default namespace

```shell
kubectl top pod ${POD_NAME}
```

### Check the memory usage of all containers in a specific pod in the default namespace

```shell
kubectl top pod ${POD_NAME} --containers
```

### Check the resource requests and limits for all pods in the default namespace

```shell
kubectl describe pods | grep -A 5 "Limits"
```

### Check the resource requests and limits for a specific pod in the default namespace

```shell
kubectl describe pod ${POD_NAME} | grep -A 5 "Limits"
```

### Check the current memory usage and available memory for each node in the cluster

```shell
kubectl describe nodes | grep -A 5 "Allocatable\|Capacity"
```


### Optimize memory usage of applications running on the cluster by tuning their memory requests and limits.

```shell
bash

#!/bin/bash



# Define the namespace and deployment name

NAMESPACE=${NAMESPACE}

DEPLOYMENT=${DEPLOYMENT}



# Get the current memory requests and limits for the deployment

MEMORY_REQUEST=$(kubectl get deploy -n $NAMESPACE $DEPLOYMENT -o jsonpath='{.spec.template.spec.containers[0].resources.requests.memory}')

MEMORY_LIMIT=$(kubectl get deploy -n $NAMESPACE $DEPLOYMENT -o jsonpath='{.spec.template.spec.containers[0].resources.limits.memory}')



# Calculate the recommended memory requests and limits based on usage statistics

MEMORY_USAGE=$(kubectl top pod -n $NAMESPACE | grep $DEPLOYMENT | awk '{print $3}')

MEMORY_RECOMMENDED=$((MEMORY_USAGE * 2))Mi



# Update the deployment with the recommended memory requests and limits

kubectl patch deployment -n $NAMESPACE $DEPLOYMENT -p '{"spec":{"template":{"spec":{"containers":[{"name":"'$DEPLOYMENT'","resources":{"requests":{"memory":"'$MEMORY_RECOMMENDED'"},"limits":{"memory":"'$MEMORY_RECOMMENDED'"}}}]}}}}'


```

### Increase available memory on the Kubernetes cluster by adding more nodes or increasing the memory allocated to existing nodes.

```shell
#AWS

aws eks update-nodegroup-config --cluster-name <cluster-name> --nodegroup-name <nodegroup-name> --scaling-config minSize=<min-size>,maxSize=<max-size>,desiredSize=<desired-size>

#Azure

az aks nodepool scale --resource-group <resource-group-name> --cluster-name <cluster-name> --name <nodepool-name> --node-count <new-node-count>

# GCP 

gcloud container clusters resize <cluster-name>  --size=<new-node-count>  --zone=<zone> 

```


Kubernetes - Available Memory for Requests in percentage Low

Overview

Parameters

Debug

Check if any nodes are in NotReady state

Check if any pods are in the Pending state

Check the memory usage of all pods in the default namespace

Check the memory usage of a specific pod in the default namespace

Check the memory usage of all containers in a specific pod in the default namespace

Check the resource requests and limits for all pods in the default namespace

Check the resource requests and limits for a specific pod in the default namespace

Check the current memory usage and available memory for each node in the cluster

Repair

Optimize memory usage of applications running on the cluster by tuning their memory requests and limits.

Increase available memory on the Kubernetes cluster by adding more nodes or increasing the memory allocated to existing nodes.

Learn more

Related Runbooks

Host Out of Memory (OOM) Incident

"Host Memory is underutilized incident"

Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Replicaset Incomplete

Support