---
id: 13a51376-feaf-44c3-b6da-cb77321f350f
---

# Container High Memory Usage Incident
---

A Container High Memory Usage Incident occurs when the memory usage of a container exceeds a certain threshold, usually 90%. This can cause performance issues and potentially lead to system crashes. The incident should be resolved by identifying the root cause of the high memory usage and taking appropriate actions to reduce it.

### Parameters
```shell
export CONTAINER_NAME="PLACEHOLDER"

export PATH_TO_BINARY="PLACEHOLDER"

export PROCESS_ID="PLACEHOLDER"

export NEW_MEMORY_LIMIT="PLACEHOLDER"

export CONTAINER_ID="PLACEHOLDER"
```

## Debug

### Check overall memory usage on the host
```shell
free -m
```

### Display the top processes by memory usage
```shell
ps aux --sort=-%mem | head
```

### Check container memory usage
```shell
docker stats ${CONTAINER_NAME}
```

### Check for memory leaks in the application
```shell
valgrind --leak-check=full ${PATH_TO_BINARY}
```

### Analyze memory usage and generate a report
```shell
pmap -d ${PROCESS_ID} | tail -1
```

### Check for memory fragmentation
```shell
cat /proc/buddyinfo
```

### Check the system logs for any memory-related errors
```shell
dmesg | grep -i memory
```

### Check kernel memory usage
```shell
cat /proc/meminfo | grep -i kernel
```

### Check for any memory-related errors in the container logs
```shell
docker logs ${CONTAINER_NAME} | grep -i memory
```

### Inefficient resource allocation for the container, causing it to consume more memory than necessary.
```shell
bash

#!/bin/bash



# Set the container ID

container_id=${CONTAINER_ID}



# Get the container's resource usage information

usage_info=$(docker stats $container_id --no-stream)



# Extract the memory usage value from the usage information

memory_usage=$(echo "$usage_info" | awk '{print $4}' | tail -n 1)



# Get the allocated memory for the container

allocated_memory=$(docker inspect $container_id | grep -i 'memory":' | awk '{print $2}' | sed 's/,//g')



# Calculate the memory usage percentage

memory_usage_percentage=$(awk "BEGIN {print ($memory_usage/$allocated_memory)*100}")



# Check if the memory usage percentage exceeds a certain threshold

threshold=90

if (( $(echo "$memory_usage_percentage > $threshold" | bc -l) )); then

    echo "Container $container_id is consuming more memory than necessary."

    echo "Memory usage: $memory_usage"

    echo "Allocated memory: $allocated_memory"

    echo "Memory usage percentage: $memory_usage_percentage%"

else

    echo "Container $container_id is not consuming more memory than necessary."

fi


```

## Repair

### Increasing the memory allocation for the container can help reduce high memory usage incidents. This ensures that the container has enough memory to handle the workload it is running.
```shell
bash

#!/bin/bash



# Set the container name and memory limit

CONTAINER_NAME=${CONTAINER_NAME}

MEMORY_LIMIT=${NEW_MEMORY_LIMIT}

# Update the container with the new memory limit
docker update -m $MEMORY_LIMIT $CONTAINER_NAME

```

A Container High Memory Usage Incident occurs when the memory usage of a container exceeds a certain threshold, usually 90%. This can cause performance issues and potentially lead to system crashes. The incident should be resolved by identifying the root cause of the high memory usage and taking appropriate actions to reduce it.


This incident type occurs when the PermGen or Metaspace usage of the Tomcat server exceeds 90%. PermGen and Metaspace are memory pools used for storing class definitions and metadata of the Java virtual machine (JVM). When these memory pools reach their maximum capacity, the server may experience performance issues or even crash. This incident requires immediate attention from the software engineer to investigate the root cause and take appropriate actions to resolve the issue.


Tomcat PermGen or Metaspace Usage exceeds 90%

Tomcat JVM OutOfMemory incident occurs when the Java Virtual Machine (JVM) running the Apache Tomcat server runs out of memory, which causes the server to crash or become unresponsive. This type of incident can happen due to a variety of reasons such as incorrect configuration settings, memory leaks, or insufficient memory allocation. When this incident occurs, it can impact the availability and performance of the web application running on the Tomcat server.


Tomcat JVM OutOfMemory Incident

This incident type refers to a situation where the Tomcat server's JDBC connection pool usage surpasses its configured limits. This can lead to a shortage of available connections for incoming requests, resulting in slow response times or even server crashes. It is important to monitor and manage the JDBC connection pool usage to prevent such incidents from occurring.


Tomcat High JDBC Connection Pool Usage Incident.

This incident type refers to an increase in the number of errors per second on a Tomcat server, which could indicate an issue with the server itself, the host, a deployed application, or an application servlet. This could include errors generated when the Tomcat server runs out of memory, can't find a requested file or servlet, or is unable to serve a JSP due to syntax errors in the servlet codebase. This incident type requires immediate attention to diagnose and address the underlying issue.


Increase of the errors/second rate for Tomcat server

A Rabbitmq memory high incident occurs when a node uses more than 90% of the allocated RAM. This can cause performance issues and impact the overall functionality of the system. It is important to promptly address this issue to prevent any further damage.


Rabbitmq memory high incident.

```shell
export CONTAINER_NAME="PLACEHOLDER"

export PATH_TO_BINARY="PLACEHOLDER"

export PROCESS_ID="PLACEHOLDER"

export NEW_MEMORY_LIMIT="PLACEHOLDER"

export CONTAINER_ID="PLACEHOLDER"
```


### Check overall memory usage on the host

```shell
free -m
```

### Display the top processes by memory usage

```shell
ps aux --sort=-%mem | head
```

### Check container memory usage

```shell
docker stats ${CONTAINER_NAME}
```

### Check for memory leaks in the application

```shell
valgrind --leak-check=full ${PATH_TO_BINARY}
```

### Analyze memory usage and generate a report

```shell
pmap -d ${PROCESS_ID} | tail -1
```

### Check for memory fragmentation

```shell
cat /proc/buddyinfo
```

### Check the system logs for any memory-related errors

```shell
dmesg | grep -i memory
```

### Check kernel memory usage

```shell
cat /proc/meminfo | grep -i kernel
```

### Check for any memory-related errors in the container logs

```shell
docker logs ${CONTAINER_NAME} | grep -i memory
```

### Inefficient resource allocation for the container, causing it to consume more memory than necessary.

```shell
bash

#!/bin/bash



# Set the container ID

container_id=${CONTAINER_ID}



# Get the container's resource usage information

usage_info=$(docker stats $container_id --no-stream)



# Extract the memory usage value from the usage information

memory_usage=$(echo "$usage_info" | awk '{print $4}' | tail -n 1)



# Get the allocated memory for the container

allocated_memory=$(docker inspect $container_id | grep -i 'memory":' | awk '{print $2}' | sed 's/,//g')



# Calculate the memory usage percentage

memory_usage_percentage=$(awk "BEGIN {print ($memory_usage/$allocated_memory)*100}")



# Check if the memory usage percentage exceeds a certain threshold

threshold=90

if (( $(echo "$memory_usage_percentage > $threshold" | bc -l) )); then

    echo "Container $container_id is consuming more memory than necessary."

    echo "Memory usage: $memory_usage"

    echo "Allocated memory: $allocated_memory"

    echo "Memory usage percentage: $memory_usage_percentage%"

else

    echo "Container $container_id is not consuming more memory than necessary."

fi


```


### Increasing the memory allocation for the container can help reduce high memory usage incidents. This ensures that the container has enough memory to handle the workload it is running.

```shell
bash

#!/bin/bash



# Set the container name and memory limit

CONTAINER_NAME=${CONTAINER_NAME}

MEMORY_LIMIT=${NEW_MEMORY_LIMIT}

# Update the container with the new memory limit
docker update -m $MEMORY_LIMIT $CONTAINER_NAME

```


Container High Memory Usage Incident

Overview

Parameters

Debug

Check overall memory usage on the host

Display the top processes by memory usage

Check container memory usage

Check for memory leaks in the application

Analyze memory usage and generate a report

Check for memory fragmentation

Check kernel memory usage

Inefficient resource allocation for the container, causing it to consume more memory than necessary.

Repair

Increasing the memory allocation for the container can help reduce high memory usage incidents. This ensures that the container has enough memory to handle the workload it is running.

Learn more

Related Runbooks

Tomcat PermGen or Metaspace Usage exceeds 90%

Tomcat JVM OutOfMemory Incident

Tomcat High JDBC Connection Pool Usage Incident.

Increase of the errors/second rate for Tomcat server

Support