---
id: 095010ae-6493-11ee-8c99-0242ac120002
---
# Host Out of Memory (OOM) Incident
---

The Host Out of Memory (OOM) Incident occurs when a server or system runs out of memory, causing it to crash or become unresponsive. This can be caused by various factors, such as an unexpected surge in traffic or insufficient resources allocated to the system. Resolving this type of incident requires identifying the root cause of the memory issue and taking appropriate measures such as optimizing system resources or increasing memory capacity.

### Parameters
```shell
export PROCESS_NAME="PLACEHOLDER"

export SERVICE_NAME="PLACEHOLDER"

export INSTANCE_ID="PLACEHOLDER"

export INSTANCE_TYPE="PLACEHOLDER"

export ZONE="PLACEHOLDER"

export RESOURCE_GROUP_NAME="PLACEHOLDER"
```

## Debug

### Check the amount of free memory
```shell
Get-WmiObject -Class Win32_OperatingSystem | Select-Object FreePhysicalMemory
```

### Check the amount of used memory by each process
```shell
Get-Process | Sort-Object -Descending WS | Select-Object ProcessName, WS -First 10
```

### Check the event viewer logs for any out of memory errors
```shell
Get-EventLog System -Source Microsoft-Windows-Resource-Exhaustion-Detector | Where-Object {$_.EventID -eq 2004}
```

### Check the system limits for the amount of memory available
```shell
Get-WmiObject -Class Win32_ComputerSystem | Select-Object TotalPhysicalMemory
```

### Check the page file usage on the host
```shell
Get-WmiObject -Class Win32_PageFileUsage | Select-Object Name, AllocatedBaseSize, CurrentUsage, PeakUsage
```

### Check the process limits for the user running the process
```shell
Get-Process | Where-Object {$_.ProcessName -eq "${PROCESS_NAME}"} | Select-Object -ExpandProperty WS
```

### Check the kernel logs for any memory-related errors
```shell
Get-EventLog System -Source Microsoft-Windows-Kernel-Power | Where-Object {$_.EventID -eq 41}
```

## Repair

### Reduce the number of applications or processes running on the host by shutting down unneeded services.
```shell


#!/bin/bash



# Stop the service

Get-Service -Name ${SERVICE_NAME} | Stop-Service 



# Disable the service from starting automatically on boot

Set-Service -Name ${SERVICE_NAME} -StartupType Disabled 


```

# Note
---

Before you proceed with changing the instance type, please be aware that the current instance will restart during the process. Changing the instance type involves stopping the current instance, resizing its resources, and then starting it again with the new configuration.

### Changing the Machine type in GCP
```shell
gcloud compute instances set-machine-type ${INSTANCE_ID} --machine-type=${INSTANCE_TYPE} --zone=${ZONE}
```

### Change the size of an Azure VM using the Azure CLI
```shell
az vm resize --resource-group ${RESOURCE_GROUP_NAME} --name ${INSTANCE_ID} --size ${INSTANCE_TYPE}
```

### Changing AWS Instance type Using AWS CLI
```shell
aws ec2 modify-instance-attribute --instance-id ${INSTANCE_ID} --instance-type ${INSTANCE_TYPE}
```



The Host Out of Memory (OOM) Incident occurs when a server or system runs out of memory, causing it to crash or become unresponsive. This can be caused by various factors, such as an unexpected surge in traffic or insufficient resources allocated to the system. Resolving this type of incident requires identifying the root cause of the memory issue and taking appropriate measures such as optimizing system resources or increasing memory capacity.


Tomcat Server Crash incident type refers to a situation where the Tomcat Server, an open-source web server and servlet container, becomes unresponsive and stops functioning. This can result in a complete or partial system outage, leading to a disruption in the delivery of web applications and services. The root cause of the crash can be attributed to a variety of factors such as misconfiguration, network or hardware issues, software bugs, or excessive server load. It is critical to address this incident quickly to minimize the impact on users and restore normal system functionality.


Tomcat Server Crash

Tomcat JVM OutOfMemory incident occurs when the Java Virtual Machine (JVM) running the Apache Tomcat server runs out of memory, which causes the server to crash or become unresponsive. This type of incident can happen due to a variety of reasons such as incorrect configuration settings, memory leaks, or insufficient memory allocation. When this incident occurs, it can impact the availability and performance of the web application running on the Tomcat server.


Tomcat JVM OutOfMemory Incident

This incident type refers to an increase in the number of errors per second on a Tomcat server, which could indicate an issue with the server itself, the host, a deployed application, or an application servlet. This could include errors generated when the Tomcat server runs out of memory, can't find a requested file or servlet, or is unable to serve a JSP due to syntax errors in the servlet codebase. This incident type requires immediate attention to diagnose and address the underlying issue.


Increase of the errors/second rate for Tomcat server

This incident type refers to a situation where Spark tasks are failing due to out of memory errors. Spark is a distributed computing system used for big data processing. When the data volume exceeds the allocated memory, the Spark tasks fail, and the system generates an out of memory error. This type of incident can cause data processing delays or even system downtime, which can impact the overall performance of the application.


Spark tasks failing due to out of memory errors.

Apache Spark Out of Memory Error is a type of incident that occurs when the Apache Spark application running on a system is unable to allocate enough memory to perform its tasks. This can happen when the system's memory is insufficient or when there is a memory leak in the application's code. When this incident occurs, the application may crash, become unresponsive, or produce incorrect results. It is important to diagnose and resolve this issue as soon as possible to ensure the smooth functioning of the application.


Apache Spark Out of Memory Error incident.

```shell
export PROCESS_NAME="PLACEHOLDER"

export SERVICE_NAME="PLACEHOLDER"

export INSTANCE_ID="PLACEHOLDER"

export INSTANCE_TYPE="PLACEHOLDER"

export ZONE="PLACEHOLDER"

export RESOURCE_GROUP_NAME="PLACEHOLDER"
```


### Check the amount of free memory

```shell
Get-WmiObject -Class Win32_OperatingSystem | Select-Object FreePhysicalMemory
```

### Check the amount of used memory by each process

```shell
Get-Process | Sort-Object -Descending WS | Select-Object ProcessName, WS -First 10
```

### Check the event viewer logs for any out of memory errors

```shell
Get-EventLog System -Source Microsoft-Windows-Resource-Exhaustion-Detector | Where-Object {$_.EventID -eq 2004}
```

### Check the system limits for the amount of memory available

```shell
Get-WmiObject -Class Win32_ComputerSystem | Select-Object TotalPhysicalMemory
```

### Check the page file usage on the host

```shell
Get-WmiObject -Class Win32_PageFileUsage | Select-Object Name, AllocatedBaseSize, CurrentUsage, PeakUsage
```

### Check the process limits for the user running the process

```shell
Get-Process | Where-Object {$_.ProcessName -eq "${PROCESS_NAME}"} | Select-Object -ExpandProperty WS
```

### Check the kernel logs for any memory-related errors

```shell
Get-EventLog System -Source Microsoft-Windows-Kernel-Power | Where-Object {$_.EventID -eq 41}
```


### Reduce the number of applications or processes running on the host by shutting down unneeded services.

```shell


#!/bin/bash



# Stop the service

Get-Service -Name ${SERVICE_NAME} | Stop-Service 



# Disable the service from starting automatically on boot

Set-Service -Name ${SERVICE_NAME} -StartupType Disabled 


```

# Note

Before you proceed with changing the instance type, please be aware that the current instance will restart during the process. Changing the instance type involves stopping the current instance, resizing its resources, and then starting it again with the new configuration.

### Changing the Machine type in GCP

```shell
gcloud compute instances set-machine-type ${INSTANCE_ID} --machine-type=${INSTANCE_TYPE} --zone=${ZONE}
```

### Change the size of an Azure VM using the Azure CLI

```shell
az vm resize --resource-group ${RESOURCE_GROUP_NAME} --name ${INSTANCE_ID} --size ${INSTANCE_TYPE}
```

### Changing AWS Instance type Using AWS CLI

```shell
aws ec2 modify-instance-attribute --instance-id ${INSTANCE_ID} --instance-type ${INSTANCE_TYPE}
```


Host Out of Memory (OOM) Incident

Overview

Parameters

Debug

Check the amount of free memory

Check the amount of used memory by each process

Check the event viewer logs for any out of memory errors

Check the system limits for the amount of memory available

Check the page file usage on the host

Check the process limits for the user running the process

Repair

Reduce the number of applications or processes running on the host by shutting down unneeded services.

Note

Changing the Machine type in GCP

Change the size of an Azure VM using the Azure CLI

Changing AWS Instance type Using AWS CLI

Learn more

Related Runbooks

Tomcat Server Crash

Tomcat JVM OutOfMemory Incident

Increase of the errors/second rate for Tomcat server

Spark tasks failing due to out of memory errors.

Support