---
id: 995c81f7-66b4-4e6a-8590-ae94900475ef
---

# High CPU Load Incident
---

A high CPU load incident occurs when the CPU usage of a host or server exceeds a certain threshold, typically 80%. This can cause slowdowns, unresponsiveness, or even crashes in the affected system. It is critical to address this issue promptly to minimize impact on users and prevent further damage to the system.

### Parameters
```shell
export CONTAINER="PLACEHOLDER"

export INTERVAL="PLACEHOLDER"

export COUNT="PLACEHOLDER"

export COMMAND="PLACEHOLDER"
```

## Debug

### Check overall CPU usage
```shell
top
```

### Check CPU usage per process
```shell
top -c
```

### Sort processes by CPU usage
```shell
ps aux --sort=-%cpu
```

### Show CPU usage in real-time
```shell
mpstat 1
```

### Show CPU usage by container
```shell
docker stats ${CONTAINER}
```

### Check CPU usage over time
```shell
sar -u ${INTERVAL} ${COUNT}
```

### Check CPU usage by interrupt
```shell
cat /proc/interrupts
```

### Check CPU usage by system call
```shell
strace -c ${COMMAND}
```

### A process or application running on the server is consuming too much CPU resources, causing the overall load to increase.
```shell


#!/bin/bash



# get the process with the highest CPU usage

highest_cpu_process=$(ps aux --sort=-%cpu | awk 'NR==2{print $11}')



# get the PID of the process with the highest CPU usage

highest_cpu_pid=$(ps aux --sort=-%cpu | awk 'NR==2{print $2}')



# get the CPU usage of the process with the highest CPU usage

highest_cpu_usage=$(ps aux --sort=-%cpu | awk 'NR==2{print $3}')



echo "The process consuming the most CPU resources is $highest_cpu_process with PID $highest_cpu_pid."

echo "It is currently using $highest_cpu_usage% of the CPU."


```

### The server is running too many applications or services simultaneously, leading to a high overall CPU load.
```shell
bash

#!/bin/bash



# Check current CPU usage

cpu_usage=$(top -b -n1 | grep "Cpu(s)" | awk '{print $2 + $4}')



# Check number of running processes

num_processes=$(ps -e | wc -l)



# Check number of running services

num_services=$(systemctl list-units --type=service --state=running | wc -l)



# Set threshold values for CPU usage, number of processes, and number of services

cpu_threshold=80

process_threshold=500

service_threshold=50



# Check if CPU usage is above threshold

if [ $(echo "$cpu_usage > $cpu_threshold" | bc) -ne 0 ]; then

  echo "High CPU usage detected. Current usage: $cpu_usage%"

fi



# Check if number of processes is above threshold

if [ $num_processes -gt $process_threshold ]; then

  echo "Too many processes running. Current count: $num_processes"

fi



# Check if number of services is above threshold

if [ $num_services -gt $service_threshold ]; then

  echo "Too many services running. Current count: $num_services"

fi


```

## Repair

### If possible, terminate any unnecessary processes or applications to free up CPU resources.
```shell


#!/bin/bash



# Set the threshold for maximum CPU usage

MAX_CPU_LOAD=80



# Get the current CPU usage

CPU_LOAD=$(top -b -n 1 | grep "Cpu(s)" | awk '{print $2+$4}')



# Check if the CPU usage is above the threshold

if [ $CPU_LOAD -gt $MAX_CPU_LOAD ]; then

  echo "High CPU load detected. Terminating unnecessary processes..."



  # Find the top CPU consuming processes

  PROCESSES_TO_KILL=$(ps aux --sort=-%cpu | awk '{if($3>50) print $2}')



  # Terminate the processes

  for PID in $PROCESSES_TO_KILL; do

    kill -9 $PID

  done



  echo "Processes terminated."

else

  echo "CPU load is within normal range."

fi


```

A high CPU load incident occurs when the CPU usage of a host or server exceeds a certain threshold, typically 80%. This can cause slowdowns, unresponsiveness, or even crashes in the affected system. It is critical to address this issue promptly to minimize impact on users and prevent further damage to the system.


This incident type refers to an increase in the number of errors per second on a Tomcat server, which could indicate an issue with the server itself, the host, a deployed application, or an application servlet. This could include errors generated when the Tomcat server runs out of memory, can't find a requested file or servlet, or is unable to serve a JSP due to syntax errors in the servlet codebase. This incident type requires immediate attention to diagnose and address the underlying issue.


Increase of the errors/second rate for Tomcat server

This incident type typically refers to a situation where there is an anomaly in the Tomcat max processing time on a host. The maximum amount of time it takes for the server to process one request, from the time an available thread starts processing the request to the time it returns a response, has exceeded the expected threshold. This could indicate that a JSP page isn’t loading or an associated process (such as a database query) is taking too long to complete.


Anomalous max processing time for Tomcat host

This incident type indicates that there is a high latency issue in the execution of a Spark job. Spark is a distributed computing framework that is used for processing large datasets. High latency in this context means that the time taken to execute the Spark job is significantly longer than expected or normal. This can result in delays in processing data and can impact the performance of the application or system that is utilizing Spark.


High Latency Incident for Spark Job Execution.

This incident type refers to a sudden and abnormal drop in the SELECT query rate on a MySQL server. This can potentially cause disruption in the service and affect the user experience. The cause of the drop could be due to a variety of factors such as database issues, server overload, or network problems. It requires immediate attention and investigation to identify the root cause and resolve the issue to prevent further impact.


Unusual drop in SELECT query rate on MySQL server

This incident type occurs when the number of concurrent connections to a MySQL database server exceeds the configured limit of 80%. This can lead to performance degradation or even database server crashes, making the application or system that relies on the database server unavailable for the end-users. This type of incident typically requires immediate attention and resolution to restore normal operation of the affected service.


MySQL too many connections incident

```shell
export CONTAINER="PLACEHOLDER"

export INTERVAL="PLACEHOLDER"

export COUNT="PLACEHOLDER"

export COMMAND="PLACEHOLDER"
```


### Check overall CPU usage

```shell
top
```

### Check CPU usage per process

```shell
top -c
```

### Sort processes by CPU usage

```shell
ps aux --sort=-%cpu
```

### Show CPU usage in real-time

```shell
mpstat 1
```

### Show CPU usage by container

```shell
docker stats ${CONTAINER}
```

### Check CPU usage over time

```shell
sar -u ${INTERVAL} ${COUNT}
```

### Check CPU usage by interrupt

```shell
cat /proc/interrupts
```

### Check CPU usage by system call

```shell
strace -c ${COMMAND}
```

### A process or application running on the server is consuming too much CPU resources, causing the overall load to increase.

```shell


#!/bin/bash



# get the process with the highest CPU usage

highest_cpu_process=$(ps aux --sort=-%cpu | awk 'NR==2{print $11}')



# get the PID of the process with the highest CPU usage

highest_cpu_pid=$(ps aux --sort=-%cpu | awk 'NR==2{print $2}')



# get the CPU usage of the process with the highest CPU usage

highest_cpu_usage=$(ps aux --sort=-%cpu | awk 'NR==2{print $3}')



echo "The process consuming the most CPU resources is $highest_cpu_process with PID $highest_cpu_pid."

echo "It is currently using $highest_cpu_usage% of the CPU."


```

### The server is running too many applications or services simultaneously, leading to a high overall CPU load.

```shell
bash

#!/bin/bash



# Check current CPU usage

cpu_usage=$(top -b -n1 | grep "Cpu(s)" | awk '{print $2 + $4}')



# Check number of running processes

num_processes=$(ps -e | wc -l)



# Check number of running services

num_services=$(systemctl list-units --type=service --state=running | wc -l)



# Set threshold values for CPU usage, number of processes, and number of services

cpu_threshold=80

process_threshold=500

service_threshold=50



# Check if CPU usage is above threshold

if [ $(echo "$cpu_usage > $cpu_threshold" | bc) -ne 0 ]; then

  echo "High CPU usage detected. Current usage: $cpu_usage%"

fi



# Check if number of processes is above threshold

if [ $num_processes -gt $process_threshold ]; then

  echo "Too many processes running. Current count: $num_processes"

fi



# Check if number of services is above threshold

if [ $num_services -gt $service_threshold ]; then

  echo "Too many services running. Current count: $num_services"

fi


```


### If possible, terminate any unnecessary processes or applications to free up CPU resources.

```shell


#!/bin/bash



# Set the threshold for maximum CPU usage

MAX_CPU_LOAD=80



# Get the current CPU usage

CPU_LOAD=$(top -b -n 1 | grep "Cpu(s)" | awk '{print $2+$4}')



# Check if the CPU usage is above the threshold

if [ $CPU_LOAD -gt $MAX_CPU_LOAD ]; then

  echo "High CPU load detected. Terminating unnecessary processes..."



  # Find the top CPU consuming processes

  PROCESSES_TO_KILL=$(ps aux --sort=-%cpu | awk '{if($3>50) print $2}')



  # Terminate the processes

  for PID in $PROCESSES_TO_KILL; do

    kill -9 $PID

  done



  echo "Processes terminated."

else

  echo "CPU load is within normal range."

fi


```


High CPU Load Incident

Overview

Parameters

Debug

Check overall CPU usage

Check CPU usage per process

Sort processes by CPU usage

Show CPU usage in real-time

Show CPU usage by container

Check CPU usage over time

Check CPU usage by interrupt

Check CPU usage by system call

A process or application running on the server is consuming too much CPU resources, causing the overall load to increase.

The server is running too many applications or services simultaneously, leading to a high overall CPU load.

Repair

If possible, terminate any unnecessary processes or applications to free up CPU resources.

Learn more

Related Runbooks

Increase of the errors/second rate for Tomcat server

Anomalous max processing time for Tomcat host

High Latency Incident for Spark Job Execution.

Unusual drop in SELECT query rate on MySQL server

Support