---
id: edc9ea7b-14f5-4e61-9c66-c86a8fdeaa4e
---

# High urgency incident related to host context switching.
---

This incident type relates to a high urgency issue regarding host context switching. The incident is triggered when the context switching grows on the node beyond a certain level, typically over 10000 per second. This issue can cause performance degradation and impact the stability of the system. The incident requires immediate attention from a software engineer to identify the root cause and take the necessary steps to resolve the issue.

### Parameters
```shell
export INSTANCE="PLACEHOLDER"

export PORT="PLACEHOLDER"

export SERVICE_NAME="PLACEHOLDER"
```

## Debug

### Check the CPU usage of the affected instance
```shell
ssh ${INSTANCE} top -bn1 | grep "Cpu(s)"
```

### Check the memory usage of the affected instance
```shell
ssh ${INSTANCE} free -m
```

### Check the number of context switches per second on the affected instance
```shell
ssh ${INSTANCE} grep ctxt /proc/stat
```

### Check the network connection status of the affected instance
```shell
ssh ${INSTANCE} netstat -anp | grep ${PORT}
```

### Check the disk usage of the affected instance
```shell
ssh ${INSTANCE} df -h
```

### Check the logs related to the affected service
```shell
journalctl -u ${SERVICE_NAME}.service
```

### Check the status of the affected service
```shell
systemctl status ${SERVICE_NAME}.service
```

### The system may be overloaded with too many requests, causing the CPU to switch between different processes frequently, leading to high context switching rates.
```shell
bash

#!/bin/bash



# Check CPU usage

CPU_USAGE=$(top -bn1 | grep "Cpu(s)" | awk '{print $2 + $4}')

THRESHOLD=80



if (( $(echo "${CPU_USAGE} > ${THRESHOLD}" | bc -l) )); then

  echo "CPU usage is high. Current usage is ${CPU_USAGE}%"

  echo "Checking context switching rates..."

  

  # Check context switching rates

  CONTEXT_SWITCHES=$(vmstat 1 2 | tail -1 | awk '{print $12}')

  THRESHOLD_SWITCHES=10000

  

  if (( ${CONTEXT_SWITCHES} > ${THRESHOLD_SWITCHES} )); then

    echo "Context switching rate is high. Current rate is ${CONTEXT_SWITCHES}"

    echo "System may be overloaded with too many requests."

  else

    echo "Context switching rate is normal. Current rate is ${CONTEXT_SWITCHES}"

    echo "No issues detected."

  fi

else

  echo "CPU usage is normal. Current usage is ${CPU_USAGE}%"

  echo "No issues detected."

fi


```

## Repair

### Consider restarting the affected service or host.
```shell
bash

#!/bin/bash

# Script to remediate high context switching rate issue



# Restart the affected service or host

echo "Restarting ${SERVICE_NAME} to remediate high context switching rate issue"

sudo systemctl restart ${SERVICE_NAME}



# Verify that the service or host has restarted successfully

echo "Verifying that ${SERVICE_NAME} has restarted successfully"

sudo systemctl status ${SERVICE_NAME}


```

This incident type relates to a high urgency issue regarding host context switching. The incident is triggered when the context switching grows on the node beyond a certain level, typically over 10000 per second. This issue can cause performance degradation and impact the stability of the system. The incident requires immediate attention from a software engineer to identify the root cause and take the necessary steps to resolve the issue.


The Host Out of Memory (OOM) Incident occurs when a server or system runs out of memory, causing it to crash or become unresponsive. This can be caused by various factors, such as an unexpected surge in traffic or insufficient resources allocated to the system. Resolving this type of incident requires identifying the root cause of the memory issue and taking appropriate measures such as optimizing system resources or increasing memory capacity.


Host Out of Memory (OOM) Incident

Tomcat High Swap Usage Incident refers to an event where Tomcat, a popular open-source web server, is experiencing high swap usage. Swap usage is a memory management technique used by the operating system to temporarily move data from RAM to the hard disk. When Tomcat is consuming high swap memory, it can cause the server to slow down or even crash, resulting in downtime for the website or application hosted on the server. This incident type requires immediate attention to identify the root cause and resolve the issue to prevent further downtime.


Tomcat High Swap Usage Incident

This incident type refers to an increase in the number of errors per second on a Tomcat server, which could indicate an issue with the server itself, the host, a deployed application, or an application servlet. This could include errors generated when the Tomcat server runs out of memory, can't find a requested file or servlet, or is unable to serve a JSP due to syntax errors in the servlet codebase. This incident type requires immediate attention to diagnose and address the underlying issue.


Increase of the errors/second rate for Tomcat server

This incident type is triggered when the percentage of busy threads on a Tomcat host exceeds a certain threshold. This can indicate that the host is experiencing high traffic or that there is a problem with the configuration of the Tomcat server. The incident is resolved when the issue is addressed and the percentage of busy threads returns to a normal level.


High % of Busy Threads on Tomcat Host (TEST)

This incident type indicates that there is a high latency issue in the execution of a Spark job. Spark is a distributed computing framework that is used for processing large datasets. High latency in this context means that the time taken to execute the Spark job is significantly longer than expected or normal. This can result in delays in processing data and can impact the performance of the application or system that is utilizing Spark.


High Latency Incident for Spark Job Execution.

```shell
export INSTANCE="PLACEHOLDER"

export PORT="PLACEHOLDER"

export SERVICE_NAME="PLACEHOLDER"
```


### Check the CPU usage of the affected instance

```shell
ssh ${INSTANCE} top -bn1 | grep "Cpu(s)"
```

### Check the memory usage of the affected instance

```shell
ssh ${INSTANCE} free -m
```

### Check the number of context switches per second on the affected instance

```shell
ssh ${INSTANCE} grep ctxt /proc/stat
```

### Check the network connection status of the affected instance

```shell
ssh ${INSTANCE} netstat -anp | grep ${PORT}
```

### Check the disk usage of the affected instance

```shell
ssh ${INSTANCE} df -h
```

### Check the logs related to the affected service

```shell
journalctl -u ${SERVICE_NAME}.service
```

### Check the status of the affected service

```shell
systemctl status ${SERVICE_NAME}.service
```

### The system may be overloaded with too many requests, causing the CPU to switch between different processes frequently, leading to high context switching rates.

```shell
bash

#!/bin/bash



# Check CPU usage

CPU_USAGE=$(top -bn1 | grep "Cpu(s)" | awk '{print $2 + $4}')

THRESHOLD=80



if (( $(echo "${CPU_USAGE} > ${THRESHOLD}" | bc -l) )); then

  echo "CPU usage is high. Current usage is ${CPU_USAGE}%"

  echo "Checking context switching rates..."

  

  # Check context switching rates

  CONTEXT_SWITCHES=$(vmstat 1 2 | tail -1 | awk '{print $12}')

  THRESHOLD_SWITCHES=10000

  

  if (( ${CONTEXT_SWITCHES} > ${THRESHOLD_SWITCHES} )); then

    echo "Context switching rate is high. Current rate is ${CONTEXT_SWITCHES}"

    echo "System may be overloaded with too many requests."

  else

    echo "Context switching rate is normal. Current rate is ${CONTEXT_SWITCHES}"

    echo "No issues detected."

  fi

else

  echo "CPU usage is normal. Current usage is ${CPU_USAGE}%"

  echo "No issues detected."

fi


```


### Consider restarting the affected service or host.

```shell
bash

#!/bin/bash

# Script to remediate high context switching rate issue



# Restart the affected service or host

echo "Restarting ${SERVICE_NAME} to remediate high context switching rate issue"

sudo systemctl restart ${SERVICE_NAME}



# Verify that the service or host has restarted successfully

echo "Verifying that ${SERVICE_NAME} has restarted successfully"

sudo systemctl status ${SERVICE_NAME}


```


High urgency incident related to host context switching.

Overview

Parameters

Debug

Check the CPU usage of the affected instance

Check the memory usage of the affected instance

Check the number of context switches per second on the affected instance

Check the network connection status of the affected instance

Check the disk usage of the affected instance

Check the status of the affected service

The system may be overloaded with too many requests, causing the CPU to switch between different processes frequently, leading to high context switching rates.

Repair

Consider restarting the affected service or host.

Learn more

Related Runbooks

Host Out of Memory (OOM) Incident

Tomcat High Swap Usage Incident

Increase of the errors/second rate for Tomcat server

High % of Busy Threads on Tomcat Host (TEST)

Support