---
id: 46329157-2417-495c-8d5c-51f1142e9acf
---

# Host requires reboot incident.
---

This incident type occurs when a host requires a reboot. It could be due to various reasons like an update, configuration changes, or malfunctioning. If the host is not rebooted, it could cause performance issues or even lead to downtime. Therefore, immediate action is necessary to resolve the issue and restore the system to its optimal state. The incident could be triggered automatically by monitoring tools or reported by users.

### Parameters
```shell
export INSTANCE_IP_ADDRESS="PLACEHOLDER"

export SERVICE_NAME="PLACEHOLDER"

export DISK_PATH="PLACEHOLDER"

export HOSTNAME="PLACEHOLDER"
```

## Debug

### 1. Check which instance requires a reboot
```shell
ssh ${INSTANCE_IP_ADDRESS} "uptime"
```

### 2. Check if the disk space is full
```shell
ssh ${INSTANCE_IP_ADDRESS} "df -h"
```

### 3. Check if there are any processes that are consuming high CPU or memory
```shell
ssh ${INSTANCE_IP_ADDRESS} "top"
```

### 4. Check if there are any logged errors related to the host or instance
```shell
journalctl -u ${SERVICE_NAME} --since "24 hours ago"
```

### 5. Check if there are any pending software updates
```shell
ssh ${INSTANCE_IP_ADDRESS} "sudo apt update && sudo apt list --upgradable"
```

### 6. Check the system logs for any hardware-related errors
```shell
dmesg | grep -i error
```

### 7. Check the network connectivity between the host and other services
```shell
ping ${SERVICE_NAME}
```

### 8. Check the status of any relevant services or daemons
```shell
systemctl status ${SERVICE_NAME}
```

### The host system might have encountered a critical error or failure, leading to a reboot requirement.
```shell


#!/bin/bash



# Check system logs for recent critical errors

if sudo grep -q 'CRITICAL' /var/log/syslog; then

    echo "System has encountered critical errors"

    echo "Reboot required"

    exit 0

fi



# Check system status for any failures

if sudo systemctl status | grep -q 'failed'; then

    echo "System has failed services"

    echo "Reboot required"

    exit 0

fi



# Check disk usage for critical levels

if [ "$(df -h ${DISK_PATH} | awk '{print $5}' | tail -n 1 | tr -d '%')" -ge 90 ]; then

    echo "Disk usage is critical"

    echo "Reboot required"

    exit 0

fi



echo "No issues found. System functioning normally."


```

## Repair

### Reboot the host
```shell


#!/bin/bash



# Define variables

HOSTNAME=${HOSTNAME}



# Schedule the reboot

echo "Rebooting the host now..."

sudo shutdown -r now



# Wait for the reboot to complete

echo "Waiting for host to reboot..."

sleep 60
```

### Verify that the host has successfully rebooted and that all services and applications have started up correctly.
```shell


#!/bin/bash



# Verify that the host has successfully rebooted

echo "Verifying host status"

if ping -c 3 ${HOSTNAME} &> /dev/null

then

    echo "Host has successfully rebooted"

else

    echo "Host reboot failed"

    exit 1

fi



# Check that all services and applications have started up correctly

if systemctl status ${SERVICE_NAME} | grep 'Active: active (running)' &> /dev/null

then

    echo "All services and applications have started up correctly"

else

    echo "Some services or applications failed to start up"

    exit 1

fi



exit 0


```

This incident type occurs when a host requires a reboot. It could be due to various reasons like an update, configuration changes, or malfunctioning. If the host is not rebooted, it could cause performance issues or even lead to downtime. Therefore, immediate action is necessary to resolve the issue and restore the system to its optimal state. The incident could be triggered automatically by monitoring tools or reported by users.


The Host Out of Memory (OOM) Incident occurs when a server or system runs out of memory, causing it to crash or become unresponsive. This can be caused by various factors, such as an unexpected surge in traffic or insufficient resources allocated to the system. Resolving this type of incident requires identifying the root cause of the memory issue and taking appropriate measures such as optimizing system resources or increasing memory capacity.


Host Out of Memory (OOM) Incident

This incident type refers to a situation where Redis, a popular in-memory data structure store, has rejected connections. This could happen due to various reasons such as server overload, network issues, or misconfiguration. When this occurs, it can cause disruptions in the performance of applications that rely on Redis for data storage or caching. Thus, it requires immediate attention and resolution to minimize the impact on the applications and users.


Redis Rejected Connections Incident

This incident type refers to the restart of a MySQL instance that has caused an alert to trigger. It may be related to issues with the MySQL database or the server hosting the instance. The incident requires investigation and resolution to ensure the proper functioning of the affected services.


MySQL instance restart incident

This incident type refers to the failure of a systemd service on a particular host instance. The incident could be triggered by various causes such as a software bug, hardware failure, or system overload. This type of incident can cause downtime or service disruption to the affected host instance, which may require immediate resolution to restore normal operations.


Host systemd service crashed (instance) incident.

Host RAID Disk Failure is an incident type that occurs when at least one device in a RAID array fails. This can result in a loss of data or service interruption. The incident requires attention and possibly a disk swap to restore the RAID array and prevent further failures.


Host RAID Disk Failure

```shell
export INSTANCE_IP_ADDRESS="PLACEHOLDER"

export SERVICE_NAME="PLACEHOLDER"

export DISK_PATH="PLACEHOLDER"

export HOSTNAME="PLACEHOLDER"
```


### 1. Check which instance requires a reboot

```shell
ssh ${INSTANCE_IP_ADDRESS} "uptime"
```

### 2. Check if the disk space is full

```shell
ssh ${INSTANCE_IP_ADDRESS} "df -h"
```

### 3. Check if there are any processes that are consuming high CPU or memory

```shell
ssh ${INSTANCE_IP_ADDRESS} "top"
```

### 4. Check if there are any logged errors related to the host or instance

```shell
journalctl -u ${SERVICE_NAME} --since "24 hours ago"
```

### 5. Check if there are any pending software updates

```shell
ssh ${INSTANCE_IP_ADDRESS} "sudo apt update && sudo apt list --upgradable"
```

### 6. Check the system logs for any hardware-related errors

```shell
dmesg | grep -i error
```

### 7. Check the network connectivity between the host and other services

```shell
ping ${SERVICE_NAME}
```

### 8. Check the status of any relevant services or daemons

```shell
systemctl status ${SERVICE_NAME}
```

### The host system might have encountered a critical error or failure, leading to a reboot requirement.

```shell


#!/bin/bash



# Check system logs for recent critical errors

if sudo grep -q 'CRITICAL' /var/log/syslog; then

    echo "System has encountered critical errors"

    echo "Reboot required"

    exit 0

fi



# Check system status for any failures

if sudo systemctl status | grep -q 'failed'; then

    echo "System has failed services"

    echo "Reboot required"

    exit 0

fi



# Check disk usage for critical levels

if [ "$(df -h ${DISK_PATH} | awk '{print $5}' | tail -n 1 | tr -d '%')" -ge 90 ]; then

    echo "Disk usage is critical"

    echo "Reboot required"

    exit 0

fi



echo "No issues found. System functioning normally."


```


### Reboot the host

```shell


#!/bin/bash



# Define variables

HOSTNAME=${HOSTNAME}



# Schedule the reboot

echo "Rebooting the host now..."

sudo shutdown -r now



# Wait for the reboot to complete

echo "Waiting for host to reboot..."

sleep 60
```

### Verify that the host has successfully rebooted and that all services and applications have started up correctly.

```shell


#!/bin/bash



# Verify that the host has successfully rebooted

echo "Verifying host status"

if ping -c 3 ${HOSTNAME} &> /dev/null

then

    echo "Host has successfully rebooted"

else

    echo "Host reboot failed"

    exit 1

fi



# Check that all services and applications have started up correctly

if systemctl status ${SERVICE_NAME} | grep 'Active: active (running)' &> /dev/null

then

    echo "All services and applications have started up correctly"

else

    echo "Some services or applications failed to start up"

    exit 1

fi



exit 0


```


Host requires reboot incident.

Overview

Parameters

Debug

1. Check which instance requires a reboot

2. Check if the disk space is full

3. Check if there are any processes that are consuming high CPU or memory

5. Check if there are any pending software updates

7. Check the network connectivity between the host and other services

8. Check the status of any relevant services or daemons

The host system might have encountered a critical error or failure, leading to a reboot requirement.

Repair

Reboot the host

Verify that the host has successfully rebooted and that all services and applications have started up correctly.

Learn more

Related Runbooks

Host Out of Memory (OOM) Incident

Redis Rejected Connections Incident

MySQL instance restart incident

Host systemd service crashed (instance) incident.

Support