---
id: adb6ca51-fb55-438a-894d-f18c2ad91ea0
---

# Host RAID Disk Failure
---

Host RAID Disk Failure is an incident type that occurs when at least one device in a RAID array fails. This can result in a loss of data or service interruption. The incident requires attention and possibly a disk swap to restore the RAID array and prevent further failures.

### Parameters
```shell
export RAID_DEVICE="PLACEHOLDER"

export DISK_DEVICE="PLACEHOLDER"

export NEW_DISK="PLACEHOLDER"

export FAILED_DISK="PLACEHOLDER"

export BACKUP_FILE="PLACEHOLDER"

export NUMBER_OF_DISKS="PLACEHOLDER"
```

## Debug

### Check if the system recognizes the RAID array
```shell
cat /proc/mdstat
```

### Check the disks status in the RAID array
```shell
mdadm --detail /dev/md${RAID_DEVICE}
```

### Check the SMART status of all disks in the RAID array
```shell
smartctl -a /dev/${DISK_DEVICE}
```

### Check the logs for disk errors
```shell
dmesg | grep -i "hard error"
```

### Check the system logs for disk errors
```shell
grep -i "I/O error" /var/log/messages
```

### Check the RAID events log
```shell
cat /var/log/messages | grep -i "mdadm"
```

### Check the disk health using S.M.A.R.T
```shell
smartctl -H /dev/${DISK_DEVICE}
```

### Check the filesystem consistency on the RAID array
```shell
fsck /dev/md${RAID_DEVICE}
```

### Check the RAID array integrity and rebuild status
```shell
mdadm --detail /dev/md${RAID_DEVICE}
```

### Check the partitions on the RAID array
```shell
lsblk
```

## Repair

### Replace the failed disk in the RAID array and rebuild the array.
```shell


#!/bin/bash



# Replace the failed disk in the RAID array

echo "Please replace the failed disk with a new one."

echo "Once you have replaced the disk, press Enter to continue."

read



# Rebuild the RAID array

echo "Rebuilding the RAID array..."

mdadm --manage /dev/md0 --remove /dev/${FAILED_DISK}

mdadm --manage /dev/md0 --add /dev/${NEW_DISK}

mdadm --grow /dev/md0 --raid-devices=${NUMBER_OF_DISKS} --backup-file=/root/${BACKUP_FILE}

echo "RAID array successfully rebuilt."


```

Host RAID Disk Failure is an incident type that occurs when at least one device in a RAID array fails. This can result in a loss of data or service interruption. The incident requires attention and possibly a disk swap to restore the RAID array and prevent further failures.


This incident type refers to the restart of a MySQL instance that has caused an alert to trigger. It may be related to issues with the MySQL database or the server hosting the instance. The incident requires investigation and resolution to ensure the proper functioning of the affected services.


MySQL instance restart incident

This incident type refers to an issue where MySQL InnoDB log writes are stalling. It may result in poor database performance, slow queries, and even server crashes in severe cases. The incident can be triggered by various factors such as high server load, large transaction logs, and insufficient disk space. It requires immediate attention and resolution to prevent any further impact on the system.


MySQL InnoDB log waits incident.

This incident type refers to an issue with a MongoDB replica set, where one or more members of the set have been marked as unhealthy. This can happen due to various causes, such as network issues, hardware failures, or configuration problems. When this occurs, it can impact the availability and performance of the database system, which can lead to data loss or corruption. Prompt resolution of this incident is necessary to prevent further damage and restore the normal functioning of the replica set.


MongoDB replica member unhealthy incident.

This incident type refers to the failure of a systemd service on a particular host instance. The incident could be triggered by various causes such as a software bug, hardware failure, or system overload. This type of incident can cause downtime or service disruption to the affected host instance, which may require immediate resolution to restore normal operations.


Host systemd service crashed (instance) incident.

This incident type occurs when a host requires a reboot. It could be due to various reasons like an update, configuration changes, or malfunctioning. If the host is not rebooted, it could cause performance issues or even lead to downtime. Therefore, immediate action is necessary to resolve the issue and restore the system to its optimal state. The incident could be triggered automatically by monitoring tools or reported by users.


Host requires reboot incident.

```shell
export RAID_DEVICE="PLACEHOLDER"

export DISK_DEVICE="PLACEHOLDER"

export NEW_DISK="PLACEHOLDER"

export FAILED_DISK="PLACEHOLDER"

export BACKUP_FILE="PLACEHOLDER"

export NUMBER_OF_DISKS="PLACEHOLDER"
```


### Check if the system recognizes the RAID array

```shell
cat /proc/mdstat
```

### Check the disks status in the RAID array

```shell
mdadm --detail /dev/md${RAID_DEVICE}
```

### Check the SMART status of all disks in the RAID array

```shell
smartctl -a /dev/${DISK_DEVICE}
```

### Check the logs for disk errors

```shell
dmesg | grep -i "hard error"
```

### Check the system logs for disk errors

```shell
grep -i "I/O error" /var/log/messages
```

### Check the RAID events log

```shell
cat /var/log/messages | grep -i "mdadm"
```

### Check the disk health using S.M.A.R.T

```shell
smartctl -H /dev/${DISK_DEVICE}
```

### Check the filesystem consistency on the RAID array

```shell
fsck /dev/md${RAID_DEVICE}
```

### Check the RAID array integrity and rebuild status

```shell
mdadm --detail /dev/md${RAID_DEVICE}
```

### Check the partitions on the RAID array

```shell
lsblk
```


### Replace the failed disk in the RAID array and rebuild the array.

```shell


#!/bin/bash



# Replace the failed disk in the RAID array

echo "Please replace the failed disk with a new one."

echo "Once you have replaced the disk, press Enter to continue."

read



# Rebuild the RAID array

echo "Rebuilding the RAID array..."

mdadm --manage /dev/md0 --remove /dev/${FAILED_DISK}

mdadm --manage /dev/md0 --add /dev/${NEW_DISK}

mdadm --grow /dev/md0 --raid-devices=${NUMBER_OF_DISKS} --backup-file=/root/${BACKUP_FILE}

echo "RAID array successfully rebuilt."


```


Host RAID Disk Failure

Overview

Parameters

Debug

Check if the system recognizes the RAID array

Check the disks status in the RAID array

Check the SMART status of all disks in the RAID array

Check the logs for disk errors

Check the system logs for disk errors

Check the RAID events log

Check the disk health using S.M.A.R.T

Check the filesystem consistency on the RAID array

Check the RAID array integrity and rebuild status

Check the partitions on the RAID array

Repair

Replace the failed disk in the RAID array and rebuild the array.

Learn more

Related Runbooks

MySQL instance restart incident

MySQL InnoDB log waits incident.

MongoDB replica member unhealthy incident.

Host systemd service crashed (instance) incident.

Support