Hetzner - DokuWiki

Festplattenaustausch im Software-RAID/en

This article uses an example to describe the necessary steps involved in exchanging a defective drive in a Software RAID (mdadm).

IMPORTANT NOTE: All commands are examples and should be adjusted accordingly!

A video how-to can be found on our youtube channel: Hetzner Youtube Channel

Inhaltsverzeichnis

Example scenario

The following configuration is assumed:

# cat /proc/mdstat
Personalities : [raid1]
md3 : active raid1 sda4[0] sdb4[1]
      1822442815 blocks super 1.2 [2/2] [UU]

md2 : active raid1 sda3[0] sdb3[1]
      1073740664 blocks super 1.2 [2/2] [UU]

md1 : active raid1 sda2[0] sdb2[1]
      524276 blocks super 1.2 [2/2] [UU]

md0 : active raid1 sda1[0] sdb1[1]
      33553336 blocks super 1.2 [2/2] [UU]

unused devices: <none>

There are four partitions in total:

  • /dev/md0 as swap
  • /dev/md1 as /boot
  • /dev/md2 as /
  • /dev/md3 as /home

/dev/sdb is the defective drive in this case. A missing or defective drive is shown by [U_] and/or [_U]. If the RAID array is intact, it shows [UU].

# cat /proc/mdstat
Personalities : [raid1]
md3 : active raid1 sda4[0] sdb4[1](F)
      1822442815 blocks super 1.2 [2/1] [U_]

md2 : active raid1 sda3[0] sdb3[1](F)
      1073740664 blocks super 1.2 [2/1] [U_]

md1 : active raid1 sda2[0] sdb2[1](F)
      524276 blocks super 1.2 [2/1] [U_]

md0 : active raid1 sda1[0] sdb1[1](F)
      33553336 blocks super 1.2 [2/1] [U_]

unused devices: <none>

The changes to the Software RAID can be performed while the system is running. If proc/mdstat shows that the drive is failing, like the example here, then an appointment can be made with the support technicians to replace the drive.

# cat /proc/mdstat
Personalities : [raid1]
md3 : active raid1 sda4[0]
      1822442815 blocks super 1.2 [2/1] [U_]

md2 : active raid1 sda3[0]
      1073740664 blocks super 1.2 [2/1] [U_]

md1 : active raid1 sda2[0]
      524276 blocks super 1.2 [2/1] [U_]

md0 : active raid1 sda1[0]
      33553336 blocks super 1.2 [2/1] [U_]

unused devices: <none>

Removal of the defective drive

Before a new drive can be added the old defective drive needs to be removed from the RAID array. This needs to be done for each individual partition.

# mdadm /dev/md0 -r /dev/sdb1
# mdadm /dev/md1 -r /dev/sdb2
# mdadm /dev/md2 -r /dev/sdb3
# mdadm /dev/md3 -r /dev/sdb4

The following command shows the drives that are part of an array:

# mdadm --detail /dev/md0

In some cases a drive may only be partly defective, so for example only /dev/md0 is in the [U_] state, whereas all other devices are in the [UU] state. In this case the command

# mdadm /dev/md1 -r /dev/sdb2

fails, as the /dev/md1 array is ok.

In this event, the command

# mdadm --manage /dev/md1 --fail /dev/sdb2

needs to be executed first, to move the RAID into [U_] status.

Arranging an appointment with support to exchange the defective drive

In order to be able to exchange the defective drive, it is necessary to arrange an appointment with support in advance. The server will need to be taken off-line for a short time.

Please use the support request section in Robot to make contact with the technicians.

Preparing the new drive

Both drives in the array need to have the exact same partitioning. Depending on the partition table type used (MBR or GPT), appropriate utilities have to be used to copy the partition table. The GPT partition table is usually used in larger then 2TiB drives (e.g. 3TB HDDs in EX4 and EX6)

Backing up the MBR/GPT

Before copying the MBR/GPT to a new drive it needs to be backed up. That way, if something goes wrong during the copying, the original can still be restored.

# Backup with MBR
sfdisk --dump /dev/sda > sda_parttable_mbr.bak
sfdisk --dump /dev/sdb > sdb_parttable_mbr.bak
sfdisk --dump /dev/sdX > sdX_parttable_mbr.bak

# Restore with MBR
sfdisk /dev/sda < sda_parttable_mbr.bak
sfdisk /dev/sdb < sdb_parttable_mbr.bak
sfdisk /dev/sdX < sdX_parttable_mbr.bak
# Backup with GPT
sgdisk --backup=sda_parttable_gpt.bak /dev/sda
sgdisk --backup=sdb_parttable_gpt.bak /dev/sdb
sgdisk --backup=sdX_parttable_gpt.bak /dev/sdX

# Restore with GPT
sgdisk --load-backup=sda_parttable_gpt.bak /dev/sda
sgdisk --load-backup=sdb_parttable_gpt.bak /dev/sdb
sgdisk --load-backup=sdX_parttable_gpt.bak /dev/sdX

Drives with GPT

There are several reduntant copies of the GUID partition table (GPT) stored on the drive, so tools that support GPT, for example parted or GPT fdisk, need to be used to edit the table. The sgdisk tool from GPT fdisk (pre-installed when using the Rescue System) can be used to easily copy the partition table to a new drive. Here's an example of copying the partition table from sda to sdb:

sgdisk --backup=sda_parttable_gpt.bak /dev/sda
sgdisk --load-backup=sda_parttable_gpt.bak /dev/sdb

The drive then needs to be assigned a new random UUID:

sgdisk -G /dev/sdb

After this the drive can be added to the array. As a final step the bootloader needs to be installed.

Drives with MBR

The partition table can be simply copied to a new drive using sfdisk:

# sfdisk -d /dev/sda | sfdisk /dev/sdb

where /dev/sda is the source drive and /dev/sdb is the target drive.

(Optional): If the partitions are not detected by the system then the partition table has to be reread from the kernel:

# sfdisk -R /dev/sdb

Naturally, the partitions may also be created manually using fdisk, cfdisk or other tools. The partitions should be Linux raid autodetect (ID fd) types.

Integration of the new drive

Once the defective drive has been removed and the new one installed, it needs to be intagrated into the RAID array. This needs to be done for each partition.

# mdadm /dev/md0 -a /dev/sdb1
# mdadm /dev/md1 -a /dev/sdb2
# mdadm /dev/md2 -a /dev/sdb3
# mdadm /dev/md3 -a /dev/sdb4

The new drive is now part of the array and will be synchronized. Depending on the size of the partitions this procedure can take some time. The status of the synchronization can be observed using cat /proc/mdstat.

# cat /proc/mdstat
Personalities : [raid1]
md3 : active raid1 sdb4[1] sda4[0]
      1028096 blocks [2/2] [UU]
      [==========>..........]  resync =  50.0% (514048/1028096) finish=97.3min speed=65787K/sec

md2 : active raid1 sdb3[1] sda3[0]
      208768 blocks [2/2] [UU]

md1 : active raid1 sdb2[1] sda2[0]
      2104448 blocks [2/2] [UU]

md0 : active raid1 sdb1[1] sda1[0]
      208768 blocks [2/2] [UU]

unused devices: <none>

Bootloader installation

If you are doing this repair in a booted system, then for GRUB2 running grub-install on the new drive is enough. For example:

grub-install /dev/sdb

In GRUB1 (legacy GRUB), depending on which drive was defective, more steps might be required.

  • Start the GRUB console: grub
  • Specify the partition where /boot is located: root (hd0,1) (/dev/sda2 = (hd0,1))
  • Install the bootloader in MBR: setup (hd0)
  • For also installing the bootloader on the second drive:
    • Map the second drive as hd0: device (hd0) /dev/sdb
    • Repeat steps 2 and 3 exactly (don't change the commands)
  • Exit the GRUB console: quit
Probing devices to guess BIOS drives. This may take a long time.


    GNU GRUB  version 0.97  (640K lower / 3072K upper memory)

 [ Minimal BASH-like line editing is supported.  For the first word, TAB
   lists possible command completions.  Anywhere else TAB lists the possible
   completions of a device/filename.]
grub> device (hd0) /dev/sdb
device (hd0) /dev/sdb
grub> root (hd0,1)
root (hd0,1)
 Filesystem type is ext2fs, partition type 0xfd
grub> setup (hd0)
setup (hd0)
 Checking if "/boot/grub/stage1" exists... yes
 Checking if "/boot/grub/stage2" exists... yes
 Checking if "/boot/grub/e2fs_stage1_5" exists... yes
 Running "embed /boot/grub/e2fs_stage1_5 (hd0)"...  26 sectors are embedded.
succeeded
 Running "install /boot/grub/stage1 (hd0) (hd0)1+26 p (hd0,1)/boot/grub/stage2 /boot/grub/grub.conf"... succeeded
Done.
grub> quit
#

For repairing via the Rescue System, the installed system has to be mounted first, as described here. All GRUB installation steps then have to be performed after chroot.



© 2018. Hetzner Online GmbH. Alle Rechte vorbehalten.