Server Surgery

Jeff Wilson — Sun, 21 Jun 2015 12:42:09 EDT

So, gremlins. I guess that’s what happens with some hard disks. One of them turned a peaceful weekend into a mad scramble for backup space and mdadm tips. Call it a fire drill without the staircases. I guess the server means to say I’ve been taking it for granted.

Some lessons learned:

You can use drives with different geometry if you have to.
An IMPI display would have been very useful for this first-timer.
When rebuilding a software RAID-1 pair, don’t forget you need to reinstall grub!

Steps used

Confirm the system still sees the new drive as /dev/sda; that’s the disk that got retired from mdadm early on Saturday, and /dev/sdb is the working unit.

root@rescue:~# parted -l

The drive /dev/sda should have no partition table. More importantly /dev/sdb should have stuff!

[snip]
Disk /dev/sdb: 2000GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End     Size    Type     File system  Flags
 1      2097kB  21.5GB  21.5GB  primary  ext3         boot, raid
 2      21.5GB  1000GB  978GB   primary               raid
 3      1000GB  1000GB  538MB   primary

Confirmed: /dev/sdb is the good drive. Copy the partition tables from /dev/sdb to /dev/sda:

root@rescue:~# sfdisk -d /dev/sdb | sfdisk /dev/sda

Add the new partitions back into the raid array:

root@rescue:~# mdadm /dev/md1 --manage --add /dev/sda1
root@rescue:~# mdadm /dev/md2 --manage --add /dev/sda2

Keep tabs on the progress (this is showing the rebuild of /dev/sdb I did after /dev/sda completed successfully):

root@rescue:~# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty]
md1 : active raid1 sdb1[1] sda1[0]
      20971456 blocks [2/2] [UU]

md2 : active raid1 sdb2[2] sda2[0]
      955260864 blocks [2/1] [U_]
      [================>....]  recovery = 84.6% (808489856/955260864) finish=17.4min speed=140544K/sec

unused devices: <none>

Redefine the swap areas.

root@rescue:~# mkswap /dev/sda3
root@rescue:~# mkswap /dev/sdb3
root@rescue:~# swapon -a

The first time around I rebooted here. That wasn’t enough. I had to setup grub, which is described here … in French:

root@rescue:~# mount /dev/md1 /mnt
root@rescue:~# chroot /mnt
root@rescue:~# grub-install --recheck /dev/sda
root@rescue:~# exit
root@rescue:~# umount /mnt

Reboot to the hard drive, and we are done.

So now we are on a new pair of drives, and for good measure I am also the proud (temporary) owner of a failover server too. Now I just need to decide which systems I want to keep running. Live backups are too good to pass up. And I might want it when I decide to tackle configuring my new terabyte of space.

μrsi

Server Surgery

Some lessons learned:

Steps used