[Cialug] Replacing failed RAID 1 drive

[Rob Cook]

That's a failed drive right there... when I got home tonight it was a
screaming unholy racket. The last pic looks to be where the head had been
machining a groove in the platter by the spindle. There are chunks on the
platter and the entire drive is dirty.

Popped in the new drive, booted up fine, rejoined the disk to the RAID via
mdadm after copying the partition scheme over from the existing drive, now
somewhere between the next 4-6 hours the array will be rebuilt. Next month
I'll get another new drive (I went from 1.5Tb to 3Tb on the replacement)
fail out the old drive and replace it with another 3Tb. Then expand the LVM
and *magic* twice the space... I hope ;)







On Wed, Oct 1, 2014 at 10:38 AM, Rob Cook <rdjcook at gmail.com> wrote:

> Well after today I may be able to give that demo. Lets hope that it all
> works properly.
>
> On Wed, Oct 1, 2014 at 8:39 AM, Matthew Nuzum <newz at bearfruit.org> wrote:
>
>> This would make a great topic for a LUG meeting sometime. Bring a box in,
>> set it up with RAID then fail and swap a drive.
>>
>> It's one of those things we all know we should do, and many of us do it,
>> but the actual hands-on experience of dealing with a failed drive is not
>> nearly as common. (I've never done in with software RAID, only done a
>> cold-swap with a hardware RAID controller)
>>
>> On Tue, Sep 30, 2014 at 5:35 PM, Rob Cook <rdjcook at gmail.com> wrote:
>>
>> > "When you say “LVM on top of RAID” I assume you mean something like
>> this:
>> >         /dev/sd* (physical block devices) => md0 (mdadm array) => pv1
>> (LVM
>> > “physical” volume) => vg1 (LVM volume group) => lv1 (LVM logical
>> volume) =>
>> > /mnt/foo (filesystem)"
>> >
>> > Yes, like that exactly.
>> >
>> > Ok, off to buy a new drive.
>> >
>> > On Tue, Sep 30, 2014 at 5:25 PM, Zachary Kotlarek <zach at kotlarek.com>
>> > wrote:
>> >
>> > >
>> > > On Sep 30, 2014, at 3:07 PM, Rob Cook <rdjcook at gmail.com> wrote:
>> > >
>> > > > I have a CentOS 6.5 box with 2 1.5Tb drives in a RAID 1 with LVM
>> > > partitions
>> > > > on top of that. One of the drives /dev/sdb has failed.
>> > > >
>> > > > I've been googling quite a bit and I think that I should be ok
>> > following
>> > > > this guide:
>> > > >
>> > > > http://www.howtoforge.com/replacing_hard_disks_in_a_raid1_array
>> > > >
>> > > > Fail then remove the drive from the array, replace with similar or
>> > larger
>> > > > then recreate. The one question I have is what to do with the LVM
>> > > > partitons? Naively they should recreate given this is a RAID 1 so
>> it's
>> > > the
>> > > > same data on both drives so I shouldn't have to worry. Or is that to
>> > > > simplistic of a view?
>> > >
>> > >
>> > > When you say “LVM on top of RAID” I assume you mean something like
>> this:
>> > >         /dev/sd* (physical block devices) => md0 (mdadm array) => pv1
>> > (LVM
>> > > “physical” volume) => vg1 (LVM volume group) => lv1 (LVM logical
>> volume)
>> > =>
>> > > /mnt/foo (filesystem)
>> > >
>> > > If that’s the case then the LVM physical volume and everything higher
>> in
>> > > the stack has no idea that you’re swapping disks and doesn’t need to
>> be
>> > > told anything.
>> > >
>> > > —
>> > >
>> > > On a related note, sometimes mdadm commands that reference physical
>> > > devices, like this:
>> > >         mdadm --manage /dev/md1 --fail /dev/sdb2
>> > > will fail with an error like:
>> > >         No such device: /dev/sdb2
>> > > because the file /dev/sdb2 no longer exists (because the disk is dead
>> or
>> > > pulled).
>> > >
>> > > But you still need to tell mdadm about it so it can update the array.
>> > > Instead you should use the short name:
>> > >         mdadm --manage /dev/md1 --fail sdb2
>> > > or whatever other device name shows up when you ask mdadm about the
>> array
>> > > or look at /proc/mdstat. That bypasses any device-file lookup and uses
>> > the
>> > > references that mdadm tracks internally.
>> > >
>> > >         Zach
>> > >
>> > >
>> > > _______________________________________________
>> > > Cialug mailing list
>> > > Cialug at cialug.org
>> > > http://cialug.org/mailman/listinfo/cialug
>> > >
>> > >
>> > _______________________________________________
>> > Cialug mailing list
>> > Cialug at cialug.org
>> > http://cialug.org/mailman/listinfo/cialug
>> >
>>
>>
>>
>> --
>> Matthew Nuzum
>> newz2000 on freenode, skype, linkedin and twitter
>>
>> ♫ You're never fully dressed without a smile! ♫
>> _______________________________________________
>> Cialug mailing list
>> Cialug at cialug.org
>> http://cialug.org/mailman/listinfo/cialug
>>
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: IMG_20141001_173653.jpg
Type: application/vnd.google-apps.photo
Size: 241635 bytes
Desc: not available
URL: <http://cialug.org/pipermail/cialug/attachments/20141001/33aeab99/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: IMG_20141001_173625.jpg
Type: application/vnd.google-apps.photo
Size: 220748 bytes
Desc: not available
URL: <http://cialug.org/pipermail/cialug/attachments/20141001/33aeab99/attachment-0004.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: IMG_20141001_173614.jpg
Type: application/vnd.google-apps.photo
Size: 173389 bytes
Desc: not available
URL: <http://cialug.org/pipermail/cialug/attachments/20141001/33aeab99/attachment-0005.bin>