[Cialug] LVM/ext3 problem

[Zachary Kotlarek]

On Oct 22, 2012, at 8:10 PM, L. V. Lammert <lvl at omnitec.net> wrote:

> It got into this state after being resized to the size of the failed PV,
> and removing the bad PV seems to been successful in the VG, however
> nothing seems to work in attempting to correct the size of the file system
> itself.
> 
> I have no information about the actual failure during resizing, .. just
> that the operation failed, so we removed the additional volume using
> vgreduce.
> 
> Unfortunately, that did not adjust the underlying filesystem. We tried to
> fix the old PV, but when we mounted, it showed corrupted, so there was no
> way to do an resize2fs on the filesystem with the extra volume in poace.


There's a layer of abstraction missing here, or maybe just crossed terminology, that's got me confused.

This is what linux sees, bottom to top:
	Block storage device (the actual storage device, /dev/md/1)
	PV (physical volume, a bit of metadata on the device to help LVM find/use it, also /dev/md/1)
	VG (volume group, a named group of 1 or more PVs)
	LV (logical volume, a named storage container, inside a VG, not tied to a specific device)
	Filesystem (ext2/3/4)

You talk about adding and removing a PV, but that just affects the amount of space available in the VG, it does not change the size of the LVs (i.e. the size of the "disk" that e2fs sees). You cannot vgreduce a VG unless there's enough space in the VG to continue to accommodate all the LVs (technically you *can* but it will argue with you -- you'd know if you did). So I'm confused about when and how the LV size changed.

I'm guessing you did something like:
	pvcreate
	vgextend
	lvextend
	resize2fs (to extend the filesystem to the new, larger size, worked)
	…something happened that makes you want to go back…
	resize2fs (to reduce the filesystem back to the old size, failed)
	lvreduce
	vgreduce
but that's just a guess. Please let me know if/where that's wrong.

If you did an `lvreduce` at some point data truncated from the end of the volume is officially lost.  However, the `vgcfgrestore` will allow you to roll back to previously backed up VG configuration, essentially reverting LV and VG changes. Backups are created by default with all the `vgchange` tools, so you should have them unless you explicitly disabled them. Of course that assumes all the old PVs are still available; obviously it doesn't do any good to restore a configuration that refers to a now-missing disk.


> Any other options come to mind?


If at all possible, consider making a block-level copy of the entire LV and attempting all these operations on that copy rather than the live disk. `dd` will work wonderfully, and once you have an image of the disk you can use standard tools to change the size of that image file, which is logically equivalent to changing the size of the LV in LVM-land. If you get a working copy you can use `dd` or even `tar` to restore back to the real disk, or you can repeat the same sequence of operations on the live data.

If you are confident that you don't have any useful data past the current end of the LV, you can probably just fake it. Re-extend the LV to match the old, larger size (using any disk -- doesn't matter what's there), then run fsck to get your filesystem back into a useable state. It will have lots of complaints once it starts reading past the end of the current LV size, but since you didn't have any useful data there you can generally ignore them. Once you're confident that the filesystem is sound and your old data is accessible, use resize2fs to reduce the filesystem and then lvreduce to reduce the LV like you tried to before.

Otherwise the options are fairly grim. Restore from backup. Use debugfs to recover the files directly (this may be a better plan than the expand-fsck-resize-reduce one I noted above, though also more work if you have lots of files). If things are really borked maybe write a clean filesystem header over the busted one but keep the old inode trees (not trivial and quite dangerous).

	Zach

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2746 bytes
Desc: not available
URL: <http://cialug.org/pipermail/cialug/attachments/20121022/4de21993/attachment.bin>