Wednesday, December 12, 2012

Extending root partition on the fly - Part 2

In my previous post I discuss some techniques to expand disks while ensuring zero downtime.

These techniques are not always viable as it will depend on the Operating System version you have, as well as LVM version.  If you don't use LVM and instead use native linux partitions, then things can get a bit uglier and you will probably need one or two reboots.  In CentOS 5.2 I can't properly re-scan the ISCSI devices.  It seems support for it may have only been added in version 5.4, as RedHat began adding shell scripts to perform this operation.

In any case, for the latest version of RedHat and Centos (5.5,6,6.2,6.3) the previously described techniques work just fine.  There is only one caveat.  In order to prevent having to disable the volume group, unmounting the filesystem and stopping services that are using them, one must create a new "Physical Volume" using pvcreate, as I've done.  The only problem with this, and it's not a big one, is that you end up with separated physical volumes all on the same LVM partition.

If you want to expand the disk but have only one physical volume in the LVM, it will be necessary to disable the volume group in order to use pvresize.  Note that this implies shutting down services and unmounting the filesystem to be expanded.

Example:

1) Extend disk in vmware

2) Rescan the disk
# echo "1" > /sys/class/scsi_device/<device number>/device/rescan

3) resize the partition in question using fdisk <device>
# fdisk /dev/sdb

4) re-read the partition table:
# partprobe

5) If your service is apache:
# service httpd stop

6) Disable the volume group
# vgchange -a n <volume group name>

7) Physical Volume Resize
# pvresize /dev/sdb1

7.1) If you check your physical volume now you should see the free extents:
# pvdisplay

8) Re-enable the volume group
# vgchange -a y <vg name>

9) Re-mount the device
# mount /var/www (for example) or mount -a

10) Restart your service
# service httpd start

11) Now note that we've only expanded the Volume Group and neither has the Logical Volume or the File System been extended, but we can do these on the fly so it's o.k. if they are mounted.
# lvextend -l +<num of free extents> /dev/<vg name>/<lv name>

13) Resize the filesystem
# resize2fs /dev/<vg name>/<lv name>

In our case we find that it's helpful to test these procedures on clones and ensuring we have the most appropriate technique for the situation.