Today there are many ways to create and maintain a large data storage system that is reliable, resizable, and cheap. The built in software RAID5/6 and LVM(Logical Volume Management) tools in Linux coupled with inexpensive drive arrays make this possible. As of 2007 you can build a Linux based system with over 12TB of usable storage for under $7,000. A NAS or SAN system with similar storage will cost you at least twice as much. In this article I'm going to go over the steps using Linux to setup resizable storage using LVM.

Like many things in linux there is more than one way to build a grow-able file system. You can use multiple hardware RAID arrays with LVM. You can use the Linux software RAID tools with LVM. Or you can use all three together. For example you could get two 16 bay SATA JBOD enclosures and use software RAID5 or 6 with LVM to make one huge storage space. Or you could buy a SATA card that does hardware RAID and all the devices would show up as one combined device under Linux. Use LVM on it and you can grow that in the future.

A proper hardware RAID array will show up in Linux as a single drive. The RAID controller is what is used to manage the array. This makes it easy for us on the Linux side since we have less variables to deal with. All we need to do is partition the device and add the partition to a logical volume group. In this example I will use two 2GB disk drives as our two hardware RAID arrays. I will create a logical volume group and use one of the disks. I will then add the other disk and expand the logical volume to include the new space.

Note: The steps in this article work under Red Hat Enterprise Linux 5 and Ubuntu 7.04. For Ubuntu users you will need to have LVM2 and resize2fs installed. Run the following command to install them.

sudo apt-get install lvm2 resize2fs

First thing to do is list the arrays in fdisk to make sure Linux sees them. Use the “fdisk -l” command.

fdisk -l

Disk /dev/sda: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 1244 9992398+ 83 Linux
/dev/sda2 1245 1305 489982+ 5 Extended
/dev/sda5 1245 1305 489951 82 Linux swap / Solaris

Disk /dev/sdb: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdb doesn't contain a valid partition table

Disk /dev/sdc: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdc doesn't contain a valid partition table  

My two drives show up as /dev/sdb and /dev/sdc. Parallel ATA drives show up as /dev/hdx. Serial ATA drives, USB , Firewire, SCSI, and most disk arrays show up as /dev/sdx. The x at the end donates the order of the drive using alphabet characters a-x.

Now we need to setup a partition on these disks.

fdisk /dev/sdb

Hit the n key then hit return. Hit the p key to choose primary partition then hit return. hit the 1 key then hit return. Hit the return key two more times to select the default of using the whole drive. Hit the t key and type 8e and hit return. This will make the partition as a Linux LVM. Hit the w key and then hit return to write the changes to disk. Repeat the above steps for the other disk array.

After you are done partitioning the disks they will be represented as a 1 next to the device. Do another fdisk -l to see the new partitions.

fdisk -l

Disk /dev/sda: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 1244 9992398+ 83 Linux
/dev/sda2 1245 1305 489982+ 5 Extended
/dev/sda5 1245 1305 489951 82 Linux swap / Solaris

Disk /dev/sdb: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 261 2096451 8e Linux LVM

Disk /dev/sdc: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdc1 1 261 2096451 8e Linux LVM

We can now add one of the disks to be seen by LVM as a physical volume. Use the pvcreate command.

pvcreate /dev/sdb1
Physical volume "/dev/sdb1" successfully created

Create the volume group using the vgcreate command. I will name mine array-pool.

vgcreate array-pool /dev/sdb1
Volume group "array-pool" successfully created

Create a logical volume using the lvcreate command.

lvcreate -L 2000 array-pool
Logical volume "lvol0" created

The above command created a logical volume of 2GB with a default name lvol0.

Format the logical volume with your desired filesystem. Here I will use ext3.

mkfs.ext3 /dev/array-pool/lvol0

mke2fs 1.40-WIP (14-Nov-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
256000 inodes, 512000 blocks
25600 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=524288000
16 block groups
32768 blocks per group, 32768 fragments per group
16000 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912

Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 32 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.

Create a mount point for the volume.

mkdir /mnt/storage

Mount the new volume.

mount /dev/array-pool/lvol0 /mnt/storage

Check to see that the new storage is available using the df -h command.

df -h

Filesystem Size Used Avail Use% Mounted on
/dev/sda1 9.4G 2.4G 6.6G 27% /
varrun 252M 104K 252M 1% /var/run
varlock 252M 0 252M 0% /var/lock
procbususb 252M 92K 252M 1% /proc/bus/usb
udev 252M 92K 252M 1% /dev
devshm 252M 0 252M 0% /dev/shm
lrm 252M 33M 219M 14% /lib/modules/2.6.20-16-generic/volatile
/dev/mapper/array--pool-lvol0
2.0G 35M 1.8G 2% /mnt/storage

We will now add our other disk to be seen by LVM. Adding /dev/sdc1 as a physical volume

pvcreate /dev/sdc1
Physical volume "/dev/sdc1" successfully created

Adding the second disk to the array-pool volume group.

vgextend array-pool /dev/sdc1
Volume group "array-pool" successfully extended

Checking the volume group using the pvs command.

pvs

PV VG Fmt Attr PSize PFree
/dev/sdb1 array-pool lvm2 a- 2.00G 44.00M
/dev/sdc1 array-pool lvm2 a- 2.00G 2.00G

To use all the space of the combined disks check to see how many physical extents are available using the vgdisplay command.

vgdisplay array-pool


--- Volume group ---
VG Name array-pool
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 3
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 2
Act PV 2
VG Size 3.99 GB
PE Size 4.00 MB
Total PE 1022
Alloc PE / Size 500 / 1.95 GB
Free PE / Size 522 / 2.04 GB
VG UUID 4YFBMT-18kP-jLRP-Pmsb-X7bU-GC0J-c10qvX

The section that shows Free PE/Size is what we are interested in. Add the extra space to the logical volume.

lvextend -l+522 /dev/array-pool/lvol0

Extending logical volume lvol0 to 3.99 GB
Logical volume lvol0 successfully resized

Unmount the volume to resize the ext3 filesystem.

umount /mnt/storage/

Grow the ext3 filesystem to use the new space.

resize2fs /dev/array-pool/lvol0

Remount the volume.

mount /dev/array-pool/lvol0 /mnt/storage/

Check the size of the volume.

df -h

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/array--pool-lvol0 4.0G 36M 3.7G 1% /mnt/storage

Note: I prefer to use RAID6 for more fault tolerance, but unfortunately at the time of this article Ubuntu 7.04 nor RHEL5 include a kernel that supports growing a RAID6. You could compile one of the newer kernels but it would not be supported by the distribution providers. I will substitute the extra parity drive I would have with a hot spare.

First thing we want to do is create partitions on each disk. In my example I have five 2GB drives.

fdisk -l

Disk /dev/sda: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 1244 9992398+ 83 Linux
/dev/sda2 1245 1305 489982+ 5 Extended
/dev/sda5 1245 1305 489951 82 Linux swap / Solaris

Disk /dev/sdb: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

Disk /dev/sdc: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

Disk /dev/sdd: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdd doesn't contain a valid partition table

Disk /dev/sde: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sde doesn't contain a valid partition table

Disk /dev/sdf: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdf doesn't contain a valid partition table

I will put one partition of Linux RAID that takes up all the space on every disk.

fdisk /dev/sdb

Hit the n key then hit return. Hit the p key to choose primary partition then hit return. hit the 1 key then hit return. Hit the return key two more times to select the default of using the whole drive. Hit the t key and then hit return to set the System ID. You can hit the L key to show all the hex code choices. The hex code choice for linux raid auto is fd. Type fd and hit return. It should say “Changed system type of partition 1 to fd (Linux raid autodetect)”. hit the w key and then hit return to write the changes to disk. Repeat these steps for the other drives you will be using in the RAID.

After you are done partitioning the disks they will be represented as a 1 next to the device. Do another fdisk -l to see the new partitions.

fdisk -l

Disk /dev/sda: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 1244 9992398+ 83 Linux
/dev/sda2 1245 1305 489982+ 5 Extended
/dev/sda5 1245 1305 489951 82 Linux swap / Solaris

Disk /dev/sdb: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 261 2096451 fd Linux raid autodetect

Disk /dev/sdc: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdc1 1 261 2096451 fd Linux raid autodetect

Disk /dev/sdd: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdd1 1 261 2096451 fd Linux raid autodetect

Disk /dev/sde: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sde1 1 261 2096451 fd Linux raid autodetect

Disk /dev/sdf: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdf1 1 261 2096451 fd Linux raid autodetect

Using the mdadm command to create the RAID.

I want to use three of the drives in a RAID5 configuration with one hot spare.

Create the RAID using a command similar to the following.

mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sdb1 /dev/sdc1 /dev/sdd1

The output should look something like below.

mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 64K
mdadm: size set to 2096384K
mdadm: array /dev/md0 started.

You can use two commands to view the status of the RAID. mdadm –detail /dev/md0 and cat /proc/mdstat

mdadm --detail /dev/md0
/dev/md0:
Version : 00.90.03
Creation Time : Thu Sep 13 23:12:59 2007
Raid Level : raid5
Array Size : 4192768 (4.00 GiB 4.29 GB)
Device Size : 2096384 (2047.59 MiB 2146.70 MB)
Raid Devices : 3
Total Devices : 3
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Thu Sep 13 23:12:59 2007
State : clean, degraded, recovering
Active Devices : 2
Working Devices : 3
Failed Devices : 0
Spare Devices : 1

Layout : left-symmetric
Chunk Size : 64K

Rebuild Status : 96% complete

UUID : f5c24ee9:27cb0d42:e368bf24:bd0fce41 (local to host ubuntu)
Events : 0.1

Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
3 8 49 2 spare rebuilding /dev/sdd1

Lets add one more disk to the RAID as a hot spare using the mdadm –add command.

mdadm --add /dev/md0 /dev/sde1
mdadm: added /dev/sde1

Use the mdadm –detail command to make sure it is added.

mdadm --detail /dev/md0
/dev/md0:
Version : 00.90.03
Creation Time : Thu Sep 13 23:12:59 2007
Raid Level : raid5
Array Size : 4192768 (4.00 GiB 4.29 GB)
Device Size : 2096384 (2047.59 MiB 2146.70 MB)
Raid Devices : 3
Total Devices : 4
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Thu Sep 13 23:35:23 2007
State : clean
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1

Layout : left-symmetric
Chunk Size : 64K

UUID : f5c24ee9:27cb0d42:e368bf24:bd0fce41 (local to host ubuntu)
Events : 0.10

Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
2 8 49 2 active sync /dev/sdd1

3 8 65 - spare /dev/sde1

Now that you have created a RAID we need to add it to a logical volume group and create a logical volume with it.

We can now add one of the disks to be seen by LVM as a physical volume. Use the pvcreate command.

pvcreate /dev/md0
Physical volume "/dev/md0" successfully created

Create the volume group using the vgcreate command. I will name mine raid-pool.

vgcreate raid-pool /dev/md0
Volume group "raid-pool" successfully created

Use the vgdisplay command to see how many physical extents are available to use.

vgdisplay

--- Volume group ---
VG Name raid-pool
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 3
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 0
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 4.00 GB
PE Size 4.00 MB
Total PE 1023
Alloc PE / Size 0 / 0
Free PE / Size 1023 / 4.00 GB
VG UUID 96bd50-9yn2-bzpc-C4kV-3W1L-rkbN-bO3o3e

Create a logical volume using the lvcreate command.

lvcreate -l 1023 raid-pool -n lv1
Logical volume "lv1" created

The above command created a logical volume using all 4GB with a default name lv1.

Format the logical volume with your desired filesystem. Here I will use ext3.

mkfs.ext3 /dev/raid-pool/lv1

mke2fs 1.40-WIP (14-Nov-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
524288 inodes, 1047552 blocks
52377 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1073741824
32 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736

Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 29 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.

Create a mount point for the volume.

mkdir /mnt/storage

Mount the new volume.

mount /dev/raid-pool/lv1 /mnt/storage

Check to see that the new storage is available using the df -h command.

df -h

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/raid--pool-lv1 4.0G 73M 3.7G 2% /mnt/storage

Now lets add another disk to the RAID.

First we should unmount or lv to be safe.

umount /mnt/storage

Use the –grow option to expand the RAID and then add another hot spare.

mdadm --grow /dev/md0 --raid-devices=4

mdadm: Need to backup 384K of critical section..
mdadm: ... critical section passed.

Again, check the status of the RAID to make sure everything is in order.

mdadm --detail /dev/md0
/dev/md0:
Version : 00.91.03
Creation Time : Thu Sep 13 23:12:59 2007
Raid Level : raid5
Array Size : 4192768 (4.00 GiB 4.29 GB)
Device Size : 2096384 (2047.59 MiB 2146.70 MB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Thu Sep 13 23:40:24 2007
State : clean, recovering
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 64K

Reshape Status : 43% complete
Delta Devices : 1, (3->4)

UUID : f5c24ee9:27cb0d42:e368bf24:bd0fce41 (local to host ubuntu)
Events : 0.648

Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
2 8 49 2 active sync /dev/sdd1
3 8 65 3 active sync /dev/sde1

You can also use the cat /proc/mdstat command.

cat /proc/mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sde1[3] sdd1[2] sdc1[1] sdb1[0]
4192768 blocks super 0.91 level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
[============>........] reshape = 60.0% (1258880/2096384) finish=1.2min speed=10877K/sec

unused devices: <none>

Once the rebuild is complete we should add the new hot spare disk also.

mdadm --add /dev/md0 /dev/sdf1

mdadm: added /dev/sdf1

Check to make sure it is really there.

mdadm --detail /dev/md0

/dev/md0:
Version : 00.90.03
Creation Time : Thu Sep 13 23:12:59 2007
Raid Level : raid5
Array Size : 6289152 (6.00 GiB 6.44 GB)
Device Size : 2096384 (2047.59 MiB 2146.70 MB)
Raid Devices : 4
Total Devices : 5
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Fri Sep 14 00:01:20 2007
State : clean
Active Devices : 4
Working Devices : 5
Failed Devices : 0
Spare Devices : 1

Layout : left-symmetric
Chunk Size : 64K

UUID : f5c24ee9:27cb0d42:e368bf24:bd0fce41 (local to host ubuntu)
Events : 0.1434

Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
2 8 49 2 active sync /dev/sdd1
3 8 65 3 active sync /dev/sde1

4 8 81 - spare /dev/sdf1

Now we can expand the logical volume to use the extra storage space. Use the pvresize command.

pvresize /dev/md0

Physical volume "/dev/md0" changed
1 physical volume(s) resized / 0 physical volume(s) not resized

You can now see the unused space with the vgdisplay comand.

vgdisplay

--- Volume group ---
VG Name raid-pool
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 5
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 6.00 GB
PE Size 4.00 MB
Total PE 1535
Alloc PE / Size 1023 / 4.00 GB
Free PE / Size 512 / 2.00 GB
VG UUID 96bd50-9yn2-bzpc-C4kV-3W1L-rkbN-bO3o3e

Use the lvresize command to add the new physical extents to the volume

lvresize -l+512 /dev/raid-pool/lv1

Extending logical volume lv1 to 6.00 GB
Logical volume lv1 successfully resized

Check the volume with e2fsck before growing the ext3 filesystem.

e2fsck -f /dev/raid-pool/lv1

e2fsck 1.40-WIP (14-Nov-2006)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/raid-pool/lv1: 11/524288 files (9.1% non-contiguous), 34912/1047552 blocks

Grow the ext3 filesystem to use the new space.

resize2fs /dev/raid-pool/lv1

resize2fs 1.40-WIP (14-Nov-2006)
Resizing the filesystem on /dev/raid-pool/lv1 to 1571840 (4k) blocks.
The filesystem on /dev/raid-pool/lv1 is now 1571840 blocks long.

Remount the volume.

mount /dev/raid-pool/lvol0 /mnt/storage/

Check the size of the volume.

df -h

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/raid--pool-lv1 6.0G 73M 5.6G 2% /mnt/storage

As you can see we went from having 4GB of storage to 6GB.