Internal Links
Main
Mac
Linux
Previous Work
Blog
Contact
External Links
The 2.6.x Linux kernel supports RAID levels 0,1,5, and 6. You can read more about them here. With todays computers software RAID can be just as fast as a dedicated hardware RAID controller and much more flexible. I use an old PowerBookG4 running Ubuntu Linux as my file server. It has an external Firewire case housing six 250GB ATA drives. I have setup all the drives in a RAID6 config. This gives me just under 1TB of usable space with two drives used for parity. That means I can have two drives fail at once before I lose any data. To me this is the best option in terms of disk usage and redundancy. I wish Apple would implement software RAID 5 and 6 into their OS! SATA drives and external enclosures make RAID6 a cheap and very reliable storage solution.
In this example I use CentOS5 running under VMware on my MacBook pro and created three 2GB virtual hard drives. I will be creating a RAID 5 using these drives and showing how to recover from a lost drive.
Note: many of these commands require being run as root.
Once you have your drives connected to your linux box check that Linux can see them using the “fdisk -l” command.
[root@localhost ~]# fdisk -l Disk /dev/hda: 15.7 GB, 15729057792 bytes 255 heads, 63 sectors/track, 1912 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/hda1 * 1 1784 14329948+ 83 Linux /dev/hda2 1785 1911 1020127+ 82 Linux swap Disk /dev/hdb: 2097 MB, 2097414144 bytes 255 heads, 63 sectors/track, 254 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk /dev/hdb doesn't contain a valid partition table Disk /dev/hdc: 2097 MB, 2097414144 bytes 255 heads, 63 sectors/track, 254 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk /dev/hdc doesn't contain a valid partition table Disk /dev/hdd: 2097 MB, 2097414144 bytes 255 heads, 63 sectors/track, 254 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk /dev/hdd doesn't contain a valid partition table [root@localhost ~]#
My three drives show up as /dev/hdb, /dev/hdc, and /dev/hdd. Parallel ATA drives show up as hdx, Serial ATA drives, USB , Firewire, and SCSI show up as /dev/sdx.
Next we want to partition each drive and give it a system ID of Linux RAID auto . Setting the system ID to linux RAID allows you to connect the drives to another Linux system and be auto detected.
fdisk /dev/hdb
Hit m on the keyboard to see all the options.
[root@localhost ~]# fdisk /dev/hdb Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable. Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite) Command (m for help): m Command action a toggle a bootable flag b edit bsd disklabel c toggle the dos compatibility flag d delete a partition l list known partition types m print this menu n add a new partition o create a new empty DOS partition table p print the partition table q quit without saving changes s create a new empty Sun disklabel t change a partition's system id u change display/entry units v verify the partition table w write table to disk and exit x extra functionality (experts only)
We need to create a partition.
Hit the n key then hit return. Hit the p key to choose primary partition then hit return. hit the 1 key then hit return. Hit the return key two more times to select the default of using the whole drive. Hit the t key and then hit return to set the System ID. You can hit the L key to show all the hex code choices. The hex code choice for linux raid auto is fd. Type fd and hit return. It should say “Changed system type of partition 1 to fd (Linux raid autodetect)”. hit the w key and then hit return to write the changes to disk. Repeat section B for the other drives you will be using in the RAID.
After you are done partitioning the disks they will be represented as a 1 next to the device. Do another fdisk -l to see the new partitions.
Disk /dev/hdb: 2097 MB, 2097414144 bytes 255 heads, 63 sectors/track, 254 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/hdb1 1 254 2040223+ fd Linux raid autodetect Disk /dev/hdc: 2097 MB, 2097414144 bytes 255 heads, 63 sectors/track, 254 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/hdc1 1 254 2040223+ fd Linux raid autodetect Disk /dev/hdd: 2097 MB, 2097414144 bytes 255 heads, 63 sectors/track, 254 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/hdd1 1 254 2040223+ fd Linux raid autodetect
Create the RAID using a command similar to the following.
mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/hdb1 /dev/hdc1 /dev/hdd1
The output will look something like this.
mdadm: layout defaults to left-symmetric mdadm: chunk size defaults to 64K mdadm: size set to 2040128K mdadm: array /dev/md0 started.
You can check the RAID using the following command.
mdadm --detail /dev/md0
The output will look something like this.
/dev/md0: Version : 00.90.01 Creation Time : Sat Mar 10 22:06:46 2007 Raid Level : raid5 Array Size : 4080256 (3.89 GiB 4.18 GB) Device Size : 2040128 (1992.31 MiB 2089.09 MB) Raid Devices : 3 Total Devices : 3 Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Sat Mar 10 22:07:06 2007 State : clean Active Devices : 3 Working Devices : 3 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 64K Number Major Minor RaidDevice State 0 3 65 0 active sync /dev/hdb1 1 22 1 1 active sync /dev/hdc1 2 22 65 2 active sync /dev/hdd1 UUID : 2a0ff162:c5e5de0a:4c65973d:28c01e83 Events : 0.2
This command will show you all the RAID's you have setup.
cat /proc/mdstat
Personalities : [raid5] md0 : active raid5 hdd1[2] hdc1[1] hdb1[0] 4080256 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU] unused devices: <none>
Format the RAID using the following command.
mkfs.ext3 /dev/md0
Create a mount point for the RAID.
mkdir /mnt/raid
If you want your users to be able to write to the RAID set the permissions.
chmod 777 /mnt/raid
Mount the RAID using the following command.
mount /dev/md0 /mnt/raid
Check to see that everything is setup and working using the following command.
df -h
You will see something like this.
Filesystem Size Used Avail Use% Mounted on /dev/hda1 14G 7.2G 5.7G 56% / none 252M 0 252M 0% /dev/shm /dev/md0 3.9G 40M 3.6G 2% /mnt/raid
You can have the raid mount automatically at boot by adding a line to your /etc/fstab file.
/dev/md0 /mnt/raid auto defaults 0 0
Note: When using RAID 0,1,5, and 6 all partitions must be the same size. The physical disks can be the same size or larger.
I copied the first install ISO of the CentOS4 distribution onto my RAID. Using the “df -h” command you can see that I have 663MB used on my raid.
[root@localhost raid]# df -h Filesystem Size Used Avail Use% Mounted on /dev/hda1 14G 7.2G 5.7G 56% / none 252M 0 252M 0% /dev/shm /dev/md0 3.9G 663M 3.0G 18% /mnt/raid
Here is an “ls -alh” of /mnt/raid.
[root@localhost raid]# ls -alh /mnt/raid/ total 624M drwxr-xr-x 3 root root 4.0K Mar 10 22:46 . drwxr-xr-x 5 root root 4.0K Mar 10 22:21 .. -rw-r--r-- 1 root root 623M Mar 10 22:48 CentOS-4.4-i386-bin1of4.iso drwx------ 2 root root 16K Mar 10 22:20 lost+found [root@localhost raid]#
Here is what the “mdadm –detail /dev/md0” command showed before I removed /dev/hdc1 from the computer.
[root@localhost raid]# mdadm --detail /dev/md0 /dev/md0: Version : 00.90.01 Creation Time : Sat Mar 10 22:06:46 2007 Raid Level : raid5 Array Size : 4080256 (3.89 GiB 4.18 GB) Device Size : 2040128 (1992.31 MiB 2089.09 MB) Raid Devices : 3 Total Devices : 3 Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Sat Mar 10 22:55:18 2007 State : clean Active Devices : 3 Working Devices : 3 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 64K Number Major Minor RaidDevice State 0 3 65 0 active sync /dev/hdb1 1 22 1 1 active sync /dev/hdc1 2 22 65 2 active sync /dev/hdd1 UUID : 2a0ff162:c5e5de0a:4c65973d:28c01e83 Events : 0.70
Now here is what you will see with a missing disk.
[root@localhost ~]# mdadm --detail /dev/md0 /dev/md0: Version : 00.90.01 Creation Time : Sat Mar 10 22:06:46 2007 Raid Level : raid5 Array Size : 4080256 (3.89 GiB 4.18 GB) Device Size : 2040128 (1992.31 MiB 2089.09 MB) Raid Devices : 3 Total Devices : 2 Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Sat Mar 10 23:01:02 2007 State : clean, degraded Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 64K
Number Major Minor RaidDevice State 0 3 65 0 active sync /dev/hdb1 1 0 0 -1 removed 2 22 65 2 active sync /dev/hdd1 UUID : 2a0ff162:c5e5de0a:4c65973d:28c01e83 Events : 0.80
Notice that the line for State: says its degraded. The data is still there and accessible.
[root@localhost ~]# ls -alh /mnt/raid/ total 624M drwxr-xr-x 3 root root 4.0K Mar 10 22:46 . drwxr-xr-x 5 root root 4.0K Mar 10 22:21 .. -rw-r--r-- 1 root root 623M Mar 10 22:48 CentOS-4.4-i386-bin1of4.iso drwx------ 2 root root 16K Mar 10 22:20 lost+found
Now lets add another drive to the linux box, partition it, and add it to the raid.
Use the same steps in section B to partition the new drive. Here is what my fdisk showed before I partitioned the replacement drive.
[root@localhost ~]# fdisk -l Disk /dev/hda: 15.7 GB, 15729057792 bytes 255 heads, 63 sectors/track, 1912 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/hda1 * 1 1784 14329948+ 83 Linux /dev/hda2 1785 1911 1020127+ 82 Linux swap Disk /dev/hdb: 2097 MB, 2097414144 bytes 255 heads, 63 sectors/track, 254 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/hdb1 1 254 2040223+ fd Linux raid autodetect Disk /dev/hdc: 2097 MB, 2097414144 bytes 255 heads, 63 sectors/track, 254 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk /dev/hdc doesn't contain a valid partition table Disk /dev/hdd: 2097 MB, 2097414144 bytes 255 heads, 63 sectors/track, 254 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/hdd1 1 254 2040223+ fd Linux raid autodetect Disk /dev/md0: 4178 MB, 4178182144 bytes 2 heads, 4 sectors/track, 1020064 cylinders Units = cylinders of 8 * 512 = 4096 bytes Disk /dev/md0 doesn't contain a valid partition table
Notice that /dev/hdc has no valid partitions.
Use the following command to add the new partition to the raid.
mdadm /dev/md0 -a /dev/hdc1 --verbose
You will get something like this.
mdadm: hot added /dev/hdc1
Use the following command to see the progress of the rebuild.
mdadm --detail /dev/md0
Note the State: line and the Rebuild Status : line.
/dev/md0: Version : 00.90.01 Creation Time : Sat Mar 10 22:06:46 2007 Raid Level : raid5 Array Size : 4080256 (3.89 GiB 4.18 GB) Device Size : 2040128 (1992.31 MiB 2089.09 MB) Raid Devices : 3 Total Devices : 3 Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Sat Mar 10 23:17:15 2007 State : clean, degraded, recovering Active Devices : 2 Working Devices : 3 Failed Devices : 0 Spare Devices : 1 Layout : left-symmetric Chunk Size : 64K Rebuild Status : 13% complete Number Major Minor RaidDevice State 0 3 65 0 active sync /dev/hdb1 1 0 0 -1 removed 2 22 65 2 active sync /dev/hdd1 3 22 1 1 spare /dev/hdc1 UUID : 2a0ff162:c5e5de0a:4c65973d:28c01e83 Events : 0.91
Thats all there is to it.