Although kernels prior to 2.0.35 shipped with some RAID support, it wasn’t until the release of 2.0.35 that support for RAID-1, RAID-4, and RAID-5 became available.
The multiple devices(md) driver that provides kernel RAID support is currently at version 0.90. The latest stable kernels in the 2.0 and 2.2 series ship with an outdated version (0.36) of the software RAID driver. Patches for the 0.90 code are available for these older kernels (see the sections below on “Kernel 2.2” and “Kernel 2.0”). Ker-nels prior to 2.0.35 ship with version 0.35 of themddriver and should not be used, except if you absolutely have to keep them in operation for legacy reasons.
The 0.90 code was finally merged with the kernel source tree during the 2.3 develop-ment phase, so 2.4 ships with the latest RAID code. So will all subsequent stable and development kernels. In general, it’s best to run at least a 2.4 series kernel when using software RAID, or a 2.2 kernel with the proper RAID patches applied. Using the old 0.36 driver is not recommended.
Along with drivers to provide software RAID for Linux, a set of utilities for manag-ing, configurmanag-ing, and tuning software arrays has emerged. These utilities, collectively called theRAID Tools(raidtoolsis the package name), are maintained by Ingo Mol-nar and Erik Troan of Red Hat and are now included with many major distribu-tions. Molnar also served as the primary developer and maintainer of the software RAID subsystem from version 0.36 through 0.90’s integration into the stable kernel tree. Today, he shares that responsibility with Neil Brown, a software engineer at the University of New South Wales. Brown also contributed many performance and reli-ability-related patches before becoming an official maintainer. In August 2001, Brown began work on his own software RAID utilities for Linux. Hismultiple devices administration(mdadm) program is a great alternative to the slightly agingraidtools.
Both packages are covered in this book. I recommend usingmdadm when possible.
Many other programmers have contributed to themddriver since its original devel-opment. Marc Zyngier wrote the original implementations of linear mode and RAID-0, along with the predecessor toraidtools. Miguel de Icaza helped develop the first implementation of RAID-1 and wrote the first version of raidtools. Gadi Oxman wrote the first 0.90 implementation of RAID-5.
Kernel 2.4
As of version 2.4, software RAID has its own subsection in the kernel configuration (in previous versions, options were found under the Block Devices submenu). The Multi-device Support (RAID and LVM) submenu contains all the configuration options for software RAID. The first option,CONFIG_MD, is a configuration switch that reveals options for RAID and Logical Volume Management (LVM). After selecting CONFIG_MD, choose CONFIG_BLK_DEV_MD, which allows the kernel to address multiple physical devices through a single logical device (see Figure 3-1). CONFIG_BLK_DEV_MD provides the foundation for themddriver and needs to be enabled on every system that uses software RAID, regardless of which RAID level is implemented.
In order to successfully build a software RAID device, one of the available RAID lev-els must also be enabled. The following kernel directives enable support for specific RAID levels:
CONFIG_MD_LINEAR CONFIG_MD_RAID0 CONFIG_MD_RAID1 CONFIG_MD_RAID5
CONFIG_MD_LINEARandCONFIG_MD_RAID0enable support for the non-RAID methods of disk grouping supported by the Linux kernel.CONFIG_MD_RAID1provides support for Figure 3-1. Enabling kernel RAID support.
data redundancy through disk mirroring. Finally, CONFIG_MD_RAID5 enables support for both RAID-4 and RAID-5 layouts. The choice between using RAID-4 or RAID-5 is made later, when creating an array. If you plan to experiment with more than one of the available RAID levels, or if you are still unsure which RAID level you plan to implement, you can enable all four levels, rather than compiling a separate kernel for each (Figure 3-2) or recompiling your kernel each time you wish to experiment with another level. This is extremely useful during testing. When you’re ready to go into production, simply rebuild the kernel with only the RAID levels you need. RAID support is also available as loadable kernel modules.
You will also notice an option for Multipath I/O support (CONFIG_MD_MULTIPATH).
This newer feature of themddriver allows the addressing of a single physical disk using multiple I/O paths (controllers or channels). So if one I/O path becomes unavailable, the disk is still accessible. Multipath I/O support is not covered in this book.
Once you have added the desired options to your kernel configuration, compile the new kernel with support for software RAID.
# make dep
# make bzImage
The new kernel should be located at .../arch/i386/boot/bzImage(for i386-based sys-tems), under the kernel source directory. If you are using a non-Intel machine, look for the compiled kernel image in the directory that is appropriate for your architec-ture (.../arch/sparc/boot/ on SPARC systems, for example).
Figure 3-2. Enable support for specific RAID levels.
Installing the new kernel
Linux kernels are traditionally stored in the/bootdirectory (usually a dedicated parti-tion) or occasionally in the rootdirectory. Copy the new kernel to the /boot direc-tory, or to the directory where you normally keep your kernels, if you have a nonstandard situation. The following example works on i386-based systems.
# cp arch/i386/boot/bzImage /boot/bzImage.raid
# cp System.map /boot/
You can name the kernel anything you like, but Ifind it helpful to name it some-thing descriptive. Others prefer to include the version number in the filename. Next, add an entry for the RAID-enabled kernel to the boot loader configuration.
On systems using the Linux Loader (LILO),/etc/lilo.confcontains information about which kernel image is loaded during the system initialization. LILO users might add the following entry to/etc/lilo.conf:
image=/boot/bzImage.raid label=linux-raid read-only root=/dev/sda1
Make sure that you enter the correctrootdevice, if it is not/dev/sda1. Either place this stanza before all the others listed in/etc/lilo.conf, or use thedefaultglobal key-word to specify which kernel to boot (see the lilo.conf manual page for further details). You can also simply select the Linux-raid kernel from the LILO prompt dur-ing startup and avoid changdur-ing the default.
# lilo
Added linux-raid * Added linux-orig
# shutdown -r now
You can also use lilo -Rto specify which kernel to load when the system restarts (after you runliloto rebuild the boot sector). In the following example, the Linux-raid kernel will be loaded the next time the system restarts, even though it is not the default kernel listed in/etc/lilo.conf.
# lilo
Added linux-raid Added linux-orig *
# lilo -R linux-raid
# shutdown -r now
lilo -Rworks for only one subsequent reboot. So, once you are satisfied with the new kernel, remember to change the default in/etc/lilo.conf or rearrange the stanzas.
Some distributions are beginning to use the Grand Unified Bootloader (GRUB) instead of LILO. GRUB users can add lines like the following to their /boot/grub/
grub.conf file:
title Linux RAID (2.4.18) root (hd0,0)
kernel /bz.2418 ro root=/dev/sda2
This entry will load the kernel found at/boot/bz.2418on the first hard disk (hd0,0) using a rootdevice of /dev/sda2, mounted initially as read-only. By default, GRUB will boot the first entry it finds ingrub.conf, so you could either place the lines above as the first entry or change thedefault parameter to reflect the correct kernel.
System initialization
When the system restarts, a message indicates that themddriver has initialized. Each RAID level compiled into the kernel is listed, along with information about the driver. On a system with support for all available RAID levels, the following message should appear:
md driver 0.90.0 MAX_MD_DEVS=256, MD_SB_DISKS=27 linear personality registered
raid0 personality registered raid1 personality registered raid5 personality registered
If your system boots faster than you can read the messages, use thedmesgcommand to examine boot messages after the system has initialized.dmesgdisplays the kernel ring buffer, allowing users to examine important kernel messages that they might have missed during kernel initialization. The kernel ring buffer stores, in memory, messages about the current state of the running kernel. Some distributions are already preconfigured to dump these messages to a file after the system boots, but you can simply redirect the output to a file (dmesg > boot.messages) or pipe it into a pager (dmesg | less) at your whim. Messages returned bydmesgmight look a little bit different from their appearance while the system is booting, but the same infor-mation is there:
# dmesg | grep md
md: linear personality registered as nr 1 md: raid0 personality registered as nr 2 md: raid1 personality registered as nr 3 md: raid5 personality registered as nr 4 md: multipath personality registered as nr 7 md: md driver 0.90.0 MAX_MD_DEVS=256, MD_SB_DISKS=27
When themddriver initializes, its version is displayed (0.90.0, in this case).MAX_MD_
DEVSindicates that the number of RAID devices is limited to 256 (with devices files named /dev/md[0-255]), and MD_SB_DISKS indicates that each software array is lim-ited to 27 member disks. However, by building hybrid arrays, users can move well beyond this limit. Hardware limitations will become an issue long before you reach the limits of themd driver.
The kernel also registers each RAID level, or personality. In this case, the kernel con-tains support for linear mode, RAID-0, RAID-1, and RAID-4/5. If RAID support was compiled as loadable kernel modules, these initialization messages will not appear, but you should see them later in your system logs when those modules are inserted.
These messages will be reported (like allmddriver messages), using thekernfacility, starting at theinfo level.
Please note that as the system remains running, the kernel ring buffer will begin to expunge old messages. So be certain to examine it soon after booting if you need to view information generated by the boot process. Most distributions are also precon-figured to place this information in /var/log/messagesafter the system boots. Please be aware that some distributions also come configured with an/etc/syslogd.confthat generates a misnamed file called/var/log/boot.log, using the log facilitylocal7. This file (and any files generated usinglocal7) contains messages from system initializa-tion scripts (rc), not kernel initializainitializa-tion messages. Kernel messages are captured by klogd and dumped to syslogd once klogd has executed. In most default configura-tions, you will find messages about the md driver in /var/log/messages, including those dumped from the kernel ring buffer byklogd. Chapter 7 covers system logging and themd driver in further detail.
If RAID-5 is supported, the kernel will also optimize the XOR routines used for par-ity checksum in a RAID-5. A message indicating that the test was performed is dis-played, along with its results:
raid5: measuring checksumming speed 8regs : 1835.600 MB/sec 32regs : 871.600 MB/sec pIII_sse : 2021.200 MB/sec pII_mmx : 2277.200 MB/sec p5_mmx : 2386.000 MB/sec
raid5: using function: pIII_sse (2021.200 MB/sec)
Since different i386-based processor architectures implement different ways to per-form the necessary XOR operations, Linux needs to determine which one is the most efficient for a particular system in advance. Notice that in this example, the fastest checksum operation was not chosen. On systems that support Streaming SIMD Extensions (SSE), that choice is selected because of its ability to circumvent the L2 cache and perform operations in parallel (see /usr/src/linux/include/asm-i386/xor.h).
This functionality improves the performance of checksum operations. Information about checksum algorithms is also recorded in the kernel ring buffer.
After the system restarts, examining/proc/mdstatwill also show that the RAID sub-system was successfully initialized with support for linear mode, RAID-0, RAID-1, and RAID-4/5.
# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid5]
read_ahead not set unused devices: <none>
See the section “Examining Arrays Using /proc/mdstat,” later in this chapter, for more information.