Theraidtoolspackage, also maintained by Ingo Molnar, provides a set of utilities for creating and managing software arrays. raidtools has been the standard software RAID management package for Linux since the inception of the software RAID driver. Over the years, raidtools has proven cumbersome to use, mostly because it relies on a configuration file (/etc/raidtab) that is difficult to maintain, and partly because its features are limited. In August 2001, Neil Brown released an alternative.
His mdadmpackage provides a simple, yet robust way to manage software arrays.
mdadmis now at version 1.0.1 and has proven quite stable after its first year of devel-opment. It has received a positive response on the Linux-raidmailing list and will likely become widespread in the future. The rest of this chapter will provide you with examples of building and managing arrays, using the raidtools package, as well as mdadm.
raidtools
raidtoolsprovides a small, simple command set that allows administrators to create, activate, and stop software arrays. Each array managed by the system is defined in a global configuration file (/etc/raidtab) that describes which physical disks are mem-bers of each array./etc/raidtabalso contains metadata about every array, such as its RAID level and failover capabilities. The commands included with the raidtools package use the information in/etc/raidtab to interface with arrays.
There are two notable versions ofraidtools. Version 0.90 is the most recent imple-mentation, and the only one being maintained. It should be used only in conjunc-tion with the 0.90 driver. An older version (0.42) ofraidtools is also still available.
This version should only be used when working with older systems that still use the 0.36 software RAID driver. Never useraidtools with the wrong driver!
Also included withraidtoolsare sample configuration files and documentation (man-ual pages and cookbooks). Unfortunately, much of the documentation is seriously outdated, sometimes by a factor of years. Adhering to the open source philosophy that “old documentation is better than no documentation,” a lot of outdated infor-mation is included with theraidtools package. It’s probably best to double-check a Linux Documentation Project mirror for the most recent information and consult additional reference material and mailing lists in lieu of relying on the documenta-tion in the package.
As of this writing, raidtools-1.0 has been released in a limited capacity. It can be found in Red Hat 8.0 and will likely become pervasive during the next few months. It fixes several minor bugs and can be used as a replacement for raidtools-0.90.
Chapter 4 presents more information about raidtools-1.0 and some new utilities included with it.
raidtools-0.90
Most modern Linux distributions ship with the raidtools package. You can install them from your installation media or download them from your vendor. In a Red Hat CD directory, for example, issue the following command:
# rpm -ihv ./raidtools-0.90-20.i386.rpm
Debian users can useapt-get to install theraidtools:
# apt-get install raidtools2
Note that under Debian,raidtools2, NOTraidtools, provides the 0.90raidtools pack-age. Please be certain you have the proper package installed.
You can also download the source code and compile the package yourself. The most recent version is available at ftp://ftp.kernel.org/pub/daemons/raid/alpha. As of this writing, the most recent version is ftp://ftp.kernel.org/pub/linux/daemons/raid/alpha/
raidtools-19990824-0.90.tar.gz. You might be aware that a seemingly newer version is available from http://people.redhat.com/mingo/raidtools/. There are several known problems with this version (raidtools-20010914), including its lack of a key manage-ment utility (raidsetfaulty). Use this version at your own risk. Theraidtoolspackage should compile without problems on any recent system, though it does require the poptlibraries. There are currently no notable configuration options. Download and unpack the archive.
# cd /usr/src/raidtools-YYYYMMDD
# ./configure
# make install
Compilation will leave you with several executables. Of particular note aremkraid, used to create new arrays, and raidstart, used to activate arrays that have already been initialized withmkraid. Installation will copy binaries to/sbinand create sym-bolic links that invoke specific functions for each utility. raidstop, for example, is simply a symbolic link that causes raidstart to invoke a case switch based on the command line. Manual pages are also installed. Chapter 4 covers each command included with theraidtools package in detail.
At the time of this writing, version 1.0 of raidtools has just appeared in Red Hat Linux 8.0. Version 1.0 of raidtools corrects the issues associated with raidtools-20010914and includes some major improvements, such as bug fixes and a couple of new utilities.lsraidwill allow you to query arrays and member disks. It will even let you generate an/etc/raidtabfile by querying a running array.raidreconfallows users
to add (or remove) disks to a RAID-0 or RAID-5, usually without data loss. Although these utilities are somewhat new, I’ll discuss them throughout this book, since they are particularly useful and will more than likely become widespread.
/etc/raidtab
/etc/raidtab is raidtools’ primary configuration file. /etc/raidtab provides a function similar to/etc/fstab. It provides the kernel with a description of each software RAID, including its RAID level and a list of member disks. All RAID devices must have an entry in/etc/raidtab to be created or activated usingraidtools.
Users create/etc/raidtabby hand, using any text editor. Once/etc/raidtabis created and contains configuration information about arrays, themkraid command can be used to construct an array based on the parameters defined in /etc/raidtab.mkraid will complain if it cannot find a valid file at/etc/raidtab. This process is similar to cre-ating the boot configuration file/etc/lilo.confand consequently running/sbin/lilo to rebuild the boot block.
Here is a typical /etc/raidtab that describes a RAID-0 with two ATA component disks:
raiddev /dev/md0 raid-level 0 persistent-superblock 1 chunk-size 64 nr-raid-disks 2 nr-spare-disks 0 device /dev/hda1 raid-disk 0 device /dev/hdb1 raid-disk 1
Each array begins with araiddeventry, followed by a list of array properties and an entry for each array member. A valid entry in /etc/raidtab must contain at least a definedraid-level, achunk-size, the total number of raid disks (nr-raid-disks) and two device/raid-disk pairs. chunk-size is a bit counterintuitive because it applies only to arrays that support disk striping (RAID-0, RAID-4, and RAID-5), yet it is required for any array you define in/etc/raidtab. That’s because the utilities in the raidtoolspackage (mkraid, most notably) check for achunk-sizeeven if the defined RAID level doesn’t support one. So when you work with linear mode or RAID-1, for example, you must provide achunk-size, even though it’s technically only a place-holder.
There are no global options in/etc/raidtab. Each section, or stanza, applies to the last parsedraiddeventry. Once an array is defined, it may be included as a member disk
in a subsequent array section. This allows you to combine multiple arrays into a hybrid array (as in RAID-10, for example).
Chapter 4 provides details about each/etc/raidtab parameter.
raidtools-0.42
It’s unlikely that you will be able to locate a package file for this outdated version of raidtools. So, in most cases, you will be stuck compiling the tools yourself. Down-load version 0.42 of the raidtools (ftp://ftp.kernel.org/pub/linux/daemons/raid/
raidtools-pre3-0.42.tar.gz) and unpack it.
One important caveat when working with this version is that it requires a patched kernel to compile. By default, theconfigurecommand assumes that your kernel is in /usr/src/linux, but you can change that by using the--with-linux parameter.
# cd /usr/src/raidtools-0.42
# ./configure --with-linux=/usr/src/linux-2.0.37
# make
# make install
Installation will leave you with an older version ofmkraid, as well as several depre-cated utilities that are used to start (mdrun), stop (mdstop), and manage (mdcreate, ckraid) arrays. Manual pages are also installed by default.
Debian users can install version 0.42 ofraidtools usingapt-get:
# apt-get install raidtools
The rest of this chapter covers the current version of raidtools and the md driver.
However, if you are working with legacy arrays, then you will have to use the 0.42 version of raidtools or mdadm to manage them. Istrongly recommend that you upgrade old arrays whenever possible.
mdadm
Over the years, many users have become quite frustrated by the problems involved in using raidtools and /etc/raidtab to manage software RAID. On small systems, the problems presented by raidtools are tolerable. But on large systems with multiple arrays, each with the potential for dozens of member disks, managing /etc/raidtab can become daunting.mdadmprovides a slightly different approach fromraidtools.
The idea behindmdadmis that the kernel, as well as administrators, should be able to manage arrays without resorting to a complicated, structured configuration file.
mdadm uses a Universally Unique ID (UUID) to identify each array and member disk. A UUID is a 128-bit number that is guaranteed to be reasonably unique on both the local system and across other systems. It is randomly generated, using sys-tem hardware and timestamps as part of its seed.mdadmuses the UUIDs found in
the array superblock to scan member disks, determining which array they belong to and what the array properties are. Many programs use UUIDs to tag devices uniqely.
In fact, kernel RAID autodetection uses these UUIDs, too. See theuuidgenand lib-uuid manual pages for more information.
mdadmprovides all the same functions thatraidtools provides, in addition to some new features. The only disadvantage ofmdadmis that it’s new. Neil Brown began working on mdadm in June of 2001, and it has undergone only a few revisions.
mdadmcan also manage legacy arrays created under the 0.36mddriver, which are normally managed with raidtools-0.42. In this book. I’ll describe how to manage arrays using both raidtools and mdadm. The final decision is yours to make, but I urge you to at least trymdadm.
You can download the most recentmdadmtarball fromhttp://www.cse.unsw.edu.au/
~neilb/source/mdadm/ and issue make install to compile and install mdadmand its documentation. In addition to the binary, some manual pages and example files are included.
# tar xvf ./mdadm-1.0.0.tgz
# cd mdadm-1.0.0.tgz
# make install
Or, you can download and install the package file found in therpmdirectory under the above URL.
# rpm -ihv mdadm-1.0.0.i386.rpm
mdadmhas five major modes of operation. The first three modes—Create, Assem-ble, and Build—are used to configure and activate arrays. Manage mode is used to manipulate devices in an active array. Follow or Monitor mode allows administra-tors to configure event notification and actions for arrays. The remaining options are used for various housekeeping tasks and are not attached to a specific mode of oper-ation, although themdadmdocumentation calls these options Misc mode. The fol-lowing list describes the major modes of operation inmdadm.
Create mode
Used to create a new array. With Create mode, you have the option to define a new array at the command line or create an array defined in/etc/mdadm.conf.
Assemble mode
Used to start an array that already exists.
Build mode
Used only for creating or starting legacy arrays (kernel version 0.36). It should never be used with the 0.90md driver.
Manage mode
Used to add and remove disks to a running array. This mode is useful for remov-ing failed disks and addremov-ing spare or replacement disks. Manage mode can also be used to mark a member disk as failed. Manage mode replicates the functions ofraidtools programs such asraidsetfaulty,raidhotremove, andraidhotadd.
Follow (or Monitor) mode
Used to implement some of mdadm’s best and unique features. With Follow/
Monitor mode, you can daemonizemdadmand configure it to send email alerts to system administrators when arrays encounter errors or fail. You can also use Follow mode to arbitrarily execute commands when a disk fails. For example, you might want to try removing and reinserting a failed disk in an attempt to correct a nonfatal failure without user intervention.
Follow/Monitor mode also allows arrays to share spare disks, a feature that has been lacking in Linux software RAID since its inception. That means you need to provide only one spare disk for a group of arrays or for all arrays. It also means that system administrators don’t have to manually intervene to shuffle around spare disks when arrays fail. Previously, this functionality was available only with hardware RAID. When Follow/Monitor mode is invoked, it polls arrays at regular intervals. When a disk failure is detected on an array without a spare disk,mdadmwill remove an available spare disk from another array and insert it into the array with the failed disk.
The remaining options, which fall under the Misc mode, are used for tasks that include stopping arrays, marking arrays as read-only or read/write, and clearing the RAID superblock from a disk.
/etc/mdadm.conf
/etc/mdadm.conf is the primary configuration file in mdadm. Unlike /etc/raidtab, mdadmdoes not rely on/etc/mdadm.confto create or manage arrays. Rather,mdadm.
conf is simply an extra way of keeping track of software RAIDs. Using a configura-tion file with mdadmis useful, but not required. Having a configuration file means that you can quickly manage arrays without spending extra time figuring out what the array properties are and where disks belong. Unlike the configuration file for raidtools,mdadm.confis concise and simply lists disks and arrays. The configuration file usually contains two types of lines, each starting with either the DEVICEorARRAY keyword. Whitespace separates the keyword from the configuration information.
DEVICElines specify a list of devices that are potential member disks.ARRAYlines spec-ify device entries for arrays, as well as identifier information. This information can include lists of one or more UUIDs,md device minor numbers, or member devices.
A simplemdadm.conf file might look like this:
DEVICE /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 ARRAY /dev/md0 devices=/dev/sda1,/dev/sdb1 ARRAY /dev/md1 devices=/dev/sdc1,/dev/sdd1
In general, it’s best to set up an/etc/mdadm.conffile after you have created an array.
You should update the file when new arrays are created. Without an mdadm.conf file, you’d need to specify more detailed information about an array on the com-mand line in order to activate it. That means you’d have to remember which devices belonged to which arrays. That could be a hassle on systems with a lot of disks.
mdadmeven provides an easy way to generateARRAYlines. The output is a single long line, but I have broken it here to fit the page:
# mdadm --detail --scan
ARRAY /dev/md0 level=raid0 num-devices=2 \ UUID=410a299e:4cdd535e:169d3df4:48b7144a
If there were multiple arrays running on the system,mdadmwould generate an array line for each one. Thus, after you’re done building arrays, you could redirect the out-put ofmdadm --detail --scanto/etc/mdadm.conf. Just make sure you manually create aDEVICE entry as well. Chapter 4 contains a complete reference formdadm.conf.