/proc/mdstatprovides a way to examine the state of themddriver, including informa-tion about active software arrays. When no arrays are running, displaying /proc/
mdstat simply shows which RAID levels the kernel supports.
# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid5]
read_ahead not set unused devices: <none>
In this example, there are no active arrays, but this kernel supports linear, RAID-0, RAID-1, and RAID-4/5. Theread_aheadvalue is not currently set, because no arrays are active.read_aheaddefines the number of sectors the kernel should cache during sequential reads. Finally,unused devicesis also empty, because there are no devices in themd subsystem that are not currently in use by an array.
If arrays are defined, /proc/mdstat provides detailed information about them, as shown in the following code:
# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid5]
read_ahead 1024 sectors
md2 : active raid1 sde1[1] sdd1[0]
17920384 blocks [2/2] [UU]
md1 : active raid0 sdc1[1] sdb1[0]
35840768 blocks 64k chunks
First, note that read_ahead is now set to 1024 sectors. That means that during sequential reads, the kernel will attempt to cache a maximum of 1024 sectors worth of data, or about 512 K (1024 sectors, with approximately 512 bytes per sector). The default value of 1024 sectors is a hard limit set by themddriver. Next, each array is listed, with the most recently activated array first. In this case,/dev/md2, a RAID-1, is listed first because it was activated most recently. Let’s examine/dev/md2one line at a time to get a better understanding of the information reported:
md2 : active raid1 sde1[1] sdd1[0]
The first line is fairly straightforward. The array/dev/md2 is an active RAID-1 con-taining two member disks:/dev/sde1and/dev/sdd1. The numbers in square brackets ([ ]) following each member disk indicate the index number of that member disk.
The information corresponds to either a raid-disk entry in/etc/raidtabor to the order
in which member disks were listed on the command line usingmdadm. Here,/dev/
sdd1 is the first raid-disk and/dev/sde1 is the second.
17920384 blocks [2/2] [UU]
This line shows information about the state of the array and its size. This array con-tains 17920384 blocks. Blocks reported in/proc/mdstatare always 1 KB in size. The next data element[2/2]shows that there are two disks in the array and that both are active. The final field [UU]shows that both disks are error-free and operating nor-mally. If a disk had failed, these fields would indicate that disks were missing from the array and which disks had failed. For example:
17920384 blocks [2/1] [_U]
Notice that there are two member disks, but only one disk is currently operational ([2/1]). The next field ([_U]) uses the underscore to indicate that the first disk has failed.
Depending on what type of array is defined, slightly different information is avail-able through/proc/mdstat.
md1 : active raid0 sdc1[1] sdb1[0]
35840768 blocks 64k chunks
This example describes a RAID-0 at/dev/md1. The first line provides the same infor-mation that the RAID-1 example provides, but the second line omits inforinfor-mation about the status of member disks and instead includes information about the chunk-size. Since RAID-0 does not support redundancy, there’s no need to provide infor-mation about how many member disks are online versus how many have failed. The failure of a single disk in a RAID-0 means that the entire array is failed.chunk-size, which isn’t a factor when working with RAID-1, is a requirement for RAID-0. So chunk-size information, instead of failure information, is provided for RAID-0 arrays.
RAID-4 and RAID-5 arrays show a combination of the information provided for RAID-0 or RAID-1 arrays:
md1 : active raid5 sde1[3] sdd1[2] sdc1[1] sdb1[0]
53761152 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
Here, a RAID-5 array defined at/dev/md1contains four member disks. The second line provides information about the chunk size and health of the array (four out of four disks are operational). In addition, the output shows the parity algorithm used, which is algorithm 2 in this case, corresponding to the left-symmetric algorithm. The numeric value reported here comes from a case switch found in the kernel RAID-5 code in the file/usr/src/linux/drivers/block/raid5.c. Each of the usable algorithms is defined there by name.
You might be wondering why the second line contains redundant information about which RAID level is in use. Let’s look at a RAID-4 example to clarify.
md1 : active raid5 sde1[3] sdd1[2] sdc1[1] sdb1[0]
53761152 blocks level 4, 64k chunk, algorithm 0 [4/4] [UUUU]
Notice thatraid5is listed as the array type on the first line, butlevel 4 is listed on the second line. That’s because RAID-4 uses the RAID-5 driver. So when working with these RAID levels, be certain to examine the second line ofmddevice output to make certain that the proper RAID level is reported. Finally, while parity algorithm0 (left asymmetric) is listed, the parity algorithm has no effect on RAID-4. The entry is simply a placeholder and can be safely ignored.
Failed disks
When a disk fails, its status is reflected in/proc/mdstat.
md1 : active raid1 sdc1[1] sdb1[0](F) 17920384 blocks [2/1] [_U]
The first line lists disks in backward order, from most recently added to first added. In this example, the(F)marker indicates that/dev/sdb1 has failed. Note on the following line that there are two disks in the array, but only one of them is active. The next element shows that the first disk (/dev/sdb1) is inactive and the second (/dev/sdc1) is in use. So aUdenotes a working disk, and an_denotes a failed disk. The output is a bit counterintuitive because the order of disks shown in the first line is the opposite of the order of U or _ elements in the second line.
Furthermore, the order in both lines can change as disks are added or removed.
Resynchronization and reconstruction
/proc/mdstat also provides real-time information about array reconstruction and resynchronization. The following mirroring array is nearly halfway done with its ini-tial synchronization.
md1 : active raid1 sdc1[1] sdb1[0]
17920384 blocks [2/2] [UU]
[= = = = = = = = =>...] resync = 46.7% (8383640/17920384) finish=5.4min speed=29003K/sec
In this example, the process is 46.7 percent complete (also indicated by the progress bar). The first number in parentheses indicates how many blocks are ready, out of the total number of blocks (the second number). The resynchronization is expected to take another 5.4 minutes, at the rate of roughly 29 MB (29003K) per second.
Recovery looks nearly identical, except that the failed disk and the newly inserted disk are both displayed.
md1 : active raid1 sdd1[2] sdc1[1] sdb1[0](F) 17920384 blocks [2/1] [_U]
[=>...] recovery = 6.1% (1096408/17920384) finish=8.6min speed=32318K/sec
Note that on the third line, the process is calledrecovery. Remember from Chapter 3 that recovery occurs when a new disk is inserted into an array after a disk fails.
Resynchronization, on the other hand, happens when a new array is created or when disks aren’t synchronized.
Although three disks are listed on the first output line, only two disks appear as array members on the second line. That’s because, when this array was built, only two member disks were defined. So the number of member disks will always remain con-stant. Each failed disk, like active member disks and spare disks, has its own disk index. When a disk in the array fails, it is removed from the raid-disk index and added to the failed-disk index, and that’s why the array disk count remains at two.
Once the recovery is complete, remove the old failed disk using raidhotremove or mdadm -r, and the/proc/mdstat entry will return to normal.
md1 : active raid1 sdd1[0] sdc1[1]
17920384 blocks [2/2] [UU]
If the spare disk also fails during recovery, themddriver will halt the reconstruction process and report both failures in /proc/mdstat, unless additional spare disks are available. In that case, the mddriver will insert the next available spare disk and restart the recovery process.
md0 : active raid5 sdh1[4](F) sdg1[3] sdd1[2] sdf1[1](F) sdb1[0]
53761152 blocks level 5, 64k chunk, algorithm 2 [4/3] [U_UU]
resync=DELAYED unused devices: <none>
/proc/sys/dev/raid
The/proc/sysdirectory provides interfaces for manipulating tunable kernel parame-ters. These parameters, which affect various aspects of the kernel, can be fine-tuned while the system is running. For more general information about /proc/sys, consult the file in the /usr/src/linux/Documentation/sysctl directory of your kernel source code, as well as/usr/src/linux/Documenation/filesystems/proc.txt.
Two files in the /proc/sys/dev/raidsubdirectory provide a way to tune the speed at which array resynchronization (and reconstruction) takes place.speed_limit_minand speed_limit_maxdefine the minimum and maximum speeds at which resynchroniza-tion occur. The latter is especially useful on slower, or heavily utilized, systems, where you might find that the resynchronization process slows down the system too much. In that case, you could change the maximum speed—in effect, throttling the resynchronization process to a more suitable level. The mddriver does provide its own I/O thottling and will attempt to perform recovery nonintrusively, but using speed_limit_minandspeed_limit_maxcan provide a quick fix if something goes hay-wire.
You’ll need to experiment with your system to see the maximum speed it can han-dle. Adjusting the minimum speed affects the low-end limit for resynchronization. It might be a good idea to simulate a drive failure and play with these values so that you aren’t surprised when a real disk failure happens.
You can simulate a disk failure by using theraidsetfaultyormdadm --fail commands, which are described later in this chapter. You can also simu-late a disk failure by disconnecting the cable connected to a drive or by removing a drive from a hot-swap enclosure. Remember, as Iwarned in the section “Hot-swap” in Chapter 2, that you should not discon-nect hardware while the system is powered on, unless you have equip-ment that supports this operation. When simulating a disk failure, the mddriver will not register that a disk has failed until an I/O operation is performed. That means you might have to access or create a file on the array before themd driver reports the failure. Running thesync command to flush filesystem buffers from memory to disk might also be required.
The default limits (100 KB and 100,000 KB, respectively) can be changed by echoing new values into the pseudofiles:
# echo "5000" > /proc/sys/dev/raid/speed_limit_min
# echo "50000" > /proc/sys/dev/raid/speed_limit_max
This changes the low and high limits to 5 MB and 50 MB, respectively. You can add the above commands to your system initialization scripts if you want to change the default values of speed_limit_min andspeed_limit_max automatically each time the system boots. Most distributions now provide the configuration file/etc/sysctl.confto preserve changes to/proc/sysacross reboots. For instance, you can alter resynchroni-zation speeds by adding the following lines to/etc/sysctl.conf:
dev.raid.speed_limit_min = 5000 dev.raid.speed_limit_max = 50000
Consult the manual pages forsysctl andsysctl.conffor further details.
raidtools
raidtools is the traditional package used to manage software arrays. Although the newermdadmis more feature-rich,raidtoolsis in wider use throughout the software RAID community. While it’s a bit more complex to manage arrays usingraidtools, the package has proved reliable for many years.raidtoolshas remained at version 0.
90 for several years, but the prerelease for version 1.0 of raidtools became available during the course of this writing, although it has not been officially released yet. The new version has some new utilities, including a tool that allows users to generate/etc/
raidtabfiles by querying an active array, which is very useful if you’ve accidentally deleted your/etc/raidtabfile. The additions to version 1.0 are covered in this section, but keep in mind that Iused a narrowly released beta version during my testing, and that some functionality might have changed since then.
Version 1.0 fixes several known bugs in the 0.90 release of raidtools. raidtools-20010914 (version 0.90.0), in addition to having some minor bugs, was released without a vital utility,raidsetfaulty.This utility is used to manually induce a disk fail-ure and had been included with previous releases. Instead, raidtools-20010914 shipped with a new program namedraidhotgenerateerror, whose name makes it look like a replacement for raidsetfaulty. Unfortunately, raidhotgenerateerror does not perform the same function asraidsetfaultyand should not be used as a replacement.
raidhotgenerateerror is merely a utility for testing error handling in the md driver.
This has caused some confusion among Linux RAID users. Because of these incon-sistencies, Iadvise against usingraidtools-20010914. Instead, download and use the previous release raidtools-19990824 from ftp.kernel.org/pub/daemons/raid/alpha. I f you are working with a version ofraidtools that came installed with your distribu-tion, check to see if you have /sbin/raidsetfaultyon your system. If it doesn’t exist, then it’s likely that you are working with a repackaged version of raidtools-20010914. In that case, I recommend installing the previous version from source, or using a newer version if one is available and tested. The beta version ofraidtools-1.0 corrects many of the problems with theraidtools-20010914release and also contains raidsetfaulty. It likely thatraidtools-1.0will be in wide release by the time this book is in print.