The Numerical Solution of Linear Systems
4.4 THE TRIANGULAR FACTORIZATION
It is possible to visualize the elimination process of Algorithm 4.2 as deriving a factorization of the coefficient matrix A into three factors,
a permutation matrix P which accounts for the row interchanges made, a unit lower-triangular matrix L containing (in its interesting part) the multipliers used, and the final upper-triangular matrix U. This point of view leads to an efficient algorithm (Choleski factorization, see Exercise 4.4-9) in case A is a symmetric positive definite matrix. It is also of value in understanding the so-called compact schemes (associated with the names of Doolittle and Crout, see Exercise 4.4-8) which are advantageous in solving linear systems on desk (or pocket) calculators, since they reduce the number of intermediate results that have to be recorded. These schemes also permit the use of double-precision accumulation of scalar products (on some machines), for a reduction of rounding-error effects. Finally, the factorization point of view of elimination makes it easy to apply backward error analysis to the elimination process (as will be done in Sec. 4.6). For these reasons, we now exhibit the triangular factorization for A as gener-ated by Algorithm 4.2.
Assume, to begin with, that no row interchanges occurred during execution of the algorithm and consider what happens to the ith equation.
For k = 1, 2, . . . , i - 1, the equation is transformed during the k th step from
t o
4.4 THE TRIANGULAR FACTORIZATION 161 by the prescription
with the multiplier
stored in the (i, k)-entry of the working array. Here, are the coefficients and right side of the pivotal equation for this step, hence are in their final form. This means, in terms of the output from Algorithm 4.2, i.e., in terms of the upper-triangular matrix U and the vector y produced in that algorithm, that
Consequently,
(4.23) (4.24) We now rewrite these equations so that the original data, A and b, appear on the right-hand side. Then we get
(4.25) and
Hence, if we let L = (lij) be the unit lower-triangular matrix given in terms of the final content of the work array W by
(4.26) then we can write these equations (for i = 1, . . . , n) in matrix form simply as
and
This demonstrates the triangular factorization, in case no interchanges occurred. If, on the other hand, interchanges did occur, then the final content of W would have been unchanged had we carried out these interchanges at the outset and then applied Algorithm 4.2 without any interchanges. This is so because all operations in the algorithm involve the subtraction of a certain multiple of one row from certain other rows in order to produce a zero in those other rows, and, for this, it does not matter in which order we have written down the rows. The only thing that matters is that, once a row has been used as pivotal row, it is not modified any further, and, for this, we must keep apart from the others those rows not yet used as pivotal rows.
Consequently, if interchanges do occur during execution of Algorithm 4.2, then the matrices L and U obtained by the algorithm satisfy
for some appropriate permutation matrix P, i.e., then
(4.27)
and also (4.28)
In terms of the vector p used in Algorithm 4.2 to record the inter-changes made, the pkth equation is used as pivot equation during the kth step. Hence P-1 should carry row pk to row k, all k. This means that Pik = iPk, all k [see Theorem 4.5(iii)], if one really wanted to know. All that matters to us, though, is that, in terms of the output p from Algorithm 4.2,
As a first application of the factorization point of view, we now look at the possibility of splitting the process of solving Ax = b into two phases, the factorization phase in which the triangular factors L and U (and a possibly different order p of the rows) are derived, and the solving phase during which one first solves the triangular system
(4.29) for y and then solves the triangular system
for x, by back-substitution. Note that the right-hand side b enters only the second phase. Hence, if the system is also to be solved for some other right-hand sides, only the second phase needs to be repeated.
According to (4.24), one solves (4.29) in Algorithm 4.2 by the steps
4.4 THE TRIANGULAR FACTORIZATION 163 In effect, this is like the back-substitution Algorithm 4.1 for solving Ux = y for x, except that the equations are gone through from first to last, since L is lower-triangular.
We record the entire solving phase in the following:
Algorithm 4.4: Forward- and back-substitution Given the final contents of the first n columns of the working array W and the n-vector p of Algorithm 4.2 (applied to the system Ax = b); also, given the right-side b.
The vector x = (xi) now contains the solution of A x = b.
Note that, once again, both sums are sometimes empty.
The practical significance of the preceding discussion becomes clear when we count (floating-point) operations in Algorithms 4.2 and 4.4. By Exercise 4.22, it takes n divisions, n(n 1)/2 multiplications, and n(n -1)/2 additions to carry out the second loop in Algorithm 4.4. The first loop takes the same number of operations, except that no divisions are required.
Hence Algorithm 4.4 takes By Exercise 4.2-4,
multiply an n × n By contrast,
this is exactly the number matrix with an n-vector.
of operations required to
are necessary to calculate the first n columns of the final contents of the working matrix W by Algorithm 4.2 (see Exercise 4.2-5). Hence the bulk of the work in solving Ax = b by elimination is needed to obtain the final content of the working matrix W, namely, additions and the same number of multiplications/divisions, for large n. The subsequent forward-and back-substitution takes an order of magnitude less operations, namely, additions and the same number of multiplications, per right side.
Hence we can solve Ax = b for many different right sides (once we know the final content of W) in the time it takes to calculate the final content of W.
In this accounting of the work, we have followed tradition and counted only floating-point operations. In particular, we have ignored
index calculations, the cost of managing DO loops and other bookkeeping costs, since these latter calculations used to be much faster than floating-point operations. This is not the case anymore on today’s computers, and this way of accounting the work done may give an inaccurate picture (see Exercise 4.2-7). On the other hand, just how the work (as measured by computing time required) depends on the bookkeeping aspect of a pro-gram varies strongly from computer to computer and is therefore hard to discuss in the generality of this textbook.
A FORTRAN subroutine, called SUBST, which incorporates the substitution Algorithm 4.4, follows.
C B IS AN N-VECTOR, GIVING THE RIGHT SIDE OF THE SYSTEM TO BE SOLVED.
C****** O U T P U T ******
C X IS THE N-VECTOR SATISFYING A*X = B . C****** M E T H O D ******
C ALGORITHM 4.4 IS USED, I.E., THE FACTORIZATION OF A CONTAINED IN C W AND IPIVOT (AS GENERATED IN FACTOR ) IS USED TO SOLVE A*X = B C FOR X BY SOLVING TWO TRIANGULAR SYSTEMS.
C
Next, we give a FORTRAN subroutine called FACTOR, which uses the elimination Algorithm 4.2, with the pivoting strategy dictated by scaled partial pivoting, to calculate a triangular factorization (if possible) for a given N × N matrix A, storing the factorization in an N × N matrix W, and storing the pivoting strategy in an N-vector IPIVOT, ready for use in the subroutine SUBST given earlier. The user must provide an additional N-vector D as a working space needed to store the “size” of the rows of A.
If there is no further need for the matrix A and storage is scarce, then A itself can be used for W in the argument list of the CALL statement (this is illegal in some FORTRAN dialects). The factorization will then replace the original matrix in the array A.
4.4 THE TRIANGULAR FACTORIZATION 165 SUBROUTINE FACTOR ( W, N, D, IPIVOT, IFLAG )
INTEGER IFLAG,IPIVOT(N), I,ISTAR,J,K
REAL D(N) ,W(N,N), AWIKOD,COLMAX,RATIO,ROWMAX,TEMP C****** I N P U T ******
C W ARRAY OF SIZE (N,N) CONTAINING THE MATRIX A OF ORDER N TO BE C FACTORED.
C N THE ORDER OF THE MATRIX C****** W O R K A R E A ******
C D A REAL VECTOR OF LENGTH N, TO HOLD ROW SIZES C****** O U T P U T ******
C W ARRAY OF SIZE (N,N) CONTAINING THE LU FACTORIZATION OF P*A FOR C SOME PERMUTATION MATRIX P SPECIFIED BY IPIVOT .
C IPIVOT INTEGER VECTOR OF LENGTH N INDICATING THAT ROW IPIVOT(K) C WAS USED TO ELIMINATE X(K) , K=l,...,N .
C IFLAG AN INTEGER,
C = 1, IF AN EVEN NUMBER OF INTERCHANGES WAS CARRIED OUT, C = -1, IF AN ODD NUMBER OF INTERCHANGES WAS CARRIED OUT, C
C = 0, IF THE UPPER TRIANGULAR FACTOR HAS ONE OR MORE ZERO DIA-GONAL ENTRIES.
C THUS, DETERMINANT(A) = IFLAG*W(1,1)*...*W(N,N) .
C IF IFLAG .NE. 0, THEN THE LINEAR SYSTEM A*X = B CAN BE SOLVED FOR
C X BY A
C CALL SUBST (W, IPIVOT, B, N, X ) C****** M E T H O D ******
C THE PROGRAM FOLLOWS ALGORITHM 4.2, USING SCALED PARTIAL PIVOTING.
C
DETERMINE PIVOT ROW, THE ROW ISTAR . COLMAX = ABS(W(K,K) )/D(K)
MAKE K THE PIVOT ROW BY INTERCHANGING IT WITH THE CHOSEN ROW ISTAR .
ELIMINATE X(K) FROM ROWS K+1,...,N . DO 19 I=K+1,N
W(I,K) = W(I,K)/W(K,K)
RATIO = W(I,K)
The preceding discussion points toward an efficient way to calculate the inverse for a given invertible matrix A of order n. As was pointed out in Sec. 4.1, for j = 1, . . . , n, the jth column A-1ij of the inverse matrix A-1 is the solution of the linear system
Hence, to calculate A-1, one calls on FACTOR once, then solves each of the n systems Ax = ij, j = 1, . . . , n, by Algorithm 4.4, that is, using SUBST. Therefore, once the elimination is carried out, it takes only n · n2 multiplications, and about the same number of additions, to find A-1.
Having given this simple prescription for calculating the inverse of a matrix, we hasten to point out that there is usually no good reason for ever calculating the inverse. It does at times happen in certain problems that the entries of A-1 have some special physical significance. In the statistical treatment of the fitting of a function to observed data by the method of least squares, for example, the entries of a certain A-1 give information about the kinds and magnitudes of errors in the data. But whenever A-1 is needed merely to calculate a vector A-1b ( a s i n s o l v i n g Ax = b) or a matrix product A-1B , A- 1 s h o u l d never be calculated explicitly. Rather, the substitution Algorithm 4.4 should be used to form these products. The reason for this exhortation is as follows: Calculating the vector A-1b f o r given b amounts to finding the solution of the linear system A x = b. Once the triangular factorization for A has been calculated by Algorithm 4.2, the calculation of A-1b can therefore be accomplished by Algorithm 4.4 in exactly the same number of multiplications and additions as it takes to form the product of A-1 with the vector b, as was pointed out earlier.
Hence, once the triangular factorization is known, no advantage for calculating A-1b can be gained by knowing A-1 explicitly. (Since forming the product A-1B amounts to multiplying each column of B by A-1, these remarks apply to calculating such matrix products as well.) On the other hand, a first step toward calculating A-1 is finding the triangular factoriza-tion for A, which is then followed by n applicafactoriza-tions of the substitufactoriza-tion algorithm; hence calculating A-1 presents a considerable initial computa-tional outlay when compared with the work of calculating A- 1b In addition, the matrix so computed is only an approximate inverse and is, in a sense, less accurate than the triangular factorization, since it is derived from the factorization by further calculations. Hence nothing can be
4.4 THE TRIANGULAR FACTORIZATION 167 gained, and accuracy can be lost, by using A-1 explicitly in the calculation of matrix products involving A-1.
Below, we have listed a FORTRAN program for the calculation of the inverse of a given N × N matrix A. This program uses the subprograms FACTOR and SUBST mentioned earlier. Sample input and the resulting output are also listed. The following remarks might help in the
understand-ing of the codunderstand-ing. The order N of the matrix A is part of the input to this program; hence it is not possible to specify the exact dimension of the matrix A during compilation. On the other hand, both FACTOR and SUBST expect matrices A and/or W of exact dimension N × N. In the FORTRAN program below, the matrix A is therefore stored in a one-di-mensional array, making use of the FORTRAN convention that the (I,J) entry of a two-dimensional (N,M) array is the ((J - 1)*N + I) entry in an equivalent one-dimensional array. The same convention is followed in storing the entries of the Jth column of A-1 in the one-dimensional array AINV: the subroutine SUBST is given the ((J - 1)*N + 1) entry of AINV as the first entry of the N-vector called X in SUBST, into which the solution of the system Ax = ij is to be stored.