Definition 2.4. (Condition Number)The condition numberκof a prob-lem P:Rn→Rm is the smallest number such that
|xˆi−xi|
|xi| ≤ε for 1≤i≤n =⇒ P(ˆx)− P(x)
P(x) ≤κε+o(ε), (2.8) whereo(ε)represents terms that are asymptotically smaller thanε.
A problem iswell conditioned ifκis not too large; otherwise the problem isill conditioned. Well-conditioned means that the solution of the problem with slightly perturbed data does not differ much from the solution of the problem with the original data. Ill-conditioned problems are problems for which the solution is very sensitive to small changes in the data.
Example 2.3. We consider the problem of multiplying two real numbers, P(x1, x2) :=x1x2. If we perturb the data slightly, say
ˆ
x1:=x1(1 +ε1), xˆ2:=x2(1 +ε2), |εi| ≤ε, i= 1,2, we obtain
ˆ
x1xˆ2−x1x2
x1x2 = (1 +ε1)(1 +ε2)−1 =ε1+ε2+ε1ε2,
and since we assumed that the perturbations are small, ε1, we can neglect the productε1ε2compared to the sumε1+ε2, and we obtain
|ˆx1xˆ2−x1x2|
|x1x2| ≤2ε.
Hence the condition number of multiplication is κ = 2, and the problem of multiplying two real numbers is well conditioned.
Example 2.4. Let A be a fixed, non-singularn×n matrix. Consider the problem of evaluating the matrix-vector product P(x) = Ax for x = (x1, . . . , xn). Suppose the perturbed vectorxˆsatisfiesˆxi=xi(1 +εi),|εi|<
ε. Then considering the infinity norm, we havexˆ−x∞≤εx∞, so that Aˆx−Ax∞
Ax∞ ≤ A∞xˆ−x∞
Ax∞ ≤ εA∞x∞
Ax∞ .
Since x=A−1Ax ≤ A−1 Ax, we in fact have Aˆx−Ax∞
Ax∞ ≤εA∞A−1∞,
so the condition number is κ∞ = A∞A−1∞. We will see in Chapter 3 that κ∞ also plays an important role in the solution of linear systems of equations.
Example 2.5. Let us look at the condition number of subtracting two real numbers,P(x1, x2) :=x1−x2. Perturbing again the data slightly as in the previous example, we obtain
|(ˆx1−xˆ2)−(x1−x2)|
|x1−x2| = |x1ε1−x2ε2|
|x1−x2| ≤ |x1|+|x2|
|x1−x2| ε.
We see that if sign(x1) =−sign(x2), which means the operation is an addi-tion, then the condition number is κ= 1, meaning that the addition of two numbers is well conditioned. If, however, the signs are the same andx1≈x2, thenκ= |x|x11|+|x−x22|| becomes very large, and hence subtraction is ill conditioned in this case.
As a numerical example, taking x1= 511 andx1= 521, we obtain for the condition numberκ= 5111+521
51−521 = 103, and if we compute with three significant digits, we obtain xˆ1 = 0.196e−1, xˆ2 = 0.192e−1, and xˆ1−xˆ2 = 0.400e−3, which is very different from the exact result x1−x2 = 0.377e−3; thus, the large condition number reflects the fact that the solution is prone to large errors due to cancellation.
In our decimal example above, the loss of accuracy is easily noticeable, since the lost digits appear as zeros. Unfortunately, cancellation effects are rarely as obvious when working with most modern computers, which use bi-nary arithmetic. As an example, let us first compute inMaple
> Digits:=30;
> a:=1/500000000000001;
1 500000000000001
> b:=1/500000000000002;
1 500000000000002
> ce:=a-b;
1
250000000000001500000000000002
> c:=evalf(a-b);
Condition of a Problem 31 3.99999999999997600000000000011×10−30 (2.9) This is the exact difference, rounded to thirty digits. We round the operands to 16-digit machine numbers:
> bm:=evalf[16](b);
1.999999999999992×10−15
> am:=evalf[16](a);
1.999999999999996×10−15 Now we calculate the difference to 30-digit precision
> c16:=am-bm;
4.0×10−30
and also the difference to 16-digit precision to emulate floating point opera-tions:
> cf16:=evalf[16](am-bm);
4.0×10−30
which agrees with to the standard modela−˜b= (a−b)(1 +r)withr= 0. In Maple, we see that the precision in this difference is reduced to two digits.
Only two digits are displayed, which indicates a serious loss of precision due to cancellation.
Performing the same operations inMatlab, we get:
format long
>> a=1/500000000000001 a =
1.999999999999996e-15
>> b=1/500000000000002 b =
1.999999999999992e-15
>> c=a-b c =
3.944304526105059e-30
Comparing with the exact difference in Equation (2.9), we see again that only the first two digits are accurate. The apparently random digits appearing in the difference stem from binary-to-decimal conversion, and it would be hard to guess that they are meaningless. Had we looked at the same computation in hexadecimal format inMatlab,
format hex
>> a=1/500000000000001 a =
3ce203af9ee7560b
>> b=1/500000000000002 b =
3ce203af9ee75601
>> c=a-b c =
39d4000000000000
we would have seen thatMatlabalso completes the result by zeros, albeit in binary.
Example 2.6. Consider once again the problem of evaluating the deriva-tive of f numerically via the formula
f(x0)≈Dx0,h(f) = f(x0+h)−f(x0)
h .
Here, the functionf acts as data to the problem. We consider perturbed data of the form
f(x) =ˆ f(x)(1 +εg(x)), |g(x)| ≤1,
which models the effect of roundoff error on a machine with precisionε. The condition number then becomes
Dx0,h( ˆf)−Dx0,h(f) Dx0,h(f)
=
f(x0+h)(1+ε g(x0+h))−f(x0)(1 +ε g(x0)) f(x0+h)−f(x0) −1
≤ ε(|f(x0)|+|f(x0+h)|)
|f(x0+h)−f(x0)| ≈ 2ε|f(x0)|
|hf(x0)|.
Thus, we have κ ≈ 2|f(x0)|/|hf(x0)|, meaning that the problem becomes more and more ill-conditioned as h→ 0. In other words, if h is too small relative to the perturbationε,it is impossible to evaluateDx0,h(f) accurately, no matter how the finite difference is implemented.
A related notion in mathematics iswell-orill-posed problems, introduced by Hadamard [60]. LetA:X → Y be a mapping of some space X toY. The problem Ax=yis well posed if
1. For eachy∈Y there exists a solutionx∈X.
2. The solutionxis unique.
3. The solutionxis a continuous function of the the datay.
If one of the conditions is not met, then the problem is said to be ill posed.
For example, the problem of calculating the derivativefbased on the values off alone is an ill-posed problem in the continuous setting, as can be seen from the fact that κ → ∞ as h → 0 in Example 2.6. A sensible way of choosing hto obtain maximum accuracy is discussed in Chapter5.
Stable and Unstable Algorithms 33
If a problem is ill-posed because condition 3 is violated, then it is also ill conditioned. But we can also speak of an ill-conditioned problem if the problem is well posed but if the solution is very sensitive with respect to small changes in the data. A good example of an ill-conditioned problem is finding the roots of the Wilkinson polynomial, see Chapter 5. It is impossible to
“cure” an ill-conditioned problem by a good algorithm, but one should avoid transforming a well-conditioned problem into an ill-conditioned one by using a bad algorithm, e.g. one that includes ill-conditioned subtractions that are not strictly necessary.