MATHEMATICAL ALGORITHMS
6. Curve Fitting
The term curve fitting (or data fitting) is used to describe the general problem of finding a function which matches a set of observed values at a set of given points. Specifically, given the points
and the corresponding values
Yl,Y2,--.,YN,
the goal is to find a function (perhaps of a specified type) such that
f(zl) = Yl, f(z2) = Y2,. . . , f(zN) = YN
and such that f(z) assumes “reasonable” values at other data points. It could be that the z’s and y’s are related by some unknown function, and our goal is to find that function, but, in general, the definition of what is “reasonable”
depends upon the application. We’ll see that it is often easy to identify
“unreasonable” functions.
Curve fitting has obvious application in the analysis of experimental data, and it has many other uses. For example,, it can be used in computer graphics to produce curves that “look nice” withlout the overhead of storing a large number of points to be plotted. A related application is the use of curve fitting
to provide a fast algorithm for computing the value of a known function at an arbitrary point: keep a short table of exact values, curve fit to find other values.
Two principal methods are used to approach this problem. The first is interpolation: a smooth function is to be found which exactly matches the given values at the given points. The second method, least squares data fitting, is used when the given values may not be exact, and a function is sought which matches them as well as possible.
6 7
68 CHAPTER 6
Polynomial Interpolation
We’ve already seen one method for solving the data-fitting problem: if f is known to be a polynomial of degree N - 1, then we have the polynomial inter-polation problem of Chapter 4. Even if we have no particular knowledge about f, we could solve the data-fitting problem by letting f(z) be the interpolating polynomial of degree N - 1 for the given points and values. This could be computed using methods outlined elsewhere in this book, but there are many reasons not to use polynomial interpolation for data fitting. For one thing, a fair amount of computation is involved (advanced N(log N)2 methods are available, but elementary techniques are quadratic). Computing a polynomial of degree 100 (for example) seems overkill for interpolating a curve through 100 points.
The main problem with polynomial interpolation is that high-degree polynomials are relatively complicated functions which may have unexpected properties not well suited to the function being fitted. A result from classical mathematics (the Weierstrass approximation theorem) tells us that it is pos-sible to approximate any reasonable function with a polynomial (of sufficiently high degree). Unfortunately, polynomials of very high degree tend to fluctuate wildly. It turns out that, even though most functions are closely approximated almost everywhere on a closed interval by an interpolation polynomial, there are always some places where the approximation is terrible. Furthermore, this theory assumes that the data values are exact values from some unknown function when it is often the case that the given data values are only ap-proximate. If the y’s were approximate values from some unknown low-degree polynomial, we would hope that the coefficients for the high-degree terms in the interpolating polynomial would be 0. It doesn’t usually work out this way; instead the interpolating polynomial tries to use the high-degree terms to help achieve an exact fit. These effects make interpolating polynomials inappropriate for many curve-fitting applications.
Spline Interpolation
Still, low-degree polynomials are simple curves which are easy to work with analytically, and they are widely used for curve fitting. The trick is to abandon the idea of trying to make one polynomial go through all the points and instead use different polynomials to connect adjacent points, piecing them together smoothly,, An elegant special case of this, which also involves relatively straightforward computation, is called spline interpolation.
A “spline” is a mechanical device used by draftsmen to draw aesthetically pleasing curves: the draftsman fixes a set of points (knots) on his drawing, then bends a flexible strip of plastic or wood (the spline) around them and traces it to produce the curve. Spline interpolation is the mathematical equivalent of this process and results in the same curve.
CURVE FITTING 6 9
It can be shown from elementary mechanics that the shape assumed by the spline between two adjacent knots is a third-degree (cubic) polynomial.
Translated to our data-fitting problem, this means that we should consider the curve to be N - 1 different cubic polynomials
Si(X) = aix3 + biX2 + cix + di, i=1,2 ,..., N - l ,
with si(x) defined to be the cubic polynomial to be used in the interval between xi and xi+17 as shown in the following diagram:
yps ... A:‘“‘“’
The spline can be represented in the obvious way as four one-dimensional arrays (or a 4-by-(N - 1) two-dimensional array). Creating a spline consists of computing the necessary a, b, c, d coefficients from the given x points and y values. The physical constraints on the spline correspond to simultaneous equations which can be solved to yield the coefficients.
For example, we obviously must have si(xi) = yi and si(xi+i) = yi+l for i=1,2 ,***, N - 1 because the spline must touch the knots. Not only does the spline touch the knots, but also it curves ;smoothly around them with no sharp bends or kinks. Mathematically, this means that the first derivatives of the spline polynomials must be equal at the knots (si-i(xi) = s:(xi) for i = 2,3,. . . , N - 1). In fact, it turns out that the second derivatives of the polynomials must be equal at the knots. These conditions give a total of 4N - 6 equations in the 4(N-1) unknown coefficients. Two more conditions need to be specified to describe the situation at the endpoints of the spline. Several options are available; we’ll use the so-called “natural” spline which derives from s;1(xr) = 0 and s&,(x~) = 0. T he s e conditions give a full system of 4N - 4 equations in 4N - 4 unknowns, which could be solved using Gaussian elimination to
calculate all the coefficients that describe the spline.
The same spline can be computed somewhat more efficiently because there are actually only N - 2 “unknowns”: most of the spline conditions are
redundant. For example, suppose that p, is the value of the second derivative of the spline at xi, so that s:)_~(x~) = .sy(xi) = pi for i = 2,. . . , N - 1, with pr = pN = 0. If the values of pl, . . . ,phr are known, then all of the a, b, c, d
coefficients can be computed for the spline segments, since we have four
70 CHAPTER 6
equations in four unknowns for each spline segment: for i = 1,2,. . . , N - 1, we must have
Si(&) = yi
%(%+1) = !A+1 ((Xi) = pi s:‘(xi+l) = pi+1.
Thus, to fully determine the spline, we need only compute the values of
P21..., pN-1. But this discussion hasn’t even considered the conditions that the first derivatives must match. These N - 2 conditions provide exactly the N - 2 equations needed to solve for the N - 2 unknowns, the pi second derivative values.
To express the a, b, c, and d coefficients in terms of the p second derivative values, then substitute those expressions into the four equations listed above for each spline segment, leads to some unnecessarily complicated expressions.
Instead it is convenient to express the equations for the spline segments in a certain canonical form that involves fewer unknown coefficients. If we change variables to t = (CC - zi)/(x,+1 - Q) then the spline can be expressed in the following way:
sip) = tyi+1 + (1 - t)yi + ( x,+1 - xd2[(t3 - qpi+1 - ((1 -t)” - (1 - t))pJ Now each spline is defined on the interval [O,l]. This equation is less formi-dable than it looks because we’re mainly interested in the endpoints 0 and 1, and either t or (1 - t) is 0 at these points. It’s trivial to check that the spline interpolates and is continuous because si-r(l) = si(O) = yi for i = 2,. . . , N-l, and it’s only slightly more difficult to verify that the second derivative is con-tinuous because s:(l) = s;+,(O) = pi+i. These are cubic polynomials which satisfy the requisite conditions at the endpoints, so they are equivalent to the spline segments described above. If we were to substitute for t and find the coefficient of x3, etc., then we would get the same expressions for the a’s, b’s, c’s, and d’s in terms of the x’s, y’s, and p’s as if we were to use the method described in the previous paragraph. But there’s no reason to do so, because we’ve checked that these spline segments satisfy the end conditions, and we can evaluate each at any point in its interval by computing t and using the above formula (once we know the p’s).
To solve for the p’s we need to set the first derivatives of the spline segments equal at the endpoints. The first derivative (with respect to x) of the above equation is
s:(t) = zi + (Xi+1 - zJ[(3t2 - l)pi+i + (3(I - t)2 + I)pi]
CURVE FITTING 7 1
where z = (yi+l-yi)/(zi+l-zi). Now, setting &(l) = s;(O) for i = 2,. . . , N-1 gives our system of N - 2 equations to solve:
(Xi - ~i-l)Pi-l+2(~i,-l - +1)p, + (Xi+1 -z&+1 = zi -z&1.
This system of equations is a simple “tridiagonal” form which is easily solved with a degenerate version of Gaussian elimination as we saw in Chapter 5. If we let ui = zi+l - zi, di = 2(zi+l -xi--i), and wi = zi - zi.-1, we have, for example, the following simultaneous equ.ations for N = 7:
In fact, this is a symmetric tridiagonal system, with the diagonal below the main diagonal equal to the diagonal above the main diagonal. It turns out that pivoting on the largest available element is not necessary to get an accurate solution for this system of equations.
The method described in the above paragraph for computing a cubic spline translates very easily into Pascal:
procedure makespline;
var i: integer;
begin readln (N) ;
for i:=l to N do readln(x[i],y[i]);
for i:=2 to N-l do d[i]:=2*(x[i+l]-x[i-11);
for i:=l to N-l do u[i]:=x[i+l]-x[i];
for i:=2 to N-l do
w[i]:=(y[i+l]--y[i])/u[i]-(y[i]-y[i-l])/u[i-11;
p[l] :=o.o; p[Nj :=o.o;
for i:=2 to N-2 do begin
w[i+l]:=w[i+;l]-w[i]*u[i]/d[i];
d[i+l]:=d[i+l]-u[i]*u[i]/d[i]
end ;
for i:=N-1 downto 2 do p[i]:=(w[i]-u[i]*p[i+l])/d[i];
end ;
7 2 CHAPTER 6
The arrays d and u are the representation of the tridiagonal matrix that is solved using the program in Chapter 5. We use d[i] where a[i, i] is used in that program, u[i] where a [i+l, i] or a[i, i+l] is used, and z[i] where a[i, N+I]
is used.
For an example of the construction of a cubic spline, consider fitting a spline to the five data points
(1.0,2.0), (2.0,1.5), (4.0,1.25), (5.0,1.2), (8.0,1.125), (10.0,l.l).
(These come from the function 1 + l/z.) The spline parameters are found by solving the system of equations
with the result p2 = 0.06590, p3 = -0.01021, p4 = 0.00443, ps = -0.00008.
To evaluate the spline for any value of 2 in the range [zr , zN], we simply find the interval [zi, zi+r] containing z, then compute t and use the formula above for si(z) (which, in turn, uses the computed values for pi and pi+r).
function eval(v: real): real;
var t: real; i: integer;
function f(x: real): red;
begin f:=x*x*x-x end;
begin
i:=O; repeat i:=i+l until v<=x[i+l];
t:=(v-x[i])/u[i];
eval:=t*y[i+l]+(l-t)*y[i]
end +u[i]*u[i]*(f(t)*p[i+l]-f(l-t)*p[i])
;
This program does not check for the error condition when v is not between x[l] and x[IVl. If there are a large number of spline segments (that is, if N is large), then there are more efficient “searching” methods for finding the interval containing v, which we’ll study in Chapter 14.
There are many variations on the idea of curvefitting by piecing together polynomials in a “smooth” way: the computation of splines is a quite well-developed field of study. Other types of splines involve other types of smooth-ness criteria as well as changes such as relaxing the condition that the spline must exactly touch each data point. Computationally, they involve exactly
CURVE FITTING 73
the same steps of determining the coefficients for each of the spline pieces by solving the system of linear equations derived from imposing constraints on how they are joined.
Method of Least Squares
A very common experimental situation is that, while the data values that we have are not exact, we do have some idea of the form of the function which is to fit the data. The function might depend on some parameters
and the curve fitting procedure is to find the choice of parameters that “best”
matches the observed values at the given points. If the function were a poly-nomial (with the parameters being the coefficients) and the values were exact, then this would be interpolation. But now we are considering more general functions and inaccurate data. To simplify the discussion, we’ll concentrate on fitting to functions which are expressed as a linear combination of simpler functions, with the unknown parameters being the coefficients:
f(x) = Clfl(X) + c2f2(x) +**a+ cA4fdx).
This includes most of the functions that we’ll be interested in. After studying this case, we’ll consider more general functions.
A common way of measuring how well a function fits is the least-squares criterion: the error is calculated by adding up the squares of the errors at each of the observation points:
E = c (f(xj)-yj)2.
l<j<N
-This is a very natural measure: the squaring is done to stop cancellations among errors with different signs. Obviously, it is most desirable to find the choice of parameters that minimizes E. It turns out that this choice can be computed efficiently: this is the so-called method of least squares.
The method follows quite directly from the definition. To simplify the derivation, we’ll do the case M = 2, N = 3, but the general method will follow directly. Suppose that we have three points xi, x2, x3 and corresponding values yi, ys, ys which are to be fitted to a function of the form f(x) = cl fi(x) + cz f2(x). Our job is to find the choice of the coefficients cl, cz which minimizes the least-squares error
E =(clfl(xl) + czfdxd - yd2 + (nf1(x2) + czfz(x2) - Y2)2 + (Clfi(X3) + c2f2(x3) - Y3)2.
74 CHAPTER 6
To find the choices of cl and c2 which minimize this error, we simply need to set the derivatives dE/dq and dE/dcz to zero. For cl we have:
g =2(cl.fl(G) + Czfz(~1) - Yl)fl(~l) 1
+ 2(Clfl(~Z) + czfz(z2) - yz)f1(52) +2(clfl(~3)+CZf2(~3)-Y3)fi(X3).
Setting the derivative equal to zero leaves an equation which the variables cl and cs must satisfy (fi(~i), etc. are all “constants” with known values):
clvl(~1vl(~l) + fl(~Z)fl(~Z) + fl(Z3)fi(53)]
+c2[f2(~l)fl(~l) + f2(~2)fl(z2) + f2(~3)fl(z3)]
= Yl.fl(G) + YZfl(52) + Y3fl(53).
We get a similar equation when we set the derivative dE/dc2 to zero.
These rather formidable-looking equations can be greatly simplified using vector notation and the “dot product” operation that we encountered briefly in Chapter 2. If we define the vectors x = (21, zz,23) and y = (yi, yz, ys) and then the dot product of x and y is the real number defined by
X-Y = z1y1+ xzy2 + 23y3
Now, if we define the vectors fi = (fl(xi),fi(xz),fi(xs)) and fs = (fs(xi),
f2(52), f2(23)) the n our equations for the coefficients cl and cz can be very simply expressed:
Clfi . fi + CZfi * fz = y . fi ClfZ.fi +c2f2 *f2 =y-f2.
These can be solved with Gaussian elimination to find the desired coefficients.
For example, suppose that we know that the data points
(1.0,2.05), (2.0,1.53), (4.0,1.26), (5.0,1.21), (8.0,1.13), (10.0,l.l).
should be fit by a function of the form cl + cz/x. (These data points are slightly perturbed from the exact values for 1 + l/x). In this case, we have fi = (l.O,l.O, l.O,l.O,l.O, 1.0) and f2 = (1.0,0.5,0.25,0.2,0.125,0.1) so we have to solve the system of equations
CURVE FITTING 7 5
with the result cl = 0.998 and c2 = 1.054 (both close to 1, as expected).
The method outlined above easily generalizes to find more than two coefficients. To find the constants ~1~2,. . . ,CM in
f(z) = clfl(z) + czfz(s) + *+* + CA4fM(Z)
which minimize the least squares error for the point and observation vectors
first compute the function component vectors
fl = (fl(d, f1(s2), . . . , fl(ZN)), f-2 = (f2(~1), f2(z2), . * *, f2(Ziv)),
Then make up an M-by-M linear system of equations AC = b with a,j = fi ” fy,
b, = fj . y.
The solution to this system of simultaneous equations yields the required coefficients.
This method is easily implemented by maintaining a two dimensional array for the f vectors, considering y as the (M + 1)st vector. Then an array a[l..M, I..M+I] can be filled as follows:
for i:=l to Mdo for j:==l to M+l do
begin it:= 0.0;
for k:=l to N do t:=t+f[i, k]*fb, k];
a[& j]:=t;
end;
and then solved using the Gaussian elimination procedure from Chapter 5.
The method of least squares can be extended to handle nonlinear func-tions (for example a function such as f(x) = cle-C2Zsincg~), and it is often
76 CHAPTER 6
used for this type of application. The idea is fundamentally the same; the problem is that the derivatives may not be easy to compute. What is used is an iterative method: use some estimate for the coefficients, then use these within the method of least squares to compute the derivatives, thus producing a better estimate for the coefficients. This basic method, which is widely used today, was outlined by Gauss in the 1820s.
CURVE FITTING 77
Exercises
1. Approximate the function lgx with a degree 4 interpolating polynomial at the points 1,2,3,4, and 5. Estimat.e the quality of the fit by computing the sum of the squares of the errors at 1.5, 2.5, 3.5, and 4.5.
2. Solve the previous problem for the function sinx. Plot the function and the approximation, if that’s possible on your computer system.
3. Solve the previous problems using a cubic spline instead of an interpolat-ing polynomial.
4. Approximate the function lgx with a cubic spline with knots at 2N for N between 1 and 10. Experiment with different placements of knots in the same range to try to obtain a better fit.
5. What would happen in least squares data fitting if one of the functions was the function Ii(x) = 0 for some i?
6. What would happen in least squares data-fitting if all the observed values were O?
7. What values of a, b, c minimize the least-squares error in using the function f(x) = ux log x + bx + c to approximate the observations f(1) = 0, f(4) = 13, f(8) = 41?
8. Excluding the Gaussian elimination phase, how many multiplications are involved in using the method of least squares to find M coefficients based on N observations?
9. Under what circumstances would the matrix which arises in least-squares curve fitting be singular?
10. Does the least-squares method work if two different observations are in-cluded for the same point?
7. Integration
Computing the integral is a fundamental analytic operation often per-formed on functions being processed on computers. One of two com-pletely different approaches can be used, depending on the way the function is represented. If an explicit representation of the function is available, then it may be possible to do symbolic integrathn to compute a similar representation for the integral. At the other extreme, the function may be defined by a table, so that function values are known for only a few points. The most common situation is between these: the function to be integrated is represented in such a way that its value at any particular point can be computed. In this case, the goal is to compute a reasonable approximation to the integral of the func-tion, without performing an excessive number of function evaluations. This computation is often called quadrature by numerical analysts.
Symbolic Integration
If full information is available about a function, then it may be worthwhile to consider using a method which involves manipulating some representation of the function rather than working with numeric values. The goal is to transform a representation of the function into a representation of the integral, in much the same way that indefinite integration is done by hand.
A simple example of this is the integ,ration of polynomials. In Chapters 2 and 4 we examined methods for “symbolically” computing sums and products of polynomials, with programs that work.ed on a particular representation for the polynomials and produced the representation for the answers from the rep-resentation for the inputs. The operation of integration (and differentiation) of polynomials can also be done in this way. If a polynomial
79
8 0 CHAPTER 7
is represented simply by keeping the values of the coefficients in an array p then the integral can be easily computed as follows:
!;;:I! downto 1 do p [i] :=p[i-II/i;
: ;
This is a direct implementation of the well-known symbolic integration rule Jc tie’ dt = xi/i for i > 0.
Obviously a wider class of functions than just polynomials can be handled by adding more symbolic rules. The addition of composite rules such as integration by parts,
/udv=uv- s v du,
can greatly expand the set of functions which can be handled. (Integration by parts requires a differentiation capability. Symbolic differentiation is some-what easier than symbolic integration, since a reasonable set of elementary rules plus the composite chain rule will suffice for most common functions.)
The large number of rules available to be applied to a particular function makes symbolic integration a difficult task. Indeed, it has only recently been shown that there is an algorithm for this task: a procedure which either returns the integral of any given function or says that the answer cannot be
The large number of rules available to be applied to a particular function makes symbolic integration a difficult task. Indeed, it has only recently been shown that there is an algorithm for this task: a procedure which either returns the integral of any given function or says that the answer cannot be