Chapter 1 Numerical approximation of data : interpolation, least squares method

Chapter 1

Numerical approximation of data :

interpolation, least squares method

I. Motivation

1

Approximation of functions

Evaluation of a function

Which functions (f : R → R ) can be effectively evaluated in any point ?

Þ the power functions : f (x) = x ^m , m ∈ N Þ the polynomial functions :

f (x) = a 0 + a 1 x + a 2 x ² + · · · + a m x ^m How can we evaluate other functions in a given point ?

for instance : f(x) = cos(x), f (x) = sin(x) exp(x),... Þ approximation by a polynomial function :

H using a Taylor series about the given point,

H searching a polynomial having the same values as the function in some close points

á Lagrange interpolation

Evaluation of a function

Which functions (f : R → R ) can be effectively evaluated in any point ?

Þ the power functions : f (x) = x ^m , m ∈ N Þ the polynomial functions :

f (x) = a 0 + a 1 x + a 2 x ² + · · · + a m x ^m

How can we evaluate other functions in a given point ? for instance : f(x) = cos(x), f (x) = sin(x) exp(x),... Þ approximation by a polynomial function :

H using a Taylor series about the given point,

H searching a polynomial having the same values as the function in some close points

á Lagrange interpolation

Evaluation of a function

Which functions (f : R → R ) can be effectively evaluated in any point ?

Þ the power functions : f (x) = x ^m , m ∈ N Þ the polynomial functions :

f (x) = a 0 + a 1 x + a 2 x ² + · · · + a m x ^m How can we evaluate other functions in a given point ?

for instance : f(x) = cos(x), f (x) = sin(x) exp(x),...

Þ approximation by a polynomial function : H using a Taylor series about the given point,

H searching a polynomial having the same values as the function in some close points

á Lagrange interpolation

Evaluation of a function

Which functions (f : R → R ) can be effectively evaluated in any point ?

Þ the power functions : f (x) = x ^m , m ∈ N Þ the polynomial functions :

f (x) = a 0 + a 1 x + a 2 x ² + · · · + a m x ^m How can we evaluate other functions in a given point ?

for instance : f(x) = cos(x), f (x) = sin(x) exp(x),...

Þ approximation by a polynomial function : H using a Taylor series about the given point,

H searching a polynomial having the same values as the function in some close points

á Lagrange interpolation

Principles of Lagrange interpolation

f (x) = sin( πx

2 )(x ² + 3)

Principles of Lagrange interpolation

f (x) = sin( πx

2 )(x ² + 3)

4 points on the curve : (−1, −4),

(1, 4),

(2, 0),

(3, −12)

Principles of Lagrange interpolation

f (x) = sin( πx

2 )(x ² + 3) Lagrange interpolating polynomial

4 points on the curve : P polynomial of degree ≤ 3 satisfying

(−1, −4), P (−1) = −4

(1, 4), P (1) = 4

(2, 0), P (2) = 0

(3, −12) P(3) = −12

Principles of Lagrange interpolation, with 6 points

f (x) = sin( πx

2 )(x ² + 3) Lagrange interpolating polynomial

6 points on the curve : P polynomial of degree ≤ 5 satisfying

(x _i , y _i ) 0≤i≤5 , P (x _i ) = y _i for 0 ≤ i ≤ 5

Remark : ouside the interval defined by the (x _i ), the Lagrange

interpolating polynomial has nothing to do with f .

I. Motivation

1

Approximation of functions

2

Curve approximation

Piecewise interpolation

f (x) = sin( πx

2 )(x ² + 3)

piecewise affine approximation

Piecewise interpolation

f (x) = sin( πx

2 )(x ² + 3) piecewise affine approximation

Piecewise interpolation

f (x) = sin( πx

2 )(x ² + 3) piecewise affine approximation

Applications : Calculation of an approximate value of the length of the curve

the area under the curve (here Z 5

−5

f(x)dx)

Þ see chapter 2

Piecewise interpolation

f (x) = sin( πx

2 )(x ² + 3) piecewise affine approximation

Applications : Calculation of an approximate value of the length of the curve

the area under the curve (here Z 5

−5

f (x)dx)

Þ see chapter 2

Cubic spline

f (x) = sin( πx

2 )(x ² + 3) s cubic spline

Principle :

between two consecutive points, s is a cubic polynomial s(x _i ) = f (x _i )

s ∈ C ²

+ two conditions on the boundary points

Cubic spline

f (x) = sin( πx

2 )(x ² + 3) s cubic spline

Principle :

between two consecutive points, s is a cubic polynomial s(x _i ) = f (x _i )

s ∈ C ²

+ two conditions on the boundary points

B´ ezier curves : principle

P ₀ , P ₁ , · · · , P _n , are n + 1 given control points

The corresponding B´ ezier curve is defined by M (t) =

n

X

i=0

B _n ⁱ (t)P i , 0 ≤ t ≤ 1 where B _n ⁱ are Bernstein polynomial defined by

B _n ⁱ = C _n ⁱ X ⁱ (1 − X) ⁿ⁻ⁱ , with C _n ⁱ = n!

i!(n − i)! .

B´ ezier curves : principle

P ₀ , P ₁ , · · · , P _n , are n + 1 given control points The corresponding B´ ezier curve is defined by

M (t) =

n

X

i=0

B _n ⁱ (t)P i , 0 ≤ t ≤ 1 where B _n ⁱ are Bernstein polynomial defined by

B _n ⁱ = C _n ⁱ X ⁱ (1 − X) ⁿ⁻ⁱ , with C _n ⁱ = n!

i!(n − i)! .

B´ ezier curves : with more points

Application of the B´ ezier curves

I. Motivation

1

Approximation of functions

2

Curve approximation

3

Fitting of statistical data

Study of statistical data

Some experimental measurements

H X : noise level in the factory (in dB),

H Y : time used to do a definite work (in minutes)

X 73 78 76 63 81 70 75 81 79 84 50 76 65 58 Y 77 85 79 67 83 73 72 83 81 82 52 77 65 58

A physical measure always contains some noise. Can we find a law linking Y and X (Y = f (X)) ?

Can we predict the value of

Y for X = 66dB ?

Linear regression

Principle

We are looking for a and b such that d(a, b) = X

(y i − (ax i + b)) ² is minimal.

The straight line y = ax + b is the linear regression line.

The polynomial P = aX + b is the least squares fitting

polynomial of the cloud of points.

II. Lagrange interpolating polynomial : theoretical study

1

Study of an example

Lagrange interpolation in 1 point

f (x) = sin( πx

2 )(x ² + 3)

1 point on the curve : Search for P 0 , such that (x 0 , y 0 ) = (1, 4) deg P 0 ≤ 0 and

P ₀ (x ₀ ) = y ₀

P ₀ = 4

Lagrange interpolation in 1 point

f (x) = sin( πx

2 )(x ² + 3)

1 point on the curve : Search for P 0 , such that (x 0 , y 0 ) = (1, 4) deg P 0 ≤ 0 and

P ₀ (x ₀ ) = y ₀

P ₀ = 4

Lagrange interpolation in 1 point

f (x) = sin( πx

2 )(x ² + 3)

1 point on the curve : Search for P 0 , such that (x 0 , y 0 ) = (1, 4) deg P 0 ≤ 0 and

P ₀ (x ₀ ) = y ₀

P ₀ = 4

Lagrange interpolation in 2 points

f (x) = sin( πx

2 )(x ² + 3)

2 points on the curve : Search for P 1 , such that (x 0 , y 0 ) = (1, 4) deg P 1 ≤ 1 and (x ₁ , y ₁ ) = (−1, −4) P ₁ (x ₀ ) = y ₀ and P ₁ (x ₁ ) = y ₁

P ₁ = 4X

Lagrange interpolation in 2 points

f (x) = sin( πx

2 )(x ² + 3)

2 points on the curve : Search for P 1 , such that (x 0 , y 0 ) = (1, 4) deg P 1 ≤ 1 and (x ₁ , y ₁ ) = (−1, −4) P ₁ (x ₀ ) = y ₀ and P ₁ (x ₁ ) = y ₁

P ₁ = 4X

Lagrange interpolation in 2 points

f (x) = sin( πx

2 )(x ² + 3)

2 points on the curve : Search for P 1 , such that (x 0 , y 0 ) = (1, 4) deg P 1 ≤ 1 and (x ₁ , y ₁ ) = (−1, −4) P ₁ (x ₀ ) = y ₀ and P ₁ (x ₁ ) = y ₁

P ₁ = 4X

Lagrange interpolation in 3 points

f (x) = sin( πx

2 )(x ² + 3)

3 points on the curve : Search for P 2 , such that (x 0 , y 0 ) = (1, 4) deg P 2 ≤ 2 and (x ₁ , y ₁ ) = (−1, −4) P ₂ (x ₀ ) = y ₀ , P ₂ (x ₁ ) = y ₁

(x 2 , y 2 ) = (3, −12) and P 2 (x 2 ) = y 2

Lagrange interpolation in 3 points

f (x) = sin( πx

2 )(x ² + 3)

3 points on the curve : Search for P 2 , such that (x 0 , y 0 ) = (1, 4) deg P 2 ≤ 2 and (x ₁ , y ₁ ) = (−1, −4) P ₂ (x ₀ ) = y ₀ , P ₂ (x ₁ ) = y ₁

(x 2 , y 2 ) = (3, −12) and P 2 (x 2 ) = y 2

Lagrange interpolation in 3 points

Idea

Search for L 0 , such that deg L 0 = 2 and

L ₀ (x ₁ ) = L ₀ (x ₂ ) = 0 and L ₀ (x ₀ ) = 1.

Search for L ₁ , such that deg L ₁ = 2 and

L 1 (x 0 ) = L 1 (x 2 ) = 0 and L 1 (x 1 ) = 1.

Search for L 2 , such that deg L 0 = 2 and

L ₂ (x ₀ ) = L ₂ (x ₁ ) = 0 and L ₂ (x ₂ ) = 1.

Prove that P ₂ = y ₀ L ₀ + y ₁ L ₁ + y ₂ L ₂ is a solution of the pb.

Give the expression of P .

á P 2 = −3X ² + 4X + 3

Lagrange interpolation in 3 points

Idea

Search for L 0 , such that deg L 0 = 2 and

L ₀ (x ₁ ) = L ₀ (x ₂ ) = 0 and L ₀ (x ₀ ) = 1.

Search for L ₁ , such that deg L ₁ = 2 and

L 1 (x 0 ) = L 1 (x 2 ) = 0 and L 1 (x 1 ) = 1.

Search for L 2 , such that deg L 0 = 2 and

L ₂ (x ₀ ) = L ₂ (x ₁ ) = 0 and L ₂ (x ₂ ) = 1.

Prove that P ₂ = y ₀ L ₀ + y ₁ L ₁ + y ₂ L ₂ is a solution of the pb.

Give the expression of P .

á P 2 = −3X ² + 4X + 3

Lagrange interpolation in 3 points

Other idea

P 1 = 4X satisfies P 1 (1) = 4, P 1 (−1) = −4 and deg P 1 = 1.

Q = (X + 1)(X − 1) satisfies Q(1) = Q(−1) = 0 and deg Q = 2 (in fact Q = −L ₂ )

Search for P ₂ under the form

P 2 = P 1 + αQ = 4X + α(X + 1)(X − 1).

P ₂ (3) = −12 ⇐⇒ α = −3 and

P ₂ = 4X − 3(X + 1)(X − 1) = −3X ² + 4X + 3

Lagrange interpolation in 3 points

Other idea

P 1 = 4X satisfies P 1 (1) = 4, P 1 (−1) = −4 and deg P 1 = 1.

Q = (X + 1)(X − 1) satisfies Q(1) = Q(−1) = 0 and deg Q = 2 (in fact Q = −L ₂ )

Search for P ₂ under the form

P 2 = P 1 + αQ = 4X + α(X + 1)(X − 1).

P 2 (3) = −12 ⇐⇒ α = −3 and

P ₂ = 4X − 3(X + 1)(X − 1) = −3X ² + 4X + 3

II. Lagrange interpolating polynomial : theoretical study

1

Study of an example

2

Existence and uniqueness of the Lagrange interpolating

polynomial

The mathematical problem

Formulation

Let (n + 1) points be given :

(x _i , y _i ) 0≤i≤n with (x _i , y _i ) ∈ R ² for all i, x i 6= x j for all i 6= j.

Is it possible to find a polynomial P with real coefficients satisfying P (x _i ) = y _i ∀0 ≤ i ≤ n?

Degree of P ?

number of equations : n + 1

Þ number of unknowns (coefficients (a i )) less that n + 1

Þ deg P ≤ n

The Lagrange basis

For 0 ≤ j ≤ n, let us define L _j =

n

Y

i=0,i6=j

X − x _i x _j − x _i . It satisfies :

L j (x j ) = 1 and L j (x i ) = 0 for all i 6= j

⇐⇒ L j (x i ) = δ i,j . and

deg L _j = n ∀0 ≤ j ≤ n.

Solution of the problem + uniqueness

Existence of a solution Let P =

n

X

j=0

y _j L _j , we have deg P ≤ n,

P (x _i ) =

n

X

j=0

y _j L _j (x _i ) = y _i for all 0 ≤ i ≤ n.

á P is a solution of the Lagrange interpolation problem.

Uniqueness

Can we find another solution to the problem, Q ? If Q exists, deg Q ≤ n and deg(P − Q) ≤ n,

P (x i ) − Q(x i ) = (P − Q)(x i ) = 0 for 0 ≤ i ≤ n.

á P − Q = 0 and the solution is unique.

Main theorem

Theorem Hypotheses :

Let us consider (n + 1) points of R ² : (x _i , y _i ) 0≤i≤n , x i 6= x j for all i 6= j

Then, there exists a unique polynomial P ∈ R n [X] satisfying P (x _i ) = y _i ∀0 ≤ i ≤ n.

P is the Lagrange interpolating polynomial that passes through the (n + 1) points (x _i , y _i ) 0≤i≤n .

In the case where y i = f (x i ) for all 0 ≤ i ≤ n (with f a given

function), P is the Lagrange interpolating polynomial of f in the

points (x i ) 0≤i≤n .

II. Lagrange interpolating polynomial : theoretical study

1

Study of an example

2

Existence and uniqueness of the Lagrange interpolating polynomial

3

Interpolation error result

Presentation of the problem

Comparison of f and P (4 points)

E(x) = f (x) − P (x)

Presentation of the problem

Comparison of f and P (4 points)

E(x) = f (x) − P (x)

with a zoom around the points

Presentation of the problem

Comparison of f and P (6 points)

E(x) = f (x) − P (x)

Presentation of the problem

Comparison of f and P (6 points)

E(x) = f (x) − P (x)

with a zoom around the points

What can be said about E(x) ? Can it be bounded ?...

Interpolation error

Theorem Hypotheses :

f : [a, b] → R , f ∈ C ⁿ⁺¹ ([a, b]),

(x i ) 0≤i≤n , n + 1 distinct real numbers of [a, b].

P n : Lagrange interpolating polynomial of f in the points (x i ) 0≤i≤n . Then, for all x ∈ [a, b], there exists ξ _x ∈ [a, b] such that

f(x) − P _n (x) = 1

(n + 1)! Π _n (x)f ⁽ⁿ⁺¹⁾ (ξ _x ), with Π _n =

n

Y

i=0

(X − x _i ).

Consequence

As a consequence, we get :

∀x ∈ [a, b] |f (x) − P _n (x)| ≤ 1

(n + 1)! M _n+1 |Π _n (x)|,

with M _n+1 = max

ξ∈[a,b] |f ⁽ⁿ⁺¹⁾ (ξ)|.

It does not imply the convergence of P n (x) towards f(x).

It is not necessary interesting to increase n.

III. Lagrange interpolating polynomial : practical computation

1

Cost of the computation of the interpolating polynomial

With the Lagrange basis

L j (x) =

n

Y

i=0,i6=j

(x − x _i )

n

Y

i=0,i6=j

(x _j − x _i ) ,

for j = 0 to n

Cost of the computation :

(n + 1) ×

2(n − 1) mult. + 1 div.

P (x) =

n

X

j=0

y j L j (x) Þ final cost ≈ 2n ² .

Other main drawback of the method : what happens if we finally want to add one more point (x n+1 , y n+1 ) ?

Þ All must be started again from zero.

With the Lagrange basis

L j (x) =

n

Y

i=0,i6=j

(x − x _i )

n

Y

i=0,i6=j

(x _j − x _i )

, for j = 0 to n

Cost of the computation : (n + 1) ×

2(n − 1) mult. + 1 div.

P (x) =

n

X

j=0

y j L j (x) Þ final cost ≈ 2n ² .

Other main drawback of the method : what happens if we finally want to add one more point (x n+1 , y n+1 ) ?

Þ All must be started again from zero.

With the Lagrange basis

L j (x) =

n

Y

i=0,i6=j

(x − x _i )

n

Y

i=0,i6=j

(x _j − x _i )

, for j = 0 to n

Cost of the computation : (n + 1) ×

2(n − 1) mult. + 1 div.

P (x) =

n

X

j=0

y j L j (x) Þ final cost ≈ 2n ² .

Other main drawback of the method : what happens if we finally want to add one more point (x n+1 , y n+1 ) ?

Þ All must be started again from zero.

With the Lagrange basis

L j (x) =

n

Y

i=0,i6=j

(x − x _i )

n

Y

i=0,i6=j

(x _j − x _i )

, for j = 0 to n

Cost of the computation : (n + 1) ×

2(n − 1) mult. + 1 div.

P (x) =

n

X

j=0

y j L j (x) Þ final cost ≈ 2n ² .

Other main drawback of the method : what happens if we finally want to add one more point (x _n+1 , y _n+1 ) ?

Þ All must be started again from zero.

With an other basis

Idea

Write the polynomial in the basis



1, (X − x 0 ), (X − x 0 )(X − x 1 ), · · · ,

n−1

Y

j=0

(X − x j )



 .

Þ P

n

= α 0 + α 1 (X − x 0 ) + · · · + α n n−1

Y

j=0

(X − x j ).

Now, if we add one point (x n+1 , y n+1 ) , we have : P _n+1 = P _n + α _n+1

n

Y

j=0

(X − x _j )

Þ we just need to calculate α _n+1 to get P _n+1 .

With an other basis

Idea

Write the polynomial in the basis



1, (X − x 0 ), (X − x 0 )(X − x 1 ), · · · ,

n−1

Y

j=0

(X − x j )



 .

Þ P n = α 0 + α 1 (X − x 0 ) + · · · + α n n−1

Y

j=0

(X − x j ).

Now, if we add one point (x n+1 , y n+1 ) , we have : P _n+1 = P _n + α _n+1

n

Y

j=0

(X − x _j )

Þ we just need to calculate α _n+1 to get P _n+1 .

With an other basis

Idea

Write the polynomial in the basis



1, (X − x 0 ), (X − x 0 )(X − x 1 ), · · · ,

n−1

Y

j=0

(X − x j )



 .

Þ P n = α 0 + α 1 (X − x 0 ) + · · · + α n n−1

Y

j=0

(X − x j ).

Now, if we add one point (x n+1 , y n+1 ) , we have : P _n+1 = P _n + α _n+1

n

Y

j=0

(X − x _j )

Þ we just need to calculate α _n+1 to get P _n+1 .

Cost of the computation of P _n (x)

P _n (x) = α ₀ + α ₁ (x − x ₀ ) + · · · + α _n

n−1

Y

j=0

(x − x _j )

= α ₀ + (x − x ₀ )

α ₁ + (x − x ₁ )

α ₂ + · · · .

H¨ orner’s algorithm for the computation of p = P n (x) p ← α _n

for k from n − 1 to 0 p ← α _k + (x − x _k )p end

Cost

n additions + n multiplications

Þ and the computation of the coefficients α _i ?

Cost of the computation of P _n (x)

P _n (x) = α ₀ + α ₁ (x − x ₀ ) + · · · + α _n

n−1

Y

j=0

(x − x _j )

= α ₀ + (x − x ₀ )

α ₁ + (x − x ₁ )

α ₂ + · · · .

H¨ orner’s algorithm for the computation of p = P n (x) p ← α _n

for k from n − 1 to 0 p ← α _k + (x − x _k )p end

Cost

n additions + n multiplications

Þ and the computation of the coefficients α _i ?

III. Lagrange interpolating polynomial : practical computation

1

Cost of the computation of the interpolating polynomial

2

The divided difference method

Calculation of the first α _i

n = 0, 1 first point (x 0 , y 0 ), P 0 = α 0 : P 0 (x 0 ) = y 0 = ⇒ α 0 = y 0

n = 1, + (x ₁ , y ₁ ), P ₁ = y ₀ + α ₁ (X − x ₀ ) : P ₁ (x ₁ ) = y ₁ = ⇒ α ₁ = y ₁ − y ₀

x ₁ − x ₀ n = 2, + (x 2 , y 2 ),

P 2 = y 0 + y 1 − y 0

x 1 − x 0

(X − x 0 ) + α 2 (X − x 0 )(X − x 1 )

P 2 (x 2 ) = y 2 = ⇒ α 2 =

y 2 − y 1

x ₂ − x ₁ − y 1 − y 0

x ₁ − x ₀

x ₂ − x ₀

Recurrence formula

Assume we have :

Interpolating points : (x 0 , y 0 ) · · · (x n−1 , y n−1 ), P n−1 = α 0 +

n−1

X

j=1

α j j−1

Y

k=0

(X − x k ), (α j ) 0≤j≤n−1 known Interpolating points : (x ₁ , y ₁ ) · · · (x _n , y _n ),

Q n−1 = β ₀ +

n−1

X

j=1

β _j

j

Y

k=1

(X − x _k ), (β _j ) 0≤j≤n−1 known Then,

X − x 0

x _n − x ₀ Q n−1 + x n − X x _n − x ₀ P n−1

= P n

and

α _n = 1

x _n − x ₀ β n−1 − 1

x _n − x ₀ α n−1 = β n−1 − α n−1

x _n − x ₀ .

Recurrence formula

Assume we have :

Interpolating points : (x 0 , y 0 ) · · · (x n−1 , y n−1 ), P n−1 = α 0 +

n−1

X

j=1

α j j−1

Y

k=0

(X − x k ), (α j ) 0≤j≤n−1 known Interpolating points : (x ₁ , y ₁ ) · · · (x _n , y _n ),

Q n−1 = β ₀ +

n−1

X

j=1

β _j

j

Y

k=1

(X − x _k ), (β _j ) 0≤j≤n−1 known Then,

X − x 0

x _n − x ₀ Q n−1 + x n − X

x _n − x ₀ P n−1 = P _n

and

α _n = 1

x _n − x ₀ β n−1 − 1

x _n − x ₀ α n−1 = β n−1 − α n−1

x _n − x ₀ .

Recurrence formula

Assume we have :

Interpolating points : (x 0 , y 0 ) · · · (x n−1 , y n−1 ), P n−1 = α 0 +

n−1

X

j=1

α j j−1

Y

k=0

(X − x k ), (α j ) 0≤j≤n−1 known Interpolating points : (x ₁ , y ₁ ) · · · (x _n , y _n ),

Q n−1 = β ₀ +

n−1

X

j=1

β _j

j

Y

k=1

(X − x _k ), (β _j ) 0≤j≤n−1 known Then,

X − x 0

x _n − x ₀ Q n−1 + x n − X

x _n − x ₀ P n−1 = P _n and

α _n = 1

x _n − x ₀ β n−1 − 1

x _n − x ₀ α n−1 = β n−1 − α n−1

x _n − x ₀ .

The divided differences

x 0 f (x 0 ) x ₁ f (x ₁ ) x ₂ f (x ₂ )

.. . .. .

.. . .. . . ..

x n−1 f (x n−1 )

. ..

x n f (x n )

· · · · f [x 0 , · · · , x n ] with

f [x ₀ , · · · , x _n ] = f[x ₁ , · · · , x _n ] − f [x ₀ , · · · , x n−1 ] x n − x 0

.

The divided differences

x 0 f [x 0 ] x ₁ f [x ₁ ] x ₂ f [x ₂ ]

.. . .. .

.. . .. . . ..

x n−1 f [x n−1 ]

. ..

x n f [x n ]

· · · · f [x 0 , · · · , x n ] with

f [x ₀ , · · · , x _n ] = f[x ₁ , · · · , x _n ] − f [x ₀ , · · · , x n−1 ] x n − x 0

.

The divided differences

x ₀ f [x ₀ ]

x 1 f [x 1 ] f [x 1 ] − f [x 0 ] x 1 − x 0

x ₂ f [x ₂ ] f [x 2 ] − f [x 1 ] x ₂ − x ₁

.. . .. . .. .

.. . . ..

x n−1 f [x n−1 ] f[x n−1 ] − f [x n−2 ] x n−1 − x n−2

. ..

x _n f [x _n ] f[x _n ] − f [x n−1 ] x n − x n−1

· · · · f [x ₀ , · · · , x _n ] with

f [x 0 , · · · , x n ] = f[x 1 , · · · , x n ] − f [x 0 , · · · , x n−1 ]

x _n − x ₀ .

The divided differences

x ₀ f [x ₀ ]

x 1 f [x 1 ] f [x 0 , x 1 ] x 2 f [x 2 ] f [x 1 , x 2 ]

.. . .. . .. .

.. . . ..

x n−1 f [x n−1 ] f [x n−2 , x n−1 ]

. ..

x _n f [x _n ] f [x n−1 , x _n ]

· · · · f[x ₀ , · · · , x _n ] with

f [x 0 , · · · , x n ] = f[x 1 , · · · , x n ] − f [x 0 , · · · , x n−1 ] x n − x 0

.

The divided differences

x ₀ f [x ₀ ]

x 1 f [x 1 ] f [x 0 , x 1 ]

x ₂ f [x ₂ ] f [x ₁ , x ₂ ] f [x ₁ , x ₂ ] − f [x ₀ , x ₁ ] x ₂ − x ₀

.. . .. . .. . .. .

. ..

x n−1 f [x n−1 ] f [x n−2 , x n−1 ] f [x n−1 , x n ] − f [x n−2 , x n−1 ] x _n − x n−2

. ..

x _n f [x _n ] f [x n−1 , x _n ] f [x n−1 , x _n ] − f [x n−2 , x n−1 ] x n − x n−2

· · · · f [x ₀ , · · · , x _n ] with

f [x 0 , · · · , x n ] = f[x 1 , · · · , x n ] − f [x 0 , · · · , x n−1 ] x n − x 0

.

The divided differences

x ₀ f [x ₀ ]

x ₁ f [x ₁ ] f [x ₀ , x ₁ ]

x 2 f [x 2 ] f [x 1 , x 2 ] f [x 0 , x 1 , x 2 ]

.. . .. . .. . .. .

. ..

x n−1 f [x n−1 ] f [x n−2 , x n−1 ] f [x n−3 , x n−2 , x n−1 ]

. ..

x n f [x n ] f [x n−1 , x n ] f [x n−2 , x n−1 , x n ]

· · · · f [x 0 , · · · , x n ] with

f [x ₀ , · · · , x _n ] = f[x ₁ , · · · , x _n ] − f [x ₀ , · · · , x n−1 ] x n − x 0

.

The divided differences

x ₀ f [x ₀ ]

x ₁ f [x ₁ ] f [x ₀ , x ₁ ]

x 2 f [x 2 ] f [x 1 , x 2 ] f [x 0 , x 1 , x 2 ]

.. . .. . .. . .. . . ..

x n−1 f [x n−1 ] f [x n−2 , x n−1 ] f [x n−3 , x n−2 , x n−1 ] . ..

x n f [x n ] f [x n−1 , x n ] f [x n−2 , x n−1 , x n ] · · · · f [x 0 , · · · , x n ] with

f[x 0 , · · · , x n ] = f[x ₁ , · · · , x _n ] − f [x ₀ , · · · , x n−1 ] x n − x 0

.

The divided differences

x 0 f [x 0 ]

x ₁ f [x ₁ ] f [x ₀ , x ₁ ]

x ₂ f [x ₂ ] f [x ₁ , x ₂ ] f [x ₀ , x ₁ , x ₂ ]

.. . .. . .. . .. . . ..

x n−1 f [x n−1 ] f [x n−2 , x n−1 ] f [x n−3 , x n−2 , x n−1 ] . ..

x n f [x n ] f [x n−1 , x n ] f [x n−2 , x n−1 , x n ] · · · · f[x 0 , · · · , x n ]

Þ P _n = f[x ₀ ] + f[x ₀ , x ₁ ](X − x ₀ ) + f [x ₀ , x ₁ , x ₂ ](X − x ₀ )(X − x ₁ ) + · · · + f [x ₀ , · · · , x _n ]

n−1

Y

j=0

(X − x _j ).

The divided differences

x ₀ f [x ₀ ]

x ₁ f [x ₁ ] f [x ₀ , x ₁ ]

x 2 f [x 2 ] f [x 1 , x 2 ] f [x 0 , x 1 , x 2 ]

.. . .. . .. . .. . . ..

x n−1 f [x n−1 ] f [x n−2 , x n−1 ] f [x n−3 , x n−2 , x n−1 ] . ..

x n f [x n ] f [x n−1 , x n ] f [x n−2 , x n−1 , x n ] · · · · f[x 0 , · · · , x n ]

Cost of the computation ≈ n ²

2 div. and n ² sub.

Result

Theorem

The Lagrange interpolating polynomial of f in the points (x _i ) 0≤i≤n

reads

P n = f [x 0 ] +

n

X

j=1

f[x 0 , · · · , x j ]

j−1

Y

k=0

(X − x _k ),

where f [ ] denotes the divided difference of f defined by induction f [x _i ] = f (x _i ) for 0 ≤ i ≤ n

f [x i , · · · , x i+k ] = f [x _i+1 , · · · , x _i+k ] − f [x _i , · · · , x _i+k−1 ] x i+k − x i

for 0 ≤ i ≤ n − k, 1 ≤ k ≤ n.

IV. A few words about Hermite interpolation

1

Presentation of the problem

One example

f (x) = sin( πx

2 )(x ² + 3)

Consider two points : (x ₀ , y ₀ ) = (−1, −4) (x ₁ , y ₁ ) = (3, −12)

P = −4 − 2(X + 1) = −6 − 2X Q = P + (X + 1)(X − 3)(αX + β)

= P + (X + 1)(X − 3)(−1)

= −X ² − 3

One example

f (x) = sin( πx

2 )(x ² + 3)

Consider two points : (x ₀ , y ₀ ) = (−1, −4) (x ₁ , y ₁ ) = (3, −12) Search P such that

P (x 0 ) = f (x 0 ), P (x 1 ) = f (x 1 )

P = −4 − 2(X + 1) = −6 − 2X Q = P + (X + 1)(X − 3)(αX + β)

= P + (X + 1)(X − 3)(−1)

= −X ² − 3

One example

f (x) = sin( πx

2 )(x ² + 3)

Consider two points : (x ₀ , y ₀ ) = (−1, −4) (x ₁ , y ₁ ) = (3, −12) Search P such that

P (x 0 ) = f (x 0 ), P (x 1 ) = f (x 1 )

P = −4 − 2(X + 1) = −6 − 2X

Q = P + (X + 1)(X − 3)(αX + β)

= P + (X + 1)(X − 3)(−1)

= −X ² − 3

One example

f (x) = sin( πx

2 )(x ² + 3)

Consider two points : (x 0 , y 0 ) = (−1, −4) (x ₁ , y ₁ ) = (3, −12) Search Q such that

Q(x 0 ) = f (x 0 ), Q(x 1 ) = f (x 1 ) Q ⁰ (x ₀ ) = f ⁰ (x ₀ ), Q ⁰ (x ₁ ) = f ⁰ (x ₁ )

P = −4 − 2(X + 1) = −6 − 2X

Q = P + (X + 1)(X − 3)(αX + β)

= P + (X + 1)(X − 3)(−1)

= −X ² − 3

One example

f (x) = sin( πx

2 )(x ² + 3)

Consider two points : (x 0 , y 0 ) = (−1, −4) (x ₁ , y ₁ ) = (3, −12) Search Q such that

Q(x 0 ) = f (x 0 ), Q(x 1 ) = f (x 1 ) Q ⁰ (x ₀ ) = f ⁰ (x ₀ ), Q ⁰ (x ₁ ) = f ⁰ (x ₁ )

P = −4 − 2(X + 1) = −6 − 2X Q = P + (X + 1)(X − 3)(αX + β)

= P + (X + 1)(X − 3)(−1)

= −X ² − 3

One example

f (x) = sin( πx

2 )(x ² + 3)

Consider two points : (x 0 , y 0 ) = (−1, −4) (x ₁ , y ₁ ) = (3, −12) Search Q such that

Q(x 0 ) = f (x 0 ), Q(x 1 ) = f (x 1 ) Q ⁰ (x ₀ ) = f ⁰ (x ₀ ), Q ⁰ (x ₁ ) = f ⁰ (x ₁ )

P = −4 − 2(X + 1) = −6 − 2X Q = P + (X + 1)(X − 3)(αX + β)

= P + (X + 1)(X − 3)(−1)

= −X ² − 3

With 2 other points

f (x) = sin( πx

2 )(x ² + 3)

P = 0 Q = − π

2 X ³ + π

4 X ² + 3π

2 X

The mathematical problem

Generalities

The Hermite interpolation takes into account

the values of the function in some points (x i ) 0≤i≤k ,

the values of the successive derivatives of the function until order α i in x i .

Formulation

f is a sufficiently smooth function defined on [a, b], x ₀ , . . . , x _k are (k + 1) given points of [a, b],

α ₀ , . . . , α _k are (k + 1) integers.

Is it possible to find P satisfying

∀0 ≤ i ≤ k, P ^(j) (x i ) = f ^(j) (x i ), ∀0 ≤ j ≤ α i ?

IV. A few words about Hermite interpolation

1

Presentation of the problem

2

Main results

Analysis of the problem

Degree of P

Number of equations :

k

X

i=0

(α i + 1) = k + 1 +

k

X

i=0

α i .

Degree : n = k +

k

X

i=0

α _i .

Definition

P is the Hermite interpolating polynomial of f in the

points (x i ) 0≤i≤k with the orders (α i ) 0≤i≤k

Theorem

Theorem : existence and uniqueness + interpolation error Hypotheses :

(x _i ) 0≤i≤k , (k + 1) points in [a, b],

(α _i ) 0≤i≤k , (k + 1) integers, n = k +

k

X

i=0

α _i f : [a, b] → R , f ∈ C ⁿ⁺¹ ([a, b]),

Then, there exists a unique polynomial P _n ∈ R n [X] such that

∀0 ≤ i ≤ k, P _n ^(j) (x _i ) = f ^(j) (x _i ), ∀0 ≤ j ≤ α _i . Furthermore, for all x ∈ [a, b], there exists ξ x ∈ [a, b] such that

f (x) − P _n (x) = 1

(n + 1)! Ω _n (x)f ⁽ⁿ⁺¹⁾ (ξ _x ), with Ω _n =

k

Y

i=0

(X − x _i ) ^α

⁺¹ .

V. Least squares method

1

The case of linear regression

Linear regression

We are looking for a 0 and b 0 such that the following distance is mini- mal :

d(a, b) =

n

X

i=1

(y i − (ax i + b)) ² .

Necessary condition

∂d

∂a (a ₀ , b ₀ ) = 0 and ∂d

∂b (a ₀ , b ₀ ) = 0

⇐⇒



 

 



 

 

 a ₀

n

X

i=1

x ² _i + b ₀

n

X

i=1

x _i =

n

X

i=1

x _i y _i

a 0 n

X

i=1

x i + b 0 n =

n

X

i=1

y i

Existence of a unique candidate (a ₀ , b ₀ )

Matrix of the linear system

A =













n

X

i=1

x ² _i

n

X

i=1

x _i

n

X

i=1

x i n













= n





 1 n

n

X

i=1

x ² _i X ¯ X ¯ 1







Invertibility det A = n ² ( 1

n

n

X

i=1

x ² _i − X ¯ ² ) = n

n

X

i=1

(x i − X) ¯ ² = n ² V(X).

Conclusion

As soon as two x i are different, det A 6= 0 and there exists a unique (a 0 , b 0 ) susceptible to be a minimum of d :

a ₀ = Cov(X, Y )

V(X) and b ₀ = ¯ Y − a ₀ X. ¯

(a ₀ , b ₀ ) is a minimizer of d

After some computations, we prove that d(a, b) − d(a ₀ , b ₀ ) =

n

X

i=1

((a ₀ − a)x _i + b ₀ − b) ² It yields

∀(a, b) ∈ R ² d(a, b) ≥ d(a ₀ , b ₀ ).

Remark on the matrix A

A =













n

X

i=1

x ² _i

n

X

i=1

x i n

X

i=1

x _i n













= B ^T B with B =









 x 1 1 x 2 1 .. . .. . x n 1











V. Least squares method

1

The case of linear regression

2

Generalization

Presentation of the problem

Given points

The cloud of points is still given by (x i ) 1≤i≤n and (y i ) 1≤i≤n . A space of functions

For some independent functions (ϕ ₁ , · · · , ϕ _m ), let us define U = {ϕ; ϕ =

m

X

i=1

u i ϕ i }

Search for a minimizer

We are looking for ϕ ^∗ ∈ U such that

n

X

i=0

|y _i − ϕ ^∗ (x _i )| ² = min

ϕ∈U n

X

i=0

|y _i − ϕ(x _i )| ² .

Main result

Theorem

As soon as two x _i are different, the least squares problem admits a unique solution

ϕ ^∗ =

m

X

i=1

u ^∗ _i ϕ i .

Futhermore, the vector u ^∗ = (u ^∗ ₁ , · · · , u ^∗ _m ) is the unique solution of the linear system

B ^T Bu ^∗ = B ^T y, with

B =







ϕ 1 (x 1 ) · · · ϕ m (x 1 )

.. . .. .

ϕ ₁ (x _n ) · · · ϕ _m (x _n )







Remark

In the linear case, m = 2 and ϕ ₁ (x) = x, ϕ ₂ (x) = 1.