On determining the fundamental matrix : analysis of different methods and experimental results

HAL Id: inria-00074777

https://hal.inria.fr/inria-00074777

Submitted on 4 May 2012

HAL is a multi-disciplinary open access

archive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

On determining the fundamental matrix : analysis of

different methods and experimental results

Quang-Tuan Luong, Rachid Deriche, Olivier Faugeras, Théodore Papadopoulo

To cite this version:

Quang-Tuan Luong, Rachid Deriche, Olivier Faugeras, Théodore Papadopoulo. On determining the fundamental matrix : analysis of different methods and experimental results. [Research Report] RR-1894, INRIA. 1993. �inria-00074777�

UNITE DE RECHERCHE

INRIA-SOPHIAANTIPOLIS

Institut National

_{de Recherche}

en Informatique

et en Automatique

2004 route des Lucioles

_{B.P. 93}

06902 Sophia-Antipolis

_France

Rapports de Recherche

N 1894

Programme 4

Robotique, Image et Vision

ON DETERMINING THE

FUNDAMENTAL MATRIX:

ANALYSIS OF DIFFERENT

METHODS AND

EXPERIMENTAL RESULTS

Quang-Tuan Luong

_{Rachid Deriche}

Olivier Faugeras

Theo Papadopoulo

On Determining the Fundamental Matrix:

Analysis of Dierent Methods

and Experimental Results

Determination de la matrice fondamentale:

Analyse de dierentes methodes

_{et resultats experimentaux}

Quang-Tuan Luong Rachid Deriche Olivier Faugeras Theodore Papadopoulo

INRIA Sophia Antipolis BP 93 - Sophia-Antipolis Cedex

France

Abstract

The Fundamental matrix is a key concept when working with uncalibrated images and multiple viewpoints. It contains all the available geometric information and enables to recover the epipolar geometry from uncalibrated perspective views. This paper ad-dresses the important problem of its robust determination given a number of image point correspondences. We rst dene precisely this matrix, and show clearly how it is related to the epipolar geometry and to the Essential matrix introduced earlier by Longuet-Higgins. In particular, we show that this matrix, dened up to a scale factor, must be of rank two. Dierent parametrizations for this matrix are then proposed to take into account these important constraints and linear and non-linear criteria for its estimation are also considered. We then clearly show that the linear criterion is unable to express the rank and normalization constraints. Using the linear criterion leads de-nitely to the worst result in the determination of the Fundamental matrix. Several examples on real images clearly illustrate and validate this important negative result. To overcome the major weaknesses of the linear criterion, dierent non-linear crite-ria are proposed and analyzed in great detail. Extensive experimental work has been performed in order to compare the dierent methods using a large number of noisy synthetic data and real images. In particular, a statistical method based on variation of camera displacements is used to evaluate the stability and convergence properties of each method.

Keywords:

Motion Analysis, Calibration, Projective Geometry.

Resume

La matrice fondamentale est un concept-cle pour toutes les questions touchant a l'emploi d'images non calibrees prises de points de vue multiples. Elle contient toute l'information geometrique disponible et permet d'obtenir la geometrie epipolaire a par-tir de deux vues perspectives non calibrees. Ce rapport est a propos du probleme im-portant de sa determination robuste a partir d'un certain nombre de correspondances ponctuelles. Nous commencons par denir precisement cette matrice, et par mettre en evidence ses relations avec la geometrie epipolaire et la matrice essentielle, introduite precedemment par Longuet-Higgins. En particulier, nous montrons que cette matrice, denie a un facteur d'echelle, doit ^etre de rang deux. Les techniques lineaires d'estima-tion de la matrice essentielle admettent une extension naturelle qui permet d'eectuer le calcul direct de la matrice fondamentale a partir d'appariements de points, au moyen d'un critere qui est lineaire. Nous montrons que cette methode soure de deux defauts, lies a l'absence de contrainte sur le rang de la matrice recherchee, et a l'absence de normalisation du critere, qui entra^nent des erreurs importantes dans l'estimation de la matrice fondamentale et des epipoles. Cette analyse est validee par plusieurs exemples reels. An de surmonter ces di cultes, plusieurs nouveaux criteres non-lineaires, dont

nous donnons des interpretations en termes de distances, sont ensuite proposes, puis plusieurs parametrisations sont introduites pour rendre compte des contraintes aux-quelles doit satisfaire la matrice fondamentale. Un travail experimental exhaustif est realise a l'aide de nombreuses donnees synthetiques et d'images reelles. En particu-lier, une methode statistique fondee sur la variation des deplacements de la camera est utilisee pour evaluer la stabilite et les proprietes de convergence des dierentes methodes.

Mots-cle:

Analyse du mouvement, calibration, geometrie projective

1 Introduction

Inferring three-dimensional information from images taken from dierent viewpoints is a central problem in computer vision. However, as the measured data in images are just pixel coordinates, there are only two approaches that can be used in order to perform this task:

The rst one is to establish a model which relates pixel coordinates to 3D coor-dinates, and to compute the parameters of such a model. This is done by camera calibration 21] 5], which typically computes the projection matrices

P

, which relates the image coordinates to a world reference frame. However, it is not always possible to assume that cameras can be calibrated o-line, particularly when using active vision systems.

Thus a second approach is emerging, which consists in using projective invariants 16], whose non-metric nature allows to use uncalibrated cameras. Recent work 8] 3] 15] 6] has shown that it is possible to recover the projective structure of a scene from point correspondences only, without the need for camera calibration. It is even possible to use these projective invariants to compute the camera calibration 4] 14]. These approaches use only geometric information which relates the dierent viewpoints. This information is entirely contained in the Fundamental matrix, thus it is very important to develop precise techniques to compute it.

In spite of the fact that there has been some confusion between the fundamental matrix and Longuet-Higgins essential matrix, it is now known that the fundamental matrix can be computed from pixel coordinates of corresponding points. Line corres-pondences are not su cient with two views. Another approach is to use linear lters tuned to a range of orientations and scales. Jones and Malik 9] have shown that it is also possible in this framework to recover the location of epipolar lines. The compu-tation technique used by most of the authors 6] 18] 20] is just a linear one, which generalizes the eight-point algorithm of Longuet-Higgins13]. After a rst part where we clarify the concept of Fundamental matrix, we show that this computation tech-nique suers from two majors intrinsic drawbacks. Analyzing these drawbacks enables us to introduce a new, non-linear computation technique, based on criteria that have a nice interpretation in terms of distances. We then show, using both large sets of simu-lations and real data, that our non-linear computation techniques provide signicant improvement in the accuracy of the Fundamental matrix determination.

2 The Fundamental Matrix

2.1 The projective model

The camera model which is most widely used is the pinhole: the camera is supposed to perform a perfect perspective transformation of 3D space on a retinal plane. In the general case, we must also account for a change of world coordinates, as well as for a change of retinal coordinates, so that a generalization of the previous assumption is that the camera performs a projective linear transformation, rather than a mere perspective transformation. The pixel coordinatesuand vare the only information we

have if the camera is not calibrated:

q

= 2 6 4 su sv s 3 7 5=

A

2 6 4 1 0 0 0 0 1 0 0 0 0 1 0 3 7 5

G

2 6 6 6 4 X Y Z 1 3 7 7 7 5

PM

(1)

whereX,Y,Z are world coordinates,

A

is a 33 transformation matrix accounting

for camera sampling and optical characteristics and

G

is a 44 displacement matrix

accounting for camera position and orientation. If the camera is calibrated, then

A

is known and it is possible to use normalized coordinates

m

=

A

;1

q

, which have a direct

3D interpretation.

2.2 The epipolar geometry and the Fundamental matrix

The epipolar geometry is the basic constraint which arises from the existence of two viewpoints. Let a camera take two images by linear projection from two dierent lo-cations, as shown in gure 1. Let

C

be the optical center of the camera when the rst image is obtained, and let

C

0 be the optical center for the second image. The line h

C



C

0

i projects to a point

e

in the rst image R

1 , and to a point

e

0 in the second imageR 2 . The points

e

,

e

0

are the epipoles. The lines through

e

in the rst image and the lines through

e

0

in the second image are the epipolar lines. The epipolar constraint is well-known in stereovision: for each point

m

in the rst retina, its corresponding point

m

0lies on its epipolar line

l

0

m. C’ C e e’ m’ m l’m lm’ R R’ Π M

Figure 1: The epipolar geometry

Let us now use retinal coordinates. The relationship between a point

q

and its corresponding epipolar line

l

0

q is projective linear, because the relations between

q

and

h

C



M

i, and

q

and h

C



M

i and its projection

l

0

q are both projective linear. We call

the 33 matrix

F

which describes this correspondence the fundamental matrix. The

importance of the fundamental matrix has been neglected in the literature, as almost all the work on motion has been done under the assumption that intrinsic parameters

are known. In that case, the fundamental matrix reduces to an essential matrix. But if one wants to proceed only from image measurements, the fundamental matrix is the key concept, as it contains the all the geometrical information relating two dierent images.

2.3 Relation with Longuet-Higgins equation

The Longuet-Higgins equation 13], applies when using normalized coordinates, and thus calibrated cameras. If the motion between the two positions of the cameras are given by the rotation matrix

R

and the translation matrix

t

, and if

m

and

m

0 are

corresponding points, then the coplanarity constraint relating

Cm

0,

t

, and

Cm

is

written as:

m

0

(

t



Rm

)

m

0T

Em

= 0 (2)

The matrix

E

, which is the product of an orthogonal matrix and an antisymmetric matrix is called an essential matrix. Because of the depth/speed ambiguity,

E

depends on ve parameters only.

Let us now express the epipolar constraint using the fundamental matrix, in the case of uncalibrated cameras. For a given point

q

in the rst image, the projective representation

l

0

q of its the epipolar line in the second image is given by

l

0

q =

Fq

Since the point

q

0

corresponding to

q

belongs to the line

l

0

q by denition, it follows

that

q

0T

Fq

= 0 (3)

It can be seen that the two equations (2) and (3) are equivalent, and that we have the relation:

F

=

A

;1T

EA

;1

Unlike the essential matrix, which is characterized by the two constraints found by Huang and Faugeras 7] which are the nullity of the determinant and the equality of the two non-zero singular values, the only property of the fundamental matrix is that it is of rank two. As it is also dened only up to a scale factor, the number of independent coe cients of

F

is seven.

2.4 Relation with the epipolar transformation

The epipolar transformation is a homography between the epipolar lines in the rst image and the epipolar lines in the second image, dened as follows. Let  be any plane containing h

C



C

0

i. Then  projects to an epipolar line l in the rst image and to an

epipolar line l 0

in the second image. The correspondences ^l and ^l 0

are homo-graphies between the two pencils of epipolar lines and the pencil of planes containing

h

C



C

0

i. It follows that the correspondancel^l 0

is a homography. In the practical case where epipoles are at nite distance, the epipolar transformation is characterized by

the ane coordinates of the epipoles

e

and

e

0and by the coe cients of the homography

between the two pencils of epipolar lines, each line being parameterized by its direction:

 7! 0 = a+b c+d (4) where  = q 2 ;e 2 q 1 ;e 1  0= q 0 2 ;e 0 2 q 0 1 ;e 0 1 (5) and

q

$

q

0, is a pair of corresponding points. It follows that the epipolar

transforma-tion, like the fundamental matrix depends on seven independent parameters.

On identifying the equation (3) with the constraint on epipolar lines obtained by making the substitutions (5) in (4), expressions are obtained for the coe cients of

F

in terms of the parameters describing the epipoles and the homography:

F 11 = be 3 e 0 3 (6) F 12 = ae 3 e 0 3 F 13 = ;ae 2 e 0 3 ;be 1 e 0 3 F 21 = ;de 0 3 e 3 F 22 = ;ce 0 3 e 3 F 23 = ce 0 3 e 2+ de 0 3 e 1 F 31 = de 0 2 e 3 ;be 3 e 0 1 F 32 = ce 0 2 e 3 ;ae 3 e 0 1 F 33 = ;ce 0 2 e 2 ;de 0 2 e 1+ ae 2 e 0 1+ be 1 e 0 1

From these relations, it is easy to see that

F

is dened only up to a scale factor. Let

c

1,

c

2,

c

3 be the columns of

F

. It follows from (6) that e 1

c

1 + e 2

c

2+ e 3

c

3 = 0. The

rank of

F

is thus at most two. The equations (6), yield the epipolar transformation as a function of the fundamental matrix:

a = F 12 (7) b = F 11 c = ;F 22 d = ;F 21 e 1 = F 23 F 12 ;F 22 F 13 F 22 F 11 ;F 21 F 12 e 3 e 2 = F 13 F 21 ;F 11 F 23 F 22 F 11 ;F 21 F 12 e 3 e 0 1 = F 32 F 21 ;F 22 F 31 F 22 F 11 ;F 21 F 12 e 0 3 e 0 2 = F 31 F 12 ;F 11 F 32 F 22 F 11 ;F 21 F 12 e 0 3

The determinant ad;bc of the homography is F 22 F 11 ;F 21 F

12. In the case of nite

epipoles, it is not null. The interpretation of equations (7) is simple: the coordinates

of

e

(resp.

e

0) are the vectors of the kernel of

F

(resp.

F

T). Writing 

0 as a function

of from the relation

y

0 1

Fy

1= 0 which arises from the correspondence of the points

at innity

y

1 = (1 0)T et

y

0 1 = (1  0

0)T, of corresponding lines, we obtain the

homographic relation.

3 The linear criterion

3.1 The eight point algorithm

Equation (3) can be written:

U

T

_f

_{= 0 (8)} where:

U

= uu 0 vu 0 u 0 uv 0 vv 0 v 0 uv1]

f

= F 11 F 12 F 13 F 21 F 22 F 23 F 31 F 32 F 33]

Equation (8) is linear and homogeneous in the 9 unknown coe cients of matrix

F

. Thus we know that if we are given 8 matches we will be able, in general, to determine a unique solution for

F

, dened up to a scale factor. This approach, known as the eight point algorithm, was introduced by Longuet-Higgins 13] and has been extensively studied in the literature 12] 22] 2] 23] 11], for the computation of the Essential matrix. It has proven to be very sensitive to noise. Our contribution is to study it in the more general framework of Fundamental matrix computation. Some recent work has indeed pointed out that it is also relevant for the purpose of working from uncalibrated cameras 18], 4] 6]. In this framework, we obtain new results about the accuracy of this criterion, which will enable us to present a more robust approach.

3.2 Implementations

In practice, we are given much more than 8 matches and we use a least-squares method to solve: min F X i (

q

0 Ti

Fq

i)2 (9)

which can be rewritten as:

min f k

Uf

~ k 2 where: ~

U

= 2 6 4

U

T 1 ...

U

Tn 3 7 5

We have tried dierent implementations. The rst one (

M-C

) uses a closed-form solu-tion via the linear equasolu-tions. One of the coe cients of

F

must be set to 1. The second one solves the classical problem:

min

f

k

Uf

~ k with k

f

k= 1 (10)

The solution is the eigenvector associated to the smallest eigenvalue of ~

U

T

_U

_{~, which}

we compute directly (

DIAG

), or using a singular value decomposition (

SVD

). The advantage of this second approach is that all the coe cients of

F

play the same role. We have also tried to normalize the projective coordinates to use the Kanatani N-vectors representation 10] (

DIAG-N

).

The advantage of the linear criterion is that it leads to a non-iterative computation method, however, we have found that it is quite sensitive to noise, even with numerous data points. The two main reasons for this are:

The constraint det(

F

) = 0 is not satised, which causes inconsistencies of the

epipolar geometry near the epipoles.

The criterion is not normalized, which causes a bias in the localization of the

epipoles.

3.3 The linear criterion cannot express the rank

cons-traint

Let l

0 be an epipolar line in the second image, computed from a fundamental matrix

F

that was obtained by the linear criterion, and from the point

m

= (uv1)T of the

rst image. We can express

m

using the epipole in the rst image, and the horizontal and vertical distances from this epipole, x and y. A projective representation forl

0 is:

l

0=

Fm

=

F

0 B @ e 1 ;x e 2 ;y 1 1 C A=

Fe

;

F

0 B @ x y 0 1 C A | {z } l1 (11) If det(

F

) = 0, the epipole

e

satises exactly

Fe

= 0, thus the last expression simplies to

l

1. It is easy to see that it denes an epipolar line which goes through the epipole

e

0 in the second image. If the determinant is not exactly zero, we see that

l

0 is the

sum of a constant vector

r

=

Fe

which should be zero but is not, and of the vector

l

1,

whose norm is bounded by p x

2+ y

2

k

F

k. We can conclude that when (xy) ! (00)

(

m

!

e

), the epipolar line of

m

in the second image converges towards a xed line

represented by

r

, which is inconsistent with the notion of epipolar geometry. We can also see that the smallerp

x 2+

y

2 is (ie the closer

m

is to the epipole), the bigger will

be the error on its associated epipolar line.

We can make these remarks more precise by introducing an Euclidean distance. If the coordinates of the point

p

are (x

0 y 0), and if l 1 x+l 2 y+l 3= 0 is the equation of

the line l, then the distance of the point

p

to the line lis: d(

p

l) = jl 1 x 0+ l 2 y 0+ l 3 j q l 2 1+ l 2 2 (12) The distance of the epipolar line l

0(

m

) given by (11) to the epipole

e

0= ( e 0 1 e 0 2 1)T is thus: d(

e

0 l 0) = jr 1 e 0 1+ r 2 e 0 2+ r 3 ;(F 11 e 0 1+ F 21 e 0 2+1) x;(F 12 e 0 1+ F 22 e 0 2+1) yj p (r 1 ;F 11 x;F 12 y) 2+ ( r 2 ;F 21 x;F 22 y) (13)

9

It is clear that when (xy)! (00),d(

e

0 l 0) ! r1e 0 1 +r 2e 0 2 +r 3 p r2 1 +r 2 2

, which is a generally a big value.

We now give a real example to illustrate these remarks. The images and the matched points are the ones of gure 8. The values of the residual vectors

r

=

Fe

and

r

0=

F

T

e

0

are:

r

= (:0000197:0000274:630 10 ;7 )T

r

0 = (:0000203:0000236:152 10 ;6 )T They seem very low, as kr k = :000033, however this is to be compared with the

residuals found by the non-linear criterions presented later, whose typical values are

kr k=:2 10

;7. The gure 2 shows a plot of the error function (13), versus the distances x and y. Units are pixels. We can see that there is a very sharp peak near the point

(xy) = 0, which represents the epipole

e

, and that the error decreases and converges

to a small value. We can conclude that if the epipole is in the image, the epipolar geometry described by the fundamental matrix obtained from the linear criterion will be inaccurate. -40 0 40 x -40 0 40 y 200 400 600

Figure 2: Distances of epipolar lines to the epipole, linear criterion

This problem can be observed directly in the images shown in the experimental part, in gure 10 for the intersection of epipolar lines, and in gure 11, for the inconsistency of epipolar geometry near the epipoles.

3.4 The linear criterion su ers from lack of

normaliza-tion

Let us now give a geometrical interpretation of the criterion (9). Using again (12), the Euclidean distance of the point

q

0 of the second image to the epipolar line

l

0 =

(l 0 1 l 0 2 l 0 3)

T ₌

_Fq

_{of the corresponding point}

_q

_{of the rst image is:}

d(

q

0 

l

0 ) = j

q

0T

l

0 j q (l 0 1) 2+ ( l 0 2) 2 (14) We note that this expression is always valid as the normalizing termk=

q (l 0 1) 2+ ( l 0 2) 2

is null only in the degenerate cases where the epipolar line is at innity. The criterion (9) can be written: X i k 2 id 2 (

q

0 i

l

0 i) (15)

This interpretation shows that a geometrically signicant quantity in the linear cri-terion is the distance of a point to the epipolar line of its corresponding point. This quantity is weighted by the coe cients k, dened above.

To see why it can introduce a bias, let us rst consider the case where the displa-cement is a pure translation. The fundamental matrix is antisymmetric and has the

form: 2 6 4 0 1 ;y ;1 0 x y ;x 0 3 7 5

where (xy1)T are the coordinates of the epipoles, which are the same in the two

images. If (uivi1)T are the coordinates of the point

q

i in the rst image, then the

normalizing factor is k 2 i =  2(( y ;vi) 2 + ( x;ui) 2), where  is a constant. When

minimizing the criterion (15), we will minimize both ki and d 2(

q

0

i

l

0

i). But minimizing

ki is the same than privileging the fundamental matrices which yield epipoles near the

image. Experimental results show that it is indeed the case. A rst example is given by the images already used: we can see that the epipoles found by the linear criterion, which are at position:

e

= (403:90603:241)T

e

0= (408

:38524:631)T

are nearer than the ones found by the non-linear criterion presented latter, as the epipolar lines obtained from the non-linear criterion are almost parallel in the images, as can be seen in gure 8. A second example is given by table 1.

In the general case, the normalizing factor is:

ki= (a(y;vi) +b(x;ui)) 2+ (

c(y;vi) +d(x;ui)) 2

To see simply its eect on the minimization, let suppose that the coe cients of the homography are xed. By computing the partial derivatives @ki

@x and @ki

@y, it is easy to

see that the minimum is obtained forx=ui andy=vi. Thus the previous observations

apply too. We can conclude that the linear criterion shifts epipoles towards the image center.

We can notice that the situation is particularly bad with the linear criterion, due to the combination of our two observations: whereas the closer the epipoles are to the images, the less accurate will be the epipolar geometry, the epipoles tend to be shifted towards the image center.

Table 1: An example to illustrate the behaviour of the linear criterion when the displacement

is a translation

R

=

I t

= 0 100 100]

image noise (pixel) coordinates of the epipoles

ex ey e 0 x e 0 y 0 246.09 1199.34 246.09 1199.34 0.1 246.09 1193.49 245.06 1193.68 0.5 239.63 1027.16 253.20 1027.23 1.0 237.22 758.55 263.33 767.01 2.0 242.54 544.70 263.01 568.16

4 Non-Linear criteria

4.1 The distance to epipolar lines

We now introduce a rst non-linear approach, based on the geometric interpretation of criterion (9) given in 3.4. The rst idea is to use a non-linear criterion, minimizing:

X

i d 2(

q

0

i

Fq

i)

However, unlike the case of the linear criterion, the two images do not play a symmetric role, as the criterion determines only the epipolar lines in the second image, and should not be used to obtain the epipole in the rst image. We would have to exchange the role of

q

i and

q

0

i to do so. The problem with this approach is the inconsistency of

the epipolar geometry between the two images. To make this more precise, if

F

is computed by minimizing P id 2(

q

0 i

Fq

i) and

F

0 by minimizing P id 2(

q

i

F

0

q

0 i), there

is no warranty that the points of the epipolar line

Fq

dierent from

q

0 correspond

to the points of the epipolar line

F

0

q

0. This remark is illustrated by gure 3. The

"corresponding" epipolar lines do not correspond at all except on the last column of the grid, where they were dened.

To obtain a consistent epipolar geometry, it is necessary and su cient that by exchanging the two images, the fundamental matrix is changed to its transpose. This yields the following criterion, which operates simultaneously in the two images:

X i ( d 2(

q

0 i

Fq

i) +d 2(

q

i

F

T

_q

0 i))

and can be written, using (14) and the fact that

q

0

Ti

Fq

i=

q

Ti

F

T

q

0 i: X i 1 (

Fq

i)2 1+ (

Fq

i) 2 2 +₍

_F

_T

_q

0 1 i)2 1+ (

F

T

_q

0 i)2 2 ! (

q

0 Ti

Fq

i)2 (16)

This criterion is also clearly normalized in the sense that it does not depend on the scale factor used to compute

F

.

Figure 3: An example of inconsistent epipolar geometry, obtained by independent search in

each image

4.2 The Gradient criterion

Taking into account uncertainty Pixels are measured with some uncertainty.

When minimizing the expression (9), we have a sum of termsCi=

q

0

Ti

Fq

i which have

dierent variances. It is natural to weight them so that the contribution of each of these terms to the total criterion will be inversely proportional to its variance. The variance of Ci is given as a function of the variance of the points

q

i et

q

0 i by:  2 Ci = h @CT i @q i @CT i @q 0 i i "



q i

0

0 

q 0 i #" @Ci @q i @Ci @q 0 i # (17) where #q i and # q 0

i are the covariance matrices of the points

q

et

q

0, respectively. These

points are uncorrelated as they are measured in dierent images. We make the classical assumption that their covariance is isotropic and uniform, that is:

#q i = #q 0 i = "  0 0  #

The equation (17) reduces to:

 2 Ci =  2 krCik 2

where rCi denotes the gradient of Ci with respect to the four-dimensional vector

(uiviu 0

iv 0

i)T built from the a ne coordinates of the points

q

i and

q

0

i. Thus: rCi= ((

F

T

q

0 i)1 (

F

T

q

0 i)2 (

Fq

i) 1 (

Fq

i) 2) T

13

We obtain the following criterion, which is also normalized: X i (

q

0 Ti

Fq

i)2 (

Fq

i)2 1+ (

Fq

i) 2 2+ (

F

T

q

0 i)2 1+ (

F

T

q

0 i)2 2 (18) We can note that there is a great similarity between this criterion and the distance criterion (16). Each of its terms has the form 1

k2 +k

0 2

C, whereas the rst one has terms

(1 k2 + 1 k0 2) C.

An interpretationas a distance We can also consider the problem of the

com-puting the fundamental matrix from the denition (3) in the general framework of surface tting, The surface S is modeled by the implicit equation g(

x



f

) = 0, where

f

is the sought parameter vector describing the surface which best ts the data points

x

i. The goal is to minimize a quantityP

id(

x

iS)

2, where

dis a distance. In our case,

the data points are the vectors

x

i = (uiviu 0

iv 0

i),

f

is one of the 7 dimensional

pa-rameterizations introduced in the previous section,, and g is given by (3). The linear

criterion can be considered as a generalization of the Bookstein distance 1] for conic tting. The straightforward idea is to approximate the true distance of the point

x

to the surface by the numberg(

x



f

), in order to get a closed-form solution. A more

pre-cise approximation has been introduced by Sampson 19]. It is based on the rst-order approximation: g(

x

)'g(

x

0) + (

x

;

x

0) rg(

x

) =g(

x

0) + k

x

;

x

0 kkrg(

x

)kcos(

x

;

x

0 rg(

x

)) If

x

0 is the point of

S which is the nearest from

x

, we have the two propertiesg(

x

0) = 0

and cos(

x

;

x

0

rg(

x

0)) = 1. If we make the further rst-order approximation that the

gradient has the same direction at

x

and at

x

0: cos(

x

;

x

0 rg(

x

0)) 'cos(

x

;

x

0 rg(

x

)), we get: d(

x

S) =k

x

;

x

0 k' g(

x

) krg(

x

)k

Il is now obvious that the criterion (18) can be written:P

id(

x

iS) 2.

It would be possible to use a second-order approximation such as the one introduced by Nalwa and Pauchon 17], however the experimental results presented in the next section show that it would not be very useful practically. We thus prefer to consider, for theoretical study, the exact distance which is now presented.

4.3 The \Euclidean" criterion

Experience with conic tting shows that when the data points are not well distributed along the conic on which they lie, the tting method using the rst order approximation of the Euclidean distance of a point to the conic gives results that are somewhat dierent of those obtained when using a full (i.e. not approximated) Euclidean distance. This is, indeed, what happens with the surface tting scheme dened in the previous paragraph : the data points are 4-D vectors

x

i = (uiviu

0

iv 0

i)T whose components

are image coordinates$ since retinas have a nite extent and since the hyper-surface

S is not bounded

1, the measures of the surface points cover only a small part of the

real \underlying" surface. This can also be seen as the following fact : estimating the fundamental matrix is also estimating the epipoles, so it involves the estimation of entities (the epipoles) that are very often far from the image space. Therefore, it seems interesting to develop a criterion based on the Euclidean distance from a 4-D point

x

i to the surfaceS in order to check if the results are noticeably dierent from those

obtained when using the gradient criterion.

Fitting a quadratic hyper-surface The hyper-surface S dened by the

equa-tion (3) in the spaceR 1

R

2(the cyclopean retina) is quadratic. Moreover, all epipolar

lines are on this hyper-surface. Let us notel 0

uivi the epipolar line in R 2 corresponding to the point (uivi)2R 1 and lu 0 iv 0

i the epipolar line in R

1 corresponding to the point

(u 0 iv 0 i)2R 2.

The computation of the 4-D Euclidean distance of a point to S relies on the fact

that :

The 4-D lines dened by u=uiv=vi(u 0 v 0) 2l 0 uivi and u 0= u 0 iv 0= v 0 i(uv)2 lu 0 iv 0 i are subsets of

S. Thus S is a ruled surface that can be parametrized by each of

these two family of lines2. This property is nothing more than writing equation (3) but

it gives us these two important parametrizations.

For example, let us parametrize S using the rst family of lines.

Every point of the surface can be represented by

q

0 = ( u

0 v

0) and a point of the

line l 0 q 0 = l 0

u0v0, so the distance of a point (

q



q

0) = ( uvu

0 v

0) to the surface is given

by the minimum of : d 2 (

q

0 

q

) +d 2 (

q

0 l 0 q 0)

when

q

0 describes the space R

2.

Thus the estimation of

F

leads to the following minimization : min F X i minq 0 fd 2 (

q

0 

q

i) +d 2 (

q

0 iL 0 q 0) g

As the previous methods, this criterion does no depend on the scale factor applied to

F

.

5 Parameterizations of the Fundamental

Ma-trix

5.1 A matrix dened up to a scale factor

The most natural idea to take into account the fact that

F

is dened only up to a scale factor is to x one of the coe cients to 1 (only the linear criterion allows us to use in

1A point ( u i v i) in the rst retina R

1may have its corresponding point that lies at innity in the second

retinaR 2

2From the point of view of this property the best 3-D analogy is the hyperboloid of one sheet

a simple manner another normalization, namely k

F

k). It yields a parameterization of

F

by eight values, which are the ratio of the eight other coe cients to the normalizing one.

In practice, the choice of the normalizing coe cient has signicant numerical con-sequences. As we can see from the expressions of the criteria previously introduced (16) and (18), the non-linear criteria take the general form:

Q 1( F 11 F 12 F 13 F 21 F 22 F 23 F 31 F 32 F 33) Q 2( F 11 F 12 F 13 F 21 F 22 F 23) whereQ 1and Q

2are quadratic forms which have null values at the origin. A well-known

consequence is that the functionQ 1

=Q

2is not regular near the origin. As the derivatives

are used in the course of the minimization procedure, this will induce unstability. As a consequence, we have to choose as normalizing coe cients one of the six rst one, as only these coe cients appear in the expression ofQ

2. Fixing the value of one of these

coe cients to one preventsQ

2 from getting near the origin.

We have established using covariance analysis that the choices are not equivalent when the order of magnitude of the dierent coe cients of

F

is dierent. The best results are theoretically obtained when normalizing with the biggest coe cients. We found in our experiments this observation to be generally true. However, as some cases of divergence during the minimization process sometimes appear, the best is to try several normalizations

We note that as the matrices which are used to initialize the non-linear search are not, in general, singular, we have to compute rst the closest singular matrix, and then the parameterization. In that case, we cannot use formulas (6), thus the epipole

e

is determined by solving the following classical constrained minimization problem

mine k

Fe

k 2 subject to k

e

k 2 = 1

which yields

e

as the unit norm eigenvector of matrix

F

T

_F

_{corresponding to the smallest}

eigenvalue. The same processing applies in reverse to the computation of the epipolee 0.

The epipolar transformation can then be obtained by a linear least-squares procedure, using equations (5) and (4).

5.2 A singular matrix

As seen in part 3, the drawback of the previous method is that we do not take into account the fact that the rank of

F

is only two, and that

F

thus depends on only 7 parameters. We have rst tried to use minimizations under the constraint det(

F

) = 0, which is a cubic polynomial in the coe cients of

F

. The numerical implementations were not e cient and accurate at all.

Thanks to a suggestion by Luc Robert, we can express the same constraint with an unconstrained minimization: the idea is to write matrix

F

as:

F

= 0 B @ a 1 a 2 a 3 a 4 a 5 a 6 a 7 a 1+ a 8 a 4 a 7 a 2+ a 8 a 5 a 7 a 3+ a 8 a 6 1 C A (19)

16

The fact that the third line is a linear combination of the two rst lines ensures that

F

is singular. Chosing such a representation allows us to represent

F

by the right number of parameters, once the normalization is done. A non-linear procedure is required, but it is not a drawback, as the criteria presented in section 4 are already non-linear.

5.3 A fundamental matrix with nite epipoles

The previous representation takes into account only the fact that

F

is singular. We can use the fact it is a fundamental matrix to parameterize it by the values that are of signicance for us. Using the formulas (6) yield:

F

= 0 B @ b a ;ay;bx ;d ;c cy+dx dy 0 ;bx 0 cy 0 ;ax 0 ;cyy 0 ;dy 0 x+ayx 0+ bxx 0 1 C A (20)

The parameters that we use are the a ne coordinates (xy) and (x 0

y

0) of the two

epipoles, and three of the four homography coe cients, which are the coe cients of the submatrix 22 obtained by suppressing the third line and the third column. We

normalize by the biggest of them. The initial parameters are obtained by computing the epipoles and the epipolar transformation by the approximations introduced in 5.1.

6 An experimental comparison

We have presented an approach to the computation of the fundamental matrix which involves several parameterizations and several criteria. The goal of this part is to pro-vide a statistical comparison of the dierent combinations.

6.1 The method

An important remark is that if we want to make a precise assessment of the performance of any method, we have to change not only the image noise, as it is often done, but also the displacements. Dierent displacements will give rise to congurations with stability properties that are very dierent.

We start from 3D points that are randomly scattered in a cube, and from a projec-tion matrix

P

. All these values are chosen to be realistic. Each trial consists of:

Take a random rigid displacement

D

, Compute the exact fundamental matrix

F

0 from

D

and

P

, Compute the projection matrix

P

0 from

D

and

P

,

Project the 3D points in the two 512512 retinas using

P

and

P

0, Add Gaussian noise to the image points,

Solve for the fundamental matrix

F

,

Compute the relative distance of the epipoles from

F

and those from

F

0.

We measure the error by the relative distance, for each coordinate of the epipole: minf jx;x 0 j min(jxjjx 0 j) 1g

It should be noted that using relative errors on the coe cients of

F

, is less appropriate, as the thing we are interested in is actually the correct position of the epipoles. We will also see later that using the value of the minimized criterion as a measure of the error is not appropriate at all: a very coherent epipolar geometry can be observed with completely misplaced epipoles. As our experimentations have shown that the average errors on the four coordinates are always coherent, we will take the mean of these four values as an error measure.

6.2 The linear criteria

We have compared the dierent implementations of the linear criterion, in the table 6.2. Each entry of the table represents the average relative distance of the results obtained by the two methods represented by the vertical entry and by the horizontal one. The abbreviations are dened in the section on the linear criterion. Conclusions are:

noise relative distances

SVD DIAG M-C DIAG-N

0.2 pixel

EXACT

0.1598 0.1543 0.1550 0.1828

SVD

0.0544 0.0543 0.1511

DIAG

0.0034 0.1375

M-C

0.1380 1 pixel

EXACT

0.4623 0.4590 0.4590 0.4929

SVD

0.1077 0.1089 0.3535

DIAG

0.0040 0.3486

M-C

0.3488 1.8 pixel

EXACT

0.6375 0.6334 0.6336 0.6587

SVD

0.1442 0.1444 0.4575

DIAG

0.0055 0.4465

M-C

0.4465

Table 2: Comparisons of the linear criteria

The normalization of projective coordinates leads to the worse results The two methods

DIAG

and

M-C

are very similar

The dierence between the st three criterions is not signicant, in comparison

with the absolute errors, which is normal as the theoretical minimum is unique.

6.3 Non-linear criteria

We have not studied extensively the Euclidean distance criterion, due to the time required for its minimization, which is several hours. However, we have found that it gives results close to, and often more precise than the ones given by the Gradient criterion. There are two dierent parameterizations, that were presented in section 5, and two dierent non-linear criteria, presented in section 4. The abbreviations for the four resulting combinations that we studied are in table 3. We have tried several minimization procedures, including material from Numerical Recipes, and programs from the NAG library.

Table 3: Non-linear methods for the computation of the fundamental matrix

abbrev. criterion parameterization

LIN

linear normalization byk

F

k

DIST-L

distance to epipolar lines 16 singular matrix 19

DIST-T

distance to epipolar lines epipolar transformation 20

GRAD-L

weighting by the gradient 18 singular matrix

GRAD-T

weighting by the gradient epipolar transformation The comparison we have done is threefold:

1. The stability of the minimum corresponding to the exact solution. When noise is present, the surface which represents the value of the criterion as a function of the parameters gets distorted, thus the coordinates of the minimum change. A measure of this variation is given by the distance between the exact epipole and the one obtained when starting the minimization with the exact epipole (gure 4). 2. The convergence properties. The question is whether it is possible to obtain a correct result starting from a plausible initialization, the matrix obtained from the linear criterion. We thus measure the distance between the exact epipole and the one obtained when starting the minimization with the linear solution (gure 5), and the distance between the epipole obtained when starting the minimization with the exact epipole and the one obtained when starting the minimization with the linear solution (gure 6) .

3. The stability of the criterion. When the surface which represents the value of the criterion as a function of the parameters gets distorted, the values of the criterion at local minima corresponding to inexact solutions can become weaker than the value of the criterion at the correct minimum (gure 7).

The conclusions are:

The non-linear criteria are always better than the linear criterion. When starting

a non-linear computation with the result of the linear computation, we always im-prove the precision of the result, even if the noise is not important. The dierence increases with the noise.

Relative distance

Image noise (pixels)

0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 2.2 2.4 2.6 0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 LIN DIST-L DIST-T GRAD-L GRAD-T

Figure 4: Relative distances obtained starting from the exact values

Relative distance

Image noise (pixels)

0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 2.2 2.4 2.6 0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 LIN DIST-L DIST-T GRAD-L GRAD-T

Figure 5: Relative distances obtained starting from the values found by the linear criterion

20

Relative distance

Image noise (pixels)

0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 2.2 2.4 2.6 0.00 0.10 0.20 0.30 0.40 0.50 DIST-L DIST-T GRAD-L GRAD-T

Figure 6: Relative distances obtained between results of the two dierent initializations

Number of false minima

Image noise (pixels)

0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 2.2 2.4 2.6 0 5 10 15 20 DIST-L DIST-T GRAD-L GRAD-T

Figure 7: Number of false minima

21

The dierence due to the choice of the criterion (

DIST

or

GRAD

) is much less

signicant than the one due to the choice of the parameterization (

L

or

T

).

The parameterization

T

yields more stable minima than the parameterization

L

,

as seen in gure 4.

However, the criterion obtained with parameterization

T

has worse convergence

and stability properties than the parameterization

L

, as seen in gures 6 and 7

As a consequence, when starting from the results of the linear criterion, the results

of the four non-linear combinations are roughly equivalent, the results obtained with the parameterization

L

and the criterion

DIST

being slightly better, as seen in gure 5.

The computation is quite sensitive to pixel noise: a Gaussian noise of variance 1

pixel yields a relative error which is about 30%.

6.4 Real data

We now illustrate the remarks made in section 3 with a pair of images. It can be seen in gure 8 that the pencils of epipolar lines obtained with the linear criterion, and those obtained with the non-linear criterion are very dierent. The epipoles obtained with the non-linear criterion are much further away. It seems at rst that if one considers a point that was used in the computation, its epipolar line lies very close to its corres-ponding point. However, the zoom of gure 9 shows that the t is signicantly better with the non-linear criterion. Figure 10 shows a set of epipolar lines obtained from the linear criterion, we can see that they don't meet exactly at a point, whereas they do by construction for the non-linear criterion. A consequence is illustrated in gure 11, which shows some more epipolar lines, drawn from points that were not used in the computation of the fundamental matrix. It can be seen that for the points on the wall, which are quite far from the epipole, the corresponding epipolar lines seem approxi-mately correct, while for the points chosen on the table, the corresponding epipolar lines are obviously very incorrect, in the sense they are very far from the corresponding points. This situation does not occur with the non-linear criterion, as it can be seen in the bottom of this gure.

7 Conclusion

In this paper, we focused on the problem of determining in a robust way the Funda-mental matrixfrom a given number of image point correspondences. Its properties and relations to the well-known Essential matrix have been made very clear. Dierent pa-rametrizations for this matrix have been proposed and a large number of criteria have been considered and analyzed in great detail to tackle e ciently this problem. The classical linear criterion has been shown to be unable to express the rank and norma-lization constraints, and dierent non-linear criteria have been proposed to overcome its major weaknesses. It has been shown that the use of non-linear criteria leads to the best results and an extensive experimental work on noisy synthetic data and real

Figure 8: Epipolar lines obtained from the linear criterion (top), and from the non-linear

criterion (bottom)

Figure 9: Zoom showing the t with the linear criterion (left) and the non-linear criterion

(right)

images has been carried out to evaluate stability and convergence properties of each method.

References

1] F. L. Bookstein. Fitting conic sections to scattered data. Computer Graphics and Image Processing, 9(1):56{71, Jan 1979.

2] J.Q. Fang and T.S. Huang. Some experiments on estimating the 3D motion pa-rameters of a rigid body from two consecutive image frames. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6:545{554, 1984.

3] O.D. Faugeras. What can be seen in three dimensions with an uncalibrated stereo rig. In Proc. European Conference on Computer Vision, pages 563{578, 1992. 4] O.D. Faugeras, Q.-T. Luong, and S.J. Maybank. Camera self-calibration: theory

and experiments. In Proc. European Conference on Computer Vision, pages 321{ 334, 1992.

5] O.D. Faugeras and G. Toscani. The calibration problem for stereo. In Proceedings of CVPR'86, pages 15{20, 1986.

6] R.I. Hartley. Estimation of relative camera positions for uncalibrated cameras. In Proc. European Conference on Computer Vision, pages 579{587, 1992.

7] T.S. Huang and O.D. Faugeras. Some properties of the E-matrix in two view motion estimation. IEEE Transactions on Pattern Analysis and Machine Intelli-gence, 11:1310{1312, 1989.

8] Koenderink J. J. and A. J. van Doorn. A ne Structure from Motion. Journal of the Optical Society of America A, 8(2):377{385, 1992.

9] D.G. Jones and J. Malik. A Computational Framework for Determining Stereo Correspondence from a Set of Linear Spatial Filters. In Proc. European Conference on Computer Vision, pages 395{410, 1992.

10] K. Kanatani. Computational projective geometry. Computer Vision, Graphics, and Image Processing. Image Understanding, 54(3), 1991.

11] C.H. Lee. Time-varying images: the eect of nite resolution on uniqueness. Com-puter Vision, Graphics, and Image Processing. Image Understanding, 54(3):325{ 332, 1991.

12] C. Longuet-Higgins. The reconstruction of a scene from two projections: con-gurations that defeat the 8-point algorithm. In Proc. 1st Conf. on Articial intelligence applications, pages 395{397, Denver, 1984.

13] H.C. Longuet-Higgins. A Computer Algorithm for Reconstructing a Scene from Two Projections. Nature, 293:133{135, 1981.

14] S.J. Maybank and O.D. Faugeras. A Theory of Self-Calibration of a Moving Camera. The International Journal of Computer Vision, 8(2):123{151, 1992. 15] R. Mohr, L. Quan, F. Veillon, and B. Boufama. Relative 3d reconstruction using

multiple uncalibrated images. Technical Report RT84-IMAG12, LIFIA, June 1992. 16] J. L. Mundy and A. Zisserman, editors. Geometric invariance in computer vision.

MIT Press, 1992.

17] V.S. Nalwa and E. Pauchon. Edgel aggregation and edge description. Computer Vision, Graphics, and Image Processing, 40(1):79{94, Oct. 1987.

18] S.I. Olsen. Epipolar line estimation. In Proc. European Conference on Computer Vision, pages 307{311, 1992.

19] P.D. Sampson. Fitting conic sections to very scattered data. an iterative renement of the Bookstein algorithm. Computer Graphics and Image Processing, 18(1):97{ 108, Jan. 1982.

20] A. Shashua. Projective structure from two uncalibrated images: structure from motion and recognition. Technical Report A.I. Memo No. 1363, MIT, Sept 1992. 21] R.Y. Tsai. An E cient and Accurate Camera Calibration Technique for 3D Ma-chine Vision. In Proceedings CVPR '86, Miami Beach, Florida, pages 364{374. IEEE, June 1986.

22] R.Y. Tsai and T.S. Huang. Uniqueness and estimation of three-dimensional mo-tion parameters of rigid objects wirth curved surfaces. IEEE Transacmo-tions on Pattern Analysis and Machine Intelligence, 6:13{27, 1984.

23] J. Weng, T.S. Huang, and N. Ahuja. Motion and structure from two perspective views: algorithms, error analysis and error estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(5):451{476, 1989.

Figure 10: Intersection of epipolar lines obtained from the linear criterion

Figure 11: Additional epipolar lines obtained with the linear criterion (top), and with the

non-linear criterion (bottom)