Reconstruction de scènes à partir d'une seule image

(1)

HAL Id: inria-00598842

https://hal.inria.fr/inria-00598842

Submitted on 7 Jun 2011

HAL is a multi-disciplinary open access

archive for the deposit and dissemination of

sci-entific research documents, whether they are

pub-lished or not. The documents may come from

teaching and research institutions in France or

L’archive ouverte pluridisciplinaire HAL, est

destinée au dépôt et à la diffusion de documents

scientifiques de niveau recherche, publiés ou non,

émanant des établissements d’enseignement et de

recherche français ou étrangers, des laboratoires

Reconstruction de scènes à partir d’une seule image

Marta Wilczkowiak

To cite this version:

Marta Wilczkowiak. Reconstruction de scènes à partir d’une seule image. Digital Libraries [cs.DL].

2000. �inria-00598842�

(2)

U.F.R.

Informatique &MathématiquesAppliquées

WydzialMatematyki

iNaukInformaty zny h

DEA IMAGERIE VISION ROBOTIQUE

Proje t presented by

Marta Wil zkowiak

3D model a quisition from single image

Prepared in the INRIA Rhne-Alpes

in the MOVI proje t

Date: JUNE 2000

Supervisors: Dr Edmond Boyer

Prof. Krzysztof Mar iniak

(3)

1 Introdu tion 3

1.1 State of the art . . . 4

1.2 Our approa h . . . 6

1.3 Overview of this report . . . 7

(4)

Introdu tion

The human eye, onne ted with the human mind, is the base tool that

allows us to per eive and interpret the world surrounding us. Our eyes let

us not only simply see obje ts we an re ognize forms, followmovements,

estimate distan es and velo ities. The omputer vision is the dis ipline,

whi h triesto apply those apabilitiesto omputers. This allows toimprove

and omplete our vision and, on the other hand, to eliminate the need of

human intera tion in ertain appli ations.

The number and importan e of appli ations of omputer vision in

to-day's world is di ult to underestimate. Surveillan e, roboti s, assistan e

in surgi al operations and CAD/CAM systems are only some in the wide

range of dis iplines whi h are developed mostly thanks to progress in

om-puter vision. The reasons for applying omputer vision are numerous: its

exibility, ability of fast treatment of large amounts of data and, above all,

the possibilityto express informationina lear and intuitive way.

One bran h of omputer visiondeals with the extra tion of

threedimen-sional information from images. The ability of interpretation of 3D data is

ne essary insome kinds of appli ations. Consider for example roboti s, the

problem of obsta leavoidan e, or appli ations for3D modelvisualisation.

Inthisworkweareinterestedinusing omputervisionto onstru t

three-dimensional models of s enes. Re onstru tion of geometri models allows us

to observe s enes from dierent points of view. Texture information, whi h

an be extra ted from images, permits to obtain photorealisti alrenderings

(5)

approa h is are onstru tion of the 3Dmodelwith the use of several images

of thes ene. Butweare notalways abletoobtaintwoormoreimagesofthe

s ene. Let us onsider for example images oming from paintings, ar hives

et .

On the otherhand some spe ial, additionalinformationabout the s ene

anprovidestrong onstraintsonthe ameraparametersandpositionsofthe

model points in the 3D spa e. Espe ially useful are geometri al

dependen- ies, like parallelism,perpendi ularity, in iden e of lines orpointsand lines.

S enes, whi h provide data of this type, are for example urban or indoor

s enes.

Theaimofthisproje tistore onstru tthe3Dmodelofthes enehaving

in disposition only one image. Ne essary steps to doso are amera

alibra-tion and extra tion of the 3D textured information. We will be interested

mainly by images of indoor and outdoor ar hite tural s enes. It is easy to

observe, that su h s enes usually ontain some ommon elements, like

dif-ferent types of ubes, triangle prisms, solids built of sets of parallelograms

et . Inthis workwewillstudythe dierentamountsofinformationprovided

by proje tions of su h primitivesin the image and exploit them for amera

alibration and re onstru tion of the 3D modelof the s ene.

1.1 State of the art

The problems of amera alibration and 3D re onstru tion have been

studied for a long time. Proposed approa hes make use, in general, of two

or moreimages,either alibratedor not,whi hlead toeither Eu lideanor a

proje tive re onstru tion. They are explained forexample inFaugerasbook

[Fau93℄.

During the last years few approa hes to the problem of Eu lidean

re- onstru tion from a single image where introdu ed. Usually the amera is

alibrated by exploitation of onstraints given by vanishing points of spa e

dire tions. To the problem of 3D re onstru tion few dierent solutions are

given whi h are des ribed below.

B.Caprile et V.Torre [CT90℄ propose a method of amera alibration

based on vanishingpoints. The proje tions of two parallel lines inthe s ene

(6)

of the amera. This fa t allows to onstru t equations useful for amera

alibration. Inthis approa hthe onstraints ome fromthe vanishingpoints

ofperpendi ulardire tions. Theprobleminsu happroa hesisthefrequently

bad numeri al onditioning inthe vanishingpoint omputation.

EdmondBoyerandRoberto Cipolla[CB98℄ propose amethodfor

re on-stru tion of urban environments based on iterative amera alibration. No

information a priori about the amera parameters is needed. Calibration is

based alsoon vanishingpoints. This approa h is basedon a linear riterion

whi h optimisesathree dimensional dire tion, the dual of a vanishingpoint

inthe3Dspa e. Todeneparalelismandperpendi ularity, uboidsareused.

Experiments show, that estimatinga spa edire tionis morerobust tonoise

perturbations than estimating an image point. This method an alibrate

severalimages. The re onstru tionisprovidedwiththe useoffew alibrated

images.

FabienVignonandEdmondBoyer[Vig99℄proposeamethodofmodelling

of urban s enes based on few images. Their method of amera alibration

is based on onstraints for amera parameters provided by the images of

uboids. The onstraints are extra ted from the proje tion matrix in ane

system oordinates atta hed to the uboid. This method of alibration

ap-plied to parallelepipeds isused in this work. The re onstru tion is based on

orrespondingpointsinfew images.

PeterSturm andStephen Maybank[SM99b℄ presentageneralalgorithm

for plane-based alibration, that an deal with arbitrary numbers of views

and alibration planes. The algorithm an simultaneously alibratedierent

views froma amerawith variableintrinsi parameters. Forsome ases they

des ribethe singularities. Experimentalresults exhibit the singularities and

reveal a good performan e in non-singular onditions. Also the proposition

of re onstru tion basedonthe knowledgeofthe metri stru ture ofplanes is

presented.

The sameauthorsin[SM99a℄propose amethodforintera tive3D

re on-stru tion of pie ewise planar obje ts from a single image. For amera

ali-brationthe methodexplainedin[CT90 ℄isused. The re onstru tionisbased

on user-provided oplanarity, perpendi ularity and parallelism onstraints.

Known intrinsi parameters of the amera allow nding proje tion rays of

points. The vanishing line of planes give information about their normal

(7)

of the sum of squared distan es between planes and points. Subsequently,

planes with unknown vanishing lines are re onstru ted in an iterative

man-ner.

A.Criminisi,I.ReidandA.Zisserman[CRZ99℄des ribeamethodof

mea-surements of distan es in the 3D s ene based on known vanishing line of a

referen e planeandthe vanishingpointofthe dire tionperpendi ulartothe

referen e plane. A new algebrai representation is developed for measuring

distan es betweenplanesparalleltothereferen eplane, omputingareaand

length ratios onplanesparallel tothe referen eplane, determiningthe

am-era position. The disadvantage of their sequential re onstru tion method is

the fa t,that errors are not spreaduniformlyin the model.

An approa h for amera alibration based on Parameterised Cuboid

Stru ture is presented in [CSC99 ℄ . This method uses a referen e

paral-lelepiped in the image to alibrate the amera and reate the oordinate

system allowing inserting virtual obje ts into the s ene. Some studies on

onstraints given by known parallelepiped parameters are provided. The

amera alibrationismadewith thequitestrongassumptionthatthe aspe t

ratio of the amerais known.

1.2 Our approa h

The aimof this work is to provide methodsfor 3D s ene re onstru tion

based on a single image. We want to extra t information of two dierent

types: geometri al information (position of points, lines, planes) and

pho-tometri (texture of surfa es). We are interested mostly in indoorand

out-doorar hite tural s enes. They usually ontain geometri alregularitiesthat

provide onstraintsuseful for amera alibrationand3Dre onstru tion.

Ex-amples are the perpendi ularity, parallelismof lines and planes and various

formsof in iden ebetweenplanes, linesand points. Ourspe ialinterestwas

to make exhaustive studies of the onstraints provided by primitives, whi h

appearveryofteninar hite turals enes: parallelepipedsandparallelograms.

The use of predened primitives whi h are ommonin s enes of interest

al-lowstominimizetheworkofauser, whointrodu esthe data,whileenabling

onsistant and a urate 3D re onstru tions.

(8)

vanishingpointsfor amera alibration, a methodbased on onstraints

pro-vided by theproje tionmatrixfromthe ane3Dspa egeneratedby uboid

stru ture tothe2D imagespa eisused. Inthebeginningwestudiedvarious

kinds of onstraintsfor amera intrinsi parametersprovided by proje tions

of orners of referen e parallelepipeds. We wanted to represent the

paral-lelepiped in a general way, allowing the optimal exploitation of ea h of its

known parameters. Some work in this area were done in [CSC99℄, but the

study was not exhaustive. Our solutionallows toderive one additional

on-straintfromeveryknownparameter. Solvingquadrati equationssystem an

do it. The amera alibration tool we developped exploits all the

informa-tion providedbyreferen eparallelepipedswhi hleadsto onstraintsoflinear

type. It gives the possibility of ombining the onstraints provided by the

proje tionsofseveralreferen e parallelepipeds. Itgivesalsothepossibilityof

using amera parameters known a priori. For example the assumption that

the prin ipalpointissituatedinthe enter oftheimageisveryreasonablein

general. Alsosometimeswe knoworwe an makeanassumption on erning

the aspe t ratio of the amera.

As se ond step weevaluated amethodof re onstru tion of the 3D

oor-dinates of points belonging to a onnex set of parallelograms. With known

ameraintrinsi parametersweare abletore onstru tthe proje tion raysof

points. Tondthe depthsofthosepointswe evaluatedamethodintrodu ed

in [CSC99 ℄ of nding the relative depths of verti es of parallelograms. We

wantedtore onstru tmanys enepointssimultaneouslytoavoidthe

in reas-ing error inthe 3D model, whi h ofteno urs in sequential approa hes.

1.3 Overview of this report

In the se ond hapter we give some denitions of proje tive geometry,

ameramodellingand alibration,and 3Dre onstru tion. Inthe third

hap-ter we introdu e the parameterisation of parallelepipeds and present

on-straints on amera intrinsi parameters provided by known parameters of

referen e parallelepipeds. We also show experimental results of tests of

ap-pli ation based on the presented ideas. In the fourth hapter we present

an algorithm for 3D re onstru tion based on sets of onnex parallelograms,

(9)

(10)

Theoreti al ba kground

Inthis hapterwepresentthetheoreti alba kgroundofthiswork. Atthe

beginningofthis hapterwegivenotations ne essarytoreadthisreport.

Af-ter this we introdu e basi denitions on erning proje tive geometry. Next

webriey des ribethe prin iples of ameramodellingand alibration. Then

we explain the on ept of 3D re onstru tion and des ribe the dieren e

be-tween the proje tive, ane and Eu lidean re onstru tion. At the end, we

briey presentthe ideaof datanormalization,introdu edin omputer vision

by Hartley, whi h an be used for numeri alstabilization of algorithms.

2.1 Notation

Bold letters represents olumn ve tors. We do not distinguish between

a point and its algebrai al representation. To represent points in 3D spa e

we use apital letters, to represent points in 2D spa e we use lower ase.

Corresponding pointsin the 3D s ene and 2D image are represented by the

same letter, e.g. Q represents the point in 3D spa e and q represents its

proje tioninthe image. Thenullve tor ofthe spa eof ndimensions willbe

represented as0

n .

Tonotematri esweuse apitalletters. Theidentitymatrixofdimension

nn is represented by I

n .

S alars, in parti ular elements of ve tors, are in lower ase, for example

we an writeQ =(q 1 ;q 2 ;q 3 ) T .

(11)

Therealspa eisrepresented byR n

. Theproje tivespa eofdimensionn

isrepresented byP

n

. Theproje tiveequalityup toas alarfa torisdenoted

as .

2.2 A few notions of proje tive geometry

2.2.1 Homogeneous oordinates

The basi mathemati al framework of our work is proje tive geometry.

The n-dimensional proje tive spa e is denoted P

n

. Points in P

n

are

rep-resented by ve tors with n+1 oordinates. They are alled homogeneous

oordinates and are dened up toa s alar fa tor:

[x 1 ;:::;x n+1 ℄ T [x 1 0 ;:::;x n+1 0 ℄ T (2.1)

if, and onlyif, 96=0 su h that

[x 1 ;:::;x n+1 ℄ T =[x 1 0 ;:::;x n+1 0 ℄ T (2.2)

Linear transformationsof a proje tive spa e are alled homographies or

ol-ineations.

2.2.2 Point at innity

The orresponden e between point oordinates in P

n and in R n is de-s ribed as 0 B B x 1 x 2 x 3 x 4 1 C C A ! 0 x 1 x 4 x 2 x4 x 3 x4 1 A (2.3)

Thistransformationisnotpossiblewhenx

4

=0. Thismightbeexplained

by the fa t that the proje tive spa e onsists of the usual ane spa e and

the plane x

4

=0, whi h is alledthe planeat innity.

To ilustrate the on ept of points at innity onsider a proje tive line.

(12)

B B x 1 x 2 x 3 1 C C A + d 0 B B x 1 x2 x 3 1 C C A + d 0 ! !inf d 0 (2.4)

Equation(??) implies,that allproje tive lines indire tiond interse t at

plane at innity in the same point [d

T ;0℄ (g. ??). 0 0 0 1 1 1 00 00 00 11 11 11 0 0 0 1 1 1 0000000000 0000000000 0000000000 0000000000 0000000000 0000000000 0000000000 0000000000 0000000000 0000000000 0000000000 0000000000 1111111111 1111111111 1111111111 1111111111 1111111111 1111111111 1111111111 1111111111 1111111111 1111111111 1111111111 1111111111 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111 11111111

P1

P2

d

π∼

x4=0

P=[d,0]

Figure2.1: The interse tion of parallel lines isa point atinnity.

2.2.3 The Absolute Coni

The aim of this se tion is to introdu e the on ept of absolute oni ,

whi h is useful for amera alibration. At the beginning we give brief

de-nitions of oni s and quadri s in proje tive spa e, whi hwill allowtodene

dieren es between proje tive, ane and eu lidean transformations of the

proje tive spa e. This willallow usto show the importan eof properties of

the imageof the absolute oni .

A oni in P

2

is dened as the urve on the proje tive plane onsisting

of points[x 1 ;x 2 ;x 3 ℄ T

that satisfy a quadrati equation:

3 X i;j=1 a ij x i x j =0 (2.5) A quadri of the P 3

(13)

[x 1 ;x 2 ;x 3 ;x 4

℄ that satisfyan equation of the type:

4 X i;j=1 a ij x i x j =0 (2.6)

The transformations of the proje tive spa e, whi h preserve the plane at

innity are alled the ane transformations. An important property is the

fa t thatthey preserveparallelismof linesand ratios ofdistan es onparallel

lines.

Let usdene the spe ial oni , onsisting of points[x

1 ;x 2 ;x 3 ;x 4 ℄ T with: 3 X i=1 x 2 i =x 4 =0 (2.7)

This oni is alled the absolute oni . It onsists of omplex points at

innity. The transformations of the proje tive spa e, whi h preserve the

absolute oni are alled the eu lidean transformations.

The eu lidean transformations preserve angles and ratios of lengths of

obje ts. Inotherwords,they analwaysbede omposedintorotation,

trans-lation and a global s aling. Thus the image of the absolute oni does not

depend on the position and orientation of the amera in spa e. It depends

only onthe intrinsi parameters of the amera. That is why the on ept of

the absolute oni is basi for amera alibration.

2.3 Camera modelling

Inthis se tionwedes ribesomebasi on eptsfor ameramodelling. At

the beginningwedes ribethe ameramodelusedinthiswork. Inthe se ond

part we briey present the transformations that are appliedto pointsin 3D

spa e to obtaintheir proje tions inthe image.

The ameramodelwe useisthe pinholemodel. Itisshown onthegure

??.

Thepinhole ameramodelofthe ameraisbasedontheproje tive

trans-formationofthe3Dspa etotheplane alledtheimageplane. Theproje tion

enterCisalso alledtheopti al enter. Thedistan ebetween theopti

(14)

Q

q

Optical axis

Image plane

Focal plane

Projection center

Principal

point

Figure 2.2: The pinhole amera model.

in spa e is the interse tion of the line XC and the image plane. This line

is alled the proje tion ray. The opti axis is the line passing through the

proje tion enter and that is perpendi ular to the image plane. Its

interse -tion with theimageplane is alledthe prin ipalpoint. Thefo alplaneisthe

plane ontaining the proje tion enter and parallel tothe image plane. It is

easy to observe, that proje tions of points in the fo al plane are points at

innity of the image plane.

Forthe mathemati aldes ription of thepinhole model, we onsider

nor-mally four dierent oordinates systems: a s ene system, a amera system,

a system in the image plane and a system related to the pixel geometry of

the imageplane. The relation between thosesystems isshown onthe gure

??:

The transformation between the point oordinates in the 3D s ene and

oordinatesof itsproje tioninthe imageis omposed ofthe following

trans-formations: 0 wu wv w 1 A = i r C r S s T 0 B B x y z t=1 1 C C A (2.8)

Tis thetransformationfromthe s ene oordinateframetothe amera

oor-dinate fram. It is a eu lidean transformation,so it an be de omposed into

(15)

Scene

Camera

Image

Pixels

Q

q

X

Y

Z

x

y

u

v

Figure 2.3: The pinhole ameramodel oordinatesystems.

points in the oordinates system onne ted with the amera to the retinal

plane. C is the ane transformation fromthe oordinate system onne ted

with the image plane tothe system onne ted with pixels. The omposition

of those transformations is usually represented by a proje tion matrix P

34 .

As the result wehave:

xP

34

X (2.9)

Matrix Pis dened up the s ale fa tor, so it has 34 1 = 11 degrees of

freedom.

The proje tion matrix may bede omposed asfollowsin ordertoexhibit

physi al propertiesof the amera:

P K( R j t) (2.10)

WhereR

3x3

istherotationmatrix(anorthogonalmatrix)andta"translation

ve tor" of size 3. The R and t have six degrees of freedom and they are

alled extrinsi parameters of the amera. They represent the orientation

and position of the amera with respe t to the 3D s ene. The matrix K is

the upper triangular and has thus ve degrees of freedom(it is dened up

to s ale). They orrespond tothe intrinsi parameters of the amera. On e

the intrinsi parameters of the amera are known, we say that amera is

alibrated.

(16)

K= 0 f s u 0 0 f v 0 0 0 1 1 A :

Its elements have the following meanings:

(u 0

;v

0

) - the position of the prin ipalpointsinthe image

- the aspe t ratio i.e. the ratio between horizontal and verti al dimensions of a pixel

s - the skew parameter whi h depends onthe anglebetween axes. Asnormally pixels are re tangular,in the rest of this work we will onsider s=0.

f - the fo aldistan e multiplied by height of a pixelin units of aretinal frame.

2.4 Camera alibration

As wementionedatthe endofthe previousse tion,by alibrationofthe

amera we understand the extra tionof its intrinsi parameters.

Therearetwomainapproa hestotheproblemofthe amera alibration:

Ifwe knowpositionsofpointsinspa esu h aspointsona alibration

patternwe an alibratethe amerausing oordinates oftheir

proje -tions in one or more images.Equation(??) gives ustwo onstraints on

the elements of the proje tion matrix for every known orresponden e

between a 3D point and itsimage. That implies that if we know

posi-tions inthe eu lideanspa eofatleastvepointsand onedire tion,we

an obtainthe proje tionmatrix. Then,usingthefa t,thatKisupper

triangular andRisorthogonal,we anextra t theintrinsi parameters

of the ameraby Cholesky de omposition.

Othermethodsof amera alibrationuse onstraintsprovidedbyknown

geometri al dependen ies, like perpendi ularity and parallelism of

di-re tions in the s ene. There exist several approa hes to this problem,

but theyleadusuallytosimilar onstraintsontheintrinsi parameters,

i.e. to equations based on the image of the absolute oni . Its link to

alibration and metri s ene stru ture was des ribed in se tion ??.

Todemonstrate this typeof onstraintsbriey des ribethe methodof

amera alibration based onvanishingpointsas introdu edin [CT90 ℄.

The vanishingpointof adire tionis theimageof the pointof

(17)

d

projection

center

vanishing

point

Figure 2.4: The vanishing point.

The main idea of the algorithm is based on the fa t, that the s alar

produ t of two perpendi ular dire tions isequal to0. Let us take two

ideal points, i.e. points at innity, [V

1 ;0℄ T and [V 2 ;0℄ T . From the

proje tions equationsv

i

KV

i

we obtain the orresponden e

V i K 1 v i (2.11)

Considering this and the fa tthat V

1 T V 2 =0 wehave: v 1 K T K 1 v 2 =0 (2.12) where the K T

is the transposition of invertion of K. The matrix

K T

K 1

isthe imageof the absolute oni .

Equation(??) implies that every pair of vanishing points of

perpen-di ular dire tions gives one onstraint on intrinsi parameters of the

amera.

This shows the utilityfor alibrationofimages of uboids. One uboid

ingeneralposition(??)bringsthree onstraintsontheintrinsi

param-eters. In many pra ti al situations some a priori information on the

intrinsi parameters is given; so the onstraints provided by a uboid

are usually su ientto alibratethe amera.

2.5 3D Re onstru tion

By the re onstru tion of the 3D model of the s ene we understand

(18)

vanishing point

cuboid

Projection center

Figure 2.5: Vanishing pointsimplied by the uboid.

obtain dierent types of re onstru tion. Usually we are interested in

re on-stru tion, whi hpreservethe anglesand atleast ratiosof lengths. This kind

of re onstru tion isnot possible if wedo not have the amera alibrated.

2.5.1 3D Proje tive re onstru tion

In the ase of the un alibrated amera we are only able to obtain a

so- alled proje tive re onstru tion. It brings some 3-dimensional information,

allowingforexampletore ognizethetypeoftheobje t(seegure??). Itwas

shown by Faugeras in[Fau92℄ and Hartleyin [HGC92℄ that itis possible to

obtain the proje tive alibration of a set of ameras without any additional

information about the s ene stru ture.

2.5.2 3D Ane and eu lidean re onstru tion

Byanere onstru tionweunderstandare onstru tion,whi hpreserves

the parallelism and the lengths ratios on parallel dire tions. To obtain this

typeofre onstru tionwehavetond thetransformationfromtheproje tive

spa e totheane spa e. Tond this transformationwehave toidentify the

plane at innity. Three points in general positions belonging to the plane

dene it. They an be obtained by use of the vanishing points (g. ??) of

three pairs of parallel lines. It was shown that using this on ept it is quite

easy to obtainthe ane re onstru tion.

(19)

model (up to the

s aleve tor).

Coplanarityandgeneralform

arepreserved.

Figure 2.6: Two threedimensionalmodels of the house.

re onstru tion, we have to extra t the intrinsi parameters of the amera.

Forthis,some additionalinformationhas tobegiven. One useful onstraint

is the onstraint of perpendi ularity ([CT90℄,[CB98 ℄). Known angles, some

knowledgeabout distan e ratios, an alsobring useful onstraints, asshown

in hapter ??

The eu lidean re onstru tion, withoutany metri informationabout the

s ene, anbeobtainedonlyuptoas alarfa tor. Thisissatisfa toryinmost

of appli ations. But it is su ient to know a single length to x the s alar

fa tor and obtainan exa t 3D re onstru tion of the s ene.

2.6 Data normalisation

One problem that we fa e almost every time when we apply ideas to

real systems is the bad onditioning of the data. Let us onsider image

points,whi harethe baseofour omputations. Theyare usuallyofthe type

x=[u;v;1℄

T

,whereu;v areofthe order10

2

;10

3

. Thetypeof onstraintsthat

(20)

of the order between 1 and 10 . The other thing is that if pixels are not

entered, their oordinates are of very dierent orders.This might auses a

bad onditioningof equation system and lead to very unstable solutions.

R.Hartleyin[Har95℄proposedanormalizationpro edureforpointssu h

as toobtain oordinates x 0 =[u 0 ;v 0 ;1℄ T ,whereu 0 ;v 0

areof theorder of1. It is

obtainedbya hangeoftheimage oordinatesysteminawaythatguarantees

that all points of interest are lying in the unit ir le. The rst step is the

translationof the enterofthe oordinatesystem tothe gravity enter ofthe

set of points. The se ond step is a s aling of oordinates to obtain lengths

of ve tors x

0

i

1.Thisidea isillustrated by g.??.

1

100

Figure 2.7: The Hartleynormalisation.

Su h transformations improve the onditionning and lead to better

(21)

Camera alibration using

parallelepipeds

In this hapterwe wouldliketointrodu e a alibration methodbasedon

parallelepipeds.

Weespe iallyare interested inurbans enes. Thebase primitiveexisting

in manmade environments isthe uboid (Fig. ??).

Figure3.1: Exampleof uboiddenition for amera alibration

The utilityof uboidstru turesindeningurbans enestru tureis

obvi-ous. Butitisalso lear,thatwe annotdenetheentires eneonlybymeans

(22)

the stru ture more interesting than uboid, but stillquiteeasyto

onstru t-the parallelepiped stru ture. We an see the examples of su h buildings on

Figures ??, ??.

(a)TorresKio,Madrid. (b) Two towers lean

towards ea h other at

anangleof15

o .

Figure 3.2: Example of parallelepiped stru ture inurbane s ene.

The se ondreason speakingfor applyingparallelepipedsisthe fa t,that

they extend the lass of obje t whi h an be used for alibration, so we an

obtain more onstraints. In the presen e of the noisy data it an

numeri- ally stabilize results. The Figure ?? shows how we an dene additional

parallelepiped stru ture, whi his not expli itlyshown onthe image.

Our methodexploitsthe onstraintsonintrinsi parametersprovided by

images of verti es of parallelepipeds in the image and known parameters of

the parallelepipeds. In order to express those onstraints, we had to hoose

a parameterization of the parallelepipeds. This is des ribed inthe rst part

of the hapter. Then we show our method for omputing an intermediate

proje tionmatrixfromtheane oordinatesystemrelatedtoparallelepiped

stru ture. Next we present onstraints on the amera intrinsi parameters

provided by olumns of this intermediate matrix. Then we des ribe our

way of using the prior knowledge onintrinsi parameters and dis uss whi h

parameters an be alibrated in degenerated ases. Finally, we des ribe our

(23)

Boston. JohnHan o ktoweris

rhomboid.

Figure 3.3: Example of parallelepiped stru ture inurbane s ene.

3.1 Parallelepiped parameterization

Our methodof amera alibration is based on mat hes between pro

je -tions oftheparallelepiped's ornersintheimageandtheir oordinatesinthe

ane spa e related tothe primitive.

Let usdene: 1 =\XY; 2 =\YZ; 3

=\ZX angles between the edges of

paral-lelepiped. 1 ; 2 ; 3 lengthsof edges

Todene onstraints onthe intrinsi parameters, we introdu ed the

param-eterization of parallelepiped shown in Figure??.

The basis hange fromane toeu lidean spa e an bedes ribed as:

E X=

4 A

(24)

bration. lelepipedstru ture.

Figure3.4: The use of parallelepipeds for additional onstraints.

where

4

is the matrix of hangeof the base and an be expressed as

4 = 0 B B x 1 x 2 x 3 0 0 y 2 y 3 0 0 0 z 3 0 0 0 0 1 1 C C A (3.2) where x 1 = 1 (3.3) x 2 = 2 os 1 (3.4) y 2 = 2 sin 1 (3.5) x 3 = 3 os 3 (3.6) y 3 = 3 os 2 os 3 os 1 sin 1 (3.7) z 3 = 3 s 1 ( os 3 ) 2 os 2 os 3 os 1 sin 1 2 (3.8)

3.2 Computation of the intermediate

proje -tion matrix

Let us x now the oordinates of a parallelepiped's verti es inan ane

(25)

y

x

z

x’

y’

z’

(0,0,0)

(x1,0,0)

(x2,y2,0)

(x3,y3,z3)

_λ3

λ2

λ1

θ3

θ2

θ1

Figure3.5: The ane oordinatesystem relatedto the parallelepiped.

x’

y’

z’

(1,1,0)

(1,0,0)

(0,0,0)

(0,1,0)

(0,0,1)

(0,1,1)

_(1,1,1)

(1,0,1)

Figure 3.6: The xed oordinates of verti es inthe ane oordinatesystem

(26)

i

parallelepiped and q

i

itsproje tion in the image. From the previous se tion

we know that the oordinates of the point Q

i

in the eu lidean oordinate

systemareexpressed by

4 Q

i

. LetPbetheproje tionmatrixfromeu lidean

spa e to the image. We havethe following dependen e:

P 4 Q i =q i (3.9)

Every dependen e of the type ??(i.e. every known proje tion of one

of the parallelepiped's verti es ) gives us the following onstraints on the

matrix M=P 4 : 0 wu wv w 1 A = 0 m 11 m 12 m 13 m 14 m 21 m 22 m 23 m 24 m 31 m 32 m 33 m 34 =1 1 A 0 B B x y z t=1 1 C C A (3.10)

This allows the onstru tion of two linear equations from every

depen-den e: m 31 xu+m 32 yu+m 33 zu+u=m 11 x+m 12 y+m 13 z+m 14 m 31 xv +m 32 yv+m 33 zv+v =m 21 x+m 22 y+m 23 z+m 24 (3.11)

whi h leads tothe following equationsystem of the type: AX=B

0 B x 1 y 1 z 1 1 0 0 0 0 x 1 u 1 y 1 u 1 z 1 u 1 0 0 0 0 0 x 1 y 1 z 1 1 x 1 v 1 y 1 v 1 z 1 v 1 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 C A 0 B B B B B B B B B B B B B B B B B B m 11 m 12 m 13 m 14 m 21 m 22 m 23 m 24 m 31 m 32 m 33 m 34 =1 1 C C C C C C C C C C C C C C C C C C A = 0 B u 1 v 1 . . . 1 C A (3.12)

(27)

depends on the rank of A.

if rk A =11 one unique solution,up toa s alarfa tor, isobtained

si rk A<11an innitefamily of solutionsis obtained.

This impliesthatweneedatleastsixpointsingeneralpositiontoobtain

thesolution. Intheoryweneedonlyvepointsandonedire tion(adire tion

provides one onstraint) to obtain the matrix. In pra ti e we an usually

identifyatleastsixproje tionsoftheprimitive'sverti es. Ontheotherhand,

to improve the results, we usually use over onstrained systems. We nd the

solutionof su ha systemby minimizationof theexpression kAX Bkinthe

sense of the least squares using the SVD method (des ribed for example in

[PTVn92 ℄.

3.3 Constraints on the intrinsi parameters

In this se tion we will des ribe the onstraints for the amera intrinsi

parameters provided by matrix P

4

introdu edinthe previous se tion.

Using the equation (??) we an write

P 4

K( R j t)

4

(3.13)

Let us onsider the matrix X

33 =KR,where = 0 x 1 x 2 x 3 0 y 2 y 3 0 0 z 3 1 A (3.14)

Let usevaluate the expression X

T K

1

X using the fa tthat R is the

orthogonal matrix. Weobtain:

X T K T K 1 X =(KR) T K T K 1 KR= T R T K T K T K 1 KR= T (3.15) The matri es T and !K T K 1

(28)

T 2 1 1 2 os 1 1 3 os 3 1 2 os 1 2 2 2 3 os 2 1 3 os 3 2 3 os 2 2 3 A (3.16) ! 0 1 0 u 0 0 2 2 v 0 u 0 2 v 0 2 f 2 +u 2 0 + 2 v 2 0 1 A : (3.17)

Weare lookingforelements of! onlyup toas alarfa tor, sowe anx

one of its elements, for example !

11

. Considering equations (??), (??) and

(??) we obtain: X T 0 ! 11 =1 0 ! 13 0 ! 22 ! 23 ! 13 ! 23 ! 33 1 A X 0 2 1 1 2 os 1 1 3 os 3 1 2 os 1 2 2 2 3 os 2 1 3 os 3 2 3 os 2 2 3 1 A (3.18)

Equation (??) allows to make some observations about what kind of

onstraints on intrinsi parameters we an obtain depending on the kind

of information about parallelepiped parameters. Considering that

T

is

known up to as alar fa tor, we have:

One known angle

ij

= 90

o

gives one linear onstraint on elements of

!: X i T !X j =0 (3.19)

One known angle

ij

givesone nonlinear onstraint onelementsof !:

os 2 ij = (X i T !X j ) 2 (X i T !X i )(X j T !X j ) (3.20)

One known length ratio r

ij =

i

j

gives one linear onstraint on !

ele-ments: X i T !X i (r ij ) 2 X j T !X j =0 (3.21)

(29)

then up tove onstraints onthe elementsof !: X i T !X j ( T ) ij ( T ) kl X k T !X l =0 (3.22)

If more parallelepipedsare available, we justin lude new equations into

an equation system. At the beginningwe add only onstraints of the linear

type whi h leads to an equation system of type AX = 0. Having at least

four equations we an obtain a unique solution for the ve tor of unknowns

[! 13 ;! 22 ;! 23 ;! 33 ℄ T

using the SVD method. If the linear system is

under- onstrained we an use non-linear onstraints to limitthe spa e of solutions

obtained by applying the SVD method for under onstrained equation

sys-tem.

On e the elements of the matrix ! are known the intrinsi amera

pa-rameters an be extra ted via [SM99b℄:

2 = ! 22 ! 11 (3.23) u 0 = ! 13 ! 11 (3.24) v 0 = ! 23 ! 22 (3.25) f 2 = ! 11 ! 22 ! 33 ! 22 ! 2 13 ! 11 ! 2 23 ! 11 ! 2 22 (3.26)

3.4 Prior knowledge about amera parameters,

degenerated ases

Veryoftenweknoworwe anmakeassumptions on erningsome

intrin-si parameters of the amera. Formost ameras, for example, the prin ipal

pointliesinthe enter of the image. Information aboutthe aspe t ratio an

usually be found in a amera's te hni al do umentation, or an sometimes

be assumed to be unit. The fo al distan e is the only parameter that we

(30)

always allow toextra t all the intrinsi parameters. It is easy to show both

algebrai allyand geometri allythat the following degenerated ases exist:

Ifonefa eoftheprimitiveisparalleltotheimageplaneanditslengths

areunknownthenedgesofthisfa egiveno onstraintsontheprin ipal

point,fo aldistan e and skew fa tor( Eq.?? isalways true).

The dire tion perpendi ular to the image plane allows to lo alize the

prin ipalpoint,but doesnot onstrain otherintrinsi parameters.

Those degenerated ases are illustrated by Figure ??

C

Figure3.7: Adegenerated ase. Proje tionsoflineslyingintheplaneparallel

to the image plane do not onstrain the intrinsi parameters if lengths are

unknown. The proje tion of the perpendi ular tothe image plane dire tion

determines the prin ipalpoint.

There are two ways to use the prior knowledge about the intrinsi

pa-rameters.

0nemethodistotranslatetheimage oordinatesystemtotheprin ipal

point and to s ale its axes su h that in further al ulus u

0

= 0;v

0 =

0; =1.

The se ond solutionisto add tothe mainequation system, depending

(31)

13 0 ! 22 = 2 (3.28) v 0 ! 22 +! 23 = 0 (3.29) 3.5 Experimental results

Toverifyrobustnessofourmethodswemadefewtests. Atthebeginning

wetestedour alibrationmethodonaseriesof syntheti data. Inthe se ond

stage wemadetest basedonphotoofthe alibrationpattern. Atthe endwe

tested our appli ation on real data. In this se tionwe des ribe our results.

3.5.1 Syntheti data

Toobtaingeneraldedu tions on erningour methodweappliedtests on

syntheti data. We appliedthe same test totwo dierent primitives:

First test was based on parallelepiped with angles (30

0 ;45 0 ;60 0 ) and lengths given.

Se ond test wasbased onparallelepiped withangles(60

0 ;90 0 ;90 0 )and unknown lengths.

Our appli ation have few types of parameters. First type are image

oordinates of primitives proje tions. The se ond type are known a priori

intrinsi parameters like position of a prin ipal point and a skew ratio. In

order toverify the sensitivity ofour methodto thoseparameters we applied

tests of the method with noisydata.

On the other side, we wanted to verify the behavior of the method in

degenerated ases. Inordertodoit,wetestedresultsof alibrationbasedon

parallelepiped rotated aroundthe verti al axisparallel tothe imageplane.

The alibration results given above were obtained as a median from a

seriesofrandomtest. Ea htestwasprovidedwithandwithoutdata

normal-izationinthepre omputationstage. Weassumedthatimagesizeis512512

and theprin ipalpointliesinthe enteroftheimage. Weformulatedseveral

(32)

but in the ase of our method the improvement of results is not very

signi ant. This an be seen on Figures ?? and ??. They present

results of alibration with and without normalization for both tested

primitives rotated aroundthe verti al axis. We an see that positions

lose todegenerated asesdoesnot allowto alibratea amerawith or

withoutnormalization. Alsoresultsof alibrationarenotverydierent.

Similardedu tionsweremadeintestsofsensitivitytonoiseonprogram

parameters.

Knowledge about lengthsof parallelepiped, thanks toadditional

equa-tions to the alibration system, improves obtained results in tests on

sensitivity to noiseon known apriori intrinsi ameraparameters and

test onrotatedprimitives.

Known lengths of edges allows to alibrate also in otherwise

degener-ated ases (des ribed in se tion ??). Figures ?? and ?? show results

of alibration of a parallelepiped, whi h is rotated around the

verti- al axis parallel to the image plane. The random noise of two pixels

was applied to primitivesverti es image oordinates. In ase of

paral-lelepiped, whi h lengths are not known, every time its fa e is parallel

to the image plane, alibration is not possible. In ase of known edges

in su h ases wesee onlyin reasing error.

In the se ond test we applied random noise onimage oordinations of

primitivesverti es proje tions.

Calibration in ase of primitive whose edges lengths are given is

pos-sible up to 3 pixels noise (Figure ??). In the se ond test results are

reasonable up to six pixels noise (Figure ??). It an be explained by

the fa t, that in ase of the rst primitive (with allparameters given)

weobtainmoreequationsbasedonnoiseddatathaninthese ondtest.

It may ause worst results.

The third test (Figures ?? and ??) wasapplied to verify the inuen e

of the error on position of a prin ipal point in the image. We tested

behavior of the method with distan e between (u

0

;v

0

) real and given

as a program parameter form 0 to 200 pixels. Experiments on both

primitives show, that the method is not very sensitive for not exa t

(33)

0

2

4

6

8

10

12

14

16

18

650

700

750

800

850

900

950 Rotation angle

f

simple

normalized

f theor.

(a)Test1,rotation

0

2

4

6

8

10

12

14

16

18

650

700

750

800

850

900

950 Rotation angle

f

simple

normalized

f theor.

(b)Test 2,rotation

Figure3.8: Test of alibrationbasedonparallelepipedrotedaroundaxis

par-allel to the image plane for primitive with known lengths(a) and unknown

lengths (b). In both ases degenerated ases are seen, but in ase (a)

(34)

0

2

4

6

8

10

12

14

16

18

400

500

600

700

800

900 1000

1100

1200

Rotation angle

f

(a)Test1,rotation

0

2

4

6

8

10

12

14

16

18

400

500

600

700

800

900 1000

1100

1200

Rotation angle

f

(b)Test 2,rotation

0

1

2

3

4

5

6

7

8

9

10

11

400

500

600

700

800

900 1000

1100

1200

Image coordinates

f

( )Test1,noisedimage oordinates

0

1

2

3

4

5

6

7

8

9

10

11

400

500

600

700

800

900 1000

1100

1200

Image coordinates

f

(d)Test2,noisedimage oordinates

Figure 3.9: Graphs of averages and standard deviations from tests on

syn-theti data.(a)- test of alibration based on primitive with angle between

fa e and image plane between 0 180, edges lengths are known; (b)- test

of alibration based on primitive with angle between fa e and image plane

between 0 180, edges lengths are unknown;( )- test with error on image

oordinates from0 10 pixels, edges lengthsare known;(d)- test with error

(35)

0

20

40

60

80

100

120

140

160

180

200

220

400

500

600

700

800

900 1000

1100

1200

Position of a principal point

f

(a)Test1,noisedposition (u0,v0)

0

20

40

60

80

100

120

140

160

180

200

220

400

500

600

700

800

900 1000

1100

1200

Position of a principal point

f

(b)Test2,noisedposition(u0,v0)

0

0.2

0.4

0.6

0.8

1

400

500

600

700

800

900 1000

1100

1200

Skew ratio

f

( ) Test 1,noisedskew ratio

0

0.2

0.4

0.6

0.8

1

400

500

600

700

800

900 1000

1100

1200

Skew ratio

f

(d)Test 1,noisedskew ratio

Figure 3.10: Graphs of averages and standard deviations from tests on

syn-theti data.(a)- test with error on position of prin ipal point, edges lengths

are known; (b)- test with error onposition of prin ipalpoint,edges lengths

are unknown;( )-test witherroronskewratio,edgeslengthsare known

(36)

then 20%. This resultshows, that assumption that theprin ipalpoint

lies inthe enter of the image is usually very reasonable and even if it

is false, alibrationshould give quitegoodresults.

Prior knowledge about the aspe t ratio improves the alibration. On

the other hand if the value given is not exa t, it spoils results. It is

shown onFigures?? and ??.

3.5.2 Calibration pattern image

(a)Familyof uboids. (b)Familyofparallelepipeds.

Figure3.11: Experiment with alibrationpattern.

For the se ond test we used the photo of a alibration pattern. We

examined two families of primitives: uboids(Fig. ??) and parallelepipeds

(Fig. ??)withdierentsizeofedges. Weappliedalsoteststo ompareresults

for every family with known and unknown lengths of edges .In all ases we

assumed that the prin ipal point lies in the enter of the image. Thanks to

this test we made more dedu tions about dependen e of alibration results

on orientation and size of the primitive. Realvalues of intrinsi parameters

where omputedbystandardmethodusingknownpositionsof ontrolpoints

of the alibration patterninthe spa e.

Intables??and??wegiveexemplary alibrationresultsforprimitives

(37)

other hand we observe, that primitivestoo big, so with proje tions of

verti es losetobordersoftheimage,giveworstresultsthanprimitives

well entered. Wegiveexemplaryresultsforprimitiveswithknownand

unknown lengths of edges.

lenght 1 4 5 6

uboids 49.5 5.6 6.2 3.7

uboids+lengths 47.4 3.2 3.6 1.6

parallelepipedes - 3.0 2.5 0.1

parallelepipedes +lengths - 2.7 2.4 0.4

Table 3.1: Relativeerrors for f in %

lenght 1 4 5 6

uboids 14.6 3.7 2.9 2.8

uboids+lengths 9.7 2.2 2.8 2.0

parallelepipedes - 0.0 2.4 2.2

parallelepipedes+lengths - 1.1 1.1 0.0

Table 3.2: Relativeerrors for in %

Calibrationisinuen edbyexisten eofsquaredfa es. Figures??show

results. Three harts are made for three dierent values of the length

of the edge (l

3

) of the uboid. On the x axis are values of length l

2 .

The y axis presents results of fo al omputationfor dierent values of

l 2

. We an observe, that the best results are not for the biggest value

of l

2

,but forthe value losest tothe lengthl

3 .

3.5.3 Real s enes

The most interesting inour work were tests onimages of real s enes. In

this ase alibration results are not really the most signi ant. In pra ti e

wedonot knowintrinsi parametersof a amera,with whi h the imagewas

(38)

1

2

3

4

5

6

7 1250

1300

1350

1400

1450

l1

f

l3=4

f theor.

l2=3

l2=4

l2=5

l2=6

l2=7

(a)l 3 =3

1

2

3

4

5

6

7 1250

1300

1350

1400

1450

l1

f

l3=5

f theor.

l2=3

l2=4

l2=5

l2=6

l2=7

(b)l 3 =4

1

2

3

4

5

6

7 1250

1300

1350

1400

1450

l1

f

l3=6

f theor.

l2=3

l2=4

l2=5

l2=6

l2=7

( )l 3 =5

(39)

We will des ribe then s enes, whi h had been re onstru ted and problems

(40)

Constru tion of textured 3D

model

The aimof this proje t was to onstru t 3D models of s enes from one

image. The onstru tion of models ontains two problems: lo alization of

model points in the 3D spa e and photo realisti rendering of surfa es. In

this hapter wedes ribeour approa hto these problems.

There are many approa hes to the onstru tion of 3D models. Having

alibratedthe amera,we aneasilyobtainproje tionraysalongwhi hmodel

points are situated. The problem of 3D re onstru tion lies in nding exa t

positions of points on proje tion rays. Most of the resear h that has been

done in this domain uses two or more images to obtain positions of model

points in spa e. For single images Peter Sturm and Stephen Maybank in

[SM99a℄ propose a method based on sets of planes and sets of pointslinked

together. Antonio Criminisi and Andrew Zisserman proposed another

ap-proa h in [CRZ99℄. It uses measurements onplanes and lines for sequential

re onstru tion of 3D oordinates of points.

Inourworkweusedtheideaintrodu edin[CSC99 ℄for omputingrelative

depths of parallelepiped's orners. Weevaluateda methodof re onstru tion

of s enes onsisting of a set of onne ted parallelograms,whi h is des ribed

in the rst se tion.

The laststage of the proje t wasthe extra tionof photometri

informa-tion. Textures obtained by simple interpolation of olors on surfa es have

(41)

of models.

Finally,wepresentresultsofsomere onstru tionsfromimagesofoutdoor

and indoors enes.

4.1 3D points re onstru tion

We assume that the amera is alibrated and so we know the matrix K

of intrinsi ameraparameters. This enablesus toba kproje t imagepoints

to 3D along their proje tion rays. The problem of re onstru tion is now to

nd their exa t positions onproje tion rays.

A 3D pointwhose image isgiven by qhas oordinates

Q= q 0 1 (4.1) where q 0 K 1 q and jjq 0 jj = 1. q 0

is alled the viewing dire tion. The

unknown expresses the distan e of Q form the opti al enter and hen e

denes its position onthe proje tion ray (Fig. ??).

000

111

000

111

000

111

000

111

000

111

000

111

00

11 00000000000000000000000000000000000000000000000

00000000000000000000000000000000000000000000000

11111111111111111111111111111111111111111111111

000000000000000000000000

111111111111111111111111

000000000000000000000000000000000

111111111111111111111111111111111

000000000000000000000000000000000000000000000000000000000000

111111111111111111111111111111111111111111111111111111111111

000000000000000000000000000000000000000000000000000000

111111111111111111111111111111111111111111111111111111

0000000000000000000

1111111111111111111

000000000000000000000000000000000000000000000000000000

111111111111111111111111111111111111111111111111111111

q’

q

Q

λ

projection center

image plane

Figure4.1: Viewing dire tions of image points.

For every parallelogram inspa e dened asshown inFigure ?? we have

(42)

Q4

Q3

Q2

Q1

Figure4.2: Constrainton 3D positions of parallelogramverti es.

Q 1 Q 2 +Q 3 Q 4 = 0 (4.2)

This impliesthat dire tions q

0

i

the parallelogram'sverti esprovidethree

equations for points depths

1 ; 2 ; 3 ; 4 : 1 0 q 0 11 q 0 12 q 0 13 1 A 2 0 q 0 21 q 0 22 q 0 23 1 A + 3 0 q 0 31 q 0 32 q 0 33 1 A 4 0 q 0 41 q 0 42 q 0 43 1 A =0 (4.3)

If we x one depth,we obtain remainingthree depths. Also,for a set of

parallelograms onne tedbyatleastonepoint,itissu ienttosetarbitrarily

one depth to ompute the positions of all verti es of parallelogram. If we

know at least one length in the s ene we an x the depths to obtain the

eu lidean modelpreserving not onlyratiosof lengths,butalsotheirabsolute

values.

In ourmethodwe reateatwodire tionalgraph,whosenodes arepoints

of parallelograms and onne tions are edges of parallelograms. Next we

hoose thelargest onnexsubsetof thegraphand weenumeratenodes. Now

(43)

equa-0 B B B B B q 0 11 q 0 21 q 0 31 q 0 41 0 0 0 0 ::: q 0 12 q 0 22 q 0 32 q 0 42 0 0 0 0 ::: q 0 13 q 0 23 q 0 33 q 0 43 0 0 0 0 ::: 0 0 0 q 0 41 q 0 51 q 0 61 q 0 71 0 ::: . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 C C C C C A 0 B B B B B B B B B B B B B B B 1 2 3 4 5 . . . . . . . . . . . . 1 C C C C C C C C C C C C C C C A = 0 B B B 0 0 . . . . . . 1 C C C A (4.4)

Afterxing onedepththe resolutionoftheequationsystemabovebrings

us informationabout depths of allpointsbelonging tothe set.

4.2 How to use photometri data for photo

re-alisti rendering

The last stage of 3D modela quisition is the extra tion of photometri

information. For this purpose we divided re onstru ted parallelograms into

triangles, adding triangles not belonging toany parallelogram, but with

re- onstru ted verti es. Textures obtained by simple interpolation of olors in

triangles in the image and trianglesin the modelare deformed. This is due

to the fa t, that the transformation from the s ene obje t to the image is

perspe tive, and the transformation from the triangle in the image to the

triangle in the 3D model is linear. Thus without any orre tion we an not

re onstru t the real pattern.

There are few approa hes tothis problem. We an for example

pre om-pute textures for ea h fa e to obtain the view as taken by a amera whose

opti al axis is parallel to the fa e's normal. In order to do this we have to

ompute the rotationbetween the opti al axis(in this work weassume that

it is the z-axis) and the fa e's normal. This allows to nd the homography

between the polygon onthe imageand polygon onnew texture.

Toobtain photorealisti renderings we applied amethodof texture

(44)

proje -The dieren ebetween rendering with and without the perspe tive

or-re tion is shown onthe Figure??.

(a) Model withtexture

or-re tion.

(b) Model without texture

orre tion.

Figure 4.3: Example of texture orre tion.

4.3 Experimental results

Totest our methods, we onstru ted several models from photos of real

s enes. We used both outdoor and indoor s enes. In this se tion we shortly

des ribe hara teristi sof every s ene, give examples of problems whi h an

o urduringre onstru tionandpresentourresults. Ea hre onstru tionwas

made with the assumption, that the prin ipal point liesin the enter of the

image.

Our rst re onstru tion was the Sioux building. It is an example of a

simple ountry house. If we do not need very detailed re onstru tion

we an dene main obje t by only two primitives. It means that

re- onstru tion is very qui k. As is shown on the Figure ?? angles are

(45)

on the modelredu e this ee t.

The Notre-Dame pla e in Grenoble. It was dened as uboid (the

tower) and set of parallelograms. During denition of the model we

had todenevery arefulythefa ebetween thetowerandhouses. It is

very narrow, soquitesensitive toin orre t data. These ond di ulty

was the fa tthat anglesare not really straight inthe real s ene- walls

of houses are not really at, and also fronts of houses are not really

paralleltothefrontofthe tower. Butasinthepreviousre onstru tion

wesee(Fig. ??),thatre onstru tedanglesarequite losetothereality.

The photo was taken by a amera that is known to have unit aspe t

ratio. The value ofthe aspe t ratio obtainedbyour methodis0:97, so

very lose tothe real one.

Torres Kio from Puerta Europa in Madrid. This is an example of a

modernstru ture whi h annotbedenedby a uboid. For alibration

we used only two onstraints provided by two pairs of perpendi ular

dire tions. Therelativeerrorobtainedforre onstru tedangleswasless

than 3%, and less then2% for lengths'ratios. Some views ofthe s ene

are shown onFigure ??.

The indoor s ene of the room under the roof (Fig. ??). To dene

ontrolpointsof uboids weused ontrolmarks. Thisphotowastaken

with thesmallestfo alofthe amera. Inresult,we an seethe ee tof

distortiononthe image. It ausedseveral problems with olinearityof

pointsand oplanarityoffa esinre onstru tedmodel(Figure??). But

general geometry of the s ene is ni ely re onstru ted. Also textures,

even onfa esnot verywellseen by the amera,are wellre onstru ted.

Generally we an see on gures above, that angles are usually well

re on-stru ted. What is also important, that we an obtain good slide of the

(46)

( ) (d)

Figure 4.4: (a)-the original image of the Sioux building; (b)-( )-(d)-photos

(47)

( ) (d)

Figure 4.5: (a)-the original image of Notre Dame pla e; (b)-( )-(d)- photos

(48)

( ) (d)

(49)

( ) (d)

Figure 4.7: (a)-the original image of indoor s ene;(b)-( )-(d)-photos from

(50)

(51)

Con lusion

Inthisworkwehaveapplied omputervisiontheorytodevelopamethod

of 3D model a quisition from single images. We were interested mostly in

outdoorand indoor urbans enes.

In aseofasingleimage,weneedadditionalinformationaboutthe

topol-ogy of the s ene to be able to alibrate the amera and then onstru t the

eu lidean 3D model. In our approa h, to obtain the ne essary data, we

employed primitivesappearinginar hite tural environments. The most

fre-quent and very useful primitivesare parallelepipeds and parallelograms.

At the rst stage we studied onstraints on ameraintrinsi parameters

provided by prior knowledge about parameters of parallelepipeds appearing

in the s ene. The method of amera alibration was based on the method

of alibration based on uboids proposed by Fabien Vignon and Edmond

Boyerin[Vig99℄. Our methodallows usingnot only onstraintsprovided by

perpendi ular dire tions, but all the information about lengths and angles

of parallelepipeds. One ompletely dened primitive allows for omplete

alibrationofthe amera. Itpermitsalsoto alibratethe amerainotherwise

degenerated ases. Experimental tests of alibration brought good results

even in the presen e of noisydata.

In the se ond stage we developed a method for re onstru ting

three-dimensional models based on sets of onne ted parallelograms. It o urs

that su h stru tures are su ient to dene many urban s enes. Results of

re onstru tions of models from photos of real s enes demonstrate good

(52)

pendent modules: alibration, re onstru tion and modelvisualization. The

module for photorealisti rendering of the three-dimensional model was

im-plemented using OpenGL.

There are several extensions that may be added to our method. For

ex-ample other primitives an be added, like uboids with triangle prisms on

the top, spheres or ylinders. On the other hand an intera tive system

as-sisting indenition of additionalprimitivesinpartiallyre onstru ted s enes

ould be developed. Also, intera tive methods for ombined re onstru tion

of twonon interse ting sets of onne ted parallelograms ouldbedeveloped.

Interesting results ould be obtained by joining models re onstru ted from

photos of dierent parts of a s ene. Dependen ies provided by known

rela-tive positions between primitives and proje tions of one primitiveinseveral