HAL Id: inria-00000127
https://hal.inria.fr/inria-00000127
Submitted on 28 Jun 2005
HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub-
L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non,
Mobile Terminals
Kadi Bouatouch, Gérald Point, Gwenola Thomas
To cite this version:
Kadi Bouatouch, Gérald Point, Gwenola Thomas. A Client-Server Approach to Image-Based Render- ing on Mobile Terminals. [Research Report] RR-5447, INRIA. 2005, pp.29. �inria-00000127�
a p p o r t
d e r e c h e r c h e
Thème COG
A Client-Server Approach to Image-Based Rendering on Mobile Terminals
Gwenola Thomas — Gérald Point — Kadi Bouatouch
N° 5447
January 2005
Unité de recherche INRIA Futurs
Gw enolaThomas
∗
,Gérald Point
∗
, Kadi Bouatouh
† ∗
ThèmeCOG Systèmesognitifs
ProjetsIPARLA
Rapportdereherhe n°5447 January200529pages
Abstrat: This report shows howto use IBRmethods to makepossible therenderingof
omplexsenesonamobileterminal,suhasaPDA,whileusingalient/serverarhiteture.
ThePDA representsthelientof aserverwhih omputesa verysmall set of keyimages
(toavoidlatenytimethatwouldaetinterativity)ofaomplex3Dseneandtransmits
them on demand to the lient arossa low bandwidth network. The lient utilizes these
imagestouseawarpingtehniquetoomputenewimagesasseenbyintermediateameras
(using an IBR tehnique) whose positions and diretions are hosen interatively by the
userbymovingthestylusofaPDA.Themostdiultproblemishowtoplaetheameras
(apturingthekeyimages)whihallowaneientwarpingavoidingartifats,suhasholes,
due to olusions and exposures. Providing a general solution to the problem of amera
plaementisahardtask. Inthisreportweaddressonlytheaseofurbansenes.
Key-words: lient/server,3Dsenes,Rendering,Warping,ImageBasedRendering,Low-
BandwidthNetwork ,PDA,CameraPlaement
∗
LaBRI,INRIAFuturs,Bordeaux
†
IRISA,Rennes
Résumé : Ce rapport montre qu'il est possible d'utiliser des méthodes IBR (rendu basé
image)poureetuerlerendudesènesomplexessurunterminalmobile,telqu'unPDA,
dans le as d'une arhiteture lient/serveur. Le PDA représente le lient d'un serveur
quialuleunensemblerestreintd'imageslés(pouréviteruntemps delateneimportant
qui pourrait dégrader l'interativité) d'une sène 3D omplexe et les transmet à travers
un réseau bas débit sur demande. Le lient utilise es images lés pour en déduire par
interpolation(enutilisantdondesméthodesIBR)denouvellesimagesintermédiairesvues
pardes amérasdont l'orientation et la position sont hoisiespar l'utilisateur de manière
interativeendéplaçantunstylet. Unproblèmedéliatest eluiduplaementdesaméras
pouréviterl'apparitiondetrousdusauxproblèmesd'olusionetd'apparitiondenouveaux
objetslorsqu'oneetue l'interpolation. Conevoirune solutiongénérale àe problèmeest
unetâhetrèsdiile. Danserapportnousproposonsunesolutiondeplaementdeaméra
dansleasdesènesurbaines.
Mots-lés : lient/serveur,Sènes 3D,Rendu,Interpolation,Rendu BaséImage,Réseau
BasDébit,PDA, PlaementdeCaméra
Contents
1 Introdution 4
2 Related works 5
2.1 3DrenderingonPDA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.2 Image-Based-Rendering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.3 Cameraplaement: Determiningrefereneimages. . . . . . . . . . . . . . . . 8
3 Overalllient-server arhiteture 9 4 Seletingthe referene imageson the server side 12 4.1 Streetsnetworkextration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
4.2 Cameraplaement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
4.3 Seletionofrefereneamerasduringnavigation . . . . . . . . . . . . . . . . 18
5 Implementationand results 19 5.1 Serverdatastrutures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.1.1 Visible buildingedges . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
5.1.2 Cameraupdateheking. . . . . . . . . . . . . . . . . . . . . . . . . . 20
5.1.3 Contributionofaamera . . . . . . . . . . . . . . . . . . . . . . . . . 21
5.2 Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
5.2.1 Ourlient/serverappliation . . . . . . . . . . . . . . . . . . . . . . . 22
5.2.2 Testsenes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
5.2.3 Someremarksaboutoating-pointandxed-pointarithmetis . . . . 23
5.2.4 Qualityoftherendering . . . . . . . . . . . . . . . . . . . . . . . . . . 24
5.2.5 Framerate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
6 Conlusion 27
1 Introdution
PDAs(PersonalDigitalAssistants)arehandheldomputersthatareinreasinglywidespread
sinethe last deade. Many appliations alreadyrun on PDAs but omplete high quality
renderingof3Dmodelsstillremainbeyondtheirapabilities. TomakerenderingonPDAs
possible,onesolutionistorelyonalient/serverarhitetureinwhihtheserveromputes
images of a3D sene then sends them to a lient, say a PDA, whih visualizes them on
its small sreen. As this solution is highly demanding in terms of network bandwith, a
preferablesolutionis to distributerendering amongthe serverandthe lient. Indeed,the
serveromputes a set of key images that are sent to a lient whih omputes in-between
imagesusingIBRtehniques(ImageBasedRendering).
Whiletraditionalrenderingmethods needdatarepresentingthegeometryandthepho-
tometryof theobjetsmaking upa3Dsene, IBR methods takeasinputaset of images
(synthetiorreal)sometimesaugmentedwithdepthmaps. Whenrenderingomplexsenes,
theomputationostoftraditionalrenderingisproportionaltothenumberofobjetswithin
asene, while itis onlyproportionalto theimage resolutionfor IBR methods. IBR teh-
niquesproved that theyarefastand easyto implement. Theyonly onsistin alulating,
forintermediateamerapositions,in-betweenframesfromkeyframesanddepth-maps.
This papershows howto use IBRmethods tomakepossiblethe renderingofomplex
senes on aPDA in the framework alient/server arhiteture. ThePDA representsthe
lient of a server whih omputes a very small set of key images (to avoid lateny time
that would aet interativity)of a omplex3D sene and transmits them ondemand to
the lient aross a low bandwidth network. The lient utilizes these images to ompute
newimagesasseenbyintermediateameras(usinganIBRtehnique)whosepositionsand
diretionsarehoseninterativelybytheuserbymovingthestylusofaPDA.
Image-Based-Renderingseemsagood ompromisebetweenlassial3Drenderingon a
PDAand streamingimages that areallomputedontheserverside. Inother words,IBR
is a good ompromise between proessing time on aPDA and time of data transmission
throughalowbandwidthnetworksuhasGPRSorwirelessnetworks(Wiorothers).
AnotheradvantageofIBRtehniquesisthepossibilitytointeratwitha3Dsenewhile
it is hardly possible when streaming images from the server to the lient. Warping key
images to ompute in-betweenones givesthe userthe feeling of navigatingthrough a3D
sene.
Themostdiultproblemishowtoplaetheameras(apturingthekeyimages)whih
allowaneientwarpingavoidingartifats,suhasholes,duetoolusionsandexposures.
Providing a general solution to the problem of amera plaement is a hard task. In this
reportweaddressonlytheaseofurbansenes.
This report is organized as follows. Setion 2 presents some related works regarding
renderingandimage-basedrenderingonPDAsaswellassolutionstotheameraplaement
problem. Our Client/Serverarhiteture is desribed in setion 3. Setion 4 presents in
detailoursolutiontotheameraplaementproblemintheaseofurbansenesonly. Some
implementationdetailsandresultsaregiveninsetion5. Finallyweonludeinsetion6.
2 Related works
Inthissetionwereport rston3DrenderingmethodsonPDAs, then onIBRtehniques
whilefousingonthesolutionsrelevanttoourmethod. Next,weaddresstheproblemofkey
imageseletion. Thisprobleman bestatedasaameraplaementproblem. Thesolution
weproposetothis problemisonlyvalid forurban senes.
2.1 3D rendering on PDA
Renderingomplex3D seneson PDAs annotbestraightforwardlyperformedbyreusing
existing software pakagesrunningon Personal Computersbeausethe PDAs are notyet
suppliedwithoatingpointunits anddediatedgraphisaeleratorshavejust beenavail-
able. Moreover, even if external memory ards of 256 Mega bytes an be used on these
devies,aesstimesarestilltoohigh.
However,3DrenderingofsmallsenesispossibleonPDAs. AnewgenerationofPDAs
basedon thenovel Intel XSalPXA 250proessorwith a400 Mhzlok speed providing
dediated support for multimedia and 3D graphis appliations has appeared. Some 3D
APIsdediated toprogrammingonPDAalreadyexist. Oneofthese toolsisa3Dgraphis
librarysimilar toOpenGL[15, 22℄. Torenderlargersenes, methods relyingonsimplied
geometry based on levels of details [27℄ or Non-Photorealisti-Modeling [9℄ an be used.
However,thesemethodsprovideimagesthatdonotseemrealistiandtheseneomplexity
isstillverylimited.
TooveromethelimitationsofPDAsandtomakepossiblerenderingofomplexseneson
thesedevies,oneanmakeuseofalient/serverarhiteture. Martin[19℄haslassiedinto
threemajorategoriesthe methodsforrendering3Dmodels inlient-serverenvironments.
Therst ategoryis alled lient-side methods [12℄. Themethods of this ategorydo not
involveany renderingonthe partof theserver. Thegeometry aswell asthe texturesare
downloaded to eah lient that requests them and the lient is responsible for rendering
it. Suh methods are not well suited for PDAs. When using the methods of the seond
ategory,alled server-sidemethods,the 3Dmodelis fullyrenderedontheserversideand
the resulting images are sent to the lients [6, 2℄. In[6℄, the sever generates the frames,
enodes andtransmitsthem to thelient. Theenoded framesare transmitted asavideo
streamtothelientwhihdeodesthestreamanddisplaysit. In[2℄,eahlientusesprevious
viewsofthesenetopreditnextviewusingimage-basedrenderingtehniques. Theserver
performsthesamepreditionandsends onlythedierenebetweenthepreditedandthe
atual views. Compressed dierene images require less bandwidth than the ompressed
imagesofeahframe. Suhmethodsgetinterestingwhenthelientshavelimitedresoures
and limitedgraphis performanes, whih is the aseof PDAs. Asfor the third ategory,
alledhybrid-sidemethods[17℄,partsofthe3Dmodelarerenderedontheserverandtheother
partsaredownloadedandrenderedonthelientside. Suh methodshavetheadvantageof
reduingthegeometri omplexityof thedata tobetransmitted byreplaing parts ofthe
geometrywithimages. However,deidingwhihpartofamodelshouldberenderedonthe
serveroron thelientisnot atrivialtask. Onepossibility isthat theserverould render
high andlow resolutionversionsof a3Dmodeland sendthe residualerrorimage andthe
low-resolution geometry to thelient [17℄. In this ase, the role of the lient is to render
the oarse model and to add the residual image to restore a full quality rendering. Suh
methodsarenotsuitedforPDAs,beauseforomplexsenes,alotofgeometry(evenoarse
models),texturesaswellasresidualimageshavetobetransmittedtothePDAs.
As seenabove, aserver-side methodhas to be used for renderingomplexsenes on a
PDA.Reallthatthiskindofmethodisbasedonstreamingimagestothelient. Streaming
imagesanoerrealismand iswell studiedto renderingonPDAs [3,6, 16,4℄. Evenwith
ahigh bandwidthnetwork,streaming isnottheperfetsolution. Alot oftime isspentby
thelienttodownloadandrenderimages[16℄,whihdramatiallyreduetheframerateon
thelientside. Moreover,whenwalkthroughisthetargetedtaskofstreamingappliations,
theproblemofinterativityisnoteientlydealtwith. NotethatintheaseofNPR(Non
PhotorealistiRendering)someworkshavebeendoneonstreamingsilhouettes,reasesand
featurelines(ratherthanimages)toaPDA[9,12℄. Inouropinion,whenrealistirendering
istargeted,streaminghastobeombinedwithIBRtehniquesasalreadydonein[16, 4℄.
Thedrawbaksof3Drendering(smallsenes,norealism)andstreaming(bottlenekfor
imagestransmissionandlakofinterativity)ledustoproposetheuseofIBRmethodsfor
renderingomplex3DenvironmentsonmobileterminalssuhasPDAs.
2.2 Image-Based-Rendering
Image-Based-Rendering(alledIBRfromnowon)isbasedontheassumptionthat thereis
aslightdierenebetweentwosuessiveimages whenwalkingthrougha3Denvironment.
Indeed,renderingataurrentpositionandanorientationanbeperformedusingIBRteh-
niqueswhihonsistinwarpingnearbypre-renderedimages(alledkeyimagesorreferene
images). IBRtehniquesareusedintwomainappliations: 3Dwalkthroughand3Dobjet
reonstrutionfromimages[7,21℄.
Afamous IBRwalkthroughsystemisQuikTimeVR[5℄whih allowsarenderingyielding
360-degreesylindrialpanoramiimages. Whenusingthismethod thedegreesoffreedom
ofnavigationarerotations,aroundtheupaxes,andzooms(inandout). Thislimitationin
degreesoffreedomfornavigationisdue tothewaytheintermediateimagesareomputed,
sayanetransformsofpanoramiimages[13℄. Warpingtehniquesoveromethisdrawbak
using3Dinformation.
Thepixelsofthewarpedimageareomputedbyre-projetionofthepixelsofkeyimages
(seegure1). Warpingisonlypossibleifdepthinformationisavailableforeahpixelofthe
keyimages [20℄, [18℄. Equation1 isthe generalequationof 3D warping,where image 1 is
thekeyimageandimage2thewrappedone.
C˙1 and C˙2 are the enters of projetion of the 2 images, P1 and P2 are the inverse
projetionmatries. Thesematriesdenetheintrisiparametersoftheamera(seegure
1).
¯ u2
δ(¯u2)=P2−1P1
¯ u1
δ(¯u1)+P2−1( ˙C1−C˙2) (1) withu¯1=
u1
v1
1
and u¯2=
u2
v2
1
(2)
a
c b
P2 =
ax bx cx ay by cy az bz cz
Figure1: 3DwarpingofapointX˙ andintrisiparametersoftheameraP2
Intheequation1,δ(¯u)is thedisparityofpixel u¯. δ(¯u) =Sz,whereS isthefoallength
andzthedepthofthepixel. Duringre-projetiononimage2,several3Dpointsanrepro-
jetontothesamepixel. TosolvethisvisibilityproblemwithoutusingaZ-Buer,MMillan
[20℄proposestowarpeahimagein olusionompatibleorder.
Warping generates exposure errors. Exposure errors ourwhen the motion of the view-
pointrevealsregionsofthemodelthatwerenotseenintherefereneimages. Tosolvethis
problem, the use of several and well hosen referene images is neessary (amera plae-
ment). LayeredDepth Images (LDI)ombine together anumber of referene images [25℄
[23℄. Multiple pairs(olor, depth) are assoiated whith eah pixel ofan LDI. Advantages
of LDIs are that they naturallyavoid redundany between referene images, they an be
warped inolusionompatible orderand areapable ofreduingthenumberof exposure
errors. Themain drawbakof LDIs methods isthat theyare demanding in memory size.
Interestingstrategiesaimingathoosingpertinentrefereneimageshavealsobeenproposed
in[10℄. There aredesribedinsetion2.3.
ThelosestrelatedworkisbyHudsonandMark[14℄whoalsoproposetouseIBRteh-
niques in the aseof a lient /serverapproah. They propose an algorithm forreferene
ameraplaement. Thisfundamentalproblemaswellasrelatedworksaredisussedinse-
tion2.3.
2.3 Camera plaement: Determining referene images
Cameraplaementonsistsinoveringeveryvisiblesurfaewithaminimumnumberofam-
erasto avoidexposuresandolusionswhenusingIBR.Thisproblemanalsoberegarded
as an extension of the Art Gallery Problem. The formal solution to this problem is the
aspetgraphs[24℄thatstoreallthevisibilityrelationsbetweenalltheobjetsinthesene.
Anaspetgraphontainsallthevisualevents(exposures,olusions)thatourin asene
andthat anthereforebeusedto pre-renderimages fromameraswhoseloationsdepend
onthevisualevents.
Thesolutionstotheproblemofameraplaementdependontheappliations. Whenthe
appliationis3D-reontrutionahugeset ofimagesareapturedbyforexampleturning a
ameraaroundthe3Dobjet[21℄orusingfewsigniantimages[7℄. Whentheappliation
isnavigation,theamerasthatproduetherefereneimagesanbeplaedat ertainpre-
denedpositions[18℄[14℄ortherefereneimagesanbeoneforallpre-omputedaording
toaertainstrategy[10℄,[1℄.
In[14℄threesetsofdepthimagesareused. Eahset ontainsfourimagesthatformthe
faesof aube. These faesare theimage planesoffour ameraspositioned at theube's
enter. Theurrentnavigationameralieswithin atrianglewhosevertiesare theenters
ofthreerefereneubes. Thisallowstoeientlywarptheurrentnavigationamera. The
targetedappliationof[14℄iswalkthroughofindoorsenvironments. Unlikethismethod,our
objetiveiswalkthroughof itymodels. With this aimin view,weproposeanewamera
plaementstrategywellsuitedtourbansenes.
OurmethodisinspiredbyFleishman'swork[10℄. Theseneisdividedintoviewingareas.
Ineahviewingarea,asmall(notneessaryminimal)setofamerassamplesishosen. Our
ameraplaementalgorithmtakestwoonsiderationsintoaount: (i)everypolygonshould
beovered(say, seenby at least onereferene amera),(ii) everyoveredpolygon should
beoveredatasuient overagerate. A overagerateof apolygonis theratiobetween
its areaand that of its projetiononto the image plane of the refereneamera that sees
thispolygon. Toavoidredundany,only oneamerais assoiatedwith agivenpolygon in
the sene. While in [10℄ amerasare plaed on the boundary of arbitrary walkingzones,
weplaerefereneamerasontheurbanstreetnetwork. Theplaedrefereneameraswill
aptureimagessimilartothoseapedestrian ouldsee whilewalkingalongastreet.
3 Overall lient-server arhiteture
The overall lient-serverarhiteture and the global algorithm of our system is desribed
in gure2. The serverowns the3D environmentand runs aameraplaementalgorithm
apable of determining pertinent referene ameras that apture referene images of the
environment, usedfor image-basedrendering. The ameraplaementalgorithm aswell as
theassoiateddatastruturesaredesribedinthenextsetion. Therefereneimages,one
omputedbytheserver,aresenttothelientthatwarpsthemtoomputenewintermediate
images.
server
− Renders 3D scenes
− Produces Reference Images
client
− Navigator
− Warps between reference images 3D data
user
2. Navigate(theta, d) 2.2 Produce()
images
2.5 [update] Send_ReferenceImages({IRefi}) 2.4 update = Update_ReferenceImages({IRefi},Mc)
1. Send_Init(IRef0) 2.3 Send_NavigationCamera(Mc)
2.1. Update_NavigationCamerat()
Figure2: Overviewoftherenderingarhiteture
Here isamoredetailed desriptionofthegure2:
1. TheIBRproessisinitializedwhentheserversendsthelientaninitialrefereneimage
togetherwithitsorrespondingameraparameters(1.Send_Init(IRef0)).Fromnow
on,IRefrepresentsarefereneimageanditsorrespondingameraparameters.
2. Onthelientside,theuserannavigatethroughthe3Denvironmentbyhangingthe
orientation and the position ofthe amera (2.Navigate(theta,d)). Inthe present
appliation,navigationisperformedinaurbanenvironment. Thepositionoftheam-
erais onstrainedtolieonahorizontalplane. Sudden hangesofameraorientation
are notallowed. These few degreesof freedom of theamera limithangesbetween
twosuessiveimagesto maketheIBRapproahpossible. Theselimitationsarealso
oherentwiththewaypeoplewalkin aity.
3. Whenevertheusermovesthenavigationamera,thelientomputesanewimageby
warpingsomeof the available referene images (2.1.Update_NavigationCamera(),
2.2.Produe()). Theavailablerefereneimagesarenotalwaysappropriateforwarp-
ing,that istosaytherefereneamerasthatproduedtheavailablerefereneimages
anbetoofarfrom theurrentnavigationamera,whihmayausethe appearane
