Christian-Albrechts-University, Kiel, Germany
9.6 RECENT TRENDS IN DYNAMIC IBR 171 For visualization of polygonal models this is obvious, but today’s programmable GPUs
allow much more complex algorithms to be executed. For nearly every method presented, a real-time version using GPU support exists. Real-time interaction in this case means, that the user can move the virtual camera interactively and the virtual views are generated with 10–50 frames per second. Most often, precomputation requires much more time, for example dense depth estimation, camera pose estimation, calculation of optimized intermediate data structures. The step towards dynamic scenes requires to reduce and speed up these steps so that video sequences from several cameras can be processed and the user can control the virtual camera at interactive rates.
Some recent real-time systems use volumetric modelling. Volumetric models are similar to medical computer tomography datasets. Typically, the 3D space is partitioned into volume elements called voxels. For each voxel it is determined if it belongs to an object or not.
This representation can also be constructed from depth maps, but the more popular approach is called ‘shape-from-silhouette’. A visual hull is computed for objects by intersecting the silhouette cones from different real views. Li et al. (2003) use shape-from-silhouette to construct the visual hull by back-projecting images and using alpha and stencil calculation.
Chapter 8 is devoted to volumetric reconstruction. To generate novel views from volumetric models different methods are available.
One common approach for IBR view synthesis of dynamic scenes is shown in Figure 9.11.
The pipeline processing starts by capturing the scene using multiple calibrated fixed cameras.
After object segmentation, shape-from-silhouette algorithms are used to create a volumetric model on the fly. After conversion to a surface model using polygon meshing, the standard polygonal rendering with view dependent texture mapping is used for visualization.
Saito et al. (1999) use 49 calibrated cameras, and compute a volumetric model. This is transferred into a polygonal surface model and during rendering it is used to generate correspondences in selected real views. From these correspondences per-pixel interpolation is performed after the determination of the disparity vectors.
Yamazakiet al.(2002) introducedbillboards. A billboard ormicro-facet is a small poly-gon, always facing the virtual camera. They approximate surface models, resample them to
rendered
Figure 9.11 IBR pipeline for view generation with visual hulls
binary voxel-models and create a multi-resolution octree. Depth maps are used for per-pixel visibility culling to prevent texturing facets with inappropriate pixel.
Goldluecke and Magnor (2003) calculate volumetric models from different views with a shape-from-silhouette approach. This model is then rendered using billboards textured from original views. It is also possible to convert volumetric models into surface models, one approach is discussed in Chapter 3.
Recently, direct depth-based view interplation for dynamic scenes was proposed by Zitnick et al. (2004). They capture dynamic scenes from a set of fixed video cameras and compute depth-compensated view interpolation from multiple views interactively. The results look promising and show that indeed the challenge of interactive free-viewpoint video can be mastered in the near future.
REFERENCES
Adelson E and Bergen E 1991 The plenoptic function and the elements of early vision.Computation models of visual processing. MIT Press, pp. 385–394.
Antone M and Teller S 2002 Extrinsic calibration of omni-directional image networks.International Journal of Computer Vision49(2–3), 143–174.
Avidan S and Shashua A 1997 Novel view synthesis in tensor space, p. 1034.
Baker S and Nayar SK 1998 A theory of catadioptric image formation. Proceedings of the IEEE Inter-national Conference on Computer Vision, Bombay, pp. 33–42.
Bakstein H and Pajdla T 2003a Non-central cameras for 3d reconstrution.Proceedings of Workshop 2003, Czech Technical University in Prague. CTU Publishing House, Faculty of Architecture of CTU, Prague, Czech Republic, pp. 240–241.
Bakstein H and Pajdla T 2003b Ray space volume of omnidirectional 180×360 deg. images. In Com-puter Vision — CVWW’03 : Proceedings of the 8th ComCom-puter Vision Winter Work-shop(ed. Drbohlav O), Czech Pattern Recognition Society, Prague, Czech Republic, pp. 39–44.
Bakstein H and Pajdla T 2003c Rendering novel views for a set of omnidirectional mosaci images.
Proceeding of Omnivis 2003: Workshop on Omnidirectional Vision and Camera Networks, IEEE computer Society Press, Los Alamitos, USA, p. 6.
Bolles RC and Baker HH 1986 Epipolar-plane image analysis: A technique for analyzing motion sequences.Technical Report 377, AICenter, SRI International, 333 Ravenswood Ave., Menlo Park, CA 94025.
Buehler C, Bosse M, McMillan L, Gortler SJ and Cohn Mf 2001 Unstructured lumigraph ren-dering. InSIGGRAPH 2001, Computer Graphics Proceedings (ed. Fiume E), ACM Press/ACM SIGGRAPH, pp. 425–432.
Chai JX, Chan SC, Shum HY and Tong X 2000 Plenoptic sampling. Proceedings of the 27th annual conference on Computer graphics and interactive techniques, ACM Press/Addison–Wesley, pp. 307–318.
Chang NL and Zakhor A 1999 A Multivalued Representation For View Synthesis.Proceedings of the International Conference on Image Processing (ICIP)Kobe, Japan, pp. 505–509.
Chen S and Williams L 1993 view interpolation for image synthesisSiggraph 1993, Computer Graphics Proceedings, pp. 279–288.
Chen SE 1995 QuickTime VR — an image-based approach to virtual environment navigation. Com-puter Graphics29(Annual Conference Series), 29–38.
Chen WC, Bouguet JY, Chu MH and Grzeszczuk R 2002 Light field mapping: Efficient representa-tion and hardware rendering of surface light fields. In SIGGRAPH 2002 Conference Proceedings (ed. Hughes J), Annual Conference Series. ACM Press/ACM SIGGRAPH, pp. 447–456.
Collins. R 1996 a space-sweep approach to true multi-image matching. Proceedings of Computer Vision and Pattern Recognition Conference, pp. 358–363.
Cooke E, Kauff P and Schreer O 2002 Imaged-based rendering for tele-conference systems. Pro-ceedings of WSCG 2001, 9th Int. Conference on Computer Graphics, Visualization and Computer Vision, Plzen, Czech Republic, p. 119.
REFERENCES 173 Debevec P 1998 Rendering Synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography. Computer Graphics 32 (Annual Conference Series), 189–198
Debevec P, Yu Y and Boshokov G 1998 Efficient view-dependent image-based rendering with pro-jective texture-mapping.Technical Report CSD-98–1003, University of California, Berkelely.
Evers-Senne JF and Koch 2003 Image based interactive rendering with view dependent geometry.
Eurographics 2003Computer Graphics Forum Eurographics Association, pp. 573–582.
Evers-Senne JF, Woetzel J and Koch R 2004 Modelling and rendering of complex scenes with a multi-camera rig, pp. 11–19.
Fehn C 2004 Depth-image-based rendering (dibr), compression and transmission for as new approach on 3D-TV.Proceedings Stereoscopic Displays and Applications, San Jose, CA, USA.
Fujii T 1994A Basic Study in the Integrated 3-D visual Communication. PhD Thesis University of Tokyo.
Fujii T and Tanimoto M 2002 Free Viewpoint TV system based on ray-space representation. Three-Dimensional TV, Video, and Display4864, 175–189.
Geyer C and Daniilidis K 2003 Omnidirectional Video.The Visual Computer, pp. 405–416.
Goldluecke B and Magnor M 2003 Real-time, free-viewpoint video rendering from volumetric geom-etry. In Visual Communications and Image Processing 2003 (ed. Ebrahimi T and Sikora T), Proceedings of SPIE,5150, pp. 1152–1158.
Gortler SJ, Grzeszczuk R, Szeliski R and Cohen MF 1996 The Lumigraph.Proceedings SIGGRAPH
’9630(Annual Conference Series), 43–54.
Heigl B, Koch R and Pollefeys M 1999 Plenoptic modeling and rendering from image sequences taken by a hand-held cameraProceedings of DAGM 1999, pp. 94–101.
Isaksen A, McMillan L and Gortler SJ 2000 Dynamically reparameterized light fields. InSiggraph 2000, Computer Graphics Proceedings(ed. Akeley K), ACM Press /ACM SIGGRAPH /Addison Wesley Longman, pp. 297–306.
Kang SB 1997 A survey of image-based rendering techniques.Technical Report, DEC Cambridge Research.
Kang SB, Uyttendaele M, Winder S and Szeliski R 2003 High dyunamic range video.ACM Trans.
Graph.22(3), 319–325.
Kawasaki H, Ikeuchi K and Sakauchi M 2001 Light field rendering for large-scale scenes.Computer Vision and Pattern Recognition (CVPR), Hawaii, USA, p. 2.
Klette R, Gimel’farb G, Wei S, Huang F, Scheibe K, Scheele M, Börner A and Reulk R 2003 On design and applications of cylindrical panoramas.Proceedings, Computer Analysis of Images and Patterns, Groningen, The Netherlands, pp. 1–8.
Koch R, Pollefeys M and Gool LV 1998 Multi viewpoint stereo from uncalibrated video sequences Proceedings ECCV’98number 1406 inLNCS. Springer, Freiburg, pp. 55–71.
Kurashima C, Yang R and Lastra A 2002 Combining approximate geometry with view-dependent texture mapping.XV Barzilian Symposium on Computer Graphics and Image Processing, Fortaleza, CE, Brazil, pp. 112–120.
Laveau S and Faugeras O 1997 3D representation as a collection of images.Proceeding of the IEEE Int. Conf. on Pattern Recogniton (CVPR’97)IEEE Publishers, pp. 689–691.
Levoy M and Hanrahan P 1996 Light field rendering. Proceedings SIGGRAPH ’96 30 (Annual Conference Series), 31–42.
Li M, Magnor M and Seidel Hp 2003 Improved hardware-accelerated visual hull rendering. Proceed-ings, Vision, Modeling, and Visualization (VMV-2003), Munich, Germany, pp. 151–158.
Lippman A 1980 Movie-maps: An application of the optical videodisc to computer graphicsProc.
ACM SIGGRAPH, pp. 32–42.
McMillan L 1999 Image-based rendering using image-warping — motivations and background.
computer Graphics (SIGGRAPH’99), course n. 39, pp. 61–64.
Naemura T, Yoshida T, and Harashima H 2001 3D computer graphics based on intergal photography.
Optics Express8, 255–262.
Peleg S and Herman J 1997 Panoramic mosaics by manifold projection.CVPR97, pp. 338–343.
Rademacher P and Bishop G 1998 Multiple-centre-of-projection images. Computer Graphics (SIGGRAPH’98),pp. 199–206.
Rousso B, Peleg S and Finci I 1997 Mosaicing with genralized strips. DARPA97, pp. 225–260.
Rousso B, Peleg S, Finci I and Rav-Acha A 1998 Univesal mosaicing using pipe projection.ICCV 98, pp. 945–952.
Saito H, Baba S, Kimura M, Vedual S and Kanade T 1999 Appearance-based virtual view genera-tion of temporally-varying events from multi-camera images in the 3d room. Proceedings of 2nd International Conference on 3D Digital Imaging and Modeling, pp. 516–525.
Schaufler G and Priglinger M 1999 Efficent displacement mapping by image warping,Proceedings of the 10th Eurographics Workshop on Rendering, pp. 175–186.
Seitz SM and Dyer CR 1996 View morphing.SIGGRAPH 96, pp. 21–30.
Shade J, Gortler S, He LW and Szeliski R 1998 Layered depth images.Proceedings ACM SIGGRAPH, ACM Press / ACM SIGGRAPH, pp. 231–242.
Shum HY and Szeliski R 1997 Panoramic image mosaics.Technical Report, Microsoft Research.
Shum HY, Wang L, Chai J and Tong X 2002 Rendering by manifold hopping.International Journal of Computer Vision (IJCV), 185–201.
Shum HY and He LW 1999 Rendering with concentric mosaics.Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, ACM Press/Addison–Wesley Pub-lishing, pp. 299–306.
Shum HY and Kang SB 2000 A review of image-based rendering techniques.Proceedings, Visual Communications and Image Processing, pp. 2–13.
Takahashi T, Kawasaki H, Ikeuchi K and Sakauchi M 2000 Arbitrary view position and direction rendering for large-scale scenes.Proceedings CVPR 2000, pp. 296–303.
Teller S, Antone M, Bodnar Z, Bosse M, Coorg S, Jethwa M, and Master N 2003 Calibrated, registered images of an extended urban area.International Journal of Computer Vision, 93–107.
Tong X, Chai J and Shum HY 2002 Layered lumigraph with lod control.Journal of Visualization and Computer Animation, 249–261.
Vogelgsang C and Greiner G 2003 Interactive range map rendering with depth interval texture slicing.
Vision, Modelling and Visualization (VMV), Munich, Germany, pp. 477–484.
Weinshall D, Lee MS, Brodsky T, Trajkovic M and Feldman D 2002 New view generation with a bi-centric camera.Proceedings of the 7th European Conference on Computer Vision, Copenhagen, DK, pp. 614–618.
Yamazaki S, Sagawa R, Kawasaki, H and Ikeuchi K and Sakauchi M 2002 Microfacet billboarding.
Rendering Techniques 2002 (Eurographics Workshop Proceedings), pp. 169–179.
Yang R, Welch G and Bishop G 2002 Real-times consensus-based scene reconstruction using com-modity graphics hardware.Proceedings of Pacific Graphics, Tsinghua University, Beijing, China, pp. 207–214.
Zitnick C, Kang S, Uyttendaele M, Winder S and Szeliski R 2004 High-quality video view interpolation using a layered representation.Proceedings, ACM SIGGRAPH, Los Angeles, CA, pp. 600–608.
Zomet A, Feldman D, Peleg S and Weinshall D 2003 Mosaicing new views: The crossed-slits projection.IEEE Trans. on PAMIpp. 741–754.