Unifying the refocusing algorithms and parameterizations for traditional and focused plenoptic cameras

(1)

HAL Id: hal-02096378

https://hal.inria.fr/hal-02096378

Submitted on 15 May 2019

HAL is a multi-disciplinary open access

archive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Unifying the refocusing algorithms and

parameterizations for traditional and focused plenoptic cameras

Charlotte Herzog, Ombeline de la Rochefoucauld, Fabrice Harms, Philippe Zeitoun, Xavier Granier

To cite this version:

Charlotte Herzog, Ombeline de la Rochefoucauld, Fabrice Harms, Philippe Zeitoun, Xavier Granier. Unifying the refocusing algorithms and parameterizations for traditional and focused plenoptic cam-eras. ICCP 2019 - IEEE International Conference on Computational Photography 2019, May 2019, Tokyo, Japan. Institute of Electrical and Electronics Engineers, 2019. �hal-02096378�

(2)

Introduction

A plenoptic camera allows to acquire and separate spatial and directional information of the light coming from a scene [1, 2]. It allows applications such as

refocusing at different depths from the one where the image has been acquired. In the literature, different refocusing algorithms are presented for several

optical plenoptic configurations [3 - 6]. We have previously shown the continuity between these optical designs, and the similarities and differences

between the associated algorithms [7]. Here we propose a unique parameterization of the light rays in a plenoptic setup, allowing the development of a

unique refocusing algorithm valid for any plenoptic configurations, based on the integration of étendues’ intersections in the image-space. With this

method, we aim at refocusing images at any distances from the camera, without the discontinuity due to the change of optical configurations.

Unifying the refocusing algorithms and parameterizations for

traditional and focused plenoptic cameras

Charlotte Herzog

1,2,3

, Xavier Granier

2,3,4

, Fabrice Harms

1 , Philippe Zeitoun

5 & Ombeline de La Rochefoucauld

1

_{Imagine Optic, Orsay/Bordeaux, France;}

2

_{Laboratoire Photonique, Numérique et Nanosciences, Talence, France}

3

_{Inria Bordeaux Sud-Ouest;}

4

_{UMS Archéovision 3D SHS, Pessac, France;}

5

Laboratoire d’Optique Appliquée, UMR7639, Palaiseau, France

2. Parametrization of the Light-Field

Conclusions

• Unification of the two historical algorithms.

• Algorithm based on the intersection of étendues

instead of using pixels as points.

• First implementation of algorithm → validation of

the idea.

• Future work: improving execution time.

References

[1] E. Y. Lam, J. Opt. Soc. Am. A, vol. 32, no. 11, pp. 2021–2032, Nov. 2015.

[2] M. Martínez-Corral and B. Javidi, Adv. Opt. Photonics, vol. 10, no. 3, p. 512, 2018. [3] R. Ng, thesis, Stanford University, 2006.

[4] T. Georgiev and A. Lumsdaine, J. Electron. Imaging, vol. 19, no. 2, p. 021106, 2010.

[5] J. Fiss, B. Curless, and R. Szeliski, in 2014 IEEE International Conference on Computational Photography (ICCP), 2014, p. 5.

[6] M. Hog, N. Sabater, B. Vandame, and V. Drazic, IEEE Trans. Comput. Imaging, vol. 3, no. 4, pp. 811–821, Dec. 2017.

[7] C. Herzog et al., in Proc. SPIE 10677, Unconventional Optical Imaging, 2018, vol. 106772U, p. 104.

VOXEL project EU’s Horizon 2020 nº 665207

LOA

1. Validity domains of historical refocusing algorithms

Bijection between object-space and

image-space for each microlens k

For each pixel 𝑿 in the image to reconstruct & For each microlens k:

• Transform of the étendue defined by pixel X and main lens aperture U :

𝑄

𝑅

= 𝑀 𝑋

𝑈

+

𝑏

𝑓

₂

0 𝑘

• For each pixel i on the sensor:

• étendues of intersection 𝑄

𝑅

&

𝐾

𝐼

defined by microlens k and pixel i

• add the weighted contribution of pixel i to reconstructed pixel X

I 𝑿 = ෍

𝒍𝒆𝒏𝒔 𝒌

෍

𝒑𝒊𝒙𝒆𝒍 𝒊

Pixel value(𝒊)

étendue 𝑸

𝑹

ځ 𝑲

𝑰

étendue 𝑸

𝑹

Traditional (or 1.0) plenoptic algorithm [3]

Focused (or 2.0) plenoptic algorithm [4]

Raw images

Reconstructed with

algorithm 1.0

Reconstructed with

algorithm 2.0

Traditional (or 1.0)

plenoptic geometry

Focused (or 2.0)

plenoptic geometry

object planes

of acquisition and reconstruction

geometry 1.0

main lens

focal length f

₁

array of microlenses

focal length f

₂

sensor

b

Z₀

Δ

A

_k

intersection

B

of

A

_k

&

B

microlens apertures sensor pixels Reconstructed pixels through each microlens Reconstructed pixels

Quality of reconstructed images depends on both the optical design

and the algorithm used.

pixel of reconstructed

image

main lens aperture

sensor pixel aperture of

microlens k

A

= volume from pixel to

main lens aperture

A

_k

=

projection of

A through

microlens

k

B = volume from

sensor pixel to

aperture of

microlens

intersection

of

A

_k

&

B

object-space ray

𝑋

𝑈

center of sensor’s pixel

points created by interpolation

microlens apertures sensor pixels α=refocusing

parameter

P=patch size

sensor pixels

only center of sensor’s pixel are used

microlens apertures

𝑄

𝑅

= 𝑀 𝑋

𝑈

+

𝑏 𝑓₂

0 𝑘

with

𝑀 =

1 𝑍₀

−𝛥

𝑍

₀

+ 𝛥 −

𝑍0 𝑓₁ 𝛥𝑏 𝑓₂

− 𝑏 + 𝛥

𝑏 + 𝑍

0

+ 𝛥 −

𝑏+𝛥 𝑍₀ 𝑓₁

−

𝑍₀+𝛥 𝑏 𝑓₂

+

𝛥𝑍₀𝑏

𝑓₁𝑓₂

Intersection of étendues in the image space

Integration is approximated on a reduced set of samples

3. Proposed algorithm

4. Comparison with historical algorithms

Plenoptic 1.0

algorithm

Plenoptic 2.0

algorithm

𝑋 𝑈