• Aucun résultat trouvé

Blur Aware Calibration of Multi-Focus Plenoptic Camera

N/A
N/A
Protected

Academic year: 2021

Partager "Blur Aware Calibration of Multi-Focus Plenoptic Camera"

Copied!
18
0
0

Texte intégral

(1)

HAL Id: hal-02537124

https://hal.archives-ouvertes.fr/hal-02537124

Submitted on 8 Apr 2020

HAL is a multi-disciplinary open access

archive for the deposit and dissemination of

sci-entific research documents, whether they are

pub-lished or not. The documents may come from

teaching and research institutions in France or

abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire HAL, est

destinée au dépôt et à la diffusion de documents

scientifiques de niveau recherche, publiés ou non,

émanant des établissements d’enseignement et de

recherche français ou étrangers, des laboratoires

publics ou privés.

Blur Aware Calibration of Multi-Focus Plenoptic

Camera

Mathieu Labussière, Céline Teulière, Frédéric Bernardin, Omar Ait-Aider

To cite this version:

Mathieu Labussière, Céline Teulière, Frédéric Bernardin, Omar Ait-Aider. Blur Aware Calibration of

Multi-Focus Plenoptic Camera. IEEE Conference on Computer Vision and Pattern Recognition, Jun

2020, Seattle, United States. �hal-02537124�

(2)

Blur Aware Calibration of Multi-Focus Plenoptic Camera

Mathieu Labussière

1

, Céline Teulière

1

, Frédéric Bernardin

2

, Omar Ait-Aider

1

1

Université Clermont Auvergne, CNRS, SIGMA Clermont,

Institut Pascal, F-63000 Clermont-Ferrand, France

2

Cerema, Équipe-projet STI, 10 rue Bernard Palissy, F-63017 Clermont-Ferrand, France

mathieu.labu@gmail.com, firstname.name@{uca,cerema}.fr

Abstract

This paper presents a novel calibration algorithm for Multi-Focus Plenoptic Cameras (MFPCs) using only raw images. The design of such cameras is usually complex and relies on precise placement of optic elements. Several calibration procedures have been proposed to retrieve the camera parameters but relying on simplified models, recon-structed images to extract features, or yet multiple calibra-tions when several types of micro-lens are used. Consider-ing blur information, we propose a new Blur Aware Plenop-tic (BAP) feature. It is first exploited in a pre-calibration step that retrieves initial camera parameters, and secondly to express a new cost function for our single optimization process. The effectiveness of our calibration method is val-idated by quantitative and qualitative experiments.

1. Introduction

The purpose of an imaging system is to map incoming light rays from the scene onto pixels of photo-sensitive de-tector. The radiance of a light ray is given by the plenop-tic functionL (x, θ, λ, τ ), introduced by Adelson et al. [1], where x ∈ R3is the spatial position of observation, θ ∈ R2 is the angular direction of observation, λ is the wavelength of the light and τ is the time. Conventional cameras capture only one point of view. A plenoptic camera is a device that allows to retrieve spatial as well as angular information.

From Lumigraph [17] to commercial plenoptic cameras [19,23], several designs have been proposed. This paper focuses on plenoptic cameras based on a Micro-Lenses Ar-ray (MLA) placed between the main lens and the photo-sensitive sensor (seeFig. 2). The specific design of such a camera allows to multiplex both types of information onto the sensor in the form of a Micro-Images Array (MIA) (see

Fig. 1(b)), but implies a trade-off between the angular and spatial resolution [9,16,10]. It is balanced according to the MLA position with respect to the main lens focal plane (i.e., focused[23,8] and unfocused [19] configuration).

Figure 1: The Raytrix R12 multi-focus plenoptic cam-era used in our experimental setup (a), along with a raw image of a checkerboard calibration target (b). The image is composed of several micro-images with different blurred levels and arranged in an hexagonal grid. In each micro-image, our new Blur Aware Plenoptic (BAP) feature is il-lustrated by their center and their blur radius (c).

The mapping of incoming light rays from the scene onto pixels can be expressed as a function of the camera model. Classical cameras are usually modeled as pinhole or thin lens. Due to the complexity of plenoptic camera’s design, the used models are usually high dimensional. Specific cal-ibration methods have to be developed to retrieve the intrin-sic parameters of these models.

1.1. Related Work

Unfocused plenoptic camera calibration. In this config-uration, light rays are focused by the MLA on the sensor plane. The calibration of unfocused plenoptic camera [19] has been widely studied in the literature [6,2,25,24,11, 36]. Most approaches rely on a thin-lens model for the main lens and an array of pinholes for the micro-lenses. Most of them require reconstructed images to extract features, and limit their model to the unfocused configuration, i.e.,

(3)

set-ting the micro-lens focal length at the distance MLA-sensor. Therefore those models cannot be directly extended to the focused or multi-focus plenoptic camera.

Focused plenoptic camera calibration. With the arrival of commercial focused plenoptic cameras [18, 23], new calibration methods have been proposed. Based on [14], Heinze et al. [13] have developed a new projection model and a metric calibration procedure which is incorporated in the RxLive software of Raytrix GmbH. Other cal-ibration methods and models have been proposed, either to overcome the fragility of the initialization [26], or to model more finely the camera parameters using depth in-formation [33, 32]. O’Brien et al. [22] introduced a new 3D feature called plenoptic disc and defined by its center and its radius. Nevertheless, all previous methods rely on reconstructed images meaning that they introduce error in the reconstruction step as well as in the calibration process. To overcome this problem, several calibration meth-ods [35, 34,29,20,3,21,31] have been proposed using only raw plenoptic images. In particular, features extrac-tion in raw micro-images has been studied in [3,21, 20] achieving improved performance through automation and accurate identification of feature correspondences. How-ever, most of the methods rely on simplified models for op-tic elements: the MLA is modeled as a pinholes array mak-ing it impossible to retrieve the focal lengths, or the MLA misalignment is not considered. Some do not consider dis-tortions [35, 21,31] or restrict themselves to the focused case [35,34,20].

Finally, few have considered the multi-focus case [13,3, 21,31] and dealt with it in separate processes, leading to different intrinsic and extrinsic parameters according to the type of micro-lenses.

1.2. Contributions

This paper focuses on the calibration of micro-lenses-based Multi-Focus Plenoptic Camera (MFPC). To the best of our knowledge, this is the first method proposing a sin-gle optimization process that retrieves intrinsic and extrinsic parameters of a MFPC directly from raw plenoptic images. The main contributions are the following:

• We present a new Blur Aware Plenoptic (BAP) feature defined in raw image space that enables us to handle the multi-focus case.

• We introduce a new pre-calibration step using BAP features from white images to provide a robust initial estimate of internal parameters.

• We propose a new reprojection error function exploit-ing BAP features to refine a more complete model, in-cluding in particular the multiple micro-lenses focal lengths. Our checkerboard-based calibration is con-ducted in a single optimization process.

Sensor image bl object // al Main Lens A O MLA (u0, v0) d D F ∆c xw yw zw Ck,l ck,l0 s f(i) ∆C ck,l

Figure 2: Focused Plenoptic Camera model in Galilean con-figuration (i.e., the main lens focuses behind the sensor) with the notations used in this paper.

A visual overview of our method is given in Fig. 3. The remainder of this paper is organized as follows: first, the camera model and BAP feature are presented inSection 2. The proposed pre-calibration step is explained inSection 3. Then, the feature detection is detailed inSection 4and the calibration process inSection 5. Finally, our results are pre-sented and discussed inSection 6. The notations used in this paper are shown inFig. 2. Pixel counterparts of metric values are denoted in lower-case Greek letters.

2. Camera model and BAP feature

2.1. Multi-focus Plenoptic Camera

We consider multi-focus plenoptic cameras as described in [8, 23]. The main lens, modeled as a thin-lens, maps object point to virtual point behind (resp., in front of) the image sensor in Galilean (resp., Keplerian) configuration. Therefore, the MLA consists of I different lens types with focal lengths f(i), i ∈ {1, . . . , I} which are focused on I different planes behind the image sensor. This multi-focus setup corresponds to the Raytrix camera system described in [23] when I = 3. The micro-lenses are mod-eled as thin-lenses allowing to take into account blur in the micro-image. Our model takes into account the MLA mis-alignment, freeing all six degrees of freedom.

The tilt of the main lens is included in the distortion model and we make the hypothesis that the main lens plane Πl is parallel to the sensor plane Πs. Furthermore, we

choose the main lens frame as our camera reference frame, with O being the origin, the z-axis coinciding with the op-tical axis and pointing outside the camera, and the y-axis pointing downwards. Only distortions of the main lens are considered. We use the model of Brown-Conrady [4,5] with three coefficients for the radial component and two for the tangential.

(4)

Furthermore, we consider the deviation of the image cen-ter and the optical cencen-ter for each micro-lens because it tends to cause inaccuracy in decoded light field. Therefore, the principal point ck,l0 of the micro-lens indexed by (k, l) is given by ck,l0 =u k,l 0 v0k,l  = d D + d u0 v0  − ck,l  + ck,l, (1)

where ck,lis the center in pixels of the micro-image (k, l),

u0 v0 >

is the main lens principal point, d is the distance MLA-sensor and D is the distance main lens-MLA, as illus-trated inFig. 2.

Finally, each micro-lens produces a micro-image onto the sensor. The set of these micro-images has the same structural organization as the MLA, i.e., in our case an hexagonal grid, alternating between each type of micro-lens. The data can therefore be interpreted as an array of micro-images, called by analogy the MIA. The MIA coor-dinates are expressed in image space. Let δc be the pixel dis-tance between two arbitrary consecutive micro-images cen-ters ck,l. With s the metric size of a pixel, let ∆c = s · δc be

its metric value, and ∆C be the metric distance between the two corresponding micro-lenses centers Ck,l. From similar

triangles, the ratio between them is given by ∆C ∆c = D d + D ⇐⇒ ∆C = ∆c · D d + D. (2) We make the hypothesis that ∆C is equal to the micro-lens diameter. Since d  D, we can make the following ap-proximation

D

D + d = λ ≈ 1 =⇒ ∆C = ∆c · D

D + d ≈ ∆c. (3) This approximation will be validated in the experiments.

2.2. BAP feature and projection model

Using a camera with a circular aperture, the blurred im-age of a point on the imim-age detector is circular in shape and is called the blur circle. From similar triangles and from the thin-lens equation, the signed blur radius of a point in an image can be expressed as

ρ = 1 s· A 2d  1 f − 1 a− 1 d  , (4)

with s the size of a pixel, d the distance between the con-sidered lens and the sensor, A the aperture of this lens, f its focal length, and a the distance of the object from the lens.

This radius appears at different levels in the camera pro-jection: in the blur introduced by the thin-lens model of the micro-lenses and during the formation of the micro-image while taking a white image. To leverage blur information,

we introduce a new Blur Aware Plenoptic (BAP) feature characterized by its center and its radius, i.e., p = (u, v, ρ). Therefore, our complete plenoptic camera model allows us to link a scene point pw to our new BAP feature p

through each micro-lens (k, l)     u v ρ 1     ∝ P (i, k, l) · Tµ(k, l) · ϕ(K(F ) · Tc· pw) , (5)

where P (i, k, l) is the blur aware plenoptic projection ma-trix through the micro-lens (k, l) of type i, and computed as P (i, k, l) = P (k, l) · Kf(i) (6) =     d/s 0 uk,l0 0 0 d/s v0k,l 0 0 0 s∆C2 −s∆C2 d 0 0 −1 0         1 0 0 0 0 1 0 0 0 0 1 0 0 0 −1/f(i) 1     .

P (k, l) is a matrix that projects the 3D point onto the sen-sor. K(f ) is the thin-lens projection matrix for the given focal length. Tc is the pose of the main lens with respect

to the world frame and Tµ(k, l) is the pose of the

micro-lens (k, l) expressed in the camera frame. The function ϕ(·) models the lateral distortion.

Finally, the projection model defined in Eq. (5) con-sists of a set Ξ of (16 + I) intrinsic parameters to be opti-mized, including the main lens focal length F and its 5 lat-eral distortion parameters, the sensor translation, encoded in (u0, v0) and d, the MLA misalignment, i.e., 3 rotations

(θx, θy, θz) and 3 translations (tx, ty, D), the micro-lens

inter-distance ∆C, and the I micro-lens focal lengths f(i).

3. Pre-calibration using raw white images

Drawing inspiration from depth from defocus the-ory [27], we leverage blur information to estimate param-eters (e.g., here our blur radius) by varying some other pa-rameters (e.g., the focal length, the aperture, etc.) in com-bination with known (i.e., fixed or measured) parameters. For instance, when taking a white picture with a controlled aperture, each type of micro-lens produces a micro-image (MI) with a specific size and intensity, providing a way to distinguish them. In the following all distances are given with reference to the MLA plane. Distances are signed ac-cording to the following convention: f is positive when the lens is convergent; distances are positive when the point is real, and negative when virtual.

3.1. Micro-image radius derivation

Taking a white image is equivalent for the micro-lenses to image a white uniform object of diameter A at a distance

(5)

White Raw Images Micro-Images Array calibration Centers{c k,l } Internal Parameters (m, q’1 … q’I ) Pre-calibration: parameters estimation N-1 R Checkerboard Raw Images Corners extraction BAP Features (u, v, ⍴) Virtual depth estimation BAP Feature extraction Optimization

Figure 3: Overview of our proposed method with the pre-calibration step and the BAP features detection that it used in the non-linear optimization process.

D. This is illustrated inFig. 4. We relate the micro-image (MI) radius to the plenoptic camera parameters. From op-tics geometry, the image of this object, i.e. the resulting MI, is equivalent to the image of an imaginary point constructed as the vertex of the cone passing through the main lens and the considered micro-lens (noted V in Fig. 4). Let a0 be the distance of this point from the MLA plane, given from similar triangles andEq. (2)by

a0= −D ∆C A − ∆C = −D  A d + D ∆cD  − 1 −1 , (7) with A the main lens aperture. Note the minus sign is due to the fact that the imaginary point is always formed behind the MLA plane at a distance a0, and thus considered as a virtual object for the micro-lenses. Conceptually, the MI formed can be seen as the blur circle of this imaginary point. Therefore, usingEq. (4), the metric MI radius R is given by

R = ∆C 2 d  1 f − 1 a0 − 1 d  = A · d 2D+  ∆cD d + D  ·d 2 ·  1 f − 1 D − 1 d  . (8) From the above equation, we see that the radius depends lin-early on the aperture of the main lens. However, the main lens aperture cannot be computed directly whereas we have access to the f -number value. The f -number of an opti-cal system is the ratio of the system’s foopti-cal length F to the diameter of the entrance pupil, A, given by N = F/A.

V V0 Main Lens - A MLA ∆C Sensor -R R b0 f d D a0

Figure 4: Formation of a micro-image with its radius R through a micro-lens while taking a white image at an aper-ture A. The point V is the vertex of the cone passing by the main lens and the considered micro-lens. V0is the image of V by the micro-lens and R is the radius of its blur circle. Finally, we can express the MI radius for each micro-lens focal length type i as

R N−1 = m · N−1+ q i (9) with m = dF 2D and qi= 1 f(i) ·  ∆cD d + D  ·d 2 − ∆c 2 . (10) Let q0ibe the value obtained by qi0 = qi+ ∆c/2.

3.2. Internal parameters estimation

The internal parameters Ω = {m, q01, . . . , qI0} are used to compute the radius part of the BAP feature and to initial-ize the parameters of the calibration process. Given several raw white images taken at different apertures, we estimate the coefficients ofEq. (9)for each type of micro-image. The standard full-stop f -number conventionally indicated on the lens differs from the real f -number calculated with the aper-ture value AV as N =√2AV.

From raw white images, we are able to measure each micro-image (MI) radius % = |R| /s in pixels for each dis-tinct focal length f(i)at a given aperture. Due to vignetting effect, the estimation is only conducted on center micro-images which are less sensitive to this effect. Our method is based on image moments fitting. It is robust to noise, works under asymmetrical distribution and is easy to use, but need a parameter α to convert the standard deviation σ into a pixel radius % = α · σ. We use the second order central moments of the micro-image to construct a covari-ance matrix. Finally, we chose σ as the square root of the greater eigenvalue of the covariance matrix. The parameter α is determined such that at least 98% of the distribution is taken into account. According to the standard normal distri-bution Z-score table, α is picked up in [2.33, 2.37]. In our experiment, we set α = 2.357.

(6)

0.04 0.06 i = 1 0.04 0.06 Radius [mm] i = 2 0.08 0.10 0.12 0.14 0.16 0.18 Inverse f -number 0.04 0.06 i = 3 0.04 0.06 0 100 i = 1 0.04 0.06 i = 2 0.04 0.06 i = 3 N=5.66 N=8.00 N=11.31 Radius [mm]

Figure 5: Microimage radii as function of the inverse f -number with the estimated lines (in magenta). Each type of micro-lens is identified by its color (type (1) in red, type (2) in green and type (3) in blue) with its computed radius. For each type an histogram of the radii distribution is given.

Finally, from radii measurements at different f -numbers, we estimate the coefficients of Eq. (9), X = {m, q1, . . . , qI}, with a least-square estimation. Fig. 5

shows the radii computed in a white image taken with an f -number N = 8, the histogram of these radii distribution for each type, and the estimated linear functions.

4. BAP feature detection in raw images

At this point, the internal parameters Ω, used to express our blur radius ρ, are available. The process (illustrated in

Fig. 3) is divided into three phases: 1) using a white raw image, the MIA is calibrated and micro-image centers are extracted; 2) checkerboard images are processed to extract corners at position (u, v); and 3) with the internal parame-ters and the associated virtual depth estimate for each cor-ner, the corresponding BAP feature is computed.

4.1. Blur radius derivation through micro-lens

To respect the f -number matching principle [23], we configure the main lens f -number such that the micro-images fully tile the sensor without overlap. In this con-figuration the working f -number of the main imaging sys-tem and the micro-lens imaging syssys-tem should match. The following relation is then verified for at least one type of micro-lens: ∆C d = A D ⇐⇒ 1 d·  ∆cD d + D  = F N D. (11) We consider the general case of measuring an object p at a distance alfrom the main lens. First, p is projected through

the main lens according to the thin lens equation, 1/F = 1/al+ 1/bl, resulting in a point p0at a distance blbehind

the main lens, i.e. at a distance a = D − blfrom the MLA.

The metric radius of the blur circle r formed on the sensor

for a given point p0 at distance a through a micro-lens of type i is expressed as r = ∆cD d + D  · d 2·  1 f(i)− 1 a− 1 d  = ∆cD d + D · d 2· 1 f(i) | {z } =q0 i(10) − ∆cD d + D · d 2· 1 d | {z } =∆C/2(2) − ∆cD d + D | {z } =∆C(2) ·d 2· 1 a =  −∆C ·d 2  ·1 a+  q0i−∆C 2  . (12)

In practice, we do not have access to the value of ∆C but we can use the approximation fromEq. (3). Moreover, a and d cannot be measured in the raw image space, but the virtual depth can. Virtual depth refers to relative depth values ob-tained from disparity. It is defined as the ratio between the object distance a and the sensor distance d:

ν = a

d. (13)

The sign convention is reversed for virtual depth computa-tion, i.e. distances are negative in front of the MLA plane. If we re-inject the latter inEq. (12), taking caution of the sign, we can derive the radius of the blur circle of a point p0 at a distance a from the MLA by

r = λ∆c 2 · ν −1+  q0i−λ∆c 2  . (14) This equation allows to express the pixel radius of the blur circle ρ = r/s associated to each point having a virtual depth without explicitly evaluating A, D, d, F and f(i).

4.2. Features extraction

First, the Micro-Images Array has to be calibrated. We compute the micro-image centers observations {ck,l} by

in-tensity centroid method with sub-pixel accuracy [30, 20, 28]. The distance between two micro-image centers δc is then computed as the optimized edge-length of a fitted 2D grid mesh with a non-linear optimization. The pixel trans-lation offset in the image coordinates, (τx, τy), and the

ro-tation around the (−z)-axis, ϑz, are also determined during

the optimization process. From the computed distance and the given size of a pixel, the parameter λ∆c is computed.

Secondly, as we based our method on checkerboard cal-ibration pattern, we detect corners in raw images using the detector introduced by Noury et al. [20]. The raw images are devignetted by dividing them by a white raw image taken at the same aperture.

With a plenoptic camera, contrarily to a classical camera, a point is projected into more than one observation onto the sensor. Due to the nature of the spatial distribution of the data, we used the DBSCAN algorithm [7] to identify the

(7)

clusters. We then associate each point with its cluster of observations.

Once each cluster is identified, we can compute the vir-tual depth ν from disparity. Given the distance ∆C1−2

be-tween the centers of the micro-lenses C1 and C2, i.e. the

baseline, and the Euclidean distance ∆i = |i1− i2|

be-tween images of the same point in corresponding micro-images, the virtual depth ν can be calculated with the inter-cept theorem: ν = ∆C1−2 ∆C1−2− ∆i = η∆C η∆C − ∆i= ηλ∆c ηλ∆c − ∆i, (15) where ∆C = λ∆c is the distance between two consecutive micro-lenses and η ≥ 1.0. Due to noise in corner detec-tion, we use a median estimator to compute the virtual depth of the cluster taking into account all combinations of point pairs in the disparity estimation.

Finally, from internal parameters Ω and with the avail-able virtual depth ν we can compute the BAP features us-ingEq. (14). In each frame n, for each micro-image (k, l) of type i containing a corner at position (u, v) in the image, the feature pnk,lis given by

pnk,l= (u, v, ρ) , with ρ = r/s. (16) In the end, our observations are composed of a set of micro-image centers {ck,l} and a set of BAP features

n

pnk,lo al-lowing us to introduce two reprojection error functions cor-responding to each set of features.

5. Calibration process

To retrieve the parameters of our camera model, we use a calibration process based on non-linear minimization of a reprojection error. The calibration process is divided into three phases: 1) the intrinsics are initialized using the in-ternal parameters Ω; 2) the initial extrinsics are estimated from the raw checkerboard images; and 3) the parameters are refined with a non-linear optimization leveraging our new BAP features.

5.1. Parameters initialization

Optimization processes are sensitive to initial parame-ters. To avoid falling into local minima during the optimiza-tion process, the parameters have to be carefully initialized not too far from the solution.

First, the camera is initialized in Keplerian or Galilean configuration. First, given the camera configuration, the internal parameters Ω, the focus distance h, and from the Eq. (17) of [23], the following parameters are set as

d ←− 2mH F + 4m and D ←− H − 2d, (17) where H is given by H = h 2 1 − r 1 ± 4F h ! , (18)

with positive (resp., negative) sign in Galilean (resp., Kep-lerian) configuration.

The focal length, and the pixel size s are set according to the manufacturer value. All distortion coefficients are set to zero. The principal point is set as the center of the image. The sensor plane is thus set parallel to the main lens plane, with no rotation, at a distance − (D + d).

Seemingly, the MLA plane is set parallel to the main lens plane at a distance −D. From the pre-computed MIA parameters the translation takes into account the offsets (−sτx, −sτy) and the rotation around the z-axis is

initial-ized with −ϑz. The micro-lenses inter-distance ∆C is set

according to Eq. (2). Finally, from internal parameters Ω, the focal lengths are computed as follows

f(i)←− d

2 · qi0 · ∆C. (19)

5.2. Initial poses estimation

The camera poses {Tcn}, i.e., the extrinsic parameters are initialized using the same method as in [20]. For each cluster of observation, the barycenter is computed. Those barycenters can been seen as the projections of the checker-board corners through the main lens using a standard pin-hole model. For each frame, the pose is then estimated us-ing the Perspective-n-Point (PnP) algorithm [15].

5.3. Non-linear optimization

The proposed model allows us to optimize all the param-eters into one single optimization process. We propose a new cost function Θ taking into account the blur informa-tion using our new BAP feature. The cost is composed of two main terms both expressing errors in the image space: 1) the blur aware plenoptic reprojection error, where, for each frame n, each checkerboard corner pnw is reprojected into the image space through each micro-lens (k, l) of type i according to the projection model ofEq. (5)and compared to its observations pnk,l; and 2) the micro-lens center repro-jection error, where, the main lens center O is reprojected according to a pinhole model in the image space through each lens (k, l) and compared to the detected micro-image center ck,l.

Let S = {Ξ, {Tcn}} be the set of intrinsic Ξ and extrinsic {Tn

c } parameters to be optimized. The cost function Θ(S)

is expressed as X pnk,l− Πk,l(pnw) 2 +Xkck,l− Πk,l(O)k2. (20)

The optimization is conducted using the Levenberg-Marquardt algorithm.

(8)

R12-A (h = 450mm) R12-B (h = 1000mm) R12-C (h = ∞)

Unit Initial Ours [20] Initial Ours [20] Initial Ours [20]

F [mm] 50 49.720 54.888 50 50.047 51.262 50 50.011 53.322 D [mm] 56.66 56.696 62.425 52.11 52.125 53.296 49.38 49.384 52.379 ∆C [µm] 127.51 127.45 127.38 127.47 127.45 127.40 127.54 127.50 127.42 f(1) [µm] 578.15 577.97 - 581.10 580.48 - 554.35 556.09 -f(2) [µm] 504.46 505.21 - 503.96 504.33 - 475.98 479.03 -f(3) [µm] 551.67 551.79 - 546.39 546.37 - 518.98 521.33 -u0 [pix] 2039 2042.55 2289.83 2039 1790.94 1759.29 2039 1661.95 1487.2 v0 [pix] 1533 1556.29 1528.24 1533 1900.19 1934.87 1533 1726.91 1913.81 d [µm] 318.63 325.24 402.32 336.84 336.26 363.17 307.93 312.62 367.40

Table 1: Initial intrinsic parameters for each dataset along with the optimized parameters obtained by our method and with the method of [20]. Some parameters are omitted for compactness.

6. Experiments and Results

We evaluate our calibration model quantitatively in a controlled environment and qualitatively when ground truth is not available.

6.1. Experimental setup

For all experiments we used a Raytrix R12 color 3D-light-field-camera, with a MLA of F/2.4 aperture. The mounted lens is a Nikon AF Nikkor F/1.8D with a 50mm focal length. The MLA organization is hexago-nal, and composed of 176 × 152 (width × height) micro-lenses with I = 3 different types. The sensor is a Basler beA4000-62KCwith a pixel size of s = 0.0055mm. The raw image resolution is 4080 × 3068.

Datasets. We calibrate our camera for three different fo-cus distance configurations hand build three corresponding datasets: R12-A for h = 450 mm, R12-B for h = 1000 mm, and R12-C for h = ∞. Each dataset is composed of:

• white raw plenoptic images acquired at different aper-tures (N ∈ {4, 5.66, 8, 11.31, 16}) with augmented gain to ease circle detection in the pre-calibration step, • free-hand calibration targets acquired at various poses (in distance and orientation), separated into two sub-sets, one for the calibration process and the other for the qualitative evaluation,

• a white raw plenoptic image acquired in the same lu-minosity condition and with the same aperture as in the calibration targets acquisition,

• and calibration targets acquired with a controlled trans-lation motion for quantitative evaluation, along with the depth maps computed by the Raytrix software (RxLive v4.0.50.2).

We use a 9 × 5 of 10mm side checkerboard for R12-A, a 8 × 5 of 20mm for R12-B, and a 7 × 5 of 30mm for R12-C. Datasets and our source code are publicly available1.

1https://github.com/comsee-research

Free-hand camera calibration. The white raw plenop-tic image is used for devignetting other raw images and for computing micro-images centers. From the set of calibra-tion targets images, BAP features are extracted, and camera intrinsic and extrinsic parameters are then computed using our non-linear optimization process.

Controlled environment evaluation. In our experimen-tal setup (seeFig. 1), the camera is mounted on a linear mo-tion table with micro-metric precision. We acquired several images with known relative motion between each frame. Therefore, we are able to quantitatively evaluate the esti-mated displacement from the extrinsic parameters with re-spect to the ground truth. The extrinsics are computed with the intrinsics estimated from the free-hand calibration. We compared our computed relative depth to those obtained by the RxLive software.

Qualitative evaluation. When no ground truth is avail-able, we evaluate qualitatively our parameters on the evalu-ation subset by estimating the reprojection error using the previously computed intrinsics. We use the Root-Mean-Square Error (RMSE) as our metric to evaluate the repro-jection, individually on the corner reprojection and the blur radius reprojection.

Comparison. Since our model is close to [20], we com-pare our intrinsics with the ones obtained under their pin-hole assumption using only corner reprojection error and with the same initial parameters. We also provide the cali-bration parameters obtained from the RxLive software.

6.2. Results

Internal parameters. Note that the pixel MI radius is given by % = |R| /s, and R is either positive if formed after the rays inversion (as inFig. 4), or negative if before. With our camera f(i)> d [12], so R < 0 implying that m and ci

are also negative. In practice, it means we use the value −% in the estimation process.

(9)

In our experiments, we set λ = 0.9931. We verified that the error introduced by this approximation is less than the metric size of a pixel (i.e., less than 1 % of its optimized value). Fig. 5shows the estimated lines (seeEq. (9)), with a cropped white image taken at N = 8, where each type of micro-lens is identified with its radius. An histogram of the radii distribution is also given for each type. From the radii measurements, the computed internal parameters are estimated and given for each dataset inTab. 2.

R12-A R12-B R12-C ∆c 128.222 128.293 128.333 m −140.596 −159.562 −155.975 q0 1 35.135 36.489 35.443 q02 40.268 42.075 41.278 q03 36.822 38.807 37.858 m∗ −142.611 −161.428 −158.292 m 1.41% 1.16% 1.52%

Table 2: Internal parameters (in µm) computed during the pre-calibration step for each dataset. The expected value for m is given by m∗and the relative error m= (m∗−m)/m∗

is computed.

As expected, the internal parameters ∆c and m are dif-ferent for each dataset, as D and δc vary with the focus dis-tance h, whereas the qi0 values are close for each dataset. Using intrinsics, we compare the expected coefficient m∗ (from the closed form inEq. (10) with the optimized pa-rameters) with its calculated m value. The mean error over all datasets is ¯m= 1.36% which is smaller than a pixel.

Free-hand camera calibration. The initial parameters for each dataset are given inTab. 1along with the optimized parameters obtained from our calibration and from the method in [20]. Some parameters are omitted for compact-ness (i.e., distortions coefficient and MLA rotations which are negligible). The main lens focal lengths obtained with the proprietary RxLive software are: Fh=450 = 47.709

mm, Fh=1000= 50.8942 mm, and Fh=∞= 51.5635 mm.

With our method and [20], the optimized parameters are close to their initial value, showing that our method pro-vides a good initialization for our optimization process. The F , d and ∆C are consistent across the datasets with our method. In contrast, the F and d obtained with [20] show a larger discrepancy. This is also the case for the focal lengths obtained by RxLive.

Poses evaluation. Tab. 3presents the relative translations and their errors with respect to the ground truth for the con-trolled environment experiment. Even if our relative errors are similar with [20], we are able to retrieve more param-eters. With our method, absolute errors are of the order of the mm (R12-A: 0.37 ± 0.15 mm, R12-B: 1.66 ± 0.58 mm and R12-C: 1.38 ± 0.85 mm) showing that the re-trieved scale is coherent. Averaging over all the datasets,

our method presents the smallest relative error, with low dis-crepancy between datasets, outperforming the estimations of the RxLive software.

R12-A R12-B R12-C All

Error [%] ¯z σz ¯z σz ¯z σz ¯z

Ours 3.73 1.48 3.32 1.17 2.95 1.35 3.33 [20] 6.83 1.17 1.16 1.06 2.70 0.86 3.56 RxLive 4.63 2.51 4.26 5.79 11.52 3.22 6.80 Table 3: Relative translation error along the z-axis with re-spect to the ground truth displacement. For each dataset, the mean error ¯z and its standard deviation σz are given.

Results are given for our method and compared with [20] and the proprietary software RxLive.

Reprojection error evaluation. For each evaluation dataset, the total squared pixel error is reported with its computed RMSE inTab. 4. The error is less than 1pix per feature for each dataset demonstrating that the computed in-trinsics are valid and can be generalized to images different from the calibration set.

R12-A (#11424) R12-B (#3200) R12-C (#9568) Total RMSE Total RMSE Total RMSE ¯ all 8972.91 0.886 1444.98 0.672 5065.33 0.728 ¯ u,v 8908.65 0.883 1345.20 0.648 5046.68 0.726 ¯ ρ 64.257 0.075 99.780 0.177 18.659 0.044

Table 4: Reprojection error for each evaluation dataset with their number of observations. For each component of the feature, the total squared pixel error is reported with its computed RMSE.

7. Conclusion

To calibrate the Multi-Focus Plenoptic Camera, state-of-the-art methods rely on simplifying hypotheses, on re-constructed data or require separate calibration processes to take into account the multi-focal aspect. This paper intro-duces a new pre-calibration step which allows us to com-pute our new BAP feature directly in the raw image space. We then derive a new projection model and a new reprojec-tion error using this feature. We propose a single calibra-tion process based on non-linear optimizacalibra-tion that enables us to retrieve of camera parameters, in particular the micro-lenses focal lengths. Our calibration method is validated by qualitative experiments and quantitative evaluations. In the future, we plan to exploit this new feature to improve metric depth estimation.

Acknowledgments. This work has been supported by the AURA Region and the European Union (FEDER) through the MMII project of CPER 2015-2020 MMaSyF challenge. We thank Charles-Antoine Noury and Adrien Coly for their insightful discussions and their help during the acquisitions.

(10)

References

[1] E. H. Adelson and J. R. Bergen. The plenoptic function and the elements of early vision. Computational Models of Visual Processing, pages 3–20, 1991.1

[2] Yunsu Bok, Hae-Gon Jeon, and In So Kweon. Geometric Calibration of Micro-Lens-Based Light-Field Cameras Us-ing Line Features-Supplementary Material. Computer Vision - Eccv 2014, Pt Vi, 8694:47–61, 2014.1

[3] Yunsu Bok, Hae-Gon Jeon, and In So Kweon. Geometric Calibration of Micro-Lens-Based Light Field Cameras Using Line Features. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(2):287–300, 2017.2

[4] Duane Brown. Decentering Distortion of Lenses - The Prism Effect Encountered in Metric Cameras can be Overcome Through Analytic Calibration. Photometric Engineering, 32(3):444–462, 1966.2

[5] Ae Conrady. Decentered Lens-Systems. Monthly Notices of the Royal Astronomical Soceity, 79:384–390, 1919.2

[6] Donald G. Dansereau, Oscar Pizarro, and Stefan B. Williams. Decoding, calibration and rectification for lenselet-based plenoptic cameras. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pat-tern Recognition, pages 1027–1034, 2013.1

[7] Martin Ester, Hans-Peter Kriegel, Jiirg Sander, and Xiaowei Xu. A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. In KDD, 1996.5

[8] Todor Georgiev and Andrew Lumsdaine. The multifocus plenoptic camera. In Sebastiano Battiato, Brian G. Rodricks, Nitin Sampat, Francisco H. Imai, and Feng Xiao, editors, Digital Photography VIII, volume 8299, page 829908, jan 2011.1,2

[9] Todor Georgiev, Ke Colin Zheng, Brian Curless, David Salesin, Shree K Nayar, and Chintan Intwala. Spatio-Angular Resolution Tradeoff in Integral Photography. Ren-dering Techniques, 2006(263-272):21, 2006.1

[10] Todor G. Georgiev and Andrew Lumsdaine. Resolution in Plenoptic Cameras. Frontiers in Optics 2009/Laser Science XXV/Fall 2009 OSA Optics and Photonics Technical Digest, page CTuB3, 2009.1

[11] Christopher Hahne, Andrew Lumsdaine, Amar Aggoun, and Vladan Velisavljevic. Real-time refocusing using an FPGA-Based standard plenoptic camera. IEEE Transactions on In-dustrial Electronics, 65(12):9757–9766, 2018.1

[12] Christian Heinze. Design and test of a calibration method for the calculation of metrical range values for 3D light field cameras. Master’s thesis, Hamburg University of Applied Sciences - Faculty of Engineering and Computer Science, 2014.7

[13] Christian Heinze, Stefano Spyropoulos, Stephan Hussmann, and Christian Perwaß. Automated Robust Metric Calibration Algorithm for Multifocus Plenoptic Cameras. IEEE Trans-actions on Instrumentation and Measurement, 65(5):1197– 1205, 2016.2

[14] Ole Johannsen, Christian Heinze, Bastian Goldluecke, and Christian Perwaß. On the calibration of focused plenop-tic cameras. Lecture Notes in Computer Science (including

subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 8200 LNCS:302–317, 2013.2

[15] Laurent Kneip, Davide Scaramuzza, and Roland Siegwart. A novel parametrization of the perspective-three-point prob-lem for a direct computation of absolute camera position and orientation. Proceedings of the IEEE Computer Society Con-ference on Computer Vision and Pattern Recognition, pages 2969–2976, 2011.6

[16] Anat Levin, William T. Freeman, and Frédo Durand. Un-derstanding camera trade-offs through a Bayesian analysis of light field projections. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 5305 LNCS(PART 4):88–101, 2008.1

[17] Gabriel Lippmann. Integral Photography. Academy of the Sciences, 1911.1

[18] Andrew Lumsdaine and Todor Georgiev. The focused plenoptic camera. In IEEE International Conference on Computational Photography (ICCP), pages 1–8. IEEE, apr 2009.2

[19] Ren Ng, Marc Levoy, Gene Duval, Mark Horowitz, and Pat Hanrahan. Light Field Photography with a Hand-held Plenoptic Camera. Technical report, Stanford University, 2005.1

[20] Charles Antoine Noury, Céline Teulière, and Michel Dhome. Light-Field Camera Calibration from Raw Images. DICTA 2017 - 2017 International Conference on Digital Image Computing: Techniques and Applications, 2017-Decem:1– 8, 2017.2,5,6,7,8

[21] Sotiris Nousias, Francois Chadebecq, Jonas Pichat, Pearse Keane, Sebastien Ourselin, and Christos Bergeles. Corner-Based Geometric Calibration of Multi-focus Plenoptic Cam-eras. Proceedings of the IEEE International Conference on Computer Vision, 2017-Octob:957–965, 2017.2

[22] Sean O’Brien, Jochen Trumpf, Viorela Ila, and Robert Ma-hony. Calibrating light-field cameras using plenoptic disc features. Proceedings - 2018 International Conference on 3D Vision, 3DV 2018, pages 286–294, 2018.2

[23] Christian Perwaßand Lennart Wietzke. Single Lens 3D-Camera with Extended Depth-of-Field. In Human Vision and Electronic Imaging XVII, volume 49, page 829108. SPIE, feb 2012.1,2,5,6

[24] Shengxian Shi, Junfei Ding, T. H. New, You Liu, and Hanmo Zhang. Volumetric calibration enhancements for single-camera light-field PIV. Experiments in Fluids, 60(1):21, jan 2019.1

[25] Shengxian Shi, Jianhua Wang, Junfei Ding, Zhou Zhao, and T. H. New. Parametric study on light field volumetric particle image velocimetry. Flow Measurement and Instrumentation, 49:70–88, 2016.1

[26] Klaus H. Strobl and Martin Lingenauber. Stepwise calibra-tion of focused plenoptic cameras. Computer Vision and Im-age Understanding, 145:140–147, 2016.2

[27] Murali Subbarao and Gopal Surya. Depth from defocus: A spatial domain approach. International Journal of Computer Vision, 13(3):271–294, 1994.3

[28] Piotr Suliga and Tomasz Wrona. Microlens array calibration method for a light field camera. Proceedings of the 2018 19th

(11)

International Carpathian Control Conference, ICCC 2018, pages 19–22, 2018.5

[29] Jun Sun, Chuanlong Xu, Biao Zhang, Shimin Wang, Md Moinul Hossain, Hong Qi, and Heping Tan. Geo-metric calibration of focused light field camera for 3-D flame temperature measurement. Conference Record - IEEE Instrumentation and Measurement Technology Conference, (51327803).2

[30] Chelsea M. Thomason, Brian S. Thurow, and Timothy W. Fahringer. Calibration of a Microlens Array for a Plenoptic Camera. 52nd Aerospace Sciences Meeting, (January):1–18, 2014.5

[31] Yuan Wang, Jun Qiu, Chang Liu, Di He, Xinkai Kang, Jian Li, and Ligen Shi. Virtual Image Points Based Geometri-cal Parameters’ Calibration for Focused Light Field Camera. IEEE Access, 6(c):71317–71326, 2018.2

[32] Niclas Zeller, Charles Antoine Noury, Franz Quint, Céline Teulière, Uwe Stilla, and Michel Dhome. Metric Calibration of a Focused Plenoptic Camera based on a 3D Calibration Target. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, III-3(July):449–456, jun 2016.2

[33] Niclas Zeller, Franz Quint, and Uwe Stilla. Calibration and accuracy analysis of a focused plenoptic camera. ISPRS An-nals of Photogrammetry, Remote Sensing and Spatial Infor-mation Sciences, II-3(September):205–212, 2014.2

[34] Chunping Zhang, Zhe Ji, and Qing Wang. Decoding and cali-bration method on focused plenoptic camera. Computational Visual Media, 2(1):57–69, 2016.2

[35] Chunping Zhang, Zhe Ji, and Qing Wang. Unconstrained Two-parallel-plane Model for Focused Plenoptic Cameras Calibration. pages 1–20, 2016.2

[36] Ping Zhou, Weijia Cai, Yunlei Yu, Yuting Zhang, and Guangquan Zhou. A two-step calibration method of lenslet-based light field cameras. Optics and Lasers in Engineering, 115(November 2018):190–196, 2019.1

(12)

(Supplementary Material)

Blur Aware Calibration of Multi-Focus Plenoptic Camera

Mathieu Labussière

1

, Céline Teulière

1

, Frédéric Bernardin

2

, Omar Ait-Aider

1

1

Université Clermont Auvergne, CNRS, SIGMA Clermont,

Institut Pascal, F-63000 Clermont-Ferrand, France

2

Cerema, Équipe-projet STI, 10 rue Bernard Palissy, F-63017 Clermont-Ferrand, France

mathieu.labu@gmail.com, firstname.name@{uca,cerema}.fr

As supplementary material, we first provide details about the camera model, second we detail in a more complete fashion the equations for the micro-image (MI) radii com-putation, third we complete the calibration results with the previously omitted parameters, and finally we give insight about the dataset images and poses.

A. Camera model

For completeness, the notations used in this paper are summarized inFig. 1.

A.1. Main lens distortions

Recall that in our model, only distortions of the main lens are considered. Distortions represent deviations from the theoretical thin lens projection model. To correct those errors, we can undistort a distorted point p =x y z> by applying a function ϕ to it, such as p−→ pϕ u. To model

the radial ϕ(r)and tangential ϕ(t)components of the lateral

distortion model, we use the model of Brown-Conrady [1, 2]. The radial component is thus expressed as

(

x(r)u = x · (1 + A0κ2+ A1κ4+ A2κ6)

y(r)u = y · (1 + A0κ2+ A1κ4+ A2κ6),

(1)

and the tangential component as (

x(t)u = B0(κ2+ 2x2) + 2B1xy

y(t)u = B1(κ2+ 2y2) + 2B0xy,

(2)

where κ =px2+ y2.

Finally, our set of intrinsic Ξ includes 5 lateral distor-tion parameters: three coefficients for the radial compo-nent, k(r) = {A

0, A1, A2}, and two for the tangential,

k(t)= {B 0, B1}. Sensor image bl object // al Main Lens A O MLA (u0, v0) d D F ∆c xw yw zw Ck,l ck,l0 s f(i) ∆C ck,l

Figure 1: Focused Plenoptic Camera model in Galilean con-figuration (i.e., the main lens focuses behind the sensor) with the notations used in this paper. Pixel counterparts of metric values are denoted in lower-case Greek letters.

A.2. F-number matching principle

The f -number of an optical system is the ratio between the system’s focal length F and the diameter of the entrance pupil, A, given by

N = F

A. (3)

The f -number accurately describes the light-gathering abil-ity of a lens only for objects an infinite distance away. In optical design, an alternative is often needed for systems where the object is not far from the lens. In these cases

(13)

the working f -number is used. The working f -number is defined as

N∗≈ 1

2NA ≈ (1 + |γ|) N , (4) where NA is the numerical aperture, i.e., the number that characterizes the range of angles over which the system can accept or emit light, and γ is the magnification of the current focus setting. Let a be the quantity that verify the following thin lens equation:

1 F = 1 a+ 1 D, (5)

where F is the focal length and D is the distance between the sensor and the lens. Then we can express the magnifi-cation as

γ = D

a, (6)

and thus we can rearrangeEq. (3)andEq. (4)to obtain the working f -number expressed as:

N∗=  1 + D a  F A = D  1 D + 1 a  F A = D A. (7)

The fundamental design principle for light-field imaging is that the working f -numbers of the micro-lenses and the main lens are matched. This condition maximizes the fill factor of the sensor while avoiding overlap between micro-images [3]. Both unfocused and focused plenoptic camera designs follow the f -number matching principle. As high-lighted in [5], the micro images generated by the micro-lenses in a plenoptic camera should just touch to make the best use of the image sensor, meaning

d ∆C = Ds− d A ⇐⇒ N ∗ µ= Nl∗− d A, (8) where d is the distance between the Micro-Lenses Array (MLA) and the sensor, Ds= D + d is the distance between

the main lens and the sensor, ∆C and A are respectively the diameters of the micro-lenses and the main lens, and, Nµ∗and Nl∗ are respectively the working f -numbers of the micro-lenses and the main lens.

Since typically d  A, we have Nµ∗ ≈ Nl∗. So, the

working f -numbers of the main imaging system and the micro lens imaging system should match. This also implies that the design of the micro lenses fixes the f -number of the main lens that is used with the plenoptic camera.

A.3. Projection model

The blur aware plenoptic projection matrix P (i, k, l) through the micro-lens (k, l) of type i is computed as

P (i, k, l) = P (k, l) · Kf(i) =     d/s 0 uk,l0 0 0 d/s v0k,l 0 0 0 s∆C2 −s∆C 2 d 0 0 −1 0         1 0 0 0 0 1 0 0 0 0 1 0 0 0 −1/f(i) 1     =      d/s 0 uk,l0 0 0 d/s v0k,l 0 0 0 s∆C2 d1d− 1 f(i)  −s∆C 2 d 0 0 −1 0      , (9)

where d is the distance between the MLA and the sensor, s is the size of a pixel, ck,l0 = uk,l

0 v k,l 0

>

is the principal point of the micro-lens (k, l), ∆C is the diameter of the micro-lens, and f(i)is the micro-lens focal length.

B. Micro-image radii computation

B.1. Image moments fitting

From raw white images, we measure each micro-image (MI) radius % = |R| /s in pixel based on image moments fitting. We use the second order central moments of the micro-image to construct a covariance matrix. Raw mo-ments and centroid are given by

Mij= X x,y xiyjI (x, y) and {¯x, ¯y} = M10 M00 ,M01 M00  ,

and the central moments are given by µpq=

X

x,y

(x − ¯x)p(y − ¯y)qI (x, y) . (10)

The covariance matrix is then given by

cov [I (x, y)] = 1 µ00 µ20 µ11 µ11 µ02  =σxx σxy σyx σyy  . (11)

Finally, we chose σ as the square root of the greater eigen-value of the covariance matrix, i.e.,

σ2= λmax= σxx+ σyy 2 + q 4σ2 xy+ (σxx− σyy)2 2 . (12) Finally, the radius % is proportional to the computed stan-dard deviation σ.

(14)

Type N = 5.66 N = 8 N = 11.31 i = 1 56.79±0.56 47.08±0.53 42.76±1.04 i = 2 53.68±0.95 41.71±0.78 35.68±1.36 i = 3 56.58±0.91 44.46±0.73 38.98±1.32

Table 1: Statistics (mean±std) over radii measurements (in µm) for each type of micro-image at different apertures.

B.2. Radii distribution analysis

An analysis of the micro-image radii distribution is given for three apertures N ∈ {5.66, 8, 11.31} inTab. 1for the dataset R12-A.

As expected, the radius decreases whilst the f -number N increases. The standard deviation is less than one-fifth of a pixel, meaning that our method provides precise results.

B.3. Coefficients estimation

From radii measurements at different f -numbers, we want to estimate the coefficients X =m q1 . . . qI

> of R N−1 = m · N−1+ q i (13) with m = dF 2D and qi= 1 f(i) ·  ∆cD d + D  ·d 2 − ∆c 2 . (14) Note that m is a function of fixed physical parameters inde-pendent of the micro-lenses focal lengths and the main lens aperture. Therefore, we obtain a set of linear equations, sharing the same slope, but with different y-intercepts. This set of equations can be rewritten as

AX = B, and then X = A>A−1 A>B where the matrix A (containing the f -numbers and a se-lector of the corresponding y-intercept coefficient) and B (containing the radii measurements) are constructed by ar-ranging the terms given the focal length at which they have been calculated. Finally, we compute X with a least-square estimation.

C. Supplements on calibration results

Some parameters have been omitted in the main paper for compactness. Tab. 2presents the complete set of intrin-sic parameters with their initial and optimized values for our method and compared to [4]. As stated, the distortions coefficients are really low, and the MLA rotations are neg-ligible. Indeed, in the case of industrial plenoptic cameras, such as Raytrix ones, a careful attention is payed to the co-planarity of the MLA and the sensor plane.

D. Insight about the datasets

We have presented three datasets in the submitted work: R12-A, R12-B, and R12-C. The devignetted images of the calibration targets from the dataset R12-A (Fig. 2), R12-B (Fig. 4), and R12-C (Fig. 6) taken at various angles and distances are presented below along with the poses at which they have been taken (Fig. 3,Fig. 5, andFig. 7).

D.1. Software and setup

All images has been acquired using the free soft-ware MultiCam Studio (v6.15.1.3573) of the company Euresys. The shutter speed has been set to 5 ms. While taking white images for the pre-calibration step, the gain has been set to its maximum value. For Raytrix data, we use their proprietary software RxLive (v4.0.50.2) to cali-brate the camera, and compute the depth maps used in the evaluation.

D.2. Dataset R12-A

The dataset has been taken at short focus distance, h = 450 mm. Therefore, the checkerboard squares size had to be decreased to 10 mm so we can observe the corner in image space. All the poses have been acquired at distances between 400 and 175 mm from the checkerboard.

Controlled evaluation. The dataset is composed of 11 poses taken with a relative step of 10 mm between each pose along the z-axis direction, at distances between 385 and 265 mm.

D.3. Dataset R12-B

The dataset has been taken at middle focus distance, h = 1000 mm. Therefore, the checkerboard squares size is set to 20 mm so we can observe the corner in image space. All the poses have been acquired at distances between 775 and 400 mm from the checkerboard.

Controlled evaluation. The dataset is composed of 10 poses taken with a relative step of 50 mm between each pose along the z-axis direction, at distances between 900 and 450 mm.

D.4. Dataset R12-C

The dataset has been taken at long focus distance, h = ∞. Therefore, the checkerboard squares size had to be in-creased to 30mm so we can observe the corner in image space. All the poses have been acquired at distances be-tween 2500 and 500 mm from the checkerboard.

Controlled evaluation. The dataset is composed of 18 poses taken with a relative step of 50 mm between each pose along the z-axis direction, at distances between 1250 and 400 mm.

(15)

R12-A (h = 450mm) R12-B (h = 1000mm) R12-C (h = ∞)

Unit Initial Ours [4] Initial Ours [4] Initial Ours [4]

F [mm] 50 49.720 54.888 50 50.047 51.262 50 50.011 53.322 A0 [×10−5] 0 23.145 6.099 0 1.686 0.023 0 18.647 1.393 −A1 [×10−6] 0 2.934 0.925 0 0.177 0.093 0 2.652 0.382 A2 [×10−8] 0 1.078 0.303 0 0.023 0.007 0 1.036 0.104 B0 [×10−5] 0 −9.870 −15.028 0 14.373 12.143 0 18.971 27.720 −B1 [×10−5] 0 5.387 5.020 0 18.947 18.164 0 7.600 4.461 D [mm] 56.658 56.696 62.425 52.113 52.125 53.296 49.384 49.384 52.379 −tx [mm] 11.293 11.129 9.771 11.299 12.503 12.672 11.302 13.213 14.159 −ty [mm] 8.411 8.186 8.334 8.416 6.305 6.114 8.418 7.256 6.231 −θx [µrad] 0 875.200 468.600 0 648.400 576.200 0 575.100 487.700 θy [µrad] 0 669.700 321.800 0 417.400 350.400 0 445.600 366.100 θz [µrad] 0.553 30.100 25.300 16.989 36.600 35.300 27.961 44.500 49.100 ∆C [µm] 127.51 127.45 127.38 127.47 127.45 127.40 127.54 127.50 127.42 f(1) [µm] 578.15 577.97 - 581.10 580.48 - 554.35 556.09 -f(2) [µm] 504.46 505.21 - 503.96 504.33 - 475.98 479.03 -f(3) [µm] 551.67 551.79 - 546.39 546.37 - 518.98 521.33 -u0 [pix] 2039 2042.55 2289.83 2039 1790.94 1759.29 2039 1661.95 1487.2 v0 [pix] 1533 1556.29 1528.24 1533 1900.19 1934.87 1533 1726.91 1913.81 d [µm] 318.63 325.24 402.32 336.84 336.26 363.17 307.93 312.62 367.40

Table 2: Initial intrinsic parameters for each dataset along with the optimized parameters obtained by our method and with the method of [4].

E. Source code and datasets redistribution

The datasets and the source code are publicly avail-able to the community. Datasets can be downloaded

from https://github.com/comsee-research/

plenoptic-datasets.

We also developed an open-source C++ library for plenoptic camera named libpleno which is available athttps:

//github.com/comsee-research/libpleno.

Along with this library, we developed a set of tools to pre-calibrate and calibrate a multifocus plenoptic cam-era, named COMPOTE (standing for Calibration Of Multi-focus PlenOpTic camEra) which is available at https:

//github.com/comsee-research/compote.

Acknowledgments

This work has been sponsored by the AURA Region and the European Union (FEDER) through the MMII project of CPER 2015-2020 MMaSyF challenge. We thank Charles-Antoine Noury and Adrien Coly for their insightful discus-sions and their help during the acquisitions.

References

[1] Duane Brown. Decentering Distortion of Lenses - The Prism Effect Encountered in Metric Cameras can be Over-come Through Analytic Calibration. Photometric Engineer-ing, 32(3):444–462, 1966.1

[2] Ae Conrady. Decentered Lens-Systems. Monthly Notices of the Royal Astronomical Soceity, 79:384–390, 1919.1

[3] Ren Ng, Marc Levoy, Gene Duval, Mark Horowitz, and Pat Hanrahan. Light Field Photography with a Hand-held Plenop-tic Camera. Technical report, Stanford University, 2005.2

[4] Charles Antoine Noury, Céline Teulière, and Michel Dhome. Light-Field Camera Calibration from Raw Images. DICTA 2017 - 2017 International Conference on Digital Image Computing: Techniques and Applications, 2017-Decem:1–8, 2017. 3,4

[5] Christian Perwaßand Lennart Wietzke. Single Lens 3D-Camera with Extended Depth-of-Field. In Human Vision and Electronic Imaging XVII, volume 49, page 829108. SPIE, feb 2012. 2

(16)

Figure 2: Devignetted images of the calibration targets (9 × 5 of 10mm side checkerboard) from the dataset R12-A taken at various angles and distances.

(17)

Figure 4: Devignetted images of the calibration targets (8 × 5 of 20mm side checkerboard) from the dataset R12-B taken at various angles and distances.

(18)

Figure 6: Devignetted images of the calibration targets (7 × 5 of 30mm side checkerboard) from the dataset R12-C taken at various angles and distances.

Figure

Figure 1: The Raytrix R12 multi-focus plenoptic cam- cam-era used in our experimental setup (a), along with a raw image of a checkerboard calibration target (b)
Figure 2: Focused Plenoptic Camera model in Galilean con- con-figuration (i.e., the main lens focuses behind the sensor) with the notations used in this paper.
Figure 4: Formation of a micro-image with its radius R through a micro-lens while taking a white image at an  aper-ture A
Figure 5: Micro-image radii as function of the inverse f - -number with the estimated lines (in magenta)
+7

Références

Documents relatifs

The ACT model considers two orthogonal components for the roughness as suggested by Ward (1992). The ACT model includes four parameters to describe the surface re fl ectance, a)

For a particular soil profile it has been found that values of AF derived from GRNN approach are closer to that of 1D linear viscoelastic seismic analysis particularly if the number

The goal of this paper is to present the association of a classical graph kernel [3], combined with a new graph kernel which takes into account chirality [1, 2], in order to predict

The same overall optimization procedure as described in section 3.3 is adopted with the clas- sical homogenization solution with the following changes: (i) the microscopic unit

By performing some text preprocess- ing and taking into account a frequency of a term in a specific window of terms (see (Duthil et al., 2011) for a full description of the process

The stereo kernel [6] which compares the minimal stereo subgraphs of two ordered graphs has one drawback : graph information are reduced to a bag of subgraphs without taking

Injection molding simulation : Taking into account the process history to predict the anisotropy in the

The forming simulation is based on a hypoelastic model for the textile reinforcement and it provides the yarn directions and therefore the warp and weft angle changes after