Vision 3D artificielle - Final exam (duration: 2h30)
P. Monasse and R. Marlet November 5th, 2013
You can choose to answer in French or English, at your convenience.
1 Disparity map with cost volume filtering
The recent method presented in this exercise comparesadaptiveimage patches.
Given two imagesul andur from a stereo pair, we define the cost volume as a 3-D image:
C(i, j, d) =|ul(i+d, j)−ur(i, j)|
wheredspans the set of possible disparities. We apply a filter to the cost volume C, yielding a volume of same dimensions C0, then we select the disparity by a simple “Winner take all” (WTA) criterion:
disp(i, j) = arg min
d C0(i, j, d)
1. Why is it necessary in practice to not useC directly in WTA (C=C0)?
2. Interpret SAD on a windowωi,j = [i−r, i+r]×[j−r, j+r] as a filter on the cost volume. In the following, we will build a more adaptive filter.
All windowsωi,j being supposed of fixed size (2·r+ 1)2, we note simply
|ω|(without index) this value.
3. Let (i0, j0) be a pixel oful. We define the energy Ei0,j0,d(a, b) = X
(i,j)∈ωi0,j0
(a·ul(i, j) +b−C(i, j, d))2+|ω|a2
where >0 is a “small” real number whose only use is to avoid the risk of dividing by 0 below.
We look for realsaandbsuch as to approximateC(i, j, d) at a givend:
(ai0,j0,d, bi0,j0,d) = arg min
a,b Ei0,j0,d(a, b).
Figure 1: Top: images ul and ur. Bottom: disparity maps computed by SAD on square windows (left) and by cost volume filtering (right).
Compute the gradient of the above energy and prove that ai0,j0,d=
P
ωi0,j0ul(i, j)C(i, j, d)/|w| −µi0,j0·C¯i0,j0,d
σi2
0,j0+ bi0,j0,d= ¯Ci0,j0,d−ai0,j0,dµi0,j0
where µi0,j0 and σ2i0,j0 are the average and variance of ul onωi0,j0, and C¯i0,j0,d the average ofC onωi0,j0× {d}.
4. Each pixel (i, j) belonging to several windowsωi0,j0, this would yield sev- eral values ofaandbfor (i, j). We take the average:
¯
ai,j,d= 1
|ω|
X ai0,j0,d
Show that we can write C0(i, j, d) =X
i0,j0
Wi,j,i0,j0C(i0, j0, d) with
Wi,j,i0,j0 = 1
|ω|2
X
(k,l)∈ωi,j∩ωi0,j0
1 +(ul(i, j)−µk,l)(ul(i0, j0)−µk,l) σk,l2 +
! .
5. Supposing ul takes only two values on ωi,j ∩ωi0,j0, comment the term under the sum.
6. See in Figure 1 the disparity maps computed from a simple search by minimization of SAD on square windows or after cost volume filtering.
Comment the better quality of the filtering method, in particular con- cerning a common drawback of local methods.
2 Multiple view constraints
The goal of this exercise is to exhibit the constraints arising from more than two views of a scene. Suppose we have a pointX in 3-D space viewed inn >2 images:
λixi=Ki Ri Ti
X, i= 1, . . . , n.
1. Recall the meaning of the different terms in such an equation.
2. Explain why we do not lose generality if we assumeR1=Id3 andT1= 0.
3. Write the above system of equations in the form a systemAY = 0 with Y = X λ1 · · · λnT
.
4. Link the existence and uniqueness of the system to the rank of matrixA.
5. Show that the rank ofAis the rank of B plus 3, where:
B =
K2T2 K2R2x1 x2 0 0 · · · 0
K3T3 K3R3x1 0 x3 0 · · · 0
... ... ... . .. . .. . .. ...
Kn−1Tn−1 Kn−1Rn−1x1 0 · · · 0 xn−1 0
KnTn KnRnx1 0 · · · 0 0 xn
6. Show that the following matrixD is of maximal rank 3(n−1):
D=
xT2 0 0 · · · 0
0 xT3 0 · · · 0
... . .. . .. . .. ...
0 · · · 0 xTn−1 0
0 · · · 0 0 xTn
[x2]× 0 0 · · · 0
0 [x3]× 0 · · · 0
... . .. . .. . .. ...
0 · · · 0 [xn−1]× 0
0 · · · 0 0 [xn]×
.
7. Deduce thatDB andB have same rank.
8. Write the contents of matrixDB.
9. Using the fact thatxi6= 0 for all i, prove that the rank of B is the rank ofM plus (n−1), withM the 3(n−1)×2 matrix:
M =
[x2]×K2R2x1 [x2]×K2T2
... ...
[xn]×KnRnx1 [xn]×KnTn
.
10. Show that the rank of M is zero if and only if all optical centers and X are aligned.
11. Show that forM to be rank 1, it is necessary that (i) for allithe vectors [xi]×KiTi and [xi]×KiRixi be proportional, (ii) which amounts to the usual two-image epipolar constraint (“bilinear constraint”).
12. We assume known that ifai6= 0 andbi 6= 0 are all vectors ofR3, the rank deficiency of the matrix
a1 b1
... ... an vn
is equivalent to the condition: aibTj −bjaTj = 0 for alli, j. Write “trilinear constraints” linking pointsx1,xi andxj.
13. Show that any possible “quadrilinear” or “multilinear” constraint is a combination of trilinear and bilinear constraints.
3.1 Multiple labels
The course presents a method based on graph cuts to compute an exact energy minimization for a label assignment problem in a case where there are multiple labels (more than two) and where the labels are linearly ordered (Boykov et al.
1998). Given a 4000×3000-pixel image and assuming we have no information on the possible range of the disparity,
1. What should be the size of the graph?
2. What would be the size of the graph if we were to use iteratedα-expansions?
3. Conclude.
3.2 Impact of pairwise potentials
1. Imagine you are facing stairs. Is it appropriate to use a Potts model as pairwise potentials? Why?
2. Imagine you are facing a pyramid. Is it appropriate to use a cap max value fonction as pairwise potentials? Why?
