3D Model Acquisition by Tr 3D Model Acquisition by Tr
acking 2D Wireframes acking 2D Wireframes
Presenter: Jing Han Shiau Presenter: Jing Han Shiau
M. Brown, T. Drummond and R. Cipolla M. Brown, T. Drummond and R. Cipolla
Department of Engineering Department of Engineering
University of Cambridge University of Cambridge
Motivation Motivation
3D models are needed in graphics, reverse 3D models are needed in graphics, reverse engineering and model-based tracking.
engineering and model-based tracking.
Want to be able to do real-time tracking.Want to be able to do real-time tracking.
System Input/Output
System Input/Output
Other Approaches Other Approaches
Optical flow/ structure from motion Optical flow/ structure from motion
(Tomasi & Kanade, 1992) (Tomasi & Kanade, 1992)
- Acquire a dense set of depth measurements - Acquire a dense set of depth measurements - Batch method: not real-time
- Batch method: not real-time
Point matching between images.Point matching between images.
- Feature extraction followed by geometric constrai - Feature extraction followed by geometric constrai nt enforcement
nt enforcement
Edge extraction followed by line matching between Edge extraction followed by line matching between 3 views using trifocal tensors
3 views using trifocal tensors
Improvement Improvement
Previous approaches used siPrevious approaches used si ngle line segments, but 2D ngle line segments, but 2D
wireframes allow high level wireframes allow high level user constraints to reduce th user constraints to reduce th e number of degrees of free e number of degrees of free dom (6 degree of freedom E dom (6 degree of freedom E uclidean motion constraint) uclidean motion constraint)
since each new line segment since each new line segment
adds 4 degrees of freedom.
adds 4 degrees of freedom.
3D Positions of Lines 3D Positions of Lines
Internal Camera parameters are known.Internal Camera parameters are known.
Initial and final camera matrices are known Initial and final camera matrices are known through querying robot (arm) for camera pose.
through querying robot (arm) for camera pose.
Edge correspondence preserved using tracking.Edge correspondence preserved using tracking.
3D position of lines computed by triangulation.3D position of lines computed by triangulation.
Single Line Tracking Single Line Tracking
Sample points are initialized along each line Sample points are initialized along each line segment.
segment.
Search perpendicular to the line for local Search perpendicular to the line for local maxima of the intensity gradient.
maxima of the intensity gradient.
New line position is chosen to minimize the New line position is chosen to minimize the sum squared distance to the measured edge sum squared distance to the measured edge
positions.
positions.
Single Line Tracking
Single Line Tracking
Triangulation (Single Line tracking) Triangulation (Single Line tracking)
Finding 3D line by intersecting the rays Finding 3D line by intersecting the rays
corresponding to the ends of the line in the corresponding to the ends of the line in the
first image with the plane defined by the line first image with the plane defined by the line
in the second image.
in the second image.
Finding 3D Line Finding 3D Line
Find the 3D line by intersecting the world line Find the 3D line by intersecting the world line defined by the point (
defined by the point (u, vu, v) in the first image, ) in the first image, with the world plane defined by the line
with the world plane defined by the line in the second, is equivalent to solving the in the second, is equivalent to solving the
linear equations linear equations
Limitations Limitations
Object edges which project to epipolar lines mObject edges which project to epipolar lines m ay not be tracked.
ay not be tracked.
In case of a pure camera translation, epipolar li In case of a pure camera translation, epipolar li nes move parallel to themselves (radially with nes move parallel to themselves (radially with respect to the epipole); but the component of a respect to the epipole); but the component of a line’s motion parallel to itself is not observable line’s motion parallel to itself is not observable
locally.
locally.
2D Wireframe Tracking 2D Wireframe Tracking
Similar to line segment tSimilar to line segment t racking, least squares m racking, least squares m ethod is used to minimiz ethod is used to minimiz e the sum of the squared e the sum of the squared
edge measurements fro edge measurements fro
m the wireframe.
m the wireframe.
2D Wireframe Tracking 2D Wireframe Tracking
The vertex image motions are stacked into the The vertex image motions are stacked into the P-dimensional vector
P-dimensional vector pp, and the measurements , and the measurements are stacked into the D-dimensional vector
are stacked into the D-dimensional vector dd00..
DD is the new measurement vector due to the m is the new measurement vector due to the m otion
otion pp, and , and MM is the DxP measurement matrix is the DxP measurement matrix ..
Least squares is used to minimize the sum squLeast squares is used to minimize the sum squ ared measurement error |
ared measurement error |dd||22..
2D Wireframe Tracking 2D Wireframe Tracking
The least squares solution is:The least squares solution is:
But in general it is not unique. It can contain arBut in general it is not unique. It can contain ar bitrary components in the right nullspace of M, bitrary components in the right nullspace of M,
corresponding to displacements of the vertex i corresponding to displacements of the vertex i mage positions that do not change the measure mage positions that do not change the measure ments. Adding a small constant to the diagona ments. Adding a small constant to the diagona
l of M prevents instability.
l of M prevents instability.
3D Model Building 3D Model Building
2D wireframe tracking preser2D wireframe tracking preser ves point correspondence.
ves point correspondence.
3D position of the vertices ca3D position of the vertices ca n be calculated from 2 views n be calculated from 2 views
using triangulation.
using triangulation.
Observations from multiple vObservations from multiple v iews can be combined by mai iews can be combined by mai ntaining a 3D pdf for each ve ntaining a 3D pdf for each ve
rtex
rtex pp(X).(X).
3D pdf is updated on the basis o f the tracked image position of t he point, and the known camera.
3D Model Building 3D Model Building
A 3D pdf has surfaces of constant probability A 3D pdf has surfaces of constant probability defined by rays through a circle in the image p defined by rays through a circle in the image p lane. This pdf is approximated as a 3D Gaussia lane. This pdf is approximated as a 3D Gaussia n of infinite variance in the direction of the ray n of infinite variance in the direction of the ray
through the image point, and equal, finit, varia through the image point, and equal, finit, varia
nce in the perpendicular plane.
nce in the perpendicular plane.
3D Model Building 3D Model Building
The 3D pdf is the likelihood of the tracked poiThe 3D pdf is the likelihood of the tracked poi nt position, conditioned on the current 3D posi nt position, conditioned on the current 3D posi
tion estimate –
tion estimate –pp(w|X). (w|X).
Multiply this by the prior pdf to get the posteriMultiply this by the prior pdf to get the posteri or pdf:
or pdf:
3D Model Building 3D Model Building
X is Gaussian with mean X is Gaussian with mean mmpp and covariance ma and covariance ma trix C
trix Cpp, , w|Xw|X is Gaussian with mean is Gaussian with mean mmll, covaria, covaria nce matrix
nce matrix CCll, and X|w is Gaussian with mean , and X|w is Gaussian with mean mm and covariance matrix and covariance matrix C.C.
These are the Kalman filter equations used to mThese are the Kalman filter equations used to m aintain 3D pdfs for each point.
aintain 3D pdfs for each point.
Triangulation (3D Model Building) Triangulation (3D Model Building)
Instead of using multiple rays that pass Instead of using multiple rays that pass
through the image point as in the case of single through the image point as in the case of single
line tracking,
line tracking, probability distributionprobability distribution is used. is used.
Combining Tracking and Model Combining Tracking and Model
Building Building
There are 6 degrees of freedom corresponding There are 6 degrees of freedom corresponding to Euclidean position in space (3 translations a to Euclidean position in space (3 translations a
nd 3 rotations) for a rigid body.
nd 3 rotations) for a rigid body.
A wireframe of P/2 points has a P-dimensional A wireframe of P/2 points has a P-dimensional vector of vertex image positions.
vector of vertex image positions.
Model-based 2D Tracking Model-based 2D Tracking
The velocity of an image point for a normalizeThe velocity of an image point for a normalize d camera moving with translational velocity d camera moving with translational velocity UU
and rotating with angular velocity
and rotating with angular velocity ww about its o about its o ptical center is
ptical center is
where
where ZZc c is the depth in camera coordinates anis the depth in camera coordinates an d (d (u, vu, v) are the image coordinates. ) are the image coordinates.
Model-based 2D Tracking Model-based 2D Tracking
Stacking the image point velocities into a P-Stacking the image point velocities into a P- dimensional vector results in
dimensional vector results in
Each vector Each vector vvii forms a basis for the 6D forms a basis for the 6D
subspace of Euclidean motions in P space.
subspace of Euclidean motions in P space.
Model-based 2D Tracking Model-based 2D Tracking
Pros:Pros:
Converting a P degree of freedom tracking problem into a 6 Converting a P degree of freedom tracking problem into a 6
degree of freedom one.
degree of freedom one.
Cons:Cons:
The accuracy of the model (and the accuracy of the subspace The accuracy of the model (and the accuracy of the subspace
of its Euclidean motion) is poor initially.
of its Euclidean motion) is poor initially.
Conclusion: Conclusion: Accumulate 3D information from Accumulate 3D information from observations and progressively apply stronger observations and progressively apply stronger
constraints.
constraints.
Probabilistic 2D Tracking Probabilistic 2D Tracking
A second Kalman filter is used to apply weightA second Kalman filter is used to apply weight ed constraints to the 2D tracking.
ed constraints to the 2D tracking.
The constraints are encoded in a full PxP prior The constraints are encoded in a full PxP prior covariance matrix.
covariance matrix.
A Euclidean motion constraint can be included A Euclidean motion constraint can be included by using a prior covariance matrix of the form by using a prior covariance matrix of the form
Probabilistic 2D Tracking Probabilistic 2D Tracking
Writing P as Writing P as and assume and assume λλii are in are in dependent to get:
dependent to get:
The variance of the image motion is large in thThe variance of the image motion is large in th e directions corresponding to Euclidean motio e directions corresponding to Euclidean motio
n, and 0 in all other directions.
n, and 0 in all other directions.
The weights can be adjusted to vary the strengtThe weights can be adjusted to vary the strengt h of the constraints.
h of the constraints.
Probabilistic 2D Tracking Probabilistic 2D Tracking
To combine tracking and model building, To combine tracking and model building,
errors due to incorrect estimation of depth are errors due to incorrect estimation of depth are
permitted, weighted by the uncertainty in the permitted, weighted by the uncertainty in the
depth of the 3D point.
depth of the 3D point.
Only components of image motion due to Only components of image motion due to camera translation depend on depth.
camera translation depend on depth.
Probabilistic 2D Tracking Probabilistic 2D Tracking
For a 1 standard deviation error in the inverse For a 1 standard deviation error in the inverse depth, the image motions are
depth, the image motions are
Stacking the image point velocities into the P-Stacking the image point velocities into the P- dimensional vector to get
dimensional vector to get
Probabilistic 2D Tracking Probabilistic 2D Tracking
LetLet
Then Then
Ignore terms due to coupling between points to Ignore terms due to coupling between points to get get
The depth variance for each point can be compThe depth variance for each point can be comp uted from its 3D pdf by
uted from its 3D pdf by σσZcZc = = uuttCuCu, where , where uu is is a unit vector along the optical axis and
a unit vector along the optical axis and CC is the is the 3D covariance matrix.
3D covariance matrix.
Probabilistic 2D Tracking Probabilistic 2D Tracking
The final form of the prior covariance matrix isThe final form of the prior covariance matrix is
Which allows image motion due to Euclidean Which allows image motion due to Euclidean motion of the vertices in 3D, and also due to motion of the vertices in 3D, and also due to
errors in the depth estimation of these vertices.
errors in the depth estimation of these vertices.
Basic Ideas Basic Ideas
1.1. Wireframe geometry specification via user input. CWireframe geometry specification via user input. C an occur at any stage, allowing objects to be recons an occur at any stage, allowing objects to be recons
tructed in parts.
tructed in parts.
Basic Ideas Basic Ideas
2. 2D tracking Kalman filter. Takes edge measureme2. 2D tracking Kalman filter. Takes edge measureme nts and updates a pdf for the vertex image positions.
nts and updates a pdf for the vertex image positions.
Maintains a full PxP covariance matrix for the image Maintains a full PxP covariance matrix for the image
positions.
positions.
Basic Ideas Basic Ideas
3. 3D position Kalman filter. Takes known camera, a3. 3D position Kalman filter. Takes known camera, a nd estimate vertex image positions, and updates a pdf nd estimate vertex image positions, and updates a pdf
for the 3D vertex positions. Maintains separate 3x3 c for the 3D vertex positions. Maintains separate 3x3 c
ovariance matrices for the 3D positions.
ovariance matrices for the 3D positions.
Algorithm Flow Algorithm Flow
Combined tracking and Combined tracking and model building
model building algorithm.
algorithm.
3D position updates are 3D position updates are performed
performed
intermittently.
intermittently.
Results Results
Real time tracking and 3D reconstruction of Real time tracking and 3D reconstruction of church image.
church image.
Results Results
ME block – constructed in 2 stages exploiting ME block – constructed in 2 stages exploiting weighted model-based tracking constraints.
weighted model-based tracking constraints.
Results Results
Propagation of 3D pdfs.Propagation of 3D pdfs.
Evolution of model from initial planar hypotheEvolution of model from initial planar hypothe sis.sis.
Results Results
Objects reconstructed Objects reconstructed using the Model
using the Model
Acquisition System, Acquisition System,
with surfaces identified with surfaces identified
by hand.
by hand.
Computer generated Computer generated image using
image using
reconstructed objects.
reconstructed objects.
Thanks!
Thanks!
Q&AQ&A
Happy Thanksgiving!!!Happy Thanksgiving!!!
