Shape processing

(1)

A local frequency analysis framework for shape processing

Yohann Béarzi, Julie Digne, Raphaëlle Chaine

LIRIS - Équipe GeoMod - CNRS

2017-11-15

(2)

Shape processing

Local analysis: Curvature estimation, patch-based analysis (denoising, super-resolution)...

Global analysis: Manifold Harmonics Transform [Vallet, Lévy 2008], [Taubin 95]

“More than global” analysis: Shape recognition in databases [ShapeNet - Chang 2015]

(3)

Application: Shape detail enhancement

(4)

Manifold Harmonics Transform and Inverse Transform

Manifold Harmonics Transform (MHT):f: function defined on the vertices of a meshf =P

ix_if_i. Then:

f˜i =hf, φii=

n

X

j=1

xihfi, φji(MHT)

f =

n

X

i=1

˜f_iφ_i (Inverse MHT).

Spherical Harmonics [Kazhdan03], Compressed Manifold Modes [Neumann14]

(5)

Manifold Harmonics Transform and Inverse Transform

Manifold Harmonics Transform (MHT):f: function defined on the vertices of a meshf =P

ix_if_i. Then:

f˜i =hf, φii=

n

X

j=1

xihfi, φji(MHT)

f =

n

X

i=1

˜f_iφ_i (Inverse MHT).

Spherical Harmonics [Kazhdan03], Compressed Manifold Modes [Neumann14]

(6)

Local analysis: local height field

Height field over a plane:

p(x,y,h=f(x,y)) Or over a quadric

[Hamdi-Cherif2017]

p(x+hn^q_x(x,y),y+hn^q_y(x,y),z+hn^q_z(x,y))

~n

~t₁

lr O

(7)

Local analysis: local height field

Height field over a plane:

p(x,y,h=f(x,y)) Or over a quadric

[Hamdi-Cherif2017]

p(x+hn^q_x(x,y),y+hn^q_y(x,y),z+hn^q_z(x,y))

~n

~t₁

lr O

(8)

Local function basis for shape representation

Two strong assumptions on the surfaceS:

S can be locally be expressed as a height field over a planar parameterization in neighborhoods of radiusr

S is smooth,C^∞

Goal

Design a function basis taking into account both the local surface derivatives and the angular oscillations of the surface around one point of the surface.

Curvature computation using quadric regression [Chen92,Hamann93] or cubic regression [Goldfeather04]

(9)

Local function basis for shape representation

S is smooth,C^∞

Goal

(10)

Local function basis for shape representation

S is smooth,C^∞

Goal

(11)

Local function basis for shape representation: Jets [Cazals03]

Surface parameterized w.r.t. P(p)Not necessarily equal toT(p) (tangent plane)

Truncated Taylor expansion

S surface locally homeomorphic to a disk in a small neighborhood around a point p, expressed asf(x,y) over a planeP(p) passing throughp. The neighborhood ofp can be expressed as a Taylor Expansion:

f(x,y) =

∞

X

k=0 k

X

j=0

f_x^k−j_yj(0,0)

(k−j)!j! x^k^−jy^j (1) wheref_x^k−j_yj =_∂xk−j^∂^k^f∂y^j.

(12)

Local function basis for shape representation: Jets [Cazals03]

Accuracy theorem [Cazals03]

Given a Taylor expansion of orderK in a neighborhood of radiusr, the precision of allk order derivatives iso(r^K−k).

In practice:

Computation of the coefficients at each vertex or point by linear system solve.

(13)

Local function basis for shape representation: Jets [Cazals03]

Accuracy theorem [Cazals03]

Given a Taylor expansion of orderK in a neighborhood of radiusr, the precision of allk order derivatives iso(r^K−k).

In practice:

Computation of the coefficients at each vertex or point by linear system solve.

(14)

Local function basis for shape representation: Zernike polynomial [Zernike34]

V_p^q(ρ, θ) =R_p^q(ρ)e^iqθ

(p,q)∈ {|q∈Z,p∈Z^≥0,|q| ≤ p,p−qis even}

R_p^q(ρ) =P

k=|p|

p−qeven

(−1)^p−k2 p+k 2 p−k

2 !^k−q₂ !^k+q₂ !ρ^k In practice: projection of

neighborhoods (disks) on the basis.

Applied to image processing and surface processing [Khotanzad 88,Maximo 2011]

(15)

Local function basis for shape representation: Zernike polynomial [Zernike34]

V_p^q(ρ, θ) =R_p^q(ρ)e^iqθ

(p,q)∈ {|q∈Z,p∈Z^≥0,|q| ≤ p,p−qis even}

R_p^q(ρ) =P

k=|p|

p−qeven

(−1)^p−k2 p+k 2 p−k

2 !^k−q₂ !^k+q₂ !ρ^k In practice: projection of

neighborhoods (disks) on the basis.

Applied to image processing and surface processing [Khotanzad 88,Maximo 2011]

(16)

Definition

Wavejets

Locally we express the function as:

f(r, θ) =

∞

X

k=0 k

X

n=−k

φk,nr^ke^inθ (2)

withφk,n=Pk j=0

1

j!(k−j)!b(k,j,n)f_xk−jy^j(0,0).

b(k,j,n) = 0 ifk andndo not have the same parity b(k,j,n) = ₂k¹i^j

P^n−k₂

h=0

k−j h

j

n−k 2 −h

(−1)^h otherwise.

(17)

φ

k,n

coefficients

φ0,0 φ2,0 |φ2,2| |φ3,1| |φ3,3| φ4,0 |φ4,2| |φ4,4|

(18)

Properties

Gaussian Curvature

K(p) = 4φ²_2,0−16φ_2,−2φ2,2

(1 + 4φ_1,−1φ_1,1)² (3)

Mean Curvature

H(p) =2φ2,0(1 + 4φ_1,−1φ1,1) + 4φ_2,−2φ1,1+ 4φ2,2φ_1,−1 (1 + 4φ_1,−1φ1,1)³²

(4)

Parameterization plane

IfP(p) =T(p), the tangent plane toS atp, thenφ1,1=φ1,−1= 0, and : K(p) = 4 φ²_2,0−φ_2,−2φ_2,2

, H(p) = 2φ_2,0 (5)

(19)

Properties

Gaussian Curvature

K(p) = 4φ²_2,0−16φ_2,−2φ2,2

(1 + 4φ_1,−1φ_1,1)² (3)

Mean Curvature

H(p) =2φ2,0(1 + 4φ_1,−1φ1,1) + 4φ_2,−2φ1,1+ 4φ2,2φ_1,−1 (1 + 4φ_1,−1φ1,1)³²

(4)

Parameterization plane

IfP(p) =T(p), the tangent plane toS atp, thenφ1,1=φ1,−1= 0, and : K(p) = 4 φ²_2,0−φ_2,−2φ_2,2

, H(p) = 2φ_2,0 (5)

(20)

Principal directions

Principal curvatures:

κ₁= 2 (φ_2,0+φ_2,2+φ_2,−2) andκ₂= 2 (φ_2,0−φ_2,2−φ_2,−2) (6) P−2≤n≤2

n even

φ_2,ne^inθ+φ_2,−ne^−inθ has 2 maxima aligned with the principal directions

(21)

Principal directions

Principal curvatures:

κ₁= 2 (φ_2,0+φ_2,2+φ_2,−2) andκ₂= 2 (φ_2,0−φ_2,2−φ_2,−2) (6) P−2≤n≤2

n even

φ_2,ne^inθ+φ_2,−ne^−inθ has 2 maxima aligned with the principal directions

(22)

Higher order principal directions

Order 3 P−n≤3

n odd

φ3,ne^inθ+φ_3,−ne^−inθ has at most 3 maxima (either 1 or 3) See also [Joshi, Séquin 2010]

Order 3 maxima directions:

(23)

Higher order principal directions

Order 3 P−n≤3

n odd

φ3,ne^inθ+φ_3,−ne^−inθ has at most 3 maxima (either 1 or 3) See also [Joshi, Séquin 2010]

Order 3 maxima directions:

(24)

Stability with respect to the parameterization plane

(p,u) =T(p)T P(p)

γ: rotation angle along (p,u) to alignP(p) toT(p) OverT(p): f(r, θ) =P∞

k=0

Pn=k

n=−k φ_k_,nr^ke^inθ; OverP(p): f(R,Θ) =P∞

k=0

Pn=+k

n=−kΦk,nR^ke^inΘ Assumeθ and Θ are computed w.r.t. directionu

Stability Theorem

The coefficients Φk,n w.r.t toP(p) can be expressed with respect to the coefficientsφk,nin the tangent plane T(p) as follows:

Φ_0,0= 0

Φ1,1= Φ^∗_1,−1=γ

2e⁻ⁱ^π² +o(γ) Φk,n=φk,n+γF(k,n) +o(γ)

(7)

whereF(k,n) is a function of theφcoefficients of order lower thank. It is independent ofR<RΦ and Θ.

(25)

Parameterization plane correction

Consequence (1)

γ= 2|Φ1,1|+o(|Φ1,1|) andarg(Φ1,1) =π 2 +o(γ)

The phase of Φ1,1 shifted byπ/2 in the planeP(p) corresponds to the axis of rotationu. Therefore, it is possible to correct the parameterization by performing a rotation ofP(p) along the axisu with rotation angle 2|Φ1,1|.

(26)

Coefficient error correction

Consequence (2)

One can recover thetruecoefficientsφk,n, starting iteratively from the lowest order coefficients:

φ_k,n= Φ_k,n−γ

k−2

X

j=1

s_j,k,n+o(γ) (8)

sj,k,n= X

p+m=n

|p|≤k−j

|m|≤j

φk−j,p

2i (φj+1,m+1(m+j+ 2) +φj+1,m−1(m−j−2)) (9)

Rk: In particular,φ_2,0= Φ_2,0+o(γ),φ_2,2= Φ_2,2+o(γ), φ_2,−2= Φ_2,−2+o(γ)

(27)

Remark on Zernike and Jets

Normal error can be derived from order 1 jets coefficients,BUTfrom a nontrivial linear combinations ofV_k^±1

Error to the surface can be derived from order 0 jets coefficients,BUTfrom a nontrivial linear combinations ofV_k⁰

Integral invariants are easy to compute with Zernike polynomialBUTnot easily written using jets.

Wavejets

Normal error, error to the surface, integral invariants are easy to write with Wavejets

(28)

First application: normal correction

PCA 2-jet 5-jet VCM Hough Hough CNN Wavejets

Normal estimation on two intersecting cylinders creating a sharp edge. First row : Noise free, Second row : Gaussian noise 1.2% - Third row : Gaussian noise 3.6%

(29)

Link with Integral Invariants

V(s) signed volume between the surface andP(p) in a small radiuss<R_φ around one pointp.

V(s) =R2π

0 A(θ,s)dθ with A(θ,s) =

Z s

0

∞

X

k=0 k

X

n=−k

r^kφ_k,ne^inθ

! rdr =

∞

X

n=−∞

a_n(s)e^inθ (10)

Withan(s) =P∞ k=|n|

φ_k,ns^k+2 k+2

Link with Integral Invariants

Vs(p): volume of the intersection of a sphere and the interior of the surface (e.g.) Manay (2006), Pottmann (2007, 2009):

Vs(p)−2πa0≈ 2

3πs³ . (11)

(30)

Link with Integral Invariants

V(s) =R2π

Z s

0

∞

X

k=0 k

X

n=−k

r^kφ_k,ne^inθ

! rdr =

∞

X

n=−∞

a_n(s)e^inθ (10)

φ_k,ns^k+2 k+2

Link with Integral Invariants

Vs(p)−2πa0≈ 2

3πs³ . (11)

(31)

Link with Integral Invariants

V(s) =R2π

Z s

0

∞

X

k=0 k

X

n=−k

r^kφ_k,ne^inθ

! rdr =

∞

X

n=−∞

a_n(s)e^inθ (10)

φ_k,ns^k+2 k+2

Link with Integral Invariants

Vs(p)−2πa0≈ 2

3πs³ . (11)

(32)

Link with Integral Invariants

V(s) =R2π

Z s

0

∞

X

k=0 k

X

n=−k

r^kφ_k,ne^inθ

! rdr =

∞

X

n=−∞

a_n(s)e^inθ (10)

φ_k,ns^k+2 k+2

Link with Integral Invariants

Vs(p)−2πa0≈ 2

3πs³ . (11)

(33)

φe2,0 φe3,1 φe2,2 φe3,3

Input point Paraboloid Hyperboloid Horse saddle Monkey saddle

9-wavejets φe0 φe1 φe2 φe3

a₀ |a_±1| |a_±2| |a_±3|

Wavejets (order 9) decomposition of a real surface.

φek,n(r, θ) =r^k φk,ne^inθ+φ_k,−ne^−inθ

andφen=P∞ k=0φek,n.

(34)

Computing Wavejets on point sets

Wavejets regression

Assume we haveLneighboring points (r_l, θ_l,z_l) then, we minimize:

E(Φ) =

L

X

l=1

z_l−

K−1

X

k=0 k

X

n=−k

r_l^ke^inθ^lφ_k_,n

2

To remove outliers we add a weight and use Iteratively Reweighted Least Squares

Solve performed by QR decomposition.

(35)

Computing Wavejets on point sets

Wavejets regression

E(Φ) =

L

X

l=1

z_l−

K−1

X

k=0 k

X

n=−k

2

(36)

Computing Wavejets on point sets

Wavejets regression

E(Φ) =

L

X

l=1

z_l−

K−1

X

k=0 k

X

n=−k

2

(37)

Results

Noisy

normals φ0,0 |φ1,1| φ2,0 |φ2,2| |φ3,1| |φ3,3|

(38)

Position enhancement filter

Unsharp Masking,inverse curvature motion, [Gabor 1965]

Position update

Movep to its new positionp⁰:

p⁰ =p+ (φ0,0−2π(α0−1)a0(s))n

Continuous motion, assumingP(p) is corrected toT(p) beforehand.

(39)

Position enhancement filter

Unsharp Masking,inverse curvature motion, [Gabor 1965]

Position update

Movep to its new positionp⁰:

p⁰ =p+ (φ0,0−2π(α0−1)a0(s))n

Continuous motion, assumingP(p) is corrected toT(p) beforehand.

(40)

Normal enhancement filter

Principle: Modifyφ_1,1 andφ_1,−1and deduce the false normal

Method

Computeφ_1,±1:

φ_1,±1=−π(α_±1−1)a_±1(s)

Deduce the normal by rotating the current normal of angle 2|φ⁰_1,1|and axis given byarg(φ⁰_1,1) +^π₂.

(41)

Normal enhancement filter

Principle: Modifyφ_1,1 andφ_1,−1and deduce the false normal

Method

Computeφ_1,±1:

φ_1,±1=−π(α_±1−1)a_±1(s)

Deduce the normal by rotating the current normal of angle 2|φ⁰_1,1|and axis given byarg(φ⁰_1,1) +^π₂.

(42)

Results

Original Enhanced normals Enhanced positions Normal and position enhancement on a bunny with 6-Wavejets. Rφis equal to 3%

of the shape diameter, andα0=α_±1= 2.

(43)

Results

α_±1= 2 α_±1=±2i α_±1=−2 α_±1 =∓2i α_±1= 0 Normal amplification

(44)

Comparisons with Unsharp masking

K = 8 K = 2 K = 8

Unsharp masking Ours Ours

28/38

(45)

Results

α0=−2 α0=−1 α0= 0 α0= 1 α0= 2 Position amplification

(46)

Resilience to noise

Input α0= 2 α±1= 0 α±1= 2

(47)

Results

Outputs of our procedures on an armadillo (Left:original; Middle: normal-based detail enhancement withK = 7, α±1= 3; Right: position-based detail

enhancement withK = 6, α0= 2).

(48)

Comparisons

Original Positions enhanced Normals enhanced

Top row: wavejets. Bottom row: high-boost filter in the manifold harmonics basis [Vallet-Lévy 2008].

(49)

Comparisons

α1= 24 α1= 24eⁱ^π⁴ α1= 24i α1= 24eⁱ^3π⁴

α₁=−24eⁱ^3π⁴ α₁=−24i α₁=−24eⁱ^π⁴ α₁=−24

Original Ours (α1= 24) [Cignoni05] [Rusinkiewicz06]

Normal enhancement on a golf ball withK = 9. (nb: α₋₁=α^∗₁)

(50)

Comparisons

Original [Rusinkiewicz06] [Cignoni05] (K = 5, α_±1= 3) Normal enhancement with different methods.

(51)

More results

Original Normal enhanced Position enhanced K = 7 (normal exaggeration) and order K = 6 filters (position filter)

(52)

More results

Original Normal enhanced Position enhanced

Applying order 9 (normal exaggeration) and order 8 filters (position filter) to the Anubis datasets withα0=α±1= 2.

(53)

Complexity per point

Given a set ofNneighbors and a Wavejet orderK: O(NK⁴)

Once the wavejet is computed, applying the filter amounts to summingK terms: O(K)

1.5M points, orderK = 6: Decomposition time 1min40s; Filtering time:

0.6s.

(54)

Conclusion

A new basis for representing the local variations of the surface

Interpretation for the third order principal directions as tensor eigenvectors (as defined by Qi 2007)

Filtering higher order derivatives usinga2,a3...

Filter design in this basis is still a work in progress.

Work funded by ANR PAPS (ANR-14-CE27-0003)

Wavejets: A Local Frequency Framework for Shape Detail Amplification, Yohann Béarzi, Julie Digne, Raphaëlle Chaine, conditionally accepted to Computer Graphics Forum (Eurographics 2018).