Markov Random Field for combined defogging and stereo reconstruction

(1)

Markov Random Field for combined defogging

and stereo reconstruction

MS45 Mathematical techniques for bad visibility restoration SIAM Conference on Imaging Science, June 5-8, 2018

Jean-Philippe Tarel and Laurent Caraffa

Universit´e Paris Est, IFSTTAR, IGN

(2)

Visual effect of fog

• Color Fades • Airlight added • Contrast and visibility decrease with distance

Clear, Visibility distance : 17km

(3)

Visual effect of fog

• Color Fades • Airlight added • Contrast and visibility decrease with distance Visibility distance : 5km

(4)

Visual effect of fog

• Color Fades • Airlight added • Contrast and visibility decrease with distance Visibility distance : 1km

(5)

Visual effect of fog

• Color Fades • Airlight added • Contrast and visibility decrease with distance Visibility distance : 500m

(6)

Visual effect of fog

• Color Fades • Airlight added • Contrast and visibility decrease with distance Visibility distance : 250m ⇒ Difficulties for object detection/recognition/identification

(7)

Model of fog visual effet

The Koschmieder law [Middleton52] :

I = I0e−βd + Is(1 − e−βd)

Foggy image I Image without fog I0 Depth map d

• Is is the sky intensity

• β is the extinction coefficient (related to the visibility distance)

⇒ For single image defogging, ambiguity between I0 and βd

(8)

Model of fog visual effet

I = I0e−βd + Is(1 − e−βd)

Foggy imageI

Image without fog I0 Depth map d

(9)

Model of fog visual effet

I =I0e−βd + Is(1 − e−βd)

Foggy image I Image without fog I0

Depth map d

(10)

Model of fog visual effet

I = I0e−βd+ Is(1 − e−βd)

Foggy image I Image without fog I0 Depth mapd

(11)

Model of fog visual effet

I = I0e−βd +Is(1 − e−βd)

(12)

Model of fog visual effet

I = I0e−βd+ Is(1 − e−βd)

(13)

Model of fog visual effet

⇒ For single image defogging, ambiguity between I0 andβd

(14)

Model of fog visual effet

(15)

Single image defoging

Foggy image Visibility restoration using CNN

AOT-Net [Li-ICCV17]

• Many variants

• Atmospheric veil Is(1 − e−βd) estimated from the pixels white

amount

• Filtering and guided filtering [He-PAMI13,Caraffa-IP15]

(16)

Single image defoging

Foggy image Visibility restoration using

[Tarel-ICCV09]

• Many variants

amount

(17)

Single image defoging

Foggy image Visibility restoration using Dark

Chanel Prior [He-CVPR09]

• Many variants

amount

(18)

Single image defoging

Foggy image Visibility restoration using

[Caraffa-IP15]

• Many variants

amount

(19)

Fog in stereo reconstruction

Left stereo image Right stereo image

• Problem : a wall is reconstructed before visibility distance due to decreasing contrast with distance

(20)

Fog in stereo reconstruction

Estimated disparity map using SGM [Hirschmuller-PAMI08]

Right stereo image

(21)

Fog in stereo reconstruction

Estimated disparity map with GC [Boykov-PAMI01]

Right stereo image

(22)

Fog in stereo reconstruction

Single Image defogging Right stereo image

(23)

Depth cues

Stereo reconstruction

• The intensity is related to the depth at far distances

• Complementary depth cues are provided by fog and stereovision

(24)

Depth cues

Virgin of the Rocks (Da Vinci, 1507)

(25)

Depth cues

Landscape of Virgin of the Rocks (Da Vinci, 1507)

(26)

Depth cues

Atmospheric veil after thresolding Stereo reconstruction

(27)

Depth cues

Atmospheric veil after thresolding Stereo reconstruction

(28)

Towards a MRF model

• Stereovision without fog

• Single image defogging knowing the depth

(29)

Towards a MRF model

(30)

Towards a MRF model

(31)

What we are looking for?

Bayesian approach : p(D, I0L|IL, IR) ∝ p(IL, IR|D, I0L) p(D, I0L) E (D, I0L|IL, IR) = E (IL, IR|D, I0L) | {z } Edata + E (D, I0L) | {z } Eprior (1)

(32)

What we are looking for?

(33)

What we are looking for?

(34)

What we are looking for?

(35)

What we are looking for?

(36)

Dense stereo reconstruction without fog

Without fog, I0L ≈ IL : E (D|IL, IR) = E (IR, IL|D) | {z } Edata stereo + E (D) | {z } Eprior stereo (2) Edata stereo = X (i ,j )∈X ρS( |IL(i , j ) − IR(i − D(i , j ), j )| σS ) Eprior stereo = λD X (i ,j )∈X X (k,l )∈N |D(i , j) − D(i + k, j + l )|

(37)

Dense stereo reconstruction without fog

(38)

Dense stereo reconstruction without fog

(39)

Dense stereo reconstruction without fog

(40)

Single image defogging knowing the depth

When depth d = _Dν is known :

E (I0|D, I ) = E (I |D, I0) | {z } Edata fog + E (I0|D) | {z } Eprior fog (3) Edata fog = X (i ,j )∈X ρP( |I0(i , j )e − βν D(i ,j ) _{+ I} s(1 − e − βν D(i ,j )_{) − I (i , j )|} σP ) Eprior fog = λI0 X (i ,j )∈X X (k,l )∈N e− βν D(i ,j ) |I0(i , j ) − I0(i + k, j + l )|

• ρp is related to the assumed noise distribution

• Factor e−

βν

(41)

Single image defogging knowing the depth

E (I0|D, I ) = E (I |D, I0) | {z } Edata fog + E (I0|D) | {z } Eprior fog (3) Edata fog = X (i ,j )∈X ρP( |I0(i , j )e− βν D(i ,j ) _{+ I} s(1 − e− βν D(i ,j )_{) − I (i , j )|} σP ) Eprior fog = λI0 X (i ,j )∈X X (k,l )∈N e− βν D(i ,j ) |I0(i , j ) − I0(i + k, j + l )|

• Factor e−

βν

(42)

Single image defogging knowing the depth

E (I0|D, I ) = E (I |D, I0) | {z } Edata fog + E (I0|D) | {z } Eprior fog (3) Edata fog = X (i ,j )∈X ρP( |I0(i , j )e− βν D(i ,j ) _{+ I} s(1 − e− βν D(i ,j )_{) − I (i , j )|} σP ) Eprior fog = λI0 X (i ,j )∈X X (k,l )∈N e− βν D(i ,j ) |I0(i , j ) − I0(i + k, j + l )|

• Factor e−

βν

(43)

Single image defogging knowing the depth

E (I0|D, I ) = E (I |D, I0) | {z } Edata fog + E (I0|D) | {z } Eprior fog (3) Edata fog = X (i ,j )∈X ρP( |I0(i , j )e− βν D(i ,j ) _{+ I} s(1 − e− βν D(i ,j )_{) − I (i , j )|} σP ) Eprior fog = λI0 X (i ,j )∈X X (k,l )∈N e− βν D(i ,j ) _|I 0(i , j ) − I0(i + k, j + l )|

• Factor e−

βν

(44)

Effect of the factor in the prior

Eprior fog = λI0 X (i ,j )∈X X (k,l )∈N e− βν D(i ,j ) |I0(i , j ) − I0(i + k, j + l ))| λI0 = 0

(45)

Effect of the factor in the prior

Eprior fog = λI0 X (i ,j )∈X X (k,l )∈N e− βν D(i ,j ) |I0(i , j ) − I0(i + k, j + l ))| λI0 = 1

(46)

Effect of the factor in the prior

Eprior fog = λI0 X (i ,j )∈X X (k,l )∈N e− βν D(i ,j ) _|I 0(i , j ) − I0(i + k, j + l ))| λI0 = 1

(47)

Effect of the factor in the prior

(48)

Effect of the factor in the prior

(49)

Effect of the factor in the prior

(50)

Stereo reconstruction and defogging : Data term

Edata fog stereo =

X (i ,j )∈X ρP( |I_0L(i , j )e −βν D(i ,j ) _{+ I} s(1 − e −βν D(i ,j )_{) − I} L(i , j )| σP ) + ρP( |I_0L(i , j )e −βν D(i ,j ) _{+ I} s(1 − e −βν D(i ,j )_{) − I} R(i − D(i , j ), j )| σP ) (4)

Edata∗ = αE_{data stereo}+ (1 − α)E_{data fog stereo}

Edata =

Edata∗ if I_L(i , j ) 6= I_s

(51)

Stereo reconstruction and defogging : Data term

Edata fog stereo =

X (i ,j )∈X ρP( |I_0L(i , j )e −βν D(i ,j ) _{+ I} s(1 − e −βν D(i ,j )_{) − I} L(i , j )| σP ) + ρP( |I_0L(i , j )e −βν D(i ,j ) _{+ I} s(1 − e −βν D(i ,j )_{) − I} R(i − D(i , j ), j )| σP ) (4) Edata∗ = αE_{data stereo}+ (1 − α)E_{data fog stereo}

Edata =

(52)

Stereo reconstruction and defogging : Data term

Edata fog stereo =

X (i ,j )∈X ρP( |I_0L(i , j )e −βν D(i ,j ) _{+ I} s(1 − e −βν D(i ,j )_{) − I} L(i , j )| σP ) + ρP( |I_0L(i , j )e −βν D(i ,j ) _{+ I} s(1 − e −βν D(i ,j )_{) − I} R(i − D(i , j ), j )| σP ) (4) Edata∗ = αE_{data stereo}+ (1 − α)E_{data fog stereo}

Edata =

(53)

Complete energy and optimization

argmin

D,I0L,σp

E =αEdata stereo(D) + (1 − α)Edata fog stereo(I0L,D,σp)

+ λDEprior stereo(D) + (1 − α)λI0LEprior fog(I0L, D)

(5)

• α and λD are hyper-parameters, λI0L = 1

• D Approximated by an initial ¨D to simplify optimization

• Alternate optimization with respect to D

(54)

Complete energy and optimization

argmin

D,I0L,σp

E =αEdata stereo(D) + (1 −α)Edata fog stereo(I0L,D,σp)

+λDEprior stereo(D) + (1 −α)λI0LEprior fog(I0L, D) (5)

• α andλD are hyper-parameters, λI0L = 1

(55)

Complete energy and optimization

argmin

D,I0L,σp

+ λDEprior stereo(D) + (1 − α)λI0LEprior fog(I0L| ¨D)

(5)

• D Approximated by an initial D¨ to simplify optimization

(56)

Complete energy and optimization

argmin

D,I0L,σp

(5)

• Alternate optimization with respect toD

(57)

Complete energy and optimization

argmin

D,I0L,σp

(5)

• Alternate optimization with respect to D and I0L

(58)

Complete energy and optimization

argmin

D,I0L,σp

E =αEdata stereo(D) + (1 − α)Edata fog stereo(I0L,D,σp) + λDEprior stereo(D) + (1 − α)λI0LEprior fog(I0L| ¨D)

(5)

• Alternate optimization with respect to D and I0L

(59)

Results on real images

Left Image Defogging I0L

(60)

Results on real images

Left Image Defogging I0L

(61)

Conclusion

• Thanks to complementary depth cues between stereovision and fog, defogging and stereo reconstruction can be combined with advantages

(62)

Conclusion

• Thanks to complementary depth cues between stereovision and fog, defogging and stereo reconstruction can be combined with advantages

(63)