Mitigation of Visibility Loss for Advanced
Camera based Driver Assistances
Nicolas Hautière, Jean-Philippe Tarel and Didier Aubert
Abstract—In adverse weather conditions, in particular in daylight fog, the contrast of images grabbed by in-vehicle cameras in the visible light range is drastically degraded, which makes the current driver assistances relying on cameras very sensitive to weather conditions. An onboard vision system should take weather effects into account. The effects of daylight fog vary across the scene and are exponential with respect to the depth of scene points. Because it is not possible in this context to compute the road scene structure beforehand contrary to fixed camera surveillance, a new scheme is proposed. Fog density is first estimated and then used to restore the contrast using a flat world assumption on the segmented free space in front of the moving vehicle. A scene structure is estimated and used to refine the restoration process. Results are presented using sample road scenes under foggy weather and assessed by computing the visibility level enhancement gained by the method. Finally, we show applications to the enhancement in daylight fog of low level algorithms used in advanced camera based driver assistances.
I. INTRODUCTION
Under adverse meteorological conditions, the contrast of images which are grabbed by a classical in-vehicle camera in the visible light range is drastically degraded, which makes current in-vehicle applications relying on such sensors very sensitive to weather conditions. An in-vehicle vision system should take fog effects into account to be more reliable. A first solution is to adapt the operating thresholds of the system or to deactivate it momentarily if these thresholds have been surpassed. A second solution is to remove weather effects from the image beforehand. Unfortunately, the effects vary across the scene. They are exponential with respect to the depth of scene points. Consequently, space invariant filtering techniques cannot be used directly to adequately remove weather effects from images. A judicious approach is to detect and characterize the weather conditions so as to estimate the decay in the image and then to remove it.
The majority of sensors dedicated to measuring visibility distances (scatterometer, transmissometer) are expensive to operate and quite often complicated to install and calibrate correctly. However, systems that meet this purpose with an onboard camera are encountered less frequently. Pomerleau [21] estimates visibility by means of measuring the contrast attenuation of road markings at various distances in front of a moving vehicle. Hautière et al. [12] detect the presence of daytime fog and estimate the meteorological visibility N. Hautière and J.-P. Tarel are with the Université Paris-Est, LEPSIS, INRETS-LCPC, 58 boulevard Lefebvre 75015 Paris, France e-mail: hau-tiere@lcpc.fr, tarel@lcpc.fr.
D. Aubert is with UniverSud, LIVIC, INRETS-LCPC, bldg 824, 14 route de la Minière, 78000 Versailles, France e-mail: didier.aubert@inrets.fr.
distance. An extension of their method using stereovision [9] or a monocular camera [3] estimates the so-called mobilized visibility distance. Mori et al. rely on a camera coupled with a millimeter wave radar to detect fog presence [18].
Methods which restore image contrast under bad weather conditions are encountered more often in the literature. Un-fortunately, they all have too strong constraints to be used onboard a moving vehicle using images taken at different times [20] or using images with different polarizing filters [22]. Recently, different methods have been proposed which rely only on a single image as input. [10] first estimates the weather conditions and approximates a 3D geometrical model of the scene which is given a priori and refined during the restoration process. The method is dedicated to in-vehicle applications. The key point is to optimize the scene model. In [24], image contrasts are restored by maximizing the contrasts of the direct transmission while assuming a smooth layer of airlight. The method may produce halos near depth discontinuities. [5] estimates the transmission in hazy scenes, relying on the assumption that the transmission and surface shading are locally uncorrelated. The method uses a local window-based operation and a graphical model. Despite their good looking results, these methods solve the problem using optimization algorithms (Powell’s method or graph cuts and belief propagation among other methods) which make them heavy to implement and run: five to seven minutes using a 600×400 image on a double Pentium 4 PC for [24] and 35 seconds using a 512×512 image on a dual core processor for [5]. Moreover, [5] can not deal with grayscale images, which is problematic for in-vehicle applications. Based on principle proposed in [24], i.e. the inference of the atmospheric veil, [13] and [25] have proposed improved algorithms. However, only [25] is fast enough to be used in real-time applications. The problem of these methods is that the depth map produced by their atmospheric veil inference may be erroneous due to the ambiguity between white objects and fog.
In this paper, we propose a scheme to restore automatically the contrast of any image grabbed by an in-vehicle camera which enables a real-time use of the proposed principle. Weather conditions are first estimated and then used to restore the contrast using a flat world assumption on the segmented free space in front of the equipped vehicle. Thanks to flat world assumption, the 3D scene structure is inferred and used to restore the contrast of the whole image.
The paper is organized as follows. We first present a model of fog visual effects. Second, our previous technique estimating the extinction coefficient of the atmosphere in the current image from a single camera is recalled from [12]
object radiance R illumination scattering direct transmission T airlight A object radiance R illumination scattering direct transmission T airlight A
Fig. 1. Fog or haze luminance is due to the scattering of daylight. Light coming from the sun and scattered by atmospheric particles towards the camera is the airlight A. It increases with the distance. The light emanating from the object R is attenuated by scattering along the line of sight. Direct transmission T of R decreases with distance.
for completeness. Once the weather condition is known, we present our free space area detection technique, see sec. IV. This one is used to reconstruct the 3D scene and to perform the contrast restoration. Then in sec. V, we present experimental results, the analysis of the method sensitivity as well as the assessment of the method with other relevant methods [13], [24], [25]. Finally in sec. VI, we give applications of the proposed contrast restoration method.
II. FOGEFFECTS ONVISION
The literature on the interaction of light with the atmosphere has been written over more than two centuries. These works still serve as reference for recent works in computer vision [4], [12], [19]. In this section, selected results dealing with fog effects on vision are presented.
A. Visual Properties of Fog
The attenuation of luminance through the atmosphere was studied by Koschmieder [17], who derived an equation relating the apparent luminance or radiance L of an object located at distance d to the luminance L0 measured close to this object:
L = L0e−βd+ L∞(1 − e−βd) (1)
This expression indicates that the luminance of the object seen through fog is attenuated in e−kd (Beer-Lambert law); it also
reveals a luminance reinforcement of the form L∞(1 − e−βd)
resulting from daylight scattered by the slab of fog between the object and the observer, also named airlight. L∞ is the
atmospheric luminance. In the presence of fog, it is also the background luminance on which the target can be detected. The previous equation may then be written as follows:
L − L∞= (L0−L∞)e−βd (2)
On the basis of this equation, Duntley developed a contrast attenuation law [17], stating that a nearby object exhibiting contrast C0with the background will be perceived at distance
d with the following contrast: C =(L
0−L∞)/L∞e−βd= C
0e−βd (3)
This expression serves to base the definition of a standard dimension called "meteorological visibility distance" Vmet, i.e.
the greatest distance at which a black object (C0 = −1) of
a suitable dimension can be seen in the sky on the horizon,
with the threshold contrast set at 5% [1]. It is thus a standard dimension that characterizes the opacity of a fog layer. This definition yields the following expression:
Vmet= − 1 β log(0.05) ≃ 3 β (4) B. Camera Response
Let us denote f the camera response function, which models the mapping from scene luminance to image intensity by the imaging system, including optic as well as electronic parts [7]. With the notations of Fig. 1, the intensity I of a pixel is the result of f applied to the sum of the airlight A and the direct transmission T , i.e:
I = f (L) = f (T + A) (5)
In this work, we assume that the conversion process between incident energy on the CCD sensor and the intensity in the image is linear. This is generally the case for short exposure times, because it prevents CCD array to be saturated. Further-more, short exposure times (1 to 4 ms) are used on in-vehicle cameras to reduce the motion blur. This assumption can thus be considered as valid and (5) becomes:
I = f (T ) + f (A) = f (L0e−βd) + f (L∞(1 − e−βd))
= f (L0)e−βd+ f (L∞)(1 − e−βd)
= Re−βd+ A∞(1 − e−βd) (6)
where R is the intrinsic intensity of the pixel, i.e. the intensity corresponding to the intrinsic luminance value of the corre-sponding scene point and A∞is the background sky intensity.
III. ESTIMATION OF THEEXTINCTIONCOEFFICIENT OF THEATMOSPHERE
For automotive applications, a camera or a LIDAR can both be used to recover the value of the atmospheric extinction coefficient β. In this section, a method to compute the extinc-tion coefficient β using a single camera behind the vehicle windshield is recalled from [12], [16].
A. Flat World Hypothesis
In the image plane, the position of a pixel is given by its (u,v) coordinates. The coordinates of the optical center projection in the image are designated by (u0,v0). In Fig. 2,
H denotes the mounting height of the camera, θ the angle between the optical axis of the camera and the horizontal, and vhthe horizon line. The intrinsic parameters of the camera are
its focal length fl, and the horizontal size tpu and vertical size
tpv of a pixel. We have also made use herein of αu= tfpul and
αv= tfpvl, and have typically considered: αu ≈αv= α. The
hypothesis of a flat road is adopted, which makes it possible to associate a distance d with each line v of the image (see [12] for further details):
d = λ
v − vh
if v > vh, where λ =
Hα
f d θ S X Y Z C x y z v u vh H M Road plane Image plane θ
Fig. 2. Modeling of the camera within its environment; it is located at a height of H in the (S,X,Y ,Z) coordinate system relative to the scene. Its intrinsic parameters are its focal length f and pixel size t. θ is the angle between the optical axis of the camera and the horizontal. Within the image coordinate system, (u,v) designates the position of a pixel, (u0,v0) is the position of the optical center C and vhis the vertical position of the horizon line.
B. Recovery of Koschmieder’s Law Parameters
Following a variable change from d to v based on (7), (6) then becomes:
I = A∞+ (R − A∞)e−β
λ
v−vh (8)
By twice taking the derivative of I with respect to v, one obtains the following:
d2I dv2 = βϕ(v)e −β λ v−vh βλ v − vh −2 (9) where ϕ(v) = λ(R−A∞) (v−vh)3 . The equation d2I dv2 = 0 has two
solutions. The solution β = 0 is of no interest. The only useful solution is given in (10):
β =2(vi−vh)/λ (10)
where videnotes the position of the inflection point of I(v). In
this manner, the parameter β of Koschmieder’s law is obtained once vi is known. Finally, thanks to vi, vh and β values, the
values of the other parameters of (6) are deduced through use of Ii and dIdv |v=vi, which are respectively the values of the
function I and its derivative in v = vi:
Rroad = Ii−(1 − e−βdi) (vi−vh) 2e−βdi dI dv|v=vi (11) A∞ = Ii+ (vi−vh) 2 dI dv|v=vi (12) where Rroad is the intrinsic intensity of the road surface.
C. Implementation
1) Estimation of the inflection point: To estimate the
pa-rameters of (6), the median intensity on each line of a vertical band is estimated and an inflection point is detected. So as to be in accordance with Koschmieder’s law assumptions, this band should only take into account a homogeneous area and the sky. Thus, a region within the image that displays minimal line-to-line gradient variation when browsed from bottom to top is identified thanks to a region growing process, illustrated in Fig. 3. A vertical band is then selected in the detected area. Finally, taking the median intensity of each segment yields the vertical variation of the intensity of the image and the position of the inflection point.
(a) (b)
Fig. 3. (a) Results of the region growing on the test images: the road and the sky are partially segmented and are painted white. (b) Results of fog detection and meteorological visibility estimation: the black segments represent the measurement bandwidth; the curve on the left gives the vertical variation of intensity within the bandwidth; the horizontal line gives the image line representative of the meteorological visibility distance.
2) Estimation of the position of the horizon line: To obtain
the values of the parameters of (6), the position of the horizon line must be estimated. It can be estimated by means of the pitching of the vehicle when an inertial sensor is available, but is generally estimated by an additional image process-ing. Generally, this type of processing seeks to intersect the vanishing lines in the image. However, under foggy weather, the vanishing lines are only visible close to the vehicle. It is thus necessary to extrapolate the position of the horizon line through the fog. Consequently, this kind of process is prone to a significant standard deviation, even if in [15] a new approach can lead to improvements. For the moment, we use the a priori sensor calibration. More precisely, by carrying out a sensitivity analysis on (10), it can be easily shown that underestimating the difference of positions between vi and
vh is more penalizing that overestimating it. To have stable
measurements, we thus chose to set the horizon line slightly above its actual position. However, a recent publication seems to show interesting results allowing expecting more accurate results. Having now the vertical positions of both the inflection point and the horizon line, the parameters of (6) can be recovered and the position of the image line representative of the meteorological visibility distance is deduced. Fig. 3 illustrates the process.
IV. CONTRASTRESTORATIONMETHOD
In this section, we present a contrast restoration method based on Koschmieder’s law. The principle of the method as well as its different steps are successively detailed in the following sections.
Fig. 4. 3D plot of the contrast restoration function (17) for β = 0.05 and A∞ = 255. One can see that objects intensity tend to become null after contrast restoration.
A. Restoration Principle
In this section, we describe a simple method to restore scene contrast from an image of a foggy scene. Let us consider a pixel with known depth d. Its intensity I is given by (6). (A∞, β) characterizes the weather condition and is estimated
thanks to section III. Consequently, contrary to (11), R can be estimated directly for all scene points from:
R = Ieβd+ A∞(1 − eβd) (13)
The previous equation can be written as follows:
R − A∞= (I − A∞)eβd (14)
and thus the contrast Cr in the restored image is:
Cr=(I−A∞)/A∞eβd= Ceβd (15)
However, R may become negative for certain values of (I, d). We can solve the equation R(d∗) = 0 and obtain:
d∗(I) = 1 β log A∞ A∞−I (16) In case of negative values during the restoration process, we propose to set these values to 0. The restoration equation becomes finally:
R = maxh0, Ieβd+ A
∞(1 − eβd)
i
(17) To properly restore the scene contrast, the remaining problem is the estimation of the depth d of each pixel. The plot of this function is shown in Fig. 4.
B. Flat World Restoration
[10] proposed a parameterized 3D model of a road scene to restore the contrast. The proposed model is relevant for most road scenes but it is not enough generic to handle all traffic scenes configurations. In a first step, we propose to use a quite opposite scheme, which consists in only assuming that the road is flat. The distance of a pixel in the image is thus assumed given by (7). Only large distances are clipped using
a parameter c. The distance dcof a pixel p of coordinates (i,j)
is thus expressed by:
dc i ∈ [0, N [, j ∈ [0, M [ =
λ
/(j−vh)if N > j > c
λ/(c−vh)if 0 ≤ j ≤ c
(18) where N ×M denotes the size of the image and c ∈ [cmin, N −
vh]. cmin is used to reduce the modeling errors. In particular,
flat world and non flat world are mixed near the horizon line. It makes sense to set the position of this clipping plane equal to the meteorological visibility distance. Indeed, nothing can be seen further than Vmet. Using (4), (7) and (10), we thus
set:
cmin=(2vi+vh)/3 (19)
By using (18) in (17), the contrast of objects belonging to the road plane is restored. Conversely, the contrast of objects out of the road plane (other vehicles, trees...) is falsely restored since their distance in the scene is overestimated. Conse-quently, according to (13), their intensity R becomes null in the restored image, what is shown in Fig. 5. This is a major inconvenient, which was tackled in [8] by underestimating the value of the horizon line.
C. Free Space Area Detection
The inconvenient of the flat world restoration can be turned into our advantage. Indeed, by detecting the pixels whose intensity is null after contrast restoration for c = cmin, we
can very easily extract the free space in front of the vehicle accordingly by looking for the biggest connected component in front of the vehicle. This region is denoted D. To improve the results of this last step, a morphological opening of the connected component is performed. Some sample results are shown in Fig. 6. The segmented region Scmin, corresponding
to the vertical objets, is overlaid in red and the segmented free space region D is overlaid in green. The proposed method allows obtaining quite good results, even if some minor improvements could be made on the segmentation of curbs and very bright objects. The quality of these results can be compared with color based or stereovision approaches [14]. The good point in our method is that we only use one gray level image. However, it only works in daytime foggy weather. The classical methods and the proposed one are thus complementary. Second, we can take advantage of this segmentation to compute a 3D model of the environment. This is the topic of the next section.
Fig. 5. Sample result of flat world restoration. The intensity of vertical objects becomes null in the restored image: (a) original image and (b) result.
Fig. 6. Free space extraction of the road scene obtained thanks to the flat world restoration: (a) original images; (b) results of vertical objects segmentation in overlaid red and free space segmentation in overlaid green.
D. 3D Reconstruction
The segmentation of the vertical objects gives a practical solution to compute a simplified 3D model of the scene, which constitutes the key point of the method. Indeed, it allows to estimate the depth of the corresponding pixels (red layer in Fig. 6, denoted Scmin). In this aim, we test different values
of c, the parameter of the clipping plane, from the biggest to the smallest one. We compute the corresponding depth map dc using (18). We restore the vertical objects with respect to
this depth map and test if some pixels are saturated. If a pixel is saturated for a given value of c, the depth of the above pixels is set to the depth of the corresponding clipping plane, i.e. λ
c−vh. We keep the biggest value of c, which correspond
to the smallest depth leading to the saturation of the pixel. The depth of the other pixels, i.e. the pixels which belong to D and the sky, is set to dcmin. This practical 3D model is
mathematically expressed by:
∄c ∈ [cmin, N − vh[, p ∈ Sc, d1(p) = dcmin(p)
∃c ∈ [cmin, N − vh[, p ∈ Sc, d1(p) = dc∗ (20)
where dc∗ is defined by:
dc∗ = min
P ∈Sc c∈[cmin,N−vh[
dc (21)
Doing this, we build a 3D model of the environment. However, this 3D model has two problems. First, the 3D model is locally false. Indeed, some distances are overestimated since the depth of the vertical objects is assumed to be the depth at which the restored pixels becomes zero. Moreover, this depth depends on the albedo of the objects in the road scene. To solve this issue, we introduce a geometric constraint. Since we know the depth of the pixels belonging to the road plane, we compute a mean correcting coefficient ν to apply to the depth of the pixels belonging to the vertical objects. This one allows reducing the depth discontinuity along the border B between the free space area D and the vertical objects Scmin:
ν =P
p∈D∩Bd(p)/P
p∈Scmin ∩Bd(p) (22)
We then correct the depth of pixels belonging to Scmin as:
∀
p(i, j) ∈ Scmin, d2(p) = ν × d1(p)
∀p(i, j) ∈ Scmin, d2(p) = d1(p)
(23)
Second, depending on the step we choose to vary the value of c from N − vh to cmin, the obtained 3D model may be
discontinuous. In this aim, we smooth the 3D model using a Gaussian kernel to obtain a smoother 3D model:
∀ p(i, j) ∈ [0, N [×[0, M [, d3(p) = d2(p) ∗ Gσ (24)
where Gσ denotes a 2D Gaussian Kernel. Finally, we correct
the depth of the pixels which is higher than d∗ (see (16)):
∀p(i, j) ∈ [0, N [×[0, M [, d2> d∗(I), d3= d∗(I) (25)
Some samples of the obtained 3D reconstruction are shown in Fig. 7c. The albedo of the objects belonging to the road surface does not influence the reconstruction of their depth since we only use a geometric constraint. The depth of the clear vertical objects may be not correct. However, the continuity between the vertical objects and the free space area is corrected thanks to (23). This type of 3D reconstruction is denser than disparity maps obtained from dense stereovision approaches [26]. Nevertheless, this 3D reconstruction is subject to the ambiguity between white objects and fog. But this is not a problem since this final 3D reconstruction is only used to restore the image, as described in the next paragraph.
E. 3D Model Based Contrast Restoration
Based on (13) and the 3D reconstruction of the road scene, the contrast of foggy images may be restored. We thus maximize the contrasts of the road surface and of the near vertical objects. However, it is not adequate to totally restore the contrast, like [24] does, since it creates some visual artefacts. It seems better, like in [13] and [25], to reduce the contrast restoration rate. In this aim, we introduce a last parameter ρ ∈]0, 1[ to control the strength of the restoration. Based on (17), the final contrast restoration equation is given by:
R = maxh0, Ieβρd3+ A
∞(1 − eβρd3)
i
(26) To finalize the restoration, we lighten the image by a constant factor equal to the difference of the means of the bottom third of the images between the original and the restored image. Indeed, the restoration process tends to darken the image. In
Fig. 7. 3D model based contrast restoration results: (a) original images; (b) results; (c) estimated depth map; (d) contrast improvement gained by the method.
this way, we ensure that the original and the restored image have the same look close from the camera, where the fog is the less visible. Finally, apart from the fog input parameters (A∞, β) and the camera calibration parameters, the proposed
contrast restoration method has only two parameters ρ and Gσ.
In the following section, we evaluate the performances of the method. In particular, we study the sensitivity of the method to some of these parameters.
Some samples of contrast restoration are shown in Fig. 7b. To facilitate the comparison with the original images, these ones have been added in Fig. 7a. On one hand, the obtained images have their contrast largely improved where the objects are located far away. In particular, the vehicles and the cyclists are now clearly visible. On the other hand, the contrast is slightly improved at a closer range, where there is less fog. The next section of the article is dedicated to the assessment of our method.
V. ASSESSMENT OF THEMETHOD
To assess the performances of our method, we first propose a methodology to quantitatively evaluate the restoration. We then analyse the sensitivity of the method to the fog inputs parameters and the method internal parameters. Finally, we analyse the performances of the method by looking at the computation time and by comparing our method with current state of the art methods.
A. Quantitative Evaluation Methodology
To quantitatively assess and rate the contrast restoration method, we use the method dedicated for visibility restoration proposed in [11]. Stating that a contrast restoration algorithm should create new visible edges, increase the contrast of existing visible edges, and not create saturated pixels, this method computes three indicators e, ¯r and s allowing to compare two gray level images: the input image and the restored image. The visible edges in the image before and after restoration are selected by a 5% contrast thresholding. This allows to compute the rate e of edges newly visible after restoration. The maps of the ratio r of the gradients at visible edges are shown in Fig. 7d for our algorithm. Then, the mean ¯r over these edges of the ratio of the gradient norms after and before restoration is computed. This indicator ¯
r estimates the average visibility enhancement obtained by the restoration algorithm. At last, the percentage of pixels s which becomes completely black or completely white after restoration is computed. We propose to define an empirical restoration score τ, which is given by the sum of the different indicators.
τ = e + ¯r + 1 − s (27)
This τ indicator is helpful to plot the different indicators on a single curve. Thanks to these different indicators, we are able to assess the sensitivity of the method to its different inputs and parameters and to rate the different existing methods with respect to our method.
3,5 3,7 3,9 4,1 4,3 4,5 4,7 4,9 5,1 5,3 5,5 195 205 215 225 235 245 255 τ 3 3,5 4 4,5 5 5,5 6 0 0,02 0,04 0,06 0,08 0,1 0,12 β τ 3,8 3,9 4 4,1 4,2 4,3 4,4 4,5 4,6 4,7 0 0,2 0,4 0,6 0,8 1 ρ τ (a) (b) (c)
Fig. 8. Sensitivity analysis of the contrast restoration method: (a) sensitivity to the background sky intensity A∞, where the restoration score τ is plotted with respect to A∞value on sample images; (b) sensitivity to the fog density β; (c) sensitivity to the parameter ρ.
B. Sensitivity Analysis
1) Sensitivity to Fog Input Parameters: Fog input
param-eters are (A∞, β). The sensitivity of the method to these
parameters is moderate but can not be neglected to optimally run the algorithm. First, we studied the sensitivity to the sky intensity A∞. Based on (15), τ should be an increasing
function of A∞. To confirm or not this assertion, we computed
τ for different values of A∞on different sample images. The
resulting plot is given in Fig. 8a and confirms the relationship which binds τ and A∞. Based on this result, we propose to
set A∞ = 2n−1 for n bits images, i.e. A∞ = 255 in our
case.
We then study the sensitivity of the algorithm to the fog density value β, which is provided by the fog detection algorithm. In this aim, we computed τ for different values of β on different sample images issued from the same video sequence to have more or less the same fog density value (β ≈ 0.06). The resulting plot is given in Fig. 8b. For small values of the fog density, τ is quite small and increases rapidly with respect to β until reaching a maximum value. This value is roughly reached for the value of β provided by the fog detection algorithm. However, this curve is flat around the
(a)
e He&al Tan Tarel&al Our
all 2.56 2.63 4.69 2.26
bottom 1.74 1.81 5.31 1.14
top 3.96 3.77 3.57 3.37
(b) ¯
r He&al Tan Tarel&al Our
all 5.43 6.08 10.62 2.8
bottom 2.24 2.06 11.06 0.23
top 14.5 18.43 11.49 15.24
(c)
s He&al Tan Tarel&al Our
all 0.03 0.015 0 0
bottom 0.025 0.03 0 0
top 0.016 0.001 0 0
TABLE I
MEAN INDICATORSe, ¯r, sPRODUCED BY THE FOUR COMPARED METHODS ON NINE IMAGES.
maximum value, which means that the sensitivity to the fog density value is small. Indeed, the algorithm needs a correct value for the product βd rather than a correct value of β. If the value of β is too small, the algorithm compensates and computes a bigger depth to saturate the intensity of the pixels and vice versa. Indeed, only the value of βd is important for the result of the contrast restoration. But it also means that the 3D reconstruction is correctly sealed if and only if a correct value of β is used.
2) Sensitivity to the Method Parameters: The method has
two parameters, ρ and Gσ. To study the sensitivity of the
method to ρ, we used the same methodology than in the previous paragraph. We thus computed τ for different values of ρ on sample images. The resulting plot are given in Fig. 8c. Based on these plots, we propose to set ρ = 0.8, the value where the maximum is obtained. The computation of τ is not relevant to study the sensitivity to the smoothing kernel Gσ, since it does not change a lot the amount of contrast
restoration. We have tried different values of Gσ. We saw that
a too small value create a lot of visual artefacts around the occluded edges. A too big value of Gσ reduces the precision
of the contrast method since the smoothing of the depth map is too important. According to our experiments a value of Gσ
between 10 and 20 pixels is a good trade off for PAL images.
C. Performance Analysis
1) Comparison with other Methods: Based on the state
of the art, few methods can be compared with our method. [24], [13] and [25] are generic enough and only need a single grayscale image to run properly. We have selected some relevant images to compare the methods. We have sent these images to the authors of the different methods, which have processed them before sending us their results. The indicators e, ¯r and s are evaluated for Tan [24], He&al. [13], Tarel&al. [25] and our method on nine images, see Tab. I. Used parameters are ρ = 0.8, A∞ = 255 and Gσ = 20. To
see the behavior of the algorithms in the near range and the far range, we have computed the indicators thanks to the entire
(a) (b) (c) (d) (e)
Fig. 9. Comparison of the different contrast restoration methods: (a) original road images in daytime fog; (b) restored images by our algorithm; (c) He&al. algorithm [13]; (d) Tan algorithm [24]; (e) Tarel&al. algorithm [25].
images, to the bottom third of the images and thanks to the top third of the images.
Results on five images of the database can be seen in Fig. 9. From Tab. I(a), we deduce that Tan, He&al. and our algorithm create the same amount of new visible edges. Tarel&al. algorithm create more visible edges.
From Tab. I(b), we can order the four algorithms in de-creasing order with respect to average increase of contrast on visible edges in the entire images: Tarel&al., Tan, He&al. and our. This confirms the observations on Fig. 9. The results in Tab. I(b) must be balanced. Indeed, if visibility restoration algorithms must increase the contrast, they must not increase the contrast where there is no fog. Tarel&al., Tan, He&al. increase the contrasts in the bottom of the images where there is less fog. Our algorithm almost does not increase the contrast in the near range. Conversely, our algorithm restores the contrast like the three other algorithms in the top of the images, where it is the most important to restore the contrast for intelligent vehicles. In this way, the restored images produced by our algorithm appear more natural.
Tab. I(c) gives the percentage of pixels which become completely black or completely white after the restoration. Compared to others, our and Tarel&al. algorithms do not create black pixels, contrary to He&al. and Tan algorithms.
2) Computation Time: The computation of the fog density
takes less than 10 ms in C++ using a 2.4 GHz Intel Core 2
Duo PC on 1/4 PAL images. On the same hardware platform, the flat world restoration takes less than 5 ms, since it only needs one single loop on the entire image, whereas the contrast restoration takes 40 ms including the 3D reconstruction. Such computation times are obtained using a few look-up-tables. Consequently, the complete image processing framework takes at most 60 ms per image. We are thus able to process 15 images per second. These results must be compared with existing contrast restoration algorithms whose computation times are given in minutes (see introduction), apart from [25].
VI. APPLICATIONS
In this section, we propose some applications of the pro-posed contrast restoration algorithm: improvement of road marking features extraction, improvement of camera-based obstacle detection and improvement of circular road signs detection. Indeed, these applications rely on gradients compu-tation. Moreover, it allows estimating the mobilized visibility distance in a very simple manner.
A. Road Marking Features Extraction
Road marking detection is a fundamental task to develop camera-based driving assistances which aims at avoiding road departure. Road marking feature extraction is the low level processing of the road marking detection. A comparison of
Fig. 10. Example of road marking features extraction enhancement: (a) road marking features extraction on the original image; (b) road marking features extraction on the restored image with the same parameters; (c) ROC curves obtained after lane-marking extraction without and with different visibility restoration algorithms. A ROC curve displays the True Positive Rate (TPR) versus the False Positive Rate (FPR) for different values of the extraction threshold.
the different existing algorithms has been proposed in [27]. According to this paper, the symmetrical local threshold filter gives the best results. We applied this filter to the image given in Fig. 10a and the restored version of this image given in Fig. 10b with the same settings of the filter. We can see an improvement of the detection range of the road markings. Using the same images than in Sect. V-C1, the ROC curves were computed without restoration and with Tan, He&al, Tarel&al and our algorithms. Fig. 10c shows the slight improvement obtained by Tan, He&al and our algorithms on the lane-marking extraction.
B. Road Signs Features Extraction
In the same way, the detection of road signs may be improved. Indeed, most approaches rely on symmetry de-tectors, like the approach proposed in [2] and could benefit from contrast improvement in the images. Based on a similar approach proposed in [6], Fig. 11 shows the detection of circular signs using a foggy image and a restored image. Thanks to the restoration, five signs have been found whereas none of them has been found using the original image. The parameters of the detection method are the same in both cases and set to typical daytime values.
C. Road Obstacle Detection
A review of existing methods for camera-based road obsta-cle detection has been proposed in [23]. Weather conditions reduce the operation range of most methods. Consequently, contrast restoration may of course improve the operation range
of these methods. For example, corner extraction, gradients extraction or texture classification are improved. On the other side, the flat world restoration method presented in section IV-B is able to segment all vertical objects (see Fig. 6), among which the road obstacles. By merging the output of this algorithm with existing obstacle detection techniques, road obstacle detection in daylight fog may be largely improved.
VII. DISCUSSION ANDPERSPECTIVES
A novel efficient algorithm for restoring the contrast of foggy images grabbed by in-vehicle cameras has been pro-posed. Based on this algorithm, we have shown that it is possible to improve the reliability of camera-based driving assistances in daylight fog. One key point of the process is a monocular 3D reconstruction based on fog properties. First, since this problem is ill posed, the proposed heuristic might be wrong despite the use of (22), in particular on bright objects like road signs or vehicle beams which are not connected to the ground. The quantitative evaluation of the proposed 3D reconstruction is difficult since we do have access to the ground truth, apart from using synthetic images which are generally less complex than actual images. Instead of using such a 3D ground truth, we think that an interesting alternative more related to our field of applications is to assess quantitatively the gain procured by contrast restoration algorithms on real applications like lane markings recognition, where it is easier to build ground truth data (e.g. [27]). Second, the method uses only one camera. It has the classical limitations of a monocular method and may be not relevant anymore in mountain areas. However, the proposed approach can be extended to the use of stereovision, like it is suggested in [10]. To go further, a temporal tracking of the contrast restoration parameters may be added, as well as a more reliable estimation of the horizon line. A fusion of Tarel&al. method and of the here proposed method would also be profitable so as to take advantages of both methods. We want also to deal with the restoration of stereo pairs and with other meteorological conditions such as rain. Finally, the extension and the improvement of the objective criterion (27) on the effectiveness of a contrast restoration algorithm are foreseen.
VIII. CONCLUSION
In this paper, a framework for restoring the contrast of images grabbed from a moving vehicle has been presented.
Fig. 11. Example of circular road signs detection improvement [6]: (a) original image; (b) restored image. The symmetry centers are marked with a red cross.
We first estimate the weather conditions and then we infer the scene structure which is refined during the restoration process. Due to our application context, the initial structure is based on a flat world assumption merged with depth heuristics to reconstruct the rest of the 3D environment, from which contrast restoration is performed. Results are assessed by measuring the visibility level enhancement gained by the contrast restoration algorithm and by analyzing the sensitivity of the algorithm to its parameters. A quantitative comparison with current state of the art is carried out. Finally, applications to the improvement of road markings, road signs and obstacle detections in daytime fog are shown. To conclude, perspec-tives are given to pave the line to new contrast restoration approaches.
ACKNOWLEDGMENTS
This work has been partly supported by the ANR (French National Research Agency) within the DIVAS and ICADAC projects.
REFERENCES
[1] International lighting vocabulary. Number 17.4. Commission Interna-tionale de l’Éclairage, 1987.
[2] N. Barnes, A. Zelinsky, and L. Fletcher. Real-time speed sign detection using the radial symmetry detector. IEEE Trans. Intell. Transp. Syst., 9(2):322–332, 2008.
[3] C. Boussard, N. Hautière, and B. d’Andréa Novel. Vehicle dynamics estimation for camera-based visibility range estimation. In Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., 2008.
[4] F. Cozman and E. Krotkov. Depth from scattering. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., Jun 1997.
[5] R. Fattal. Single image dehazing. ACM Trans. Graphics, 27(3), 2008. [6] P. Foucher, P. Charbonnier, and H. Kebbous. Evaluation of a road sign
pre-detection system by image analysis. In Proc. Int. Conf. Comput. Vis. Theory and Application, 2009.
[7] M. D. Grossberg and S. K. Nayar. Modeling the space of camera re-sponse functions. IEEE Trans. Pattern Anal. Mach. Intell., 26(10):1272– 1282, 2004.
[8] N. Hautière and D. Aubert. Contrast restoration of foggy images through use of an onboard camera. In Proc. IEEE Conf. Intell. Transp. Syst., pages 1090–1095, 2005.
[9] N. Hautière, R. Labayrade, and D. Aubert. Real-time disparity contrast combination for onboard estimation of the visibility distance. IEEE Trans. Intell. Transp. Syst., 7(2):201–212, 2006.
[10] N. Hautière, J.-P. Tarel, and D. Aubert. Towards fog-free in-vehicle vision systems through contrast restoration. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2007.
[11] N. Hautière, J.-P. Tarel, D. Aubert, and E. Dumont. Blind contrast enhancement assessment by gradient ratioing at visible edges. Image Analysis and Stereology, 27(2):87–95, 2008.
[12] N. Hautière, J.-P. Tarel, J. Lavenant, and D. Aubert. Automatic fog detection and estimation of visibility distance through use of an onboard camera. Machine Vision and Applications, 17(1):8–20, 2006. [13] K. He, J. Sun, and X. Tang. Single image haze removal using dark
channel prior. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., Jun 2009.
[14] Y. He, W. H., and B. Zhang. Color-based road detection in urban traffic scenes. IEEE Trans. Intell. Transp. Syst., 5(4):309–318, 2004. [15] H. Kong, J.-Y. Audibert, and J. Ponce. Vanishing point detection for
road detection. In Proc. IEEE Conf. Comput. Vis. Pattern Recog., Jun 2009.
[16] J. Lavenant, J.-P. Tarel, and D. Aubert. Procédé de détermination de la distance de visibilité et procédé de détermination de la présence d’un brouillard. French patent 0201822, LCPC / INRETS, 2002.
[17] W. Middleton. Vision through the atmosphere. University of Toronto Press, 1952.
[18] K. Mori, T. Takahashi, I. Ide, H. Murase, T. Miyahara, and Y. Tamatsu. Recognition of foggy conditions by in-vehicle camera and millimeter wave radar. In Proc. IEEE Intell. Veh. Symp., 2007.
[19] S. G. Narasimhan and S. K. Nayar. Vision and the atmosphere. Int. J. Comput. Vis., 48(3):233–254, 2002.
[20] S. G. Narasimhan and S. K. Nayar. Contrast restoration of weather degraded images. IEEE Trans. Pattern Anal. Mach. Intell., 25(6):713– 724, 2003.
[21] D. Pomerleau. Visibility estimation from a moving vehicle using the RALPH vision system. In Proc. IEEE Conf. Intell. Transp. Syst., pages 906–911, 1997.
[22] Y. Schechner, S. Narasimhan, and S. Nayar. Polarization-based vision through haze. Applied Optics, Special issue, 42(3):511–525, Jan 2003. [23] Z. Sun, G. Bebis, and R. Miller. On-road vehicle detection: A review.
IEEE Trans. Pattern Anal. Mach. Intell., 28(5):694–711, 2006. [24] R. T. Tan. Visibility in bad weather from a single image. In Proc. IEEE
Conf. Comput. Vis. Pattern Recog., 2008.
[25] J.-P. Tarel and N. Hautière. Fast visibility restoration from a single color or gray level image. In Proc. IEEE Int. Conf. Comput. Vis., 2009. [26] W. van der Mark and D. Gavrila. Real-time dense stereo for intelligent
vehicles. IEEE Trans. Intell. Transp. Syst., 7(1):38–50, 2006. [27] T. Veit, J.-P. Tarel, P. Nicolle, and P. Charbonnier. Evaluation of road
marking feature extraction. In Proc. IEEE Conf. Intell. Transp. Syst., pages 174–181, 2008.
Nicolas Hautièrereceived the M.S. degree in civil
engineering from the National School of State Public Works (ENTPE), Lyon, France in 2002 and the M.S. and Ph.D. degrees in computer vision from the University Jean Monnet, Saint-Etienne, France, in 2002 and 2005, respectively. From 2002 to 2005, he was a Ph.D. student at the Interactions Vehicle-Infrastructure-Driver Research Unit (LIVIC), which is a mixed research unit between the French National Institute for Transportation and Safety Research (IN-RETS) and the French Public Works Research Lab-oratory (LCPC). Since 2009, he has been a research leader at the LabLab-oratory for road Operation, Perception, Simulations and Simulators (LEPSiS), which is recent research unit between INRETS and LCPC. His research interests cover the modelling of the meteorological phenomena reducing the highway visibility, the detection of visibility conditions and the estimation of the visibility range.
Jean-Philippe Tarel graduated from the Ecole
Nationale des Ponts et Chaussées (ENPC), Paris, France (1991). He received his PhD degree in Ap-plied Mathematics from Paris IX-Dauphine Univer-sity in 1996. He was with the Institut National de Recherche en Informatique et Automatique (INRIA) from 1991 to 1996 and from 2001 to 2003. From 1997 to 1998, he worked as a research associate at Brown University, USA. From 1999, he is a researcher in the French Public Works Research Lab-oratory (LCPC), Paris, France. His research interests include 3D reconstruction, pattern recognition and detection.
Didier Aubertreceived the M.S. and Ph.D. degree
respectively in 1985 and 1989 from the National Polytechnical Institut of Grenoble (INPG). From 1989-1990, he worked as a research scientist on the development of an automatic road following system for the NAVLAB at Carnegie Mellon Uni-versity. From 1990-1994, he worked in the research department of a private company (ITMI). During this period he was project leader of several projects dealing with computer vision. In 1995 he joined IN-RETS (French National Institute for Transportation and Safety Research). He is currently a senior researcher, deputy director of LIVIC and he is the manager of the perception team. He works on road traffic monitoring, on crowd monitoring, on automated highway systems and on driving assistance systems. He is an image processing expert for several compagnies and teaches in several Universities (Paris VI, Paris XI, Paris XII, Evry, Versailles) and Engineer schools (ENPC, ENST, ENSMP).
