HAL Id: hal-02319750
https://hal.archives-ouvertes.fr/hal-02319750
Submitted on 18 Oct 2019
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of
sci-entific research documents, whether they are
pub-lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
LiDAR-driven spatial regularization for hyperspectral
unmixing
Tatsumi Uezato, Mathieu Fauvel, Nicolas Dobigeon
To cite this version:
Tatsumi Uezato, Mathieu Fauvel, Nicolas Dobigeon. LiDAR-driven spatial regularization for
hyper-spectral unmixing. IEEE International Geoscience and Remote Sensing Symposium (IGARSS 2018),
Jul 2018, Valencia, Spain. pp.1740-1743. �hal-02319750�
Any correspondence concerning this service should be sent
to the repository administrator:
tech-oatao@listes-diff.inp-toulouse.fr
This is an author’s version published in:
http://oatao.univ-toulouse.fr/22390
Official URL
DOI :
https://doi.org/10.1109/IGARSS.2018.8518741
Open Archive Toulouse Archive Ouverte
OATAO is an open access repository that collects the work of Toulouse
researchers and makes it freely available over the web where possible
To cite this version:
Uezato, Tatsumi and Fauvel, Mathieu and
Dobigeon, Nicolas LiDAR-driven spatial regularization for
hyperspectral unmixing. (2018) In: IEEE International Geoscience
and Remote Sensing Symposium (IGARSS 2018), 23 July 2018 - 27
July 2018 (Valencia, Spain).
LIDAR-DRIVEN
SPATIAL REGULARIZATION FOR HYPERSPECTRAL UNMIXING
Tatsumi Uezato
(1), Mathieu Fauvel
(2)and Nicolas Dobigeon
(1)(1)
University of Toulouse, IRIT/INP-ENSEEIHT, 31071 Toulouse, France
(2)
University of Toulouse, Dynafor/INP-ENSAT, 31326 Castanet-Tolosan, France
firstname.lastname@enseeiht.fr, mathieu.fauvel@ensat.fr
ABSTRACT
Only a few research works consider LiDAR data while con-ducting hyperspectral image unmixing. However, the digital surface model derived from LiDAR can provide meaningful information, in particular when spatially regularizing the in-verse problem underlain by spectral unmixing. This paper proposes a general framework for spectral unmixing that in-corporates LiDAR data to inform the spatial regularization applied to the abundance maps. The proposed framework is validated and compared to existing unmixing methods that incorporate spatial information derived from the hyperspec-tral image itself using two different simulated data and digital surface models. Results show that the spatial regularization incorporating LiDAR data significantly improves abundance estimates.
Index Terms— Hyperspectral imagery, spectral unmix-ing, edge, lidar, classification
1. INTRODUCTION
Due to a relatively low spatial resolution of the sensors, hy-perspectral images are generally composed of mixed pixels, i.e., composed by spectral mixtures of several elementary ma-terials in unknown proportions. Spectral unmixing aims at deconvolving the spectral mixtures into a collection of refer-ence spectra, known as endmembers, and their corresponding proportions or abundances [1]. Most of conventional spec-tral unmixing methods ignore the intrinsic 2D+λ structure of the hyperspectral datacube (where λ refers to the wavelength) but exploit only the spectral information. However, recent advances in spectral unmixing enable spatial information to be incorporated into the unmixing process [2, 3]. The key idea consists in promoting identical or similar abundance es-timates in a given neighborhood of the pixels of interest. This strategy relies on the assumption that neighboring pixels gen-erally show smooth transitions in abundances. These meth-ods incorporate this spatial information as a regularization to
Part of this work has been funded by EU FP7 through the ERANETMED JC-WATER program, MapInvPlnt Project ANR-15-NMED-0002-02 and by the MUESLI IDEX ATS project, Toulouse INP.
describe the spatial variations of abundances in a local neigh-borhood [4, 5].
One of the well-known limitations of spatial regulariza-tions consists in not properly preserving the edges between homogeneous areas, even when using total variation (TV)-like penalizations. Instead, they tempt to overly smooth edges, finally leading to poor abundance estimates in these specific areas. One solution is to build a so-called guidance mapwhich localizes these edges. The spatial regularization for these pixels can be subsequently adjusted accordingly, by resorting to a weighted spatial regularization [6]. This side information is commonly estimated from the hyperspectral image to be unmixed directly. However, this estimation may be greatly affected by noise or illumination variations [7], which may lead to incorrect weighting of the spatial reg-ularization. One alternative consists in exploiting external data, if available, to derive this guidance map. Such strategy is expected to be robust to the aforementioned problem. In particular, LiDAR data have a great potential to extract com-plementary spatial information that can be used to derive a reliable guidance map. Indeed, LiDAR data can discriminate different materials and/or areas using height information, even if they are spectrally similar [8]. Due to its intrinsic nature, it is robust to any illumination variations during the acquisition. While LiDAR data have been successfully used for hyperspectral image classification, only a few studies proposed to exploit LiDAR data for spectral unmixing. In [9], the authors investigated whether weighting the spatial regularization thanks to the LiDAR data can improve the accuracy when estimating abundances, especially in shaded pixels. However, it is still unclear whether LiDAR data or a combination of LiDAR data and another guidance map can lead to more accurate estimates of abundances. This paper proposes to fill this gap.
More precisely, the contributions of this paper are twofold: 1) to develop a new spectral unmixing framework that incor-porates LiDAR data into the weighting of the spatial regu-larization; and 2) to conduct a comprehensive comparison of the weighting functions derived from LiDAR data or from its combination with another guidance map. The proposed framework has been validated using two simulated data to
as-sess whether LiDAR data can lead to significant improvement of abundance estimates.
2. LIDAR DATA-DRIVEN UNMIXING 2.1. Unmixing with spatial regularizations
The linear mixture model (LMM) has been widely used to de-compose a mixed spectrum into a collection of endmembers and their abundances. LMM represents a mixed spectrum as a linear combination of the endmember spectra
yi= Sai+ ni (1)
where yi ∈ RL×1 is a mixed spectrum of the ith pixel,
S ∈ RL×M is the matrix of endmember signatures, a i =
[a1i, . . . , aM i] T
∈ RM ×1 represents abundance fractions at
the ith pixel, ni ∈ RL×1represents noise or modeling error,
L is the number of spectral bands and M is the number of endmembers. Abundance non-negativity constraint (ANC) and the abundance sum-to-one constraint (ASC) are usually imposed as follows ∀m, i, ami≥ 0 and ∀i, M X m=1 ami= 1. (2)
When endmembers S have been identified thanks to a pri-ori knowledge or extracted from the hyperspectral image using a dedicated endmember extraction algorithm [1], the abundance vectors ai(i= 1, . . . , N ) can be estimated
pixel-by-pixel by solving the N following optimization problems, where N is the number of pixels,
min ai 1 2kyi− Saik 2 2 s.t. (2). (3)
In the optimization problem, abundances are estimated for each pixel independently, ignoring the spatial information in-herent to the abundance maps to be recovered. One conven-tional way to regularize the associated inverse problem con-sists in incorporating a spatial regularization into the mini-mization problem, leading to
min ai 1 2kyi− Saik 2 2+ λ X j∈N(i) wijkai− ajkpps.t. (2) (4)
where λ is a parameter to control the balance between the data fitting term and the spatial regularization,N (i) is the set of the neighboring pixels1of the ith pixel, w
ij is a weight
de-scribing the expected similarity between the ith and jth pix-els. Popular ℓp-norms considered in spatial regularizations
include p= 2 and p = 1, which promotes smooth variations and piecewise constant behaviors of the abundance maps, re-spectively. In this work, without loss of generality of the main contribution presented in this paper, the case p = 1 will be considered. Moreover, when the weights are tuned to wij= 1 (∀i, j) with p = 1, the resulting spatial
regulariza-tion is known as a specific instance of the total variaregulariza-tion (TV)
1In this study, a 4-order neighborhood will be considered.
penalization. In this case, each neighboring pixel equally con-tributes to the spatial regularization term. However, this may be inappropriate, in particular for pixels located in edges be-tween several distinct areas characterized by different materi-als and/or composition. Thus, choosing appropriate weights describing the spatial relationships between neighboring pix-els can greatly improve abundance estimates [6]. The wij
can be adjusted according to a guidance image which sum-marizes this spatial information, such as the edge locations. This guidance image can be derived directly from the hyper-spectral image to be unmixed. However, this choice can be significantly affected by variations in illumination or sensor noise. Moreover, when distinct areas are composed of spec-trally similar materials, such a guidance map will hardly be able to encode the presence of edges. Conversely, digital sur-face model (DSM) computed from LiDAR data represents a great opportunity to overcome these limitations. If the height derived from DSM is different for each region belonging to a particular mixture of endmembers, DSM can correctly ex-tract the edge information even under different illumination conditions. This property is particularly useful in urban or vegetated areas where the height of materials plays an impor-tant role [8, 10]. In what follows, various guidance maps are presented, based on the hyperspectral image, DSM or a com-bination of both. Then, the spatially regularized optimization problem is solved using the alternating direction method of multipliers, following the strategy in [4]. More information regarding the optimization procedure is given in [11]. 2.2. Different types of weights
A variety of guidance images can be used to adjust the weights in the spatial regularization. In this section, five different approaches are described. Each weight is calculated by using a normalized squared difference of features between a target pixel and the neighborhood pixels.
w-HI: Weights are chosen from the spectral similarity be-tween neighboring pixels
wij= 1 Qi exp − 1 σ2 y kyi− yjk22 kyi+ yjk22 (5) where yiis the spectrum of the ith pixel and σ2yis a parameter
controlling the weight range and Qirepresents the
normaliza-tion constant such thatP
j∈N(i)wij= 1.
w-PC1: Weights are adjusted from the similarity between pixels of the first principal component (PC) recovered by PC analysis wij= 1 Qi exp − 1 σ2 p (pi− pj)2 (pi+ pj)2 . (6)
where piis the value of the ith pixel of the first PC1 and σp2is
a parameter controlling the weight range.
w-DSM: When DSM derived from LiDAR data is available, the guidance map can be computed from the similarity be-tween the heights of neighboring pixels
wij = 1 Qi exp − 1 σ2 h (hi− hj)2 (hi+ hj)2 . (7) where hiis the height of the ith pixel and σ2his a parameter
controlling the weight range. Moreover, this DSM informa-tion can be combined with the previous guidance maps, as detailed in what follows.
w-HI-DSM: Coupling DSM with previous guidance maps computed from the hyperspectral image leads to
wij = 1 Qi exp − 1 σ2 y kyi− yjk22 kyi+ yjk22 + exp − 1 σ2 h (hi− hj)2 (hi+ hj)2 .
w-PC1-DSM: Similarly, the weights can be adjusted as wij = 1 Qi exp − 1 σ2 p (pi− pj)2 (pi+ pj)2 + exp − 1 σ2 h (hi− hj)2 (hi+ hj)2 . 3. EXPERIMENTS
3.1. Experiment using simulated data 1 (SIM1)
Generation of SIM1: SIM1 has been generated to assess whether the weighting function derived from DSM can im-prove the abundance estimates in the specific situations when each spatially coherent region is characterized by a different height. The simulated hyperspectral image and DSM data have been generated as follows. Five endmember spectra have been randomly selected from the USGS spectral library. A synthetic100 × 100 discrete-value image has been randomly generated as in [12] to identify spatially coherent areas. Each area is assigned a different height to build synthetic DSM (Fig. 1a). Within each area, statistically consistent synthetic mixtures of the M = 5 endmembers are randomly generated, and then corrupted by an additive Gaussian noise leading to a signal-to-noise ratioSNR = 20dB. A color composition of the resulting hyperspectral image is depicted in Fig. 1b. Validation of methods: Quantitative validation has been conducted using the root mean square error (RMSE) of the abundance estimates RMSE = v u u t 1 N M N X i=1 M X m=1 (ami− ˆami)2 (8)
where ami andˆami are the actual and estimated abundance
fractions. This quantitative analysis can be also conducted by restricting the computation of RMSE for the pixels specifi-cally localized in the edge areas of the hyperspectral image. Interested readers are invited to consult [11] for details. Results: For the weighting functions proposed in Section 2.2, RMSE as functions of the regularization parameter λ are depicted in Fig. 2. As expected, RMSE estimated from
(a) (b)
Fig. 1: SIM1: (a) Synthetic DSM. (b) Color composition of the synthetic hyperspectral data.
no-weight performs poorly compared to other methods incor-porating DSM information. In particular, RMSE of the no-weight approach significantly decrease for large values of λ. This shows that this method is very sensitive to the value of the regularization parameter λ. Conversely, the methods that incorporate DSM information perform favorably. The meth-ods estimate accurate abundances even when a large value (>1) of λ is chosen. Among all these methods, those rely-ing on DSM perform better than their DSM-free counterparts. This shows that the use of DSM can lead to more accurate estimates of abundances especially where the height allows spatially discrete regions to be distinguished.
0.001 0.05 0.1 0.5 1 1.5
(Parameter controlling the spatial regularization) 0.005 0.01 0.015 0.02 0.025 RMSE w-HI w-PC1 w-DSM w-HI-DSM w-PC1-DSM 0.001 0.05 0.1 0.5 1 1.5 0 0.05 0.1 0.15 No-weight
Fig. 2: SIM1: Abundance RMSE as a function of λ.
3.2. Experiment using simulated data 2 (SIM2)
Generation of SIM2: This second experiment has been de-signed to assess whether DSM can still be useful for estimat-ing abundances when DSM describes only partially the spa-tial information. To do so, a synthetic yet more realistic hy-perspectral image has been generated and coupled with real DSM. More precisely, a real hyperspectral image and its
cor-(a) (b)
Fig. 3: SIM2: (a) Real DSM. (b) Color composition of the synthetic hyperspectral data.
responding DSM (depicted in Fig. 3a) have been acquired during a flight campaign conducted in June 2016, over the city of Saint-Andr´e, France. First, M = 4 endmembers have been extracted from this real image using the n-Dimensional Visualizer provided by the ENVI software. To build ground-truth abundance maps, this real hyperspectral image has been unmixed following the LMM. Finally, based on these end-member spectra and abundance maps, the synthetic hyper-spectral image referred to as SIM2 is generated according to LMM and corrupted by an additive Gaussian noise with SNR= 20dB (see Fig. 3b).
10-3 5*10-3 0.01 0.05 0.1
(Parameter controlling the spatial regularization) 0.025 0.03 0.035 0.04 0.045 0.05 0.055 RMSE w-HI w-PC1 w-DSM w-HI-DSM w-PC1-DSM 10-3 5*10-3 0.01 0.05 0.1 0.02 0.04 0.06 0.08 0.1 No-weight
Fig. 4: SIM2: Abundance RMSE as a function of λ. Results: RMSE is depicted as a function of the regularization parameter in Fig. 4. The method with no weight performs poorly compared with other methods that incorporate edge information. RMSE obtained by the w-DSM unmixing model is smaller than those derived from w-PC1 or w-HI when an optimal value of λ is used. The combinations of DSM and other guidance images (w-PC1-DSM and w-HI-DSM) also show smaller RMSE than those derived from the DSM-free methods (w-HI or the w-PC1) for a wide range of λ. However, RMSE derived from the DSM-informed methods are larger
for a large value of the regularization parameter, i.e., when the unmixing problem is more spatially regularized. This is probably due to the fact that DSM does not capture all the spatial information.
4. CONCLUSION
In this study, a new spectral unmixing framework that in-corporates DSM information derived from LiDAR data was proposed. The proposed method was compared to the meth-ods incorporating spatial information derived from the hyper-spectral image directly. Results showed that the use of Li-DAR data with other guidance maps can lead to significant improvement of abundance estimation and robustness.
5. REFERENCES
[1] J. M. Bioucas-Dias, A. Plaza, N. Dobigeon, M. Parente, D. Qian, P. Gader, and J. Chanussot, “Hyperspectral unmixing overview: Ge-ometrical, statistical, and sparse regression-based approaches,” IEEE J. Sel. Topics Appl. Earth Observations Remote Sens., vol. 5, no. 2, pp. 354–379, 2012.
[2] O. Eches, N. Dobigeon, and J. Y. Tourneret, “Enhancing hyperspectral image unmixing with spatial correlations,” IEEE Trans. Geosci. Remote Sens., vol. 49, no. 11, pp. 4239–4247, 2011.
[3] C. Shi and L. Wang, “Incorporating spatial information in spectral un-mixing: A review,” Remote Sens. Environment, vol. 149, no. 0, pp. 70–87, 2014.
[4] M. D. Iordache, J. M. Bioucas-Dias, and A. Plaza, “Total variation spatial regularization for sparse hyperspectral unmixing,” IEEE Trans. Geosci. Remote Sens., vol. 50, no. 11, pp. 4484–4502, 2012. [5] T. Uezato, R. J. Murphy, A. Melkumyan, and A. Chlingaryan,
“Incor-porating spatial information and endmember variability into unmixing analyses to improve abundance estimates,” IEEE Trans. Image Pro-cess., vol. 25, 2016.
[6] J. Liu, J. Zhang, Y. Gao, C. Zhang, and Z. Li, “Enhancing spectral unmixing by local neighborhood weights,” IEEE J. Sel. Topics Appl. Earth Observations Remote Sens., vol. 5, no. 5, pp. 1545–1552, 2012. [7] T. Uezato, R. J. Murphy, A. Melkumyan, and A. Chlingaryan, “A novel
spectral unmixing method incorporating spectral variability within end-member classes,” IEEE Trans. Geosci. Remote Sens., vol. 54, no. 5, pp. 2812–2831, 2016.
[8] L. Ni, L. Gao, S. Li, J. Li, and B. Zhang, “Edge-constrained markov random field classification by integrating hyperspectral image with li-dar data over urban areas,” Journal of Applied Remote Sensing, vol. 8, no. 1, pp. 085 089–085 089, 2014.
[9] A. Castrodad, T. Khuon, R. Rand, and G. Sapiro, “Sparse modeling for hyperspectral imagery with lidar data fusion for subpixel mapping,” in Proc. IEEE Int. Conf. Geosci. Remote Sens. (IGARSS), 2012, pp. 7275– 7278.
[10] M. Dalponte, L. Bruzzone, and D. Gianelle, “Fusion of hyperspectral and lidar remote sensing data for classification of complex forest areas,” IEEE Trans. Geosci. Remote Sens., vol. 46, no. 5, pp. 1416–1427, 2008. [11] T. Uezato, M. Fauvel, S. May, and N. Dobigeon, “Hyperspectral image unmixing with LiDAR data-aided spatial regularization,” submitted. [Online]. Available: http://arxiv.org/abs/1712.07862/
[12] T. Uezato, R. J. Murphy, A. Melkumyan, and A. Chlingaryan, “A novel endmember bundle extraction and clustering approach for capturing spectral variability within endmember classes,” IEEE Trans. Geosci. Remote Sens., vol. 54, no. 11, 2016.
