HAL Id: hal-02112180
https://hal.archives-ouvertes.fr/hal-02112180
Submitted on 26 Apr 2019
HAL is a multi-disciplinary open access
archive for the deposit and dissemination of
sci-entific research documents, whether they are
pub-lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d’enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
extension to the non-negative setting
Thi Thanh Nguyen, Charles Soussen, Jérôme Idier, El-Hadi Djermoune
To cite this version:
Thi Thanh Nguyen, Charles Soussen, Jérôme Idier, El-Hadi Djermoune. NP-hardness of ℓ0
minimiza-tion problems: revision and extension to the non-negative setting. 13th Internaminimiza-tional Conference on
Sampling Theory and Applications, SampTA 2019, Jul 2019, Bordeaux, France. �hal-02112180�
NP-hardness of
`
0
minimization problems:
revision and extension to the non-negative setting
Thanh T. N
GUYEN1, Charles S
OUSSEN2, J´erˆome I
DIER3, El-Hadi D
JERMOUNE11
CRAN, Universit´e de Lorraine, CNRS, F-54506 Vandœuvre-l`es-Nancy, France
2L2S, CentraleSup´elec-CNRS-Universit´e Paris-Sud, Universit´e Paris-Saclay,
F-91192 Gif-sur-Yvette, France
3
LS2N, UMR 6004, F-44321 Nantes, France
{thi-thanh.nguyen,el-hadi.djermoune}@univ-lorraine.fr;
charles.soussen@centralesupelec.fr; jerome.idier@ls2n.fr
Abstract—Sparse approximation arises in many appli-cations and often leads to a constrained or penalized `0
minimization problem, which was proved to be NP-hard. This paper proposes a revision of existing analyses of NP-hardness of the penalized `0 problem and it introduces a
new proof adapted from Natarajan’s construction (1995). Moreover, we prove that `0minimization problems with
non-negativity constraints are also NP-hard.
I. INTRODUCTION
Sparse approximation appears in a wide range of appli-cations, especially in signal processing, image processing and compressed sensing [1]. Given a signal data y ∈ Rm and a dictionary A of size m×n, the aim is to find a signal x ∈ Rnthat gives the best approximation y ≈ Ax and has the fewest non-zero coefficients (i.e., sparsest solution). This task leads to solving one of the following constrained or penalized `0 minimization problems:
min ky−Axk2≤ kxk0 (`0C) min kxk0≤K ky − Axk2 2 (`0C0) min x ky − Axk 2 2+ λkxk0 (`0P )
in which , K and λ are positive quantities related to the noise standard deviation, the sparsity level and regulariza-tion strength, respectively. Letters C and P respectively indicate that the problem is constrained or penalized. Depending on application, the appropriate statement will be addressed. It is noteworthy that n and K often depend on m when one considers the size of problem. (`0C) and
(`0C0) are well known to be hard [2, 3]. The
NP-hardness of (`0P ) was claimed to be a particular case of
more general complexity analyses in [4, 5]. However, we point out that these complexity analyses do not rigorously apply to (`0P ) as claimed. In this paper, we justify the
complexity analyses in [4, 5] do not apply to problem (`0P ), and we provide a new proof for the NP-hardness
of (`0P ) adapted from Natarajan’s construction [2].
In several applications such as geoscience and remote sensing [6, 7], audio [8], chemometrics [9] and computed
This work was supported by the project ANR BECOSE (ANR-15-CE23-0021).
This work was carried out in part while T. T. Nguyen was visiting L2S during the academic years 2017-2019.
tomography [10], the signal or image of interest is non-negative. In such contexts, one often addresses a mini-mization problem with both sparsity and non-negativity constraints [10–12]. Adding non-negativity constraints to `0 minimization problems yields the following problems:
min x≥0,ky−Axk2≤ kxk0 (`0C+) min x≥0,kxk0≤K ky − Axk2 2 (`0C0+) min x≥0 ky − Axk 2 2+ λkxk0 (`0P +)
Several papers address non-negative `0minimization
prob-lems in the literature (see, e.g., [13–16]). However, to the best of our knowledge, the complexity of these problems has not been addressed yet, the question of their NP-hardness being still open. Here we show that these prob-lems are NP-hard and the proof can be derived from the NP-hardness of `0 minimization problems.
The rest of paper is organized as follows. In Section II, we discuss the issues related to NP-hardness of (`0P ) in
existing analyses and we present our proof. In Section III, we discuss about the NP-hardness of non-negative `0
minimization problems. We draw some conclusions in Section IV.
II. HARDNESS OF`0MINIMIZATION PROBLEMS
A. Background on constrained `0 minimization problems
Let us recall that an NP-complete problem is a prob-lem in NP to which any other probprob-lem in NP can be reduced in polynomial time. Thus NP-complete problems are identified as the hardest problems in NP. An NP-complete problem is strongly NP-NP-complete if it remains NP-complete when all of its numerical parameters are bounded by a polynomial in the length of the input. NP-hard problems are at least as hard as NP-complete problems. However, NP-hard problems do not need to be in NP and do not need to be decision problems. Formally, a problem is NP-hard (respectively, strongly hard) if a complete (respectively, strongly NP-complete) problem can be reduced in polynomial time to it. The reader is referred to [17, 18] for more information on this topic.
In the literature, problem (`0C), called SAS in [2],
is well known to be hard [2, Theorem 1]. The NP-hardness of (`0C) is a valuable extension of an earlier
result: the problem of minimum weight solution to linear equations (equivalent to (`0C) with = 0) is NP-hard [17,
p. 246]. Davis et al. proved that (`0C0), called M -optimal
approximation in [3], is NP-hard for any K < m [3, Theorem 2.1]. Both analyses of Natarajan and Davis were made by a polynomial time reduction from the “exact cover by 3-sets” problem1 which is known to be
NP-complete [17, p. 221].
B. Existing analyses on penalized`0 minimization
In [4, 5], the NP-hardness of (`0P ) is deduced as
a particular case of more general complexity analyses. However, it turns out that the latter do not apply to (`0P ), as explained hereafter. Chen et al. [4] address the
unconstrained `q-`p minimization problem, defined by:
min x ky − Axk q q+ λkxk p p (`q-`p)
where λ > 0, q ≥ 1 and 0 ≤ p < 1. The authors showed that problem (`q-`p) is NP-hard with any λ > 0, q ≥ 1
and 0 ≤ p < 1 [4, Theorem 3]. Obviously, (`0P ) is the
case where q = 2 and p = 0. The proof was done by i) introducing an invertible transformation which scales any instance of problem (`q-`p) to the problem (`q-`p)
with λ = 1/2, and ii) establishing a polynomial time reduction from the partition problem which is known to be NP-complete [17] to the problem (`q-`p) with λ = 1/2.
In other words, they showed that problem (`q-`p) with
λ = 1/2 is NP-hard and, because there exists an invertible transformation from any problem (`q-`p) to the one with
λ = 1/2, every problem (`q-`p) is NP-hard. Similarly, they
showed that (`q-`p) is strongly NP-hard [4, Theorem 5] by
a reduction from the 3-partition problem which is known to be strongly NP-hard [17]. The invertible transform used in [4] is defined by:
˜
x = (2λ)1/px, A = (2λ)˜ −1/pA. (1) Unfortunately, (1) is not well-defined when p = 0. There-fore, [4, Theorems 3 and 5] do not apply to (`0P ) when
λ 6= 1/2.
Using a different approach, Huo and Chen’s paper [5] addresses the penalized least-squares problem defined by:
min x ky − Axk 2 2+ λ n X i=1 φ(|xi|), (PLS)
where φ is a penalty function mapping non-negative values to non-negative values. The authors showed that (PLS) is NP-hard if the penalty function φ satisfies the following four conditions [5, Theorem 3.1]:
C1. φ(0) = 0 and ∀ 0 ≤ τ1< τ2, φ(τ1) ≤ φ(τ2).
C2. There exists τ0> 0 and a constant d > 0 such that
φ(τ ) ≥ φ(τ0) − d(τ0− τ )2
1The latter problem, denoted by X3C in [2, 17], is stated as follows:
Given a set S and a collection C of 3-element subsets of S (called triplets), is there a subcollection of disjoint triplets that exactly covers S?
for every 0 ≤ τ < τ0.
C3. For the aforementioned τ0, if τ1, τ2< τ0 then
φ(τ1) + φ(τ2) ≥ φ(τ1+ τ2).
C4. For every 0 ≤ τ < τ0,
φ(τ ) + φ(τ0− τ ) > φ(τ0). (2)
The proof of [5, Theorem 3.1] is by a reduction from the NP-complete problem X3C to the decision version of (PLS); this leads to the NP-completeness of the decision version of (PLS) and so the NP-hardness of (PLS) [5, Appendix 1]. The authors claimed that the `0 penalty
function satisfies conditions C1-C4 for τ0 = d = 1.
Therefore, the (PLS) problem with the `0penalty function
is NP-hard [5, Corollary 3.2]. Unfortunately, it turns out that the `0penalty does not fulfill condition C4 as claimed.
Indeed, for τ = 0 the strict inequality (2) becomes φ(0) > 0. Besides, in the proof [5, Appendix 1], the inputs of the decision problem are not guaranteed to have rational values. This might also violate the polynomiality of the reduction. Therefore, [5, Theorem 3.1] does not apply to (`0P ).
In [5], the authors also mention an alternate proof of NP-hardness of (`0P ) from Huo and Ni’s earlier paper
[19] as a special case of their results. In this proof [19, Appendix A.1], the relation between (`0P ) and (`0C) is
es-tablished using the principle of Lagrange multiplier. More precisely, the authors introduce an instance of (`0C) in
which is defined from the minimizer of (`0P ) and argue
that solving (`0P ) is equivalent to solving the mentioned
instance of (`0C), which is known to be NP-hard [2].
There are a number of issues in the NP-hardness proof in [19]. For instance, the proposed transformation between (`0P ) and (`0C) is not a polynomial time reduction.
Besides, it is well known that (`0P ) and (`0C) are not
equivalent [20].
C. New analysis on penalized `0 minimization problems
To prove that a problem T is NP-hard, one must estab-lish a polynomial time reduction (briefly called reduction hereafter) from some known NP-hard or NP-complete problem to T [18]. Roughly speaking, the reduction from a problem T1 to another problem T2 implies that T1 is not harder than T2. Therefore, if there exists a reduction from T1 to T2 and if T1 is NP-hard, T2 must be NP-hard too. The NP-hardness proofs in [2] and [3] use this principle. As an adaptation of Natarajan’s construction, we prove the NP-hardness of (`0P ) using the same principle as follows.
Theorem II.1. Problem (`0P ) is NP-hard for 0 < λ < 3.
The proof is by a reduction from the known NP-complete problem X3C to (`0P ). The proof contains three
steps: (1) Construct an instance of (`0P ) from a given
instance of X3C; (2) Construct a solution of (`0P ) from
a solution of X3C; (3) Construct a solution of X3C from a solution of (`0P ).
1) Construction of an instance of (`0P ) from a given
instance of X3C: Given an instance of X3C: S = {s1, s2, ..., sm} is a set of m elements. C is a collection
can assume that m is a multiple of 3 since otherwise there is trivially no exact cover so no solution of X3C.
We now construct an instance of (`0P ). Let y =
[1, 1, ..., 1]T ∈ Rm. Let A = (a
ij)1≤i≤m,1≤j≤n where
aij = 1 if si ∈ cj and aij = 0 otherwise. Let λ ∈ Q,
0 < λ < 3. Let
F (x) := ky − Axk22+ λkxk0. (3)
2) Construction of a solution of (`0P ) from a
solu-tion of X3C: Assume that there is a subcollection of disjoint triplets ˆC which exactly covers S. Let x∗ = [x∗1, x∗2, ..., x∗n]T where x∗j = 1 if cj ∈ ˆC and x∗j = 0
otherwise. We will prove that x∗ is a solution of (`0P ).
Since ˆC exactly covers S, | ˆC| = m/3 and y = Ax∗. Thus, kx∗k0= m/3 and
F (x∗) = 0 + λm
3 = λ
m 3. Suppose that there exists ¯x such that
F (¯x) < F (x∗) = λm
3. (4)
Let us show that this leads to a contradiction.
Since F (¯x) ≥ λk¯xk0, from (4) we have k¯xk0< m/3.
Therefore, we can rewrite k¯xk0 = m/3 − q for some
q ∈ N, 1 ≤ q < m/3. Note that A¯x has m entries. Since the number of non-zero entries of A¯x identifies with the number of elements si recovered by the subcollection
corresponding to ¯x, this number cannot exceed 3k¯xk0=
m − 3q. As a result, the number of zero entries of A¯x must be between 3q and m. Since y is the all-one vector, y − A¯x has at least 3q entries valued 1, which implies
ky − A¯xk22≥ 3q. (5) Hence, F (¯x) ≥ 3q + λm 3 − q = λm 3 + (3 − λ)q > λ m 3, (6) which contradicts (4). Therefore, x∗is a solution of (`0P ).
3) Construction of a solution of X3C from a solution of (`0P ): Assume that x∗is a solution of (`0P ). We will
consider four cases as follows. a) Casekx∗k
0> m/3: We deduce that X3C has no
solution. Indeed, assume that ˆC is an exact cover for S. Define x = [x1, x2, ..., xn]T where xj = 1 if cj∈ ˆC and
xi= 0 otherwise. Then we have
F (x) = λm 3 < λkx
∗k
0≤ F (x∗)
which contradicts the fact that x∗ is a solution of (`0P ).
b) Case kx∗k
0 < m/3: We deduce that X3C has
no solution. Indeed, assume that ˆC is an exact cover for S. Let x = [x1, x2, ..., xn]T where xj = 1 if cj ∈ ˆC
and xi = 0 otherwise. Then we have F (x) = λ
m 3. Since kx∗k
0< m/3, we can write kx∗k0= m/3 − q for some
q ∈ N and 1 ≤ q < m/3. Similar to (6), we have F (x∗) > λm
3. Since F (x) = λ m
3, we obtain F (x
∗) > F (x) which
contradicts the fact that x∗ is a solution of (`0P ).
c) Case where kx∗k
0 = m/3 and y 6= Ax∗: We
deduce that X3C has no solution. Indeed, assume that ˆC is an exact cover for S. Define x = [x1, x2, ..., xn]T where
xj= 1 if cj∈ ˆC and xi= 0 otherwise. Then we have
F (x) = λm
3 < ky − Ax
∗k2
2+ λkx∗k0= F (x∗)
which contradicts the fact that x∗ is a solution of (`0P ).
d) Case where kx∗k
0= m/3 and y = Ax∗: Let ˆC
be the collection of triplets cj such that the jth entry of
x∗ is non-zero. Obviously, ˆC is an exact cover for S so a solution of X3C.
Thus Theorem II.1 is proved.
It is notable that the proof above is also valid when F (x) := ky − Axkp
p+ λkxk0 for any p ≥ 1. Indeed,
one only needs to check whether (5) still holds when the `2 norm is replaced by the `p norm with p ≥ 1. This
is the case since y − A¯x has at least 3q entries equal to 1. Therefore, we have the following generalization of Theorem II.1.
Theorem II.2. Problem minxky −Axkpp+ λkxk0is
NP-hard for p ≥ 1 and 0 < λ < 3.
III. HARDNESS OF NON-NEGATIVE`0MINIMIZATION PROBLEMS
The NP-hardness of non-negative `0minimization
prob-lems is a consequence of NP-hard proofs of (`0C) [2],
(`0C0) [3] and (`0P ) (Theorem II.1). Indeed, all these
proofs consist in a reduction from X3C and the solu-tion that established equivalence is binary. Therefore, the additional non-negativity constraints do not change the validity of these proofs. In other words, one can repeat the same proofs as in [2, 3] and that of Theorem II.1 for the corresponding non-negative `0minimization problems
(`0C+), (`0C0+) and (`0P +). Another way to prove the
NP-hardness of non-negative `0minimization problems is
by a reduction from the corresponding `0 minimization
problems which are known to be NP-hard. In this reduc-tion, the instance of non-negative problems is defined by
˜
y = y, A = [A, −A],˜ x =˜ x
+
x−
where x+ = max{x, 0}, x− = max{−x, 0}. Naturally,
by this construction, one gets ˜x ≥ 0, k˜xk0 = kxk0 and
˜
A˜x = Ax. The proofs (skipped for brevity) contain three steps similar to that of Theorem II.1.
Therefore, we can state the following theorem without proof.
Theorem III.1. (`0C+), (`0C0+) are NP-hard. The same
for (`0P +) with 0 < λ < 3.
In the same spirit and using the same argument as at the end of Section II-C one can directly extend Theorem II.2 to the non-negative setting.
Theorem III.2. Problem minx≥0 ky − Axkpp+ λkxk0 is
IV. CONCLUSION
NP-hardness of penalized `0 minimization problems
cannot be deduced from previous complexity analyses, as stated in [4, 5]. Here, we introduced a new proof of NP-hardness of penalized `0minimization problems when
the regularization parameter λ is smaller than 3, by an adaptation of Natarajan’s construction [2], while the case λ ≥ 3 is still open. Besides, we showed that the `0
minimization problems with non-negative constraints are also NP-hard.
This work can be extended in several directions. For instance, researchers interested in what makes NP-hard problems even harder might be interested in the strong NP-hardness of the aforementioned optimization problems. As it is widely believed that X3C is strongly NP-complete, one might easily deduce the strong NP-hardness of (`0C),
(`0C0) and other problems which are reduced from X3C.
However, to the best of our knowledge, X3C is only proved to be complete [17, pp. 53, 221] and the strong NP-completeness has not been rigorously shown yet. There-fore, we believe that the question of strong NP-hardness of (non-negative) `0 minimization problems is not trivial
and needs more work in future.
Besides, as (non-negative) `0minimization problems are
NP-hard, it would be interesting to know if the associated decision problems are in NP (so being NP-complete). Let us consider the decision problem associated with (`0C):
given y ∈ Qm, A ∈ Qm×n, a positive rational number and a positive integer K, does there exist x ∈ Rnsuch that ky − Axk2 ≤ and kxk0 ≤ K? This decision problem
should be in NP since if one can guess a rational solution x, it can be verified in polynomial time if ky − Axk2≤
and kxk0≤ K. Similarly, we conjecture that the decision
version of other optimization problems mentioned in the paper are in NP as well.
Another perspective is the approximability of aforemen-tioned NP-hard problems. The hardness of approximating (`0C) was discussed in [21, 22]. It was shown that
approxi-mating (`0C) to within a factor of (1−α) ln(n), 0 < α < 1
is NP-hard [22]. Examining whether similar results can be obtained on other NP-hard problems presented in the paper would require more involved theoretical analysis, which is left for future work.
V. ACKNOWLEDGMENT
The authors would like to thank M. Mongeau (ENAC, Toulouse) for helpful discussions and two anonymous reviewers for constructive and valuable comments.
REFERENCES
[1] M. Elad, Sparse and redundant representations: From theory to applications in signal and image processing, Springer, New York, Aug. 2010.
[2] B. K. Natarajan, “Sparse approximate solutions to linear systems”, SIAM J. Comput., vol. 24, no. 2, pp. 227–234, Apr. 1995. [3] G. Davis, S. Mallat, and M. Avellaneda, “Adaptive greedy
approx-imations”, Constructive Approximation, vol. 13, no. 1, pp. 57–98, Mar. 1997.
[4] X. Chen, D. Ge, Z. Wang, and Y. Ye, “Complexity of unconstrained l2-lp minimization”, Mathematical Programming, , no. 143, pp. 371–383, 2014.
[5] X. Huo and J. Chen, “Complexity of penalized likelihood estima-tion”, J. Statist. Comput. Simul., , no. 80:7, pp. 747–759, 2010.
[6] M.-D. Iordache, J. M. Bioucas-Dias, and A. Plaza, “Sparse unmixing of hyperspectral data”, IEEE Trans. Geosci. Remote Sensing, vol. 49, no. 6, pp. 2014–2039, June 2011.
[7] E. Esser, Y. Lou, and J. Xin, “A method for finding structured sparse solutions to nonnegative least squares problems with applications”, SIAM J. Imaging Sci., vol. 6, no. 4, pp. 2010–2046, Oct. 2013. [8] T. Virtanen, J. F. Gemmeke, and B. Raj, “Active-set Newton
algorithm for overcomplete non-negative representations of audio”, IEEE Trans. Audio, Speech, Language Process., vol. 21, no. 11, pp. 2277–2289, Nov. 2013.
[9] R. Bro and S. De Jond, “A fast non-negativity-constrained least squares algorithm”, J. Chemometrics, vol. 11, no. 5, pp. 393–401, 1997.
[10] S. Petra and C. Schn¨orr, “Average case recovery analysis of tomographic compressive sensing”, Linear Alg. Appl., vol. 441, pp. 168–198, 2014.
[11] P. O. Hoyer, “Non-negative matrix factorization with sparseness constraints”, J. Mach. Learning Res., vol. 5, pp. 1457–1469, 2004. [12] J. Rapin, J. Bobin, A. Larue, and J.-L. Starck, “Sparse and non-negative BSS for noisy data”, IEEE Trans. Signal Process., vol. 61, no. 22, pp. 5620–5632, Nov. 2013.
[13] A. M. Bruckstein, M. Elad, and M. Zibulevsky, “On the uniqueness of nonnegative sparse solutions to underdetermined systems of equation”, IEEE Trans. Inf. Theory, vol. 54, no. 11, pp. 4813– 4820, Nov. 2008.
[14] R. Peharz and F. Pernkopf, “Sparse nonnegative matrix factorization with l0 constraints”, Neurocomputing, vol. 80, pp. 38–46, 2012. [15] M. Yaghoobi, D. Wu, and M. E. Davies, “Fast non-negative
orthogonal matching pursuit”, IEEE Signal Process. Lett., vol. 22, no. 9, pp. 1229–1233, Sept. 2015.
[16] Z. Wang, R. Zhu, K. Fukui, and J. Xue, “Cone-based joint sparse modelling for hyperspectral image classification”, Signal Processing, vol. 144, pp. 417–429, 2018.
[17] M. R. Garey and D. S. Johnson, Computers and Intractability: A Guide to the Theory of NP-Completeness, W. H. Freeman, New York, 1979.
[18] J. v. Leeuwen, Handbook of theoretical computer science (vol. A): algorithms and complexity, Elsevier Science Publishers, 1990. [19] X. Huo and X. Ni, “When do stepwise algorithms meet subset
selection criteria?”, Annals Statist., vol. 35, no. 2, pp. 870–887, 2007.
[20] M. Nikolova, “Description of the minimizers of least squares regularized with `0 norm. Uniqueness of the global minimizer”,
SIAM J. Imaging Sci., vol. 6, no. 2, pp. 904–937, May 2013. [21] E. Amaldi and V. Kann, “On the approximability of minimizing
nonzero variables or unsatisfied relations in linear systems”, The-oretical Computer Science, vol. 209, no. 1, pp. 237 – 260, 1998. [22] A. M. Tillmann, “On the computational intractability of exact and
approximate dictionary learning”, IEEE Signal Processing Letters, vol. 22, no. 1, pp. 45–49, Jan 2015.
