A necessary and sufficient condition for exact recovery by l1 minimization.

HAL Id: hal-00164738

https://hal.archives-ouvertes.fr/hal-00164738v2

Preprint submitted on 25 Nov 2011

HAL

is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L’archive ouverte pluridisciplinaire

destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

A necessary and sufficient condition for exact recovery by l1 minimization.

Charles Dossal

To cite this version:

Charles Dossal. A necessary and sufficient condition for exact recovery by l1 minimization.. 2011.

�hal-00164738v2�

A necessary and sufficient condition for exact sparse recovery by ℓ

1 minimization

Charles Dossal

,

aIMB, Université Bordeaux 1, 351, cours de la Libération, F-33405 Talence cedex (FRANCE)

Abstract

In this paper, a new sharp sufficient condition for exact sparse recovery byℓ1-penalized minimization from linear measurements is proposed. The main contribution of this paper is to show that, for most matrices, this condition is also necessary. Moreover, when theℓ1 minimizer is unique, we investigate its sensitivity to the measurements and we establish that the application associating the measurements to this minimizer is Lipschitz-continuous.

Résumé

Une condition nécessaire et suffisante d’identifiabilité parcimonieuse par minimisationℓ1. Dans cet article, une nouvelle condition suffisante pour l’identifiabilité parcimonieuse par minimisationℓ1pénalisée à partir de mesures linéaires est proposée. La contribution majeure de ce travail est de prouver que pour la plupart des matrices, cette condition est aussi nécessaire. Par ailleurs, lorsque le minimiseur du problème ℓ1 est unique, sa sensibilité aux mesures est étudiée est il montré que l’application qui envoie les mesures sur ce minimiseur est Lipschitz-continue.

1. Introduction

Let x⁰ ∈ R^N be a vector and A be a real matrix with n rows and N columns. Let y⁰ be n linear measurements of x⁰, i.e.y⁰ =Ax⁰ ∈Rⁿ, where typicallyn < N. In this paper, we propose a necessary and sufficient condition ensuring thatx⁰ can be recovered fromy⁰ by solving the following optimization problemP1(y⁰)

x∈minRNkxk₁ such that y=Ax. P1(y)

There is of course a huge literature on the subject, and covering it fairly is beyond the scope of this paper.

We restrict our overview to those works pertaining to ours. For instance, our new condition can be seen as an extension of [3] and [?]. Indeed, in [3], the following optimization problem (so-called Lasso problem)

x∈minRN

1

2ky−Axk²₂+γkxk₁. P1(y, γ)

is studied. It is proved that anyx⁰ belonging to the setF is the unique solution ofP1(Ax⁰), where F=

xsuch that rank(AI) =|I|and∀j /∈I, |haj, AI(A^t_IAI)⁻¹sign (xI)i|<1 ,

whereIis the support ofx,AI the active matrix associated tox, whose columns are those ofAindexed by I,xI are the non-zero components ofxandajis the column ofAindexed byj. In the sequel,Span(ai)i∈I

will be denoted VI. The sufficient condition developed in [3] plays a pivotal role in several papers which investigate support recovery in presence of noise by solving P1(y, γ), e.g. [1,4,2].

In [?], the Exact Recovery Condition ERC is defined. this condition does not depend on the sign but only on the support ofx⁰ and provides results about the recovery ofx⁰ and stability to noise.

The goal of this paper is to show that the set of vectors that can be recovered by ℓ1 minimization is exactly, for most matrices,F the closure ofF. This result highlights the fact that

maxj /∈I |haj, AI(A_I^tAI)⁻¹sign (xI)i|and

AI(A^t_IAI)⁻¹sign (xI) 2

are good indicators of the identifiablility, and brings arguments justifying the success of algorithms de- veloped in [?] to find very sparse but non-identifiable vectors.

Email address:charles.dossal@math.u-bordeaux1.fr(Charles Dossal).

Before proceeding, let us fix some terminology and definitions. A vectorx⁰ is said identifiable if and only if it is the unique solution ofP1(Ax⁰).

Definition1.1 Let (xⁱ)i6N be N points of Rⁿ. These points(xⁱ)i6N are said in general position (GP) if all affine subspaces ofRⁿ which dimensionk < ncontain at most k+ 1pointsxⁱ. A matrixA satisfies condition(GP)if for all sign vectorS∈ {−1,1}^N, points(S[i]ai)i6N are in general position.

It can be noticed that for any matrix A, the matrix A+E satisfies condition (GP)with probability 1 ifE is any random perturbation with a an absolutely continuous density with respect to the Lebesgue measure: with this definition most matrices satisfy condition(GP).

2. Contributions

The contributions of this paper are summarized as follows.

Theorem 2.1 Ifx⁰∈ F, thenx⁰ is identifiable.

Theorem 2.2

(i) Ifx⁰is identifiable, and if for allyin a neighborhood ofy⁰=Ax⁰P1(y)has only one solution, then x⁰∈ F.

(ii) If Asatisfies(GP), for ally∈Im (A),P1(y)has a unique solution andxis identifiable if and only if x∈ F.

Theorem 2.3 If for all y ∈ Im (A) the solution of P1(y) is unique, the application φ associating y to this solution is Lipschitz.

3. Preliminary Lemmas

The two following Lemmas can be found in [3].

Lemma 3.1 A vectorx^∗ is a solution ofP1(y, γ)if and only if

A^t_I(y−Ax^∗) =γsign (x^∗_I) and∀j /∈I|haj, y−Ax^∗i|61 where the support ofx^∗ is denoted byI.

Lemma 3.2 If for a vectorx^∗ the matrix AI satisfies rank(AI) =|I| and

A^t_I(y−Ax^∗) =γsign (x^∗_I) and∀j /∈I,|haj, AI(A^t_IAI)⁻¹sign (x^∗_I)i|<1 thenx^∗ is the unique minimizer of P1(y, γ).

Lemma 3.3 Ifx⁰ is the unique solution ofP1(Ax⁰)then rank(AI) =|I|.

Proof: If there existsh∈Ker(A)that is supported onI=I(x⁰), one has for allt∈R,A(x⁰+th) = Ax⁰. Consequently the applicationt7→

x⁰+th

₁is locally affine in a neighborhood of 0. It follows that there existst6= 0such that

x⁰+th ₁6

x⁰ ₁.

Lemma 3.4 For ally∈Im (A) there exists a solutionxtoP1(y)such that rank(I) =|I|.

Proof: Letx⁰be a solution ofP1(y). If rank(I)<|I|, there existsh∈Ker(A)that is supported on I. For a suitablet,I(x⁰+th)(I(x⁰)andx⁰+this a solution ofP1(y).

Lemma 3.5 Ifx⁰∈ F, thenx⁰ is the unique solution ofP1(Ax⁰)

Proof: Lemma 3.2 shows that x⁰ is the unique solution of P1(AI(x⁰_I+γ(A^t_IAI)⁻¹sign x⁰_I ), γ) if γ >0 is small enough.

Lemma 3.6 Let y∈Im (A)and(γn)n∈N a sequence of positive real numbers tending vers 0. If(xⁿ)n∈N

are some solutions of P1(y, γn)sucht that limn→∞xⁿ=x⁰ one gets thatx⁰ is a solution ofP1(y).

4. Proofs

4.1. Proof of theorem 2.1

From Lemma 3.5 one has that vectors inF are identifiable. SinceAI(A^t_IAI)⁻¹sign (xI)only depends on the sign and the support ofx,F is a union of cones of various dimensions. It follows that the closure F ofF is exactly equal to the set of vectorsx⁰ that can extented into a vector ofF :

2

F=

x⁰ such that∃x¹such thatI(x⁰)∩I(x¹) =∅ andx⁰+x¹=x²∈ F

Let us now suppose thatx⁰∈ F, and that there exists x¹ such thatI(x⁰)∩I(x¹) =∅ andx²=x⁰+x¹ belongs toF.

Choosex³∈R^N such thatAx⁰=Ax³ and define x⁴:=x³+x¹. We haveAx⁴=Ax² and sincex²∈ F, x²is the unique solution of P1(Ax²)which implies thatkx²k1<kx⁴k1. It follows that

x² ₁=

x⁰ ₁+

x¹ ₁<

x⁴ ₁6

x³ ₁+

x¹

₁, (1) which implies

x⁰ ₁<

x³

₁. That is,x⁰is the unique solution of P1(Ax⁰).

4.2. Proof of theorem 2.2

Proof: Lety∈Im(A)and(γn)n∈Na sequence of positive real numbers tending to 0. From Lemma 3.4 we can always choose a solution x(γn) ofP1(y, γ)such that the associated active matrix AIn has a full rank. Since the number of possible supports and signs is finite, up to an extraction of a subsequence, we can suppose that allx(γn)share the same support Iand the same signS. From Lemma 3.1, we know thatx(γn)satisfies

x(γn)I =x⁰_I−γn(A^t_IAI)⁻¹S and∀j /∈I |haj, y−Ax(γn)i|6γ (2) where x⁰ is the vector supported on I such that x⁰_I = (A_I^tAI)⁻¹A^t_Iy. Using Lemma 3.6, x⁰ is a solu- tion of P1(y). From (2) we deduce that I(x⁰) ⊂ I for γn small enough. It follows thaty−Ax(γn) = γAI(A^t_IAI)⁻¹S =γdI,S and thus∀j /∈I,|haj, dI,Si|61

Let us defineJ to be the set of all indices such that|haj, dI,Si|= 1. One can first notice thatI⊂J. We now show that

(i) either(aj)j∈J are linearly dependent and one can build vectorsx¹close tox⁰such thatP1(x¹)has several solutions, in which case condition(GP)cannot be satisfied;

(ii) or(aj)j∈J are linearly independent andx(γn)∈ F andx⁰∈ F andx⁰ is identifiable.

(i) Suppose that (aj)j∈J are linearly dependent and let ε > 0. Define K ( J such that I ⊂ K and (ak)k∈K is a basis ofVJ = Span(aj)j∈J andj0∈J∩K^c. DefineSK∈R^|K|bySK =A^t_KdI,S. Then, SK is a sign vector andx¹ the vector supported onKdefined by

x¹_K =x⁰_K+ ε kSKk₂SK

Now chooseε >0small enough to ensure thatsign x¹_I(x0)

= sign x⁰_I(x0)

. From definition ofx¹, it follows that

x⁰−x¹

₂ =ε. For γ small enough the support and the sign ofx¹ and the vector x¹(γ)supported inK and defined byx¹(γ)K=x¹_K−γ(A^t_KAK)⁻¹SK are identical.

We shall prove thatx¹(γ)is a solution ofP1(Ax¹, γ)using Lemma 3.1. Indeed A^t_K(Ax¹−Ax¹(γ)) =γSK

The vector S = A^t_IdI,S is the common sign of vectors (x(γn))n∈N. Since limn→∞x(γn) = x⁰ it follows that for alli∈I(x⁰)⊂I,sign x⁰[i]

=S[i]. Since for alli∈I, S[i] =hai, dI,Si=SK[i]it follows thatSK= sign x¹_K

. Moreoversign x¹_K

= sign x¹(γ)K

which yields that A^t_K(Ax¹−Ax¹(γ)) =γsign x¹(γ)K

SincedI,S∈VI ⊂VK, one hasdI,S =AK(A^t_KAK)⁻¹A^t_KdI,S=AK(A_K^t AK)⁻¹SK. Ifj /∈K,

|haj, Ax¹−Ax¹(γ)i|=γ|haj, AK(A_K^t AK)⁻¹SKi|=γ|haj, dI,Si|< γ,

and by Lemma 3.1 it follows thatx¹(γ)is a solution ofP1(Ax¹, γ). Now Lemma 3.6 shows thatx¹is a solution ofP1(Ax¹). Choose someh∈Ker(A)that is supported onK∪ {j0}and such thath[j0] = S[j0] =a^t_j0dI,S. DenotehK the restriction ofhon indicesK. Since0 =Ah=S[j0]aj0+AKhK and SK =A^t_KdI,S one has

hsign x¹_K

, hKi=S_K^t hk=d^t_I,SAKhK =−S[j0]d^t_I,Saj0 =−1 Moreover, for allt∈R,A(x¹+th) =Ax¹ and for small non negativet

x¹+th ₁=

x¹

₁+t+thsign x¹

, hKi= x¹

₁, which shows thatx¹is not the unique minimizer ofP1(Ax¹).

Notice that for each l∈K∪ {j0}, S[l]al∈VK andS[l]al belongs to the affine hyperplan HdI,S =

{u, such thathu, dI,Si = 1}. Hence at least |K|+ 1 points S[l]al belong to an affine subspace VK∩ HdI,S which dimension equals to|K| −1. Therefore,(GP)is not satisfied.

(ii) Let us now suppose that the(aj)j∈J are linearly independent. DefineSJ =A^t_JdI,S,∀i∈I, S[i] = SJ[i]. Let x¹ be the vector supported on J such that for all i ∈ I(x⁰), x¹[i] = x⁰[i] and for all j ∈J ∩I(x⁰)^c, x¹[j] = SJ[j]. One has sign x¹_J

=SJ and since dI,S ∈ VI ⊂VJ, one hasdI,S = AJ(A^t_JAJ)⁻¹A^t_JdI,S=AJ(A^t_JAJ)⁻¹SJ. It follows that for alll /∈J,

|hal, AJ(A^t_JAJ)⁻¹SJi|=|hal, dI,Si|<1 implying thatx¹∈ F and hencex⁰∈ F.

4.3. Proof of theorem 2.3

The proof relies on the following lemma

Lemma 4.1 For all y⁰ ∈ Im (A), there exists ε0 such that for all y ∈ Im (A)∩ B(y⁰, ε0), one has I(φ(y⁰))⊂I(φ(y))

Proof: To prove this lemma we will prove that for any y⁰ ∈ Im (A) and any sequence (yⁿ)n∈N

tending to y⁰, any subsequence (y^un)n∈N such that I(φ(y^un)) = Jun = J is constant, the support J satisfiesJ ⊃I=I(φ(y⁰)).

Denote byx⁰=φ(y⁰)andI=I(x⁰). One hasx⁰(I) = (A^t_IAI)⁻¹A^t_Iy⁰=A_I⁺y⁰. Let(yⁿ)n∈Na sequence of elements ofIm (A)tending toy⁰andxⁿ=φ(yⁿ). Up to an extraction of a subsequence, one can suppose that for alln∈N,I(xⁿ)is constant. Denote byJ this common support. Thenxⁿ_J = (A^t_JAJ)⁻¹A^t_Jyⁿ = A⁺_Jyⁿ. Letx^∞= limn→∞xⁿ andx^∞_J =A⁺_Jy⁰. Letzⁿ be the sequence of vectors supported onI defined by zⁿ_I =A⁺_Iyⁿ. By the definiton of zⁿ, Azⁿ = PVIyⁿ. Let K be a set such that (ak)k∈K is a basis of Im (A), andvⁿ the vector supported onK such thatvⁿ_K =A⁺_K(P_VI⊥(yⁿ)). One has

A(zⁿ+vⁿ) =yⁿ and kzⁿ+vⁿk₁6kzⁿk₁+kvⁿk₁ Noticing that

kvⁿk₁=

A⁺_K(P_VI⊥(yⁿ−y⁰)) ₁ −→

n→∞0 One deduces that limn→∞kzⁿ+vⁿk₁ =

x⁰

₁. Moreover limn→∞kxⁿk₁ = kx^∞k₁ and since kxⁿk₁ 6 kzⁿ+vⁿk₁ we deduce that kx^∞k₁ 6

x⁰

₁ and finally x⁰ = x^∞ since x⁰ = φ(y⁰). Thus one has I = I(x⁰) =I(x^∞)⊂J which concludes the proof of the lemma.

Lety⁰∈Im (A). There existsε0 >0 such that for ally ∈Im (A)∩ B(y⁰, ε0), there exists J satisfying I ⊂J. Moreover x=φ(y) is supported inJ andxJ =A_J⁺y. SinceI ⊂J, one also hasx⁰_J =A⁺_Jy⁰ and thus

x−x⁰ 2=

A⁺_J(y−y⁰)

2. One deduces that

∀y∈Im (A)∩ B(y⁰, ε0),

φ(y⁰)−φ(y) 26C

y⁰−y

2 withC= max

J,rank(AJ)=|J|λmax(A⁺_J) which concludes the proof of the theorem.

References

[1] E.J. Candès and Y. Plan. Near ideal model selection vis l1 minimization. Ann Statit, 2007.

[2] Ch. Dossal, M.L. Chabanol, G. Peyré, and j. Fadili. Sharp suppôrt recovery from noisy random measurements by l1 minimization. Applied and Computational Harmonic Analysis, 2011.

[3] J.J. Fuchs. On sparse representations in arbitrary redundant bases. IEEE-I-IT, 2002.

[4] Wainwright. Sharp thresholds for high-dimensional and noisy sparsity recovery using ℓ1-constrainted quadratic programming(lasso). IEEE Trans. Info. Theory, 55(5):2183–2202, 2009.

4