-ORACLE INEQUALITY FOR THE LASSO IN MULTIVARIATE FINITE MIXTURE OF MULTIVARIATE GAUSSIAN REGRESSION MODELS

DOI:10.1051/ps/2015011 www.esaim-ps.org

AN

-ORACLE INEQUALITY FOR THE LASSO IN MULTIVARIATE FINITE MIXTURE OF MULTIVARIATE GAUSSIAN REGRESSION MODELS

Emilie Devijver

Abstract. We consider a multivariate finite mixture of Gaussian regression models for high- dimensional data, where the number of covariates and the size of the response may be much larger than the sample size. We provide an 1-oracle inequality satisfied by the Lasso estimator according to the Kullback−Leibler loss. This result is an extension of the 1-oracle inequality established by Meynet in [ESAIM: PS 17(2013) 650–671]. in the multivariate case. We focus on the Lasso for its 1-regularization properties rather than for the variable selection procedure.

Mathematics Subject Classification. 62H30.

Received October 17, 2014. Revised April 23, 2015.

1. Introduction

Finite mixture regression models are useful for modeling the relationship between response and predictors, arising from different subpopulations. Due to recent improvements, we are faced with high-dimensional data where the number of covariables can be much larger than the sample size. We have to reduce the dimension to avoid identifiability problems. Considering a mixture of linear models, an assumption widely used is to say that only a few covariates explain the response. Among various methods, we focus on the 1-penalized least squares estimator of parameters to lead to sparse regression matrix. Indeed, it is a convex surrogate for the non-convex0-penalization, and it produces sparse solutions. First introduced by Tibshirani in [11] in a linear model Y = Xβ+, where X ∈ R^p, Y ∈ R, and ∼ N(0, σ²), the Lasso estimator is defined in the linear model by

βˆ^Lasso(λ) = argmin

β∈R^p

||Y −Xβ||²2+λ||β||1

, λ >0.

Many results have been proved to study the performance of this estimator. For example, [1,4], for studying this estimator as a variable selection procedure in the linear model. Note that those results need strong assumptions on the Gram matrixX^tX, as the restrictive eigenvalue condition, that can be not fulfilled in practice. A summary of assumptions and results is given by B¨uhlmann and van de Geer in [12]. One can also cite van de Geer in [13]

and discussions, who precises a chaining argument to perform rate, even in a non linear case.

If we assume that (x_i, y_i)1≤i≤n arise from different subpopulations, we could work with finite mixture re- gression models. Indeed, the homogeneity assumption of the linear model is often inadequate and restrictive.

Keywords and phrases.Finite mixture of multivariate regression model, Lasso,1-oracle inequality.

1 Laboratoire de Math´ematiques d’Orsay, Facult´e des Sciences d’Orsay, Universit´e Paris-Sud, 91405 Orsay, France.

[email protected]

Article published by EDP Sciences c EDP Sciences, SMAI 2015

This model was introduced by St¨adleret al.in [10]. They assume that, for i ∈ {1, . . . , n}, the observationy_i, conditionally to X_i =x_i, comes from a conditional density s_ξ0(.|x_i) which is a finite mixture of K Gaussian conditional densities with proportion vectorπ, where

Y_i|X_i=x_i ∼s_ξ0(y_i|x_i) = K k=1

π_k⁰

√2πσ⁰_kexp

−(y_i−β_k⁰x_i)² 2(σ_k⁰)²

for some parametersξ⁰= (π⁰_k, β_k⁰, σ_k⁰)1≤k≤K. They extend the Lasso estimator by

ˆ

s^Lasso(λ) = argmin

s^K_ξ

⎧⎨

⎩−1 n

n i=1

log(s^K_ξ (y_i|xi)) +λ K k=1

π_k p j=1

|σ⁻_k¹[β_k]_j|

⎫⎬

⎭, λ >0 (1.1) For this estimator, they provide an 0-oracle inequality satisfied by ˆs^Lasso(λ), according to the restricted eigenvalue condition also, and margin conditions, which leads to link the Kullback−Leibler loss function to the 2-norm of the parameters.

Another way to study this estimator is to look after the Lasso for its1-regularization properties. For example, [6,8,9]. Contrary to the0-results, some1-results are valid with no assumptions, neither on the Gram matrix, nor on the margin. This can be achieved due to the fact that they are looking for rate of convergence of order 1/√

nrather than 1/n. For finite mixture Gaussian regression models, we could cite Meynet in [8] who gives an 1-oracle inequality for another extension of the Lasso estimator, defined by

ˆ

s^Lasso(λ) = argmin

s^K_ξ

⎧⎨

⎩−1 n

n i=1

log(s^K_ξ (y_i|x_i)) +λ K k=1

p j=1

|[β_k]_j|

⎫⎬

⎭, λ >0. (1.2) In this article, we extend this result to finite mixture of multivariate Gaussian regression models. We will work with random multivariate variables (X, Y)∈R^p×R^q. As in [8], we shall restrict to the fixed design case, that is to say non-random regressors. We observe (x_i)1≤i≤n. Without any restriction, we could assume that the regressors x_i ∈[0,1]^p for all i ∈ {1, . . . , n}. Under only bounded parameters assumption, we provide a lower bound on the Lasso regularization parameterλwhich guarantees an oracle inequality.

This result is non-asymptotic: the number of observations is fixed, and the number pof covariates can grow.

Remark that the numberK of clusters in the mixture is supposed to be known. Our result is deduced from a finite mixture multivariate Gaussian regression model selection theorem for 1-penalized maximum likelihood conditional density estimation. We establish the general theorem following the one of Meynet in [8], which combines Vapnik’s structural risk minimization method (see Vapnik in [15]) and theory around model selection (see Le Pennec and Cohen in [3] and Massart in [5]). As in Massart and Meynet in [6], our oracle inequality is deduced from this general theorem, the Lasso estimator being viewed as the solution of a penalized maximum likelihood model selection procedure over a countable collection of1-ball models.

The article is organized as follows. The model and the framework are introduced in Section2. In Section3, we state the main result of the article, which is an1-oracle inequality satisfied by the Lasso in finite mixture of multivariate Gaussian regression models. Section4 is devoted to the proof of this result and of the general theorem, deriving from two easier propositions. Those propositions are proved in Section5, whereas details of lemma states in Section6.

2. Notations and framework

2.1. Finite mixture regression model

We observenindependent couples (x,y) = (x_i, y_i)1≤i≤n ∈([0,1]^p×R^q)ⁿ, withy_i∈R^qa random observation, realization of variableY_i, andx_i ∈[0,1]^p fixed for alli∈ {1, . . . , n}. We assume that, conditionally to thex_is,

theY_is are independent and identically distributed with conditional densitys_ξ0(.|x_i), which is a finite mixture of K Gaussian regressions with unknown parameters ξ⁰. In this article, K is fixed, then we do not precise it with unknown parameters. We will estimate the unknown conditional density by a finite mixture of K Gaussian regressions. Each subpopulation is then estimated by a multivariate linear model. Detail the conditional densitys_ξ. For ally∈R^q, for allx∈[0,1]^p,

s_ξ(y|x) = K k=1

π_k

(2π)^q/2det(Σ_k)^1/2exp

−(y−β_kx)^tΣ_k⁻¹(y−β_kx) 2

(2.1) ξ= (π1, . . . , π_K, β1, . . . , β_K, Σ1, . . . , Σ_K)∈Ξ=

Π_K×(R^q×p)^K×(S⁺⁺_q )^K Π_K=

(π1, . . . , π_K);π_k >0 for allk∈ {1, . . . , K} and K k=1

π_k= 1

S⁺⁺_q is the set of symmetric positive definite matrices onR^q.

We want to estimate ξ⁰ from the observations. For all clusterk∈ {1, . . . , K}, βk is the matrix of regression coefficients, and Σ_k is the covariance matrix in the mixture component k, whereas the π_ks are the mixture proportions. Forx∈[0,1]^p, we define the parameterξ(x) of the conditional densitys_ξ(.|x) by

ξ(x) = (π1, . . . , π_K, β1x, . . . , β_Kx, Σ1, . . . , Σ_K)∈]0,1[^K×(R^q)^K×(S⁺⁺_q )^K. For allk∈ {1, . . . , K}, for allx∈[0,1]^p, for allz∈ {1, . . . , q}, [βkx]_z =_p

j=1[β_k]_z,j[x]_j, and thenβ_kx∈R^q is the mean vector of the mixture componentkfor the conditional densitys_ξ(.|x).

2.2. Boundedness assumption on the mixture and component parameters

Denote, for a matrix A, m(A) the modulus of the smallest eigenvalue of A, and M(A) the modulus of the largest eigenvalue of A. We shall restrict our study to bounded parameters vector ξ = (π,β,Σ), where π = (π1, . . . , π_K),β= (β1, . . . , β_K),Σ = (Σ1, . . . , Σ_K). Specifically, we assume that there exists deterministic positive constantsA_β, a_Σ, A_Σ, a_π such thatξbelongs to ˜Ξ, with

Ξ˜=

ξ∈Ξ: for allk∈ {1, . . . , K}, max

z∈{1,...,q} sup

x∈[0,1]^p|[β_kx]_z| ≤A_β,

a_Σ ≤m(Σ_k⁻¹)≤M(Σ_k⁻¹)≤A_Σ, a_π≤π_k

. (2.2) LetS the set of conditional densitiess_ξ,

S=

s_ξ, ξ∈Ξ˜

.

2.3. Maximum likelihood estimator and penalization

In a maximum likelihood approach, we consider the Kullback−Leibler information as the loss function, which is defined for two densitiessandt by

KL(s, t) =

⎧⎪

⎨

⎪⎩

R^qlog s(y)

t(y)

s(y)dy ifsdytdy;

+∞otherwise.

(2.3)

In a regression framework, we have to adapt this definition to take into account the structure of conditional densities. For the fixed covariates (x1, . . . , x_n), we consider the average loss function

KL_n(s, t) = 1 n

n i=1

KL(s(.|x_i), t(.|x_i)) = 1 n

n i=1

R^qlog

s(y|x_i) t(y|xi)

s(y|x_i)dy.

Using the maximum likelihood approach, we want to estimates_ξ0 by the conditional densitys_ξ which maxi- mizes the likelihood conditionally to (x_i)1≤i≤n. Nevertheless, because we work with high-dimensional data, we have to regularize the maximum likelihood estimator. We consider the 1-regularization, and a generalization of the estimator associated, the Lasso estimator, which we define by

ˆ

s^Lasso(λ) := argmin

sξ∈S ξ=(π,β,Σ)

⎧⎨

⎩−1 n

n i=1

log(s_ξ(y_i|x_i)) +λ K k=1

q z=1

p j=1

|[β_k]_z,j|

⎫⎬

⎭; whereλ >0 is a regularization parameter.

We define also, fors_ξ defined as in (2.1), and with parametersξ= (π,β,Σ),

N₁^[2](s_ξ) =||β||1= K k=1

p j=1

q z=1

|[β_k]_z,j|. (2.4)

3. Oracle inequality

In this section, we provide an1-oracle inequality satisfied by the Lasso estimator in finite mixture multivariate Gaussian regression models, which is the main result of this article.

Theorem 3.1. We observe ncouples (x,y) = ((x1, y1), . . . ,(x_n, y_n))∈([0,1]^p×R^q)ⁿ coming from the condi- tional density s_ξ0, whereξ⁰∈Ξ˜, where

Ξ˜=

ξ∈Ξ: for allk∈ {1, . . . , K}, max

z∈{1,...,q} sup

x∈[0,1]^p|[β_kx]_z| ≤A_β,

a_Σ ≤m(Σ_k⁻¹)≤M(Σ_k⁻¹)≤A_Σ, a_π≤π_k

. Denote bya∨b= max(a, b).

We define the Lasso estimator, denoted bysˆ^Lasso(λ), for λ≥0, by ˆ

s^Lasso(λ) = argmin

sξ∈S

−1 n

n i=1

log(s_ξ(y_i|x_i)) +λN₁^[2](s_ξ)

; (3.1)

with

S=

s_ξ, ξ∈Ξ˜

and where, for ξ= (π,β,Σ),

N₁^[2](s_ξ) =||β||1= K k=1

p j=1

q z=1

|[βk]_z,j|.

Then, if

λ≥κ

A_Σ∨ 1

a_π 1 + 4(q+ 1)A_Σ

A²_β+log(n) a_Σ

K n

1 +qlog(n)

Klog(2p+ 1)

with κan absolute positive constant, the estimator (3.1)satisfies the following 1-oracle inequality.

E[KL_n(s_ξ0,sˆ^Lasso(λ))]≤(1 +κ⁻¹) inf

sξ∈S

KL_n(s_ξ0, s_ξ) +λN₁^[2](s_ξ)

+λ +κ

K n

e⁻¹²π^q/2a_π A^q/_Σ²

2q

+κ K

n

A_Σ∨ 1

a_π 1 + 4(q+ 1)A_Σ

A²_β+log(n) a_Σ

×K

1 +A_β+ q a_Σ

2

whereκ is a positive constant.

This theorem provides information about the performance of the Lasso as an1-regularization algorithm. If the regularization parameterλis properly chosen, the Lasso estimator, which is the solution of the1-penalized empirical risk minimization problem, behaves as well as the deterministic Lasso, which is the solution of the 1-penalized true risk minimization problem, up to an error term of orderλ.

Our result is non-asymptotic: the numbernof observations is fixed while the numberpof covariates and the sizeq of the response can grow with respect tonand can be much larger thann. The numberKof clusters in the mixture is fixed.

There is no assumption neither on the Gram matrix, nor on the margin, which are classical assumptions for oracle inequality for the Lasso estimator. Moreover, this kind of assumptions involve unknown constants, whereas here, every constants are explicit. We could compare this result with the0-oracle inequality established in [10], which needs those assumptions, and is therefore difficult to interpret. Nevertheless, they get faster rate, the error term in the oracle inequality being of order 1/nrather than 1/√

n.

The main assumption we make to establish the Theorem 3.1 is the boundedness of the parameters, which is also assumed in [10]. It is needed to tackle the problem of the unboundedness of the likelihood (see [7] for example).

Moreover, we let regressors to belong to [0,1]^p. Because we work with fixed covariates, they are finite. To simplify the reading, we choose to rescalexto get||x||_∞≤1. Nevertheless, if we not rescale the covariates, and the regularization parameterλbound and the error term of the oracle inequality depend linearly of||x||∞.

The regularization parameterλbound is of order (q²+q)/√

nlog(n)²

log(2p+ 1). Forq= 1, we recognize the same order, as regards to the sample sizenand the number of covariatesp, to the1-oracle inequality in [8].

A great attention has been paid to get a lower bound ofλwith optimal dependence onp, which is the number of regressors, but we are aware that dependences inqandK may not be optimal. Indeed, even if roles ofpand q are not symmetric, we can wonder if a dependence of order logarithm inq could be expected, which is not achieved here. For the number of components, a dependence in√

Kcould be envisaged, see [8]. Those optimal rates are open problems.

Van de Geer, in [13], gives some tools to improve the bound of the regularization parameter to

log(p) n . Nevertheless, we have to control eigenvalues of the Gram matrix of some functions (ψ_j(x_i))1≤j≤D

1≤i≤n,D being the number of parameters to estimate, whereψ_j(x_i) satisfies

|log(s_ξ(y_i|x_i))−log(sξ˜(y_i|x_i))| ≤^D

j=1

|ξ_j−ξ˜_j|ψ_j(x_i).

In our case of mixture of regression models, control eigenvalues of the Gram matrix of functions (ψ_j(x_i))1≤j≤D 1≤i≤n

corresponds to make some assumptions, as REC, to avoid dimension reliance onn, K andp. Without this kind of assumptions, we could not guarantee that our bound is of order

log(p)

n , because we could not guarantee that eigenvalues does not depend on dimensions. In order to get a result with smaller assumptions, we do not use the chaining argument developed in [13]. Nevertheless, one can easily compute that, under restricted eigenvalue condition, we could perform the order of the regularization parameter toλ

log(p) n log(n).

4. Proof of the oracle inequality

4.1. Main propositions used in this proof

The first result we will prove is the next theorem, which is an1-ball mixture multivariate regression model se- lection theorem for1-penalized maximum likelihood conditional density estimation in the Gaussian framework.

Theorem 4.1. We observe(x_i, y_i)1≤i≤n with unknown conditional Gaussian mixture densitys_ξ0.

For allm∈N^∗, we consider the1-ball S_m={sξ ∈S, N₁^[2](s_ξ)≤m} for S={sξ, ξ∈Ξ}, and˜ Ξ˜ defined by

Ξ˜=

ξ∈Ξ: for allk∈ {1, . . . , K}, max

z∈{1,...,q} sup

x∈[0,1]^p|[β_kx]_z| ≤A_β,

a_Σ ≤m(||Σ_k⁻¹||)≤M(Σ_k⁻¹)≤A_Σ, a_π≤π_k

.

Forξ= (π,β,Σ), let

N₁^[2](s_ξ) =||β||1= K k=1

p j=1

q z=1

|[β_k]_z,j|.

Let sˆ_m anη_m-log-likelihood minimizer inS_m, for η_m≥0:

−1 n

n i=1

log(ˆs_m(y_i|xi))≤ inf

sm∈Sm

−1 n

n i=1

log(s_m(y_i|xi))

+η_m.

Assume that for all m∈N^∗, the penalty function satisfies pen(m) =λmwith

λ≥κ

A_Σ∨ 1

a_π 1 + 4(q+ 1)A_Σ

A²_β+log(n) a_Σ

K n

1 +qlog(n)

Klog(2p+ 1)

for a constant κ. Then, ifmˆ is such that

−1 n

n i=1

log(ˆsmˆ(y_i|x_i)) + pen( ˆm)≤ inf

m∈N^∗

−1 n

n i=1

log(ˆs_m(y_i|x_i)) + pen(m)

+η

for η≥0, the estimator ˆsmˆ satisfies

E(KL_n(s_ξ0,sˆmˆ))≤(1 +κ⁻¹) inf

m∈N^∗

sminf∈SmKL_n(s_ξ0, s_m) + pen(m) +η_m

+η +κ

K n

e⁻¹²π^q/2 A^q/_Σ²

2qa_π

+κ K

nK

A_Σ∨ 1

a_π 1 +4(q+ 1) 2 A_Σ

A²_β+log(n) a_σ

×

1 +A_β+ q a_Σ

2

;

whereκ is a positive constant.

It is an 1-ball mixture regression model selection theorem for 1-penalized maximum likelihood conditional density estimation in the Gaussian framework. Its proof could be deduced from the two following propositions, which split the result if the variableY is large enough or not.

Proposition 4.2. We observe (x_i, y_i)1≤i≤n, with unknown conditional density denoted by s_ξ0. Let M_n > 0, and consider the event

T :=

i∈{max1,...,n} max

z∈{1,...,q}|[Y_i]_z| ≤M_n

.

For allm∈N^∗, we consider the1-ball

S_m={s_ξ ∈S, N₁^[2](s_ξ)≤m}

whereS ={s_ξ, ξ∈Ξ}˜ and

N₁^[2](s_ξ) =||β||1= K k=1

p j=1

q z=1

|[βk]_z,j|

for ξ= (π,β,Σ).

Let sˆ_m anη_m-log-likelihood minimizer inS_m, for η_m≥0:

−1 n

n i=1

log(ˆs_m(y_i|x_i))≤ inf

sm∈Sm

−1 n

n i=1

log(s_m(y_i|x_i))

+η_m.

Let C_Mn = max 1

aπ, A_Σ+¹₂(|Mn|+A_β)²A²_Σ,^q(|Mn|+₂^Aβ)AΣ

. Assume that for all m ∈ N^∗, the penalty function satisfiespen(m) =λm with

λ≥κ4C_Mn

√n

√K

1 + 9qlog(n)

Klog(2p+ 1)

for some absolute constantκ. Then, any estimatesˆmˆ withmˆ such that

−1 n

n i=1

log(ˆsmˆ(y_i|x_i)) + pen( ˆm)≤ inf

m∈N^∗

−1 n

n i=1

log(ˆs_m(y_i|x_i)) + pen(m)

+η

for η≥0, satisfies

E(KL_n(s_ξ0,ˆsmˆ)^½_T)≤(1 +κ⁻¹) inf

m∈N^∗

sminf∈Sm

KL_n(s_ξ0, s_m) + pen(m) +η_m

+κK³√^/2qC_Mn n

1 +

A_β+ q a_Σ

2

;

whereκ is an absolute positive constant.

Proposition 4.3. Let s_ξ0,T andsˆmˆ defined as in the previous proposition. Then, E(KL_n(s_ξ0,sˆmˆ)^½_T^c)≤ e^−1/2π^q/2

A^q/_Σ²

2Knqa_πe^{−1/4(Mn2−2MnAβ)aΣ}.

4.2. Notations

To prove those two propositions, and the theorem, begin with some notations.

For any measurable functiong:R^q →R, we consider the empirical norm

g_n:=

!1 n

n i=1

g²(y_i|x_i);

its conditional expectation

E(g(Y|X)) =

R^qg(y|x)s_ξ⁰(y|x)dy;

its empirical process

P_n(g) := 1 n

n i=1

g(y_i|xi);

and its normalized process

ν_n(g) :=P_n(g)−E_X(P_n(g)) = 1 n

n i=1

"

g(y_i|x_i)−

R^q

g(y|x_i)s_ξ0(y|x_i)dy

# .

For allm∈N^∗, for all modelS_m, we defineF_m by F_m=

f_m=−log s_m

s_ξ0

, s_m∈S_m

.

LetδKL>0. For all m∈N^∗, letη_m ≥0. There exist two functions, denoted by ˆsmˆ and ¯s_m, belonging toS_m, such that

P_n(−log(ˆsmˆ))≤ inf

sm∈Sm

P_n(−log(s_m)) +η_m; KL_n(s_ξ0,¯s_m)≤ inf

sm∈Sm

KL_n(s_ξ0, s_m) +δKL. (4.1) Denote by ˆf_m:=−log

sˆm

s_ξ0

and ¯f_m:=−log ¯sm

s_ξ0

. Letη≥0 and fixm∈N^∗. We define the setM(m) by M(m) ={m∈N^∗|P_n(−log(ˆs_m)) + pen(m)≤P_n(−log(ˆs_m)) + pen(m) +η}. (4.2)

4.3. Proof of the Theorem 4.1 thanks to the Propositions 4.2 and 4.3

1

aπ, A_Σ+¹₂(|M_n|+A_β)²A²_Σ, q(|M_n|+A_β)A_Σ/2

. Assume that, for allm∈N^∗, pen(m) =λm, with

λ≥κC_Mn K

n

1 +qlog(n)

Klog(2p+ 1)

.

We derive from the two propositions that there existsκ such that, if ˆmsatisfies

−1 n

n i=1

log(ˆsmˆ(y_i|x_i)) + pen( ˆm)≤ inf

m∈N^∗

−1 n

n i=1

log(ˆs_m(y_i|x_i)) + pen(m)

+η;

then ˆsmˆ satisfies

E(KL_n(s_ξ0,sˆmˆ)) = E(KL_n(s_ξ0,sˆmˆ)^½_T) + E(KL_n(s_ξ0,sˆmˆ)^½_T^c)

≤(1 +κ⁻¹) inf

m∈N^∗

sminf∈SmKL_n(s_ξ0, s_m) + pen(m) +η_m

+κC_Mn

√n K^3/2q

1 + (A_β+ q a_Σ)²

+η +κe^−1/2π^q/2

A^q/_Σ²

2Knqa_πe^{−1/4(Mn2−2MnAβ)aΣ}.

In order to optimize this equation with respect toM_n, we considerM_nthe positive solution of the polynomial log(n)−1

4(X²−2XA_β)a_Σ = 0;

we obtainM_n=A_β+

A²_β+^{4 log(}_a ⁿ⁾

Σ and√

ne^{−1/4(Mn2−2MnAβ)aΣ} = ^√1_n. On the other hand,

C_Mn ≤

A_Σ∨ 1 a_π

"

1 +q+ 1

2 A_Σ(M_n+A_β)²

#

≤

A_Σ∨ 1 a_π

"

1 + 4(q+ 1)A_Σ

A²_β+log(n) a_Σ

#

·

We get

E(KL_n(s_ξ0,sˆmˆ)) = E(KL_n(s_ξ0,sˆmˆ)^½_T) + E(KL_n(s_ξ0,sˆmˆ)^½_T^c)

≤(1 +κ⁻¹) inf

m∈N^∗

sminf∈SmKL_n(s_ξ0, s_m) + pen(m) +η_m

+η +κ

K n

e⁻¹²π^q/2 (qA_Σ)^q/2

2qa_π

+κ K

n

A_Σ∨ 1

a_π 1 + 4(q+ 1)A_Σ

A²_β+log(n) a_Σ

×K

1 +

A_β+ q a_Σ

2

·

4.4. Proof of the Theorem 3.1

We will show that there exists η_m ≥0, andη ≥0 such that ˆs^Lasso(λ) satisfies the hypothesis of the Theo- rem4.1, which will lead to Theorem3.1.

First, let show that there exists m∈ N^∗ andη_m≥0 such that the Lasso estimator is an η_m-log-likelihood minimizer inS_m.

For allλ≥0, if m_λ=N₁^[2](ˆs(λ)), ˆ

s^Lasso(λ) = argmin

s∈S N[2]

1 (s)≤mλ

−1 n

n i=1

log(s(y_i|xi))

.

We could takeη_m= 0.

Secondly, let show that there existsη≥0 such that

−1 n

n i=1

log(ˆs^Lasso(λ)(y_i|x_i)) + pen(m_λ)≤ inf

m∈N^∗

−1 n

n i=1

log(ˆs_m(y_i|x_i)) + pen(m)

+η.

Taking pen(m_λ) =λm_λ,

−1 n

n i=1

log(ˆs^Lasso(λ)(y_i|x_i)) + pen(m_λ) =−1 n

n i=1

log(ˆs^Lasso(λ)(y_i|x_i)) +λm_λ

≤ −1 n

n i=1

log(ˆs^Lasso(λ)(y_i|x_i)) +λN₁^[2](ˆs^Lasso(λ)) +λ

≤ inf

sξ∈S

−1 n

n i=1

log(s_ξ(y_i|x_i)) +λN₁^[2](s_ξ)

+λ

≤ inf

m∈N^∗ inf

sξ∈Sm

−1 n

n i=1

log(s_ξ(y_i|xi)) +λN₁^[2](s_ξ)

+λ

≤ inf

m∈N^∗

sξinf∈Sm

−1 n

n i=1

log(s_ξ(y_i|x_i))

+λm

+λ

≤ inf

m∈N^∗

−1 n

n i=1

log(ˆs_m(y_i|x_i)) +λm

+λ;

which is exactly the goal, withη=λ. Then, according to the Theorem4.1, with ˆm=m_λ, and ˆsmˆ = ˆs^Lasso(λ), for

λ≥κ

A_Σ∨ 1

a_π 1 + 4(q+ 1)A_Σ

A²_β+log(n) a_Σ

K n

1 +qlog(n)

Klog(2p+ 1)

,

we get the oracle inequality.

5. Proof of the theorem according to T or T

5.1. Proof of the Proposition 4.2

This proposition corresponds to the main theorem according to the event T. To prove it, we need some preliminary results.

From our notations, reminded in Section4.2, we have, for all m∈N^∗ for allm ∈M(m), P_n( ˆf_m) + pen(m)≤P_n( ˆf_m) + pen(m) +η≤P_n( ¯f_m) + pen(m) +η_m+η;

E(P_n( ˆf_m)) + pen(m)≤E(P_n( ¯f_m)) + pen(m) +η_m+η+ν_n( ¯f_m)−ν_n( ˆf_m);

KL_n(s_ξ0,ˆs_m) + pen(m)≤ inf

sm∈SmKL_n(s_ξ0, s_m) +δKL+ pen(m) +η_m+η+ν_n( ¯f_m)−ν_n( ˆf_m); (5.1) thanks to the inequality (4.1).

The goal is to bound−ν_n( ˆf_m) =ν_n(−fˆ_m).

To control this term, we use the following lemma.

Lemma 5.1. Let M_n>0. Let T =

i∈{max1,...,n}

z∈{max1,...,q}|[Y_i]_z|

≤M_n

.

Let C_Mn= max 1

aπ, A_Σ+¹₂(|Mn|+A_β)²A²_Σ,^q(|Mn|+₂^Aβ)AΣ

and

Δ_m =mlog(n)

Klog(2p+ 1) + 6

1 +K

A_β+ q a_Σ

·

Then, on the eventT, for allm ∈N^∗, for allt >0, with probability greater than1−e^−t,

f_msup∈F_m

|ν_n(−f_m)| ≤ 4C_Mn

√n

9√

KqΔ_m +√ 2√

t

1 +K

A_β+ q a_Σ

Proof. Page663

From (5.1), on the eventT, for allm∈N^∗, for allm ∈M(m), for all t >0, with probability greater than 1−e^−t,

KL_n(s_ξ0,sˆ_m) + pen(m)≤ inf

sm∈SmKL_n(s_ξ0, s_m) +δKL+ pen(m) +ν_n( ¯f_m) +4C_Mn

√n

9√

KqΔ_m +√ 2√

t

1 +K

A_β+ q a_Σ

+η_m+η

≤ inf

sm∈SmKL_n(s_ξ0, s_m) + pen(m) +ν_n( ¯f_m) + 4C_Mn

√n

9√

KqΔ_m + 1 2√

K

1 +K

A_β+ q a_Σ

2

+√ Kt

+η_m+η+δKL,

the last inequality being true because 2ab≤ ^√1_Ka²+√

Kb². Letz >0 such thatt=z+m+m. On the eventT, for allm∈N, for allm∈M(m), with probability greater than 1−e^−(z+m+m),

KL_n(s_ξ⁰,ˆs_m) + pen(m)≤ inf

sm∈SmKL_n(s_ξ⁰, s_m) + pen(m) +ν_n( ¯f_m) + 4C_Mn

√n

9√

KqΔ_m + 1 2√

K

1 +K

A_β+ q a_Σ

2

+ 4C_Mn

√n √

K(z+m+m)

+η_m+η+δKL. KL_n(s_ξ0,ˆs_m)−ν_n( ¯f_m)≤ inf

sm∈Sm

KL_n(s_ξ0, s_m) + pen(m) + 4C_Mn

√n

√Km +

"

4C_Mn

√n

√K(m+ 9qΔ_m)−pen(m)

#

+4C_Mn

√n

1 2√

K

1 +K

A_β+ q a_Σ

2

+√ Kz

+η_m+η+δKL. Letκ≥1, and assume that pen(m) =λmwith

λ≥ 4C_Mn

√n

√K

1 + 9qlog(n)

Klog(2p+ 1)

Then, as

Δ_m =mlog(n)

Klog(2p+ 1) + 6

1 +K

A_β+ q a_Σ

, with

κ⁻¹=4C_Mn

√n

√K1

λ≤ 1

1 + 9qlog(n)

Klog(2p+ 1), we get that

KL_n(s_ξ0,sˆ_m)−ν_n( ¯f_m)≤ inf

sm∈Sm

KL_n(s_ξ0, s_m) + (1 +κ⁻¹) pen(m) +4C_Mn

√n 1 2√ K

1 +K

A_β+ q a_Σ

2

+4C_Mn

√n

54√ Kq

1 +K

A_β+ q a_Σ

+√

Kz

+η+δKL+η_m

≤ inf

sm∈SmKL_n(s_ξ0, s_m) + (1 +κ⁻¹) pen(m) +4C_Mn

√n

27K^3/2+27 + 1/2

√K

1 +K

A_β+ q a_Σ

2

+√ Kz

+η_m+η+δKL. Let ˆmsuch that

−1 n

n i=1

log(ˆsmˆ(y_i|x_i)) + pen( ˆm)≤ inf

m∈N^∗

−1 n

n i=1

log(ˆs_m(y_i|x_i)) + pen(m)

+η;