Selection of GLM mixtures: a new criterion for clustering purpose

HAL Id: hal-00957880

https://hal.archives-ouvertes.fr/hal-00957880

Preprint submitted on 11 Mar 2014

HAL

is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from

L’archive ouverte pluridisciplinaire

destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de

Selection of GLM mixtures: a new criterion for clustering purpose

Olivier Lopez, Milhaud Xavier

To cite this version:

Olivier Lopez, Milhaud Xavier. Selection of GLM mixtures: a new criterion for clustering purpose.

2014. �hal-00957880�

Selection of GLM mixtures: a new criterion for clustering purpose

Olivier Lopez

, Xavier Milhaud

aENSAE ParisTech, 3 Avenue Pierre Larousse, 92245 Malakoff Cedex, France

bCREST (LFA lab), 15 Boulevard Gabriel P´eri, 92245 Malakoff Cedex, France

cSorbonne Universit´es, UPMC Universit´e Paris VI, EA 3124, LSTA, 4 place Jussieu 75005 Paris, France

Abstract

Model-based clustering from finite mixtures of generalized linear models is a challenging issue which has undergone many recent developments (Hen- nig and Liao (2013), Hannah, Blei, and Powell (2011), Aitkin (1999)). In practice, the model selection step is usually performed by using AIC or BIC penalized criteria. Though, simulations show that they tend to overesti- mate the actual dimension of the model. These evidence led us to consider a new criterion close to ICL, firstly introduced in Baudry (2009). Its defi- nition requires to introduce a contrast embedding an entropic term: using concentration inequalities, we derive key properties about the convergence of the associated M-estimator. The consistency of the corresponding classi- fication criterion then follows depending on some classical requirements on the penalty term. Finally a simulation study enables to corroborate our the- oretical results, and shows the effectiveness of the method in a clustering perspective.

Keywords: Conditional classification likelihood, GLM, Model selection.

1. Introduction

The use of mixture modeling has boomed since the publication of Demp- ster, N.M., and D.B. (1977), who provided with how to estimate the pa-

∗Corresponding author

Email addresses: olivier.lopez@ensae.fr(Olivier Lopez), xavier.milhaud@ensae.fr (Xavier Milhaud)

rameters of a finite mixture model thanks to the EM algorithm. Applica- tions involving finite mixtures are usually focused on dividing explicitly the population structure into subpopulations, given that we originally do not know to which subpopulation each individual belongs. This is what is com- monly called an incomplete data problem, where the crucial point lies in determining the right number of subpopulations (or components) in order to perform a model-based clustering at the end. Meanwhile, generalized linear models (GLM) have undergone vigorous development since the early 1980’s.

Their great popularity may be due to their high flexibility and ability to con- sider both categorical and continuous risk factors (see McCullagh and Nelder (1989) and references therein). In many situations, finite mixtures of GLM can be very useful to deal with a large heterogeneity concerning the impact of those risk factors on some phenomenon in the population under study.

This could be interpreted as complex interactions between a response and some covariates.

We focus in this paper on the topic of selecting the “best” GLM mixture;

where “best” should be understood as the best trade-off between the fit and the clustering confidence (this notion will be more detailed further). In this view, we study a criterion which was originally proposed by Baudry (2009).

We derive general exponential bounds for this type of criterion, and show how these results can be applied to the selection of the order (number of components) of GLM mixtures. Selecting one model within a collection is an important statistical problem that has undergone vigorous developments in the literature. In the mixture framework, a universal solution has failed to emerge to answer the question of selecting the right order. Many articles have been dedicated to the implementation of algorithmic mixture calibra- tion techniques. Nevertheless, they often suffer from a lack of theoretical justification with respect to their convergence properties.

Based on the information theory, Oliviera-Brochado and Vitorino Martins (2005) point that there are basically two main approaches to infer the order of a mixture: hypothesis tests, and information and classification criteria.

Garel (2007) highlights the difficulty of establishing multimodality by means

of the generalized likelihood ratio test in the first approach, because the clas-

sical result according to which the test statistic is χ

-distributed is generally

not applicable where mixtures are concerned. Azais, Gassiat, and Mercadier

(2006) and Azais, Gassiat, and Mercadier (2009), building on Gassiat (2002),

offer a detailed solution to overcome this issue. When using information cri-

teria, most authors (McLachlan and Peel (2000), Fraley and Raftery (1998))

agree that the BIC criterion gives better results than AIC since it seeks to minimize the Kullback-Leibler (KL) divergence to the true distribution (Raftery (1994), Ripley (1995)). For example, the convergence of BIC to esti- mate the order of gaussian mixtures has been proved in Keribin (1999). More generally, Gassiat and Van Handen (2013) unrolls the convergence properties (towards the theoretical model) of a model selected by a likelihood-penalized criterion (where the penalty linearly depends on the model dimension) in the context of mixtures. In practice, it is a well known fact that these two cri- teria tend to overestimate the theoretical number of components, especially when the model is misspecified (Baudry (2009)). This statement is not re- ally surprising: AIC, BIC and their variations were originally proposed for model selection problems in regular statistical models, and thus their usage is not well-supported or motivated for model selection in non-standard mod- els such as mixtures. Celeux and Soromenho (1996) and Biernacki (2000) were the firsts to introduce a classification criterion to avoid this overesti- mation, namely the ICL criterion. However, ICL consistency has not been proved in the context of maximum likelihood theory. Baudry (2009) recently demonstrated the consistency of a slightly modified version of ICL under the gaussian mixture framework; but no such property has been established in the context of GLM mixtures.

The aim of this paper is two-fold: obtaining new theoretical results con-

cerning the classification criterion introduced by Baudry (2009), and develop-

ing its application in the context of GLM mixtures. In section 2, we consider

a general mixture framework and define the conditional classification likeli-

hood. By maximizing this contrast, we obtain an estimator of the mixture

parameters which differs from the maximum likelihood estimator. Indeed,

its purpose is to find a compromise between a small classification error and

a good fit to data. We obtain a general bound for the estimation error based

on concentration inequalities. In section 3, general penalized criteria are con-

sidered to select the order of the mixture, and we determine conditions for

the consistency of such procedures. The application of these results to GLM

mixtures follows in section 4. The practical behavior of this approach is then

investigated through simulation studies in section 5.

2. The maximum conditional classification likelihood estimator 2.1. Context of mixtures

Let ( Y , F ) be a measurable space and let (f

)

be a parametric family of densities on Y . The parameter θ is assumed to range over a set Θ ∈ B(R

);

where B(.) denotes the Borel sets and d ≥ 1. For any probability measure ν on (Θ, B(Θ)), the mixture density f

is defined on Y by

f

(y) = Z

Θ

f

(y) ν(dθ) = Z

Θ

f (y; θ) ν(dθ).

ν is the mixing distribution and (f

) is called the mixands. If ν has finite support, f

is a finite mixture density. In the present paper, our interest lies in discrete mixtures: for any y ∈ Y , the density f

is assumed to belong to a collection of densities M

defined as

M

= n

f(y; ψ

) =

ng

X

i=1

π

f

(y; θ

) | ψ

= (π

, ..., π

, θ

, ..., θ

) ∈ Ψ

o , (1)

where Ψ

= Π

× Θ

, with Π

⊂

(π

, ..., π

) : P

i=1

π

= 1 and π

≥ 0 and Θ

= (θ

, ..., θ

).

We will denote K

the dimension of the parameter set Ψ

. Based on i.i.d.

observations Y = (Y

, . . . , Y

), the corresponding likelihood is given by

∀ ψ

∈ Ψ

, L(ψ

; Y

, ..., Y

) = L(ψ

) = Y

j=1 ng

X

i=1

π

f

(Y

; θ

). (2) Let us note that the true density function, f

(y), may not belong to any M

in a misspecified case. The maximum likelihood estimator (M LE) ˆ ψ

is defined as the maximizer of L(ψ

) over Ψ

(in full generality, it may not be unique). Under some regularity conditions, ˆ ψ

converges towards ψ

, which is the true parameter when the model is correctly specified.

2.2. A new contrast: the conditional classification likelihood

To figure out in what the optimization by conditional classification likeli-

hood consists, we first have to define the conditional classification likelihood

itself. This function is derived from the general principle of the EM algorithm

and approximates the j

individual likelihood of the complete data (Y

, δ

),

where δ

= (δ

)

is the latent component indicator (more precisely δ

equals 1 if observation j belongs to component i, 0 otherwise).

Several authors have attempted to exploit the link between the likelihood of the observed data and the likelihood of the complete data (Celeux and Gov- aert (1992)), originally noted by Hathaway (1986). A specific term appears while writing the likelihood relatively to the complete data (Y, δ), hereafter named the classification likelihood: ∀ ψ

∈ Ψ

,

lnL_c(ψg;Y, δ) = Xn

j=1 ng

X

i=1

δ_ij ln (πif_i(Yj;θ_i))

= Xn

j=1 ng

X

i=1

δ_ij ln π_if_i(Yj;θ_i) Png

k=1π_kf_k(Y_j;θ_k)

| {z }

τi(Yj;ψg)

+

Xn

j=1

=1

z }| {

ng

X

i=1

δ_ijln

ng

X

k=1

π_kf_k(Y_j;θ_k)

!

| {z }

lnL(ψg;Y)

= Xn

j=1 ng

X

i=1

δ_ij ln

τ_i(Y_j;ψ_g)

+ lnL(ψ_g;Y) (3)

τ

(Y

; ψ

) is the a posteriori probability that observation j belongs to com- ponent i. The term that binds the two likelihoods is very close to what is commonly called the entropy:

∀ ψ

∈ Ψ

, ∀ y

∈ R

, Ent(ψ

; y

) = −

ng

X

i=1

τ

(y

; ψ

) ln

τ

(y

; ψ

) . This function results from the expectation (w.r.t.δ) taken in the first member of the right-hand term in (3), hence the “conditional classification likelihood”

denoted further by L

:

lnL_cc(ψ_g;Y) =E_δ[lnL_c(ψ_g;Y, δ)] = lnL(ψ_g;Y) + Xn

j=1 ng

X

i=1

E_δ[δ_ij|Y_j] ln τ_i(Y_j;ψ_g)

= lnL(ψg;Y)−Ent(ψg;Y), (4)

where Ent(ψ

; Y) = P

j=1

Ent(ψ

; Y

).

The entropy is maximum in case of equiprobability (τ

(Y

; ψ

) = ... = τ

(Y

; ψ

));

and minimum when one of the a posteriori probabilities is worth 1. As high-

lighted by equation (4), this term can be seen as a penalization of the observed

likelihood: the bigger the lack of confidence when making the a posteriori

classification (via the Bayes rule), the greater the penalization (and vice versa). In fact, the entropy has a zero limit when τ

tends to 0 or 1.

However, it is not differentiable at 0: consider the function h(τ

) = τ

ln τ

, then we have lim

h

(τ

) = −∞ . This will be a key point in the definition of the parameters space that is acceptable to ensure the convergence of the estimator based on the conditional classification likelihood. We must there- fore avoid the a posteriori proportions of the mixture tending to zero. Also essential is to keep in mind that the mixture model should be identifiable, see McLachlan and Peel (2000) (p.26) for a further discussion on this issue.

More generally, the L

expression enables to deduce additional constraints to be imposed on the parameters space (so that L

does not diverge) by studying its limits. Most of time, this suggests that critical situations corre- spond mainly to parameters that would not be bounded (Baudry (2009)).

Define the maximum conditional classification likelihood estimator (M L

E) ψ ˆ

= arg max

ψg∈Ψg

1 n

X

j=1

ln L

(ψ

; y

). (5) It should converge towards ψ

= arg max

E

[ln L

(ψ

, Y )].

Baudry (2009) provides us with the following example so as to catch in what ψ

differs from ψ

. Recall that the latter aims at minimizing the KL divergence between f(.; ψ

) and the theoretical distribution f

(.).

Example. f

is the normal density N (0, 1). Consider the model M =

1

2 f

(.; − µ, σ

) + 1

2 f

(.; µ, σ

) ; µ ∈ R, σ

∈ R

,

where no further condition is imposed. There is no closed-form expression for ψ

in this example (even when σ

is fixed!). However one can compute it numerically: (µ

, σ

) = (0.83, √

0.31). This means that there exists a unique maximizer of E

[ln L

(µ, σ

)] in Ψ

(up to a label switch), which is obviously different from ψ

. Indeed, ψ

= (µ

, σ

) = (0, 1), which leads to nothing else than the theoretical distribution itself.

It shows that the M L

E does not aim at recovering the theoretical distri- bution, even when contained in the model under consideration. Here, M LE has no rule to designate two suitable classes (components) for this model:

it would therefore construct the same two exactly superimposed f

(.; 0, 1).

The allocation of observations to one or any of these components would then be completely arbitrary (with probability 0.5, hence maximum entropy). In contrast, the compromise sought by the M L

E, which penalizes such exces- sive entropy, leads to find another estimator resulting in greater confidence in the assignment of observations to mixture components.

2.3. Exponential bound for the M L

E

To shorten the notation, let ψ

= ψ

and ˆ ψ

= ˆ ψ

. Define φ(ψ

; y) = ln L

(ψ

; y) − ln L

(ψ

; y), and d(ψ

, ψ

) = − E [φ(ψ

; Y )].

If the model is correctly specified, that is if f

( · ) = f ( · ; ψ

), the function d is the Kullback-Leibler divergence between f ( · ; ψ

) and f ( · ; ψ

). In full generality, this quantity will be different from the Kullback-Leibler, but will express some pseudo-distance between the parameters ψ

and ψ

. Let us observe that, by definition of ψ

, d(ψ

, ψ

) ≥ 0 for all ψ

∈ Ψ

.

The main result of this section is to provide an exponential bound for the deviation probability of the L

contrast, centered by its expectation in the case where ln L

is bounded. If the contrast is unbounded, up to some additional moment condition, the exponential bound is perturbed by a polynomial term multiplied by a constant which is a decreasing function of the sample size. As a corollary, we deduce bounds for d( ˆ ψ

, ψ

). Let us note that, in view of applying our result to GLM inference, we require to have a result which is adapted to unbounded contrasts ln L

(for many GLM distributions, the logarithm of the response density is unbounded).

To obtain the exponential bound, we first require an assumption which ensures a domination of the components f

as well as of their derivatives.

Assumption 1. Assume that Θ is a compact subset of R

. Denote by

∇

f

(y; θ

) (resp. ∇

f

(y; θ

)) the vector (resp. the matrix) of partial deriva- tives of f

with respect to each component of θ

. Assume that, for all i = 1, ..., n

and all θ ∈ Θ,

f

(y; θ) ≤ Λ ˜

(y) < ∞ ,

f

(y; θ) ≥ Λ ˜

(y) > 0,

k∇

f

(y; θ) k

≤ Λ ˜

(y),

k∇

f

(y; θ) k

≤ Λ ˜

(y),

with sup

Λ ˜

(y)˜ Λ

(y)

≤ A(y). ˜

In the case where the functions f

are not bounded, we require some mo- ment assumptions on both ˜ A(y) defined in Assumption 1 and some functions related to the contrast evaluated at the true parameter.

Assumption 2. Using the notations of Assumption 1, assume that there exists m > 0 such that

E [ ˜ A(Y )

] + E [ |∇

ln f(Y ; ψ

) |

] + E [ sup

i=1,...,ng

| g

(Y ) |

] < ∞ , where the g

(y) corresponds to the notations of Lemma B1.

We now state the main result of this section.

Theorem 1. Let

P (x; g) = P sup

ψg∈Ψg

X

j=1

ln L

(ψ

; Y

) − ln L

(ψ

; Y

) + d(ψ

, ψ

) k ψ

− ψ

k

> x

! , where Ψ

is a set of parameters such that, for all ψ

= (π

, ..., π

, θ

, ..., θ

) ∈ Ψ

, for all 1 ≤ i ≤ n

, π

≥ π

> 0. Assume that ψ

is an interior point of Ψ

. Under Assumptions 1 and 2 with m − ε ≥ 2 for some ε ≥ 0, there exists four constants A

, A

, A

and A

(depending on the parameter space Θ and on the functions f

only) such that

P (x; g) ≤ 4

exp

− A

x

n

+ exp

− A

x n

+ A

x

, for x > A

n

[ln n]

.

Proof. Let φ

(y) = {

}

kψg−ψ_g^bk

.

We can decompose φ

(y) = φ

(y) − φ

(y), where φ

(y) =

ln f(y; ψ

) − ln f (y; ψ

)

k ψ

− ψ

k , φ

(y) = Ent(ψ

, y) − Ent(ψ

, y)

k ψ

− ψ

k .

The proof consists of applying the concentration inequality of Proposition A1,

along with Proposition A2 to the classes of functions F

= { φ

: ψ

∈ Ψ

}

for l = 1, 2. To apply Proposition A2, we have to check that polynomial

bounds on the covering numbers of these two classes hold (condition (i) in

Proposition A2). This is done in the first step of the proof. Nevertheless, Proposition A1 and A2 require the boundedness of the class of functions that one considers. Therefore it can only be obtained for a truncated version of these two classes, that is F

1

for some M going to infinity at some reasonable rate, and some appropriate function F

. The application of the concentration inequality to the truncated version is performed in the second step of the proof. In a third step, the difference between the truncated version and the remainder term is considered. Finally, in a fourth step of the proof, all the results are gathered.

Step 1: covering numbers requirements.

Let A

= { π f

(y; θ) : θ ∈ Θ, π ∈ [π

, 1] } . Due to Assumption 1, a first order Taylor expansion shows that

ln f(y; ψ

) − ln f (y; ψ

) k ψ

− ψ

k

≤ n

d Λ ˜

(y) Λ ˜

(y) ,

where we recall that d is the dimension of Θ. So the class F

admits the en- velope F

(y) = ∇

ln f(y; ψ

) + n

d A(y)diam(Ψ ˜

), where diam(Ψ

) denotes the diameter of Ψ

with respect to k · k . Observe that, from Assumption 1, it follows from a second order Taylor expansion and Lemma 2.13 in Pakes and Pollard (1989) that N

(ε, A

) ≤ C

ε

, for some constants C

> 0 and V

> 0. Since F

= P

i=1

A

, Lemma 16 in Nolan and Pollard (1987) applies, so that N

(ε, F

) ≤ C

n

ε

.

For the class F

, the bound on the covering number is a consequence of Lemma B2. The assumptions of Lemma B1, required to obtain Lemma B2, clearly hold from Assumption 1, with Λ

(y) = ˜ Λ(y), Λ

(y) = ˜ Λ

(y), Λ

(y) = d Λ ˜

(y) + ˜ Λ

(y), Λ

(y) = 2

d

Λ ˜

(y). The envelope of F

is F

(y) = n

[Λ

(y)diam(Ψ

) + sup

| g

(y) | ], with Λ

(y) = CA(y)

.

Step 2: concentration inequality for truncated classes.

Introduce a constant M

> 0, and consider the classes F

= F

1

for l = 1, 2, where the functions F

are the envelope functions defined in Step 1. Observe that the covering number of F

can be bounded using the same bound as in Step 1, since truncation does not alter this bound (this can be seen as a consequence of Lemma A.1 in Einmahl and Mason (2000)). Let

P

(x; g ) = P sup

ψg∈Ψg

X

j=1

{ φ

(Y

) − E[φ

(Y )] } 1

> x

!

.

To bound this probability, we combine Proposition A1 and Proposition A2.

The requirements (ii) and (iii) of Proposition A2 hold with M = M

, σ

= E [F

(Y )

], while the requirement (i) follows from Step 1. Observe that M

can be taken large enough so that M

≥ σ (in Step 4 of the proof, we will make M

tend to infinity). Following the notations of Proposition A2, introducing a sequence of Rademacher variables (ε

)

independent from (Y

)

, we get

E

"

sup

ψg∈Ψg

X

j=1

ε

φ

(Y

)1

#

≤ C

n

[log(M

)]

, (6) where C

is a constant depending on K

, the dimension of the model.

Taking u = x(2A

)

in Proposition A1, we get, for x > 2A

C

n

[log M

]

, that the probability P

(x; g) is bounded by

P sup

ψg∈Ψg

X

j=1

{ φ

(Y

) − E[φ

(Y )] }

> A

E

"

sup

ψ∈Ψg

X

j=1

ε

φ

(Y

)1

# + u

!!

, where φ

(y) = φ

(y)1

. Hence, from Proposition A1 with σ

l

=

σ

, we get

P

(x; g) ≤ 2

exp

− C

x

n

+ exp

− C

x M

, with C

= A

[4A

σ

]

, and C

= A

[2A

]

.

Step 3: remainder term.

Define φ

(y) = φ

(y)1

. We have

X

j=1

φ

(Y

) ≤

X

j=1

F

(Y

)1

=: S

.

Hence, from Markov’s inequality, P(S

> x) ≤

E[F

(Y )

1

].

Next, from Cauchy-Schwarz inequality,

E[F

(Y )

1

] ≤ E[F

(Y )

]

P(F

(Y ) > M

)

. Again, from Markov’s inequality, P(F

(Y ) > M

) ≤

n

.

This finally leads to

P(S

> x) ≤ n

x

M

E[F

(Y )

]

E[F

(Y )

]

. (7)

Take M

= n

. Then n

M

is equal to 1 provided that k

= 2k + ε. We take k

= m, which corresponds to k = m/2 − ε/2. Next,

E[φ

(Y )]

≤ E

F

(Y )

P(F

(Y ) > M

)

≤ E[F

(Y )

]

M

E[F

(Y )

]

. Therefore, since (m − ε) ≥ 2,

X

j=1

E[φ

(Y )]

≤ E[F

(Y )

]

E[F

(Y )

]

=: C

. Hence, for x > C

,

P sup

ψg∈Ψg

X

j=1

E[φ

(Y )]

> x

!

= 0. (8)

Let

P

(x; g) = P sup

ψg∈Ψg

X

j=1

{ φ

(Y

) − E[φ

(Y )] }

> x

! . It follows from (7) and (8) that

P

(x; g) ≤ P(S

> x/2) + P sup

ψg∈Ψg

X

j=1

E[φ

(Y )]

> x/2

!

≤ C

x

, for x > C

.

Step 4: summary.

We have

P (x; g) ≤ X

l=1

P

(x/4; g) + P

(x/4; g).

From Step 2 and 3, we deduce that P (x; g ) ≤ 4

exp

− C

x

16n

+ exp

− C

x 4M

+ C

x

,

for x > max(C

, 2A

C

n

log M

). The result follows from the fact that we

can impose C

large enough so that C

≤ 2A

C

n

log M

, and from the

fact that we imposed M

= n

at Step 3.

Remark 1: in the bound of P (x; g), two terms decrease exponentially, while a third one decreases in a polynomial way. This additional term is the price to pay for considering potentially unbounded variables Y (see Gassiat (2002) and Gassiat and Van Handen (2013) for related bounds in the bounded case).

If we increase the assumptions on Y, by assuming the existence of an expo- nential moment for ˜ A(y) instead of a finite m

moment for m large enough in Assumption 2, a better bound can be obtained. This will especially be the case when one considers bounded variables Y, which lead to a bounded function ˜ A(y). In appendix Appendix C, we show how this bound can be obtained under this more restrictive assumption.

Remark 2: it is easy to see, from the proof of Theorem 1, that a similar bound holds if ln L

is replaced by the log-likelihood, and ψ

is the limit of the M LE. Indeed, the proof is divided into proving bounds for the classical log-likelihood, and for the entropy term. In this last situation, note that the restriction of the probabilities π

to values larger than π

is not required.

This restriction in Theorem 1 was imposed by the behavior of the derivative of the entropy near 0, which could explode otherwise. This problem does not appear when one only considers the log-likelihood.

2.4. Almost sure rates for the M L

E

Corollary 1. Assume that ln L

is twice differentiable with respect to ψ

, and denote by H

the Hessian matrix of E[ln L

(ψ

; Y )] evaluated at ψ

. Assume that, for some c > 0, ψ

H

ψ

> c k ψ

k

for all ψ

∈ Ψ

, where k · k

denotes the L

− norm. Then, under the assumptions of Theorem 1, for m ≥ 2 in Assumption 2 and for the norm k · k

, we have

k ψ ˆ

− ψ

k

= O

1 n

. If m > 4,

k ψ ˆ

− ψ

k

= O

[ln n]

n

.

Proof. Observe that, from a second order Taylor expansion, d( ˆ ψ

, ψ

) ≥ c k ψ ˆ

− ψ

k

. By definition of ˆ ψ

, we have

X

j=1

ln L

( ˆ ψ

; Y

) − ln L

(ψ

; Y

)

k ψ ˆ

− ψ

k

≥ 0.

Therefore, X

j=1

{ ln L

( ˆ ψ

; Y

) − ln L

(ψ

; Y

) } k ψ ˆ

− ψ

k

+ nd( ˆ ψ

, ψ

) k ψ ˆ

− ψ

k

≥ nd( ˆ ψ

, ψ

) k ψ ˆ

− ψ

k

≥ cn k ψ ˆ

− ψ

k

. Applying Theorem 1, we get, for x > A

n

[ln n]

,

P

cn k ψ ˆ

− ψ

k

> x

≤ P (x; g) ≤ 4

exp

− A

x

n

+ exp

− A

x n

+ A

x

. Define E

(u) = P(n

k ψ ˆ

− ψ

k

> u[ln n]

). We have P(E

(u)) ≤ P (x; g)

with x = ucn

[ln n]

, if u > A

. Proving the almost sure rate of Corollary 1 is done by applying the Borel-Cantelli Lemma to the sets { E

(u) }

, for some u large enough. We need to show that for some u large enough, P

n≥1

P(E

(u)) < ∞ . We have, for u > A

, X

n=1

P(E

(u)) ≤ X

n=1

4 n

+

X

n=1

4 exp − A

n

[ln n]

u +

X

n=1

A

u

c

n

[ln n]

. We see that the first sum in the right-hand side is finite provided that u >

A

. The second sum is finite if ε > 0. The third is finite if m > 4 and ε taken sufficiently small.

To prove the O

− rate of Corollary 1, we need to show that p

(u) = P(E

(u/[ln n]

)) tends to zero when u tends to infinity. Using the same arguments as before, for m ≥ 2,

p

(u) ≤ 4 exp( − A

u

) + 4 exp( − A

[ln n]

u) + 2

A

c

u

, where the right-hand side tends to zero when u tends to infinity.

Remark 3: it also follows the proof of Corollary 1 the stronger result 1

n X

j=1

{ ln L

( ˆ ψ

; Y

) − ln L

(ψ

; Y

) }

k ψ ˆ

− ψ

k + d( ˆ ψ

, ψ

)

k ψ ˆ

− ψ

k = O

[ln n]

n

. This implies

1 n

X

j=1

{ ln L

( ˆ ψ

; Y

) − ln L

(ψ

; Y

) } + d( ˆ ψ

, ψ

) = O

[ln n]n

. (9)

3. A new penalized selection criterion: I CL

As mentioned before, a crucial issue in clustering and mixture analysis is to determine the appropriate order of the mixture to correctly describe the data. Biernacki (2000) tried to circumvent the challenge faced by BIC as for selecting the right number of classes, especially in the case of a misspecified mixture model. He wanted to emulate the BIC approach by replacing the observed likelihood by the classification likelihood, and eliminate the prob- lem of overestimating the order in the mixture. This way, he expected to find a criterion that allows achieving a better compromise between the classifica- tion quality and the fit to data. This criterion, called ICL, is henceforth well suited to issues of population clustering. But a particular attention should be paid to the definition of the penalty: early works used to consider en- tropy as part of the penalty. Unfortunately no theoretical result could be demonstrated from this viewpoint, despite promising results in practical ap- plications (Biernacki et al. (2006)). Baudry (2009) then proposed to redefine the ICL criterion by combining it to the L

contrast. The penalty thus be- comes identical to that of BIC, and the estimator (M L

E) used to express the “new” ICL criterion differs from the maximum likelihood estimator. In this regard, we have in the previous section shown the strong convergence of this estimator towards the theoretical parameter of the underlying distribu- tion under particular regularity conditions. We now focus on the selection process from a finite collection of nested models M

, g = { 1, ..., G } .

3.1. Previous works on ICL criteria

The ICL criterion was defined on the same basis as the BIC criterion:

Biernacki (2000) suggests to select in the collection the model satisfying M

= arg min

Mg∈{M1,...,MG}

− max

ψg∈Ψg

ln L

(ψ

; Y, δ) + K

2 ln n .

In practice, one approximates arg max

L

(ψ

; Y, δ) by ˆ ψ

when n gets

large, which is clearly questionable since the contrast is different from the

classical likelihood. Besides, the label vector δ is not observed, so that the

Bayes rule is used on a posteriori probabilities to assign observations to each

mixture component: the predicted label is denoted ˆ δ

and also depends on

the MLE. This leads to consider the following procedure:

M^ICLa = arg min

Mg∈{M1,...,MG}

−lnL_c( ˆψ^{M LE}_g ;Y,δˆ^B) +K_g 2 lnn

= arg min

Mg∈{M1,...,MG}

−lnL( ˆψ^{M LE}_g ;Y)− Xn

j=1 ng

X

i=1

δˆ_ij^Blnτ_i(Y_j; ˆψ_g^{M LE}) +K_g 2 lnn

.

McLachlan and Peel (2000) suggest to use the a posteriori probabilities τ

(y; ˆ ψ

) instead of ˆ δ

:

M

= arg min

Mg∈{M₁,...,MG}

− ln L

( ˆ ψ

; Y, τ ( ˆ ψ

)) + K

2 ln n

= arg min

Mg∈{M₁,...,MG}

− ln L( ˆ ψ

; Y) + Ent( ˆ ψ

) + K

2 ln n

| {z }

pen^ICLb(Kg)

.

In fact, ICL

and ICL

are really different in practice only if ∀ i, τ

(Y

; ˆ ψ

) ≃ 1/n

. Some basic algebra shows that ICL

≥ ICL

: this means that ICL

penalizes to a greater extent a model whose observations allocation is un- certain than does ICL

. Biernacki (2000) and McLachlan and Peel (2000) have shown, through various simulated and real-life examples, that the ICL criterion is more robust than the BIC criterion when the model is misspeci- fied (which is often the case in reality). Granted, BIC and ICL have similar behaviors when the mixture components are distinctly separated; but ICL severely penalizes the likelihood in the reverse case, still taking into account its complexity. However, there is no clear relationship between the maxi- mum likelihood theory and the entropy. In addition, the criterion defined as such is not fully satisfactory from a theoretical viewpoint. Indeed, its prop- erties have not been proved yet: for instance it is not consistent in the sense that BIC is, because its penalty does not satisfy Nishii’s conditions (Nishii (1988)). In particular, it is not negligible in front of n:

1

n Ent(ψ

; Y)

−→

E

[Ent(ψ

; Y )] > 0

It follows that Ent(ψ

; Y) = O(n). Until very recently, there was therefore

clearly a gap between the practical interest aroused by ICL and its theoretical

justification. This was partly plugged by Baudry (2009) who introduced a

new version of ICL integrating a “BIC-type penalty”:

M

= arg min

Mg∈{M1,...,MG}

− ln L

( ˆ ψ

) + K

2 ln n .

In the context of gaussian mixtures, Baudry (2009) has rigorously shown that the number of components selected using this criterion converges weakly towards the theoretical one, but only in the bounded case.

3.2. Consistency of selection criteria

Still in the mixture modelling framework, let M

denote the model with smallest dimension K

such that E[ln L

(ψ

)] = max

E[ln L

(ψ

)].

The following theorem provides consistency properties of a class of penalized estimators. Related results can be found in Baudry (2009).

Theorem 2. Consider a collection of models (M

, ..., M

) satisfying the as- sumptions of Theorem 1.Consider a penalty function pen(M

) = K

u

, and

ˆ

g = arg max

g=1,...,G

1 n

X

j=1

ln L

(ψ

; Y

) − pen(M

)

! . Then, if m > 2 in Assumption 2 and if nu

→ ∞ , we get ∀ g 6 = g

P(ˆ g = g) = o(1).

If m > 4 in Assumption 2, there exists some constant C such that, if nu

>

C ln n, almost surely, g ˆ 6 = g for n large enough and for all g 6 = g

. Proof. Let

ε

= E

ln L

(ψ

; Y )

− E

ln L

(ψ

; Y ) . Decompose

1 n

Xn

j=1

lnL_cc( ˆψ_g∗;Y_j)−lnL_cc( ˆψ_g;Y_j) =ε_g+ 1 n

Xn

j=1

nlnL_cc(ψ_g^b∗;Y_j)−E[lnL_cc(ψ^b_g∗;Y)]o

− 1 n

Xn

j=1

nlnL_cc(ψ^b_g;Y_j)−E[lnL_cc(ψ_g^b;Y)]o

+ 1 n

Xn

j=1

nlnL_cc( ˆψ_g^∗;Y_j)−lnL_cc(ψ_g^b∗;Y_j) +d( ˆψ_g^∗, ψ^b_g∗)o

− 1 n

Xn

j=1

n

lnL_cc( ˆψ_g;Y_j)−lnL_cc(ψ^b_g;Y_j) +d( ˆψ_g, ψ_g^b)o . (10)

It follows from the remark following Corollary 1 that the last two terms in (10) are O

([ln n]n

) (or O

(n

)). We now distinguish two cases: ǫ

> 0 and ǫ

= 0.

Step 1: ǫ

> 0.

It follows from the law of iterated logarithm that

1 n

X

j=1

ln L

(ψ

; Y

) − E[ln L

(ψ

; Y )]

+ 1 n

X

j=1

ln L

(ψ

; Y

) − E[ln L

(ψ

; Y )]

= O

([ln ln n]

n

).

Note that these two terms are O

(n

) if we only focus on O

− rates.

If ˆ g = g, we have 1

n X

j=1

ln L

( ˆ ψ

; Y

) − ln L

( ˆ ψ

; Y

) − pen(g

) + pen(g) < 0. (11) However, due to the previous remarks, if we take u

= o(1), the left-hand side in (11) converges almost surely towards ε

(in probability rates, is equal to ε

+ o

(1)). This ensures that M

is almost surely not selected for n large enough (in probability rates, P(ˆ g = g) = o(1)).

Step 2: ǫ

= 0.

Since ψ

= ψ

, 1

n X

j=1

ln L

(ψ

; Y

) − E[ln L

(ψ

; Y )] = 1 n

X

j=1

ln L

(ψ

; Y

) − E[ln L

(ψ

; Y )] ,

and ε

= 0, which shows that the first three terms in (10) are zero. This leads to

1 n

X

j=1

ln L

( ˆ ψ

; Y

) − ln L

( ˆ ψ

; Y

) − pen(g

) + pen(g) ≥

u

+ O

u

+ O

n

,

since K

− K

> 1. This shows that there exists a constant C > 0 such

that, if nu

> C ln n, M

is almost surely not selected when n tends to

infinity. To obtain that P(ˆ g = g) = o(1), it is sufficient to have nu

tending

to infinity.