The breakdown bahavior of the TCLUST procedure

(1)

The breakdown behavior of the TCLUST procedure Ch. Ruwet Definition Parameters A real example Breakdown Conclusions

The breakdown behavior of the TCLUST

procedure

Joint work with L.A. García-Escudero, A. Gordaliza and A. Mayo-Iscar from the University of Valladolid (Spain)

Ch. Ruwet

University of Liège

(2)

Introduction - Simulated dataset

−20 −10 0 10 20 30

−20

−10

0

(3)

Introduction - Estimation by trimmed k -means

−20 −10 0 10 20 30

−20

−10

0

(4)

Outline

Definition of the TCLUST procedure Choice of the different parameters A real example

Breakdown behavior Conclusions

(5)

Outline

Definition of the TCLUST procedure

Choice of the different parameters A real example

(6)

Notations

k the fixed number of clusters;

α ∈ [0,1[the trimming size;

(PR) Xn= {x1, . . . ,xn} ∈ Rp a dataset that is not

concentrated on k points after removing a mass equal toα;

R0,R1, . . . ,Rk a partition of{1, . . . ,n}with|R0| = ⌊nα⌋; ϕ (·; µ, Σ)the probability density function (pdf) of the

p-variate normal distribution with meanµand

(7)

Clustering procedures based on trimming (1)

The trimmed k -means:k centers T1, . . . ,Tk that

minimize k X j=1 X i∈Rj kxi−Tjk2 (Cuesta-Albertos et al., 1997)

The trimmed determinant criterion: k centers T1, . . . ,Tk

and a p×p scatter matrix S that maximize

k X j=1 X i∈Rj logϕ xi;Tj,S

(8)

Clustering procedures based on trimming (2)

Heterogeneous clustering: k centers T1, . . . ,Tk and k

p×p scatter matrices S1, . . . ,Sk that maximize

k X j=1 X i∈Rj logϕ xi;Tj,Sj

under the constraint det(S1) = . . . =det(Sk)

(9)

TCLUST procedure E

c,k,α García-Escudero et al., 2008

k centers T1, . . . ,Tk, k p×p scatter matrices

S1, . . . ,Sk and k weights pj ∈ [0,1],j =1, . . . ,k with

Pk j=1pj =1 that maximize k X j=1 X i∈Rj log pjϕ xi;Tj,Sj

Eigenvalues-ratio restriction (ER):

Mn

mn

= maxj=1,...,kmaxl=1,...,pλl(Sj) minj=1,...,kminl=1,...,pλl(Sj)

≤c

for a constant c ≥1 and whereλl(Sj)are the

eigenvalues of Sj, l =1, . . . ,p and j =1, . . . ,k . Θc = n θ ={pj}kj=1, {Tj}kj=1, {Sj}kj=1 :(ER) is OK o

(10)

An

R

package

> library(tclust) > tclust(data, k = 3 , alpha = 0.05, restr = "eigen", restr.fact = 12, equal.weights = FALSE)

restris the type of restriction to be applied: "eigen"

(default), "deter" and "sigma"

restr.factis the constant c that constrains the

allowed differences among group scatters

equal.weightsleads to a model without estimation

(11)

Example - Simulated dataset

−20 −10 0 10 20 30

−20

−10

0

(12)

Example - Trimmed k -means

restr = "eigen", restr.fact = 1, equal.weights = TRUE −20 −10 0 10 20 30 −20 −10 0 10

(13)

Example - Trimmed determinant criterion

restr = "sigma", restr.fact = 1, equal.weights = TRUE −20 −10 0 10 20 30 −20 −10 0 10

(14)

Example - Heterogeneous clustering

restr = "deter", restr.fact = 1, equal.weights = TRUE −20 −10 0 10 20 30 −20 −10 0 10

(15)

Example - TCLUST

restr = "eigen", restr.fact = 50, equal.weights = FALSE −20 −10 0 10 20 30 −20 −10 0 10

(16)

Outline

Definition of the TCLUST procedure

Choice of the different parameters

A real example Breakdown behavior Conclusions

(17)

Choice of c

The choice of c should depend on prior knowledge of type of clusters we are searching for;

Large values of c lead to rather unrestricted solutions; Small values of c yield similarly structured clusters; This constant can be viewed as a "robustness" constant.

(18)

Classification trimmed likelihood function

García-Escudero et al., 201x For fixed c≥1, For k ≥1 andα ∈ [0,1[, Lc(α,k) := max {Rj}k_j=0,θ∈Θc k X j=1 X i∈Rj log pjϕ xi;Tj,Sj ∆c(α,k) = Lc(α,k +1) − Lc(α,k) ≥0 is the "gain" achieved by increasing the number of clusters from k to k +1

k∗ should be the smallest value of k such that ∆c(α,k) ≈0, except for small values ofα α∗should be the smallest value ofαsuch that ∆c(α,k∗) ≈0 for allα ≥ α∗

(19)

Classification trimmed likelihood function

García-Escudero et al., 201x For fixed c≥1, For k ≥1 andα ∈ [0,1[, Lc(α,k) := max {Rj}k_j=0,θ∈Θc k X j=1 X i∈Rj log pjϕ xi;Tj,Sj ∆c(α,k) = Lc(α,k +1) − Lc(α,k) ≥0 is the "gain"

achieved by increasing the number of clusters from k to

k +1

k∗ should be the smallest value of k such that ∆c(α,k) ≈0, except for small values ofα α∗should be the smallest value ofαsuch that ∆c(α,k∗) ≈0 for allα ≥ α∗

(20)

Classification trimmed likelihood function

k +1

k∗ should be the smallest value of k such that ∆c(α,k) ≈0, except for small values ofα

α∗should be the smallest value ofαsuch that ∆c(α,k∗) ≈0 for allα ≥ α∗

(21)

Classification trimmed likelihood function

k +1

k∗ should be the smallest value of k such that ∆c(α,k) ≈0, except for small values ofα

α∗should be the smallest value ofαsuch that ∆c(α,k∗) ≈0 for allα ≥ α∗

(22)

The

R

package

R > ctl <- ctlcurves (data, k = 1:4, alpha = seq (0, 0.2, length = 6),restr.fact = 50) R > plot(ctl)

(23)

Example - Simulated dataset

0.00 0.05 0.10 0.15 −14000 −13000 −12000 −11000 CTL−Curves Objectiv e Function V alue α Restriction Factor = 50 1 1 1 1 1 1 1 2 2 2 2 2 2 2 3 3 3 3 3 3 3 4 4 4 4 4 4 4 5 5 5 5 5 5 5

(24)

Outline

Definition of the TCLUST procedure Choice of the different parameters

A real example

(25)

Swiss bank notes data (1)

Flury and Riedwyl, 1988

6 variables (measurements on the bank notes)

200 observations divided in 2 groups: 100 genuine and 100 forged old Swiss 1000-franc bank notes

(26)

Swiss bank notes data (2)

R > plot(ctlcurves(Swiss, k = 1:4 , alpha = seq(0,0.3,by=0.025)) 0.00 0.05 0.10 0.15 0.20 0.25 0.30 −900 −800 −700 −600 −500 −400 −300 CTL−Curves Objectiv e Function V alue α Restriction Factor = 50 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4

(27)

Swiss bank notes data (3)

R > plot(tclust(Swiss, k = 2 , alpha = 0.15, restr.fact = 50)

Classification

First discriminant coord.

Second discr

iminant coord.

(28)

Outline

Breakdown behavior

(29)

Intuition about Breakdown

Breakdown point (BDP): the fraction of outliers needed to bring the estimator to its bounds

Replacement BDP (RBDP): observations are replaced by outliers

Addition BDP (ABDP): outliers are added

Explosion of the centers ˆ

pj =0 (sign of a badly chosen k )

Implosion or explosion of the scatter matrices

Some of them : impossible due to (ER)

All of them : impossible due to existence under (PR) (García-Escudero et al., 2008)

(30)

Replacement breakdown point

Proposition

The replacement breakdown point of the TCLUST procedure satisfies the optimistic relation

RBDP≤min ⌊nα⌋ +1 n ,j=min1,...,k |Cj| n . Data dependent

Same upper bound as the trimmed k -means (García-Escudero and Gordaliza, 1999) even if we expect a smaller RBDP for the TCLUST (estimation of weights and scatters)

(31)

Ideal model of "well-clustered" data sets

Hennig, 2004

n1< . . . <nk and Ajm= {x(nj−1+1),m, . . . ,xnj,m},

j =1, . . . ,k ;

Xm =Sk_j₌₁Ajm is said to be"well k -clustered"if∃b< ∞

s.t.,∀m∈ N, (1) max 1≤j≤k_x_i max ,m,xl,m∈A j m kxi,m−xl,mk <b (2) lim m→∞_x min i,m∈A h m,xl,m∈A j m,j6=h kxi,m−xl,mk = ∞; Addition of r outliers y1,m, . . . ,yr,m: (3) lim m→∞minkyi,m−xl,mk = ∞ (4) lim m→∞mini6=l kyi,m−yl,mk = ∞.

(32)

Addition breakdown point

Proposition

Let Xm,m∈ N,be an ideal sequence of data sets inRpthat

are "well k -clustered" in clusters A1_m, . . . ,Ak_m verifying

conditions(1)and(2). The addition of r ≤ ⌊nα⌋outliers

verifying conditions(3)and(4)does not break down the

TCLUST procedure with trimming sizeα:

ABDP≥ ⌊nα⌋

n+ ⌊nα⌋.

Better than fitting mixtures of t distributions or adding a noise component in normal mixtures (Hennig, 2004).

(33)

Dissolution point

Hennig, 2008

Ec,k,α(Xn)the TCLUST clustering of Xn;

E_c∗_,_k_,α(Xn+g)the clustering of Xninduced by

Ec,k,α(Xn+g); P a partition of Xn; For C ∈ P1and D∈ P2,γ(C,D) = |C∩D| |C∪D|; A cluster C ∈ P1is dissolved inP2if max D∈P2 γ(C,D) ≤ 1 2.

(34)

Example (1)

3 4 5 6 7 3 4 5 6

(35)

Example (2)

3 4 5 6 7 3 4 5 6 C

(36)

Example (3)

P 3 4 5 6 7 3 4 5 6

(37)

Example (4)

|C∩D′| =6 and |C∪D′| =10+8 3 4 5 6 7 3 4 5 6 D’

(38)

Example (5)

|C∩D′′| =4 and |C∪D′′| =10+13 3 4 5 6 7 3 4 5 6 D’’

(39)

Example (6)

|C∩D′′′| =0 and |C∪D′′′| =10+8 3 4 5 6 7 3 4 5 6 D’’’

(40)

Example (7)

max D∈P γ(C,D) =1/3<1/2 → C is dissolved inP 3 4 5 6 7 3 4 5 6

(41)

Dissolution point

Hennig, 2008

For C∈Ec,k,α(Xn), thedissolution point of C is given by

∆(Ec,k,α,Xn,C) =min g ( g |C| +g : ∃xn+1, . . . ,xn+g : max D∈E∗ c,k,α(Xn+g) γ(C,D) ≤1/2 ) .

(42)

Intuition about the dissolution point theorem

g ≤ ⌊nα⌋

Xna dataset for which there is no high concentration in

Xn+gwhatever the g added outliers

C ∈Ec,k,α(Xn)with|C| >g

If there are g points among the trimmed observations that are fitted well enough by the TCLUST clustering, then the cluster C can not be dissolved by the addition of g outliers.

(43)

Isolation robustness

Hennig, 2008

A clustering procedure is said to beisolation robustif for any dataset Xnand for any "well-isolated" cluster C of the

partition,

C is be stable under the addition of points, i.e. for all g,

any cluster of the partition of Xn+g should not join

observations of C and Xn\C

and

there is at least one cluster in the new partition containing some observations of C.

(44)

Example

−10 −5 0 5 10 15 −10 −5 0 5 10 15 X1 X2

(45)

Choice of k and

α

0.00 0.05 0.10 0.15 0.20 −7000 −6000 −5000 −4000 −3000 CTL−Curves Objectiv e Function V alue α Restriction Factor = 12 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4

(46)

k

₌

3,

_{α =}

0 .

1, c

₌

12

−10 −5 0 5 10 15 −10 −5 0 5 10 15 X1 X2

(47)

The TCLUST E

c,k,α

is not isolation robust

0 20 40 60 80 100 −100 −80 −60 −40 −20 0 X1 X2

(48)

The "2-steps" procedure E

c

is isolation robust !

0.00 0.05 0.10 0.15 0.20 −10000 −8000 −6000 −4000 CTL−Curves Objectiv e Function V alue α Restriction Factor = 12 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 5 0 20 40 60 80 100 −100 −80 −60 −40 −20 0 X1 X2

(49)

Outline

Breakdown behavior

(50)

Conclusions

A flexible clustering procedure; A completeRpackage;

A graphical tool to chose the parameters;

Good breakdown behavior under the ideal model of "well-clustered" dataset;

Isolation robustness of the "2-steps" procedure. Moreover, the influence functions (not presented here) are bounded.

(51)

References (1)

Cuesta-Albertos J.A., Gordaliza A. and Matrán C. (1997) Trimmed k-means: an attempt to robustify quantizers. Ann. Statist., 25(2):553-576

Gallegos M.T. (2002) Maximum likelihood clustering with outliers. Classification, clustering, and data analysis (Cracow, 2002), Stud. Classification Data Anal. Knowledge Organ., pages 247-255

Gallegos M.T. and Ritter G. (2005) A robust method for cluster analysis. Ann. Statist., 33(1):347-380

García-Escudero L.A. and Gordaliza A. (1999) Robustness properties of k -means and trimmed k -means. J. Amer. Statist. Assoc., 94(447):956-969

García-Escudero L.A., Gordaliza A., Matrán C. and

Mayo-Iscar A. (2008) A general trimming approach to robust cluster analysis. Ann. Statist., 36(3):1324-1345

(52)

References (2)

García-Escudero L.A., Gordaliza A., Matrán C. and Mayo-Iscar A. (201x) Exploring the number of groups in robust model-based clustering. Stat Comput

Flury B. and Riedwyl H. (1988) Multivariate Statistics. A practical approach. Chapman and Hall, London

Hennig C. (2004) Breakdown points for maximum likelihood estimators of location-scale mixtures. Ann. Statist.,

32(4):1313-1340

Hennig C. (2008) Dissolution point and isolation robustness: robustness criteria for general cluster analysis methods. J. Multivariate Anal., 99(6):1154-1176

RDevelopment Core Team (2010). R: A language and

environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, URL http://www.R-project.org/.