High breakdown inference for mixed linear models

(1)

Thesis

Reference

High breakdown inference for mixed linear models

COPT, Samuel

Abstract

Mixed linear models are used to analyze data in many settings. These models have in most cases a multivariate normal formulation. The maximum likelihood estimator (MLE) or the residual MLE (REML) are usually chosen to estimate the parameters. However, the latter are based on the strong assumption of exact multivariate normality. Welsh and Richardson (1997) have shown that these estimators are not robust to small deviations from the multivariate normality. This means, in practice, that a small proportion of data (even only one) can drive the value of the estimates on their own. We present some of the most used models in the analysis of variance. We introduce the mixed linear model formulation and see that in most cases it is possible to extract independent subvectors of observation. The structure of the covariance matrix is derived for a great variety of models. Since the model is multivariate, we propose in this thesis a high breakdown multivariate robust estimator for very general mixed linear models, that include, for example, covariates. This robust estimator belongs to the class of S-estimators (Rousseeuw and Yohai [...]

COPT, Samuel. High breakdown inference for mixed linear models. Thèse de doctorat : Univ. Genève, 2004, no. SES 563

URN : urn:nbn:ch:unige-120533

DOI : 10.13097/archive-ouverte/unige:12053

Available at:

http://archive-ouverte.unige.ch/unige:12053

Disclaimer: layout of this document may differ from the published version.

(2)

HIGH BREAKDOWN INFERENCE FOR MIXED LINEAR MODELS

Samuel Copt

Submitted for the degree of Ph.D in Econometrics and Statistics Faculty of Economics and Social Sciences

University of Geneva Switzerland

Accepted on the recommendation of : Prof. Gérard Antille, University of Geneva, Prof. Chris Field, Dalhousie University, Canada,

Prof. Elvezio Ronchetti, University of Geneva,

Prof. Maria-Pia Victoria-Feser, University of Geneva, supervisor.

Thesis number 563 Geneva, 2004

(3)

La Faculté des sciences économiques et sociales,

sur préavis du jury, a autorisé l’impression de la présente thèse, sans entendre, par là, émettre aucune opinion sur les propositions

qui s’y trouvent énoncées et qui n’engagent que la responsabilité de leur auteur.

Genève, le 2 juillet 2004

Le Doyen : Pierre ALLAN

(4)

Abstract

Mixed linear models are used to analyze data in many settings. These models have in most cases a multivariate normal formulation. The maximum likelihood estimator (M LE) or the residual M LE (REM L) are usually chosen to estimate the parameters. However, the latter are based on the strong assumption of exact multivariate normality. Welsh and Richardson (1997) have shown that these estimators are not robust to small deviations from the multivariate normality.

This means, in practice, that a small proportion of data (even only one) can drive the value of the estimates on their own.

We present some of the most used models in the analysis of variance. We introduce the mixed linear model formulation and see that in most cases it is possible to extract independent subvectors of observation. The structure of the covariance matrix is derived for a great variety of models. Since the model is multivariate, we propose in this thesis a high breakdown multivariate robust estimator for very general mixed linear models, that include, for example, covariates.

This robust estimator belongs to the class of S-estimators (Rousseeuw and Yohai 1984) from which we can derive the asymptotic properties for inference. We also use it as a diagnostic tool to detect outlying subjects. We derive the estimating equation de…ning the high breakdown estimator and we describe how it can be computed via a simple iterative algorithm. We study the behavior of the robust estimator through an extensive simulation study. It is compared to the maximum likelihood estimator under a great variety of con…guration implying di¤erent models, di¤erent contamination patterns and di¤erent samples size. We also discuss the advantages of this estimator and illustrate its performance with the analysis of four datasets.

We also consider robust inference for multivariate hypotheses as an alternative to the classical F-test by using a robust score type test statistic proposed by Heritier and Ronchetti (1994) and study its properties by means of simulations and real data analysis.

(5)

Résumé

Les modeles lin. mixtes sont utilisés pour analyser les données issues de recherches dans de nombreux domaines des sciences humaines et sociales. Pour estimer ces modèles et tester des hypothèses, nous proposons dans ce travail une approche dite robuste qui protège les analyses de bias potentiels dus à la présence dans les échantillons d’une minorité de données atypiques.

Nous partons d’une formalisation multivariee de ces modeles et proposons un estimateur robuste appartenant à la classe des estimateurs S (Rousseeuw et Yohai 1984) dont nous dérivons les fonctions de scores ainsi que les propriétés asymptotiques pour l’inférence. Nous pouvons également l’utiliser comme outil de diagnostique a…n de pouvoir détecter de potentielles valeurs extrêmes. Les avantages de cet estimateur ainsi que son comportement sont illustrés au travers d’une étude de simulation ainsi que par l’analyse de quatre jeux de données réels.

Basée sur cet estimateur, une partie inférence robuste est également développée. Nous proposons une alternative robuste au test classique de Fisher. Il s’agit d’un test du score robuste proposé par Hériter et Ronchetti (1994). Le comportement de ce test est étudié aux moyens de simulations ainsi que de jeux de donnée réels.

(6)

Aknowledgments

I would like to express my gratitude to Prof. Maria-Pia Victoria-Feser for her valuable suggestions and her generous guidance throughout the course of this research. Her con…dence and her encouragements have made this work possible.

I am also grateful to Prof. Gérard Antille, Prof. Chris Field and Prof. Elvezio Ronchetti for their helpful comments and for taking the time and e¤ort to ensure my work was of quality.

Their valuable suggestions improved the clarity of presentation.

My thanks also go to my friends and colleagues of the Faculty of Economics and Social Sciences of the University of Geneva for their stimulating discussions and their moral support.

Finally, my thanks go to my wife Sonia, who deserves an award for her patience and under- standing during my PhD study and the writing of this thesis.

(7)

To my parents,

(8)

Introduction

Statistics is concerned with the variability that is inevitably present in any sets of data. The traditional method for isolating the sources of variability in a set of measurements is known as the analysis of variance. It has long been used in statistics and provides a simple and optimal description of a complete data set by means of models characterized by a certain number of parameters. Its purpose is to determine the extent to which the e¤ect of an independent variable, often called a factor, is a major component of the variability. The partition of the variability is summarized in what is known as an ANOVA table.

Historically, the original work of Fisher (1925) is at the beginning of the development of the analysis of variance. Fisher’s description of the analysis of variance methodology was based on sums of squares of di¤erences among observed means. In the recent decades, the trend has been to present many of the ideas behind the analysis of variance in terms of what are called nowadays linear models and more particularly that class of linear models which is called …xed e¤ects models. Variations among data can be studied through di¤erent classes of linear models, those known as random e¤ects models and also those called mixed e¤ects models, which are models that have mixed features of …xed and random models. In general, data analysis using mixed linear models is closely related to the classical analysis of variance, but this is not always true for linear and random e¤ects models. To have a thorough history of the development of the analysis of variance or to have a good documented bibliography on the subject, consider the excellent survey of Sahai, Khuri, and Kapadia (1985).

(12)

Considering a more standard terminology, an ANOVA model is also de…ned through an experimental design. An experimental design refers to a plan for assigning subjects to experimental conditions. Such designs involve a certain number of speci…cations such as the determination of the experimental conditions (independent variable) to be used, the measurements (dependent variables) to be recorded and the nuisance variable that must be controlled. Of course, those designs are also de…ned through the statistical hypothesis that are made about the parameters and the population from which the subjects are drawn. Given the choices made about the design and the statistical hypothesis, the model will be referred to as a …xed, random or mixed e¤ects model. In this work, the terms model and design will be used indi¤erently. For a review of the various designs used in the statistical literature see e.g. Kirk (1982).

In this work we consider more particularly designs that involve repeated measurement of the same group of individuals. Such design is called a repeated measures design. More speci…cally, when separate groups of individuals are studied, the number of individuals required is some multiple of the combined levels of each factor that is introduced. With repeated measures, rather than randomly assigning the individuals to the various conditions, the same individuals can be measured at all levels. The advantage of such studies is that di¤erences in measures cannot be attributed to individual characteristics (for example, motivation, intelligence). An- other advantage is that we need fewer individuals, the same individual being measured at all levels. In either case, the design is termed as repeated measures, because the same individuals are measured on a number of occasions corresponding to each treatment level. It is also referred to as randomized block design, with each individual designated as a “block”.

Repeated measures studies have been introduced in virtually all behavioral and social sciences:

psychology, medicine, education, sociology, political science, economics, business and industry.

There are many research hypotheses that can be tested using repeated measures designs, such as hypotheses that compare the same subjects under di¤erent treatments, or those that follow performance over time. Repeated measures designs are quite versatile and are called by many di¤erent names. For example, a one-way repeated measures ANOVA model may be known

(13)

as a one factor within-subjects ANOVA model, a treatments-by-subjects ANOVA model, or a randomized blocks ANOVA model. A two-way repeated measures ANOVA model may be referred to as a two factors within-subjects ANOVA model, a two-way ANOVA model with repeated measures on both factors, a multiple treatments-by-subjects ANOVA model. In fact, repeated measures designs are special cases of the randomized complete block design wherein each subject is considered to be a block and is observed under all treatment levels.

Actually, the …rst insight of this work was to develop a robust estimator for designs involving repeated measurements only. We will see that the results of this research are applicable not only to repeated measures models but also to other kind of models. As long as it is possible to write the corresponding models as a multivariate normal model with constrained covariance matrix, we can apply the robust method we propose. These models belong to the class of mixed linear models and include, for example, hierarchical or multilevel models (random nested models), longitudinal data (repeated measures) and others. The aim of this research is thus to investigate robust procedures for ANOVA models which can be described as multivariate normal with constrained covariance matrix and to propose new robust inference tools. In terms of robustness, most of the proposals in the …eld of ANOVA (or mixed linear models) are based on a weighted version of the corresponding log-likelihood function. Welsh and Richardson (1997) present a survey of the various robust methods developed so far.

Here we propose a di¤erent approach that we hope will improve the performance of the robust estimators especially in terms of breakdown point (see later). It starts from a reformulation of the models as multivariate normal distributions. For example, consider the one factor within- subject ANOVA model given by the following structural equation

y_ij = + _j+s_i+"_ij i= 1; ::; n j= 1; :::; l

where y_ij is an observation for subject i at treatment level j, is the grand mean, _j is a

…xed e¤ect for the jth treatment levels and ^l_j=1 j = 0,si is a random variable that explains the random e¤ect of the ith subject on the response variable y_ij and "_ij is a residual error

(14)

term. We suppose that the unobservable random variablessi and"ij are independent and have independent N(0; ²_s) and N(0; ²_")respectively. An equivalent multivariate formulation of the model is given by

y_i= +e_ls_i+"_i i= 1; :::; n

where y_i = [y_i1;:::;y_il]^T, = [ + ₁; + ₂: : : : ; + _l]^T ,e_l is a vector of ones of length l and

"i = ["i1; :::"_il]^T. Given the assumptions on si and "ij, the yi are independent multivariate

normalN( ; ) random variables with

= ²_"Il+ ²_sJl

withJl being thel l matrix of ones. For other models, the covariance matrix has always a particular structure (see Chapter 2). The problem is then reduced to the estimation of and and our aim is to propose a robust estimator for a structured or constrained . We can say that the covariance matrix is constrained to have a particular structure given the assumptions on the structural model and we will use this structure in the de…nition of our robust estimator.

Estimating the parameters in mixed linear models is only a …rst step into inference. Therefore in a second part of our research, we will also consider robust testing for comparing treatment levels or in other words the elements of the mean vector. Fisher (1925) developed the widely used F-test. Unfortunately, the F-test is known as being not robust to small deviations such as extreme observations (see for example Braun and McNeil, 1981). It is true however that the robustness of the classical F-test is still a subject of research. While it can remain stable in presence of misspeci…ed models, it can break completely in some other cases. As we will see later, there are two main reasons that a¤ect the robust properties of the F-test. To our knowledge, no formal robustness study has been done in the repeated measures settings and we will show that small amounts of extreme observations can considerably bias the decisions taken on the basis of theF-test. With this idea in mind, we adapt a robust testing procedure based on the results of Heritier and Ronchetti (1994), who propose robust versions of the classical Wald test, score type test and likelihood ratio test for general parametric models. Since those

(15)

small, we will also investigate the small sample properties of the robust tests.

This work is organized as follows. In Chapter 2 we present some of the most used models in the analysis of variance. We introduce the mixed linear models formulation and see that in most cases it is possible to extract independent subvectors of observation. The structure of the covariance matrix is derived for a great variety of models. In Chapter 3 we review both the classical and robust methods of estimation that exist so far. The estimating equation de…ning the high breakdown estimator are derived in Chapter 4. We describe how it can be computed via a simple iterative algorithm. In Chapter 5, we study the behavior of the robust estimator through an extensive simulation study. It is compared to the maximum likelihood estimator under a great variety of con…guration implying di¤erent models, di¤erent contamination patterns and di¤erent samples size. Robust inference is developed in Chapter 6 and analyzed through a simulation experiment in Chapter 7. Chapter 8 analyzes four data sets using classical and robust estimators. In Chapter 9, we summarize the questions that have been left open during the course of this work and suggest di¤erent directions for further research. Finally, Chapter 10 concludes.

(16)

Chapter 2

Models formulation

In this chapter, we review some of the most used designs in analysis of variance through the mixed linear models formulation. Indeed, the mixed linear models formulation has the advantage of being very ‡exible for complex ANOVA designs. We present the formulation of mixed linear models as multivariate normal distribution with constrained covariance matrices. We ex- plore di¤erent kinds of models and see that in each case it is always possible to extract random independent subvectors that are multivariate normal.

2.1 mixed linear models

mixed linear models can be expressed generally by the regression type equation

y=X +Z +" (2.1)

where y is the N vector of all measurements (observations), X is a N q₀ design matrix for the …xed e¤ects component , a q0-vector of unknown …xed e¤ects, Z is theN q design matrix for the random e¤ect vector and" is the N-vector of independent residual errors.

can actually be partitioned into a series ofr sub-vectors,

= [ ^T₁; ^T₂; :::; ^T_r]^T (2.2)

(17)

E(y) =X

var(") = ²_"I_N (2.3)

We suppose that for each _j

var( _j) = ²_jI_q_j (2.4)

withq_j being the number of elements in _j. Moreover

cov( _j; _jT) = 0 j6=j^T (2.5) and similarly

cov( ;") = 0 (2.6)

Using (2.4)-(2.6), the covariance structure of is

D=var( ) = 2 66 66 66 4

21I_q₁

22Iq2

. ..

2rI_q_r 3 77 77 77 5

(2.7)

Then partitioning Zin submatrices as in (2.2) i.e.

Z= [Z1;Z2; :::;Zr] with each submatrix Zj of dimensionN qj, (2.1) becomes

y=X + Pr j=1

Zj j+" (2.8)

Hence,

(18)

V=var(y) =ZDZ^T + ²_"I_N = Pr j=1

2

jZ_jZ^T_j + ²_"I_N

A useful extension of this formulation is to treat"just as another _j, say ₀, and incorporate it into (2.8) by de…ning

0 =", Z0 =I_N, and ²₀ = ²_"

and so we have

y=X + Pr j=0

Zj j (2.9)

and

V= Pr j=0

2

jZjZ^T_j (2.10)

We assume that all the q0+r+ 1 e¤ects are identi…able and concentrate on models for which we can write

V=diag( _i) (2.11)

with i = ;8i= 1; : : : ; n. For such models, we have an equivalent multivariate formulation for (2.1) which is

yi N( ; ) (2.12)

with yi the p-vector of independent observations obtained by partitioning y according to the covariance structure in (2.11) and =x withx a p q₀ matrix obtained by partitioning X according to the covariance structure in (2.11). The case in which = _i=xi , i.e. with the presence in the model of covariates, will also be discussed. We can actually write

= Xr j=0

2

jz_jz^T_j (2.13)

withz_j being the design matrices that de…ne the structure of each block ofV.

(19)

Most of the well-known models can actually be written as in (2.12) with a covariance matrix de…ned with (2.13) and we now present in detail some of them.

2.2 One factor within-subject ANOVA model

We begin with a simple repeated measures model with only one experimental factor that is usually termed as one factor within-subject ANOVA model. It is not of great interest in practice but has the advantage of being simple, easy to describe and theoretically interesting. There are actually two possible models and we will de…ne them by means of structural equations.

The …rst model, which is called Model I, expresses the response variable as a function of the

…xed experimental factor and a random factor “subject”. For the second model, Model II, the response variable is a function of the …xed experimental factor, a random factor “subject”

and an interaction random factor which represents the interaction between the subject and the experimental factor. We treat here the case where the model is balanced i.e. all y_ij for all i and j are observed.

2.2.1 Model I

The one factor within-subject ANOVA model can be written as

yij = _j+si+"ij i= 1; ::; n j = 1; :::; l (2.14) where y_ij is the response of subject i at treatment level j. There are l levels for the experimental factor (in this casep=l) and lmeasurements on n subjects are taken on the response variable. _j is a …xed e¤ect for each levelj of the experimental factor and s_i is a variable that represents the random e¤ect of theith subject on the response variable. The unobservable random variabless_iand"_ij are supposed independent and have independentN(0; ²_s)andN(0; ²_") respectively8ij.

Usually, in the literature, an alternative formulation of model (2.14) is proposed. This model can be written as

(20)

y_ij = + _j+s_i+"_ij (2.15) where is the overall mean and _j is the …xed e¤ect deviation from the overall mean for level (or treatment) j. We need ^l_j=1 j = 0. The mean _j in the structural model (2.14) is related to and _j with

j = + _j

Here we will use formulation (2.15). Given the assumption ons_i and"_ij, it follows that they_ij are jointly normally distributed with mean + j and

cov(yij; y_lk) = 8>

>>

<

>>

>:

2"+ ²_s

2s

0

i=l; j =k i=l; j 6=k otherwise

The equivalent multivariate formulation is obtained by making n l-vectors of observations yi

which we can write as

y_i= +e_ls_i+"_i i= 1; :::; n (2.16)

with = vec( + _j). The yi are then independent multivariate normal N( ; ) random variables with

= var(yi) =var(elsi+"i) =ele^T_l ²_s+ ²_"Il (2.17)

= ²_sJ_l+ ²_"I_l = 2 66 66 66 4

2"+ ²_s ²_s ²_s

2s . .. ...

... . .. ...

2s 2

"+ ²_s 3 77 77 77 5

To match the notation of (2.13) we can write

(21)

= Xr j=0

2

jz_jz^T_j = ²_"I_l+ ²_sJ_l

with z₁ =e_l andz₀=I_l. Note that can also be expressed as =x . Using the constraint

l

j=1 j = 0, the …xed e¤ect vector is de…ned as

= [ ; 1; 2; :::; _l ₁]^T (2.18)

and hence the design matrix xis

x= 2

4 e_l ₁ I_l ₁ 1 e^T_l ₁

3

5 (2.19)

Finally, in the mixed linear models formulation, we have

y = (en x) + (In e_l) +" (2.20)

= X +Z

whereyis anl-vectors of responses, "is a nl-vectors of residual errors, and is the Kronecker product. We have ₁ = (s1; : : : ; sn)^T, Z1 = In el, so that Z1Z^T₁ = (In el) (In el)^T. It follows thatE(y) =X , and that

V = var(y) = (In 1l) ²_sIl(In 1l)^T + ²_"Inl

= ²_s(In Jl) + ²_"(In Il) =In ( ²_sJl+ ²_"Il)

2.2.2 Model II

Suppose now that we have to following structural model

yij = + j +si+ ( s)ij+"ij i= 1; ::; n j= 1; :::; l (2.21)

(22)

where yij, , j and si are de…ned as before. We suppose here that there is an interaction between the random e¤ect of the model (i.e. the subject) and the …xed experimental factor. A new random variable( s)ij is thus included in the model. The unobservable random variables s_i,"_ij,( s)_ij are supposed independent and have independentN(0; ²_s),N(0; ²_")andN(0; ²_s) respectively.

As before we can use a multivariate formulation to get

yi= +e_lsi+ ( s)i+"i i= 1; :::; n (2.22)

and ( s)i = [( s)ij; :::;( s)_il]^T. The yi are then independent multivariate normal N( ; ) random variables with

= 2 66 66 66 4

2"+ ²_s+ ²_s ²_s ²_s

2s . .. ...

... . .. ...

2s 2

"+ ²_s+ ²_s 3 77 77 77 5

(2.23)

Note that it is clear from (2.23) that ²_s cannot be separated form ( ²_s+ ²_s), so that model II cannot be estimated. In other words, the interaction e¤ect is confounded with the residual errors. This is not necessary the case with models that include more …xed e¤ects.

2.3 Within-subject ANOVA models with more than one factor

As we want to use the particular covariance structure of the di¤erent models (either model I or model II) we now enlarge the analysis to more complex models. Suppose we have a structural equation following the model II but this time with two …xed e¤ects factors and with respectivelyl and g levels, that is

yijk= + j+ _k+ ( )jk+si+ ( s)ij + ( s)ik+"ijk (2.24)

(23)

withi= 1; ::; n; j= 1; :::; l and k= 1; :::; g (hence in this case p=gl). For the parameters to be identi…able, we need

l

j=1 j = 0; ^g_k=1 _k= 0 and ^l_j=1 ^g_k=1( )jk = 0:

Note that an alternative model is given by (2.24) in which the random interaction ( s)ij and ( s)_ikare omitted. The unobservable random variabless_i,"_ij,( s)_ij and( s)_ikare independent and have independent N(0; ²_s) , N(0; ²_"), N(0; ²_s) and N(0; ²_s) respectively. Using the formulation given in (2.12) we have that the y_i are multivariate normal N( ; ) with

=vec( + _j+ _k+ ( )_jk) 8i

can also be de…ned as =x . An example of the structure of xand is given below.

For the covariance matrix, de…ne

1 = (s₁; : : : ; s_n)^T Z₁=I_n e_gl

2 = (( s)₁₁;( s)₁₂; : : : ;( s)_nl)^T Z2=In Il eg 3 = (( s)₁₁;( s)₁₂; : : : ;( s)_ng)^T Z₃=In e_l Ig

so that Z2Z^T₂ =In Il Jg and Z3Z^T₃ =In Jl Ig, and therefore

= ²_sJgl+ ²_s(Il Jg) + ²_s(Jl Ig) + ²_"Igl

withz1=egl,z2=Il eg andz3= el Ig.

To get an idea of the structure of the covariance matrix and x , suppose now thatl= 2 and g= 2. Then would have the form

= 2

4 [ ₁] [ ₂] [ ₂] [ ₁]

3 5 with

(24)

1= 2

4 ²¹ ²²

2

2 2

1

3

5; ₂ = 2

4 ²³ ²⁴

2

4 2

3

3 5 and

2

1 = ²_"+ ²_s+ ²_s+ ²_s

2

2 = ²_s+ ²_s

2

3 = ²_s+ ²_s

24 = ²_s

For the …xed e¤ect vector =x , one would get

= [ ; ₁; ₁;( )₁₁]^T and

x= 2 66 66 66 4

1 1 1 1

3 77 77 77 5

Note that in this case the interaction variance ²_s and ²_s can be separated from the residual variance.

The general structure of the covariance matrix remains actually the same when the number of

…xed e¤ects increases. Take a model with three …xed e¤ect factors , and . One would get for example the (rather undigest) structural equation,

y = + + + +( ) +( ) +( ) +( ) +s+( s) +( s) +( s) +"

(25)

withi= 1; ::; n j= 1; :::; l and k= 1; :::; g; m= 1; :::; hand we deduce that

=vec( + j+ _k+ _m+ ( )_jk+ ( )jm+ ( )_km+ ( )_jkm) 8i

can also be written as =x with the structure ofx and depending of l,g and m. The covariance matrix has the form

= ²_sJ_glq+ ²_s(I_l J_gq) + ²_s(J_l I_g J_q) + ²_s(J_gl I_q) + ²_"I_glq Suppose that each factor has two levels, one gets the following covariance matrix

= 2 66 66 66 4

[ 1] [ 2] [ 3] [ 4] [ ₂] [ ₁] [ ₄] [ ₃] [ 3] [ 4] [ 1] [ 2] [ ₄] [ ₃] [ ₂] [ ₁]

3 77 77 77 5 with

1 = 2

4 ²¹ ²²

2

2 2

1

3

5; ₂ = 2

4 ²³ ²⁴

2

4 2

3

3 5; ₃ =

2

4 ²⁵ ²⁶

2

6 2

5

3 5; ₄ =

2

4 ²⁷ ²⁸

2

8 2

7

3 5

and

2

1= ²_"+ ²_s+ ²_s+ ²_s+ ²_s ²₅ = ²_s+ ²_s+ ²_s

22= ²_s+ ²_s+ ²_s ²₆ = ²_s+ ²_s

2

3= ²_s+ ²_s+ ²_s ²₇ = ²_s+ ²_s

24= ²_s+ ²_s ²₈ = ²_s

We can see that for classical repeated measures models (having only crossed within-subject factors), the block structure of the covariance matrix remains always the same.

(26)

2.4 Multilevel models

In a multilevel model or random nested model, the random factors are only nested in random ones, leading to models of the type

y_ijk= + _j+s_i+ _i(k)+"_i(k)j

with + _j; j= 1; : : : ; lthe …xed e¤ect, ands_i; i= 1; : : : ; nand _i(k); k= 1; : : : ; g nthe random e¤ects. The distributional hypothesis onsi, _i(k)and"_i(k)j areN(0; ²_s),N(0; ²)andN(0; ²_") respectively. A simple example is the case in whichgmeasures are taken on each subjectiand each experimental conditionj. With this model, we have that =eg vec( + j), j = 1; :::; l, Z₁=I_n e_gl,Z₂ =I_n e_l I_g, so that

= ²_sJ_gl+ ²(J_l I_g) + ²_"I_gl:

For example, suppose thatj= 2 and g= 2 then would have the form

= 2

4 [ 1] [ 2] [ ₂] [ ₁]

3 5 with

1= 2 4

2s+ ² + ²_" ²_s+ ²

2s+ ² ²_s+ ² + ²_"

3

5; ₂ = 2 4

2s 2 s 2s 2 s

3 5

2.5 Hierarchical models

Until now, we have only presented models in which each level of a factor is combined with every level of another factor. Hierarchical models are models where only some levels of a factor are combined with the levels of another factor. More formally, suppose that we have two treatments and with respectively l and g levels. In the language of experimental design, if each level of treatment appears with onlyone level of treatment , is said to be nested in . Experi- mental designs with one or more nested treatments are particularly well suited for research in

(27)

in which two types of instruction materials (treatment levels 1 and 2) are to be evaluated using students in four classes (treatment levels ₁, ₂, ₃ and ₄). Two classrooms are randomly assigned to one type of programmed material. For obvious reasons, all children in a particular classroom are subject to the same type of material. We assume that each classroom contains the same number of children. Each classroom _k appears with only one level of instruction material. Thus treatment is nested in .

One can also extend the models so as to include covariates. For example, we have the typ- ical experiment in which a measure is taken from n1 samples of typej = 1 and n2 samples of typej= 2, and in each sample the measure is taken ong“objects". For example, the “objects"

can be rats, the samples cages, n1 of which are given treatmentj = 1 and treatmentj = 2to the n₂ others. This type of design is called a nested design. The covariate is here a dummy variable for the type of treatment. The corresponding model can be written as

y_ijk = + Ji(j) + _j(i)+"_j(i(k))

with

J_i(j) = 8<

:

0 j= 1 1 j= 2

+ Ji(j) the …xed e¤ect and _j(i); i = 1; : : : n (n=n1+n2), with k= 1; : : : ; g n. We then have

i =e_g( + J_i(j)) =e_g (1; J_i(j)) ( ; )^T =x_i

1 = ( ₁₍₁₎; : : : ; _1(n₁₎; _2(n₁₊₁₎; : : : ; _2(n))^T,Z₁ =I_n e_g and therefore

= ²J_g+ ²_"I_g

Note that the structure of is the same as with the one factor within-subject ANOVA model, the di¤erence lies in the mean that depends on the “sample" which plays here the role of observation.

(28)

A more complicated example is a design reported by Fellner (1986) and also analyzed by Richardson and Welsh (1995) on the content of one type (j = 1) of metallic oxide measured in n1 = 18 lots and another type of metallic oxide (j = 2) measured in n2 = 13 other lots. Two samples were drawn from each lot and duplicate analyses were then performed by each of two chemists randomly selected for each sample. The model is

y_ijklm= + J_i(j) + _j(i)+ _j(i(k))+ _j(i(k(l)))+"j(i(k(l(m)))) (2.25) with + J_i(j) the …xed e¤ect and _j(i); i= 1; : : : n(n=n₁+n₂) the random e¤ect due to the lot, _j(i(k)); k= 1; : : : ;2n, the random e¤ect due to the sample and _j(i(k(l))); l = 1; : : : ;4n, the random e¤ect due to the chemist. We then have

i =e₈( + J_i(j)) =e₈ (1; J_i(j)) ( ; )^T =x_i and Z₁ =I_n e₈,Z₂ =I_n I₂ e₄,Z₃ =I_n I₄ e₂, so that

= ²J₈+ ²I₂ J₄+ ²I₄ J₂+ ²_"I₈

Thus the parameters to be estimated are the means for each type of metallic oxide and the variances respectively associated with lots, samples and chemists. Suppose that we have 248 observations. We then can maken=31 independent sub-vectors y_i of size 8. To give a better idea of the structure of each independent subvectoryi = [yi111; :::; yi222]^T, the sub-vector y1 is de…ned as

(29)

2 66 66 66 66 66 66 66 66 66 64

y₁₁₁₁ y1112

y₁₁₂₁ y1122

y₁₂₁₁ y1212

y₁₂₂₁ y1222

3 77 77 77 77 77 77 77 77 77 75

= 2 66 66 66 66 66 66 66 66 66 64

1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 0

3 77 77 77 77 77 77 77 77 77 75

2 4

3 5+

2 66 66 66 66 66 66 66 66 66 64

1 1 1 1 1 1 1 1

3 77 77 77 77 77 77 77 77 77 75

1+ 2 66 66 66 66 66 66 66 66 66 64

1 0 1 0 1 0 1 0 0 1 0 1 0 1 0 1

3 77 77 77 77 77 77 77 77 77 75

2 4 ¹¹

12

3 5+

2 66 66 66 66 66 66 66 66 66 64

1 0 0 0 1 0 0 0 0 1 0 0 0 1 0 0 0 0 1 0 0 0 1 0 0 0 0 1 0 0 0 1

3 77 77 77 77 77 77 77 77 77 75

2 66 66 66 4

111 112 121 122

3 77 77 77 5

+I₈ 2 66 66 66 66 66 66 66 66 66 64

"1111

"₁₁₁₂

"1121

"₁₁₂₂

"1211

"₁₂₁₂

"1221

"₁₂₂₂ 3 77 77 77 77 77 77 77 77 77 75

The covariance matrix is given by

var(y_i) = P3 r=1

2

rz_rz^T_r + ²_"I₈ = 2

4 ¹ ²

2 1

3 5

with

1 = 2 66 66 66 4

21 2

2 2

3 2

2 2

1 2

3 2

3 23 2

3 2

1 2

2 2

3 2

2 2

1

3 77 77 77 5

2 = 2 66 66 66 4

24 2

4 2

4 24 2

4 2

4

3 77 77 77 5

(30)

2

1 = ² + ² + ²+ ²_e

2

2 = ² + ² + ²

2

3 = ² + ²

2

4 = ²

2.6 ANCOVA Models

In psychology, ANCOVA models denote ANOVA models with covariates (like a pre-measurement).

As an example consider the model in which a covariate is added to a within-subject ANOVA model as in (2.15) for each level of the within-subject factor, i.e.

yij = + jxij + _j+si+"ij (2.26) with + jxij+ _j the …xed e¤ect andsi the random e¤ect. The parameter _j is added when it is supposed that the intercepts of the regression lines are di¤erent. Similarly, one could add the constraint of equal regression slopes with j = 8j. We then have

i = vec + _jxij + _j = [e_l;diag(x_ij);Il] ( ; ₁; : : : ; _l; ₁; : : : ; _l)^T

= x_i ;

and Z1 =In el, so that

= ²_sJ_l+ ²_"I_l:

Here, the primary goal of this research is to de…ne a robust estimator for mixed linear models.

In this chapter we have seen that it is possible to extract independent multivariate normal N( ; ) vectors of observations y_i for some of the well-known mixed linear models, so that a robust estimator based on the multivariate normal modelN( ; ) with constrained covariance

(31)

Chapter 3

Estimation of mixed linear models

3.1 Classical estimators

The purpose of analysis of variance is to test di¤erences in means for statistical signi…cance.

This is accomplished by analyzing the variance, that is, by partitioning the total variance into the component that is due to the random error and the components that are due to di¤erences between means. Another classical approach to the estimation of the variance components is the maximum likelihood approach.

Suppose that we observe a response variable y_ij described through the structural equation (2.15). The analysis of sources of variability between observations and the grand mean proceeds with estimation of each component of the structural equation. Hence, the grand mean is estimated by y = (ln) ¹ ⁿ_i=1 ^l_j=1yij. The e¤ect of each level of the treatment factor j is estimated by the di¤erence between each treatment’s level mean and the grand mean y_j y withyj =n ¹ ⁿ_i=1yij .The e¤ect of each subjectsi is estimated by the di¤erence between each subject’s mean and the grand meany_i y withy_i =l ¹ ^l_j=1y_ij . This component,between- subjects, re‡ects di¤erences among the subjects. Random errors, "ij, would be estimated by y_ij y_i, the di¤erence between a score and the subject’s mean. This component,within-subjects, re‡ects variability within each subject. The simple estimators of ²_" and ²_s are obtained by

(32)

^²_s = 1

n 1

Xn i=1

(y_i y )² 1

l^²_"; (3.1)

^²_" = 1 n(l 1)

Xn i=1

Xl j=1

(yij yi)² (3.2)

These estimators are unbiased estimators of the variance components. For other types of structural equation it is always possible to proceed with the classical computation of the various sums of squares that can become quite tedious when the complexity of the design increases.

We will not pursue this issue here, and consider the maximum likelihood to derive estimates for the various parameters.

3.1.1 Maximum likelihood estimator

As we have said, when the models become more complicated, it is more convenient to use the multivariate formulation used in mixed linear models. Recall that we can write a mixed linear models as

y=X +Z (3.3)

withZ partitioned as

Z = [Z₀;Z1;Z₂; :::;Z_r] 2 66 66 66 4

0 1

...

r

3 77 77 77 5

= Pr j=0

Zj j (3.4)

where _j is the vector for random factorj, and ₀ ="and Z0=IN. The number of level for random factor j is denoted byq_j. Recall that the random e¤ects _j have the properties

E( _j) = 0 and var( _j) = ²_iI_q_i with ² = ²_" and

(33)

cov( _j; _jT) = 0 forj6=j^T Thus

var( ) =D withD de…ned in (2.7).

Using these assumptions we know that

E(y) =X

and

V=var(y) = Pr j=0

2jZ_jZ^T_j (3.5)

Given that y N(X ;V), the likelihood function is

L=L( ;Vjy) = exp ¹²^{(y X} ⁾^T^V ¹^{(y X} ⁾ (2 )¹²^NjVj¹²

(3.6) wherejVjstands for the determinant of V. The log-likelihood

logL=l( ;Vjy) = 1

2Nlog(2 ) 1

2logjVj 1

2(y X )^TV ¹(y X ) (3.7) Recall that the unknown parameters are and ²_"; ²₁; :::; ²_r included in V. To maximize l( ;Vjy), we di¤erentiate (3.7), …rst with respect to which yields (for rules of matrix di¤erentiation, see Graybill, 1983)

l( ;Vjy)

=X^TV ¹y X^TV ¹X (3.8)

Second, di¤erentiating (3.7) with respect to ²_j with

(34)

V

2 j

=Z_jZ^T_j gives forj= 0; :::; r

l( ;Vjy)

2 j

= 1

2tr(V ¹Z_jZ^T_j) +1

2(y X )^TV ¹Z_jZ^T_jV ¹(y X ) (3.9) Equation (3.9) is a complicated function of the variance components. Hence expressions for the solutions have to be obtained numerically, usually using either an iteratively reweighted least squares or a Newton-Raphson procedure. In the mixed linear theory this problem has been extensively studied and numerous algorithms have been developed (see e.g. Jennrich and Schluchter, 1986 or Lindstrom and Bates, 1988).

3.1.2 REML in the general mixed linear models

Patterson and Thompson (1971) introduced restricted maximum likelihood estimation (REML) as a method of estimating variance components in the context of unbalanced incomplete block designs. REML is often preferred to maximum likelihood estimation because it takes into account the loss of degrees of freedom in estimating the mean and hence produces unbiased estimating equations for the variance parameters. Alternative and more general derivations of REML are given by Harville (1977).

As for the maximum likelihood estimator we …rst de…ne the REML in the case of the general mixed linear models. Rather than using the data vector y directly, REML is based on linear combinations of elements ofy, chosen in such a way that the resulting model does not contain any …xed elements. This arises from starting with a set of valuesk^Ty where the vectors k^T of size1 N are chosen so thatk^Ty= k^TX +k^TZucontains no term in , i.e. so that

k^TX =0 (3.10)

Hence

(35)

The di¤erentiation with respect to ²_j gives forj= 0; :::; r lR( ;Vjy)

2j

= 1

2tr(PZjZ^T_j) +1

2y^T PZjZ^T_jV ¹Py (3.13) with (see Kathri, 1966),

P=K(K^TVK) ¹K^T

The transformation ofyintoK^Tyis not suitable for a multivariate normal formulation. Indeed, for example, in the one factor within-subject ANOVA model, X=en Il ( = ) thenM= I_N (e_n I_l) (e_n I_l)⁺and( )⁺is the Moore-Penrose inverse. It is clear that it is not possible to recover a structure like in (2.12) from the REML formulation, so that we won’t pursue this route here. In terms of robustness, Richardson and Welsh (1994) have shown that the REML is not robust and hence have proposed a robust version (see later).

3.2 Robust estimators

Robust statistics is an extension of parametric statistics, taking into account that parametric models are only approximation of the reality. It is concerned with the behavior of statistical procedures (tests, estimators,...) under small model deviations. Moreover, robust statistics must provide statistical procedures which are reliable and reasonably e¢ cient under deviations from the assumed parametric models. We de…ne a few basic concepts developed in robust statistics that will be used in the present work. We …rst de…ne a local concept, namely the in‡uence function. It measures the asymptotic bias caused to an estimator by an in…nitesimal amount of contamination at some particular point. This local concept is com- plemented by a global notion, the breakdown point which measures the maximal proportion of contamination that an estimator can tolerate without taking arbitrarily large values. For example, Maronna (1976) showed that robust estimator based on a weighting scheme that is not redescending (no weight of zero) fails to be robust in high dimension. This happens because for such estimators, their breakdown point is at most 1=(p+ 1), p being the dimension of the data. When working in high dimension it is therefore crucial to consider high

(36)

breakdown estimators. An estimator can be robust in the in…nitesimal sense i.e. to have a bounded in‡uence function but not in the global one i.e. to have a low breakdown point.

For more details and formulae on robustness measures, see e.g. Hampel et al. (1986).

Most of the work about robustness in the variance components estimation and/or repeated measures analysis has been done through the mixed linear models. There is an extensive literature about robust analysis of variance in mixed linear models. A review of the various methods can be found in Stahel and Welsh (1997) and in Welsh and Richardson (1997). Several di¤erent robust procedures are proposed but most of them are based on the maximum likelihood principle. In fact, most of these methods propose the use of a robusti…ed log-likelihood instead of the classical log-likelihood objective function. To robustify the log-likelihood, one can for example replace the quadratic function in the log-likelihood by a slower growing one, see Huggins (1993a, 1993b) and Huggins and Staudte (1994). Another solution is to modify the estimating equations rather than the likelihood itself, see Richardson (1997). Richardson and Welsh (1995) propose two robust versions of the restricted maximum likelihood estimator in the general mixed linear models. All these methods are based on the fact that variance component models are based on an additive decomposition of variability into several components. Rocke (1983) states that this decomposition is a particular property of the variance which is not shared by other measures of spread. This is natural when the estimators are derived by modifying de…nitions of this form but other methods of de…ning estimators can be used as well. In particular, an estimator can be de…ned by the algorithm used to compute it and by modifying the algorithmic de…nition of such estimators to construct robust estimator. Rocke (1983) proposes an estimator of this type.

3.2.1 Estimation by maximizing a robusti…ed likelihood

A general method for obtaining robust estimators is to de…ne a robusti…ed likelihood by replac- ing the quadratic function in (3.7) by a slower growing one in the sense that it has bounded derivatives. In the context of mixed linear models, Huggins (1993a) and Huggins and Staudte (1994) propose using

(37)

k^TX= 0 (3.11) Harville (1977) refers tok^Ty fork^T satisfying (3.11) as being an “error contrast”: its expected value is zero,

E(k^Ty) =k^TX =0

The maximum possible number of linearly independent error contrasts k^T is N v where v = rank(X). For example, a particular set of N v linearly independent error contrast is given byK^TywhereK^T is a(N v) N matrix whose rows are anyN vlinearly independent rows k^T of the matrixM=I X(X^TX) X^T:

ForK^TX= 0, we then have the model

K^Ty N(0;K^TVK)

The REML equations can be derived from the ML equations (3.9), in which the terms are transformed. Indeed by making use of the transformation by K^T, we replace

y byK^Ty X byK^TX= 0

Zby K^TZ V by K^TVK so that tr(V ¹Z_jZ^T_j)in (3.9) becomes

tr[(K^TVK) ¹K^TZjZ^T_jK] =y^TK(K^TVK) ¹K^TZjZ^T_jK(K^TVK) ¹K^Ty (3.12) Finally, letLRbe the likelihood function of K^Tyand de…ne the log-likelihoodlR as

l_R( ;Vjy) = logL_R( ;Vjy) = 1

2(N r) log(2 ) 1

2log K^TVK 1

2(y^TK(K^TVK) ¹K^Ty)

High breakdown inference for mixed linear models

Thesis

Reference

High breakdown inference for mixed linear models

HIGH BREAKDOWN INFERENCE FOR MIXED LINEAR MODELS

Contents

Chapter 1

Introduction

Chapter 2

Models formulation

2.1 mixed linear models

2.2 One factor within-subject ANOVA model

2.3 Within-subject ANOVA models with more than one factor

2.4 Multilevel models

2.5 Hierarchical models

2.6 ANCOVA Models

Chapter 3

Estimation of mixed linear models

3.1 Classical estimators

3.2 Robust estimators