Bootstrapping Mixed Models

In what follows, we review some of the bootstrapping procedures for Mixed Models cate-gorized by the analogies one can make to the schemes available in the LM. First, we focus on proposals having links to the Residual Bootstrap for LM, yet accounting for the ran-dom structure that is particular to LMM. A second set of schemes relying on simulation and therefore mimicking the Wild and Parametric procedures is then described, while a third batch consisting in resampling data clusters, an analogue to the Pairs procedure, is studied at the last subsection.

1.4.1 Bootstraps based on Random Effect Predictions

If the random structure of a LMM can be decomposed as in equation (1.6), one can consider resampling random errors and Predictions of the random effects to create the bootstrap samples. This procedure, often named Random Effect Bootstrap (REB), see e.g Davison and Hinkley[1997, Chapter 3.8], requires an initial fit of the model providing parameter estimates β^b and Empirical Predictions (EP) for the Random Effects ub_r and the Residuals b = y−Xβ^b −^Pq_r=1σ_brZ_ru_br as well. Then, one can construct bootstrap samples u^∗_r and ^∗ via simple random sampling with replacement from the vectors of EP, yielding bootstrap samples of the outcome, as in:

y^∗ =Xβ^b +^Xq

r=1

σb_rZ_ru^∗_r+^∗. (1.18) In practice, it is useful to consider a supplementary step, since predictions are shrunk towards 0, implying that they are less spread than the random effects [Robinson,1991]. To illustrate this point, and following Morris [2002], let us consider the very simple example of a model with one random effect for the intercept. In this case, the expression of the Best Linear Unbiased Predictor (BLUP) is given by ub_i = _niⁿ_σ^i2σ_+φ(¯y_i −x¯^T_i β), where

¯

y_i = _n¹_i ^Pni=1ⁱ y_ij, with a varianceVar(ub_i) = _niⁿ_σⁱ2^σ+φ(1−¹_n)σ², showing two shrinkage factors with respect toσ², the variance of the RE. To counter this effect one can considerreflating the vectors of RE by dividing them with respect to an empirical estimate of their dispersion e.g. v_r =ub^T_ru_br/q_r thus ensuring that the resulting predictions have a standard variance, a suggestion by Carpenter et al. [2003]. The resulting reflated predictions u^e_r = u^b_r/√

v_r are subsequently resampled to create the bootstrap samples.

1.4. Bootstrapping Mixed Models 11 Another scheme that generalizes the RB is theTransformation Bootstrap. Write model (1.8) asy=Xβ+Σ^1/2δ, whereδ is drawn from a multivariate standard normal distribu-tion and Σ=Z∆Z^T+φI_N. In this setting, one can define predictions of the standardized random components of the model δ^b =Σ^b−1/2(y−Xβ^b) and resample with replacement on this vector to construct bootstrap samples of the residuals δ^∗ and consequentlyy^? via the relationship:

y^∗ =Xβ^b +Σ^1/2

bθ δ^∗. (1.19)

This procedure, can be improved by the same Reflation step as for the REB to avoid the effects of shrinkage.

Outside of the framework of LMM, it is not obvious how to define a residual/prediction-based bootstrap. On one hand there is a problem of choice of the prediction method for the random effects: either Conditional Modes (CM) or Empirical Best Predictors (EBP).

On the other hand, predictions of the random effects tend to fail to reproduce the normal distribution, since they depend on raw residuals that will present discreteness for some types of outcomes (e.g. Mixed Logit). For the sake of comparison, we could define the following bootstrap procedure following a suggestion found in Shao and Tu [2012].

Consider Pearson Residuals ˆij = [yij−µˆij]/^qφv(ˆµij) centered around an empirical mean for the whole sample and concatenated into a vector ˆe_P, and a prediction of the random effects (either CM or EBP) ˆu_i subsequently concatenated in a vector ˆu. After resampling with replacement over both vectors, creating η_ij^∗ = x^T_ijβ^b +z^T_ijD

b^σ

u^∗_i and µ^∗_ij = g⁻¹(η_ij^∗), bootstrap samples of the outcome can be constructed as follows :

y^∗_ij =µ^∗_ij +^hφv^b µ^∗_ij^i1/2e^∗_ij. (1.20)

1.4.2 Bootstraps based on Distributional Assumptions

If there is enough confidence in the assumptions of the data generating processs, one can consider generating replicates of random effects and errors using the estimates of the model parameters via simulation, in the spirit of the Parametric Bootstrap for LM.

In LMM, bootstrap-samples for random effects (and sampling errors) are generated via simulation using the estimated model parameters, i.e. u^∗_i and^∗_i are drawn fromN_q(0,I_q) andN_ni(0,φI^b _ni) respectively, allowing the construction of the following bootstrap samples for the outcome vectors:

y^∗_i =X_iβ^b +Z_iD

bσu^∗_i +^∗_i. (1.21) This intuitive scheme has been used in many contexts with fair results, see e.g. Butar and Lahiri [2003], Lahiri et al. [2003], Gonz´alez-Manteiga et al. [2007] for an illustration of its use in the estimation of the uncertainty of predictions of random effects.

The extension to a non-Gaussian context is straightforward, since it is apparent that the link between estimates θ^b, predictions ub_i and the conditional CDF of the outcome is mediated through the fitted linear predictor ηb_ij, and thus the fitted mean µb_ij via the link function g. It follows that bootstrap samples of the outcomes y^∗_ij can be drawn by simulation from the conditional distribution F

bµ^∗_ij, where g(µb^∗_ij) = x^T_ijβ^b +z^T_ijD

bσu^∗_i with replicates u^∗_i drawn from a N(0,I_q). Variations to the method, such as a Wild Bootstrap procedure which does not require the multivariate normality ofu^∗_i can also be considered, see e.g. Gonz´alez-Manteiga et al. [2008].

1.4.3 Bootstraps based on Data Clusters

As we have reviewed in the formulation (Section 1.2), the observations in a LMM have a dependence structure induced by the random components of the model reducing their exchangeability, yet it is assumed that there is independence between observational units or clusters, which in return can be seen as exchangeable. Using this remark, Davison and Hinkley [1997, Chapter 3.8] describe the steps of a scheme consisting on resampling the indexes corresponding to observational units with replacement, imitating the principle behind the Pairs Bootstrap, but this time with data clusters [y_i,X_i], hence the denom-ination of Cluster Bootstrap (CB). They further propose two strategies in order to add more perturbation, the first relying on resampling observations within the selected cluster (Strategy 1, a two-stage bootstrap of sorts) and a second strategy relying on permutation (Strategy 2). Variations to the method, such as the Reverse two stage Bootstrap which also takes into account the hierarchical structure of the components of variation by invert-ing the resamplinvert-ing steps in the two-stage bootstrap have also been proposed [McCullagh, 2000]. The extension to non-Gaussian GLMM is of course, straightforward.

TheGeneralized Cluster Bootstrap(GCB) (seeField et al. 2010,Pang and Welsh 2014) is an extension of this approach to the framework of estimators resulting from a set of estimating equations, i.e. θ^b = argθ{¹_n^Pn_i=1Ψi(y_i,X_i;θ) = 0}, where Ψi denotes a Score function. In this formulation, the CB corresponds to the resampling of the Scores and the solution of the resampled estimating equations, however it can also be characterized as drawing random weights w^∗_i following a multinomial distribution M(n, p₁ = · · · =p_n = 1/n) independently of either y_i or X_i, and solving a set ofrandomly weighted estimating equations, as in :

It is possible to link this formulation with the one at the basis of theGeneralized Bootstrap for Estimating Equations of Chatterjee and Lahiri [2011], a procedure that encompasses a wide variety of bootstrap schemes by allowing these random weights to be generated by distributions other than the Multinomial, yet all fullfilling a crucial condition, namely E[w^∗_i] =Var[w^∗_i] = 1. In the context of LMM,Field et al. [2010],Pang and Welsh [2014], for example, choose weights drawn from a E(1) distribution.

1.4.4 Discussion

The theoretical properties of some of these schemes have only recently been subject of study. Field and Welsh[2007] consider a simple one-way modely_ij =µ+σu_i+_ij and focus their analysis on the ability of the bootstrap schemes to properly estimate the variance and covariance of the sums of squaresSSW, andSSB(forBetweenandWithin, respectively), since these quantities determine the asymptotic distribution of the parameters. The authors compare different bootstrap schemes on the grounds of their ability to estimate these moments under different specifications of the data generating mechanism. On this particular aspect, they find that the REB provides proper results for these variances under joint asymptotics, i.e. when n_i, n → ∞ and correcting the predictions to counter shrinkage and when the data generation mechanism is exactly represented in the bootstrap scheme (Field and Welsh 2007, Corollary 2). However, there is empirical evidence of the shortcomings of this scheme in presence of random effects with a small number of levels even with the supplementary reflation step. Intuitively, this could be due to the fact

1.5. A Random Weighted Bootstrap for GLMM 13