We conducted three simulation studies to evaluate the real type I error level and the statistical power of hierarchical and crossed models when data was simulated under a crossed model5. The design of the simulation study was carried out under different settings to investigate the impact on the inference of 1) the number of participants, 2) the number of items, 3) the item to residual variance ratio, 4) the distribution of the IV, 5) the distribution of the random effects, 6) the violation of assumptions of the application of the crossed model and 7) for continuous and ordinal DVs.
8.4.1 Analyses
Once the data sets were created, for each data set parameters were estimated by a hierarchical model with the subject as the random effect and by a crossed model with the item and subject as random effects.
We wanted to test if the nominal error rate of 5% was preserved when testing the significance of a variable which brings no new information to the model (once with a variable related to the other variables in the model, and once with a variable independent of the others variable (no correlation)).
Three types of variables bringing no new information to the model were tested for each data set : a level 2 subject variable (L2su), an level 2 item variable (L2it) and a level 1 variable (L1).
We compared the number of times that the parameter related to the variables of each type was significant (with a nominal level fixed at 5%). If the model and the inference procedure were correct, by definition, approximately 5% of the dataset will find a variable significant. And we calculated the power of the models to detect a variable when it is informative for the model. For further information see appendix 8.6.1.
8.4.2 Study 1 Generating the data
Simulations6 were conducted using R 2.8. Three types of variables were created : a level subject one (in figure 8.2, level 2 subject), a level item one (in figure 8.2, level 2 item) and a level measure one (in figure 8.2, level 1 response). Several variables were generated under a multivariate distribution correlated or not by level. The variables composing the dependent variables were multivariate normally distributed (the fixed part of the model).
Once the fixed effects defined, we generated the random effects related to each level. As we si-mulated under a crossed levels (subject and item levels) model, we needed a subject effect, an item effect and a residual effect, which, when the assumptions were satisfied, followed a normal distribution centered in 0 and each with its own variance.
From these variables, two dependent variables were computed : a continuous variable (y) defined by the linear combination of the variables of level 1, of level 2 subject and of level 2 item and an
5. Des simulations préliminaires ont servi de base à ces trois études. Elles ont été présentée lors des XVIIIème Journées Internationales de Psychologie Différentielle à Genève en 2008. L’article qui en a découlé se trouve dans l’annexe 19 (Iglesias et al., 2010a).
6. The code for the generation of the data and for the models estimated is available on the MAD website : http ://www.unige.ch/fapse/mad/iglesias/index.html
ordinal variable (y discretized into a 5 points Likert-type by defining 4 thresholds values : –1.5 time the standard deviation (SD) (-1.5 SD), -0.5 SD, +0.5 SD and r1.5 SD (Beal & Dawson, 2007)) of the continuous variable.
The variance and covariance of the variables used in the simulation were defined from the distribu-tion of actual variables in the job satisfacdistribu-tion field. The variables were simulated as close as possible to the believed reality ; and therefore correlated variables were tested as is usual in real variables. The amount of variance of the fixed effect of each level was equal. For further information see appendix 8.6.2.
We also wanted to test different settings ; therefore we first varied the numbers of subjects (20 and 40) and the number of item (6, 10 and 15) of the sample and the item to residual variance ratio of the random effects (1 (subject, item and residual variances equal to 4), 1/3 (subject and item variances equal to 4 and residual variance equal to 12) and 3 (subject and residual variances equal to 4 and item variance equal to 12))7.
Results
Firstly, we can see in figure 8.3 that using a hierarchical or a crossed model to test the level of significance of the parameters of a level subject variable (L2su) does not change the amount of type I error rate no matter the number of items (6, 10 and 15), nor what the item to residual variance ratio is (1, 1/3 and 3) nor ifyis continuous or ordinal. More ever, the amount of type I error tends to become closer to the threshold of 5%, when the number of subjects increased. Secondly, figure 8.3 shows that using a hierarchical model to test the level of significance of the parameters of a level item variable (L2it) or a level 1 variable (L1) is clearly worse than using a crossed model, especially when there is an increase in the number of subjects and items, and in the item to residual variance ratio. Thirdly, for the ordinal y the type I error rate was slightly lower or equal to the continue y in all conditions and for both models.
8.4.3 Study 2 Generating the data
Based on the results of the first study, we compared the real type I error rate of the two models when the target was α= 5% for the item to residual variance ratio equal to 1.
In study 2, we varied the distribution of the fixed part of the IV with three kinds of distribution : multivariate normal (as in study 1), minor multivariate asymmetrically distributed IV and strong multivariate asymmetrically distributed IV (for the fixed part the distribution of the residuals stayed normal). Finally, we varied the distribution of the random part by simulating asymmetrically residuals (the fixed part being normally distributed). This last point was a way to test the impact of the violation of the assumptions of applications for the crossed model (i.e. the normal distribution of the elements of the random part).
We wanted to test these four settings, when the elements for the random part were under the assumption of the model, denominated by CS for compound symmetry, and when violating the as-sumptions on the variance-covariance matrix of the random effect as follows : 1) variance-covariance matrix per subject not compound symmetric (denominated by NCSsu) and 2) variance-covariance matrix per item not compound symmetry (denominated by NCSit). To violate the implied variance-covariance matrix of the crossed model we used a variance-variance-covariance matrix with more than two eigenvalues. For further information see appendix 8.6.3.
7. The item, the subject and the residual variances values correspond to the variances remaining in the model after adding the explicative variables
Results
For the CS part, (i.e. when the variance-covariance matrix satisfied the condition of application to the crossed model), in figure 8.4, three major findings could be pointed out. Firstly, using a hierarchical or a crossed model when the data were simulated by a crossed model to estimate the parameters of a level subject variable, does not change the amount of type I error whatever the distribution of the IV (multivariate normal distributed, minor multivariate asymmetrically distributed and strong multivariate asymmetrically distributed) or the residual variance (residual asymmetrically distributed) is and ify is continuous or ordinal. Secondly, as before, using a hierarchical model when the data was simulated by a crossed model to estimate the parameters of a level item variable or a level 1 variable was clearly worse than using a crossed model. For the ordinal y, the error type I rate was slightly lower or equal than for the continuousyin all conditions and for both models. Thirdly, a multivariate non-normal distribution of the IV does not affect the real type I error whatever the level variable tested and the nature ofy. However, when testing level item variables or level 1 variables, a violation of the distribution of the residuals decreases the type I error rate. The same results come out for 10 and 15 items and for 40 subjects, therefore no graph was drawn.
For the NCSsu and NCSit parts, (i.e. when the variance-covariance matrices did not satisfy the condition of application for the crossed model), in figure 8.4, the results were very similar to those of the CS part. There was a low impact on the inference when violating the assumptions on the variance-covariance matrix whatever the model.
Power for Studies 1 and 2
A necessary condition to compare the power of both methods is to observe similar type I error rate. Therefore in our simulation, only the level 2 subject IV power rate could be compared. Figure 8.5 depicts the results for the number of subjects (20 and 40) and the number of items (6, 10 and 15) and the distribution of the IV and the distribution of the residual effect (multivariate normally distributed, minor multivariate asymmetrically distributed, strong multivariate asymmetrically distributed and residual asymmetrically distributed) of power and type I error rates for the level 2subject variables. In the Figure 8.5, we only presented the bloc compound symmetric, the item to residual variance ratio of 1 case andycontinuous, because no differences between the three types of variance-covariance matrix of the three item to residual variance ratios and for ordinal or continuousy were found.
With similar type I error rates, the crossed models were significantly more powerful than the hierarchical ones whatever the number of subjects and of items, the distribution of the IV or of the residual effect and ify was continuous or ordinal.
8.4.4 Study 3 Generating the data
In the third study, data even closer to the field of job satisfaction was simulated. For the item level, dichotomized variables were created. And for the measurement level the interaction of level subject and level item variables were computed.
This time, the amount of variance for the fixed effect of the item level 2 wasn’t controlled because of the dichotomized nature of the variables from the item level. For further information see appendix 8.6.4.
Firstly, we varied the numbers of subjects (20 and 40) and the number of items (4, 8 and 12) of the sample, balanced and unbalanced number of items (half-half versus a fourth-three fourth) and the item to residual variance ratio of the random effects (1, 1/3 and 3), with the IV multivariate normally distributed and random effects normally distributed. Secondly, we varied the distribution of the fixed part with two kinds of asymmetry : minor and strong multivariate asymmetrically distributed
level subject IV. And thirdly, we tested the impact of the violation of assumptions of application of the crossed model and therefore looked at the impact of a non normal distribution of the residuals (residuals asymmetrically distributed).
Results
Firstly, we can see in figure 8.6 that using a hierarchical or a crossed model to test the level of significance of the parameters for a level subject variable, did not change the level of type I error no matter what the number of items (4, 8 and 12), the distribution of the IV (multivariate normally distributed, minor multivariate asymmetrically distributed and strong multivariate asymmetrically distributed) or of the residual effect (residual asymmetrically distributed) nor if y was continuous or ordinal. Secondly, using a hierarchical model to test the level of significance of the parameters of an item level variable or a level 1 variable was clearly worse (or equal) than using a crossed model8. Thirdly, using a crossed model to test the level of significance of the parameters of an item level unbalanced dichotomous variable when the residual effect was asymmetrically distributed, the type I error rate was greater when y was continuous than when y was ordinal. And this effect increased with the number of items. And fourthly, using a hierarchical model to test the level of significance of the parameters of a level 1 variable when the IV was asymmetrically distributed (minor and strong asymmetry), the type I error rate was strongly higher than when the IV were normally distributed.
Concerning the power, only the level 2 subject IV had a similar type I error rate and thus the power rate is compared only in this case. The same conclusion as for the previous simulation could be done : the crossed models are more powerful in detecting significant IV than the hierarchical ones when the models are comparable ; therefore the results are not shown.