2.3 Proportional reduction in deviance measure
2.4.2 Alternative measures
2.4.3.3 Analysis of the results
The results are presented in the form of Tables composed of a first column giving the name of the considered measure. Then, the other columns contain the frequencies, the means (and standard deviations) of the sensitivities and specificities for the fixed part, random part and overall model, respectively. In particular, the columns freqF, sensFand specFgive respectively (a) the frequency of choosing the correct fixed part, (b) the mean, and the standard deviation in parentheses, of the sensitivities of the fixed part, and (c) the mean, and the standard deviation in parentheses, of the specificities of the fixed part. Thus, it allows for evaluating the ability of the measures to identify the correct set of fixed effects.
Then, the columns freqR, sensR and specR give respectively (a) the frequency of choosing
the correct random part, (b) the mean, and the standard deviation in parentheses, of the sensitivities of the random part, and (c) the mean, and the standard deviation in parentheses, of the specificities of the random part. Thus, it allows for evaluating the ability of the measures to identify the correct set of random effects. Finally, the columns freq, sens and spec give respectively (a) the frequency of choosing the correct model (the population model), (b) the mean, and the standard deviation in parentheses, of the sensitivities of the entire model, and (c) the mean, and the standard deviation in parentheses, of the specificities of the entire model. Thus, it allows for evaluating the ability of the measures to identify jointly the correct set of fixed effects and the correct set of random effects. With these information, we are able to distinguish the measures that are appropriate for selecting the fixed effects, those that are appropriate for selecting the random effects, and those that are appropriate for selecting both the fixed and random effects. The Tables are given for case 1 first, and then for cases 2 and 3. For each case, we give the Table for the Normal distribution, then, for the Poisson distribution, then, for the Bernoulli distribution and finally, for the Binomial distribution.
For ease of reading, we will use a notation similar to that introduced in the Tables of this Section. Thus, (a) freqF, freqR and freq are used to speak about the frequencies of choosing the correct fixed part, random part and entire model, respectively; (b) sensF, sensRand sens are used to speak about to the average sensitivity of the fixed part, random part and entire model, respectively; (c) specF, specR and spec are used to speak about to the average specificity of the fixed part, random part and entire model, respectively.
Furthermore, instead of speaking about PRDpen corresponding to the IC (e.g., mAIC or BICN) and obtained by considering the null model n0 (either n0 = 1 or n0 = 2), we replace this text by the notation PRDICpen(n0).
This Section is organized as follows. We first discuss the results obtained in case 1, in which the fixed and random parts have the same weight and second, we discuss the differences between case 1 and cases 2 and 3, in which the random, respectively fixed, part has less weight than the fixed, respectively random, part. Third, the differences between the considered values for λn? for PRDpen and the marginal IC are noted.
Case 1
In the subsequent paragraphs, we compare the performance in terms of model selection of our proposal PRDpen with that of the marginal and conditional IC. We aim at demon-strating that PRDpen is competitive with the considered IC. First, we discuss the results for the Normal distribution and second, those for the Poisson, Bernoulli and Binomial distributions.
For the Normal distribution (cf. Table 2.5), the different versions of PRDpenhave freqF, specF, freqR, specR, freq and spec that are all smaller than those of the corresponding IC (e.g., freq = 28 for PRDmAICpen (2) and freq = 75 for the mAIC). However, the different versions of PRDpen have sensR and thus sens (as sensF = 1 for all PRDpen and all the corresponding IC) that are all larger than those of the corresponding IC (e.g., sensR = 0.943 for PRDmAICpen (2) and sensR = 0.853 for the mAIC). Thus, as sensR for PRDpen is always bigger than sensR for the corresponding IC, and as specR for PRDpen is always lower than specR for the corresponding IC (e.g., specR = 0.763 for PRDmAICpen (2) and specR = 0.963 for the mAIC), PRDpen tends to select more complex random parts than those selected by the corresponding IC. The same applies for the cAIC that has sensR which is bigger than sensR for all the marginal IC (e.g., sensR = 0.912 for the cAIC and
Table 2.5: Normal distribution - case 1. (1) corresponds to the null model 1 and (2) corresponds to the null model 2. The different versions of PRDpen are computed with k = dim(θ). The columns sensF, specF, sensR, specR, sens and spec, contain the mean, and the standard deviation in parentheses, of the sensitivities and specificities.
Fixed part Random part Model
Measure freqF sensF specF freqR sensR specR freq sens spec
PRDpen(1) mAIC 68 1(0) 0.508(0.49) 66 0.947(0.123) 0.607(0.448) 3 0.987(0.031) 0.514(0.465) mAICCm 157 1(0) 0.989(0.023) 33 0.85(0.166) 0.805(0.307) 10 0.963(0.042) 0.978(0.03) BICN 186 1(0) 0.997(0.013) 9 0.742(0.14) 0.937(0.143) 2 0.935(0.035) 0.993(0.016) BICm 175 1(0) 0.994(0.017) 20 0.8(0.164) 0.87(0.238) 5 0.95(0.041) 0.986(0.022)
PRDpen(2) mAIC 99 1(0) 0.735(0.421) 96 0.943(0.126) 0.763(0.379) 28 0.986(0.031) 0.737(0.397) mAICCm 172 1(0) 0.993(0.02) 56 0.857(0.165) 0.87(0.245) 37 0.964(0.041) 0.985(0.023) BICN 196 1(0) 0.999(0.006) 15 0.718(0.121) 0.973(0.091) 11 0.93(0.03) 0.998(0.008) BICm 185 1(0) 0.996(0.013) 38 0.785(0.16) 0.935(0.166) 28 0.946(0.04) 0.993(0.017)
Drand 0 1(0) 0.004(0.063) 6 1(0) 0.03(0.171) 0 1(0) 0.006(0.06)
mrc,a 3 1(0) 0.015(0.122) 118 0.987(0.11) 0.605(0.49) 0 0.997(0.028) 0.051(0.121) crc,a 27 0.861(0.108) 0.914(0.262) 0 0.792(0.162) 0.755(0.377) 0 0.844(0.121) 0.904(0.261) mAIC 173 1(0) 0.993(0.021) 100 0.853(0.166) 0.963(0.166) 75 0.963(0.041) 0.991(0.022) mAICCm 190 1(0) 0.998(0.012) 70 0.792(0.162) 0.988(0.084) 60 0.948(0.04) 0.997(0.012) cAIC 157 0.996(0.031) 0.979(0.079) 19 0.912(0.147) 0.453(0.467) 7 0.975(0.042) 0.946(0.083)
BICN 200 1(0) 1(0) 19 0.698(0.098) 1(0) 19 0.925(0.024) 1(0)
BICm 195 1(0) 0.999(0.007) 44 0.743(0.141) 0.997(0.033) 39 0.936(0.035) 0.999(0.007)
sensR = 0.853 for the mAIC, which is the largest sensR for the different marginal IC), and specR which is even lower than specR for all the versions of PRDpen (e.g., specR = 0.453 for the cAIC and specR = 0.607 for PRDmAICpen (1), which is the smallest specR for the different versions of PRDpen). Thus, PRDpen, and even more the cAIC, tend to select models with a too complex random part, and thus have weaker performances than the marginal IC in terms of model selection in that setup. For PRDpen, it is likely due to the estimation of φ, that is necessary to compute the marginal log-likelihood under the saturated model Pmi=1ls,i( ˆφs | yi), as explained in Section 2.3.1.3. For the cAIC, it is surprising, as the cAIC was initially proposed for random effects selection. Thereby, it shows that our proposition is competitive with the existing IC.
Furthermore, for the Normal distribution (cf. Table 2.5), computing PRDpen(2) is recommended over PRDpen(1), as freq for PRDpen(2) are all higher than freq for PRDpen(1) (e.g., freq = 10 for PRDmAICCpen m(1) and freq = 37 for PRDmAICCpen m(2)). These higher freq are due to the increase of specF (e.g., specF = 0.989 for PRDmAICCpen m(1) and specF = 0.993 for PRDmAICCpen m(2)) and mainly to the increase of specR(e.g., specR = 0.805 for PRDmAICCpen m(1) and specR = 0.87 for PRDmAICCpen m(2)). Thus, the null model 2 seems especially more appropriate than the null model 1 for the random effects selection, as observed for the Poisson, Bernoulli and Binomial distributions in the subsequent para-graphs. Hence, computing PRDpen(2) rather than PRDpen(1) allows for reducing the difference in the specR between PRDpen and the corresponding IC (e.g., specR = 0.87 for PRDmAICCpen m(2) is closer to specR = 0.988 for the mAICCm than specR = 0.805 for PRDmAICCpen m(1)).
What is discussed in the two following paragraphs applies for the Poisson, Bernoulli and Binomial distributions (cf. Tables 2.6, 2.7 and 2.8). However, the values reported in the text to provide some examples all come from Table 2.6 (Poisson distribution) to facilitate the reading. For the Poisson, Bernoulli and Binomial distributions, the different versions of PRDpen(1) are equivalent in terms of model selection to the corresponding
Table 2.6: Poisson distribution, case 1. (1) corresponds to the null model 1 and (2) corresponds to the null model 2. The different versions of PRDpen are computed with k = dim(θ). The columns sensF, specF, sensR, specR, sens and spec, contain the mean, and the standard deviation in parentheses, of the sensitivities and specificities.
Fixed part Random part Model
Measure freqF sensF specF freqR sensR specR freq sens spec
PRDpen(1) mAIC 169 1(0) 0.99(0.025) 91 0.84(0.167) 0.965(0.155) 64 0.96(0.042) 0.989(0.027) mAICCm 189 1(0) 0.998(0.01) 62 0.78(0.158) 0.99(0.057) 52 0.945(0.04) 0.997(0.01)
BICN 200 1(0) 1(0) 16 0.693(0.091) 1(0) 16 0.923(0.023) 1(0)
BICm 198 1(0) 1(0.004) 33 0.723(0.126) 0.998(0.024) 31 0.931(0.031) 0.999(0.004)
PRDpen(2) mAIC 100 1(0) 0.703(0.441) 116 0.942(0.127) 0.812(0.367) 49 0.985(0.032) 0.71(0.421) mAICCm 136 0.997(0.033) 0.972(0.08) 114 0.92(0.143) 0.893(0.26) 67 0.978(0.042) 0.967(0.078) BICN 162 0.988(0.061) 0.964(0.158) 79 0.842(0.167) 0.927(0.222) 49 0.952(0.056) 0.962(0.152) BICm 150 0.998(0.024) 0.903(0.278) 100 0.89(0.157) 0.89(0.28) 65 0.971(0.042) 0.902(0.268) Drand 0 0.986(0.055) 0.065(0.247) 11 0.978(0.082) 0.12(0.326) 0 0.984(0.062) 0.068(0.247) mrc,a 155 1(0) 0.779(0.415) 18 0.225(0.419) 0.865(0.343) 0 0.806(0.105) 0.785(0.405) crc,a 9 0.994(0.035) 0.134(0.334) 27 0.99(0.057) 0.195(0.384) 0 0.993(0.04) 0.138(0.322) cAIC 160 0.986(0.055) 0.991(0.027) 50 0.822(0.167) 0.858(0.312) 21 0.945(0.054) 0.982(0.033)
IC, as explained in Section 2.3.1.3. This demonstrates that PRDpen is efficient for model selection. Furthermore, the way of computing PRDpen(2) (we evaluate τ at ˆτnull to compute the marginal log-likelihood under the null model) has the beneficial effect of identifying more often the correct random part than the corresponding IC. Indeed, freqR are all higher for PRDpen(2) than for PRDpen(1) (e.g., freqR = 16 for PRDBICpenN(1) and freqR = 79 for PRDBICpenN(2)). However, the correct fixed part is less often identified with PRDpen(2) than with PRDpen(1) (e.g., freqF = 200 for PRDBICpenN(1) and freqF = 162 for PRDBICpenN(2)). Thus, the null model 1 seems more appropriate to identify the relevant fixed effects, while the null model 2 seems more appropriate to identify the relevant random effects.
Due to the computational burden (cf. Section 2.4.2), we computed the cAIC only for the Poisson distribution in case 1 (and for the Normal distribution, which is discussed in the previous paragraphs) and the results are given in Table 2.6. freqF is equal to 160 for the cAIC and is relatively close to 160 for all the different versions of PRDpen (e.g., freqF = 150 for PRDBICpenm(2)). The performance of the cAIC and of PRDpen for selecting the correct set of fixed effects is thus similar. However, the performance of the cAIC for selecting the correct random part is lower than that of all the different versions of PRDpen(2). Indeed, freqR is equal to 50 for the cAIC and is for instance equal to 100 for PRDBICpenm(2). Thereby, the correct model is more often selected by PRDpen(2) than by the cAIC (e.g., freq = 65 for PRDBICpenm(2) and freq = 21 for the cAIC). The cAIC is obtained using the conditional log-likelihood function and is thus supposed to be more appropriate for random effects selection. However, the results obtained in this simulation study indicate that the marginal log-likelihood function (with which PRDpenis computed) is more appropriate than the conditional log-likelihood function for that purpose.
Another aim of this simulation study is to demonstrate that the existing measures of model adequacy are not appropriate for model selection and hence, that PRDpen outper-forms them. The results clearly confirm this point, as exposed in the subsequent para-graphs, in which we discuss the results for (1) the marginal indices and (2) the conditional indices.
Orelien and Edwards (2008) advised to use the marginal measures of model adequacy
Table 2.7: Bernoulli distribution, case 1. (1) corresponds to the null model 1 and (2) corresponds to the null model 2. The different versions of PRDpen are computed with k = dim(θ). The columns sensF, specF, sensR, specR, sens and spec, contain the mean, and the standard deviation in parentheses, of the sensitivities and specificities.
Fixed part Random part Model
Measure freqF sensF specF freqR sensR specR freq sens spec
PRDpen(1) mAIC 174 0.99(0.057) 0.993(0.023) 26 0.49(0.245) 0.992(0.078) 7 0.865(0.079) 0.993(0.022) mAICCm 185 0.982(0.076) 0.999(0.006) 11 0.413(0.181) 1(0) 7 0.84(0.077) 0.999(0.006)
BICN 91 0.82(0.167) 1(0.003) 1 0.338(0.053) 1(0) 0 0.7(0.127) 1(0.003)
BICm 163 0.942(0.127) 1(0.004) 2 0.36(0.102) 1(0) 0 0.796(0.102) 1(0.004)
PRDpen(2) mAIC 166 0.99(0.057) 0.98(0.103) 67 0.602(0.308) 0.993(0.074) 39 0.893(0.091) 0.981(0.097) mAICCm 183 0.983(0.073) 0.998(0.011) 37 0.487(0.263) 1(0) 28 0.859(0.086) 0.998(0.01)
BICN 84 0.808(0.165) 1(0.003) 3 0.345(0.084) 1(0) 1 0.693(0.127) 1(0.003)
BICm 154 0.927(0.138) 1(0.004) 18 0.4(0.195) 1(0) 15 0.795(0.121) 1(0.004)
Drand 0 1(0) 0(0) 8 1(0) 0.04(0.196) 0 1(0) 0.002(0.012)
mrc,a 0 1(0) 0(0) 68 1(0) 0.34(0.475) 0 1(0) 0.021(0.029)
crc,a 21 0.971(0.075) 0.298(0.452) 12 0.968(0.098) 0.162(0.366) 0 0.97(0.077) 0.29(0.438)
Table 2.8: Binomial distribution, case 1. (1) corresponds to the null model 1 and (2) corresponds to the null model 2. The different versions of PRDpen are computed with k = dim(θ). The columns sensF, specF, sensR, specR, sens and spec, contain the mean, and the standard deviation in parentheses, of the sensitivities and specificities.
Fixed part Random part Model
Measure freqF sensF specF freqR sensR specR freq sens spec
PRDpen(1) mAIC 174 1(0) 0.992(0.023) 85 0.833(0.167) 0.948(0.204) 63 0.958(0.042) 0.989(0.027) mAICCm 188 1(0) 0.997(0.01) 71 0.793(0.162) 0.985(0.107) 60 0.948(0.041) 0.997(0.012)
BICN 200 1(0) 1(0) 14 0.692(0.088) 0.998(0.024) 14 0.923(0.022) 1(0.001)
BICm 200 1(0) 1(0) 38 0.732(0.132) 0.998(0.024) 38 0.933(0.033) 1(0.001)
PRDpen(2) mAIC 148 1(0) 0.958(0.158) 105 0.89(0.157) 0.925(0.215) 68 0.973(0.039) 0.956(0.148) mAICCm 182 1(0) 0.995(0.019) 90 0.845(0.167) 0.965(0.131) 78 0.961(0.042) 0.993(0.02)
BICN 200 1(0) 1(0) 43 0.74(0.138) 0.998(0.024) 43 0.935(0.035) 1(0.001)
BICm 197 1(0) 0.999(0.009) 73 0.798(0.163) 0.987(0.087) 71 0.95(0.041) 0.998(0.01)
Drand 0 1(0) 0.004(0.063) 9 1(0) 0.045(0.208) 0 1(0) 0.007(0.06)
mrc,a 1 1(0) 0.005(0.071) 111 1(0) 0.555(0.498) 1 1(0) 0.039(0.075)
crc,a 31 0.85(0.104) 0.914(0.271) 1 0.775(0.157) 0.77(0.384) 0 0.831(0.117) 0.906(0.269)
for fixed effects selection. However, the results for the Normal, Bernoulli and Binomial distributions in Tables 2.5, 2.7 and 2.8 do not support this advice, as illustrated by the marginal rc,a. Indeed, all freqF are tiny (freqF = 3 for the Normal distribution and freqF = 1 for the Binomial distribution) or null (for the Bernoulli distribution). The reason why freqF are so small is given by sensF, sensR and sens, and by specF. sensF, sensR and sens for the marginal rc,a indicate that the selected model contains all the effective predictors and variance components, as they are all equal to 1 for the Bernoulli and Binomial distributions, or very close to 1 for the Normal distribution (sensF = 1, sensR = 0.987 and sens = 0.997). Furthermore, specF for the marginal rc,a indicates that the selected model contains all the ineffective predictors for most of the 200 generated samples, as all specF are close or equal to 0 (specF = 0.015 for the Normal distribution, specF = 0 for the Bernoulli distribution and specF = 0.005 for the Binomial distribution).
It thus means that the models with the most complex fixed part (models 5 and 15) are selected by the marginal measures for the Normal, Bernoulli and Binomial distributions.
Table 2.6 shows that for the Poisson distribution, the marginal rc,a selects 155 times the correct set of fixed effects. Its performance is however lower than that of all PRDpen(1) (e.g., freqF = 169 for PRDmAICpen (1)).
For the Normal, Poisson, Bernoulli and Binomial distributions (cf. Tables 2.5, 2.6, 2.7, 2.8), the conditional indices of model adequacy, except the conditionalrc,a, which are illustrated byDrand, mostly select the most complex model (model 15). This is the reason why, for Drand, (a) sensF, sensR and sens are all equal to 1 for the Normal, Bernoulli and Binomial distributions, or very close to 1 for the Poisson distribution (sensF = 0.986, sensR = 0.978 and sens = 0.984), and (b) specF, specR and spec are all very close to 0 (e.g., for the Bernoulli distribution in Table 2.7, spec = 0.002). Only the conditional rc,a, thanks to its adjustment for the number of fixed effects, has higher specF, specR and spec than the other measures of model adequacy (e.g., for the Bernoulli distribution in Table 2.7, spec = 0.29), meaning that the model selected by the conditionalrc,ais closer to the population model than the model selected by the other measures of model adequacy.
However, the conditional rc,a is far from being competitive, as it never identifies the population model (freq = 0). Thereby, neither the marginal nor the conditional measures of model adequacy are able to perform model selection. In comparison with PRDpen, their performance in terms of model selection is clearly lower, which demonstrates the usefulness of our proposal.
Cases 2 and 3
We saw previously that the choice of the null model impacts the performance of PRDpen in terms of model selection when fixed and random parts of the population model have the same weight (case 1). We further note in the subsequent paragraphs what is the influence of the modification of the balance between fixed and random parts on PRDpen according to the considered null model. To this end, we compare the results obtained in cases 2 (in which the random part has less weight than the fixed part) and 3 (in which the fixed part has less weight than the random part) with those obtained in case 1 (in which the fixed part and the random part have the same weight).
In case 2, the correct fixed part is more often selected by PRDpen(1) than in case 1 for the Normal and Poisson distributions (cf. Tables 2.9 and 2.10). For instance, for the Normal distribution, freqF is equal to 157 in case 1 (cf. Table 2.5) and to 165 in case 2 (cf.
Table 2.9) for PRDmAICCpen m(1). Two exceptions are noted for PRDBICpenm(1) for the Normal distribution and for PRDBICpenN(1) for the Poisson distribution, for which freqF are the same
Table 2.9: Normal distribution - case 2. (1) corresponds to the null model 1 and (2) corresponds to the null model 2. The different versions of PRDpen are computed with k = dim(θ). The columns sensF, specF, sensR, specR, sens and spec, contain the mean, and the standard deviation in parentheses, of the sensitivities and specificities.
Fixed part Random part Model
Measure freqF sensF specF freqR sensR specR freq sens spec
PRDpen(1) mAIC 103 1(0) 0.69(0.449) 46 0.893(0.156) 0.64(0.449) 1 0.973(0.039) 0.687(0.427) mAICCm 165 1(0) 0.991(0.021) 26 0.795(0.163) 0.872(0.265) 4 0.949(0.041) 0.984(0.026) BICN 195 1(0) 0.999(0.007) 1 0.712(0.114) 0.953(0.13) 0 0.928(0.029) 0.996(0.011) BICm 175 1(0) 0.994(0.016) 17 0.758(0.149) 0.927(0.171) 2 0.94(0.037) 0.99(0.019)
PRDpen(2) mAIC 35 1(0) 0.293(0.444) 141 0.995(0.041) 0.77(0.402) 17 0.999(0.01) 0.322(0.418) mAICCm 125 1(0) 0.967(0.046) 126 0.952(0.118) 0.875(0.276) 66 0.988(0.029) 0.962(0.047) BICN 188 1(0) 0.996(0.019) 105 0.872(0.163) 0.963(0.133) 94 0.968(0.041) 0.994(0.019) BICm 158 1(0) 0.941(0.208) 116 0.918(0.144) 0.908(0.236) 82 0.98(0.036) 0.939(0.199) Drand 0 0.933(0.102) 0.3(0.459) 0 0.9(0.153) 0.3(0.459) 0 0.925(0.115) 0.3(0.459)
mrc,a 8 1(0) 0.058(0.23) 116 0.962(0.19) 0.62(0.487) 0 0.99(0.047) 0.092(0.22)
crc,a 8 0.803(0.071) 0.977(0.141) 0 0.705(0.107) 0.922(0.246) 0 0.779(0.08) 0.973(0.142) mAIC 166 1(0) 0.991(0.023) 84 0.828(0.167) 0.958(0.18) 55 0.957(0.042) 0.989(0.025) mAICCm 188 1(0) 0.997(0.01) 64 0.783(0.159) 0.983(0.11) 52 0.946(0.04) 0.997(0.012) cAIC 166 1(0) 0.986(0.033) 11 0.897(0.155) 0.452(0.469) 5 0.974(0.039) 0.954(0.046)
BICN 200 1(0) 1(0) 18 0.693(0.102) 1(0) 18 0.923(0.026) 1(0)
BICm 198 1(0) 1(0.004) 32 0.722(0.124) 0.998(0.024) 30 0.93(0.031) 0.999(0.004)
as in case 1. Rather, the correct random part is less often selected by PRDpen(1) than in case 1 for the Normal and Poisson distributions, as the signal of the random part is smaller than that of the fixed part in case 2 (e.g., for the Poisson distribution, freqR = 62 in case 1 in Table 2.6 and freqR = 35 in case 2 in Table 2.10 for PRDmAICCpen m(1)).
Tables 2.9, 2.10, 2.11 and 2.12 show that, for PRDpen(2) in case 2, the decrease of signal of the random part results in smaller freqFfor the four distributions (e.g., for the Binomial distribution, freqF = 200 in case 1 in Table 2.8 and freqF = 188 in case 2 in Table 2.12 for PRDBICpenN(2)). Some exceptions are however noted for the Bernoulli distribution for PRDmAICCpen m(2) (for which freqF = 183 in both cases 1 and 2, in Tables 2.7 and 2.11), PRDBICpenN(2) and PRDBICpenm(2) (for which freqF are slightly larger in case 2 than in case 1).
Furthermore, for PRDpen(2) in case 2, the decrease of signal of the random part results in an increase of sensR for the four distributions (e.g., for the Binomial distribution, sensR = 0.74 in case 1 in Table 2.8 and sensR = 0.825 in case 2 in Table 2.12 for PRDBICpenN(2)). As a result, freqR are higher than in case 1, except for the Poisson distribution for PRDmAICpen (2), PRDmAICCpen m(2) and PRDBICpenm(2), which is due to the decrease of all specR (e.g., for the Poisson distribution, specR = 0.927 in case 1 in Table 2.6 and specR = 0.655 in case 2 in Table 2.10 for PRDBICpenN(2)). These higher freqR are explained by the decrease of signal of the random part, which results in the decrease of the variance of the random intercept (and of the variance of the random slope). Thereby, the other relevant variance components are less masked by the variance of the random intercept. For the Normal distribution, the increase of freqR are such that all the different versions of PRDpen(2), except PRDmAICpen (2) (discussed below), identify more often the correct model than the corresponding IC, while these latter were better than all the versions of PRDpen for model selection in case 1. For instance, freq is equal to 28 in case 1 (cf. Table 2.5) and to 82 in case 2 (cf. Table 2.9) for PRDBICpenm(2). In comparison, freq for the BICm is equal to 39 in case 1 and to 30 in case 2.
Thereby, when the signal of the fixed part is more important than that of the random part, PRDpen(2), except PRDmAICpen (2), outperforms all the considered IC, included the cAIC (cf. Table 2.9). Indeed, the cAIC only identifies 5 times the correct model in case 2.
Table 2.10: Poisson distribution, case 2. (1) corresponds to the null model 1 and (2) corresponds to the null model 2. The different versions of PRDpen are computed with k = dim(θ). The columns sensF, specF, sensR, specR, sens and spec, contain the mean, and the standard deviation in parentheses, of the sensitivities and specificities.
Fixed part Random part Model
Measure freqF sensF specF freqR sensR specR freq sens spec
PRDpen(1) mAIC 179 1(0) 0.993(0.022) 60 0.77(0.178) 0.968(0.162) 41 0.943(0.045) 0.992(0.023) mAICCm 197 1(0) 0.999(0.009) 35 0.72(0.155) 0.988(0.084) 32 0.93(0.039) 0.998(0.01)
BICN 200 1(0) 1(0) 4 0.618(0.135) 1(0) 4 0.905(0.034) 1(0)
BICm 200 1(0) 1(0) 19 0.67(0.149) 0.997(0.033) 19 0.918(0.037) 1(0.002)
PRDpen(2) mAIC 41 1(0) 0.343(0.465) 86 0.983(0.093) 0.552(0.469) 7 0.996(0.023) 0.356(0.44) mAICCm 97 1(0) 0.768(0.398) 85 0.967(0.12) 0.668(0.41) 24 0.992(0.03) 0.762(0.379) BICN 104 0.997(0.033) 0.697(0.448) 83 0.96(0.128) 0.655(0.425) 34 0.988(0.04) 0.695(0.431) BICm 88 0.998(0.024) 0.636(0.469) 79 0.965(0.122) 0.63(0.429) 22 0.99(0.035) 0.636(0.448) Drand 0 0.836(0.098) 0.754(0.431) 9 0.753(0.147) 0.795(0.401) 0 0.815(0.11) 0.757(0.426) mrc,a 83 1(0) 0.424(0.494) 59 0.587(0.493) 0.71(0.455) 0 0.897(0.123) 0.441(0.479) crc,a 6 0.928(0.104) 0.429(0.49) 35 0.892(0.157) 0.523(0.493) 0 0.919(0.117) 0.434(0.477)
Table 2.11: Bernoulli distribution, case 2. (1) corresponds to the null model 1 and (2) corresponds to the null model 2. The different versions of PRDpen are computed with k = dim(θ). The columns sensF, specF, sensR, specR, sens and spec, contain the mean, and the standard deviation in parentheses, of the sensitivities and specificities.
Fixed part Random part Model
Measure freqF sensF specF freqR sensR specR freq sens spec
PRDpen(1) mAIC 179 0.997(0.033) 0.99(0.072) 30 0.475(0.251) 0.993(0.074) 12 0.866(0.069) 0.991(0.068) mAICCm 187 0.987(0.065) 0.999(0.007) 11 0.405(0.18) 0.998(0.024) 6 0.841(0.069) 0.999(0.007)
BICN 105 0.842(0.167) 1(0) 1 0.337(0.047) 1(0) 1 0.715(0.126) 1(0)
BICm 167 0.948(0.121) 1(0.004) 6 0.37(0.133) 1(0) 4 0.804(0.1) 1(0.004)
PRDpen(2) mAIC 156 0.995(0.041) 0.934(0.228) 87 0.653(0.322) 0.99(0.1) 46 0.91(0.09) 0.937(0.214) mAICCm 183 0.988(0.061) 0.997(0.015) 56 0.553(0.298) 0.998(0.024) 47 0.88(0.093) 0.997(0.015)
BICN 93 0.822(0.167) 1(0) 12 0.377(0.161) 1(0) 10 0.71(0.139) 1(0)
BICm 160 0.938(0.13) 0.999(0.009) 29 0.443(0.239) 1(0) 26 0.815(0.125) 0.999(0.008) Drand 0 0.991(0.044) 0.044(0.205) 9 0.988(0.061) 0.08(0.272) 0 0.99(0.047) 0.047(0.204)
mrc,a 0 1(0) 0(0) 54 1(0) 0.27(0.445) 0 1(0) 0.017(0.027)
crc,a 15 0.943(0.097) 0.403(0.485) 19 0.938(0.13) 0.303(0.452) 0 0.942(0.1) 0.397(0.476)
Table 2.12: Binomial distribution, case 2. (1) corresponds to the null model 1 and (2) corresponds to the null model 2. The different versions of PRDpen are computed with k = dim(θ). The columns sensF, specF, sensR, specR, sens and spec, contain the mean, and the standard deviation in parentheses, of the sensitivities and specificities.
Fixed part Random part Model
Measure freqF sensF specF freqR sensR specR freq sens spec
PRDpen(1) mAIC 177 1(0) 0.993(0.021) 81 0.828(0.167) 0.947(0.204) 60 0.957(0.042) 0.991(0.022) mAICCm 194 1(0) 0.999(0.007) 61 0.773(0.156) 0.995(0.041) 55 0.943(0.039) 0.998(0.007)
BICN 200 1(0) 1(0) 15 0.692(0.094) 0.998(0.024) 15 0.923(0.024) 1(0.001)
BICm 200 1(0) 1(0) 34 0.727(0.128) 0.997(0.033) 34 0.932(0.032) 1(0.002)
PRDpen(2) mAIC 94 1(0) 0.785(0.383) 117 0.962(0.107) 0.807(0.348) 41 0.99(0.027) 0.787(0.363) mAICCm 157 1(0) 0.985(0.032) 98 0.913(0.147) 0.887(0.235) 67 0.978(0.037) 0.979(0.033) BICN 188 1(0) 0.997(0.012) 77 0.825(0.167) 0.97(0.096) 67 0.956(0.042) 0.995(0.013) BICm 179 1(0) 0.994(0.019) 85 0.868(0.163) 0.933(0.157) 69 0.967(0.041) 0.99(0.02) Drand 0 0.993(0.038) 0.03(0.171) 5 0.99(0.057) 0.055(0.229) 0 0.993(0.043) 0.032(0.171)
mrc,a 2 1(0) 0.014(0.118) 109 1(0) 0.545(0.499) 1 1(0) 0.047(0.113)
crc,a 20 0.819(0.087) 0.967(0.171) 0 0.728(0.13) 0.862(0.323) 0 0.796(0.097) 0.96(0.171)
Table 2.13: Normal distribution - case 3. (1) corresponds to the null model 1 and (2) corresponds to the null model 2. The different versions of PRDpen are computed with k = dim(θ). The columns sensF, specF, sensR, specR, sens and spec, contain the mean, and the standard deviation in parentheses, of the sensitivities and specificities.
Fixed part Random part Model
Measure freqF sensF specF freqR sensR specR freq sens spec
PRDpen(1) mAIC 71 1(0) 0.538(0.488) 68 0.947(0.123) 0.623(0.441) 4 0.987(0.031) 0.543(0.462) mAICCm 158 1(0) 0.989(0.023) 35 0.85(0.166) 0.812(0.302) 11 0.963(0.042) 0.978(0.029) BICN 172 0.983(0.059) 0.997(0.013) 10 0.742(0.14) 0.938(0.142) 2 0.923(0.059) 0.993(0.016) BICm 172 0.997(0.027) 0.994(0.017) 21 0.798(0.163) 0.88(0.222) 5 0.947(0.046) 0.987(0.021)
PRDpen(2) mAIC 116 1(0) 0.799(0.383) 85 0.925(0.14) 0.777(0.363) 28 0.981(0.035) 0.797(0.361) mAICCm 174 1(0) 0.993(0.019) 49 0.835(0.167) 0.887(0.228) 32 0.959(0.042) 0.987(0.022) BICN 184 0.987(0.053) 0.999(0.006) 14 0.713(0.116) 0.977(0.085) 7 0.918(0.047) 0.998(0.008) BICm 183 0.998(0.022) 0.996(0.013) 33 0.775(0.157) 0.94(0.152) 22 0.942(0.042) 0.993(0.016)
Drand 0 1(0) 0.004(0.063) 6 1(0) 0.03(0.171) 0 1(0) 0.006(0.06)
mrc,a 1 1(0) 0.005(0.071) 120 0.997(0.047) 0.605(0.49) 0 0.999(0.012) 0.042(0.074) crc,a 35 0.883(0.111) 0.881(0.304) 0 0.825(0.167) 0.675(0.413) 0 0.869(0.125) 0.868(0.302) mAIC 173 1(0) 0.993(0.021) 100 0.853(0.166) 0.963(0.166) 75 0.963(0.041) 0.991(0.022) mAICCm 190 1(0) 0.998(0.012) 70 0.792(0.162) 0.988(0.084) 60 0.948(0.04) 0.997(0.012) cAIC 144 0.983(0.059) 0.977(0.079) 25 0.923(0.141) 0.435(0.469) 6 0.968(0.058) 0.944(0.083)
BICN 182 0.98(0.064) 1(0) 21 0.702(0.102) 1(0) 15 0.91(0.049) 1(0)
BICm 190 0.994(0.035) 0.999(0.007) 45 0.745(0.142) 0.997(0.033) 37 0.932(0.041) 0.999(0.007)
In case 3, the fixed part has less weight than the random part, and this results in lower freqRthan in case 1 for all PRDpen(2) for the four distributions (cf. Tables 2.13, 2.14, 2.15 and 2.16). Indeed, the increase of signal of the random part implies the increase of signal
In case 3, the fixed part has less weight than the random part, and this results in lower freqRthan in case 1 for all PRDpen(2) for the four distributions (cf. Tables 2.13, 2.14, 2.15 and 2.16). Indeed, the increase of signal of the random part implies the increase of signal