To illustrate the use of PRDpen as both a measure of model selection and of model ade-quacy, we analyze a sub-sample of the data from the 1988-89 Bangladesh fertility survey (Huq and Cleland, 1990), previously analyzed by Ng et al. (2006). In this survey, 1934 women from 60 districts were surveyed on their use of contraception. These data are unbalanced, as the number of women for each district ranges between 2 and 118, with an average of 32 women for each district. The response is binary with a value of 1 indicating contraceptive use and of 0 otherwise. The predictors are the age, the number of children (factor livch with 4 levels: 0, 1, 2, 3+) and the type of residence (factor urban with 2 levels: urban, rural) at the moment of the survey. The interest of this survey is in study-ing the relationship between the use of contraceptive and the above mentioned predictors.
As in the Section 2.4.3.3, instead of speaking about PRD obtained by considering the
age
use
0.0 0.4 0.8
−1 0 1 2
rural
−1 0 1 2
urban livch
0 1 2 3+
Figure 2.1: Contraception use in function of the age standardized (age) and of the number of children (livch) categorized by the type of residence (urban). The curves are obtained by LOESS (locally weighted scatterplot smoothing).
age
use
0.0 0.4 0.8
−1 0 1 2
rural
−1 0 1 2
urban ch
N Y
Figure 2.2: Contraception use in function of the age standardized (age) and of the presence of children (ch) categorized by the type of residence (urban). The curves are obtained by LOESS (locally weighted scatterplot smoothing).
null model n0 (either n0 = 1 or n0 = 2), or PRDpen corresponding to the IC (e.g., mAIC or BICN) and obtained by considering the null model n0 (either n0 = 1 or n0 = 2), we replace this text by the notation PRD(n0) and PRDICpen(n0), respectively.
We fitted 16 different models (GLMM with yi | bi ∼ Bernoulli(µi) and logit link function) considering the predictors age that is standardized for numerical stability (vari-able age), age (standardized) squared, urban and livch. We further create the factor ch, that is the factor livch dichotomized into 0 versus 1 or more (factor ch with levels:
N=no child, Y=1 or more children), and the interaction between age and ch. Figure 2.1 justifies the inclusion of the predictor age squared and the creation of the factor ch that dichotomizes the factor livch. Indeed, we can see a quadratic effect of age and there appears to be no significant difference between the levels 1, 2 and 3+ of the factorlivch (we will formally test this). Figure 2.2 justifies the inclusion of the interaction between age and ch, as it seems that the effect of the age on the contraception use depends on the presence of children. Three different random parts are considered: (1) a random
in-Table 2.17: Description of the fitted models for analyzing the data on the use of contra-ception by women in Bangladesh. uncorr.=uncorrelated; corr.=correlated.
Model
random uncorr. random corr. random
intercept intercept + slope intercept + slope
Predictor 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
age × × × × × × × × × × × × × × × ×
age2 × × × × × × × × ×
urban × × × × × × × × × × × × × × ×
livch × × × × × ×
ch × × × × × ×
age×ch × × ×
tercept only; (2) a random intercept and a random slope for urbanthat are uncorrelated;
and (3) a random intercept and a random slope for urbanthat are correlated. The fitted models are described in Table 2.17. The fixed part of the model 1 contains the predictor age; that of the models 2, 7 and 12 further contains the predictor urban; that of the models 3, 8 and 13 further adds the predictor livch to age and urban in comparison with the models 2, 7 and 12; that of the models 4, 9 and 14 further contains the predictor age squared in comparison with the models 3, 8 and 13; that of the models 5, 10 and 15 contains the same predictors as the models 4, 9 and 14, but livch is replaced bych; and that of the models 6, 11 and 16 further contains the interaction agebych in comparison with the models 5, 10 and 15. Concerning the random part, the models 1 to 6 contain a random intercept; the models 7 to 11 contain a random intercept and a random slope for urbanthat are uncorrelated; and the models 12 to 16 also contain a random intercept and a random slope but that are correlated. We do not write all the equations of these different models but, for instance, the model 16 can be written as
logit(µij) =β0+β1ageij +β2age2ij +β3urbanij +β4chij +β5ageij·chij +bi0+bi3urbanij,
bi0 bi3
!
∼N
0 0
!
, σ2int σint,urban σint,urban σ2urban
!!
, (2.11)
for i=1,· · · ,60, j = 1,· · · , ni.
Then, we computed for all the 16 models the measures considered in the simulation study (cf. Table 2.1 for the different versions of PRDpen, defined in equation (2.4), and Section 2.4.2 for the alternative measures). To this end, we further fitted both null models:
the null model 1 that contains a fixed intercept and the null model 2 that further contains a random intercept. As in the simulation study, the results for PRDmAICCpen N, PRDmCAICpen N, PRDmCAICpen m, the mAICCN, mCAICN and mCAICm are not presented. Also, we do not present the values of PRDpen obtained with k = dim(β) and k = dim((τ0, φ)0). The reasons of these omissions are the same as those exposed in Section 2.4.3.2. In comparison with the simulation study, we further present PRDBICpenh and the BICh for completeness (it illustrates that the values of PRDBICpenh, or of the BICh, are between those of PRDBICpenN, or of the BICN, and those of PRDBICpenm, or of the BICm, respectively). The values of
the cAIC are also given (it took 52 hours to compute the 16 values of the cAIC on a personal computer). All the considered measures of model adequacy are presented for completeness. Indeed, with this illustration, we are not only interested in model selection (as it was the case in the simulation study in Section 2.4), but also in assessing model adequacy. We therefore consider also the (non adjusted) marginal and conditional rcand we differentiate both null models for Drand and Prand (which was not the case in the simulation study, as the results for Drand and Prand in terms of model selection are the same regardless of the null model).
The subsequent paragraphs are organized as follows. Firstly, we use PRDpen and the IC to identify the model that best fits the data among the 16 alternatives (the models that best fit the data are those with the largest values of PRDpen and with the smallest values of the IC). Secondly, we give the parameters estimates of the selected model. According to this model, we discuss the relationship between the use of contraceptive and the selected predictors, along with the variation across districts. Thirdly, we use PRD, PRDpen and the other considered measures of model adequacy to evaluate the adequacy of the selected model. Finally, we give a summary.
Table 2.18 provides the values of the considered measures for the 16 different models.
The R code used to obtain these results is given in the online supplementary material.
In order to identify the relevant fixed effects, we will focus on the versions of PRDpen(1), based on the simulation study results (cf. Section 2.4.3.3). Furthermore, we consider the cAIC and, even if the marginal IC give equivalent results for model selection to PRDpen(1), as explained in Section 2.4.3.2, we also consider them to illustrate this equivalency. The comparison of the values of the different versions of PRDpen(1) as well as all of the IC for models 1 and 2 clearly shows that the predictor urban is relevant. For instance, PRDmAICpen (1) is equal to 0.021 for model 1 and to 0.033 for model 2. Then, as can be seen in Table 2.17, the models 2, 7 and 12; 3, 8 and 13; 4, 9 and 14; 5, 10 and 15; 6, 11 and 16 have the same fixed part, respectively. Thus, the usefulness of the predictor livch is highlighted comparing the values of any of the measures of model selection above mentioned for the models 2, 7 and 12 with those for the models 3, 8 and 13, respectively. For instance, the difference between the cAIC for model 2 (3425.107) and the cAIC for model 3 (3408.685) is equal to 16.422, which is considered as very important.
The quadratic effect of age is also highlighted by all these indices when comparing the models 3, 8 and 13 with the models 4, 9 and 14, respectively. Indeed, PRDBICpenh(1) is for instance around 0.06 for models 3, 8 and 13 and around 0.07 for models 4, 9 and 14. Then, we compare the models 4, 9 and 14 with the models 5, 10 and 15 to see if the dichotomized factor ch should be preferred over the factor livch. PRDpen(1) is slightly larger when including ch rather than livch (relative increase between 1.2 and 8.9% depending on the version of PRDpen(1)). Thus, with less information, we are able to explain the same proportion of deviance. Thereby, we prefer the predictor ch overlivch. The IC all agree with that choice, as they are smaller for the models containing the dichotomized factor ch. The decision of including the interaction between ageand ch is made by comparing the values of the different measures of model selection for the models 5, 10 and 15 with those for the models 6, 11 and 16, respectively. This decision depends on the considered value for λn? in PRDpen(1). Indeed, slightly larger values of PRDmAICpen (1), PRDmAICCpen m(1) and PRDBICpenm(1) are observed, but we rather observe equal values for PRDBICpenN(1) and PRDBICpenh(1). The same is true regarding the different marginal IC. For instance, the BICm is equal to 2397.752 for model 5 and to 2393.842 for model 6 (the difference is considered as important, as it is close to 4), while the BICN is equal to 2418.59 for model 5 and
to 2418.153 for model 6 (the difference is unimportant, as it is practically 0). The cAIC further indicates not to include the interaction, as it is around 2 points larger for models 6, 11 and 16 in comparison with models 5, 10 and 15.
In order to identify the best set of random effects, the results of the simulation study obtained for the Bernoulli distribution (cf. Section 2.4.3.3) indicate to consider PRDpen(2), as the frequency of choosing the correct random part is higher for PRDpen(2) than for PRDpen(1) (and thus, than for the marginal IC) and than for the cAIC. However, we also consider the IC to illustrate the similarities and differences between these measures. We see in Table 2.17 that the models 2, 7 and 12 have the same fixed part and differ only in their random part and that the same is true for the models 3, 8 and 13; 4, 9 and 14; 5, 10 and 15; and 6, 11 and 16, respectively. Thereby, the comparison of models 2 and 7; 3 and 8; 4 and 9; 5 and 10; and 6 and 11 allows for evaluating the relevance of a random slope for urban that is uncorrelated with the random intercept. The values of the different versions of PRDpen(2) are equal or smaller when the models contain the random slope.
Indeed, PRDmAICCpen m(2) is for instance equal to 0.043 for both models 3 and 8. Thus, the models with the random intercept are preferred over those that contain uncorrelated random intercept and slope. The same applies for the IC, except for the cAIC that favors the inclusion of the random slope. When we compare model 2 with 12; 3 with 13; 4 with 14; 5 with 15; and 6 with 16, we compare the models with a random intercept with those containing correlated random intercept and slope for urban. The values of PRDpen(2) are slightly larger for the models with a random slope (e.g., PRDmAICCpen m(2) is equal to 0.047 for model 13, while it is equal to 0.043 for model 3), except for PRDBICpenN(2). The choice of the random part will thus depend on the choice of λn?. The same conclusion is true for all the IC, which could be surprising, as in the simulation study (cf. Section 2.4.3.3), PRDpen(2) selects more often the correct random part than the corresponding IC (the selected random part is thus different). However, this is explained by the strong signal of the random part. In particular, we will see below that the variance of the random slope for the type of residence, as well as the correlation between the random intercept and the random slope, are important.
We know from the results of the simulation study in Section 2.4.3.3 that PRDBICpenN and the BICN are less appropriate than PRDBICpenm and the BICm, respectively. Thus, PRDBICpenN and the BICN are not advised for these data. Thereby, we decide to include the interaction between ageandch considering the values of PRDmAICpen (1), PRDmAICCpen m(1), PRDBICpenm(1), the mAIC, mAICCmand BICm rather than the PRDBICpenN(1) (and indirectly PRDBICpenh(1)), the BICN (and indirectly the BICh) and the cAIC. Furthermore, only PRDBICpenN(2) and the BICN favor models with a random intercept, in comparison with models that contain a random intercept and a random slope for urban that are correlated. Thus, the model that contains the predictorsage,urban,agesquared,ch and the interaction betweenage and ch, and that further contains a random intercept and a random slope forurbanthat are correlated is selected, which corresponds to the model 16 (cf. equation (2.11)). Note that the model 16 is not the most complex existing model, as we retain the dichotomized factor ch rather than the factor livch.
Table2.18:MeasuresformodelsofthedataontheuseofcontraceptionbywomeninBangladesh.(1)correspondstothenullmodel1 and(2)correspondstothenullmodel2.ThedifferentversionsofPRDandPRDpenarecomputedwithk=dim(θ). Model RandominterceptRandominterceptandslope-uncorrelatedRandominterceptandslope-correlated Measure123456789101112131415 PRD(1)0.0230.0350.0680.0840.0840.0870.0370.070.0850.0850.0880.0410.0740.0890.0890.092
PRD
(1) pen
mAIC0.0210.0330.0640.0790.080.0820.0330.0640.0790.080.0820.0370.0680.0820.0830.085 mAICCm0.0210.0320.0630.0780.080.0820.0330.0630.0770.0790.0810.0370.0660.080.0820.084 BICN0.0170.0260.0510.0640.0690.0690.0250.0490.0610.0670.0670.0270.0510.0620.0680.068 BICm0.020.030.0590.0730.0760.0780.030.0590.0720.0750.0770.0330.0610.0750.0780.079 BICh0.0220.0320.0560.0690.0750.0750.0330.0570.070.0750.0750.0360.060.0720.0770.077 PRD(2)0.0010.0140.0480.0640.0640.0670.0150.0490.0640.0640.0670.020.0540.0690.0690.072
PRD
(2) pen
mAIC00.0120.0440.0590.060.0630.0130.0440.0590.060.0630.0170.0480.0630.0640.066 mAICCm00.0120.0430.0580.060.0620.0120.0430.0580.0590.0620.0160.0470.0610.0630.065 BICN-0.0020.0080.0320.0460.0510.0510.0060.0310.0430.0490.0490.0080.0330.0450.0510.051 BICm-0.0010.010.0390.0540.0570.0580.010.0390.0530.0560.0570.0140.0430.0560.0590.06 BICh0.0010.0110.0350.0480.0540.0540.0120.0360.0490.0550.0550.0150.040.0520.0580.058 mDrand-0.0010.0190.0510.0660.0660.0690.0190.050.0650.0650.0690.0190.0510.0660.0660.069 Drand(1)0.0580.0650.10.1160.1160.1190.080.1130.1260.1260.1280.0890.1220.1350.1350.137 Drand(2)0.0010.0090.0460.0630.0630.0660.0250.0590.0740.0730.0760.0340.0690.0830.0830.085 Prand(1)0.0450.0530.0880.1040.1040.1070.0630.0960.110.110.1130.0710.1040.1170.1170.12 Prand(2)-0.013-0.0040.0330.050.050.0530.0070.0420.0570.0560.0590.0150.0490.0640.0640.067 R2 m0.0020.0260.0870.1150.1150.1250.0320.0910.1170.1170.125 R2 c0.0720.080.1420.1720.1720.180.1180.1750.1980.1970.204 mrc0.0020.0480.1240.1570.1570.1640.0470.1230.1560.1560.1630.0540.1280.1610.1610.167 mrc,a0.0010.0460.1210.1540.1550.1610.0450.120.1530.1540.160.0530.1260.1580.1590.165 crc0.1130.1330.2050.2360.2360.2420.1570.2240.250.250.2550.1720.2380.2630.2630.267 crc,a0.1120.1310.2020.2340.2340.2390.1560.2210.2480.2480.2520.170.2350.2610.2610.265 mAIC2537.9272508.3932427.6162388.7292385.1862379.1812506.1452426.2992388.6842385.0752379.3392496.0272417.0172380.6112376.9762371.53 mAICCm2538.3552509.122429.772391.5522386.7712381.3352507.2562429.1222392.2842387.2292382.1632497.6122420.6172385.1012379.82375.13 cAIC3429.1623425.1073408.6853398.2843393.9573396.1383408.3433396.5963388.7443384.6243386.8093406.5763393.833386.2893380.7863383.514 BICN2554.6292530.6622466.5872433.2672418.592418.1532533.9822470.8382438.792424.0462423.8782529.4312467.1232436.2842421.5152421.637 BICm2544.212516.772442.2762405.4832397.7522393.8422516.6172443.0542407.5332399.7352396.0942508.5932435.8662401.5542393.7312390.38 BICh2547.6832523.7162459.6412426.3212411.6442411.2072520.092456.9462424.8982410.1542409.9862512.0662449.7582418.9192404.152404.272
Table 2.19: Estimates of the parameters of the model 16 defined in equation (2.11). The standard errors of the estimates of the fixed effects are given in parentheses.
Parameter estimate Predictor
ageij age2ij urbanij chij ageij·chij
βˆ0 βˆ1 βˆ2 βˆ3 βˆ4 βˆ5 σˆ2int σˆurban2 ρˆint,urban σˆint,urban
-1.344 -0.416 -0.459 0.790 1.211 0.599 0.378 0.526 -0.793 -0.354
(0.224) (0.197) (0.069) (0.163) (0.209) (0.230)
The estimates of the parameters of the model 16, that is defined in equation (2.11), are given in Table 2.19. The value of these estimates allows for understanding the relationship between the use of contraceptive and the predictors, and also for bringing to light the variability across districts. We especially follow Rodr´ıguez (2016) and Ng et al. (2006) in order to interpret the selected model. Ng et al. (2006) aimed at comparing different estimation methods, and thus only analyzed our model 13 and compared it with our model 3 to justify the inclusion of the random slope for urbanusing the LRT. Rodr´ıguez (2016) also analyzed our models 3 and 13, but no comparison of models was conducted.
Among the predictors of the model 16, urban gives the information that women of the same age and having the same status regarding the children have a higher probability of using a contraceptive if they live in an urban, rather than a rural, area with an odds that is more than two times greater (exp(0.790) = 2.203). Furthermore, the dichotomized factorchindicates that within a type of residence and age, the odds of using contraception is more than 3 times larger for women having one or more children than those having no children (exp(1.211) = 3.357). The effect of the age on the probability of contraceptive use depends on the factor ch. Indeed, for women living in the same type of residence, the probability of using a contraceptive declines with age if they have no children, with 3.4% lower odds per year (exp(−0.416)−1 =−0.340). If these women have one or more children, the odds of using contraception rather very slightly increases and is 1.2 times higher per year (exp(−0.416 + 0.599) = 1.201).
The random part of the model 16 gives us complementary information. Indeed, the random intercept allows for capturing the variability of contraception use between dis-tricts. The standard deviation of the random intercept ˆσint= 0.615 indicates that women in a district which is one standard deviation above the mean have odds of using contracep-tion that are around 85% higher than women in an average district (exp(0.615) = 1.849), which is considerable. The random slope for the predictor urban further indicates that the influence of the type of residence on the use of contraceptive varies greatly by dis-trict. For instance, the difference in logits between an urban area and a rural area is close to zero in a district with differential one standard deviation below the mean (0.790− σˆurban = 0.790 −0.725 = 0.065). And this difference for a district with dif-ferential one standard deviation above the mean is equal to 0.790 + 0.725 = 1.515, which corresponds to an odds of using contraception that is more than 4 times greater (exp(1.515) = 4.549) for urban area (compared with rural area).
We finally evaluate the adequacy of the model 16 considering the measures of model adequacy presented in Table 2.18. PRD(1) and PRDpen(1) indicate that the proportional reduction in deviance in comparison with the null model 1 (fixed intercept) ranges be-tween 7 and 9%. With the null model 2 (fixed and random intercepts), PRD(2) and
PRDpen(2) range between 5 and 7%, which means that around 2% of the proportion of explained deviance is due to the random intercept.
The marginalDrand indicates that the proportional reduction in joint deviance due to the fixed effects is around 7%, and Drand with the null model 1 indicates that the propor-tional reduction due to both fixed and random effects is around 14%. It is interesting to note that the value of PRD(1) is in between. An explanation is that PRD is computed using the marginal log-likelihood and is thus close to a marginal measure. However, the estimates of the variance components are integrated in the formula of PRD (cf. equation (2.2)), which allows for taking into account the random component of the model, which is closer to a conditional measure. The difference between the value of Drand for the null model 1 (0.137) and that for the null model 2 (0.085) confirms the high variability between districts. The results obtained with Prand, which measures the proportional reduction in PQL, are close to those obtained with Drand.
The values of R2m and R2c, that measure the proportion of variance explained by the fixed effects and by both fixed and random effects, respectively, are higher than that of the marginal Drand and Drand (we do not expect equal values, as these indices measure different quantities). We further observe that the value of the conditional measure Rc2 is around 2 times higher than that of the marginal measure R2m, which is also observed when comparing the marginalDrandandDrand, and which again shows the large variability between districts.
The non adjusted and adjusted concordance correlation coefficients (marginal and conditional rc and rc,a) are close and their values are higher than those of the other measures of model adequacy. Different values for these latter measures are expected, as they are not measures of explained variance or deviance, but are rather concordance correlation coefficients between observed and predicted values.
The values of all the considered measures of model adequacy are rather small, which means that the model 16 does not capture much of the information contained in the data.
Thus, it means that it is difficult to provide a good prediction of contraception use based on this model.
In summary, we identified the model 16 as the best among the 16 alternatives using PRDpen and the IC. Model 16 indicates that the probability of using a contraceptive is higher in urban area (compared with rural area), is higher for women having children (compared with women having no children), declines with age for women having no chil-dren, and is close to be constant for women having children. Furthermore, the model 16 indicates that on average, there is variability across districts, and that the effect of the type of residence on the probability of contraception use is different between districts.
However, all the considered measures of model adequacy indicate that we are not able to explain a large proportion of deviance or variance, or that the concordance between observed and predicted values is low, respectively.
2.6 Discussion
In this Chapter, we propose, in the context of the GLMM, a penalized measure of the proportional reduction in deviance due to the model of interest in comparison with a prespecified null model, named PRDpen. PRDpen is demonstrated to be appropriate for model selection in Section 2.3.3, as PRDpen is asymptotically loss efficient and consistent for fixed effects selection, and in the simulation study in Section 2.4. Indeed, the results of our simulation study confirm that the ability of PRDpen to perform model selection is
similar to that of the IC. These results further demonstrate that the existing measures
similar to that of the IC. These results further demonstrate that the existing measures