“Why does it often seem that the rate of theoretical and empirical progress in our field [psychology]
is agonizingly slow ?” wonders DeShon (2002, p.189). Different responses were proposed : some call into question the responsibility of the use of null hypothesis testing (Cohen, 1994 ; Schmidt, 1996), which was refuted by many good arguments supporting the use of null-hypothesis testing (Cortina & Dunlap, 1997 ; DeShon, 2002 ; Frick, 1996). More specifically in organizational psychology and in particular in the job satisfaction field (which is one of the concepts most used and studied in organizational behavior research), several authors point to problems that are precursors to the null-hypothesis testing “issue” : one problem is in explaining the lack of progress is due to the focus on how to measure job satisfaction instead of focusing on what job satisfaction is (Locke, 1969) ; a second problem is due to the way in
69
which job satisfaction is defined and to a lack of consensus on the definition (Brief, 1998 ; Mignonac, 2004 ; H. M. Weiss, 2002) ; and a third problem is due to a problem of measurement (Scarpello &
Campbell, 1983 ; Schleicher et al., 2004 ; Van Saane et al., 2003). Another problem explaining the slow progress in the job satisfaction field as in psychology in general is a problem related to the quality of the data and the selection model used to represent the relations among variables as mentioned by DeShon et Morris (2002).
In this article, we first wanted to go through some statistical tools available (classical and less clas-sical models) to analyze job satisfaction and to evaluate their appropriateness to adequately modelize the structure of the data. After presenting examples of what is needed to analyze human resources (HR) data and in particular job satisfaction item’s scale, we will proposed a different method to analyze job satisfaction item’s scale : the mixed effect models (MEM). Secondly, on the bases of the appropriateness of the use of MEM, we will present three simulation studies in order to evaluate the actual type I error rate level and the power of two types of models when the structure of the data are close to an item scale data. Many articles evaluate these models based on simple design with only one independent variable (IV). The novelty of this article is to simulate models close to the one used in research with many IVs correlated and not necessarily normally distributed. These simulation studies enable us to study the quality of inference when using different models to analyze item scale data.
8.2.1 How to Analyze a Job Satisfaction Item Scale ?
In this section, we highlight the problem linked to each method usually used to analyze a measure of job satisfaction and then what is really needed to correctly modelize HR data and more generally item scales data. To illustrate our purpose, we will take the short version of the Minnesota Satisfaction Questionnaire (MSQ, D. J. Weiss et al., 1967), which is an item-scale made up of 20 questions. The 20 questions refer to different facets/dimensions of job satisfaction such as pay, security, social status and working conditions. This scale could be divided into two subscales : intrinsic and the extrinsic job satisfaction.
To analyze job satisfaction, we need some information on the employees, i.e. our subjects. In the literature two groups of factors have been principally studied in relation to job satisfaction : factors related to the job itself (job characteristics (Hackman & Oldham, 1976), organizational constraints, role ambiguity and role conflict, schedules, etc.) and factors related to the worker’s personality (positive and negative affectivity (Watson & Slack, 1993), self-esteem, self-efficacy, locus of control and neuroticism (Judge & Bono, 2001), etc.). These variables could explain the variability between workers. We also need information on the questions to explain the variability between the responses to the scale questions as for example the information related to the two subscales of job satisfaction.
To illustrate our purpose, we use a data set of 293 Swiss workers (aged between 18 and 65 years).
In this data set, among others, the following variables are gathered : MSQ, positive affectivity (PA), utilization of skills (US), role conflicts (RC) and trust in the direct chief (TC) (the last four variables being related to job satisfaction in the literature). A classical approach, and the more frequently done, is to aggregate the 20 questions of MSQ (scoreyi of job satisfaction for each subject), and then analyze the new indicator by a multiple linear regression.
The model tested is the following : Yi =β0+β1T Ci+β2U Si+β3RCi+β4P Ai+εi where εi is the residuals,itheith subject,T Ci the score for the trust in direct chief for theith subject. Running this regression on our data, trust in the direct chief ( ˆβ1=0.32, t(289)=11.91, p<0.001), utilization of skills ( ˆβ2=0.38, t(289)=8.37, p<0.001), role conflict ( ˆβ3=-0.09, t(289)=-4.18, p<0.001) and positive affectivity ( ˆβ4=0.17, t(289)=4.31, p<0.001) are all highly significant and explain 61.4% of the variance of the job satisfaction.
With the regression, the variability of job satisfaction between individuals is analyzed, but the variability within subjects (i.e. the variance between the responses to the different questions on job satisfaction) is lost (Snijders & Bosker, 1999). Taking into account the information related to the
variability of the responses will help us to better understand job satisfaction by testing, for example, variables related to the items. The idea of keeping the information related to variability inside the subjects is not new and research already goes in this direction with the structural equation modeling (SEM). Another way to take into account the information related to the within variability, is to use multilevel models, also called hierarchical models or MEM. The multilevel models are more and more used in organizational research in order to simultaneously take into account the macro level and the micro level (for example at the macro level sectors, companies or business units and at the micro level the employees) or possibly to study longitudinal data. These models are applied in many other fields such as psycholinguistic for repeated measures, developmental psychology, etc. (e.g. Baayen, 2008 ; Baayen et al., 2008 ; Gilet et al., 2009).
The multilevel models analyzed simultaneously the macro and the micro levels, taking into account contextual factors such as the organizational structure as well as individual aspects. These models allowed us to more accurately and realistically model the organizational phenomena (Bliese et al., 2007). Kozlowski et Klein (2000) pointed out the importance of these models in the development of organizational research, by offering the opportunity to develop “a paradigm that can bridge the micro-macro gap in the theory and research.” (p.4). During the last two decades, the field of organizational science has progressed with the development of an integrated conceptual and possible methodological paradigm with the multilevel modelization.
Until now, multilevel models have been used in organizational research to take into account the subjects at a micro level (i.e. level 1, and for macro, level 2). But how can we use the multilevel models to take into account the variability within subjects ? As mentioned before, the multilevel can also be used for repeated measures1. In this case, the level 2 is constituted of the subjects and the level 1 of the repeated measures. Let us return to our 20 measures of job satisfaction. At the level 2, we have the 293 subjects (denotedi) and at level 1, the 20 measures of job satisfaction constituting the MSQ questionnaire (denoted j) measured for each subject, each item contributing equally to measure job satisfaction (yij). Note for the people familiar with SEM ; this model is equivalent to a SEM with job satisfaction as the latent variable, the 20 questions as the manifest variables, the loadings being equal to one and the uniquenesses being equal. This multilevel could also be named a hierarchical model, because the measures of job satisfaction are in the subjects. Two kinds of explicative variables could be used to analyze the data : variables at level 2 and variables at level 1. The variables at level 2 are the variables explaining the response variability between subjects, e.g. the factors related to job satisfaction at the personal level of positive affectivity or the personal level of trust in his/her chief. The variables at level 2 are variables explaining the variability between responses as the question/item with the idea that some questions can be more easily evaluated to be extreme or belonging to a dimension (e.g. for the MSQ, the intrinsic or the extrinsic satisfaction).
As for any statistical model in psychology, the subjects are modelized as a random effect (random as opposed to fixed) in order to generalize to the population of subjects. The idea is that the 293 subjects of our sample were theoretically drawn randomly from the population, or more precisely as the statistician will define it each person of the subject population to which we want to generalize our results has a non null chance of belonging to our sample. Here, since we have several measures (yij
per subject), one must add this subject specific intercept bi that captures the between variance and treats it as a random effect.
When testing the four variables (related to the subjects), we tested the following hierarchical model with, subject specific interceptbsui andij, the residual that captures the unexplained variance at level 1 (loosely the within subject variance) :Yij =β0+β1T Ci+β2U Si+β3RCi+β4P Ai+bsui +ij. This model assumes that each measure scores (Yij) is a function of a particular subject (T Ci, U Si, RCi, P Ai) on the variable TC, US, RC and PA plus some amount of measure level error (ij) and some amount of subject level specificity (bsui ). In other words, in our application the job satisfaction measure will be
1. When applicable, the business unit can be defined as level 3.
a function of the subject’s characteristics plus some amount of error measure and subject variations.
A measure of the quality of the model is the log-likelihood (LogLik). The greater the LogLik, the better the model fits the data. To evaluate the model we compared it to the empty model (i.e. without any explicative variables :Yij =β0+bsui +ij). In our example, adding the four explicative variables increases the LogLik by 129 points (with the empty model LogLik equal to -5997). The deviance been equal to 258 (-2*logLik) with 4 degrees of freedom.
An important advantage of this method, as we do not aggregate the questions (items) of the MSQ, is that we can take into account additional information in the model, such as the two subscales of job satisfaction (modeled through a dummy variable) : intrinsic and extrinsic satisfaction. We can also add interaction variables between level 2 (called subject level 2) and level 1 variables as the interaction between intrinsic-extrinsic satisfaction (IE) (variable of level 1) and trust in chief (TC) (level 2 variable) (see figure 8.1). The new model is :
Figure8.1 – Hierarchical Model Representation
Subject 1
…
…
Level 2 subjects
…
Level 1
response
… … … …
Subject 2 Subject 3 Subject i
Level 2 explicative variables (IV): TC, US, RC and PA
Level 1 IV: IE and interaction between IE and TC (IE*TC)
When applying this model to the data we found that the higher the trust in chief ( ˆβ1=0.40, χ2(1)=164, p<0.001), the more a person uses his/her skills ( ˆβ2=0.38,χ2(1)=80, p<0.001), there are less role conflicts ( ˆβ3=-0.09, χ2(1)=14, p<0.001), the higher the positive affectivity will be ( ˆβ4=0.17, χ2(1)=18, p<0.001), then the higher job satisfaction will be. Moreover, the intrinsic job satisfaction mean is lower than the extrinsic job satisfaction mean ( ˆβ5=-0.12,χ2(1)=20, p<0.001) and the effect of trust in the chief is less strong for the intrinsic satisfaction than for the extrinsic satisfaction ( ˆβ6=0.14, χ2(1)=28, p<0.001). Adding, these two new variables to the model increased the log-likelihood from -5859 to -5835, with a deviance of 48 and 2 degrees of freedom. This is a very large amount, showing the importance of level 1 variables.
8.2.2 Are the models correct ?
As mentioned in the previous section to best modelize job satisfaction we need to take into account the most comprehensive structure possible. This means the variability between workers (between subjects), the differences between the responses to the scale questions and, the information available on the subject and on the questions levels. With the hierarchical models, we take into account the differences between subjects, the information on the subject and the information on items. However, the variability between questions is not taken into account in this model. This has two drawbacks for the results. Firstly, the model does not take into account the variability of items and therefore does not optimally fit the data. In fact, if the responses to an item are always higher than the other items,
they will not be modelized2. Secondly the model does not allow the results to be generalized to other item scales. In research, one aim is to be able to generalize our results to the whole subject population independently of the sample used to obtain the results, which is possible with the regression or the hierarchical model used by modelizing a subject random effect. Symmetrically, to allow the results to be independent to the job satisfaction scale chosen, we need to take into account the variability between items and modelize it as an item random effect. In fact, if we want to compare the results of two studies with two different samples measured on the same item scale, it won’t be a problem because as mentioned before the subjects of each sample are modelized as a random effect. But if two different scales are used to measure job satisfaction with a regression or a hierarchical model, and we found a difference, we can not be sure if the difference is due to the scale used or to substantial differences. It has been shown that the bottom line is that these models inflate the type I error (this point will be discussed in more detail below in the simulation studies)3.
Theoretically we need an extra level, an item level (called item level 2), in order to generalize the results to the item scale as we do for subjects. If this effect exists in the data, and if it is forgotten, it will impact the estimation of the model and may lead to incorrect inference (Luo & Kwok, 2009 ; Meyers & Beretvas, 2006). This idea is not only theoretically grounded but also sustained by real data.
We analyzed about twenty databases (not presented here) in order to be sure that this additional level appears significantly informative to the model, which was the case each time. One additional point is that, if we decided not to take into account the item level, it means there is only one way to measure the concept of job satisfaction, i.e. the scale we used (Iglesias et al., 2010b).
In conclusion, we have to add an item level to the hierarchical model used before in order to comprehensively modelize the data. This new model is called a mixed effects model with crossed random effects, which we will call crossed models. The subject and the item are crossed random effects ; crossed in contrast with embedded or nested as it is the case for the subject in the business units for example.
The crossed model corresponding to the last hierarchical model tested will be :Yij =β0+β1T Ci+ β2U Si +β3RCi+β4P Ai +β5IEij +β6T CiIEij +bsui +bitj +ij with bitj the item specific intercept capturing the item variance and could be represented as in figure 8.2.
In our data, the estimation of the crossed model parameters are really close to those found for the hierarchical model (i.e. multilevel). This is due to the fully balanced design of our experiment.
As the data is fully balanced, the items are orthogonal to the subjects and therefore adding the item random effect does not affect the variance of subject. In an unbalanced design, the estimation would probably be different. In contrast, the inference (i.e. the values of the p-values) is affected by the model choice. In our example, the p-values for the crossed model are in general, slightly higher (therefore less significant). An exception occurred with the variable concerning the subscale of job satisfaction, that was highly significant in the hierarchical model (p<0.001) and not significant in the crossed model (p= 0.360). In an unbalanced design, the inference will probably be clearly worse for the hierarchical model.
Thus, the use of the crossed model as the hierarchical model, in contrast to the linear regression, allows us to take into account the variability within subjects and information on the response and to test variables of interaction between level 2 subject variables and level 1 variables, as is the case for the interaction between IE and TC. Furthermore the crossed model better modelizes the job satisfaction allowing the generalizability to the subjects and to the items, allowing the comparison between studies
2. We can estimate a mean by items (it mean to consider item as a fixe effect), but in this case, no other variable related to item could be estimated in the model as it would be collinear to the information already in the model
3. It is true that in the hierarchical model we could use the items as explicative variables of level 1 to modelize a different mean by item, i.e. modelizing items as fixed effects. In this case, we could not test any other variables of level 1 as IE nor the interaction variable of level 1 and 2 as IE*TC because the new information will be dependent of the item already in the model. The debate concerning treating effects as fixed or as random belongs not only to the MEM, but can also be found in the literature on meta-analysis and on Generalizability Theory.
Figure8.2 – Mixed effects model with crossed random effects. representation
…
Question
1 Question
2 Question
j
…
…
… …
Level 2 subject
Level 1 response
Level 2 item
… … … …
Subject 1 Subject 2 Subject 3 Subject i
Level 2 subject IV: TC, US, RC and PA
Level 2 item IV: IE
Level 1 IV: interaction between IE and TC
using different measures of job satisfaction and more precisely fitting the data than the hierarchical model.
As mentioned earlier the hierarchical model presented was equivalent to a SEM with job satisfaction as the latent variable, the 20 questions as the manifest variables, the loadings being equal to one and the uniquenesses being equal. For the crossed model, there is no equivalence in SEM4.
As mentioned previously, the hierarchical models are mostly newly used in the field of HR, but we found no mention of the use of crossed models except in the Generalizability Theory, also called G-Theory. The G-theory was originally introduced by Cronbach and colleagues (Cronbach et al., 1972, 1963). One purpose was to calculate indices to define if a measure was accurate and how much it could be generalized. In the G-theory, items are also modelized as a random effect.