populations
Samuel Turcotte, Maude Beaudoin, Catherine Vallée, Claude Vincent et François Routhier.
4.1 Résumé
Cette revue systématique synthétise l’état et la qualité des connaissances portant sur les qualités métrologiques du Community Integration Questionnaire pour les personnes vivant avec un handicap autre qu’un traumatisme cérébral selon les lignes directrices PRISMA. Les bases Medline, Embase, CINAHL, OTseeker et PsycINFO ont été consultées. L’extraction des données et l’évaluation méthodologique critique des articles (liste de contrôle MacDermid, liste de contrôle COSMIN) ont été effectuées. Dix études représentant 3000 personnes répondaient aux critères d’inclusion. Cinq populations ont été documentées. De nombreuses propriétés psychométriques du Community Integration Questionnaire sont encore mal évaluées pour les adultes vivant avec un handicap autre qu’une lésion cérébrale traumatique.
59 4.2 Abstract
This systematic review the psychometric evidence concerning the utilisation of the Community Integration Questionnaire for individuals living with a disability other than a traumatic brain injury. Medline, Embase, CINAHL, OTseeker and PsycINFO were searched. PRISMA guidelines were used. Studies that investigated at least one psychometric property of the Community Integration Questionnaire for individuals living with a disability other than traumatic brain injury were included. Data extraction and critical methodological appraisal of the articles (MacDermid checklist, COSMIN checklist) were independently performed and validated by the 2 first authors. Ten studies representing 3000 individuals met the inclusion criteria. Five populations were documented. Many psychometric properties of the Community Integration Questionnaire are still poorly evaluated for adults living with a disability other than a traumatic brain injury.
60 4.3 Introduction
Community integration is the engagement of an individual following a disability in social roles that are culturally and developmentally appropriate1. Community
integration is described as the ultimate goal of rehabilitation interventions2, 3.
The reason is that successful community integration is often synonymous with physical accessibility, sufficient and satisfactory opportunities for social participation, a network of reciprocal social relations, a sense of belonging to one’s environment and fewer identity issues. Because of that, one must have sufficient knowledge for evidence-based decision-making regarding the assessment of community integration. It is essential to generate knowledge that helps clinicians and researchers make evidence-based decisions regarding their assessment. The Community Integration Questionnaire is one of the most used and renowned assessment for the evaluation of community integration4-7, even
if other questionnaires exist (e.g. Community Integration Measure or Reintegration to Normal Living Index). However, either recent reviews were done or there weren’t sufficient papers to conduct a review about psychometric properties of these tools.
At this time, two systematic literature reviews were conducted, summarizing the evidence on the psychometric properties of the Community Integration Questionnaire. The first one, undertaken in 1997, exclusively addressed this question in the context of traumatic brain injury8. One of the conclusions
suggested further research on sensitivity temporal variability and reliability for people with more severe cognitive deficits8. The second was conducted in 2008
and was also focused solely on traumatic brain injury6. The objective of that
study was to review assessment methods of community integration for this population and evaluate the psychometric and administrative properties6. The
main conclusion was that the Community Integration Questionnaire was the most reliable and objective measure of community integration outcome for this population.
61
Even though these studies bring useful information for clinicians and researchers, there is still a gap in the evaluation of the psychometric properties of the Community Integration Questionnaire for adults living with disabilities other than traumatic brain injury. Therefore, the objective of our study is to review the content and the quality of the psychometric evidence relating to the use of the Community Integration Questionnaire for adults living with disabilities other than traumatic brain injury.
Psychometric properties addressed in this review are: 1) validity (criterion validity, structural validity/factor analysis, construct validity/hypothesis testing, content validity, cross-cultural validity, floor and ceiling effect, item response theory), 2) reliability (test-retest reliability, internal consistency, inter-rater reliability), 3) acceptability, 4) feasibility and 5) responsiveness.
4.4 Methods
A systematic review was conducted to identify the articles studying the psychometric properties (validity, reliability, acceptability, feasibility, responsiveness) of the Community Integration Questionnaire for adults living with disabilities other than traumatic brain injury. This systematic review was written in accordance with the Preferred Reporting Items for Systematic reviews and Meta-analyses9.
A database search using Medline, CINAHL, OTseeker, Embase and PsycINFO was conducted on in early June 2019. A search strategy was applied systematically to each database. Controlled vocabulary was added to the equation for databases using such vocabulary. During the search, there was no restriction on the year of publication. The search strategies for CINAHL and PsycINFO are presented as examples in Appendix 1.
Studies were included if they documented at least one psychometric property of the Community Integration Questionnaire. Studies were excluded if they: 1) were written in a language other than English or French or 2) only included
62
individuals living with traumatic brain injury. The two first authors together screened the titles and the abstracts of all retrieved articles in regard to inclusion and exclusion criteria. Then, the preselected articles were independently read in full by the two same authors to ensure their eligibility. A perfect consensus was reached for the final selection of articles. There was no issue to discuss for the inclusion.
The following information was extracted: authors, year of publication, country, design, population and psychometric properties (criterion validity, structural validity/factor analysis, construct validity/hypothesis testing, content validity, cross-cultural validity, floor and ceiling effect, item response theory, test-retest reliability, internal consistency, inter-rater reliability, acceptability, feasibility and responsiveness). Two authors (ST, MB) conducted an independent extraction for half of the articles and validated the extraction performed on the other half. Extraction table was built in light of each of the five populations identified through the search; which, as far as the data allow, opened the door for a certain comparison between populations.
Two complementary checklists were used to appraise the quality of the psychometric evidence. First, the general methodological quality of the selected studies was evaluated with the Critical appraisal of study design for psychometric articles from Annexe H in Law and MacDermid10 (MacDermid checklist hereafter).
To the authors' knowledge, no guideline exists to interpret the MacDermid’s scores. Therefore, the scores were transformed in percentages and interpreted as follows: poor quality (scores between 0 and 50%), fair (scores between 51 and 80%), and good (scores above 80%).
Second, the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) checklist11, 12 was used. Quality level in the
COSMIN checklist is either poor, fair, good or excellent. The specific criteria per item of each COSMIN item are described in a 4-point scale available from the web site www.cosmin.nl.
63
Two authors (ST, MB) evaluated each selected study independently with the MacDermid checklist and the COSMIN checklist. A consensus for each item of the MacDermid checklist and the COSMIN checklist was then reached by discussion between the authors (when needed), as suggested in previous studies13, 14. To be judged trustable, data had to be issued from studies with fair
to good quality according to the MacDermid checklist and from fair to excellent quality according to the COSMIN checklist.
64 4.5 Results
After duplicates were removed, a total of 126 articles were identified. Ten articles representing a little bit more than 3000 individuals met the selection criteria.
Figure 3. Flow diagram: selection process of the studies
Figure 3 presents the flow diagram detailing this process. Characteristics of included studies are shown in Table 5. Research on the psychometrics properties of the Community Integration Questionnaire for adults living with a disability other than traumatic brain injury is quite recent, as 60% of the studies identified were published after 2010. Quality appraisal according to the MacDermid
4 Clinical Rehabilitation 00(0)
struct validity. These data are all trustable accord- ing to the checklists.
Severity of the aphasia is negatively associated with the scores of the Community Integration
Questionnaire.15 Functional status for adults with
multiple sclerosis (as measured by the Expanded
Disability Status Scale)20 and for those with apha-
sia (as measured by the COOP-WONCA charts and
the Barthel Index)15 are positively correlated with
the scores of the Community Integration Questionnaire. All these data reach a sufficient level of methodological quality.
Finally, construct validity is also documented by the presence of significant and positive asso- ciations between the Community Integration Questionnaire and other questionnaires that assess this rehabilitation outcome. In fact, the Community Integration Questionnaire is associated with the
65
checklist (general methodological quality) and the COSMIN checklist (design behind the measurement of each psychometric property) are presented in Table 6.
Validity
Construct Validity
Data show positive correlations between CIQ score and age (being younger) in samples of adults dealing with aphasia15, with spinal cord injury16, 17 and in a
mixed sample18. It is the same for the correlations with gender (being a woman)
in samples of adults living with multiple sclerosis19, with spinal cord injury16, 17
and in mixed populations18, 20. Ethnicity (being white) in a sample of adults living
with spinal cord injury16, living arrangement (living with others) and records of
subsequent employment (positive history of employability) in a mixed sample20
also positively correlates with the Community Integration Questionnaire scores. However, only the association between gender and community integration in adults with multiple sclerosis19 and the one between age and community
integration in adults with aphasia15 reaches a sufficient level of methodological
quality according to the MacDermid and COSMIN checklists.
Table 5. Characteristics of the studies
Characteristic Number of studies Country United States 6 (60%) Iran 2 (20%) Netherlands 1 (10%) Switzerland 1 (10%) Population
Spinal cord injury 2 (20%) Multiple sclerosis 2 (20%)
Aphasia 1 (10%)
Burns 1 (10%)
66
Included studies Populations MacDermid checklists’ scores
Quality appraisal behind the measurement of psychometric properties (COSMIN checklist) Internal consistency Test- retest Reliability Content
validity Structural validity Hypotheses testing
Cross- cultural validity Respo nsi- venes s Dalemans et al.
2010 136 Aphasia n=150 (92%) Good Fair Poor ¾ Good Fair ¾ ¾
Taheri et al.
2016 137 n=265 MS (90%) Good ¾ ¾ ¾ ¾ ¾ ¾ Fair
Rintala et al.
2002 138 Mixed n=99 Fair (79%) Poor Fair ¾ ¾ Fair Poor ¾
Kratz et al.
2015 139 n=627 SCI Fair (79%) Fair ¾ ¾ Poor Poor ¾ ¾
Hirsh et al.
2011 140 n=751 Mixed Fair (77%) Excellent ¾ ¾ Excellent Good ¾ ¾
Negahban et al. 2013 88
MS
n=105 Fair (75%) Excellent Fair ¾ Excellent Good Fair ¾
Tomaszewski et
al. 2016 141 Mixed n=54 Fair (68%) Fair ¾ ¾ ¾ Poor ¾ ¾
Gerrard et al.
2015 142 n=492 Burns Fair (67%) Fair ¾ Poor Fair ¾ ¾ ¾
Gontkovsky et
al. 2009 6 n=28 SCI Fair (59%) Poor ¾ ¾ ¾ Poor ¾ ¾
Corrigan et al.
1995 143 n=461 Mixed Poor (42%) Poor ¾ ¾ ¾ Poor ¾ ¾
67
Also, there were significant associations between the Community Integration Questionnaire and three questionnaires on quality of life (Multiple Sclerosis Quality of Life Questionnaire19, Short-Form Health Survey-36 in a mixed
population sample21 and Life Satisfaction Questionnaire in adults with aphasia15)
which indicate good construct validity. These data are all trustable according to the checklists.
Severity of the aphasia is negatively associated with the scores of the Community Integration Questionnaire15. Functional status for adults with
multiple sclerosis (as measured by the Expanded Disability Status Scale)19 and
for those with aphasia (as measured by the COOP-WONCA charts and the Barthel Index)15 are positively correlated with the scores of the Community Integration
Questionnaire. All these data reach a sufficient level of methodological quality.
Finally, construct validity is also documented by the presence of significant and positive associations between the Community Integration Questionnaire and other questionnaires that assess this rehabilitation outcome. In fact, the Community Integration Questionnaire is associated with the Mayo-Portland Adaptability Inventory in mixed population20. It is also associated with the Craig
Handicap Assessment of Reporting Technique in adults with spinal cord injury16
and in mixed population22. However, only this last association reaches a
sufficient level of quality according to the checklists’ completion.
Structural Validity
Structural validity has been established with factorial analysis, which is a method to statistically determine the underlying theoretical dimensions in an outcome tool 23. Items of an outcome tool are regrouped under the theoretical dimensions
they represent 23. For the Community Integration Questionnaire, with the
multiple sclerosis population, the analysis revealed a 4-factor structure that accounted for 66.63% of variance19. In the exploratory factor analysis with the
burn survivor’s population a 2-factor structure was found and confirmed in the validation phase. Six items loaded on factor 1 and six items loaded on factor 2.
68
One item cross-loaded on both factors24. A confirmatory factor analysis was
conducted only with the burn survivor’s population, which is an important step to ensure the quality of a factorial analysis. The methodological quality is sufficient according to the MacDermid and COSMIN checklists.
In the exploratory factor analysis with the mixed population, a 4-factor structure accounting for 63% of the variance was identified. Four items loaded on factor 1, four items loaded on factor 2, three items loaded on factor 3 and two items loaded on factor 4. The authors decided to conduct a confirmatory factor analysis, however, they decided to exclude one item (child care) to found a 3- factor structure where six items loaded on factor 1, four items loaded on factor 2, two items loaded on factor 3 and three items cross-loaded on two factors 21. With the aphasia population, the factor analysis revealed a 2-factor
structure. Seven items loaded on factor 1 and four items loaded on factor 2. Since one item cross-loaded on the two factors and that factor 2 had low internal consistency, factor analysis was conducted again with one factor. These authors also decided to remove the items with low factor loading (childcare, frequency of leisure activities, leisure activities, having a best friend, job/school) to increase internal consistency of the factor15. Even though the methodological
quality is sufficient according to the MacDermid and COSMIN checklists with aphasia and mixed populations, to remove items in the factorial analysis is to modify the structure of the questionnaire and therefore, conclusions cannot be made on the structural validity of the Community Integration Questionnaire but on a modified version of it.
For the spinal cord injury population, the results on structural validity must be interpreted with even more caution since items were removed and modified during the factorial analysis and the methodological quality is insufficient according to the COSMIN checklist. In the exploratory factor analysis, the authors excluded from analyses item 4 regarding childcare because of the low response rate. A 3-factor structure was obtained. However, some factors did not load strongly on any factor or cross-loaded on multiple factors. The authors
69
decided to remove the school item and to add an item relating to entertaining inside the home. In the confirmatory factor analysis, the 3-factor identified in the exploratory factor analysis was not a good fit. Consequently, a post hoc exploratory factor analysis was conducted, and it resulted in a 3-factor structure including the same new items17.
Floor and Ceiling Effects
Few authors studied this psychometric property15, 17-19. There were no floor or
ceiling effects for adults dealing with aphasia (Total, Home Integration subscale, Social Integration subscale, Productive Activities subscale)15 and with one of the
mixed population sample (Total, Home Integration subscale, Social Integration subscale)18. In this last sample, authors found some inconsistencies in the
hierarchy of ratings for the Productive Activities subscale that undermined its clarity. For individuals with multiple sclerosis, the percentage of participants that obtained the maximum or the minimum score was always lower than the cut point of 15%. However, the percentage is three times higher for the Productive Activities subscale for maximum scores (Home Integration subscale=2.9%, Social Integration subscale=2.9%, Productive Activities subscale=9.5%, total=0%) and the minimum scores (Home Integration subscale=6.7%, Social Integration subscale=0%, Productive Activities subscale=11.4%, total=0%)19.
In adults living with spinal cord injury, high rates of ceiling effects were found for the Productive Activities subscale, while the other subscales had low rates of floor and ceiling effects17. Only the data concerning the mixed population aren’t
trustable according to the MacDermid and COSMIN checklists.
Cross-Cultural Validity
Cross-cultural validity was discussed for the studies in which translations were made. According to COSMIN, “an adequate procedure contains multiple forward and backward translations with at least two translators per step” 25. In the
70
it difficult to judge if the process led to an adequate adaptation of the questionnaire.
With multiple sclerosis populations, no major modifications were made by forward and backward translators. Explanations were added to the item concerning volunteering activities19. The agreement between Spanish and
English version of the Community Integration Questionnaire for the mixed population was evaluated with Kappa statistics. Kappa coefficient were in the moderate to a substantial range: five items (shopping, social arrangements, frequency of leisure activities, visiting others, having a best friend) are in the 0.41–0.60 range and five items (meal preparation, housework, child care, personal finances, travel outside home) are in the 0.61 to 0.80 range22. Two
items (frequency of shopping, leisure activities) had only slight to fair (≤0.40) agreement. One item (job/school) had an excellent agreement (>0.80)22. The
quality of the study was sufficient for the multiple sclerosis population only according to the MacDermid and COSMIN checklists. The quality of the study with the mixed population is insufficient according to the COSMIN checklist.
Reliability
Internal consistency
As shown in Table 7, internal consistency, determined with Cronbach coefficient alpha (α), was always acceptable (α ≥ 0.70)26 or approaching acceptable limits
for the Community Integration Questionnaire total scores for adults with multiple sclerosis, spinal cord injury or burn survivors and with mixed population. Home Integration subscale had the highest internal consistency between the three subscales and always reached acceptable limits. Productive Activities subscale never reached acceptable limits for internal consistency. Social Integration subscale reached an acceptable α once with a mixed population sample but in a study where the methodological quality is insufficient.
71
Item-to-total correlations are also shown in table 8. Total-Home Integration subscale and Total-Social Integration subscale correlations reached high levels
27. Between the three subscales, the Productive Activities subscale always had
the lowest correlations with the Community Integration Questionnaire total score and only showed low to moderate correlation levels.27 If the methodological
quality of Hirsh et al. (2011) and Tomaszewski et al. (2016) is sufficient for the two mixed population samples, data concerning spinal cord injury must be used with caution.
Test-Retest Reliability
The test-retest reliability of the Community Integration Questionnaire with multiple sclerosis is acceptable. The intraclass correlation coefficient for the total scale is 0.96 and is located between 0.91 and 0.97 for each subscale19. For mixed
population, the agreement was in the moderate to a substantial range. Four items (social arrangements, frequency of shopping, frequency of leisure activities, visiting others) were in the 0.41–0.60 range and seven items (shopping, housework, childcare, personal finances, travel outside home, having a best friend, job/school) in the 0.61 to 0.80 range. One item (leisure activities) had slight to fair (≤0.40) agreement and one item (meal preparation) had an excellent agreement (>0.80)22.Test-retest reliability with the aphasia population
was considered excellent by the authors with an intraclass correlation coefficient of 0.9615.The methodological quality of the study is sufficient for multiple
sclerosis and mixed population according to the MacDermid and COSMIN checklists but the methodological quality with the aphasia population is insufficient according to the COSMIN checklist.
Responsiveness
Responsiveness is the ability of an instrument to detect change over time 23. An
acceptable responsiveness, as measured with the receiver operating characteristic (ROC), was achieved by the Community Integration Questionnaire with multiple sclerosis population: the area under the ROC curve is 0.81