Development of a progress monitoring tool for upper primary students’ reading comprehension

(1)

Development of a progress monitoring tool for upper primary students’ reading comprehension

Authors: Rielke Bogaert – Koen Aesaert - Emmelien Merchie – Hilde Van Keer Keywords:

Reading comprehension Primary education

Assessment methods and tools Student learning

Domain:

Assessment and Evaluation Specialist interest group

SIG 02 - Comprehension of Text and Graphics

Abstract

Within the context of our information society, reading comprehension is a key competence. However, many students experience difficulties with it. This is especially true in upper primary education, which is a critical period in the development of this competence. In this respect, the reading literature increasingly emphasizes the importance of monitoring students’ progress in reading comprehension.

However, appropriate measurement instruments to realize this progress monitoring are lacking. In this study, therefore, a new tool was developed to monitor upper primary students’ progress in reading comprehension (i.e., RC-PM tool), especially focusing on comprehension of expository texts. The test development process followed a multistep process consisting of a) defining the construct of reading comprehension, b) creating and operationalizing a test framework, c) developing the test items and texts, and d) reviewing the quality of the reading comprehension tests. 3269 students participated in the development of the tool. Automated test assembly was conducted in R to compile six equivalent test versions. The theoretical and educational significance of the RC-PM tool is discussed.

(2)

Extended summary

Problem statement

Reading comprehension is a key competence to participate successfully in the 21^st century information society. A critical period in the development of these reading comprehension skills is upper primary education (Keresteš et al., 2019). Unfortunately, many upper primary students struggle with reading comprehension, especially with comprehending expository texts (e.g., Rasinski, 2017). In this respect, the research interest in monitoring the progress of students’ reading comprehension is increasingly growing (e.g., Fuchs, 2017). Moreover, prior reading research indicated that progress monitoring has a high potential to promote students’ comprehension performance (e.g., Förster &

Souvignier, 2014). However, there is a lack of appropriate, theoretically and empirically supported progress monitoring tools for reading comprehension. Furthermore, the tools that are available focus mostly on reading fluency or on general reading skills instead of on reading comprehension (e.g., CBM- Maze test; Muijselaar et al., 2017). Therefore, the aim of the present study is to develop a Reading Comprehension – Progress Monitoring (RC-PM) tool for upper primary students, especially focusing on comprehending expository texts.

Method

Participants. 3269 Flemish (Belgium) upper primary school students from 167 classes in 68 schools participated in this study (51.6% fifth-graders, 48.4% sixth-graders; 49.4% girls, 50.6% boys).

Students overall mean age was 11.39 years (SD = 0.92).

Test development. The development of the RC-PM tool followed the guidelines provided in the Standards for Educational and Psychological Testing (AERA, APA, & NMCE, 2014). First, based on the comprehensive and leading construction-integration model (CI model) of Kintsch (1998, 2005), reading comprehension was clearly defined. More specifically, the test framework was developed based on the three comprehension levels distinguished in the CI model: a) surface structure (e.g., literal representation of the text), 2) textbase model (e.g., propositions at micro and macrolevel), and 3) situation model (e.g., integration of text information with prior knowledge/experiences). In view of the operationalization of this framework, example test questions were created for each comprehension level (e.g., surface model: What does the word ‘…’ mean here?). Second, short (i.e., +/- 100 words) and long (i.e., 250 – 500 words) representative expository texts were collected (n = 60), covering different subject domains of Flemish secondary education (e.g., language and culture, art and creation, sport). Eight to fifteen multiple choice (MC) questions covering the three comprehension levels were created for each text by a panel of reading experts. More particularly, three-option MC questions were opted for based on the meta-analysis of Rodriguez (2005). Furthermore, several steps were taken to guarantee the quality of the RC-PM tool. The test items were reviewed by a panel of upper primary school teachers and educational experts on reading comprehension. As to the texts, the reading fluency level as well as the comprehension level of the texts were analyzed by the Dutch center of educational measurement (CITO). Texts which were found too easy or too difficult and test items which were found to be of insufficient quality were removed. Next, one fifth-grade (n = 22) class pilot tested three texts with 30 test items in total on their comprehensibility. Finally, the best 35 texts remained, consisting of 10 short and 25 long texts with each eight to twelve MC items. Since it is

(3)

practically impossible to have students complete a reading comprehension test of 35 texts with each eight to twelve MC items, a stepwise system was used (see Figure 1). More specifically, the first two reading texts with 21 test items in total were completed by all students (n = 3269) (i.e., anchor texts).

For the other 33 texts, the participating students were divided in 20 groups of each minimum 150 students. An overlap in texts across the different groups was provided so that each text would be read by a minimum of 300 students.

Data analysis. Automated test assembly was conducted in R to develop the RC-PM tool, consisting of a set of equivalent test versions (Diao & van der Linden, 2011). The two-parameter logistic model (2PLM) was used to estimate the item parameters (e.g., item difficulty and discrimination) (Baker, 1987).

Results

Six equivalent (i.e., based on the item difficulty) test versions were compiled. Each test version consists of two to three texts with 17 to 18 test items in total. Each text and each test item are only part of one of the test versions. A detailed description of the results will be presented at the conference.

Theoretical and educational significance

This study responds to the need for a theoretically and empirically substantiated tool to monitor upper primary students’ progress in reading comprehension. Furthermore, this tool can support teachers to make substantiated instructional decisions based on their students’ specific needs.

(4)

Appendix

Figure 1. Stepwise system of the original test pool.

TEXT ^Anchortext 1 Anchor

text 2 Text 1 Text 2 Text 3 Text 4 Text 5 Text 6 Text 7 Text 8 Text 9 Text 1 Text 2 Text 3 Text 4 Text 5 Text 6 Text 7 Text 8 Text 9Text 10 Text

11 Text 12 Text

13 Text 14 Text

15 Text 16 Text

17 Text 18 Text

19 Text 20 Text

21 Text 22 Text

23 Text 24

Group 1 x x x x x

Group 4 x x x x x x

Anchor texts Short texts Long texts

(5)

References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing.

Washington, DC: Author.

Baker, F. B. (1987). Methodology Review: Item Parameter Estimation Under the One-, Two-, and Three-Parameter Logistic Models. Applied Psychological Measurement, 11(2), 111–141.

https://doi.org/10.1177/014662168701100201

Diao, Q., & van der Linden, W. J. (2011). Automated test assembly using lp_solve version 5.5 in r.

Applied Psychological Measurement, 35(5), 398–409.

https://doi.org/10.1177/0146621610392211

Förster, N., & Souvignier, E. (2014). Learning progress assessment and goal setting: Effects on reading achievement, reading motivation and reading self-concept. Learning and Instruction, 32, 91–

100. https://doi.org/10.1016/j.learninstruc.2014.02.002

Fuchs, L. S. (2017). Curriculum-Based Measurement as the Emerging Alternative: Three Decades Later. Learning Disabilities Research & Practice, 32(1), 5–7. https://doi.org/10.1111/ldrp.12127 Keresteš, G., Brkovic, I., Siegel, L. S., Tjus, T., & Hjelmquist, E. (2019). Literacy development beyond

early schooling: a 4-year follow-up study of Croatian. Reading and Writing, 32, 1955–1988.

https://doi.org/10.1007/s11145-018-9931-9

Muijselaar, M. M. L., Kendeou, P., de Jong, P. F., & van den Broek, P. W. (2017). What Does the CBM- Maze Test Measure? Scientific Studies of Reading, 21(2), 120–132.

https://doi.org/10.1080/10888438.2016.1263994

Rasinski, T. V. (2017). Readers Who Struggle: Why Many Struggle and a Modest Proposal for Improving Their Reading. Reading Teacher, 70(5), 519–524. https://doi.org/10.1002/trtr.1533 Rodriguez, M. C. (2005). Three options are optimal for multiple-choice items: A meta-analysis of 80

years of research. Educational Measurement: Issues and Practice, 24(2), 3–13.

https://doi.org/10.1111/j.1745-3992.2005.00006.x