This chapter presents the PIRLS 2011 achievement results in reading for all participating countries and Canadian provinces. First, the results of Grade 4 students in reading achievement for Canada and participating provinces will be compared to the other participating countries. The provincial results will also be compared to the Canadian average. Then the results for provinces will be presented by language for the provinces that sampled students in the English and French school systems separately.
Next, the reading performance of boys and girls across provinces will be reported. Then the results for the two main aspects of reading will be described: Purposes for Reading (i.e., reading for literary experience, and reading to acquire and use information) and Processes of Comprehension (i.e., retrieving and straightforward inferencing, and interpreting, integrating, and evaluating). Finally, for the provinces that participated in previous PIRLS assessments, results will be reported over time.
The PIRLS 2011 average scores in reading are reported on the PIRLS scale, which has a range of 0 to 1000. In the fi rst administration in 2001, the international mean was set at 500 with a standard deviation of 100. This has been used as a baseline for the subsequent administrations. In 2011, the centrepoint of the 0 to 1000 scale (i.e., 500) is used as the international reference point.
It may be misleading to compare and rank the students’ performance based on the average scores only.
When comparing the results, it is important to take into account the sampling error and the error of measurement associated with each average score. This will determine whether the differences in the average scores are statistically signifi cant (see the statistical note below for details).
20
Terminology Used in the Charts and Tables
Differences
In this report, the terms “difference” or “different,” used in the context of achievement levels, benchmarks, and percentages, refer to a difference in a technical sense. They refer to a statistically signifi cant difference. A difference is statistically different when there is no overlap of confi dence intervals between different measurements being compared. Throughout this report, average scores that are signifi cantly different from the Canadian average score are highlighted in bold.
Confi dence intervals
The purpose of PIRLS is to report results on the Grade 4 student population. A random sample of Grade 4 students was selected to write the test. The average scores were computed based on the responses of these students. Thus, the reported achievement scores provide estimates of the achievement results that would have been demonstrated if all students in the population had participated in this assessment. However, this process introduces what is known in statistical terms as a sampling error. In addition, a degree of error is associated with the scores describing student reading skills because these scores are estimated, based on student responses to test items.
This error is called the error of measurement. Because an estimate that is based on a sample is rarely exact, and because the error of measurement exists, a standard error (SE) is computed. In large-scale assessments such as PIRLS, it is a common practice when reporting mean scores to provide a range of scores within which the “true” achievement level might fall. This range of scores expressed for each average score is called a confi dence interval. A 95 per cent confi dence interval is used in this report to represent the high- and low-end points between which the actual average score should fall 95 per cent of the time (and is computed as ± 1.96 SE). It is important to consider the standard error when comparing the results among groups in order to determine if the scores are statistically different from one another.
In other words, one can be confi dent that the actual achievement level of all students would fall somewhere in the established range 19 times out of 20, if the assessment were repeated with different samples randomly drawn from the same student population. In the charts in this report, confi dence intervals are represented by the symbol . If the confi dence intervals overlap, the differences are defi ned as not statistically signifi cant. When the confi dence intervals overlapped slightly, an additional test of signifi cance (t-test) was conducted in order to determine whether the difference was statistically signifi cant.
Results in Reading for Participating Countries and Canadian Provinces
The following chart provides the average scores in reading for each participating country and
Canadian provinces in the PIRLS 2011 for Grade 4 students. In total, 45 countries participated at the Grade 4 level. These countries and Canadian provinces are listed in this chart in descending order according to the average reading achievement.
Chart 1.1 Average scores and confi dence intervals in reading for countries and Canadian provinces
H o n g K o n g S A R 300 350 400 450 500 550 600 Countries and Provinces SEAverage
Score
22
Overall, Canadian students performed quite well on PIRLS 2011, with higher achievement than most participating countries. Canada had an average score of 548, which is well above the PIRLS scale centrepoint of 500. Among all participating countries in PIRLS 2011, seven obtained an average score signifi cantly higher than Canadian students: Hong Kong SAR, Russian Federation, Finland, Singapore, Northern Ireland, United States, and Denmark. There are six countries performing as well as Canada: Croatia, Chinese Taipei, Ireland, England, Netherlands, and Czech Republic.
Most students in Canada are performing well in reading, with the average scores for all provinces being above the PIRLS centrepoint of 500. In relation to the Canadian average, British Columbia performed above the Canadian average, while Ontario, Nova Scotia, Alberta, and Newfoundland and Labrador performed at the Canadian average. The average scores for Quebec and New Brunswick French are signifi cantly lower than that of Canada overall.
Canadian Results in Reading by Language
The performance of students enrolled in English- and French-language school systems is also examined. Four provinces (British Columbia, Ontario, Quebec, and Nova Scotia) oversampled these populations separately in order to examine the difference between the two language groups.
The table below presents the average scores and differences in the reading performance for students enrolled in English- and French-language school systems.
Table 1.1 Average scores and differences in reading by language English-language
school system
French-language school system
Difference Average score SE Average score SE
BC 556 3.2 513 6.2 43
AB 548 2.8 – – –
ON 554 2.7 506 3.5 48
QC 545 3.6 537 2.4 8
NBf – – 514 2.7 –
NS 551 2.5 500 3.7 51
NL 547 2.8 – – –
CAN 553 2.0 533 2.1 20
Note: Alberta and Newfoundland and Labrador did not oversample students in French; New Brunswick did not assess students in English. However, the results for these provinces are included in this table, so that they can be compared with either the Canadian English or Canadian French average score.
In all provinces that sampled by language, students enrolled in English-language schools are
performing at a level that is not statistically different from the Canadian English average. The results are different for francophones. More precisely, only Quebec students are performing at the Canadian French average; for the other provinces (i.e., British Columbia, Ontario, New Brunswick French, and Nova Scotia), the average scores of students enrolled in French-language schools are signifi cantly lower than the Canadian French average.
Overall, there is a clear pattern in the difference in reading results between students enrolled in the English-language school systems and those in the French-language school systems. In most provinces, students enrolled in a majority-language school system signifi cantly outperform those enrolled in a minority-language school system. The differences for these provinces range from 43 points to 51 points. In Quebec, there is no signifi cant difference in the student performance between these two groups.
Canadian Results in Reading by Sex
Results obtained from multiple studies have shown that girls usually perform better than boys in reading. This was the case in PISA 200911 and PCAP 200712 (when reading was the major domain), and the differences in the results were noticed in all Canadian provinces. Table 1.2 shows the average scores and differences in PIRLS 2011 reading results by sex for each participating province.
Table 1.2 Average scores and differences in reading by sex
Girls Boys
Difference Average score SE Average score SE
BC 564 3.5 548 3.7 16
AB 553 3.1 543 3.1 10
ON 558 3.1 546 2.8 13
QC 544 2.6 531 2.4 14
NBf 520 3.5 507 4.4 13
NS 556 2.6 543 2.8 13
NL 555 3.1 538 3.1 16
24
In line with the previous studies, the PIRLS 2011 results demonstrate that girls continue to perform better than boys in reading, and this is consistent across Canadian provinces. In Canada overall, girls are outperforming boys by 12 points. Across provinces, the difference in the average scores gap ranges from 10 points in Alberta to 16 points in British Columbia and Newfoundland and Labrador.
This pattern is also consistent across almost all participating countries. The difference in the average scores between boys and girls is smaller in Canada than the international average (12 vs. 16 points).
Canadian Results for Reading Purposes and Comprehension Processes
As described in the introduction of this report, the two aspects of students’ reading literacy — Reading Purpose and Comprehension Processes — represent the foundation of the PIRLS reading assessment.
For the fi rst aspect, Reading Purpose, PIRLS focuses on two scales: literary reading and informational reading. A total of fi ve literary passages and fi ve informational passages made up the PIRLS 2011 assessment. The literary texts were fi ctional stories, and the information passages included various types of content and organizational structures.
For the second aspect, PIRLS assessed four major Comprehension Processes: focusing on and retrieving explicitly stated information; making straightforward inferences; interpreting and
integrating ideas and information; and fi nally, examining and evaluating content, language, and textual elements. Owing to the low number of items in each process, results have been aggregated to two combined process scales only:
1. Retrieving and straightforward inferencing: this scale combines “focusing on and retrieving explicitly stated information” and “making straightforward inferences”;
2. Interpreting, integrating, and evaluating: this scale combines “interpreting and integrating ideas and information” and “examining and evaluating content, language, and textual elements.”
Previous PIRLS assessments have shown that most countries tend to perform relatively better in either
“literary reading” or “informational reading” for the fi rst aspect (reading purpose) and, similarly, in either “retrieving and straightforward inferencing” or “interpreting, integrating, and evaluating for the second aspect” (comprehension processes) (Mullis et al., 2012). In this context, it is important to examine Canadian results for each aspect and to compare results between different scales. Thus, in what follows, results and differences for the two aspects described above will be reported: Reading Purpose (literary reading and informational reading) and Comprehension Processes (retrieving and straightforward inferencing, and interpreting, integrating, and evaluating).
Achievement of Canadian Students by Reading Purpose — Canada and the Provinces
At the international level, countries with the highest average scores for reading overall also obtain the highest average scores in both literary and informational reading, compared to other countries. Also, within each country, most countries obtain a relatively higher average score in either literary reading or informational reading. For example, Hong Kong SAR and Chinese Taipei are performing better in informational reading than in literary reading. Conversely, Northern Ireland, United States, and Canada are performing better in literary reading than in informational reading.
Chart 1.2 illustrates the results for literary reading and informational reading for Canada and the provinces.
Chart 1.2 Average scores in literary and informational reading
Results for Canada overall show that students are performing signifi cantly better in literary reading than in informational reading. This pattern of higher achievement in literary reading is also consistent across provinces.
561552 552545 558549 539536 516511 555545 552543 553545 500
520 540 560 580 600
BC AB ON QC NBf NS NL CAN
Literary Reading InformaƟonal Reading
26
Achievement of Canadian Students in Reading Purposes, by Sex
Chart 1.3 presents the results by sex for the two reading purposes, literary and informational, for Canada overall.
Chart 1.3 Average scores in literary and informational reading, by sex
The results provide interesting fi ndings in that Canadian girls are performing better in literary reading than informational reading, while there is no signifi cant difference between the two reading purposes for Canadian boys. In both literary and informational reading, girls outperform boys by 18 and 7 points respectively. Internationally, girls also outperform boys on both literary and informational reading.
Achievement of Canadian Students in Comprehension Processes — Canada and the Provinces
Internationally, most of the top performing countries are performing signifi cantly better in the interpreting, integrating, and evaluating process than in reading overall. For instance, the difference was signifi cant for eight of the twelve highest performing countries: Canada, Hong Kong SAR, Russian Federation, Singapore, Northern Ireland, United States, Chinese Taipei, and England.
542 544
549 562
500 520 540 560 580 600
InformaƟonal ReadingLiterary Reading
Girls Boys
Chart 1.4 presents the results by comprehension process for Canada and the provinces.
Chart 1.4 Average scores in retrieving and straightforward inferencing, and in interpreting, integrating, and evaluating
550561 542554 545559 538538 514513 543555 540553 543554 500
520 540 560 580 600
BC AB ON QC NBf NS NL CAN
Retrieving and Straighƞorward Inferencing InterpreƟng, IntegraƟng, and EvaluaƟng
There are signifi cant differences between the two scales for comprehension processes in Canada overall. Generally, as is the case internationally, average scores in the interpreting, integrating, and evaluating process tend to be higher across provinces.
Achievement of Canadian Students in Comprehension Processes, by Sex
As seen previously, students tend to perform better in interpreting, integrating, and evaluating than in retrieving and straightforward inferencing. Chart 1.5 presents the results for interpreting, integrating, and evaluating, and for retrieving and straightforward inferencing, by sex.
28
Chart 1.5 Average scores in retrieving and straightforward inferencing, and in interpreting, integrating, and evaluating, by sex
The results demonstrate that both girls and boys in Canada are performing better in interpreting, integrating, and evaluating than in retrieving and straightforward inferencing. As for the sex
differences, girls outperform boys on both scales, which is consistent with the international average.
Trends in Reading Achievement
Canadian participation in large-scale assessments allows meaningful comparisons with other countries. It also provides valuable information about important features of our education systems.
Ministries and departments of education consider results from these assessments and other contextual information when making political decisions to improve their education system. As many decisions and changes are implemented over time (e.g., changes to the curriculum) based in part on the results of large-scale assessments, it is important to report the results over time. In this section, the trends in reading achievement are reported for those provinces that participated in previous PIRLS assessments (PIRLS 2001 and/or PIRLS 2006). More precisely, two 5-year comparisons (2001 vs. 2006 vs. 2011) can be made for two provinces, Ontario and Quebec, since they both started their participation in 2001. One fi ve-year comparison (2006 vs. 2011) can be made for three provinces, British Columbia, Alberta, and Nova Scotia, which joined PIRLS in 2006. No results over time can be reported for New Brunswick French, Newfoundland and Labrador, or Canada overall, since they participated in PIRLS for the fi rst time in 2011.
548 538
560 549
500 520 540 560 580 600
Interpreting, Integrating, and EvaluatingRetrieving and Straightforward Inferencing
Girls Boys
Table 1.3 compares the results in reading for PIRLS 2001, 2006, and 2011 for the fi ve provinces mentioned above.
Table 1.3 Comparison of results in reading between PIRLS 2001, 2006, and 2011
2001 2006 2011 Difference with 201113
Average
score SE Average
score SE Average
score SE 2001 2006
BC – – 558 2.6 556 3.2 – -2
AB – – 560 2.4 548 2.9 – -12
ON 548 3.3 555 2.7 552 2.6 4 -3
QC 537 3.0 533 2.8 538 2.1 0 5
NS – – 542 2.2 549 2.4 – 7
Ontario and Quebec have participated in PIRLS since 2001, and their performance in reading has remained stable throughout the years. There are no signifi cant differences in the average scores between 2011 and previous years. For the three provinces that participated in PIRLS 2006 and PIRLS 2011, the performance in reading remained the same for British Columbia, while results decreased by 12 points for Alberta and increased by 7 points for Nova Scotia over the past fi ve years.