Rankings can tell us how countries and provinces compare with each other overall. Rankings tell us nothing, however, about what students can actually do. We can elicit more information from the data if we are able to describe what can be done at specific score levels. For this reason, reading achievement was divided into five levels.
As expected, the highest-ranking jurisdictions overall also tend to have the highest proportion of students at Level 5 (Figure 1.14). However, a few substantial shifts are apparent. For example, Japan has relatively few Level 5 students, although with a large proportion at Level 4 it has a high ranking on overall performance. On the other hand, Australia ranks higher on this scale than on the overall performance scale, because of a relatively high proportion of students at Level 5. Generally speaking, the positions of Canada and the provinces do not shift much.
Finland Saskatchewan Alberta Japan British Columbia Prince Edward Island Manitoba Nova Scotia Ontario CANADA Quebec Newfoundland New Brunswick Australia Mexico Sweden United Kingdom United States Switzerland Italy France Germany Russian Federation Belgium
FIGURE 1.12
Percent of Students Above 90th, 75th and 50th International Percentiles: MATHEMATICS
0 2 0 4 0 6 0 8 0 100
Percent
90th percentile (top 10%) 75th percentile (top 25%) 50th percentile (top 50%) FIGURE 1.11
Percent of Students Above 90th, 75th and 50th International Percentiles: READING
0 2 0 4 0 6 0 100
Percent
8 0
90th percentile (top 10%) 75th percentile (top 25%) 50th percentile (top 50%) Alberta
Finland British Columbia Australia CANADA Ontario Manitoba Quebec United Kingdom Saskatchewan Nova Scotia Newfoundland Prince Edward Island Belgium United States Sweden Japan New Brunswick Switzerland Germany France Italy Russian Federation Mexico
Japan Alberta Quebec Switzerland Belgium Australia United Kingdom British Columbia Manitoba CANADA Finland France Ontario Saskatchewan Sweden United States Nova Scotia Germany Russian Federation Prince Edward Island Newfoundland New Brunswick Italy Mexico
FIGURE 1.14
Reading Proficiency Scales: Percent of Students at Each Level
0 2 0 4 0 6 0 8 0 100
Below level 1 Level 1 Level 2 Level 3 Level 4 Level 5 Percent
0 2 0 4 0 6 0 100
Percent
8 0
90th percentile (top 10%) 75th percentile (top 25%) 50th percentile (top 50%) FIGURE 1.13
Percent of Students Above 90th, 75th and 50th International Percentiles: Science
Japan Alberta United Kingdom Quebec Finland Australia British Columbia CANADA Manitoba Ontario Belgium France Newfoundland Sweden United States Saskatchewan Switzerland Nova Scotia Prince Edward Island Germany New Brunswick Italy Russian Federation Mexico
Alberta Finland British Columbia Australia CANADA Ontario Manitoba Quebec United Kingdom Saskatchewan Nova Scotia Newfoundland Prince Edward Island United States Belgium Sweden Japan New Brunswick Switzerland Germany France Italy Russian Federation Mexico
Five levels of reading literacy
Reading achievement was divided into five levels.
Essentially, these levels represent the most difficult test items that a student could answer. Therefore, a student at one level could be assumed to be able to answer questions at all lower levels. To help in interpretation, these levels were linked to specific score ranges on the original scale. Because the five levels are complex to describe, an example from each level is given for the reading retrieving scale. Tasks of similar complexity were required for each level of the other reading scales.
Level 1 (score from 335 to 407)
Students were shown a notice from a personnel department about a service that would help with job mobility. They were asked to find a single explicitly stated piece of information—how to find out more about the service—which was signalled by a heading in the text that matched the term used in the question.
Level 2 (score from 408 to 480)
Students were required to state how to check that a bicycle seat was in the right position, by finding two pieces of connected information in an assembly manual. The placement of the relevant information was clearly stated in the question.
Level 3 (score from 481 to 552)
Looking at a complex international airline timetable, with prominent competing information, students had to find a single piece of information that satisfied three conditions—time, destination and connecting city. For information about one of the conditions, the reader had to refer to a separate list of abbreviations.
Level 4 (score from 553 to 626)
Presented with a relatively long, dense extract from a play, students had to use information embedded in a stage direction in order to mark the positions of two actors on a diagram of the stage.
Level 5 (score above 626)
Students were given a complex and unfamiliar set of instructions about how to make telephone calls from a hotel room, and a letter with the phone number of a friend in a different country. They were required to find and organise in correct sequence four pieces of information and to draw inferences to work out exactly how to dial the number.
Performance below level 1
Students performing below Level 1 (total reading score below 335) are not able to routinely show the most basic type of knowledge and skills that PISA seeks to measure. Such students have serious difficulties in using reading literacy as a tool to advance their knowledge and skills in other areas. Placement at this level does not mean that these students have no literacy skills.
Most of these students are able to correctly complete some of the PISA items. Their pattern of responses to the assessment is such that they would be expected to solve less than half of the tasks from a test composed of only level 1 items.
Interpreting differences in PISA reading scores
A difference of 73 points between two average scores could be thought of as representing about one proficiency level in reading literacy. A difference of one proficiency level can be considered a comparatively large difference in student performance in substantive terms. For example, on the interpreting scale, Level 3 distinguishes students who can typically integrate several parts of a text, understand a relationship or construe the meaning of a word or phrase, and can compare, contrast and categorise competing information according to a range of criteria.
At Level 2, students can be expected only to identify the main idea in a text, to understand relationships, make and apply simple categories, and construe meaning within a limited part of a text where information is not prominent but only low-level inferences are required.