The average scores reported in the previous section provide a useful but limited way of comparing performance of different groups of students. Another way to look at performance is to examine the proportions of students who can accomplish tasks at various proficiency or skill levels. This kind of analysis allows a further breakdown of average scores and an examination of groups of students who show similar abilities. In PISA, mathematics skill is a continuum – that is, mathematics skill is not something a student has or does not have, but
rather every 15-year-old shows a certain level of mathematics skill. The mathematics skill or proficiency levels used in PISA 2003 are described in the text box
‘Mathematics Proficiency levels’.
Figure 1.6 (based on data from Table B1.7) shows the distribution of students by skill level by country, and includes the Canadian provinces. Results for countries and provinces are presented in descending order according to the proportion of the 15-year-olds who performed at level 2 or higher.
Mathematics proficiency levels
Mathematics achievement was divided into six proficiency levels representing a group of tasks of increasing difficulty, with Level 6 as the highest and Level 1 as the lowest. Students performing below Level 1 (mathematics score below 359) are not able to show routinely the most basic type of knowledge and skills that PISA seeks to measure. Such students have serious difficulties in using mathematical literacy as a tool to advance their knowledge and skills in other areas. Placement at this level does not mean that these students have no mathematics skills. Most of these students are able to correctly complete some of the PISA items. Their pattern of responses to the assessment is such that they would be expected to solve less than half of the tasks from a test composed of only Level 1 items.
In PISA, students were assigned to a proficiency level based on their probability of answering correctly the majority of items in that range of difficulty. A student at a given level could be assumed to be able to correctly answer questions at all lower levels. To help in interpretation, these levels were linked to specific score ranges on the original scale. Below is a description of the abilities associated with each proficiency level. (Source: Organisation for Economic Cooperation and Development, Programme for International Student Assessment, PISA, 2003).
Level 6 (score above 668)
At Level 6 students can conceptualise, generalise, and utilise information based on their investigations, and modelling of complex problem situations. They can link different information sources and representations and flexibly translate among them. Students at this level are capable of advanced mathematical thinking and reasoning. These students can apply this insight and understanding along with a mastery of symbolic and formal mathematical operations and relationships to develop new approaches and strategies for attacking novel situations. Students at this level can formulate and precisely communicate their actions and reflections regarding their findings, interpretations, arguments, and the appropriateness of these to the original situations.
Level 5 (score from 607 to 668)
At Level 5 students can develop and work with models for complex situations, identifying constraints and specifying assumptions. They can select, compare, and evaluate appropriate problem solving strategies for dealing with complex problems related to these models. Students at this level can work strategically using broad, well-developed thinking and reasoning skills, appropriate linked representations, symbolic and formal characterisations, and insight pertaining to these situations. They can reflect on their actions and formulate and communicate their interpretations and reasoning.
Level 4 (score from 545 to 606)
At Level 4 students can work effectively with explicit models for complex concrete situations that may involve constraints or call for making assumptions. They can select and integrate different representations, including symbolic ones, linking them directly to aspects of real-world situations. Students at this level can utilise well-developed skills and reason flexibly, with some insight, in these contexts. They can construct and communicate explanations and arguments based on their interpretations, arguments, and actions.
Level 3 (score from 483 to 544)
At Level 3 students can execute clearly described procedures, including those that require sequential decisions. They can select and apply simple problem-solving strategies. Students at this level can interpret and use representations based on different information sources and reason directly from them. They can develop short communications reporting their interpretations, results, and reasoning.
Level 2 (score from 421 to 482)
At Level 2 students can interpret and recognise situations in contexts that require no more than direct inference. They can extract relevant information from a single source and make use of a single representational mode. Students at this level can employ basic algorithms, formulae, procedures, or conventions. They are capable of direct reasoning and of making literal interpretations of the results.
Level 1 (score from 359 to 420)
At Level 1 students can answer questions involving familiar contexts where all relevant information is present and the questions are clearly defined. They are able to identify information and to carry out routine procedures according to direct instructions in explicit situations. They can perform actions that are obvious and follow immediately from the given stimuli.
Figure 1.6
Percentage of students at each level of proficiency on the combined mathematics scale
Finland Serbia and Montenegro (Ser.) Uruguay
Using these proficiency levels, students with very high and very low levels of proficiency can be identified.
Listed in Table 1.3 are the percentages of students who performed at Level 1 or below and the percentages of students who performed at Level 5 or above for each country and the ten provinces.
The lower group includes students who would have great difficulty continuing studies in mathematics and in daily life activities involving the application of mathematics skills. In contrast, the students in the upper group are likely to be well qualified to do so.
Compared to the OECD average, a significantly smaller proportion of Canadian students performed at Level 1 or below in mathematics. The Canadian proportion at Level 1 or below was approximately half the proportion of the OECD average (10% versus 21%
respectively). Only Finland had a significantly smaller proportion of students at Level 1 or below than Canada.
In contrast, a significantly higher proportion of Canadian students performed at Level 5 or above in mathematics. The OECD average was approximately 15%, five percentage points lower than the average for Canada. Four countries (Hong Kong-China, Belgium, Liechtenstein and the Netherlands) had significantly greater percentages of students with higher skills than Canada.
Turning to the provinces, the percentages of students who performed at Level 1 or below on the combined mathematics scale were, with the exception of New Brunswick and Prince Edward Island, similar to the percentage for Canada. The percentages of students in New Brunswick performing at Level 1 or below (14%) were significantly higher than the Canadian percentage performing at Level 1 or below but lower than the percentage observed for the OECD average. The percentages of students in Prince Edward Island performing at Level 1 or below (18%) were significantly higher than the Canadian percentage performing at Level 1 or below and statistically the same as the percentage observed for the OECD average.
The proportion of students in Alberta at Level 5 or above (27%) was significantly greater than the Canadian percentage (20%). The proportion of students in Quebec, British Columbia, Manitoba, and Ontario who performed at Levels 5 or higher were comparable to the proportion for Canada.
Lower percentages of students in Newfoundland and Labrador, Prince Edward Island, Nova Scotia, New Brunswick and Saskatchewan performed at Level 5 or above compared to the Canadian percentage (Table 1.3).
However, with the exception of Prince Edward Island, the provincial percentages were statistically the same as the OECD average.
Table 1.3
Percentage of students with high level skills in mathematics and low level skills in mathematics, by country and province
Percentage of students with low level skills (Level 1 or below) Percentage of students with high level skills (Level 5 or above)
Country and province % Country and province %
Finland 7
Newfoundland and Labrador 13
Japan 13
Prince Edward Island 18
Austria 19
Serbia and Montenegro (Ser.) 42
Uruguay 48
Newfoundland and Labrador 14
New Brunswick 14
Prince Edward Island 10
United States 10
Serbia and Montenegro (Ser.) 2
Thailand 2 than the Canadian percentage
Percentage significantly lower than the Canadian percentage Percentage not significantly different
from the Canadian percentage