The average scores reported in the previous section provide a useful but limited way of comparing performance of different groups of students. Another way to compare performance is to examine the proportions of students who can accomplish tasks at various proficiency levels. This kind of analysis allows a further breakdown of average scores and an examination of groups of students who show similar abilities. In PISA, science proficiency is a continuum – that is, science proficiency is not something a student has or does not have, but rather every 15-year-old shows a certain level of proficiency. The science proficiency levels used in
PISA 2006 are described in the text box 5 ‘Science Proficiency levels’.
Chart 1.6 (based on data from Appendix table B.1.9) shows the distribution of students by proficiency level by country and includes the Canadian provinces. Results for countries and provinces are presented in descending order according to the proportion of 15-year-olds who performed at Level 2 or higher. The OECD defined Level 2 proficiency as a baseline as it represents a critical level of science literacy on the PISA test. It is at Level 2 that students begin to demonstrate the kind of science knowledge and skills that enable them to actively and effectively use science competencies.
Chart 1.5
Difference in average scores in science between students who performed in the bottom quarter of performance and students who performed in the
top quarter of performance
score points difference (top quarter-bottom quarter) score points difference (top quarter-bottom quarter)
115 120 125 130 135 140
115 120 125 130 135 140
N.L. P.E.I. N.S. N.B. Que. Ont. Man. Sask. Alta. B.C. Can.
Chapter 1 / The performance of Canadian students in mathematics in an international context Chart 1.6
Percentage of students at each level of proficiency on the combined science scale
Finland
Level 2 Level 3 Level 4 Level 5 Level 6
Box 5
Science proficiency levels
Science achievement was divided into six proficiency levels representing a group of tasks of increasing difficulty with Level 6 as the highest and Level 1 as the lowest. Students performing below Level 1 (science score below 334.5) are not able to demonstrate routinely the most basic type of knowledge and skills that PISA seeks to measure. Such students have serious difficulties in using science literacy as a tool to advance their knowledge and skills in other areas. Placement at this level does not mean that these students have no science knowledge and skills. Most of these students are able to correctly complete some of the PISA items. Their pattern of responses to the assessment is such that they would be expected to solve less than half of the tasks from a test composed of only Level 1 items.
In PISA 2006, Level 2 has been identified as the ‘baseline proficiency’ level or the level of achievement on the PISA scale at which students begin to demonstrate the scientific competencies that will enable full participation in life situations related to science and technology.
Students were assigned to a proficiency level based on their probability of answering correctly the majority of items in that range of difficulty. A student at a given level could be assumed to be able to correctly answer questions at all lower levels. To help in interpretation, these levels were linked to specific score ranges on the combined science scale. Below is a description of the abilities associated with each proficiency level. (Source:
Organisation for Economic Cooperation and Development, Programme for International Student Assessment, PISA 2006).
Level 6 (score above 707.81)
At Level 6, students can consistently identify, explain and apply scientific knowledge and knowledge about science in a variety of complex life situations. They can link different information sources and explanations and use evidence from those sources to justify decisions. They clearly and consistently demonstrate advanced scientific thinking and reasoning, and they are willing to use their scientific understanding in support of solutions to unfamiliar scientific and technological situations. Students at this level can use scientific knowledge and develop arguments in support of recommendations and decisions that centre on personal, social or global situations.
Level 5 (score from 633.14 to 707.81)
At Level 5, students can identify the scientific components of many complex life situations, apply both scientific concepts and knowledge about science to these situations, and can compare, select and evaluate appropriate scientific evidence for responding to life situations. Students at this level can use well-developed inquiry abilities, link knowledge appropriately and bring critical insights to these situations. They can construct evidence-based explanations and arguments based on their critical analysis.
Level 4 (score from 558.48 to 633.14)
At Level 4, students can work effectively with situations and issues that may involve explicit phenomena requiring them to make inferences about the role of science or technology. They can select and integrate explanations from different disciplines of science or technology and link those explanations directly to aspects of life situations. Students at this level can reflect on their actions and they can communicate decisions using scientific knowledge and evidence.
Level 3 (score from 483.81 to 558.48)
At Level 3, students can identify clearly described scientific issues in a range of contexts. They can select facts and knowledge to explain phenomena and apply simple models or inquiry strategies. Students at this level can interpret and use scientific concepts from different disciplines and can apply them directly. They can develop short communications using facts and make decisions based on scientific knowledge.
Level 2 (score from 409.14 to 483.81)
At Level 2, students have adequate scientific knowledge to provide possible explanations in familiar contexts or draw conclusions based on simple investigations. They are capable of direct reasoning and making literal interpretations of the results of scientific inquiry or technological problem solving.
Level 1 (score from 334.48 to 409.14)
At Level 1, students have such a limited store of scientific knowledge that it can only be applied to a few, familiar situations. They can present scientific explanations that are obvious and follow concretely from given evidence.
Chapter 1 / The performance of Canadian students in mathematics in an international context
Using these proficiency levels, students with high and low levels of proficiency can be identified. Listed in Table 1.3 are the percentages of students who performed at Level 1 or below and the percentages of students who performed at Level 5 or 6 for each country and the ten provinces.
Students performing at Level 1 or below would have great difficulty continuing studies in science and in daily life activities involving the application of science skills. In contrast, the students performing at Level 5 or above are likely to be well qualified to do so.
Compared to the OECD average, a significantly smaller proportion of Canadian students performed at Level 1 or below in science. The Canadian proportion at Level 1 or below was almost half the proportion of the OECD average (10% versus 19% respectively). Only Finland and Estonia had a significantly smaller proportion of students at Level 1 or below than Canada.
In contrast, a significantly higher proportion of Canadian students performed at Level 5 or above in science. The OECD average was approximately 9%, six percentage points lower than the average of 15% for Canada. Two countries (Finland and New Zealand) had significantly greater percentages of students with higher skills than Canada.
Turning to the provinces, the percentages of students who performed at Level 1 or below on the combined science scale were similar to the percentage for Canada in six of the Canadian provinces (Newfoundland and Labrador, Nova Scotia, Quebec, Ontario, Manitoba and British Columbia). The percentages of students in Alberta who performed at Level 1 or below was significantly lower than the Canadian average. While the percentage of students who performed at Level 1 or below was higher that the Canadian average in Saskatchewan, New Brunswick and Prince Edward Island, the percentage was below the OECD average.
The percentages of students in Alberta at Level 5 or higher (18%) were significantly greater than the Canadian percentage (15%). The percentages of students in Newfoundland and Labrador, Quebec, Ontario, Manitoba and British Columbia who performed at Levels 5 or higher were comparable to the percentage for Canada.
Lower percentages of students in Prince Edward Island, Nova Scotia, New Brunswick and Saskatchewan performed at Level 5 or higher compared to the Canadian percentage (Table 1.3). However, the provincial percentages were statistically the same or higher than as the OECD average.
Percentage of students with low level Percentage of students with high level
proficiency (level 1 or below) proficiency (level 5 or above)
Country and province percentage Country and province percentage
Finland 4 Finland 21
Alberta 6 Alberta 18
Estonia 8 New Zealand 18
Hong Kong-China 9 British Columbia 16
British Columbia 9 Hong Kong-China 16
Ontario 9 Japan 15
Canada 10 Australia 15
Macao-China 10 Chinese Taipei 15
Korea 11 Canada 15
Quebec 11 Ontario 14
Chinese Taipei 12 Quebec 14
Nova Scotia 12 United Kingdom 14
Newfoundland and Labrador 12 Newfoundland and Labrador 14
Japan 12 Netherlands 13
Manitoba 12 Slovenia 13
Australia 13 Manitoba 13
Netherlands 13 Liechtenstein 12
Liechtenstein 13 Germany 12
Saskatchewan 14 Czech Republic 12
New Zealand 14 Estonia 12
Slovenia 14 Saskatchewan 11
Hungary 15 Switzerland 11
New Brunswick 15 Korea 10
Germany 15 Belgium 10
Ireland 15 Nova Scotia 10
Czech Republic 15 Austria 10
Switzerland 16 Prince Edward Island 10
Prince Edward Island 16 Ireland 10
Austria 16 United States 9
Sweden 16 France 8
United Kingdom 17 New Brunswick 8
Croatia 17 Sweden 8
Poland 17 Denmark 7
Belgium 17 Hungary 7
Latvia 17 Poland 7
Denmark 18 Iceland 6
Spain 20 Norway 6
Slovak Republic 20 Luxembourg 6
Lithuania 20 Slovak Republic 6
Iceland 20 Macao-China 5
Norway 21 Croatia 5
France 21 Israel 5
Luxembourg 22 Lithuania 5
Russian Federation 22 Spain 5
Greece 24 Italy 5
Table 1.3
Percentage of students with high and low proficiency levels in science, by country and province
Percentage significantly higher than Percentage not significantly different Percentage significantly lower than
the Canadian percentage from the Canadian percentage the Canadian percentage
Chapter 1 / The performance of Canadian students in mathematics in an international context
United States 24 Russian Federation 4
Portugal 24 Latvia 4
Italy 25 Greece 3
Israel 36 Portugal 3
Serbia 38 Bulgaria 3
Chile 39 Chile 2
Uruguay 42 Uruguay 2
Bulgaria 43 Turkey 1
Jordan 44 Serbia 1
Thailand 46 Brazil 1
Turkey 46 Jordan 1
Romania 47 Argentina 1
Montenegro 50 Romania 1
Mexico 51 Thailand 0
Argentina 56 Mexico 0
Colombia 60 Montenegro 0
Brazil 61 Qatar 0
Indonesia 61 Colombia 0
Tunisia 63 Tunisia 0
Azerbaijan 72 Azerbaijan 0
Qatar 79 Indonesia 0
Kyrgyzstan 86 Kyrgyzstan 0
Table 1.3 (concluded)
Percentage of students with high and low proficiency levels in science, by country and province
Percentage significantly higher than Percentage not significantly different Percentage significantly lower than
the Canadian percentage from the Canadian percentage the Canadian percentage
Percentage of students with low level Percentage of students with high level
proficiency (level 1 or below) proficiency (level 5 or above)
Country and province percentage Country and province percentage