2. Reporting on Indicator 4.1.1
2.2 Reporting on Indicator 4.1.1
regional – and national assessments could be used to inform Indicator 4.1.1. Naturally, the cross-national assessments (aiming at both global and regional coverage) have been first in line to report for SDG 4.1.1 as they are designed for cross-national comparisons, measure common subject areas or assessment domains (minimum core denominator) and are expressed on a common scale. Unfortunately many regions do not have a regional assessment, nor have the countries joined any cross-national initiative.
This represents a challenge if the options are to be restricted to cross-national assessments.
2.2.1 How global and regional
assessments are distributed globally
Figures 2.2 and 2.3 map the current distribution of assessments by category. In terms of subjects assessed, reading and mathematics are the most common areas of study. As previously explained, all cross-national and national assessments could be used to inform Indicator 4.1.1. In addition, household-survey based assessments, in general those measuring foundational skills such as Multiple Indicator Cluster Surveys (MICS), Early Grade Reading Assessment (EGRA) and Early Grade Mathematics Assessment (EGMA) can be used as well.9According to UIS estimates, 80%of countries have conducted a national learning assessment or participated in a cross-national initiative in the last five years (UIS, 2016). This represents a significant increase in the number of student assessments undertaken globally over the past decade. This increase is largely due to the growing number of countries interested in monitoring their progress in a regional context, leading to a rapid growth of regional assessments during this period. However, due to differences
9 See Treviño and Ordenes, 2017.
Box 2.1 Where and how to find SDG 4 data
m The Quick Guide to Education Indicators for SDG 4 describes the process of developing and producing the global monitoring indicators while explaining how they can be interpreted and used. This is a hands-on, step-by-step guide for anyone who is working on gathering or analysing education data.
m The SDG 4 Data Book: Global Education Indicators 2018 ensures that readers have the latest available data for the global monitoring indicators at their fingertips, and will be regularly updated.
m The SDG 4 Data Explorer, displays data by country, region or year; by data source; and by sex, location and wealth. It allows users to explore the measures of equality that are crucial for the achievement of SDG 4.
m UIS.Stat is the world’s most comprehensive database on education. It enables users to search and extract data from across UIS’s many databases.
m The SDG 4 database contains data on key indicators needed for global monitoring, including data on learning outcomes. It presents the assessment undertaken by each country as well as the share of children who reached minimum proficiency levels in reading and mathematics.
Source: UNESCO Institute for Statistics (UIS).
Figure 2.2 In-school based assessment
Figure 2.3 Foundational skills assessments – countries implement-ing household-based assessments in basic education
Figure 2.2 School-based assessment
Note: Areas shaded in orange correspond to the existence of national assessments.
Source: UNESCO Institute for Statistics (UIS).
Figure 2.3 Foundational skills assessments – countries implementing household-based assessments in basic education
Source: UNESCO Institute for Statistics (UIS).
PISA TIMSS PIRLS LLECE PASEC SACMEQ PILNA SEA-PLM
EGRA EGMA MICS Citizen-led Young Lives
in the measurement constructs and frameworks, these assessments are not always comparable across countries and many technical challenges remain. Thus, it is difficult at this stage to compare countries in terms of learning achievement across regions due to the lack of comparability using the same scale.
Knowing what currently exists in countries in terms of assessment systems is important for charting a way forward for Indicator 4.1.1 and finding a feasible, cost-effective way of reporting. Indicator 4.1.1.a – that is, for early grades – is classified in Tier III, while Indicators 4.1.1.b and c – that is, primary and lower
secondary levels – are classified in Tier II. Expanding the linking and reporting options could be the way forward to upgrading the sections of the indicator in Tiers II and III. 10
2.2.2 Understanding the current configuration of school-based assessments
For the purpose of analysis, and given their relevance for each of the three educational levels of Indicator 4.1.1, we will limit our discussion to school-based assessments. It is useful to consider four types of school-based assessments, each offering specific opportunities and challenges: i) the three large international programmes (PISA, TIMSS and PIRLS); ii) the five regional cross-national assessments (LLECE, PASEC, PILNA, SACMEQ and SEA-PLM);
iii) national assessments for monitoring purposes (either sample- or census-based); and iv) national examinations for certification or selection purposes.
Figures 2.4 and 2.5 focus on differentiating coverage in terms of the three educational levels and the four types of assessments with two criteria of coverage (number of countries and population). Figure 2.4 focuses on the number of countries by region and level regardless of the size of the countries. On the other hand, Figure 2.5 refers to population coverage.
As the SDG indicators follow a tier classification - that means all regions need a reasonable coverage - the analysis is presented in terms of regions and not as a particular assessment.
Figure 2.4 indicates that the three global assessments provide the best coverage at the lower secondary level if only international assessments are considered.
Yet even here, fewer than one-half of the world’s countries are covered, though participating countries represent 76% of the world’s population (as shown in Figure 2.5). Adding the five regional programmes expands coverage for the end of primary education.
10 Tier 2: Indicator is conceptually clear, internationally-established methodology and standards are available, but data are not regularly produced by countries.
Tier 3: No internationally-established methodology or standards are yet available for the indicator, but methodology/standards are being (or will be) developed or tested.
Table 2.1 Options for reporting on Indicator 4.1.1 School-based End of primary
education
EGMA: Early Grade Mathematics Assessment EGRA: Early Grade Reading Assessment
LLECE: Latin American Laboratory for Assessment of the Quality of Education MICS6: Multiple Indicator Cluster Surveys, Round 6
PAL: People’s Action for Learning
PASEC: Programme d’analyse des systems éducatifs de la CONFEMEN (Programme of Analysis of Education Systems of CONFEMEN)
PILNA: Pacific Islands Literacy and Numeracy Assessment PIRLS: Progress in International Reading Literacy Study PISA: Programme for International Student Assessment
PISA-D: Programme for International Student Assessment for Development SACMEQ: Southern and Eastern Africa Consortium for Monitoring Educational Quality SEA-PLM: Southeast Asia Primary Learning Metrics
TIMSS: Trends in International Mathematics and Science Study Source: UNESCO Institute for Statistics (UIS).
Figure 2.4 Country coverage by type of assessment
Figure 2.5 Population coverage by type of assessment and region
Europe & North America Pacific
North Africa & West Asia Sub-Saharan Africa East and South East Asia
South Asia Latin America & Caribbean
Caucasus & Central Asia
Countries Countries International assessments (PISA, TIMSS, and PIRLS)
World 223
countries Early grades
62 countries End of primary
0 countries End of lower secondary 84 countries
World 223
countries Early grades
135 countries End of primary
101 countries End of lower secondary 108 countries
World 223
countries Early grades
106 countries End of primary
62 countries End of lower secondary 84 countries
World 223
countries Early grades
135 countries End of primary
110 countries End of lower secondary 129 countries
….plus national assessments
….plus regional assessments ….plus national examinations
Figure 2.4 Country coverage by type of assessment
Source: Gustafsson, 2018.
Europe & North America Pacific
North Africa & West Asia Sub-Saharan Africa East and South East Asia
South Asia Latin America & Caribbean
Caucasus & Central Asia Source: Gustafsson, 2018.
Population Population International assessments (PISA, TIMSS, and PIRLS)
World
0 students End of lower secondary
….plus regional assessments ….plus national examinations
Figure 2.5 Population coverage by type of assessment and region
Very large gains are visible following the addition of national assessments, though it is not clear if all meet the minimum procedural quality or have enough alignment in terms of content coverage to make them comparable. This would be a general problem unless some minimum content coverage is developed.
However, they could still report Indicator 4.1.1 with footnotes referring to the shortcomings that could serve as a warning in terms of comparability.
Catering for the examinations of countries adds relatively little coverage, though at the lower
secondary level the 7 percentage point gain (86% to 93%) is substantial.
2.2.3 Why is it relevant to define the minimum level?
The Education 2030 Framework for Action commits all countries to establish benchmarks for measuring progress towards SDG 4 targets. In response, the UIS has proposed the development of scales that describe the progression of learning skills and thereby help countries to identify and agree on the benchmarks needed to define minimum proficiency levels for reporting purposes (see Section 2.3.2).
It is important to recognise that several years will be required to resolve all of the methodological and political issues needed to report on SDG indicators on the same scale. The challenges are primarily due to the fact that learning assessment initiatives use different definitions of performance levels, while discussions continue on an interim reporting strategy. The UIS has published a database that links different assessments to the same scale and to report on Indicator 4.1.1 using two alternative benchmarks.
The two benchmarks belong to two different assessments that reflect the contexts of countries with different income levels. For example, SACMEQ is a regional survey used to assess students at the end of primary school. The decision was therefore made to use the SACMEQ benchmark (referred to as the basic proficiency level) for reading and mathematics at the primary level for all countries in the database (see Box 2.2).
In addition, the database includes results using the minimum proficiency level defined by the International Association for Evaluation of Educational Achievement (IEA) for the Progress in International Reading Literacy Study (PIRLS) and Trends in International Mathematics and
Box 2.2 What are children expected to know at the primary level?
According to the SACMEQ benchmarks, children in Grade 6 who have achieved the minimum proficiency level in reading can “interpret meaning (by matching words and phrases completing a sentence, matching adjacent words) in a short and simple text by reading forwards or backwards” (SACMEQ III).
In mathematics, students can “translate verbal information (presented in a sentence, simple graph or table using one arithmetic operation) in several repeated steps”. Moreover, he/she “translates graphical information into fractions, interprets place value of whole numbers up to thousands and interprets simple common everyday units of measurement” (Hungi et al., 2010).
The IEA benchmarks used in PIRLS and TIMSS are more demanding. For example, “when reading Informational Texts, students can locate and reproduce explicitly stated information that is at the beginning of the text” (Mullis et al., 2012). For mathematics, “students can add and subtract whole numbers. They have some recognition of parallel and perpendicular lines, familiar geometric shapes and coordinate maps. They can read and complete simple bar graphs and tables” (Mullis et al., 2016b).
Source: UIS, 2017g.
Figure 2.6 Proportion of students not reaching the basic and mini-mum proficiency levels in reading by SDG region
Science Study (TIMSS). Both of these international assessments have global coverage primarily involving middle- and high-income countries.
Figure 2.6 shows the percentage of primary and lower secondary students not achieving the basic proficiency level and the minimum proficiency level.
The minimum proficiency level is more difficult and requires a higher level of skills and concepts, which explains why fewer students are achieving it. Hence, less children achieve the minimum proficiency level than the basic level.
It is also important to note the variation in rates between regions. The change in percentage of students below the basic and the minimum proficiency level is not linear. Linearity could occur if there were a similar distribution of pupils for all possible scores between countries. A high proportion of students concentrated around the basic proficiency level implies that a minor change in the levels of the threshold to the minimum proficiency level will produce a dramatic reduction in the proportion of children who reach minimum proficiency levels. There are regions with a high proportion of children with very basic sets of skills for whom the minimum proficiency
level is too high a bar. This explains why such a high proportion is not reaching the benchmark.
The differences in the results highlight the need to accelerate discussions on benchmarks. Is it possible to define appropriate benchmarks for all? There is a clear need to define concepts as well as to examine the feasibility and utility of setting benchmarks at different levels of monitoring. Both the technical and political aspects of the process must be taken into account in these discussions.