7. Supporting countries to produce learning data for Indicator 4.1.1
7.2 How to implement a learning assessment in my country?
Different learning assessments have different purposes and characteristics. Countries aiming to introduce learning assessments should be aware of these differences and should select the assessment that best fits their national education goals, needs and resources. In all likelihood, more than one type of assessment may be needed, especially if one counts both highly-formal assessments and less-formal assessments used in the classroom or by ministry providers of quality assurance services to schools.
Learning assessments may measure different subject areas at different levels and grades of the education cycle. They may vary in the frequency of administration and on the costs of implementing the assessment. National assessments are usually better-fitted for measuring the national curriculum, whereas regional and cross-national assessments
Immigration status Disability Indigenous background Wealth Location Language spoken at home Sex
Age 20
20 18 16 14 10 8 7
Figure 7.4 Availability of disaggregated data out of a total of 20 assessments
Box 7.2 How do learning assessments define location?
Slight distinctions appear when examining school locations. Large-scale assessments provide no information about the location of students’ homes. They only provide basic information regarding the location of schools.
Another issue relates to the definition of rural and urban areas. In some assessments, the distinction between locations of schools is based on the number of people living in the area, while in other assessments the definition is more subjective.
In PISA, information concerning the definition of location is provided in the questionnaire to avoid potential misunderstanding of rural/urban areas. On the contrary, the distinction between locations of schools is more complicated for PASEC and SACMEQ assessments. This difficulty may be explained by the geographical structure of sub-Saharan African countries. For instance, the questionnaire asks school directors if the school is located in “a city”, “a suburb of a large city”, “a big town” or “a small town”. It may be complicated to differentiate between “a big town” and “a suburb of a large city”. The same observation can be made for SACMEQ, where it is not clear how to differentiate between an “isolated area” and a “rural area”.
Source: UNESCO Institute for Statistics (UIS).
Source: UNESCO Institute for Statistics (UIS), http://uis- azr-prod-wordpress-eus1.azurewebsites.net/gaml/capacity-development/
allow for making international comparisons. Regional assessments are available for countries that usually share common geographic, cultural, linguistic or historical backgrounds; they are usually available in the subject areas and grades that are considered a priority for those regions. Cross-national assessments have a more global presence and their results have been considered for reporting in the beginning stages of SDG 4 (see Chapter 2). As methodologies are developed, all assessments will report to SDG 4 in a harmonised way so that they are comparable.
7.2.1 Options for implementing a national assessment
As shown in Figure 7.5, countries that decide to conduct school-based national assessments can follow different strategies. One strategy is to develop
a brand new assessment. This strategy is the most common and has the advantage of greater ownership by stakeholders. A new assessment usually ensures better alignment with the national curriculum, which is important when reporting if students are reaching the curriculum objectives.
Another strategy is to adapt a national assessment already being used in another country. For instance, Mozambique put in place its national assessment by adapting an assessment programme from Brazil (Provinha Brasil) to measure reading in the first cycle of primary education. This South-South collaboration saved Mozambique time and resources by not having to “reinvent the wheel”.
A third strategy that countries should consider is to adapt learning assessments that are freely available
Secondary education:
PISA: Reading, mathematics, science (15-year-old students) Primary education:
PIRLS: Reading (Grade 4) TIMSS: Mathematics
and science (Grades 4 and 8)
Figure 7.5 Options to consider when deciding what type of learning assessment to implement
Develop a brand new assessment
Africa:
PASEC: Language and mathematics
(Grades 2 and 6) SACMEQ: Language,
mathematics, health (Grade 6) Adapt a national
assessment from another country National
assessment Cross-national
assessment Regional
assessment
Source: Ramirez, 2018a.
Adapt free tools from other assessments
Latin America:
LLECE: Language, mathematics, science
(Grades 3 and 6)
Pacific Islands:
PILNA: Language and mathematics
(Grades 4 and 6)
Southeast Asia:
SEA-PLM: Language, mathematics, citizenship (Grade 5)
Figure 1.1 Interim reporting of SDG 4 indicators
online, i.e. that are part of the public domain.
Instruments and procedures from assessment programmes, such as EGRA and EGMA, ASER and Uwezo, can be used to monitor learning in mathematics/numeracy and language/literacy, and are available in different languages. For example, Gambia has been administering its own local adaptation of EGRA/EGMA to nationally-representative samples of students. Pakistan has been administering its refined version of the ASER household test annually to nationally-representative samples of children and youth. This approach has the advantage of offering free and ready-made tools for measuring learning. The disadvantage is less ownership by stakeholders and less flexibility to address national curriculum considerations. One possible disadvantage of simply borrowing downloaded tools is the loss of the technical assistance and quality assurance that accompanies the development of such tools.
However, this kind of assistance can be obtained cheaply or for free through bilateral or multilateral development agencies.
Different assessments can complement each other.
For example, a country may administer a national assessment in Grade 3 and a cross-national assessment in Grade 6. However, given scare resources, countries may have to opt for one or the other. To economise, the country may use a much more informal assessment, still capable of producing useful information but not of the accuracy of the Grade 6 assessment, at Grade 3 level. (However, minimum levels of reliability and validity need to be assured.)
Figure 7.5 shows the options a country may consider when deciding what type of learning assessment to implement. The first decision concerns whether to conduct a national, regional or cross-national assessment. There are pros and cons for each of these assessment types (see Table 7.1), and countries should weigh them according to their own local context.
7.2.2 Key stages in implementing a learning assessment
Figure 7.6 presents the key stages of the
assessment cycle that should be taken into account during implementation. Most of these stages apply to all assessment types (national, regional or cross-national), with different emphasis and somewhat different activities. Implementing the entire assessment cycle may take around three years, although this varies considerably from country to country.
To inform Indicator 4.1.1, countries will need to provide evidence that each one of these stages was implemented, meeting technical criteria.
7.2.3 What are the alternative institutional arrangements for a learning assessment unit?
Different institutional arrangements are possible to implement a national, regional or cross-national assessment. Common arrangements include:
m Unit within the ministry or department of education.
This is probably the most typical arrangement.
Having the learning assessment unit within the ministry or department has the advantage of facilitating coordination among curriculum, pedagogy and assessment teams. This facilitates alignment among these components and uses of the assessment results. It has the disadvantage of being more vulnerable to political interference and corruption from high levels or from colleagues and peers including teachers (e.g. not publishing or altering poor results).
m Semi-autonomous public institution. Several countries have national institutes of statistics, research centres or quality assurance agencies leading national, regional or cross-national assessments. These institutions have their own budget and are accountable to the minister of education or congress. They have the advantage
Table 7.1 Pros (+) and cons (-) of national assessments vis-à-vis cross-national and regional assessments
National assessments Cross-national and
regional assessments Politics
(-) More likely to be affected by country politics.
Results may not be published or they may not be trusted.
(+) Independent of country politics. Results are more likely to be trusted
Local stakeholders (+) Allows for involving local stakeholders in the
assessment and therefore more likely to ensure their support.
(-) Local stakeholders are less involved in the assessment and therefore may be less likely to support it.
Curriculum (+) Usually more aligned with the national curriculum and its learning objectives.
(-) Usually less aligned with the national curriculum and its learning objectives.
Capacity (-) The local team may not have access to appropriate
training to implement the assessment. (+) The local team can benefit from high quality, hands-on training in each step of the assessment.
Very valuable to build local capacity.
Costs (+) May be cheaper than a cross-national assessment.
Countries need to cost for test development, data analysis and reporting.
(-) May be more expensive than a national
assessment. Countries need to cost for participation fees and travels, and assessment implementation.
Source: UIS, 2017d.
Box 7.3 Main challenges when conducting a large-scale assessment
m Failure to secure political support and stable funding. As a consequence, the assessment stability is at risk.
Involving stakeholders and transparency are essential to minimise this risk.
m Need to secure sufficient staff. It is important to agree on the number of staff and the amount of time they put into the assessment, and to plan the assessment accordingly.
m Need to develop local capacity. The best way to do so is by providing hands-on training while implementing an assessment. Efforts should be made to retain the trained staff.
m Poor sampling. Sampling and fieldwork should be planned in detail and well in advance.
m Lack of standardised procedures. Manuals, training and quality control procedures are important tools to ensure standardisation.
m Assessment results are not comparable. Major issue when the aim is to report changes in learning across years.
In design, make sure it has the technical features needed for comparing results.
m Assessment results are not published. Planning for a communication strategy where poor results are used as a baseline to promote improvements.
m Lack of an assessment culture. Produce simple reports, flyers and websites that address a few key research questions, and offer workshops to explain results and the assessment in general.
Source: Ramirez, 2018a.
Figure 7.6 Stages and activities typically needed to implement a learning assessment
of being more independent of political or collegial influence. The risk is a lack of coordination and misalignment of the assessment with other components and policies of the education system (e.g. the assessment team not communicating with the curriculum team to ensure that the tests measure the curriculum objectives).
m Examination board or unit. Countries that have examinations for certification (e.g. secondary school diploma) or selection purposes (e.g.
university entrance examinations) may benefit from having the same institution in charge of national, regional or cross-national assessments. The advantage is the benefit of the institutional capacity and expertise of the examination team. A disadvantage is the possibility of overwhelming an institution that already has a clear mandate.
m Outsourcing to a university, NGO or equivalent.
Some countries outsource the implementation of their national, regional or cross-national assessments. They do so by forming strategic alliances of five or ten years, or by signing contracts with one or more institution to be in charge of the whole or a part (e.g. field operation) of the assessment. This arrangement is more common for regional and cross-national assessments. In national assessments, it is used more often during the initial introduction of the assessment or during the first years of its implementation (e.g. a university is in charge of a national pilot assessment). A limitation for low- and middle-income countries is the lack of local institutions with the technical capacity to implement the assessment.
Figure 7.6 Stages and activities typically needed to implement a learning assessment
Specifies why, what, who, how, when... to assess.
Test and questionnaire specifications stating the content, skills, competencies to measure and the background variables to measure.
Includes item writing/adaptation/translation, piloting and pychometric analyses, defining proficiency levels, design of final instruments and printing.
Sampling, field operation plan, manuals, recruiting and training administrators and supervisors, logistics, contacting schools and administration of instruments.
Data capturing and cleaning, psychometric analyses, scaling, standard setting, data analysis, computation of test and questionnaire results.
Specify policy questions to be answered; design results reports, brochures, videos, media toolkit and other; disseminate results; offer training, seminars and workshops to ensure that stakeholders have access to, understand, value and use results and information.
Assessment framework
Instrument development
Field operation
Data processing
Communication
Source: Ramirez, 2018a.
Year 1 Year 2 Year 3
Regardless of the arrangement, the institution leading the assessment should be accountable to a clearly recognisable body (e.g. the minister of education, congress or a national education commission) that is itself accountable.91
7.2.4 How much does a learning assessment cost?
A national learning assessment may cost between US$200,000 and US$1,000,000, depending on several factors, such as target population (in-school versus out-of-school children/youth), number of students tested, administration mode (e.g. group versus individual administration), local costs of services (e.g. printing) and personnel (e.g. test administrators). It is important to estimate total costs and secure sufficient and stable funding (e.g.
from government and donors). Perhaps the most important cost factor is whether the sampling frame of the assessment allows specific inference about performance of sub-national jurisdictions (states, provinces) given that a larger sample size is needed for valid results at the sub-national level. In other words, sometimes the results are valid for the national aggregate and allow statistical and policy inferences at that level, but the sampling does not allow a conclusion for some disaggregation, either at the sub-national level or according to another classification.
The cost of regional and cross-national assessments is likely to be around USD$1,500,000. Again, this may vary greatly depending on the assessment programme, implementation plan and local costs (UIS, 2018a).
When comparing the financial costs of the large-scale assessment to the cost of running an education system for an “average” country, we realise that a large-scale assessment is an investment. According to the UIS database, the average cost for low- and middle-income countries to run their pre-primary to secondary education system is about US$5.8 billion
91 See UIS, 2018a.
per year. If we assume (as studies have shown) that education systems have at least 10% inefficiency, the average cost of inefficiencies in a country would be around US$580 million per year. If 5% of this inefficiency, in a conservative scenario, is addressed by having and properly using learning assessment data, then the benefit is about US$30 million per year in an average country.
With an estimated annual cost of US$250,000 for two assessments every four years, the benefit/cost ratio would be 30/0.25=US$120 million per year. This exercise would produce stunning results at the global level. For every 100 countries that invest, the benefits are clear (100 * US$120 million per year). In other terms, the approximately US$1 million invested every four years for an assessment (or US$250,000 per year) amounts to just one-tenth of 1% of the running costs for the entire education system (UIS, 2018a).
Any modern organization spends at least that much on quality control systems, relative to its revenue.