Developing evaluation capacity and action research in Africa

CAPACITY AND ACTION RESEARCH IN AFRICA

This section reports on an evaluation capacity development model in the context of the emerging,

85 This section is based on Bolly, 2018.

results-oriented culture in Africa. It is not meant to be prescriptive, but focuses on the relevant elements of this process and the experience of Recherche-action sur la mesure des apprentissages des bénéficiaires des programmes d’alphabétisation (Action Research on Measuring Literacy Programme Participants’

Learning Outcomes) (RAMAA).

RAMAA was initiated by the UIL at the request of certain African countries. It focuses on the field of non-formal education and aims to provide policymakers and development partners with reliable and comparable data adapted to the quality of literacy programmes.⁸⁶

Its objective is to assist countries in setting up a system for monitoring and evaluating the quality of

86 Literacy programmes refer to organized learning arrangements that target young people and adults who are illiterate or have low literacy skills.

Female Male Aged 15-39 Aged 40-64 More than

primary

Primary or less

%

0 10 20 30 40 50 60 70 80 90 100

Armenia Ukraine Georgia Serbia Viet Nam Colombia Bolivia Kenya Ghana Armenia Ukraine Serbia Georgia Viet Nam Colombia Bolivia Kenya Ghana Armenia Ukraine Viet Nam Serbia Colombia Georgia Bolivia Kenya Ghana

By gender By age group By mother's education

Figure 6.7 Percentage of working-age population who are at or above the minimum literacy threshold, by sex, age and mother's education, 2011-2016

Notes: Data are based on the latest availability from STEP Skills Measurement Program. STEP is representative of the urban population, aged 15 to 64. Those who are considered at or above the minimum literacy threshold have demonstrated literacy proficiency at Level 1 or higher.

Those at this proficiency levels can execute tasks that require the respondent to read relatively short print continuous, non-continuous or mixed texts to locate a single piece of information which is identical to or synonymous with the information given in the question or directive.

Source: STEP Skills Measurement Program, 2011-2016 (http://microdata.worldbank.org/index.php/catalog/step/about).

literacy provision. Starting from the development of a standardised methodological framework, it aims to provide a better understanding of the proficiency levels of reading, writing, mathematics and problem-solving skills acquired by youth and adults aged 15 years and older who are participating in literacy programmes. This is supported by an assessment of the determinants of quality that specify the contextual variables that explain the different outcomes between participants and countries.⁸⁷ This information

mechanism is designed to effectively and regularly guide literacy policies.

In short, RAMAA focuses on the participants of literacy programmes as part of a population. Its reports results at the global level, rather than the individual level, in order to assess the results of the literacy sector in the RAMAA countries and their evolution over time. In terms of impact, this review of literacy outcomes by RAMAA will test whether literacy programmes provide participants with a common core of basic skills, thus making literacy one of the sub-sectors of education that contributes to a more inclusive and just society. Taken in terms of quality of education, as well as skills and learning outcomes, RAMAA is contributing to Indicator 4.6.1 and broader aims of the SDGs.

RAMAA is designed in the spirit of action research;

the project itself is under development. The experience of the first phase (2011-2014), which started with five countries (Burkina Faso, Mali, Morocco, Niger and Senegal) led to the expansion to seven other countries (Benin, Côte d’Ivoire, Cameroon, Central African Republic, Chad, Democratic Republic of Congo and Togo) in the current phase (2016-2020). It was decisive for at least three reasons:⁸⁸ i) three out of five countries demonstrated strong political commitment that has resulted in substantial financial and technical mobilisation; ii) the mobilisation of national experts with multidisciplinary profiles led to a collective and

87 The determinants of quality are essential in the sense that the production function of literacy programmes is not uniform but variable in terms of populations, operators and approaches, among others.

88 See Bolly and Jonas, 2015.

successful learning dynamic; and iii) the results have helped to reshape national literacy strategies in some countries (Morocco, Niger and Senegal), as well as the potential development of master’s degrees in education sciences at national universities (e.g.

Senegal).

We have chosen to focus on modalities for the development of evaluative capacity because of its poorly documented nature, unlike the technical aspects that are widely debated in many scientific papers (Varone, 2007). The evaluative capacity development model in RAMAA is original and differs from the top-down, vertical logic that characterises most international surveys. Conducted horizontally in the context of Action Research, this approach provides a solid foundation for the effective implementation of a results-oriented culture.

6.4.1 Methodological framework of RAMAA

The quality of the youth and adult literacy

programmes in RAMAA is captured at three distinct levels: i) the level of learning of beneficiaries upon beginning and end of the literacy programmes; ii) the sustainability of learning over time and space; and iii) the impacts of literacy (see Figure 6.8). In the current phase, the 12 RAMAA countries are engaged in the production of standardised measurement tools in accordance with the first level of analysis.

The organization of the survey is spread over a period of five years (2016-2020) and sub-divided into four phases: i) the first year focuses on the consolidation of the partnership framework; ii) the next two years are devoted to the updating of contextual data, the adjustment of measurement instruments and collection methods;⁸⁹ iii) the fourth year gives way to the pilot testing of measuring instruments and collection procedures; and iv) the fifth year gives rise to the execution of the assessment itself.

89 This includes the updating of the harmonised competency framework, the development of the assessment framework as well as the elaboration of the items/questionnaires.

Figure 6.8 Different levels of analysis of RAMAA learning out-comes

Figure 6.9 Methodology for the development of learning measure-ment tests towards standardised measuremeasure-ment tools

Currently, RAMAA is in the adjustment stage of the measurement tools, including the assessment framework that specifies the skills to be assessed, the content standards for these skills and the psychometric method for determining proficiency levels.⁹⁰

The originality of the RAMAA assessment framework is that it relies initially on fine contextual analysis of the common frames of reference associated with the literacy provision of the RAMAA countries, which, in turn, will allow it to incorporate good practices in literacy assessment (see Figure 6.9).

This first descriptive construct, which we call the RAMAA Harmonised Competency Framework (HCF), aims to identify the profile in terms of literacy skills of a so-called “literate” person in the context of the RAMAA countries, by pooling:

m Competency frameworks available in the literacy programmes of the countries concerned;

m Competency frameworks mobilised in social and professional activities; and

m The competency framework describing the ideal profile of the literate citizen as reflected in sectoral social policy documents (education, employment, health, social development, etc.).

90 See Mally (2018) for details.

A second RAMAA specificity relates to the consideration of writing skills, which are poorly reflected in international surveys, due in part to the difficulty of comparing performances in different writing systems (Jeantheau, 2015). The first phase of RAMAA included twelve national languages. However, the heterogeneity of the exercises developed during this phase made the data processing more complex.

In terms of remediation, this phase of RAMAA will standardise and limit the collection of information. It will largely rely on the methodological approach of the survey on information exchange and daily life.

The last RAMAA specificity lies in the measurement, in declarative form, of the socio-educational and professional skills common to the context of RAMAA countries. In the first phase of the project, they

Source: Diagram developed by Sobhi Tawil, UNESCO.

Figure 6.8 Different levels of analysis of RAMAA learning outcomes

Learning outcomes

1 2

3

and usages

Literacy impacts

Factors determining quality

Note: * IVQ is the Survey on Information Exchange and Daily Life Source: UIL.

Figure 6.9 Methodology for the

development of learning measurement tests towards standardised measurement tools

Contextual and harmonised competency framework (HCF) is a pooling of three competency frameworks mobilized in: i) literacy

programmes; ii) social and professional activities; and iii) sectoral policy documents

Standardised assessment framework which takes into account the HCF, good practices in the field

of assessments (IVQ*, PIAAC, PASEC, etc.) and recent

research in psychology

Items/questionnaires

focused on four areas, namely health/well-being, citizenship, environment and work.

Proficiency levels will not be fixed in advance but according to an empirical data mining procedure. It is the success of the participants that will allow us to assess the levels of control defined not in a dichotomy but on a continuum (see procedure adopted by PIAAC).

Thus, from the answers given by the participants to the test items, a performance scale will be developed using an item response model. As per the theory related to this type of model, the scores of the respondents as well as the level of difficulty of the items are measured on the same scale. This makes it possible to build groups of levels and to associate them with sets of items of increasing difficulty (Rocher, 2015).

6.4.2 RAMAA model of developing

assessment capacities for anchorage in a results-oriented culture

By “assessment capacities” in RAMAA, we mean the ability to put in place a sustainable mechanism for monitoring and assessing the quality of the literacy programmes. The viability of such an enterprise is based on interdependent guiding principles, which include the following:

m Institutional: a firm political will which translates into the signing of a memorandum of understanding and a technical (making use of a set of national skills) as well as financial commitment (taking charge of national activities) of the countries in RAMAA.

m Strategic/organizational: action research that favours a dynamic co-construction based on the active participation of national teams at all stages of the RAMAA programme. The goal is to promote a results-oriented culture.

Overall, skills development should not be seen as a goal in itself, but as a process that takes time to make a real impact. The investment will yield future returns.

This co-construction operates:

i) Throughout the RAMAA implementation by pooling experiences in the form of an interactive and iterative cycle with three stages (see Figure 6.10):

m Development of the frameworks (guidelines);

m Development of measurement tools and collection tools; and

m Evaluation and adjustment of measurement tools and country perspective (national reporting). The multi-cultural professionalism of national experts, external experts and UNESCO are a source of mutual enrichment in the sense stressed by Courtois (2013).

ii) South-South cooperation boils down to pooling expertise and strengthening inter-country reciprocal links.

Interactive and iterative cycle with three steps:

i) outline development;

ii) development of measurement tools and

collection tools;

iii) evaluation, adjustment of measurement tools and

a perspective of the products.

Preparatory 1.

phase:

Figure 6.10 RAMAA model for assessment capacity development

Source: UIL.

Figure 1.1 Interim reporting of SDG 4 indicators

6.4.3 Conclusion

RAMAA offers a standardised measure of reading, writing, mathematics and problem-solving skills targeted at youth and adults (aged 15 years and older) who benefit from literacy programmes. The data and analyses produced by RAMAA will be made available to policymakers in the countries concerned, as well as the educational community of researchers and civil society, with the aim of contributing to the debate on the quality of education and the governance of literacy programmes. While emphasising the assessment of functional writing skills and generic skills, RAMAA also can contribute to monitoring progress towards SDG Target 4.6 and, in particular, Indicator 4.6.1. The collaboration of a wide range of national and international experts and the application of rigorous standardised procedures offer guarantees of reliability and sustainability.

The real challenge today is funding. The inclusion of assessment in national budgets remains modest and has slowed the implementation of RAMAA. Achieving the goal of a genuine institutional anchoring of a national assessment policy requires stronger political and financial support. The latter is a fundamental necessity for achieving SDG Target 4.6.

7. Supporting countries to