ClimDev-Africa monitoring framework evaluation approach from inception to implementation

(1)

ClimDev-Africa Monitoring Framework Evaluation Approach from Inception to

Implementation

05

(2)

(3)

ClimDev-Africa Monitoring Framework Evaluation Approach from Inception to

Implementation

This document summarizes the work that has been done since 2012/2013 to build the Monitoring Framework and Evaluation Approach (MFEA) of the ClimDev-Africa programme with a specific focus on the most recent updates: implementation in 2014 and 2015 including baselines, case studies, mapping, evaluation exercises protocols. The possible route forwards for MFEA under ClimDev- Africa Phase 2 is also proposed.

(4)

(5)

1. Building the MFEA since 2012

For a programme such as ClimDev-Africa, it is crucial to have a credible and effective outcome- oriented monitoring and evaluation system to streamline activities and enhance result-based management. The Monitoring Framework and Evaluation Approach (MFEA) implemented by ClimDev-Africa with the support of the International Institute for the Environment and Development (IIED) clearly tracks how outputs aggregates to result areas, and result areas to outcome for all the areas of work of the programme.

Transition from the original logframe to the MFEA :

The implementation of the ClimDev-Africa MFEA system is a long and critical process that started in 2012 through the establishment of a Theory of Change (TOC). At the outset, it was necessary for the three partner institutions to collectively consider and define their respective roles and responsibilities and to identify the scope of what was to be monitored and evaluated within programme. The purpose of the TOC process was to set out and examine the logic underlying the ways that ClimDev-Africa sought to achieve its result areas, and thereby, contribute to the desired outcomes and impact. Based on the programme’s objectives, the ToC enabled the three institutions to track results vis a vis achievement of outputs, and to test the assumptions that underpin the causal links between outputs, outcomes and impact.

This first step, ToC, clarified the role and responsibilities of each core partner in achieving

2

1. Building the MFEA since 2012

For a programme such as ClimDev-‐Africa, it is crucial to have a credible and effective outcome-‐

oriented monitoring and evaluation system to streamline activities and enhance result-‐based management. The Monitoring Framework and Evaluation Approach (MFEA) implemented by ClimDev-‐Africa with the support of the International Institute for the Environment and Development

(IIED) clearly tracks how outputs aggregates to result areas, and result areas to outcome for all the areas of work of the programme.

Transition from the original logframe to the MFEA :

The implementation of the ClimDev-‐Africa MFEA system is a long and critical process that started in 2012 through the establishment of a Theory of Change (TOC). At the outset, it was necessary for the three partner institutions to collectively consider and define their respective roles and responsibilities and to identify the scope of what was to be monitored and evaluated within

Pre-‐2012:

2008: Old logframe established before the creation of the ClimDev-‐Africa functional bodies. This Logframe included ClimDev-‐

Africa activities and related indicators.

2010: Functional bodies of ClimDev-‐Africa in place

2011: ClimDev-‐Africa activities seemed not completely well aligned with the 2008 logframe so we encountered some

reporting issues: it was hard for the donors to clearly see the link between the

logframe and our activities.

2011/2012: Discussions with donors and partners led to the decision of reviewing the logframe to be able to better report against it. The Theory of Change (ToC) exercise was then chosen as a way to clearly understand the roles and responsibilities of each partner.

MFEA establishment 2012-‐

2015:

2012/2013: ToC established to have a basis for the logframe revision.

2014: Logframe reviewed and agreed by the three partners + MFEA narrative established (2 M&E workshop with the partners)

2014/2015: Implementation of the MFEA through different actions:

-‐ Evaluation approach set up -‐ Case studies started -‐ Baseline collection started -‐ Reporting against the

indicators in place -‐ Thinking around an MFEA

online tool

(8)

2

project outcome as well as the expected programme impacts on research, policy-making and practice communities involved in the programme. The time allocated to set up the theory of change narrative was necessary to make sure that ClimDev-Africa would have a bespoke and adapted MFEA system based on a common understanding of the project, taking into account different partner organizations’ institutional reporting requirements.

The structure of the MFEA is based upon a logframe (derived from the ToC) that outlines ClimDev-Africa’s impact, outcome, result areas, outputs and functioning activities. This logframe analyses the different levels of action by setting up baselines, indicators, milestones and targets for each of the partners. The indicators guide the partners on tracking their activities as well as the general functioning of the programme. The Monitoring Framework (MF) records the impacts of the assumptions and risks identified and generates data and information to test the hypotheses on how activities lead to outputs. The Evaluation Approach (EA) will bring a broader overview of the project efficiency and achievements through different timely evaluation exercises (mid-term review, output to outcome review, impact evaluation).

The MFEA is also a strong tool that allows ClimDev-Africa to address issues of socio- economic returns on climate information investments to put a development value on climate investments. This system provides a metric to measure the returns on different types of investments and the effectiveness of observed changes. More specifically, the MFEA is setting out how knowledge on the returns on investments in climate information services will be generated through the tracking and assessment of the projects supported by the ClimDev Special Fund.

The system aims to enable continuous learning and decision making to improve the focus and management of the programme during implementation, and more particularly after the completion of each cycle. In short, the MFEA plays two particularly important roles:

à Demonstrated value for money: address issues of socio-economic returns on climate information investments

à Learning dimension: adjust the programme according to the lessons learnt from the ClimDev tracked activities and projects

(9)

The following scheme displays the MFEA process and its two main outcomes for the ClimDev-Africa programme:

The timeline diagram below represents the main steps and achievements in the MFEA since 2012:

4

ClimDev-‐

Africa MFEA Logic

Adjust and review the ClimDev

Strategy

Theory of Change Process

Establishement of the Logframe

(baselines, indicators, targets) Repor\ng on an

online pla]rom (data collec\on, tracking av\vi\es) Sharing lessons

learnt through MFEA

Socio-‐economic returns on climate

information investments highlighted

Theory of change and MFEA

narra\ve (2012-‐2013)

Logframe reviewed (MFEA workshop-‐

March 2014)

Evalua\on exercises set up

through protocols documents (MFEA workshop-‐

June 2014)

Case studies exercises started

(July 2014)

Logframe Baseline at Result

Areas set up for the secretariat

(2015)

CIS CBA framework

produced (December 2014-‐

March 2015) MFEA progress since 2012:

MFEA logic:

ClimDev-‐

Africa MFEA Logic

Adjust and review the ClimDev

Strategy

Theory of Change Process

Establishement of the Logframe

(baselines, indicators, targets) Repor\ng on an

online pla]rom (data collec\on, tracking av\vi\es) Sharing lessons

learnt through MFEA

Socio-‐economic returns on climate

information investments highlighted

Theory of change and MFEA

narra\ve (2012-‐2013)

Logframe reviewed (MFEA workshop-‐

March 2014)

Evalua\on exercises set up

through protocols documents (MFEA workshop-‐

June 2014)

Case studies exercises started

(July 2014)

Logframe Baseline at Result

Areas set up for the secretariat

(2015)

CIS CBA framework

produced (December 2014-‐

March 2015) MFEA progress since 2012:

MFEA logic:

(10)

4

2. Main achievements of the ClimDev- Africa MFEA in 2014-2015

Agreement on the Logframe and setting up of the evaluation approach

Two workshops were organized in 2014 to train and build the capacity of the ClimDev- Africa partners on the MFEA. These workshops consolidated the TOC and MFEA narrative as well as advanced the initiative toward the concrete implementation of the MFEA. The first workshop (March 2014) focused on reviewing the logframe and sought agreement amongst ClimDev-Africa Institutions on the way forward. The second workshop (June 2014) additionally looked in detail at the Evaluation Approach. The main outcomes were:

- Logframe reviewed and agreed by the 3 ClimDev-Africa partners

- Evaluation Protocol documents established for all the evaluation exercises planned for the ClimDev-Africa programme

- Case studies exercises for ClimDev-Africa launched

- Agreement to recruit a consultant to work within the Secretariat on the baseline collection

Evaluation protocols

The evaluation protocols are the main outcome of the MFEA workshop organized in June 2014. They explain the methodology for the different evaluation exercises required for the ClimDev-Africa programme (mid-term review, result area to outcome review, impact evaluation, case studies). Each planned evaluation exercise included: the specific objective of the evaluation exercise, its timing for implementation, issues to be addressed, the criteria to be considered for the exercise (e.g. effectiveness, efficiency, relevance), the role of the different stakeholders in the exercise, the steps, the data collection methodology and the communication of the results.

Baselines work at result area level for the secretariat:

Task / Purpose:

An external consultant has been engaged to work on establishing the baseline for the selected indicators in the logframe at each of these levels of action. This exercise has begun in earnest, and is initially being undertaken for all the indicators for which the Secretariat and ACPC are responsible.

The collection of the remaining baselines are expected to be planned and organized by the partners before the end of the year with completion anticipated before Phase 2 begins in 2016. The logframe delineates roles and responsibilities for the collection of data against each

(11)

indicator, accordingly, each institution will work on the baseline for their respective indicators Given that the baseline establishment task is taking place under the auspices of the Secretariat i.e. the indicators have been developed to measure not only the national situation/impact, but also monitor and evaluate the resultant programmatic impact of ClimDev-Africa across the continent.

Methodology:

The work to date has principally focused on the development of a robust methodology that is informed by best practice in Monitoring and Evaluation (M&E), and is closely tied to the ClimDev-Africa ToC. The full methodology was also developed in consultation with ACPC experts and stakeholders to ensure that the definitions relating to Climate Information Systems (CIS) are accurate and will help ensure rigorous measurement of the programme impact.

The methodology makes use of both quantitative and qualitative baseline information. The initial step was to refine and evolve a number of the indicators to measure not only the national situation/impact, but also offer help to monitor and evaluate the programmatic impact of ClimDev-Africa across the continent. In order to better establish/measure the programmatic impact of the ClimDev-Africa program, the methodology seeks to measure activities/changes attributed directly to ClimDev-Africa program, and those that are taking place outside of the program. This will help establish a counter factual i.e. an indication of what would happen or is happening outside of, and hence, in the absence of the ClimDev- Africa program.

Interim Status & Findings:

The baseline establishment is currently underway, but the baselines are only partially completed.

The indicators for which the baselines have been worked on are universally established at zero for ClimDev-Africa; in 2011 the baseline monitoring data for the indicators as defined in the logframe can be recorded as zero for ClimDev-Africa impact.

This can be attributed to two factors;

• The methodology that has been developed seeks to determine attribution of the programme, so the logical starting point is zero, and

• Programming did not really begin in earnest until after 2011, and there were no concrete activities at this time

A full policy mapping was undertaken for all 54 African countries. This identified that in 2011, 15 countries across the continent had produced a dedicated climate change policy, and revealed that none of these countries gained support from ClimDev in that year. Genuine integration and mainstreaming of climate change into key sectoral policies (as identified by the countries themselves) - meaning clear provision within sectoral policies for the management of climate change risks and opportunities – was relatively low across Africa in 2011. Integration into development policies is slightly higher across the continent, but

(12)

6

there is still relatively limited evidence that climate change was considered and integrated effectively into development planning in 2011. As mentioned above, a full narrative for each country is offered to substantiate the scores assigned.

A policy mapping of the legal and regulatory framework to support responses to climate change and regional harmonized development framework mainstreaming climate change responses is currently underway for the Regional Economic communities. This is targeted for completion by mid-year, 2015.

A subset of 10 countries for establishment of the baseline and then subsequent measurement at the milestones has also been recommended. This will offer an indication of the situation in 2011 and then changes over time. The rationale behind choosing 10 countries for ongoing measurement is to ensure that M&E of the ClimDev-Africa programme is practical (and practicable). Undertaking such an exercise for all 54 countries may make the baseline and the subsequent milestone measurement of the programme unwieldy and overly time consuming.

Agreement between the partners on the selection of subset of countries must be reached in order to complete the baselines establishment exercise. This is envisaged for June 2015.

The following selection criteria have provisionally been devised for the selection of the 10 candidate countries:

- ClimDev-Africa planned activities - Hazard profile - Development context - Geographical spread

Using these criteria, the following countries, small island states and Regional Economic Communities have been provisionally suggested:

1) Ethiopia 2) The Gambia 3) Rwanda 4) Senegal

5) Southern African Development Community 6) Economic Community of Central African States 7) Cape Verde

8) One country receiving funding under the ClimDev-Africa Special Fund - to be decided once funds have begun to be allocated.

9) & 10) Counterfactual countries - the counterfactual countries will need to be investing in CIS to ensure a like-for-like CIS investment comparison can be undertaken, but is not receiving support from the ClimDev-Africa program.

A confidence level for each of the indicators will also be included that will be determined by the quality/credibility of the data gathered from each of the countries. This will be clearly marked and recorded in the master sheet.

(13)

Case studies and the Cost-Benefit Analysis framework (CBA) for year 2015

The case studies which fall under the first result area (widely available climate information, well packaged and disseminated) are part of the evaluation exercises. The programme assumes that climate risk management should increase/improve in order to maintain desired development trajectories. With this in mind, CIS (Climate Information Services) could provide a means to maintain development whilst simultaneously facilitating adaptive behavioural change under climate stress. This would be via provision and analysis of relevant and appropriate climate information and used for bespoke decision-making. Therefore, improvements in the quality and targeting (i.e. vulnerable groups) of climate risk management should lead to vulnerability reduction, greater resilience and sustainable livelihoods. The case study approach assesses the impact of introducing and improving CIS¹ for organisations, government/policy officials (immediate beneficiaries) and urban and rural climate sensitive communities (ultimate beneficiaries). The case studies are the first step in operationalizing the evaluation approach.

The first step in the case study is an ex-ante evaluation to set up an evaluation framework based on the project beneficiaries’ expectations. This assessment framework is established using the Tracking Adaptation and Measuring Development (TAMD) twin-track approach². Bespoke ToC exercises define the context, identify criteria for progress, and assist when selecting indicators and enable continuity across ex-ante, mid-term and ex-post assessments. Based on the case study methodology, a case study exercise has begun in Ethiopia creating a specific evaluative framework for the project.

In addition to these case studies, Cost-Benefit Analysis (CBA) will be carried out. They will complement ClimDev-Africa’s evaluation systems by estimating (ex-ante) or establishing (ex-post) monetary values for likely or observed outcomes – enabling comparison with costs.

Second, Net Present Values (NPV) (inflation adjusted and time discounted monetary values) provide a common metric to compare across CIS investments of similar value. Third, Benefit- Cost Ratios [each dollar invested returns X over the lifetime of the investment (under discounting and inflationary constraints)] or Internal Rates of Return (the discount rate that will bring NPV to zero) assess the magnitude of returns and thus facilitate comparison across CIS investments, or adaptation options more generally. Fourth, by documenting fluctuations in net benefits over time, cost-benefit analysis can determine the optimal scale of projects – i.e. where net benefits are maximized.

IIED in collaboration with ACPC established a framework document that presents and explains the different CBA approaches that could be used to complement the case studies and more broadly the ClimDev-Africa evaluation approach. These case studies complemented with CBA will generate evidences to inform the future CDSF investments in Africa.

1 Focusing on three result areas: a) packaging and disseminating CIS (availability); b) improving the quality of CIS (quality);

and c) awareness and advocacy improvements (knowledge).

2 http://www.iied.org/tracking-adaptation-measuring-development-tamd

(14)

8

ACPC 2015 Workplan MFEA mapping

The mapping exercise aims at matching ClimDev-Africa institutions activities with the ClimDev logframe indicators at output level. This is to clarify the responsibilities between the 3 institutions in terms of on-going reporting, ensuring coherence with MFEA system to make the reporting more coherent both in terms of producing annual reports and reporting online.

To report correctly against the 2015 workplan, the various activities suggested under each result area had to be aligned with the different outputs listed in the logframe. This work has been done in collaboration with ACPC experts responsible for each result area.

MFEA Online tool: ClimDev Monitoring and Evaluation Tool

An online web-based platform accessible by the three institutions - based on the logframe should be implemented in 2015 in order to:

- Enable result-based management of the project by reporting on ClimDev-Africa activities and generating evidence and knowledge to develop, implement and adjust the programme

- Help the three partners to track progress and to carry out the different evaluation exercises when needed.

So far, the best options for the development of the CMET have been identified.This CMET should also be considered as a learning tool, and will enable the institutions to draw lessons from it.

3. Way Forward and the Transition to ClimDev-Africa phase 2

ACPC recently increased its team of experts working on ClimDev-Africa projects and activities. In March 2015, the MFEA was presented to them offering an opportunity for initial discussions and exchanges about the relevance of the current logframe (especially at output level) for the ClimDev-Africa Phase 2 (planned to start in 2016). The second phase of ClimDev-Africa (2015-2020) and its related activities will have to be reported against a common logframe for the programme. In this regard, the current logframe content might have to be redesigned and adapted especially at the output level to fit with the expected achievements of the ClimDev-Africa phase 2. Any changes and modifications of the logframe would also mean an alignment of the baseline in terms of date and value.

Other issues were raised around the reporting mechanisms: level of reporting and how to ensure durable and reliable reporting mechanisms across the 3 institutions. Any future

(15)

thinking and work on linking the MFEA and its logframe with the ClimDev-Africa Phase 2 should be organized and developed in collaboration with the three ClimDev-Africa partners and with the funding agencies.

In summary, the main points that will need to be discussed and agreed between the 3 institutions about the MFEA adaptation for the ClimDev-Africa phase 2, are:

1. At which level should the re-design of the logframe happen? (Outcome? Result areas?

Outputs?)

E.g. new outputs better corresponding to the expected outcomes of ClimDev-Africa Phase 2 could be added in the logframe if needed.

2. In the case new outputs are added, their baseline should start in 2015. The institutions should also discuss possible segregation of the outputs level by institution (i.e.: separate outputs per institution).

4. Agreement on the reporting level for ClimDev- Africa phase 2: should ClimDev-Africa report only at Result Area or also at Output level to the funding agencies?

5. The necessity for each partner to appoint a focal M&E person within that institution to regularly report against the logframe indicators

Recommendations on how to proceed:

- Indicate potential re-design of the MFEA according to the objectives of phase 2 at the Steering Committee

- Organize a meeting with the 3 institutions to discuss the key changes needed for the - Devise an MFEA working plan aligned with the ClimDev-Africa phase 2 business MFEA

plan

(16)

10

4. Planned MFEA activities for 2015

Expected MFEA activities for 2015 Status/Implementation

Map the 2015 activities with the logframe The ClimDev-Africa planned activities for 2015 have been mapped to the MFEA logframe. They now match with the ClimDev logframe indicators at output level and result area level to clarify and delineate reporting responsibilities between the 3 institutions for on-going reporting

Baselines collection and methodology established

The baselines collection and methodology establishment for the indicators for which the Secretariat and ACPC are responsible started early 2015 for the programme to be able to correctly report and measure progress. The process to collect the remaining baselines should be agreed by the 3 institutions and carried out in the year 2015.

Case studies for CDSF project

One or two case studies will be carried out looking at ClimDev-Africa CDSF CIS investments in countries. They will bring evidence on how well ClimDev-Africa is progressing, and gather information that is not readily available elsewhere creating new knowledge on returns on investment in CIS and provide recommendations to improve the programme and direct future CDSF CIS investments.

MFEA online tool implemented (CMET)

Best options for the development of the CMET (ClimDev Monitoring and Evaluation Tool) have been identified early 2015 in order to have an online web-based platform accessible by the three institutions that:

- Enable result-based management of the project by reporting on ClimDev-Africa activities and generating evidence and knowledge to develop, implement and adjust the programme - Help the three partners to track progress and to carry out the different evaluation exercises when needed.

This CMET should also be considered as a learning tool, and will enable the institutions to draw lessons from it. The platform should be put in place by the end of 2015.

Report against the logframe for year 2015 The ClimDev-Africa institutions should be able to report against the logframe for the activities of the year 2015 informing the relevant indicators. This task should be carried out at the end of 2015, beginning of 2016.

Prepare the transition towards ClimDev-Africa Phase 2

With the ClimDev-Africa Phase 2,, the logframe will remain the central element of the MFEA against which all reporting will be done. In this regard, the 3 institutions might have to adapt the logframe and amend current indicators if needed.

In case the logframe is redesigned, the baselines should then be established immediately and an agreement between three institutions will have to be reached for the appropriate activity level for reporting. The thinking and necessary work for the transition of the MFEA towards the ClimDev-Africa Phase 2 will be done during the year 2015.

(17)

(18)

About ClimDev-Africa

The ClimDev-Africa Programme is an initiative of the African Union Commission (AUC), the United Nations Economic Commission for Africa (ECA) and the African Development Bank (AfDB). It is mandated at the highest level by African leaders (AU Summit of Heads of State and Government). The Programme was established to create a solid foundation for Africa’s response to climate change and works closely with other African and non-African institutions and partners specialised in climate and development.

Contacts

Africa Climate Policy Center

United Nations Economic Commission for Africa Menelik II Ave., P.O. Box 3001

Addis Ababa, Ethiopia [email protected]

www.climdev-africa.org

ClimDev-Africa monitoring framework evaluation approach from inception to implementation

ClimDev-Africa Monitoring Framework Evaluation Approach from Inception to

Implementation

05

ClimDev-Africa Monitoring Framework Evaluation Approach from Inception to

Implementation

Contents

1. Building the MFEA since 2012

2. Main achievements of the ClimDev- Africa MFEA in 2014-2015

Agreement on the Logframe and setting up of the evaluation approach

Evaluation protocols

Baselines work at result area level for the secretariat:

Methodology:

Interim Status & Findings:

Case studies and the Cost-Benefit Analysis framework (CBA) for year 2015

ACPC 2015 Workplan MFEA mapping

MFEA Online tool: ClimDev Monitoring and Evaluation Tool

3. Way Forward and the Transition to ClimDev-Africa phase 2

4. Planned MFEA activities for 2015

Supported by