ECONOMIC COMMISSION FOR AFRICA
2019 EVALUATION REPORT:
for a
prosperous Africa
IdeastoAction
Key Findings, recommendations
and Lessons Learned
Content
FOREWORD
ECA EVALUATION FUNCTION AT A GLANCE
PROGRAMME/PROJECT EVALUATIONS UNECA Partners Survey
Mid-term Evaluation: Support to boosting intra-African trade (ATPC III)
Review of the intergovernmental structure of the Economic Commission for Africa pursuant to resolutions 943 (XLIX) and 957 (LI)
Development Account Project Evaluations SELF-EVALUATION OF MAJOR EVENTS
52nd Session of the ECA Conference of Ministers (COM)
Africa Regional Forum on Sustainable Development (ARFSD 5)
5th Conference of African Ministers Responsible for Civil Registration
AUDIT FINDINGS AND RECOMMENDATIONS UN Board of Auditors Annual Audit of ECA OIOS Audit of the renovation of the Africa Hall OIOS Audit of the Management of Policy Centers
KEY LESSONS LEARNED FROM EVALUATIONS AND AUDITS
Legend
“Putting ideas into action for an empowered and transformed Africa” is ECA’s vision since the year 2018. ECA clocked 60 years of existence then and a repositioning of activities and strategy was necessary in order to be responsive and relevant to its member States’ needs and demands. The vision is anchored by three main facets of the Commission namely its think tank function, convening function and its operational function. In this context, evaluation and audit are key for organizational learning, accountability and strategic decision-making as ECA strives to promote inclusive and sustainable development in support of accelerating the economic diversification and structural transformation of Africa, in line with the 2030 Agenda for Sustainable Development and Agenda 2063: The Africa we want. In addition, more now than ever, is evaluation critical in the context of the COVID-19 pandemic as it fundamentally helps to ensure responses that are built on evidence and hence effective.
Along the ECA reform is the repositioning of the UN development system which was mandated by the UN General Assembly in May 2018. The reform aims to strengthen the UN development system to be able to better support countries in achieving Sustainable Development Goals. The resolution and follow-up guidelines and the development system changes, recognize the key role of independent system wide evaluations and the need for strengthening national evaluation systems for assessing results achieved at the global, regional and national levels.
During the reporting period, a number of evaluations and reviews were commissioned, and a series of internal and external audits were undertaken. Evaluation activities were done to help identify good practices and lessons that are informing planning, design and implementation in addition to management strategic decision making.
Evaluations further provided objective assessments of performance, accountability of how resources were used and promoted transparency. Findings, lessons and recommendations were drawn from evaluations undertaken in the areas of trade and regional integration, data and statistics, social development, gender and women empowerment, macroeconomics and governance, the overall relevance and effectiveness of ECA’s interventions and the functioning of its intergovernmental structure, among other areas.
Lessons from the evaluations have helped inform the design and planning of extra budgetary projects and programmes, Development Account projects and the 2021 programme plan. Lessons and recommendations also informed aspects of implementation to ensure management for results, strengthened relevance including contribution to SDGs and Agenda 2063, more effectiveness and efficiency, sustainability of results, strengthened partnerships as well as strengthened mainstreaming of gender equality and integration of human rights. For example, ECA is building on the findings of the partners’ survey in which partners acknowledged the main role of ECA’s research products to inform on topics, such as statistics, economic development, green and blue economy, industrialization, trade, energy issues and gender, to strengthen its think tank and operational activities.
Additionally, ECA has increased efforts in strengthening the capacities of national monitoring and evaluation systems in the follow-up, monitoring, reporting and evaluation of SDGs and Agenda 2063 as a result of the feedback received from the same partners’ survey.
Lessons and recommendations drawn from the review of the intergovernmental structure of ECA were presented at the Conference of Ministers of Finance, Planning and Economic Development (COM) of 2019 and informed the re-arrangement of the intergovernmental structure to make it fit for purpose and to be able to support a more integrated, coherent and multidimensional policy advice support and programming. Further, major events of ECA such as the Conference of Ministers, the Africa Regional Forum on Sustainable Development and the Conference of African Ministers Responsible for Civil Registration were self-evaluated to draw lessons for improvement on the effectiveness, efficiency and outcomes of ECA’s convening function.
Similarly, UN oversight bodies- Office of Internal Oversight Services (OIOS), UN Board of Auditors (BoA) and Joint Inspection Unit (JIU) with the collaboration of the Evaluation function undertook audit and review of some ECA activities to help improve the effectiveness of risk management, internal controls and governance processes.
This report presents a synopsis of key findings, lessons learned and recommendations from evaluations, reviews and audits undertaken by ECA in the year 2019.
Foreword
ECA evaluation function at a glance
ECA’s evaluation function is aligned to the Regulations and Rules Governing Programme Planning, the Programme Aspects of the Budget, the Monitoring of Implementation and the Methods of Evaluation (PPBME), the United Nations Evaluation Group (UNEG) Norms and Standards for evaluation and United Nations Secretariat Self-Evaluation Policy.
It aims to provide an impartial and objective assessment of the Commission’s contributions to Africa’s development results based on their relevance, effectiveness, efficiency, impact, partnerships sustainability and gender equality and human rights integration.
(1) to contribute to organizational learning by continuously generating and distributing information about what works or what does not, for whom and under what circumstances;
THE EVALUATION FUNCTION AT ECA SERVES THREE MAIN PURPOSES:
(2) to produce data and evidence base on which credible decisions can be made for better development results; and
(3) to strengthen accountability and transparency to stakeholders in ECA’s capacity in managing for results.
EVALUATION PRINCIPLES AND NORMS
UTILITY
TRANSPARENCY
INDEPENDENCE
CREDIBILITY
PROFESSIONALISM
ETHICS
GENDER EQUALITY AND HUMAN RIGHTS
EVALUATION CRITERIA
Consistency of intended objectives/outcomes with the priorities of ECA, SDGs and Agenda 2063, development strategies and priorities of
governments and requirements of the target groups;
The extent to which human and financial resources were used in the best possible way to deliver
activities and outputs, and facilitate the achievement of results in coordination with other stakeholders
Extent to which the expected objectives and results have been achieved as planned in the results framework
(likelihood of impact): fundamental and significant long term changes and effects, positive and negative, planned and unforeseen, that have resulted from the intervention with respect to the target groups and other affected stakeholders
The likelihood that the positive effects of the
sub-programme, theme or project will continue in the future.
Collaborative implementation with UN agencies, Pan-African institutions and other stakeholders to enhance efficiency and effectiveness and minimize duplication of efforts.
The extent to which issues of gender and human
rights have been incorporated into the design,
planning, implementation, and monitoring and
evaluation practices of the intervention, as well as the
results achieved.
UNECA Partners Survey
AREAS FOR IMPROVEMENT
METHODOLOGY
OBJECTIVE OF SURVEY
PROFILE OF RESPONDENTS
To understand perception of partners on relevance and effectiveness of ECA interventions
30 KEY QUESTIONS &
120+ SUB - QUESTIONS
Sent by email to 1,731 partners
RESPONSE RATE OF 27%
Most respondents worked in regional integration and trade, and poverty and development sectors
26% F
EMALEECA can better address SDG-related data requirements Ensure more effective collaboration between UN and other Pan-African institutions through enhanced dialogue, and clearer definition of roles and strategies
ECA’s advisory services can become more influential.
Mainstreaming gender in the area of private sector development and finance
Develop new partnerships, strengthen interaction with, and increase involvement of a broader and more diversified range of actors
Have targeted communications to publicize ECA’s activities, and disseminate with relevance its research products to broader audiences, in view of improving its visibility.
Need to work more on building national capacities in monitoring and evaluation.
Improved support to development of innovative financing mechanisms
Improved dissemination and outreach of ECA knowledge products Human resource
capacities within ECA and partner organizations
}
•ECA’s 5 strategic directions are supportive to their own work and results (especially ECA’s research output on the following areas: statistics, economic development, industrialization, green and blue economy, trade, and gender)
•ECA’s research outputs are of high quality
•ECA Convening function
•ECA 100% Think-tank function
•ECA is an effective partner for regional and multi-stakeholder engagement
79% 80%
SURVEY RESULTS Overall, partners’ perception of ECA is HIGHLY POSITIVE
=
137
INSTITUTIONSOFM
EMBERSTATES84%
• ECA’s development of macroeconomic and structural policy options is moderately up to highly supportive to their own work and results
RESOURCESHAVEBEEN SUFFICIENTLYUTILIZED. ATPC ISHIGHLYREGARDEDBY
ALLSTAKEHOLDERS.
RELEVANCE
RELEVENTTO:
Identified strategic needs and
priorities, the SDGs and ECA mandate, new vision, strategic directions, core functions and ECA reforms.
• Measuring contribution is challenging.
THEIMPLEMENTATIONOF ATPC III
HASGENERALLYBEENHIGHLYEFFECTIVE.
EFFECTIVENESS
PROGRESS TOWARDS OUTCOMES
Successes include:
• Progress of the AfCFTA negotiations;
• Increased positive reputation of the Centre and ECA at large;
• Increased knowledge, budgets and other resources by organizations engaged in advancing the AfCFTA agenda
ATPC SERVICESHAVEAVERY HIGHDEMAND.
• To strengthen programme management and monitoring.
• To expedite processing of agreements with
implementing partners
• To better disseminate research products
MORECANBEDONE
EFFICIENCY
• A large but manageable number of partners that strengthens capacity and delivery
• Excellent relationships and overarching partnerships with the AUC
• Partners including RECs
appreciate high quality outputs and technical support provided.
OPTIMALANDSTRATEGIC CHOICEOFPARTNERS
PARTNERSHIPS
• Institutionalization of ATPC within ECA structures.
• ECA’s in-kind and financial contributions to the Centre.
• Securing further external funding to extend the centre's work
THEREISASTRONGINDICATIONOF PROGRESSTOWARDSSUSTAINABILITY
SUSTAINABILITY
• There was focus on raising awareness to consider gender and human rights
• Good progress in putting gender, equality and equity issues at the center of trade work
• Low financial and human resources investment in the gender related outcome
ATPC’SWORKONINCLUSIONANDTRADE HASBEENGROUNDBREAKINGANDTIMELY
GENDER AND HUMAN RIGHTS
RE COMMEN D ATIONS
Mid-term Evaluation: Support to boosting intra-African trade (ATPC III)
ATPC III is focusing on the negotiation and implementation of the Africa Continental Free Trade Area (AfCFTA) and on the Action Plan on Boosting Intra African Trade (BIAT) Initiative to ensure that trade works is an effective tool for the structural transformation and diversification of African economies.
EVALUATION RESULTS
Refine the results framework: including developing more robust measurements at the immediate outcome level;
developing more precise definitions of indicators; expanding data collection strategies.
Capitalize on ECA reforms to ensure that the trade reforms are closely linked to industrialization and economic diversification.
Strengthen project management: including areas such as managing and monitoring risks, financial planning, seamless and formal mechanism to incorporate lessons learned into new work.
Leverage partnerships: develop a partnership strategy that includes mapping of ATPC workplan against partner competencies and internal ATPC capacity to deliver Ensure sustainability: via training, and building capacity and knowledge for partners/ institutions on the ground.
Greater investment in human rights & gender: invest in additional expertise to develop gender & human rights strategy.
CONTRIBUTINGFACTORSTOPROJECT’SSUCCESS INCLUDES:
• Reputation for quality work;
• The expertise and commitment of its team;
• Its responsiveness to emerging needs and opportunities;
• Initial traction in its training program;
• Its increasing integration into ECA, and;
• Close collaboration with the AUC
6
Review of the intergovernmental structure of the Economic Commission for Africa pursuant to resolutions 943 (XLIX) and 957 (LI)
1st review period: Nov 2016 - Jan 2017 Areas of review: the intergovernmental structure of the Commission.
Preliminary findings presented at its 51st session in May 2018..
2nd review period: May 2018 - Dec 2018 Areas of review: The focus on the effectiveness and efficiency of the Commission.
CONFERENCE OF MINISTERS (together with the Committee of Experts)
Overall, functioning well:
• Annual Commission sessions are well-planned in advance and executed as projected.
• Well-attended by Member States.
• The statutory nature of the meeting; opportunity for networking; and opportunity for advocacy and consensus-building are main reasons for attendance
• Knowledge resources presented at the Conference of Minsters were of high quality
• Delegates value consultation on theme of the Conference
• Allocation of time for presentations and discussions on substantive issues
• Greater gender and geographical representation needed
• Systematic consideration of provisional budgetary implications needed
Sectoral subsidiary organs of the Commission Committee on Gender and Social Development;
Committee on Sustainable Development; Committee on Regional Cooperation and Integration; Committee on Statistics
Themes of the sectoral subsidiary organs are relevant to Member States
Agenda 2030 & Agenda 2063 well covered
& discussed
Level of participation is high Frequency of committee meetings is adequate
• Promote a greater sense of ownership through increased & regular interaction with secretariat
• More timely circulation of documents in relevant languages needed
• Greater involvement of, and consultations with, Member States in preparation of meetings
• Follow-up on implementation of
recommendations and outcomes of committee meetings
Subregional intergovernmental committees of experts
• Number of participants has increased since 2007
• Duration and frequency of meetings are adequate
• Joint meetings of various subregional intergovernmental committees of experts would help to strengthen
South-South collaboration and peer-learning
Extend participation to senior officials in Member States delegations to ensure effective uptake of policy recommendations at country level
• Strengthen monitoring & evaluation of the impact of discussions and outcomes
• IDEP- new strategic plan & info on new e-learning programme to be disseminated to all relevant stakeholders.
• Strengthen outreach & engagement with permanent missions of Member States based in Addis Ababa and members of the Bureau of the Commission
AREAS OF IMPROVEMENT
AREAS OF IMPROVEMENT
AREAS OF IMPROVEMENT
KEY FINDINGS KEY FINDINGS KEY FINDINGS
RECOMMENDATIONS
CONFERENCE OF MINISTERS
African Governments to coordinate in a coherent & effective manner the participation of their delegates from sectoral ministries at the Commission and its subsidiary organs sessions
Key policy decisions of the Commission session to be complemented with provisional budgetary implications Permanent representatives to engage more proactively in the work of the Commission
ECA to systematically engage with the Bureau of the Commission and the Bureaus of the subsidiary bodies with a view to ensuring more effective follow up and implementation of policy decisions
SECTORAL SUBSIDIARY ORGANS
To be reorganized to better reflect the restructured programme of the Commission by leveraging on internal synergies to deliver its integrated analytical, policy tools, convening platforms and expertise.
Streamline their work programmes & allocate enough time for discussions on technical and statutory matters Ensure timely distribution of documents in all the working languages, including electronically
Systematically administer self-evaluation as a way to assess the relevance, effectiveness and efficiency of each session.
SUBREGIONAL INTERGOVERNMENTAL COMMITTEES OF EXPERTS
To be renamed the “intergovernmental committees of senior officials and experts” to encourage participation of senior government officials
To meet jointly across the subregions to strengthen South-South cooperation and peer-learning
Consider streamlining the work programmes of their meetings so that sufficient time is allocated to discussions on technical and statutory matters
Ensure timely distribution of documents in all the working languages, including electronically
Systematically administer self-evaluation as a way to assess the relevance, effectiveness and efficiency of each session.
AFRICAN INSTITUTE FOR ECONOMIC DEVELOPMENT AND PLANNING (IDEP) The strategic plan to be disseminated to all relevant stakeholders
Member States in arrears of financial contributions are strongly encouraged to make the required payments
Continue in sharpening tools and improving the knowledge products and their sharing Invest in building the capacity of national institutions
Better integration of gender and human rights during design and project inception stages, as well as throughout the project cycle
Strengthening of project level monitoring and evaluation systems
Allocate budget allowing more robust, effective and efficient data collection during implementation Utilize and encourage greater involvement of national skills and institutions to ensure project sustainability through transfer of skills
Increase sharing of experiences (challenges and best practices) among member states
Improved risk analysis and mitigation measures should be developed at the outset of the project.
Further streamline processes to ensure timely disbursement of funds to avoid handicapping project activities.
RE COMMEN D ATIONS
DEVELOPMENT ACCOUNT PROJECT EVALUATIONS
KEY FINDINGS
1. All projects were relevant to the needs of the member states and were able to demonstrate at a limited scale the benefits of the interventions.
2. Majority of them were implemented in an efficient manner,
3. Projects implemented were effective, short duration of projects limits capture progress towards
outcomes,
4. Projects evidenced making progress towards attainment of outcomes.
5. Sustainability of projects needs more
attention. Need for robust exit
strategy at the outset.
52 ND
S ESSION OF THE
ECA C ONFERENCE
OF M INISTERS
(COM 2019)
key ways to
improve
conference:
3
IMPROVE TIMING
CONDUCT PEER-LEARNING DISCUSSIONS
SHARE SDG DATA
KNOWLEDGE
SHARING REFLECTION
FEEDBACK ON EVENT
understand the challenges of administering the current revenue frameworks in the digital era very
66.7%
INSIGHTFUL PRESENTATIONS in discussing outcomes of the Commission’s subsidiary organs’sessions.
57%
EFFECTIVE DELIBERATIONS discussion on statutory issues was sufficient50%
TIME ALLOCATIONthe theme of the conference.
76.5%
EFFECTIVE DELIBERATIONS Sessions dedicated to statutory issues of the Commission were61.5%
INFORMATIVE SESSIONSconference.
statutory issues
work of secretariat (86.7% - of which 33.3% found them very
86.7%
EFFECTIVE DELIBERATIONS Key policy issues discussed was veryrelevant to helping African governments maximize efficiency a digital economy
.
73%
RELEVANT DISCUSSIONSHigh-level ministerial policy dialogue and high-level roundtable
77.8%
EFFECTIVE DELIBERATIONSSide events gave fresh viewpoints and insights (76.7%)
76.7%
INSIGHFUL SIDE EVENTSthe event’s theme gave
80%
INSIGHTFUL PRESENTATIONSrelated to the benefits of fiscal policy moderately relevant (85.7% - of which 35.7% found them very useful)
85.7%
RELEVANT DISCUSSIONSthrough private sector and trade
that guided the discussions on statutory issues were of quality (of which 33.3% felt that they were of high quality)
90.5%
QUALITY DOCUMENTATIONshowcase how fiscal policy can
management were useful (33.3% of the 86.7% found them very useful)
86.7%
USEFUL POLICY OPTIONSstatutory issues increased ECA
94.8%
EFFECTIVE DELIBERATIONS and outcome of the sessions dedicated to statutory issues (90.5% - of which 42.9% found90.5%
EFFECTIVE DELIBERATIONS(56.6% said full, while 36.7% said agreeing that they are very likely to shared during the event. 76% of
93.3%
OVERALL OBJECTIVES MET3
key words to to describe most preferredaspects
Experts
A FRICA
R EGIONAL F ORUM
ON S USTAINABLE D EVELOPMENT
(ARFSD 5)
PROFILE OF RESPONDENTSon improving
Government officials
49%
16%
Major groups
14%
12%
Others9%
KEY HIGHLIGHTS
FROM PARTICIPANTS
A multi-stakeholder platform convened by the UNECA to advance the implementation of the 2030 Agenda and the 2063 Africa Agenda
90%
SURVEY RESULTS:
• Overall, the Forum and side events were well-organized.
• Participants were most satisfied with the translation and transmission of documents, including
programme booklets and information package, and invites.
• The Forum gave space and opportunity for people with disabilities to share their thoughts during discussion, but more can still be done to include civil society organisations in the plenary sessions.
• A major area of improvement is timeliness and time management of the Forum.
87%
EFFECTIVE SUPPORTThe in-session support and
94%
SAY OBJECTIVES WERE METSide events provided fresh viewpoints and insights (90%).
90%
INSIGHTFUL SIDE EVENTSkey messages generated reflected the discussions held during the Forum.
76%
ALIGNED RECOMMENDATIONS56%+
USEFUL SESSIONS Sessions were very useful and of very good quality (>37%).81%
INCENTIVISED TO PARTICIPATE90%
USEFUL SESSIONSClarify the role of specialized agencies in developing background papers for the Forum
Consider sharing documents in electronic format to ensure that the Forum is environment-friendly
Participants (90%) found two sessions exceptionally useful:
• Roundtable panel on Voluntary National Reviews (VNRs) and peer-learning on implementation, follow-up and review of the 2030 Agenda and 2063 Africa Agenda
• Parallel panel meetings for in-depth review, peer-learning and dialogue on the sub-themes of ARFSD
RECOMMENDATIONS
5 TH C ONFERENCE OF A FRICAN M INISTERS R ESPONSIBLE FOR
C IVIL R EGISTRATION
Theme: Innovative Civil Registration and Vital Statistics System:
Foundation for Legal Identity Management
KEY HIGHLIGHTS FEEDBACK ON EVENT
RECOMMENDATIONS
94%
were either satisfied or very satisfied with the conference• The conference fully brought Identity Management and Civil Registration Government stakeholders and partners to a single table for the first time.
indicated that the overall objective of the conference was fully achieved
77%
indicated that they are very likely to use the knowledge acquired from the conference in their institutions.
indicated that the session on “Towards accelerated improvement of CRVS systems in African countries” was good to excellent in terms of its quality
The session on “Civil registration as the foundation for legal identity systems” was highly rated it terms of its usefulness.
73%
97%
• The conference helped participants to better understand the role of CRVS in legal identity.
• The conference provided tools that could help countries reach SDG target 16.9: “By 2030, provide legal identity for all, including birth registration.”
• The diversity of the topics made it possible to address all aspects related to the central theme.
• There was need to exhaustively discuss holistic innovative CRVS approaches considering the theme of the conference.
• Country case studies should have been selected from those that are best practices and also those that are still struggling so that lessons and experiences could be exchanged effectively.
• The program could have been less packed so that there is room for in-depth discussions on the prioritized areas of discussions.
There is need to put in place a monitoring system for countries to report on performance on agreed recommendations of the conference using a standard set of indicators. This would also help to keep the conference alive in between conferences.
For continuity and a better follow-up on recommendations and actions in the member countries, there is need to invite experts present at previous sessions in the next sessions.
There is need to ensure that countries that are participating in panel discussions are those that have practical experiences to share and not share from a theoretical standpoint.
In terms of meeting arrangements logistics (specifically transport to and from the venue) and quality of in session interpretation could be improved.
Maintain a project schedule, updated regularly and capturing risk analysis Take steps to ensure that owner-requested change orders are minimized, including by ensuring that only changes within the scope of the Africa Hall project as approved by the General Assembly are processed.
Establish a quarterly mechanism to discuss all change orders with the independent risk management firm to assess whether changes are in alignment with the risk mitigation strategy.
Ensure adequate oversight of the procurement for the Africa Hall renovation and visitors center construction projects through updated procurement plans coupled with appropriate support such as training.
UN Board of Auditors Annual Audit of ECA
Sub-programmes to systematically align respective result areas against relevant SDGs Renewed focus on supporting countries with low capacity in field of data collection Increased advocacy efforts for increased participation by Member States in Voluntary National Reviews (VNRs).
Further strengthen internal controls regarding project formulation, risk management, monitoring of projects
RECOMMENDATIONS
OIOS Audit of the renovation of the Africa Hall
RECOMMENDATIONS
OIOS Audit of the Management of Policy Centers
Improved guidelines for establishing and operating current and future policy centers Develop targeted resource mobilization strategy for policy centers
Establish key performance indicators for all stages of processing of donor and grant documents Clear provision in letter of agreement with IPs to regulate management fees charged by IPs.
RECOMMENDATIONS
14
