Statistical challenges in monitoring and reporting on MDGs in African countries

40. With regard to the development and availability of MDG indicators, there are a number of challenges that hamper the capacity of African countries to report on progress towards achievement of the MDGs. Some of these challenges are outlined below:

Lack of data for compilation of some indicators

41. Some African countries lack baseline data for the development of some indicators, e.g. pov­

erty indicators. This may be because their administrative data are not developed in that area or be­

cause surveys have not been undertaken in it, making it impossible to develop the MDG indicator.

Examples of this are sanitation and measuring slum populations.

Data discrepancies between national and international organizations

42. The increased demand for data to measure and monitor indicators requires evaluation of the different sources, organizations and methods for producing MDG-related data. Most data are col­

lected through NSSs, some of which are not coordinated. National data enter the international statis­

tical system in a process through which specialized agencies review and further standardize national data in order to produce certain Indicators. This approach has sometimes aroused controversy as countries complain that they were not consulted when certain computations were made, while, in some cases international estimates contradicted national estimates (United Nations Statistical Com­

mission, 2005) [3]. This is a public relations challenge that can be resolved by close consultation between countries and international agencies. Data exchange between national and international organizations could eliminate some of the problems.

Statistical coordination within the NSS and between NSSs and international organizations

43. Coordination among statistical agencies within countries is essential in order to achieve consistency and efficiency in the statistical system (Lievesley, 2001) [41. However, many African countries lack mechanisms for coordination among producers of MDG data. For example, there is lack of coordination for the harmonization of definitions, concepts and classifications among dif­

ferent sources. Inter-institutional collaboration, both at the national and international levels, can go some way towards enhancing coordination. In this way, duplication of data and dissemination of contradictory and non-comparable data can be minimized and the use of scarce resources maxi­

mized (Economic Commission for Latin America and the Caribbean (ECLAC)) [5]. It has often been

10 Best Practices Report on Millennium Development Goals Monitoring and Reporting at National and Subnational Levels in

argued that "competing, inconsistent results on the same issue introduce scepticism and doubts in users' minds and the quality of such data is viewed with suspicion. This undermines confidence in statistics" (Everaers, 2002) [6].

44. In order for international organizations to receive quality MDG-related information, closer partnerships need to be developed between international agencies and national statistics systems.

Mechanisms should be put in place to harmonize requests for metadata. Coordination of MDG data could also facilitate the interlinking and standardization of data from different sources for those countries where this aspect is still a problem.

Methodological issues

45. Since population census data are used as denominators for most MDG indicators, the meth­

odological challenges concerning the conduct of population censuses need to be examined, par­

ticularly census mapping, the pilot census and post-enumeration surveys.

46. Census mapping: Some African countries have conducted censuses without completing the mapping exercise covering the whole country. In this case, some enumeration areas (EAs) were de­

marcated while the enumerators were in the field. This meant producing rough sketches of EA maps with poorly defined boundaries and very rough estimates of population sizes. Such sloppy handling of the delineation of EAs resulted in under-coverage in some areas owing to missing boundaries and some duplication owing to the overlap of some EAs. In principle, EAs should be mutually exclu­

sive and cover the whole country. However, under-coverage is the main problem associated with many African censuses. Where post-enumeration surveys have been conducted, under-coverage rates have been as high as 1 7 per cent. This would have had an impact on the reliability of the results used to estimate some MDG indicators. Under-coverage is one of the components of non-sampling errors associated with censuses. It is therefore important to control non-sampling errors at every stage of census activities, from planning to analysis of results. With regard to minimizing coverage error, it is incumbent upon African countries to carry out a comprehensive census-mapping exercise that produces well-defined EAs and covers the whole country or the part of the country designated for the conduct of the census. Census enumerators and supervisors should be well trained in the art of identifying and covering their assigned EAs.

47. Pilot census: Prior to conduct of the census, a pilot census should be carried out, preferably under similar census conditions, a year before the census. The pilot census is a rehearsal of the ac­

tual census that allows the field conditions, logistics, draft census questionnaire, data processing, and so forth to be tested. The results are used to refine and finalize the census questionnaire; deter­

mine workloads of field staff; review logistics; and determine the data-processing strategy. All these efforts are intended to enhance the quality of census results, including data used in computing MDG indicators, by minimizing non-sampling errors (errors not caused by sampling but human errors such as data-entry errors, biased questions in the questionnaire, biased processing, false information pro­

vided by respondents, etc.).

48. Post-enumeration surveys: A census is traditionally a massive operation making errors inevi­

table, whatever precautions are put in place; the difference among countries is the degree of error.

The primary objective of a census evaluation programme is to determine the sources and magnitude

of coverage and content errors for some selected variables. For many developing countries, the post-enumeration survey (PES) has become a plausible independent evaluation programme. This is partly because other independent sources of data with relevant, comprehensive and reliable information are still rare (ECA, 1999) [7], for example, civil registration.

49. The PES is a complete re-enumeration of a representative sample of a census population matching each individual enumerated in the PES with information from the census enumeration (United Nations, 2008) [8]. Thus, the results of the comparison are mainly used to measure cover­

age and content error in the context of the census. Coverage error refers to people missed in the census or those included erroneously. On the other hand, content errors are identified by evaluating the response quality of selected questions in a census. It is also a basis for evaluating the reliability of some characteristics reported in the census. Evaluation of the magnitude and direction of errors in a census is necessary in order to present to users the extent of reliability and accuracy of some characteristics reported in the census; and it is advisable to conduct a post-enumeration survey as part of the pilot census and immediately after the census.

50. In addition to evaluating coverage and content errors for some census items, the results of a PES have other practical uses: they offer an opportunity to learn from procedural and conceptual limitations in the census that need improvement in future censuses; a PES can identify erroneous procedures used in a census; in conducting subsequent censuses, some lessons learned from the PES can be used to improve implementation and methods of future censuses.

Statistical capacity

51. Statistical capacity encompasses a number of elements such as: the organizational structure of the NSS; human and financial resources; statistical training, and data collection, processing, analysis and dissemination capabilities. In many African countries, the statistical capacity is weak.

As stated earlier, collecting a myriad of MDG statistics requires a mixture of sound data sources such as civil registration, sample surveys, censuses and administrative records, all of which require viable national statistical systems. In some countries, NSOs and other producers of statistics, such as line ministries, do not have the capacity to produce high-quality MDG-related statistics as a result of the lack of trained human resources, high staff turnover, and inadequate resources 19]. In July 2006, the United Nations Economic and Social Council adopted its resolution 2006/6 on strengthening statistical capacity in countries and included a set of recommendations to improve the coverage, transparency and reporting on all indicators.

Creation of databases

52. The major sources of data for MDG indicators include censuses, surveys and administrative records, so the relevant data needs to be in one or more databases. In this regard, the NSO would be the appropriate agency to maintain the database as in most countries it is the institution responsible for statistical coordination.

53. Advances in information and communication technology, coupled with improved coordina­

tion in establishing priorities and standards by the national statistical system, can make a big

dif-12 Be s t Practices Report on Millennium Development Goals Monitoring and Reporting at National and Subnational Levels in

ference. Statistical outcomes will show more consistency and better comparability over space and time.

54. At the international level, the IAEG set up a Task Force to work out a mechanism for the exchange of MDG data within a country and between international agencies. The Task Force has been working on the Data Structure Definition (DSD) and Code Lists for MDG indicators. The Sta­

tistical Data and Metadata Exchange (SDMX) standards give technical specifications for exchanging statistical and metadata. They have created the DSD, which is flexible, enabling agencies to report data using SDMX (UNSD, 2010) [10]. The challenge is to use SDMX universally as a basis for data exchange between countries and international organizations.

Comparability of MDG statistics

55. One of the major challenges to MDG statistics is ensuring comparability of data over a period of time within a country and at international level. Comparability of data is a problem especially when different sources are combined. In general, statistics have greater usefulness when they are amenable to comparison over space and time (Depoutot, 1998) [11]; this is certainly true for most MDG statistics. In order to monitor change across geographic, sectoral, and temporal dimensions, the comparability of MDG statistics needed requires the use of common concepts, definitions and, to some extent, methodologies for data collection and analysis.

56. While the collection of comparable MDG statistics is difficult, their importance is increasing.

Problems associated with assembling cross-national comparable data include the need for the low­

est common denominator, the burden created on responding countries while the cross-national data may not be specific to national needs. There is a lack of metadata supporting most cross-national data, making the interpretation and comparison of MDG data problematic.

Specific compilations such as CO, data

57. Countries are obviously facing problems in the compilation of specific indicators that are not regularly produced by national statistical authorities because they are not relevant in the national context or are not among their specific priorities. An example of this is CO emissions data. Inter­

national organizations responsible for the MDGs and this particular indicator should assist African countries in addressing the specific data required and the recommended method for computing the indicator.