Data and Indicators

Data can be defined as individual points of observation that are collected on a particular subject. Collecting data on environmen-tal health services is not an evaluation goal, but is instead a means to gain an understanding of these services. Although critical to evaluation, data are only part of the evaluation process. By them-selves, data are a useless mass of figures (for example, on health status); but when they are put into a proper context and inter-preted, they become information that is useful to managers and decision-makers.

Data and Indicators

I

:

Indicators measure change. They are variables and are used as sur-rogates for measuring a parameter of interest, when practical diffi-culties prevent the exact measurement of that parameter. In a sense, an indicator is an empirical model that accurately reflects an event, even though it is only an indirect or partial measure of a complex situation. For example, the presence of high concentrations of sulfur dioxide in air alone is not necessarily a health risk, but sulfur diox-ide concentrations are often used as an indicator of overall air quality.

Indicators are particularly useful when measured over a period of time, so that they track direction, magnitude and rate of change.

They are also useful for comparing different areas (or groups of people) at the same moment in time. Though indicators are usually indirect measures, they can assist managers in making inferences and projections that can facilitate decision-making and other man-agement functions.

Three major applications for indicators can be identif ied:

decision-making, communication and policy follow-up.

Indicators support decision-making and comparisons by provid-ing information about efficiency, effectiveness and problem situa-tions. Decisions about priority of services, sequence of activities and allocation of resources are often based on a series of indica-tors.

Indicators are critical for productive communication because they promote a common understanding through the use of a consistent framework for measurement. They allow international organizations and national and local authorities to communicate with each other and with their many partners (or stakeholders), including communi-ties, institutes and NGOs.

Indicators also help determine the degree to which goals are met and policies followed. In particular, performance indicators are be-ing used increasbe-ingly to monitor implementation. For example, the Protocol on Water and Health, signed by the Member States of the

European Region at the Third Ministerial Conference on Environ-ment and Health in London, United Kingdom, in 1999 (1) mandates that the signing parties will:

Collect and evaluate data on:

(a)their progress towards the achievement of the targets [set forth in the protocol]; and

(b) indicators that are designed to show how far that progress has contrib-uted towards preventing, controlling or reducing water-related disease.

O

This section presents a set of qualities (or objectives) that evaluators should seek for indicators (2). Only some of these qualities can be maximized simultaneously, because no single indicator can posses them all. A suite of indicators is commonly used to avoid the limita-tions of any one indicator.

Validity

An indicator should accurately measure what it is supposed to meas-ure. If the data from which the indicator is calculated are inaccurate or are subject to unforeseen variations, the indicator will not be valid.

For instance, the validity of past data from the CCEE has been ques-tioned because the Soviet system fostered inaccurate record keep-ing. An example of this is discussed in Chapter 6 in a case study entitled Assessment of environmental monitoring in Estonia (page 145). Compliance with international standards (such as those recommended by the International Organization for Standardization (ISO)) helps secure data validity.

Internal reliability

When an indicator is reliable, it means that under similar conditions it allows the same inferences to be made by different people. A dis-tinction can be made between internal and external reliability. Inter-nal reliability means that an indicator should provide the same basis for conclusions within one population or geographical area. For ex-ample, the number of telephone complaints received by a local au-thority may not be a reliable indicator in some underdeveloped or

impoverished areas because it presumes that citizens who wish to make complaints have access to a telephone.

External reliability

External reliability demands that the same conclusions be drawn from indicators, irrespective of the population’s size or location – that is, gen-eralizations can be made with these indicators. For example, using the number of students per 100 inhabitants as an indicator of the level of training of the inhabitants in a particular city could be highly mislead-ing, either because of the existence (or lack of) a university in the city or because of the degree of prestige and popularity of the universities in the city. So, a city of the size of Oxford (home of a university) but situated 50 km away will have many fewer students per 100 inhabitants than does Oxford, although the overall level of training of the inhabit-ants may not be very different. Another indicator would probably be required to measure the average training level of the population.

Specificity

An indicator is specific if its value remains stable when other data with a similar context change – that is, if it reflects only the parameter in question. The indicator must change when the parameter changes and must remain constant when the parameter remains constant. Being as-sured of specificity is difficult, mainly because so many factors (such as socioeconomic status and unemployment rate) interact with one an-other and are difficult to separate. For instance, life expectancy is not a specific indicator of good health or of long life. Life can be long, but very unhealthy. A more specific indicator would be the average life expectancy at the age of 40 and might include a list of disabilities.

Sensitivity

An indicator is sensitive if small fluctuations in the parameter are re-flected by small fluctuations in the indicator itself. Again, life expect-ancy is not a particularly sensitive indicator of health status because it is unlikely that small gains or fluctuations in overall health status are re-flected accurately.

Relevance

Indicators are said to be relevant if they relate to the appropriate data or phenomenon studied. Relevance is not as much a property of an indicator as a property of a group of indicators within a given

framework. Some indicators are thought to indicate ideas, which they in fact may not. For instance, the percentage of solid waste recycled in a city is certainly not an indicator of sustainability, although it might be argued that more recycling implies more sustainability; this is not the case because recycling by itself is not a particularly rel-evant indicator of sustainability. Nevertheless, combining recycling with the level of waste produced can provide a good measure of sustainability (less waste and more recycling is more sustainable).

Easily obtainable

Minimizing the burden on those providing data is desirable. Meas-urements are limited by such practicalities as cost and ease of col-lection. When the choice is available, it is better to collect data that are more easily obtainable than not. Sometimes “easily obtainable”

is translated to “already collected,” but such an approach is too sim-ple. In order to follow the new development model and promote overall sustainability, simply relying on existing data by default is problematic. As mentioned earlier, sulfur dioxide is often used to indicate air quality because most localities, if they measure any air pollutants, measure sulfur dioxide. But just because data for sulfur dioxide are very common does not necessarily mean that they are the best indicator of air quality. Evaluators must make trade-offs among indicators that are readily available and those that are most desirable. The available indicators may not be exactly suitable for what is desired, but collecting additional information is not always feasible, owing to expense, political will and other factors.