Techniques for evaluating projects and programs

6.1. Economic evaluation

Economic evaluation is an important tool for assessing the micro efficiency and effectiveness of phar-maceutical programs in terms of costs and outcomes, as well as cost/effect relationships for individual medicines. In seeking to maximise both effectiveness and efficiency despite constraints on resources, governments and industry have increasingly used economic evaluations to compare alternative courses of action in terms of both their costs and health outcomes. Economic evaluations generally draw upon decision analytic models so that the cost and health consequences of alternative treatment and probable pathways can be simultaneously analysed. Decision trees are useful tools for calculating and comparing the expected cost per patient outcome of alternative treatment strategies. Using a decision tree, an ana-lyst is able to model and compare the costs and outcomes of a medical intervention (e.g., drug therapy) against that of alternative forms of treatment and (or) other medical or surgical interventions. This type of evaluation can simultaneously take into account the alternative means of delivering pharmaceutical care in terms of method, place, timing or quantity, at all phases of treatment [20].

There are essentially three basic types of economic evaluation: benefit analysis (CBA), cost-effectiveness analysis (CEA) and cost-utility analysis (CUA). The debate is still on going as to which of these approaches is to be preferred.

Cost-benefit analysis(CBA) measures both cost and benefits in monetary terms as it seeks to identify interventions that will bring about potential improvements in welfare. This technique involves the need to place a monetary value on items such as health benefit. A number of means of doing this exist (e.g., the human capital method or measures of an individual’s willingness-to-pay such as revealed preferences or contingent valuation), but to some extent the figures are unavoidably artificial. For example, the human capital approach does not measure the individual’s willingness to pay for their own health and survival [21]. Although revealed preferences and contingent valuation do attempt to measure willingness-to-pay, the former involves an uncoupling of the valuation of consequences from the context [14], while the latter is still considered experimental.

Cost-effectiveness analysis(CEA) again measures costs in monetary units but it expresses effectiveness using a biological yet quantifiable unit of effect such as the number of lives saved or life years gained.

CEA assesses the incremental costs and incremental effectiveness of one medical intervention relative to another. Unlike CBA, CEA ratios must be compared to an external standard and outcomes common to both alternatives are needed in order to judge relative desirability. A special form of CEA known as cost-minimisation analysis is essentially limited to comparing costs between alternative forms of treatment where the effectiveness of each has been demonstrated or can reasonably be assumed to be the same.

Cost-utility analysis (CUA), like CEA, is concerned with incremental analysis. CUA, however, quan-tifies effect in terms of “utility”. Utility in this sense represents subjective values (i.e. quality weights) given to their overall health status; over a period of time one may express the health status in “quality adjusted life years” (QALYs) or “health related quality of life (HRQL). These quality weights have the merit of combining both the positive and negative effects of a treatment into a single summary score that usually ranges between 0 (equal to death) and 1 (equal to perfect health). It is possible to compare health outcomes in different diseases using the elicited values. Combining health state utilities with cost information renders it possible to calculate cost-utility ratios for alternative treatments.

In spite of advances made in carrying out such measurements, the application of CUA is currently limited by problems associated with the determination of utility values and QALYs. In many instances, for example, utility scores cannot be obtained from patients themselves for such reasons as severity of illness, mental capacity or age, and they have to be derived by the observer from other groups in society.

But this may be a good thing, as many observers believe that in systems where the public pays for health care, QALYs should be elicited from a random sample of the population.

Conducting an economic evaluation requires data on costs (i.e. unit costs or prices of resources), phys-ical quantity of resources consumed (by the treatments being compared) and outcomes (i.e. defining comparative treatment effects and how treatment affects quality of live) [7]. The basic data requirements for both health and economic outcomes were discussed in Chapter 2. The data needed to conduct an economic evaluation will depend on the perspective or viewpoint used to determine the range of costs.

Commonly chosen perspectives for economic evaluations are those of the health care provider, the pa-tient, the community and society generally. The societal perspective includes all costs born by patients, insurers or other parties, while the patient perspective includes only those costs born by the patient.

Therefore, changing the viewpoint will affect the range of costs considered and could affect the choice of programme or treatment being evaluated.

The data needed also depends on the time frame used. The time frame used for collecting data on costs and outcomes should not be misleading and bias the analysis in favour of one intervention over another. If costs are followed over a lifetime then it is standard practice to discount these future costs to present values. Methodological debates surrounding economic evaluations and how they are used by decision-makers is discussed in Chapter 5.

6.2. Evaluating quality

Several methods exist for evaluating the quality of pharmaceutical care provided from prescribing through to delivery. The narrowest of these is the Drug Utilisation Review (DURs). DURs are used to evaluate the level and pattern of drug use. They were originally motivated by concerns about excessive or inappropriate prescribing [8]. DURs can be used to document the association between inappropriate prescribing and its adverse clinical and/or economic consequences. Appropriateness of drug use can be assessed on three levels:

(i) whether any medication is warranted;

(ii) if indicated, which drug would be the agent of choice;

(iii) whether the chosen drug is used in an appropriate manner (dosage, duration, type and frequency of monitoring and the risk of drug interactions) [5].

However, it is not always easy to make such an objective assessment at any of these three levels. To determine acceptability one will have to set the identified pattern of use against some recognised standard

which is applicable to that particular therapy (for example as defined in an authoritative therapeutic guideline). This can be time-consuming [11] and even controversial but necessary.

A second common method of evaluating quality is that of audit. Audit is used to determine whether an optimal quality of service is being achieved within the resources available. The evaluation can be used to uncover areas of problem or concern to prescribers and their patients, to determine whether there is an effective use of resources to achieve desired outcomes or to address equity considerations in terms of geographical, social or ethnic differences in service use and prescribing patterns in relation to population needs. There are two main types of audit;medical auditevaluates the quality of medical care, including procedures used for treatment, the use of resources and the resulting outcome and quality of life enjoyed by patients [22] whileclinical auditfocuses on the quality of professional care by doctors, nurses, pharmacists and other health care providers (e.g., in the delivery of pharmaceutical services).

If the findings are indeed favourable, health providers and users can be reassured; if not, changes will be needed. Adjustments to service made in the light of medical audit findings can lead to improved patient outcomes, better quality of life and the more cost-effective use of resources. A prerequisite to audit is the availability of accurate patient information [2]. The main disadvantage of audit is that it takes a considerable amount of money and professional time.

Finally, quality assurance provides a systematic approach to evaluating health and pharmaceutical care.

A WHO working group [25] defined four components of quality assurance in health care: patient satis-faction with services provided; professional performance (technical quality); resource use (efficiency);

and risk management (the risk of illness or injury associated with the service). Quality assurance eval-uations of prescribing for example have focused variously onprescribing patterns to assess physician competence and appropriateness in prescribing (i.e. performance) [15], on drug utilisation to evaluate different strategies to contain costs (i.e. resource use) [17], oncompliancewith prescribing guidelines to monitor the continuum of quality use of medicines between in-patient and out-patient care [19], and on themonitoring of preventable adverse outcomes(risk management). Important in performing a quality assurance assessment will be appropriate and valid indicators as well as a meaningful time-frame. For example, one study that was evaluating adherence to prescribing guidelines found that indicators evalu-ated at an individual patient level give differing results to aggregate data, and self-reported instruments were found to overestimate adherence [24].

6.3. Production and cost models

The extent to which efficiency has been achieved can be assessed using production and cost models.

These models apply production and cost functions to any production process; they determine the particu-lar mix of inputs which will result in the lowest costs for any given level of output and the optimal size of the production if costs there are to be kept to a reasonable minimum. These models have been applied to physician, hospital and insurance services to determine the extent to which production efficiencies have been achieved and how they can be further improved [1].

6.4. International comparisons

In an attempt to gain a perspective on the macro efficiency of the drug system, helpful comparisons can be made to other politically and economically similar countries. One must however beware of possible pitfalls in making international comparisons because of cultural, demographic and political differences between countries, as well as, methodological issues associated with the data used in the comparisons;

the matter has been previously discussed in Chapter 2 and is touched on again in Section 7 below.