Limitations of monitoring and the use of indicators

The use of monitoring and indicators is not without problems. Some specific pitfalls have been noted above, but there are also more general difficulties. While for example performance, as judged on the basis of a single indicator, may have improved, it is important to put this indicator back into the wider policy context and consider whether the overall policy change which brought about this improvement was on balance beneficial. It is particularly important to consider whether using the selected indicators may have introduced perverse incentives that shift problems from one area into another that is not mon-itored. Making an effort to meet indicator targets can mean that other important issues get neglected, a dilemma known as Goodhart’s Law of government policy indicators [9]. According to Goodhart’s law, if some variable is made the focus of measuring policy performance, then people and institutions will devote a disproportionate amount of time and effort to meeting this target at the expense of other aspects of their performance which are not subject to the same level of scrutiny. The result can be disbalance with negative effects. This phenomenon is familiar: for example, although the objective of a negative list may have been to reduce drug costs, it could be that obliging physicians to prescribe other products will result in cost increases in unexpected areas (e.g., because of adverse reactions resulting in hospital admissions).

The obvious solution would be to monitor more variables. However, if too many indicators are selected the evaluation may be rendered so complex or the measured outcomes may prove so contradictory that the findings are difficult to understand and act on. Finally, introducing too many indicators can render the entire process excessively costly and labour-intensive, the phenomenon that has been termed an “audit explosion”.

A further risk is that in selecting indicators there is sometimes a tendency to choose an indicator or target that is not relevant to a particular policy context or health care system. The indicators may not be relevant despite being simple to understand and may have been selected because they were likely to sug-gest a favourable outcome. This tendency to isolate clusters from their context is known in epidemiology as the Texas-sharpshooter fallacy, referring to the story of a Texas sharpshooter who shoots at the side of a barn and then draws a bull’s-eye around the bullet holes.

It is important not too adopt policies from abroad too rapidly merely because an audit carried out there has indicated that they are successful. Even if the foreign audit seems methodologically sound, the same policies implemented in one’s own country may prove disappointing when the results are assessed, be-cause of differences in the medical and organisational context or other difficulties pointed out in earlier sections.

It is important too that any results arising from evaluations be disseminated and action taken so that necessary changes are made. Results should be disseminated to the appropriate decision-makers in a timely manner. Strong leadership and governance principles can be useful to ensure that appropriate changes are made in a uniform manner at a macro level and where applicable at local levels. Problems associated with the dissemination of evaluations are discussed in more depth in Chapter 5 of this book.

8. Conclusion

Before planning any policy evaluation one needs to consider carefully how the pharmaceutical service is structured, how it functions (i.e. its dynamics) and how the goals and objectives of intended policy measures (whether related to cost containment or not) could affect it. Policy priorities will have to be set in advance and means identified of assessing progress. Cost containment policies can themselves be

costly as well as time-consuming, and only with good planning, the setting of clear priorities and careful assessment over time can one ensure that they are worthwhile and will do more good than harm.

The assessment of progress may well lead to the need to adjust, extend or even abandon particular policies. Since both the policies themselves, and any amendments to them as they are implemented, can have widespread implications for local structure and resources, health service managers should be involved in setting the agenda for the evaluations. In addition, the ultimate recipients of service – patients and professionals – should also be involved in assessing the effects of policies in the field. Most important of all, these ongoing measures of progress need to be studied, regularly and promptly, by the policy makers themselves, so that whatever policy changes are called for can be introduced without undue delay.

References

[1] L.A. Aday, C.E. Begley, D.R. Lairson and C.H. Slater, Evaluating the medical care system: effectiveness, efficiency and equity, Foundation of the American College of Healthcare Executives, Ann Arbor, Michigan, 1993.

[2] Audit Commission, A prescription for improvement: Towards more rational prescribing in general practice, HMSO, Lon-don, 1994.

[3] R.H. Brook and E.A. McGlynn, Maintaining quality of care, in:Health Services Research, E. Ginzberg, ed., Harvard University Press, Cambridge, 1991.

[4] P. Brudon-Jakobowictz, J.-D. Rainhorn and M.R. Reich,Model Indicators for Monitoring National Drug Policies, World Health Organization, Geneva, 1995.

[5] E.A. Chrischilles and K. Gondek, Do practice guidelines augment drug utilisation review?,Pharmacoeconomics 12(6) (1997), 648–666.

[6] A.J. Culyer and A. Wagstaff, Equity and equality in health and health care,Journal of Health Economics12(1993), 431–457.

[7] M.F. Drummond, Experimental versus observational data in the economic evaluation of pharmaceuticals,Medical Deci-sion Making18(2) (1998), Suppl. S12–S18.

[8] M.N.G. Dukes, Introduction, in:Drug Utilisation Studies – Methods and Uses, M.N.G. Dukes, ed., WHO Regional Office for Europe, Copenhagen, 1993.

[9] P. Dunleavy, So, what do political scientists do?,New Statesman9(9 October 1998), 3–5, 31.

[10] L. Greenfield, B.D. Colen, P.D. Cleary and S. Greenfield,Evaluating the Quality of Health Care: What Research Offers Decision Makers, Milbank Memorial Fund, New York, 1996.

[11] I.C. Gyssens and B.J. Kullberg, Improving the quality of antimicrobial drug use can result in cost containment,Pharmacy World & Science17(1995), 163–167.

[12] H. Hansluwka, Measuring the health of populations: Indicators and interpretations, Social Science and Medicine 20 (1985), 1207–1224.

[13] W.W. Holland,Evaluation of Health Care, Oxford University Press, Oxford, 1983.

[14] M. Johannesson, B. Joensson and G. Karlsson, Outcome measurement in economic evaluation,Health Economics5(4) (1996), 279–296.

[15] D.M. Krumholz, S.D. Jaanus and R.L. Urquidi, Prescribing patterns of optometrists newly licensed in therapeutic drug use in New York State,Optometry72(5) (2001), 292–298.

[16] J. Le Grand, Equity and Choice: An Essay in Economics and Applied Philosophy, HarperCollins Academic, London, 1991.

[17] C.R. MacIntyre, D. Sindhusake and G. Rubin, Modelling strategies for reducing pharmaceutical costs is hospital, Inter-national Journal of Quality in Health Care13(1) (2001), 63–69.

[18] Management Science for Health,Managing Drug Supply, Kumarian Press, West Hartford, Connecticut, 1997.

[19] A. Mant, W.C. Rotem, L. Kehoe and K.I. Kaye, Compliance with guidelines for continuity of care in therapeutics from hospital to community,Medical Journal of Australia174(6) (2001), 277–280.

[20] A. Maynard and T.A. Sheldon, Health economics: Has it fulfilled its potential?, in:Non-random Reflections on Health Ser-vices Research: On the 25th Anniversary of Archie Cochrane’s Effectiveness and Efficiency, A. Maynard and I. Chalmers, eds, The Nuffield Provincial Hospitals Trust, London, 1997.

[21] E.J. Mishan,Cost Benefit Analysis, 4th edn, Routledge, London, 1988.

[22] A.S. St Leger, H. Schnieden and J.P. Walsworth-Bell,Evaluating Health Services’ Effectiveness, Open University Press, Milton Keynes, 1992.

[23] C.D. Shaw, Health-care league tables in the United Kingdom,Journal of Quality in Clinical Practice17(1997), 215–219.

[24] C.C. Veninga, P. Denig, L.G. Pont and F.M. Haaijer-Ruskamp, Comparison of indicators assessing the quality of drug prescribing for asthma,Health Services Research36(1 Pt 1) (2001), 143–161.

[25] World Health Organisation,Quality Assurance in Health Care, Vol. 1, No. 2/3, WHO, Geneva, 1989.

[26] World Health Organisation/Action Programme on Essential Drugs,How to investigate drug use in health facilities: Se-lected drug use indicators, WHO/DAP, Geneva, 1993.

IOS Press

Chapter 5