Application of patient safety indicators
internationally: a pilot study among
seven countries
SASKIA E. DRO¨ SLER1, NIEK S. KLAZINGA2,3, PATRICK S. ROMANO4,5, DANIEL J. TANCREDI5,
MARIA A. GOGORCENA AOIZ6, MOIRA C. HEWITT7, SARAH SCOBIE8, MICHAEL SOOP9,
EUGENE WEN10, HUDE QUAN11, WILLIAM A. GHALI11, SOEREN MATTKE12AND EDWARD KELLEY13 1
Faculty of Industrial Engineering and Health Care, Niederrhein University of Applied Sciences, Reinarzstrasse 49, Krefeld, Germany,
2
Academic Medical Center, University of Amsterdam, Amsterdam, The Netherlands,3Health Division, Organization for Economic Co-operation and Development, Paris, France,4Division of General Medicine, University of California Davis School of Medicine, 4150 V Street, PSSB Suite 2400, Sacramento, CA, USA,5Center for Healthcare Policy and Research and Department of Pediatrics, University of California, Davis, Sacramento, CA, USA,6Institute of Health Information, Ministry of Health, Madrid, Spain,7Health Care Safety and Quality Unit, Australian Institute of Health and Welfare, Canberra, Australia,8National Patient Safety Agency, London, UK,9Unit for Qualifications and Patient Safety, Swedish National Board of Health and Welfare, Stockholm, Sweden,10Canadian Institute for Health Information, Toronto, Canada,11Department of Community Health Sciences, University of Calgary, Calgary, Canada,12Rand Corporation, Washington, DC, USA, and13World Health Association’s Patient Safety Programme, Geneva, Switzerland
Abstract
Objective. To explore the potential for international comparison of patient safety as part of the Health Care Quality Indicators project of the Organization for Economic Co-operation and Development (OECD) by evaluating patient safety indicators originally published by the US Agency for Healthcare Research and Quality (AHRQ).
Design. A retrospective cross-sectional study.
Setting. Acute care hospitals in the USA, UK, Sweden, Spain, Germany, Canada and Australia in 2004 and 2005/2006. Data sources. Routine hospitalization-related administrative data from seven countries were analyzed. Using algorithms adapted to the diagnosis and procedure coding systems in place in each country, authorities in each of the participating countries reported summaries of the distribution of hospital-level and overall (national) rates for each AHRQ Patient Safety Indicator to the OECD project secretariat.
Results. Each country’s vector of national indicator rates and the vector of American patient safety indicators rates published by AHRQ (and re-estimated as part of this study) were highly correlated (0.821 – 0.966). However, there was substantial sys-tematic variation in rates across countries.
Conclusions. This pilot study reveals that AHRQ Patient Safety Indicators can be applied to international hospital data. However, the analyses suggest that certain indicators (e.g. ‘birth trauma’, ‘complications of anesthesia’) may be too unreliable for international comparisons. Data quality varies across countries; undercoding may be a systematic problem in some countries. Efforts at international harmonization of hospital discharge data sets as well as improved accuracy of documen-tation should facilitate future comparative analyses of routine databases.
Keywords: patient safety, quality indicators, International Classification of Diseases
Introduction
International comparisons of health system performance are gaining popularity. Over the past 5 years, the Health Care Quality Indicators (HCQI) project of the Organization for Economic Co-operation and Development (OECD) has,
despite several methodological hurdles, made progress in developing and reporting on internationally comparable indi-cators of quality of care [1, 2].
Patient safety is considered an important aspect of quality of care, and many countries have expressed interest in com-parable information to facilitate benchmarking and enhance Address reprint requests to: Saskia E. Dro¨sler, Faculty of Industrial Engineering and Health Care, Niederrhein University of Applied Sciences, Reinarzstrasse 49, Krefeld, Germany. Tel:þ49 21518226643; Fax: þ49 21518226660; E-mail:
mutual learning. As a first step in OECD’s HCQI project, an international expert panel selected in 2004 potential patient safety indicators from the literature based on three criteria: importance for patient safety, scientific soundness and potential feasibility for international data collection. Through a structured ranking process, the expert panel gen-erated a list of 21 patient safety indicators from 59 candidate indicators [3, 4]. An important source was the patient safety indicator set of the US Agency for Healthcare Research and Quality (AHRQ), which contributed the 12 indicators shown in Table 1. The next step in the HCQI project was to assess the feasibility of using these 12 indicators to study patient safety across multiple countries. Seven OECD countries agreed in 2006 to join such a feasibility study, and the results are described in this article.
Methods
Patient safety indicators
The AHRQ Patient Safety Indicators were developed on behalf of the AHRQ by a team at the University of California and Stanford University. Precise documentation on indicator definitions and on the process for indicator devel-opment, selection and continuous review is available online in the public domain [5, 6]. The indicators exclusively rely on routinely collected hospital data that report the diagnoses, procedures, diagnosis-related groups (DRG) and selected patient-related data elements pertaining to each hospitaliz-ation. Since a separate but related set of pediatric quality indi-cators was created by the AHRQ in 2005, all of the AHRQ Patient Safety Indicators except obstetric injuries have been limited to adults aged 18 or more years at admission. In the USA, selected indicators are used for comparative hospital
ranking in several states [7] and country-wide public report-ing [8]. The indicator definitions published by the AHRQ are based on the International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM). The AHRQ offers a subset of indicators for application on area-level data to assess the quality of care in certain geographical regions or other defined populations (e.g. members of a specific insur-ance plan). The area-level indicators are calculated differently from the hospital-level indicators as the definition of numer-ator events at the area level is broader. In the OECD project, hospital-level indicators are used as there is no infor-mation available about geographical regions within the parti-cipating countries.
Data collection
The following seven OECD member countries volunteered to participate in the project: Australia, Canada, Germany, Spain, Sweden, the UK (England only) and the USA. The structure of administrative hospital databases is similar in all participating countries: principal diagnosis and secondary diagnoses are precisely defined and coded according to various versions of the ICD. The number of available sec-ondary diagnosis fields ranges from 6 to 50, whereas for pro-cedure coding between 12 and 100 fields are available. Six different procedure classification systems are in use, so only two of the seven participants use the same procedure catalog. Information on each patient’s age, admission and discharge status (e.g. discharged home, admitted from or transferred to another acute care facility, deceased) and length of stay is also available. To achieve comparability in indicator calculation, several methodological issues had to be taken into account: (i) the participating countries use different ICD versions and procedure classifications,
. . . .
. . . .
Table 1 Overall population patient safety indicator rates (%) for seven OECD member countries
Patient safety indicator Country Ratio between
highest and lowest rate
A B C D E F G
Complications of anesthesia (1) 0.009 0.135 0.145 0.079 0.003 0.023 0.102 48.3 Decubitus ulcer (3) 0.796 0.24 1.264 2.661 0.208 0.765 2.52 12.8 Foreign body left during procedure (5) 0.005 0.007 0.004 0.006 0.002 0.004 0.009 4.5 Selected infections due to medical care (7) 0.146 0.095 0.073 0.281 0.029 0.089 0.251 9.7 Postoperative hip fracture (8) 0.005 0.017 0.033 0.033 0.054 0.005 0.03 10.8 Postoperative pulmonary embolism or deep
vein thrombosis (12)
0.261 0.101 0.617 0.333 0.129 0.116 1.079 10.7 Postoperative sepsis (13) 0.418 0.377 0.317 0.96 0.052 0.246 1.151 22.1 Accidental puncture or laceration (15) 0.166 0.392 0.077 0.144 0.167 0.075 0.356 5.2 Transfusion reaction (16) 0.0003 0.0009 NA 0.0002 0.0002 0.0001 0.0004 9 Birth trauma—injury to neonate (17) 0.521 0.192 0.151 1.448 0.132 0.891 0.261 11 Obstetric trauma—vaginal delivery (18) 1.199 4.104 NA 2.386 4.013 2.289 4.072 3.4 Obstetric trauma—cesarean delivery (20) 0.281 0.783 NA 0.199 0.161 0.096 0.436 8.2 Numbers in column 1 refer to AHRQ Patient Safety Indicators.
(ii) documentation rules, such as definitions of principal and secondary diagnoses, vary among countries and (iii) not all participants use a DRG system for hospital payment. To minimize the impact of these differences, the OECD pro-vided a detailed technical manual to facilitate each country’s calculation of its patient safety indicator rates [9].
Five of the seven participating countries use ICD-10 (or country-specific modifications thereof ) instead of ICD-9-CM. A German project tested the feasibility of patient safety indi-cator calculation in ICD-10 and found high concordance between indicator rates based on American and German hospital data [10]. A British study applied ICD-10 indicator definitions to English data and confirmed the potential for monitoring safety events [11]. An international research con-sortium called the International Methodology Concon-sortium for Coded Health Information evaluated the accuracy of the exist-ing ICD-10 translations and provided an internationally harmonized ICD-10 version for 15 indicators [12, 13]. That existing translation was a major prerequisite to perform com-parisons across countries and across coding systems.
AHRQ Patient Safety Indicator definitions distinguish between the principal diagnosis and secondary diagnoses. If the critical event is coded as the principal diagnosis, it is assumed—according to American coding standards [14]— that the patient was admitted with that condition and that the condition did not occur during the hospitalization. Two participating countries, Canada and Sweden, use a different definition of the principal diagnosis, based on the disease consuming the most resources during the hospitalization. If a hospital-acquired complication turns out to be the principal diagnosis, the case would not be counted as a safety event using the AHRQ logic and would thereby turn false negative. A diagnosis type indicator in the administrative data, indicat-ing whether a condition was present at admission or acquired during hospitalization, can solve this conflict. Canada, using a diagnosis type indicator, was able to reconfigure its admin-istrative database to achieve comparability in calculation of indicator rates.
Although the selected patient safety indicators mainly rely on diagnosis codes, procedure codes are used to help define certain indicators (e.g. postoperative hip fracture or post-operative pulmonary embolism or deep vein thrombosis). As there is no uniform international system for procedure classi-fication, the countries were provided lists of specific pro-cedures and asked to find the corresponding codes and to accomplish the calculation accordingly.
In comparison to the original AHRQ definitions of the patient safety indicators, three indicators were slightly modified for this project. ‘Complications of anesthesia’ cases assigned to MDC 14 (Major Diagnostic Category, obstetrics DRG chapter) are excluded here but not in the AHRQ definition. In addition to certain drug intoxications, this indicator is sup-posed to capture cases in which a failed intubation occurred. The ICD classification carries separate codes to differentiate failed intubations during non-obstetric clinical situations from those that occur during childbirth. The existing ICD codes for intubation failure in obstetric patients are not included in the AHRQ definition. Therefore, it seems inconsistent to include
obstetric cases in the denominator of this indicator. Regarding the indicator ‘obstetric trauma—vaginal delivery’, the AHRQ definition distinguishes between deliveries with tion (forceps or vacuum) and deliveries without instrumenta-tion. To avoid conflicts due to different procedure classifications, these indicators were joined together in the OECD project [15]. Again, due to the lack of a uniform pro-cedure classification system, the propro-cedure-based exclusions used to identify cases with operative treatment were omitted for indicator ‘decubitus ulcer’.
The lead agency in each of the participating countries cal-culated rates after converting the ICD-10-WHO code list to the country-specific version of ICD-10. As the structures of the classification trees are similar, this task was manageable. For each indicator, countries were asked to provide popu-lation rates based on overall national numerator and denomi-nator counts, as well as mean hospital rates and their standard deviations. A separate questionnaire explored tech-nical details about the structure of each country’s administra-tive database and country-specific documentation issues (e.g. mean number of secondary diagnoses).
Analyses
Between-country variations in indicator rates are estimated from ratios between highest and lowest value (displayed in Table 1) and by estimation of the coefficient of variation (CV: ratio between weighted standard deviation and weighted mean, displayed in Table 2). Between-country concordance in the relative magnitude of indicator rates was tested by computing the Pearson’s correlation coefficient between each country’s vector of indicator rates and the USA vector. The American results were regarded here as the ‘reference stan-dard’ because the USA developed the methods of patient safety indicator calculation and has the longest experience in collecting ICD-coded data for payment and surveillance. Statistical analyses were performed using Release 12.0 of SPSS for Windows (Copyright 1989 – 2003, Chicago: SPSS Inc.) and Version 9.1 of the SAS System for Windows.
Between-country comparisons in the relative magnitudes of the vector of national indicator rates were performed in order to identify countries with unusually low, high or mixed results. The 12 indicator rates reported for each country were log-transformed and then treated as components of a geo-metric vector in a multi-dimensional space. The Euclidean distance between each country’s vector and the weighted cen-troid of the vectors from the other countries was computed [16]. The formula used for this calculation is:
ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð p1 q1Þ2þ ð p2 q2Þ2þ þ ð pn qnÞ2 q ¼ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi Xn i¼1 ð pi qiÞ2 s
For the ith indicator (i ¼ 1, . . . , n), the variable pi represents
the logarithmized population rate in the country under investi-gation. The variable qi reflects the weighted mean of the
logarithmized population rates for the ith indicator in the remaining countries. Population rates were log-transformed before computing Euclidean distances to stabilize variations across the indicators. A relatively large Euclidean distance indi-cates that a country is unusual compared with other countries in at least one and generally more than one indicator.
Results
All seven countries returned indicator rates to the OECD secretariat; however, one country did not use the OECD technical manual but instead used a previous version of the AHRQ manual, and therefore had to omit three indicators due to major differences in definitions.
Measures
The sample sizes were estimated from the indicator ‘foreign body left during procedure’, as the denominator of this indi-cator captures nearly all medical and surgical cases. The term ‘procedure’ in the title of this indicator reflects surgical oper-ations as well as medical or interventional treatments (e.g. endo-scopy). Relative to the overall number of hospital discharges [17], five participating countries used more than 70% of avail-able data and one country used 68%. Taking into account that inpatients younger than 18 years and patients treated in psychia-tric institutions are excluded from calculation of the indicators, six countries included a majority of adult inpatients.
Table 1 depicts the population rates that countries reported to the OECD secretariat, with countries randomly assigned labels A – G. For privacy reasons, single numerator and denominator counts are not shown. These numbers were used for further descriptive analyses, including the weighted means and weighted standard deviations shown in Table 2.
We found systematic variation in indicator rates across countries (Tables 1 and 2), ranging from a 3.4-fold difference across countries for ‘Obstetric trauma—vaginal delivery’ (CV ¼ 22%) and a 4-fold difference for ‘foreign body left in’ (CV ¼ 29%) to an 11-fold difference across countries for ‘birth trauma’ (CV ¼ 75%) and a 48-fold difference for ‘com-plications of anaesthesia’ (CV ¼ 57%). Reliability analyses (not displayed) showed that only one indicator ‘birth trauma’ was not positively associated with other rates at the national level.
In the concordance analysis for relative magnitudes of each country’s vector of indicator rates with the American vector, we found significant correlations with Pearson’s coef-ficients (P 0.01), ranging from 0.821 to 0.966. These results suggest that the within-country relative magnitudes of various indicators are concordant to US data.
Regarding the raw data given in Table 1, the question arises whether there is systematic variation at the country level. For example, is there a country showing overall high or low rates compared with other countries? The raw data reveal that Country G has the highest or second highest rates for eight indicators. Rather low rates were reported by countries E (lowest rates for six indicators) and F (lowest or second lowest rates for eight indicators). The grey columns in Fig. 1 show the studentized Euclidean distances for the participating countries using all indicator rates. Our hypoth-esis that countries E and F are outliers compared with the other countries is supported, as they show the greatest dis-tances from the centroid. To verify this result, the compu-tation was repeated using divided data with two sets of six indicators each. Set 1 includes indicators with rather high variability (1, 3, 8, 12, 15 and 17), whereas set 2 (5, 7, 13, 16, 18 and 20) contains more stable indicators. The white and dark columns in Fig. 1 represent the Euclidean dis-tances generated from both data sets. For both data sets, the ranking of the Euclidean distances is concordant with the unsplit data. The low Euclidean distances for Country
. . . .
Table 2 Weighted mean population rates (%) and weighted standard deviations for patient safety indicator rates across participating countries
Patient safety indicator n(countries) Rate (%) weighted mean Rate (%) weighted standard deviation CV (%) Complications of anaesthesia 7 0.07 0.04 57 Decubitus ulcer 7 1.9 0.9 47
Foreign body left during procedure 7 0.007 0.002 29 Selected infections due to medical care 7 0.2 0.1 50
Postoperative hip fracture 7 0.02 0.01 50
Postoperative pulmonary embolism or deep vein thrombosis
7 0.6 0.4 67
Postoperative sepsis 7 0.8 0.4 50
Accidental puncture or laceration 7 0.3 0.1 33
Transfusion reaction 6 0.0003 0.0002 67
Birth trauma—injury to neonate 7 0.4 0.3 75
Obstetric trauma—vaginal delivery 6 3.7 0.8 22
C must be interpreted with caution due to three missing values.
To understand the variation at the institutional level, countries were asked to report mean hospital rates and their standard deviations. Table 3 shows the means and standard deviations of within-country hospital-level rates for selected indicators. Five of seven countries reported these data; the number of institutions included in this analysis ranged from 237 to 698 acute care hospitals. For all indicators, the stan-dard deviations are relatively large; however, Country E shows the highest standard deviations for most of the indi-cators. As these calculations were performed without denomi-nator weighting, the results should be interpreted cautiously. Owing to their unweighted calculation, mean hospital rates in Table 3 differ from population rates shown in Table 1.
Discussion
Although it would be premature to make inferences about the safety of patient care in the participating countries, our study demonstrates that the AHRQ Patient Safety Indicators can be applied to hospital data from multiple countries. One main methodological challenge, the consistent translation of
indicator definitions from ICD-9-CM to ICD-10, was over-come through the international research consortium. However, differences in ICD coding guidelines and practices, such as optional versus mandatory use of certain codes to describe external causes of disease, affect the rates of those indicators that are defined using E-codes (e.g. ‘complications of anaesthesia’ and ‘foreign body left during procedure’).
This project selected only patient safety indicators primar-ily defined on diagnoses, as there is no common procedure classification across countries. If a reference procedure classi-fication issued by WHO was implemented across countries, it would permit electronic linkage between different country-specific procedure classifications using information technol-ogy methods such as cross walking. A recent validation study from the USA revealed that those indicators built largely on procedure codes are relatively accurate (e.g. ‘postoperative wound dehiscence’) [18].
Two of the participating countries, the USA and Canada, have implemented a marker in their hospital data sets, indicat-ing whether or not each coded diagnosis was present at admission or occurred during hospitalization. However, the American data used in this analysis predated the implemen-tation of this data element. This marker facilitated the ana-lyses of Canadian data, as extensive rearrangements had to be performed to account for the fact that Canada uses a differ-ent definition of principal diagnosis, and generally increases the usefulness of administrative data for performance mea-surement [19]. To improve the validity of safety event identifi-cation using coded hospital discharge data, the present-on-admission indicator should be introduced internationally.
Substantial variations in rates for some indicators could not be elucidated, as access to the underlying databases is restricted to the responsible persons in the participating countries. Hospital reimbursement relies on coding of diag-noses and procedures (e.g. DRGs) in all participating countries except Country E. For this country, underreporting is likely, as hospitals lack any clear financial incentive to code diagnoses thoroughly. This factor could also explain the unusually large variation in rates across hospitals in Country E. The fact that patient safety indicator rates heavily rely on the quality and completeness of ICD coding is supported by Fig. 2. Countries Figure 1 Studentized Euclidian distances of logarithmized
patient safety indicator rates (three missing values for Country C).
. . . . . . . .
Table 3 Variation across hospitals (mean hospital rates [%, unweighted] and standard deviations) of selected patient safety indicators from five countries
A B C E F
Mean SD Mean SD Mean SD Mean SD Mean SD Decubitus ulcer 1.37 2.95 0.12 0.27 1.38 1.57 0.21 14.70 0.94 1.87 Foreign body left during procedure 0.004 0.008 0.027 0.036 0.004 0.009 0.001 0.780 0.002 0.010 Selected infections due to medical care 0.14 0.25 0.03 0.08 0.09 0.41 0.02 7.20 0.04 0.08 Postoperative pulmonary embolism or
deep vein thrombosis
0.25 0.46 0.19 0.78 0.60 0.54 0.11 0.97 0.25 1.75 Postoperative sepsis 0.34 0.63 0.63 4.15 0.30 0.29 0.05 0.51 0.19 0.22 Accidental puncture or laceration 0.13 0.12 0.34 0.27 0.06 0.08 0.10 66.21 0.04 0.07
with a higher mean number of secondary diagnoses report increased rates of the indicator ‘postoperative deep vein thrombosis and pulmonary embolism’. The surprisingly high number of secondary diagnoses in Country E is related to its specific documentation rules. Any medical condition, regard-less whether or not it has an impact on medical treatment, can be noted. Other factors, such as access to health care or ethnic disparities, might also affect rates within and across countries [20]. To evaluate systematic variations across countries, the method of estimating Euclidian distances is feasible and shows similar trends in repeated measurements.
The participating countries are now interested in investi-gating the possible causes of lower or higher indicator rates. For example, questions have been raised about differences in documentation, differences in average length of stay (leading to a shorter or longer time at risk) and differences in coding practice (resulting from how ICD codes are used to set hos-pital payments). The focus of this investigation was simply on establishing the feasibility of performance measurement using administrative hospital data in an international setting. Further analyses using stratified or risk-adjusted data are now underway. Secondary data analysis is a very feasible method for quality reporting within countries, but the quality of the collected data is a major concern. For this reason, validation studies based on medical record reviews are currently under-way in the USA and elsewhere [21].
The AHRQ Patient Safety Indicators are designed to monitor in-hospital quality of care; however, the declining length of in-hospital stay must be taken into account in the future. Quality management and concerns about patient safety should not stop at the hospital doors. Data systems should be developed to permit tracking post-hospital compli-cations that are closely related to the preceding hospital stay, such as postoperative pulmonary embolism. The general introduction of a unique patient identifier, already introduced in several Northern European countries, will be essential to permit quality management across settings of patient care.
Conclusion
The results demonstrated feasibility of implementation and quantified the amount of variation in patient safety indicator rates. The next challenge, however, is now to figure out how to interpret the variation seen, because it relates to either (i) true variation in patient safety, (ii) variation in coding and data quality or (iii) a combination of (i) and (ii). The way forward is to now conduct validation studies and harmoniza-tion of coding rules and data quality across countries. OECD member countries have expressed high interest in continuing the project and 10 additional countries partici-pated in the 2008 calculation round. In this study, more detailed information on patient populations will be captured to gain greater insight into the safety of hospital care interna-tionally. If this research proves to be successful, the OECD anticipates publishing comparable data on patient safety in future publications such as Health at a Glance.
Acknowledgements
This investigation was initiated by the OECD as part of the HCQI project. Furthermore, appropriate health agencies in Australia, Canada, Germany, Spain, Sweden, the UK and the USA supported this research by performing the calculations and providing the data. This article reflects the opinion of the authors and does not represent an official position of the OECD, its member countries or institutions participating in the project. Members of the patient safety indicator interest group of the International Methodology Consortium for Coded Health Information (www.imecchi.org) are acknowl-edged for providing an internationally harmonized version of indicator definitions in ICD-10.
Funding
Organization for Economic Co-operation and Development (OECD).
References
1. Organization for Economic Co-Operation and Development. OECD Health Data 2008. http://www.irdes.fr/EcoSante/ DownLoad/OECDHealthData_FrequentlyRequestedData.xls (27 February 2009, date last accessed).
2. Organization for Economic Co-Operation and Development. Health at a Glance. Paris: OECD, 2007.
3. Mattke S, Epstein A, Leatherman S. The OECD Health Care Quality Indicators Project: history and background. Int J Qual Health Care2006;18(Suppl. 1):1 – 4.
4. McLoughlin V, Millar J, Mattke S et al. Selecting indicators for patient safety at the health system level in OECD countries. Int J Qual Health Care2006;18(Suppl. 1):14 – 20.
Figure 2 Dependency between mean number of secondary diagnoses and population rate (%) of indicator postoperative pulmonary embolism (PE) or deep vein thrombosis (DVT).
5. Agency for Healthcare Research and Quality. Quality Indicators— Guide to Patient Safety Indicators. Version 2.1, revision 3 (17 January 2005). Rockville, MD: AHRQ, 2003. http:// www.qualityindicators.ahrq.gov/psi_archive.htm (27 February 2009, date last accessed).
6. McDonald K, Romano P, Geppert J et al. Measures of patient safety based on hospital administrative data. The Patient Safety Indicators. Technical Review 5. Rockville, MD: AHRQ, 2002. 7. State of Florida. Agency for Health Care Administration. http://
www.floridahealthfinder.gov/CompareCare/SelectChoice.aspx (27 February 2009, date last accessed).
8. Agency for Healthcare Research and Quality. National Healthcare Quality Report, 2005. Rockville, MD: AHRQ, 2005. http:// www.ahrq.gov/qual/nhqr05/fullreport/ (27 February 2009, date last accessed).
9. Dro¨sler SE. Facilitating cross-national comparisons of indi-cators for patient safety at the health-system level in the OECD countries. OECD Health Technical Paper No. 19, 2008. http:// www.oecd.org/dataoecd/24/48/40401929.pdf. Accessed (27 February 2009, date last accessed).
10. Dro¨sler SE, Cools A, Ko¨pfer T et al. Are quality indicators derived from routine data suitable for evaluation hospital per-formance? First Results Using the AHRQ Patient Safety Indicators in Germany [in German]. Z Arztl Fortbild Qualitatssich2007;101:35 – 42.
11. Raleigh VS, Cooper J, Bremner SA et al. Patient safety indi-cators for England from hospital administrative data: case-control analysis and comparison with US data. Br Med J 2008;337:a1702.
12. De Coster C, Quan H, Finlayson A et al. Identifying priorities in methodological research using ICD-9-CM and ICD-10 administrative data: report from an international consortium BMC Health Serv Res2006;6:77.
13. Quan H, Dro¨sler S, Sundararajan V et al. Adaptation of AHRQ Patient Safety Indicators for Use in ICD-10 Administrative Data by an International Consortium. Agency of Healthcare Research and Quality 2008. http://www.ahrq.gov/downloads/pub/advances2/
vol1/Advances-Quan_52.pdf (27 February 2009, date last accessed).
14. Centers for Medicare and Medicaid Services (CMS). ICD-9 CM Official Guidelines for Coding and Reporting. http://www.cdc.gov/ nchs/datawh/ftpserv/ftpICD9/icdguide07.pdf (27 February 2009, date last accessed).
15. Millar J, Mattke S the members of the OECD Patient Safety Panel. Selecting indicators for patient safety at the health systems level in OECD countries. OECD Health Technical Paper No. 18, 2004. http://www.oecd.org/dataoecd/53/26/ 33878001.pdf (27 February 2009, date last accessed).
16. Hartigan JA. Clustering Algorithms. New York: Wiley, 1975. 17. Organization for Economic Co-Operation and Development.
OECD Health Data2007. http://www.oecd.org/document/10/ 0,3343,en_2649_201185_38976778_1_1_1_1,00.html (27 February 2009, date last accessed).
18. Romano PS. HCQI Patient Safety Indicators Validity Report. Selecting Indicators for Patient Safety at the Health Systems Level in OECD Countries. Summary of Recent US Experience. OECD 2007. http://www.oecd.org/dataoecd/44/29/39495326.pdf (27 February 2009, date last accessed).
19. Glance LG, Osler TM, Mukamel DB et al. Impact of the present-on-admission indicator on hospital quality measure-ment: experience with the agency for healthcare research and quality (AHRQ) inpatient quality indicators. Med Care 2008;46:112 – 9.
20. Coffey RM, Andrews RM, Moy E. Racial, ethnic, and socioeco-nomic disparities in estimates of AHRQ Patient Safety Indicators. Med Care 2005;43(Suppl.):I48 – 57.
21. Romano PS, Mull HJ, Rivard PE et al. Validity of selected AHRQ Patient Safety Indicators based on VA National Surgical Quality Improvement Program data. Health Serv Res 2009;44:182 – 204.
![Table 3 Variation across hospitals (mean hospital rates [%, unweighted] and standard deviations) of selected patient safety indicators from five countries](https://thumb-eu.123doks.com/thumbv2/123doknet/14884862.646238/5.918.131.482.533.736/variation-hospitals-hospital-unweighted-standard-deviations-indicators-countries.webp)
