• Aucun résultat trouvé

A quality evaluation of guidelines on five different viruses causing public health emergencies of international concern

N/A
N/A
Protected

Academic year: 2022

Partager "A quality evaluation of guidelines on five different viruses causing public health emergencies of international concern"

Copied!
15
0
0

Texte intégral

(1)

Article

Reference

A quality evaluation of guidelines on five different viruses causing public health emergencies of international concern

ZHAO, Siya, et al.

Abstract

This project aims to evaluate the methods and reporting quality of practice guidelines of five different viruses that have caused Public Health Emergencies of International Concern (PHEIC) over 20 past years: the severe acute respiratory syndrome coronavirus (SARS-CoV), Ebola virus, Middle East respiratory syndrome coronavirus (MERS-CoV), Zika virus and the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). We systematically searched databases, guideline websites and government health agency websites from their inception to February 02, 2020 to extract practice guidelines for SARS-CoV, Ebola virus, MERS-CoV, Zika virus, SARS-CoV-2 and the diseases they caused. The literature was screened independently by four researchers. Then, fifteen researchers evaluated the quality of included guidelines using the AGREE-II (Appraisal of Guidelines for Research and Evaluation II, for methodological quality) instrument and RIGHT (Reporting Items for practice Guidelines in Healthcare, for reporting quality) statement. Finally, a total of 81 guidelines were included, including 21 SARS-CoV guidelines, 11 Ebola virus (EBOV) [...]

ZHAO, Siya, et al . A quality evaluation of guidelines on five different viruses causing public health emergencies of international concern. Annals of Translational Medicine , 2020, vol. 8, no. 7, p. 500

DOI : 10.21037/atm.2020.03.130 PMID : 32395544

Available at:

http://archive-ouverte.unige.ch/unige:151753

Disclaimer: layout of this document may differ from the published version.

(2)

Page 1 of 13

A quality evaluation of guidelines on five different viruses causing public health emergencies of international concern

Siya Zhao1,2#, Jin Cao1#, Qianling Shi2,3, Zijun Wang2, Janne Estill4,5, Shuya Lu2,6,7, Xufei Luo1,2, Junxian Zhao1, Hairong Zhang2, Jianjian Wang1,2, Qi Wang8,9, Yangqin Xun1,2, Jingyi Zhang1,2, Meng Lv1,2, Yunlan Liu1, Xiaomin Nie10, Ling Wang1,2, Xianzhuo Zhang2,3, Weiguo Li11,12, Enmei Liu11,12, Xiaohui Wang1, Yaolong Chen2,13,14,15,16; on behalf of COVID-19 evidence and recommendations working group

1School of Public Health, Lanzhou University, Lanzhou 730000, China; 2Evidence-based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou 730000, China; 3The First Clinical Medical College of Lanzhou University, Lanzhou 730000, China; 4Institute of Global Health, University of Geneva, Geneva, Switzerland; 5Institute of Mathematical Statistics and Actuarial Science, University of Bern, Bern, Switzerland; 6School of medicine, University of Electronic Science and Technology, Chengdu 611731, China; 7Department of Pediatric, Sichuan Provincial People’s Hospital, Chengdu 610072, China; 8Health Policy PhD Program, Faculty of Health Sciences, McMaster University, Hamilton, Canada; 9McMaster Health Forum, McMaster University, Hamilton, Canada; 10The Second Clinical Medical College of Lanzhou University, Lanzhou 730030, China; 11Department of Respiratory, Ministry of Education Key Laboratory of Child Development and Disorders, National Clinical Research Center for Child Health and Disorders, China International Science and Technology Cooperation base of Child development and Critical Disorders, Children’s Hospital of Chongqing Medical University, Chongqing 400014, China; 12Chongqing Key Laboratory of Pediatrics, Chongqing 400014, China; 13WHO Collaborating Centre for Guideline Implementation and Knowledge Translation, Lanzhou 730000, China;

14GIN (Guidelines International Network) Asia, Lanzhou 730000, China; 15Key Laboratory of Evidence Based Medicine and Knowledge Translation of Gansu Province, Lanzhou University, Lanzhou 730000, China; 16Chinese GRADE Center, Lanzhou 730000, China

Contributions: (I) Conception and design: Y Chen, X Wang, J Estill; (II) Administrative support: None; (III) Provision of study materials or patients:

S Zhao, J Cao; (IV) Collection and assembly of data: S Zhao, J Cao; (V) Data analysis and interpretation: X Wang, S Zhao, J Cao; (VI) Manuscript writing: All authors; (VII) Final approval of manuscript: All authors.

#These authors contributed equally to this work.

Correspondence to: Xiaohui Wang. School of Public Health, Lanzhou University, Lanzhou 730000, China. Email: wangxiaohui@lzu.edu.cn; Yaolong Chen. Evidence-based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou 730000, China.

Email: chenyaolong@vip.163.com.

Abstract: This project aims to evaluate the methods and reporting quality of practice guidelines of five different viruses that have caused Public Health Emergencies of International Concern (PHEIC) over 20 past years: the severe acute respiratory syndrome coronavirus (SARS-CoV), Ebola virus, Middle East respiratory syndrome coronavirus (MERS-CoV), Zika virus and the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). We systematically searched databases, guideline websites and government health agency websites from their inception to February 02, 2020 to extract practice guidelines for SARS- CoV, Ebola virus, MERS-CoV, Zika virus, SARS-CoV-2 and the diseases they caused. The literature was screened independently by four researchers. Then, fifteen researchers evaluated the quality of included guidelines using the AGREE-Ⅱ (Appraisal of Guidelines for Research and Evaluation Ⅱ, for methodological quality) instrument and RIGHT (Reporting Items for practice Guidelines in Healthcare, for reporting quality) statement. Finally, a total of 81 guidelines were included, including 21 SARS-CoV guidelines, 11 Ebola virus (EBOV) guidelines, 9 MERS-CoV guidelines, 10 Zika Virus guidelines and 30 SARS-CoV-2 guidelines. The evaluation of the methodological quality indicated that the mean scores of each domain for guidelines of each virus were all below 60%, the scores for guidelines in the domains of “clarity of presentation” being the highest and in the “editorial independence” lowest. The mean reporting rate of each domain for guidelines of each virus was also less than 60%: the reporting rates for the domain “background”

were highest, and for the domain “funding and interests” lowest. The methodological and reporting quality of the practice guidelines for SARS-CoV, Ebola virus, MERS-CoV, Zika virus and SARS-CoV-2 guidelines Review Article

(3)

Introduction

In 2019, an outbreak of a new coronavirus, named the severe acute respiratory syndrome coronavirus 2 (SARS- CoV-2; previously known as 2019 novel coronavirus, 2019- nCoV) occurred in Wuhan, Hubei province, China. To February 22, 2020, a total of 77,794 patients had been diagnosed with coronavirus disease-19 (COVID-19), the disease caused by SARS-CoV-2, in 29 countries and regions, causing 2,359 deaths (1). World Health Organization (WHO) considers the epidemic as a high-risk Public Health Emergency of International Concern (PHEIC) (1). PHEIC is defined as an extraordinary situation that constitutes a public health risk to member states other than the country where the situation originated through the international spread of disease, and potentially requires a coordinated international response (2). Locally outbreaking infectious diseases, often known as public health emergency at first, without effective control measures may easily lead to PHEIC. A study found that more than 260 local public health emergencies occurred in Africa between 2016 and 2018, and 87% of African countries occurred at least one public health emergency event (3). Among them, the Ebola virus (EBOV), which outbroke in south African in 2014, has killed over 11,000 people in ten countries and regions (4).

This was also one of the PHEICs with the widest influence and largest death toll over the past 20 years.

WHO constantly focuses on how to effectively control local public health emergencies and prevent them from developing into PHEICs. In 2013, WHO published the Emergency Response Framework (ERF) (5), which details how to respond to local public health emergencies, and emphasizes the importance of guidelines in the context of public health emergencies. WHO regards providing guidance for the world in case of public health emergencies as its responsibility (5). The key of controlling public health emergencies is to guide and organize people to quickly and effectively respond to health threats (6,7). High- quality guidelines and recommendations are the premise to

effectively respond to public health emergencies. To guide guidelines developers to develop high-quality guidelines, WHO published a handbook for guideline development in 2014 (8), and then published a framework and toolkit in 2017 to guide the development process and improve the quality of guidelines developing in context of public health emergencies (9). Some scholars have also discussed the applicability and feasibility of the WHO rapid guideline development standards (10) from the WHO Handbook for Guideline Development (8) under public health emergencies, others investigated the reporting of declarations and conflicts of interest in WHO guidelines (11), and considered that the WHO rapid guideline development process still had some problems, for example related to the roles of different members of the guideline development group. It is necessary to reduce the bias and improve the method and reporting quality of rapid guidelines (10) in particular for reporting of funders and their role, and declaration processes (11) to make sure that they can better guide the response to public health emergencies.

In response to public health emergencies, such as infectious diseases, WHO and many other countries have issued a great number of guidelines to guide clinicians and public health professionals how to treat and control epidemics. Some researchers have investigated the quality of guidelines for viruses that have caused PHEICs (12), but none of them have so far studied the development and reporting quality of SARS-CoV-2 guidelines. We therefore aimed to examine the methodological and reporting quality of guidelines on five selected viruses that had caused PHEIC in the past 20 years: the severe acute respiratory syndrome coronavirus (SARS-CoV;

2003), Ebola virus (EBOV,2014), Middle East respiratory syndrome coronavirus (MERS-CoV; 2015), Zika virus (2016), and severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2; 2020) (4), hereinafter referred to as “the five viruses”. The overarching goal of this evaluation was to inform WHO and other organizations developing such tend to be low. We recommend to follow evidence-based methodology and the RIGHT statement on

reporting when developing guidelines.

Keywords: Coronavirus; Ebola virus (EBOV); emergency; practice guideline; Zika virus

Submitted Feb 23, 2020. Accepted for publication Mar 25, 2020.

doi: 10.21037/atm.2020.03.130

View this article at: http://dx.doi.org/10.21037/atm.2020.03.130

(4)

guidelines about the weaknesses for producing these types of guidelines.

Methods Search strategy

We systematically searched MEDLINE (via PubMed), China Biomedical Literature database (CBM), China National Knowledge Infrastructure (CNKI) and Wanfang Data. Websites of international organizations government health institutions developing guidelines were also searched, including National Institute for Health and Care Excellence (NICE), National Guideline Clearinghouse (NGC), Healthcare Improvement Scotland (SIGN), World Health Organization (WHO) and Guidelines International Network (GIN). All databases were searched from their inception to February 02, 2020, and the languages were restricted to Chinese and English. The search terms included “Coronavirus”, “Middle East Respiratory Syndrome Coronavirus”, “MERS”, “SARS”, “2019- nCoV”, “Wuhan-Cov”, “Ebola virus”, “Zika”, “Practice guideline”, “Recommendation*” and “Statement*”. (Full search strategies are presented in Supplementary Appendix 1). In addition, supplementary searches were conducted on Google Scholar, Medlive, the official website of the National Health Commission of the People’s Republic of China (http://www.nhc.gov.cn/), the official website of the Centers for Disease Control and Prevention of the United States (https://www.cdc.gov/), the official website of the Chinese Center for Disease Control and Prevention (http://

www.chinacdc.cn/), and preprint platforms.

Inclusion and exclusion criteria

We included practice guidelines for SARS-CoV, Ebola virus, MERS-CoV, Zika virus, SARS-CoV-2 and the diseases they caused. As practice guidelines we considered documents that contain recommendations aiming to help clinicians, public health professionals, patients, susceptible populations, or any other target audience, to make appropriate decisions in a specific clinical or public health emergency background.

The following types of documents were excluded: (I) documents containing diagnosis and treatment consensus or standards; (II) comprehensive guidelines on more general topics, containing information about the five viruses; (III) translations or summaries of guidelines; (IV) protocols of guidelines; (V) guidelines where the full text was not

available; and (VI) old versions of guidelines with updated version available.

Screening

The search results were imported into an EndNote library (version X9.1). Four researchers (Siya Zhao; Xufei Luo; Junxian Zhao; Jin Cao) screened the guidelines independently in pairs following the pre-defined exclusion criteria, and the results were cross-checked after screening.

Disagreements were discussed and solved with another researcher (Yaolong Chen).

Data extraction

Fifteen researchers (Siya Zhao, Jin Cao, Zijun Wang, Qianling Shi, Junxian Zhao, Shuya Lu, Hairong Zhang, Yangqin Xun, Ling Wang, Jianjian Wang, Qi Wang, Jingyi Zhang, Yunlan Liu, Xiaomin Nie and Xianzhuo Zhang) were divided into five groups of three. Each group extracted the data from the retrieved guidelines independently by using a standardized data collection form. Disagreements were resolved by consensus. Extracted data included title, scope and purpose, publication year, publication format, developing organization, development country or region, development method (evidence-based or not) and funding.

The publication year was dichotomized depending on if the guideline was published in the year of the outbreak of the respective virus was confirmed (2003 for SARS-CoV, 2014 for Ebola virus, 2015 for MERS-CoV, 2016 for Zika virus and 2020 for SARS-CoV-2) (4), or thereafter. The guideline development method was defined as evidence-based, if the recommendations were based on a comprehensive search of evidence.

Methodological quality appraisal of guidelines

We employed the latest version of the AGREE- Ⅱ instrument (13) to evaluate the methodological quality of each included guideline. Five groups of appraisers with three appraisers in each group independently evaluated the guidelines. For each guideline, 23 items within six domains (“scope and purpose”, “stakeholder involvement”, “rigour of development”, “clarity and presentation”, “applicability”

and “editorial independence”) were evaluated on a scale ranging from 1 (absence of item) to 7 (reported with exceptional quality). For each guideline, we calculated the quality score for each domain (the scaled domain score)

(5)

following the AGREE-II manual (13). Then we calculated and reported the mean score over all guidelines for a given virus. Figure 1 show an example calculation for the score of Domain 1 for guidelines on one virus.

Reporting quality appraisal of guidelines

We employed the RIGHT Statement (14)to evaluate the reporting quality of the included guidelines. Two researchers evaluated the guidelines independently. For each guideline, we evaluated 22 items (35 subitems in total) grouped into the following seven domains: “basic information”,

“background”, “evidence”, “recommendations”, “review and quality assurance”, “funding and declaration and management of interests” and “other information”. Each item was evaluated as either “reported” (guideline was reported with exceptional quality containing the majority information), “not reported” (relevant information on the item was lacking), or “not applicable” (the item did not need to be evaluated). In the case of conflict, another researcher (Yaolong Chen) was consulted and agreement was reached by consensus. The domain reporting rate for each guideline was then calculated by dividing the number of items reported by the total number of items in the domain, including those evaluated as “reported”, “not reported” and

“not applicable”. Finally, we calculated the mean reporting

rate of each domain for guidelines of each virus.

Statistical analysis and quality control

Before the formal extraction and evaluation, 15 researchers were trained on a random sample of two guidelines (from outside of the sample) to improve consistency. Consistency was evaluated by intra-class correlation coefficient (ICC).

We carried out subgroup analyses for the publication year since outbreak, country and region, format of publication (peer-reviewed journal, or website only), and developing organization. Statistical analyses were conducted by using SPSS version 25.0 through t value of student’s t-test, variance analysis or Kruskal-Wallis H test depending on the type of data. We analyzed the relationship between evaluation results of methodological and reporting quality for all 81 included guidelines by Spearman correlation.

Results Search results

We identified 1,752 records from the databases and 41 records from guideline websites, manual searches and other additional sources. Twenty-two duplicates were excluded, and 1,771 records were considered to be potentially relevant. After screening for titles, abstract and full-text, a Figure 1 Example of three appraisers give the following scores for Domain 1 (Scope & Purpose) for one guideline.

Item1 Item2 Item3 Total

Appraiser 1 5 6 6 17

Appraiser 2 6 6 7 19

Appraiser 3 2 4 3 9

Total obtained score 13 16 16 45

Maximum possible score = 7 (strongly agree) × 3 (items) × 3 (appraisers) = 63 Minimum possible score = 1 (strongly disagree) × 3 (items) × 3 (appraisers) = 9 Then,

(45 9)

100% 100% 66.7%

(63 9) (Obtained Score

The Scaled domain score = × = × =

- Minimum possible Score)

Maximum Score - Minimum possible Score) (Scaled domain score of an individual guideline) Average Score of Domain 1 =

Total number of guidelines

(6)

total of 81 guidelines were included (Figure 2).

Guideline characteristics

A total of 81 guidelines were included, including 21 SARS- CoV guidelines, 11 EBOV guidelines, 9 MERS-CoV guidelines, 10 Zika virus guidelines and 30 SARS-CoV-2 guidelines. The characteristics of the 81 included guidelines are shown in Table 1.

A total of 63 (77.8%) guidelines were published in the calendar year the respective virus outbreak was confirmed (2003 for SARS-CoV, 2014 for Ebola virus, 2015 for MERS-CoV, 2016 for Zika virus and 2020 for SARS- CoV-2) (4). Only fifteen (18.5%) of the guidelines followed

the principles of evidence-based guideline development, meaning that the recommendations were based on a comprehensive search of the available evidence. Only four (4.9%) guidelines reported funding sources and the sponsor.

Quality assessment of guidelines Methodological quality

The ICC value for the pilot test using AGREE-II was 0.81, indicating a high reliability on the scores.

None of the 81 guidelines scored more than 85.0% in any domain of the AGREE-II assessment. The mean scores of the six domains over all guidelines for a particular virus were 3.7–32.2% (SARS-CoV), 12.7–44.4% (MERS-CoV), Figure 2 Flow diagram of the selection process.

Records identified through database searching

(n=1,752)

Additional records identified through other sources

(n=41)

Excluded for duplicated (n=22)

Excluded for title and abstract, with reasons:

1.Not guideline (n=1,494) 2.Not relevant (n=146) 3.Not for people (n=32)

4.Not published in English or Chinese (n=7)

Excluded for full-text articles, with reasons:

1.Guidelines translation (n=7) 2.Old version (n=3)

3.Full-text was not available (n=1) Screened for title and abstract

(n=1,771)

Screened for full-text articles (n=92)

Included guidelines of the five different viruses

(n=81)

(7)

Table 1 Characteristic of 81 included guidelines

Categories SARS-CoV (n=21) EBOV (n=11) MERS-CoV (n=9) ZIKV (n=10) SARS-CoV-2 (n=30)

Publication year since outbreak

≤1 year 13 (61.9%) 5 (45.5%) 5 (55.6%) 10 (100.0%) 30 (100.0%)

>1 year 8 (38.1%) 6 (54.5%) 4 (44.4%) 0 (0.0%)

Publication country/region

International 1 (4.8%) 6 (54.5%) 5 (55.6%) 6 (60.0%) 7 (23.3%)

China 15 (71.4%) 2 (18.2%) 1 (11.1%) 0 (0.0%) 13 (43.3%)

America 3 (14.3%) 1 (9.1%) 0 (0.0%) 4 (40.0%) 9 (30.3%)

Others 2 (9.5%) 2 (18.2%) 3 (33.3%) 0 (0.0%) 1 (3.3%%)

Organization

WHO 1 (4.8%) 5 (45.4%) 5 (55.5%) 5 (50.0%) 7 (23.3%)

CDC 2 (9.5%) 2 (18.1%) 0 (0.0%) 4 (40.0%) 10 (33.3%)

Government 2 (9.5%) 1 (9.0%) 2 (22.2%) 0 (0.0%) 8 (26.6%)

Society/association 7 (33.3%) 1 (9.0%) 2 (22.2%) 1 (10.0%) 1 (3.3%)

Hospital 3 (14.3%) 0 (0.0%) 0 (0.0%) 0 (0.0%) 4 (13.3%)

University 0 (0.0%) 1 (9.0%) 0 (0.0%) 0 (0.0%) 0 (0.0%)

Not reported 6 (28.6%) 1 (9.0%) 0 (0.0%) 0 (0.0%) 0 (0.0%)

Scope & purpose

Overall management 5 (23.8%) 1 (9.0%) 3 (33.3%) 2 (20.0%) 7 (23.3%)

Prevention 2 (9.5%) 3 (27.2%) 2 (22.2%) 2 (20.0%) 7 (23.3%)

Diagnosis 0 (0.0%) 0 (0.0%) 1 (11.1%) 1 (10.0%) 1 (3.3%)

Prevention and treatment 1 (4.8%) 0 (0.0%) 1 (11.1%) 0 (0.0%) 3 (10.0%)

Screening 0 (0.0%) 0 (0.0%) 1 (11.1%) 0 (0.0%) 0 (0.0%)

Diagnosis and treatment 0 (0.0%) 0 (0.0%) 0 (0.0%) 0 (0.0%) 3 (10.0%)

Nursing 0 (0.0%) 2 (18.1%) 0 (0.0%) 0 (0.0%) 1 (3.3%)

Others 11 (52.3%) 5 (45.4%) 1 (11.1%) 5 (50.0%) 8 (26.6%)

Publication format

Journal 17 (81.0%) 4 (36.3%) 3 (33.3%) 3 (30.0%) 1 (3.3%)

Website 4 (19.0%) 7 (63.6%) 6 (66.6%) 7 (70.0%) 29 (96.7%)

Development method

Evidence-based 1 (4.8%) 4 (36.3%) 4 (44.4%) 3 (30.0%) 3 (10.0%)

Non-evidence-based 20 (95.2%) 7 (63.6%) 5 (55.5%) 7 (70.0%) 27 (90.0%)

Funding

Reported 0 (0.0%) 2 (18.1%) 1 (11.1%) 0 (0.0%) 1 (3.3%)

Not reported 21 (100.0%) 9 (81.8%) 8 (88.8%) 10 (100%) 29 (96.6%)

Publication year within an outbreak: 2003 (SARS-CoV), 2014 (Ebola virus), 2015 (MERS-CoV), 2016 (Zika virus), 2020 (SARS-CoV-2). CDC, Centers for Disease Control and Prevention of the United States; Chinese Center for Disease Control and Prevention; SARS-CoV, Severe Acute Respiratory Syndrome coronavirus; EBOV, Ebola virus; MERS-CoV, Middle East Respiratory Syndrome coronavirus; ZIKV, Zika virus;

SARS-CoV-2, severe acute respiratory syndrome coronavirus 2.

(8)

9.1–47.5% (Ebola virus), 9.1–47.5% (Zika virus) and 3.2–

34.6% (SARS-CoV-2) (Figure 3).

In the domains of “Scope and purpose” and “Clarity and presentation”, most guidelines for the five viruses had relatively high scores, but only up to about 80.0%. The mean score of Zika virus guidelines scored highest in the

“Scope and purpose” domain (50.2%), and the EBOV guidelines in the domain of “Clarity and presentation”

(47.5%).

In the domains of “Rigour of development” and

“Editorial independence”, none of the guidelines scored more than 85.0%. The lowest mean score of the “Scope and purpose” domain belong to the SARS-CoV-2 guidelines (3.2%), and the SARS-CoV guidelines scored (4.8%) the least mean score in the domain of “Clarity and presentation”. In addition, the majority of included guidelines for all five viruses [nineteen (95.0%) of the SARS-CoV, six (66.7%) of the MERS-CoV, eight (72.7%) of the Ebola virus, six (60.0%) of the Zika virus, and twenty- seven (90%) of the SARS-CoV-2 guidelines] did not report

any information related to “Editorial independence”.

Reporting quality

In the domains of “Basic information” and “Background”, most of guidelines had relatively high reporting rates, however no more than 90.0%. The mean reporting rates of the Zika virus guidelines are the most in these two domains (41.7% and 60.0%), having scores of 90.0% in the items of

“Brief description of the health problem(s)” and “Aim(s) of the guideline and specific objectives” (Figure 4).

In the domains of “Evidence”, “Review and quality assurance” and “Funding and declaration and management of interests”, most of the guidelines for the five viruses are not over 80.0% (except for a Zika virus guideline and a SARS- CoV-2 guideline scored 100% in the domain of “Review and quality assurance”). The SARS-CoV and SARS-CoV-2 guidelines had the lowest mean reporting rates (3.8% and 4.7%, respectively) in the domain of “Evidence”. None of the SARS-CoV and EBOV guidelines reported any items in the domain of “Review and quality assurance”.

Figure 3 Mean AGREE-Ⅱ scores of guidelines for the five viruses in the six domains.

60.0%

50.0%

40.0%

30.0%

20.0%

10.0%

0.0%

SARS-CoV guidelines MERS-CoV guidelines Ebola virus guidelines Zika virus guidelines SARS-CoV-2 guidelines

29.5%

44.4%

40.7%

50.2%

34.0%

13.4%

18.9%

25.4%

32.4%

14.0%

4.8%

12.7%

12.5%

21.1%

8.3%

32.2%

40.5%

47.5%

47.0%

34.6%

19.2%

28.2%

26.5%

22.6%

18.9%

3.7%

15.4%

9.1%

20.8%

3.2%

Scope and purpose Stakeholder Rigour of development Clarity of presentation Applicability Editorial independence involvement

(9)

The mean reporting rate of the SARS-CoV-2 guidelines did not exceed 50% for any domain. In the domain of

“Review and quality assurance”, only 6.7% of the items were reported in the SARS-CoV-2 guidelines, and for the specific items “Executive summary”, “Guideline development groups” and “Health care questions”, the reporting rates were below 5%.

Subgroup analysis

The results of the subgroup analysis are shown in Tables 2,3.

In the subgroup analysis of publication year, we compared the evaluation results for methodological and reporting quality between guidelines published within one year (≤1 year) and more than 1 year (>1 year) from the outbreak.

For SARS-CoV, the guidelines published at least one year after the outbreak had a higher quality of methodology (P=0.005) and reporting (P=0.026). In the subgroup analysis of publication country/region, results of the variance analysis showed that both the methodological (P=0.001) and reporting (P=0.005) quality of the SARS-CoV guidelines differed across countries and/regions. Guidelines on SARS-

CoV-2 in journals had a higher quality of methodology (P=0.003) and reporting (P=0.001) than guidelines published on websites, and the quality also differed across the developer organizations (P=0.006).

Correlation analysis of evaluation results

The results showed that there was a positive correlation between AGREE-II and RIGHT scores (R=0.716, P<0.001) (Figure 5). The guidelines with high methodological quality therefore tend to also have high reporting quality.

Discussion

Over the past two decades, serious outbreaks caused by the SARS-CoV, MERS-CoV, EBOV and Zika viruses have occurred around the world and triggered PHEIC (4), and the SARS-CoV-2 is currently threatening to become the next global public health emergency (1). We found that the methodological and reporting quality of the guidelines on these five viruses tended to be low. The mean scores of each domain of AGREE-Ⅱ for the guidelines Figure 4 Mean RIGHT scores of guidelines for the five viruses in seven domains.

60.0%

50.0%

40.0%

30.0%

20.0%

10.0%

0.0%

SARS-CoV guidelines MERS-CoV guidelines Ebola virus guidelines Zika virus guidelines SARS-CoV-2 guidelines

38.9%

40.7%

40.9%

41.7%

37.8%

32.7%

48.6%

43.2%

60.0%

35.4%

3. 8%

15.6%

12. 7%

18.0%

4.7%

15.17%

31.8%

33.8%

37.1%

23.8%

0.0%

22.2%

0.0%

25.0%

6.7%

1.2%

11.1%

4.6%

20.0%

5.0%

6.4%

48.2%

21.2%

40.0%

41.1%

Basic information Background Evidence Recommendations Review and Funding and Other information quality assurance declaration

management of interests

(10)

of each virus were all below 60%. The mean reporting quality scored also below 60% for all viruses and RIGHT domains. In AGREE-Ⅱ, the domains of “Rigour of development” and “Editorial independence” scored very

low. In RIGHT, the domains of “Evidence”, “Review and quality assurance” and “Funding and interests” also had poor scores. Especially none of the guidelines for SARS- CoV and EBOV reported any information in the domain Table 2 Subgroup analysis of AGREE-Ⅱ scores for the guidelines of the five viruses (mean ± SD)

Categories SARS (n=21) MERS (n=9) EBOV (n=11) ZIKV (n=10) SARS-CoV-2 (n=30)

Publication year since outbreak

≤1 year 0.14±0.05 0.29±0.19 0.18±0.11

>1 year 0.22±0.07 0.24±0.05 0.34±0.15

t/F/H −3.195 0.459 −1.991

P 0.005 0.660 0.078

Publication country/region

International 0.19±0.00 0.23±0.08 0.32±0.17 0.38±0.18 0.22±0.08

China 0.14±0.05 0.14±0.00 0.16±0.10 0.19±0.12

America 0.22±0.03 0.14±0.00 0.24±0.03 0.16±0.05

Others 0.31±0.06 0.38±0.18 0.27±0.15 0.13±0.00

t/F/H 8.118 1.942 0.627 1.742 0.724

P 0.001 0.224 0.620 0.137 0.547

Publication format

Journal 0.17±0.08 0.32±0.23 0.35±0.20 0.24±0.01 0.44±0.00

Website 0.23±0.10 0.24±0.08 0.23±0.11 0.36±0.18 0.18±0.08

t/F/H −0.217 0.572 1.305 −1.698 3.307

P 0.830 0.619 0.224 0.140 0.003

Organization

WHO 0.19±0.00 0.23±0.08 0.26±0.11

CDC 0.16±0.07 0.12±0.02

Government 0.22±0.04 0.22±0.12 0.25±0.00

Society/association 0.23±0.07 0.41±0.24 0.17±0.00

Hospital 0.12±0.05

University 0.59±0.00

Not reported 0.12±0.03 0.4±0.00

t/F/H 3.119 1.589 3.402 2.455* 14.367*

P 0.040 0.279 0.103 0.29 0.006

All guidelines of Zika virus and SARS-CoV-2 were published in the same year, the data cannot carry out subgroup analysis for the publication year since outbreak. *, Kruskal-Wallis Test was used due to uneven variance; , publication year within an outbreak: 2003 (SARS-CoV), 2014 (Ebola virus), 2015 (MERS-CoV), 2016 (Zika virus), 2020 (SARS-CoV-2). SARS-CoV, Severe Acute Respiratory Syndrome coronavirus; EBOV, Ebola virus; MERS-CoV, Middle East Respiratory Syndrome coronavirus; ZIKV, Zika virus; SARS-CoV-2, severe acute respiratory syndrome coronavirus 2; CDC, Centers for Disease Control and Prevention of the United States; Chinese Center for Disease Control and Prevention; t, t value of Student’s t-test; F, F value of analysis of variance; H, H value of Kruskal-Wallis H test.

(11)

Table 3 Subgroup analysis of RIGHT scores for the guidelines of the five viruses (mean ± SD)

Categories SARS (n=21) MERS (n=9) EBOV (n=11) ZIKV (n=10) SARS-CoV-2 (n=30)

Publication year since outbreak

≤1 year 0.15±0.07 0.37±0.22 0.18±0.13

>1 year 0.24±0.09 0.28±0.07 0.36±0.21

t/F/H −2.414 0.870 −1.633

P 0.026 0.424 0.137

Publication country/region

International 0.26±0.00 0.34±0.14 0.30±0.17 0.45±0.20 0.25±0.11

China 0.15±0.07 0.17±0.00 0.10±0.06 0.25±0.18

America 0.26±0.06 0.29±0.00 0.28±0.03 0.22±0.05

Others 0.31±0.00 0.37±0.25 0.39±0.34 0.11±0.00

t/F/H 6.247 0.444 0.765 2.025 0.361

P 0.005 0.661 0.549 0.096 0.782

Publication format

Journal 0.17±0.08 0.34±0.27 0.41±0.23 0.26±0.03 0.63±0.00

Website 0.23±0.10 0.32±0.13 0.20±0.13 0.43±0.19 0.22±0.11

t/F/H −1.161 0.147 1.915 −2.428 3.557

P 0.260 0.888 0.088 0.048 0.001

Organization

WHO 0.24±0.14 0.25±0.13

CDC 0.17±0.16

Government 0.21±0.06 0.14±0.00

Society/association 0.43±0.32 0.14±0.00

Hospital

University 0.57±0.00

Not reported 0.63±0.00

t/F/H 0.287* 0.732 3.068 6.632* 10.248*

P 0.962 0.519 0.122 0.036 0.036

Publication year within an outbreak: 2003 (SARS-CoV), 2014 (Ebola virus), 2015 (MERS-CoV), 2016 (Zika virus), 2020 (SARS-CoV-2). All guidelines of Zika virus and SARS-CoV-2 were published in the same year, the data cannot carry out subgroup analysis for the publication year since outbreak. *, Kruskal-Wallis Test was used due to uneven variance. , publication year within an outbreak: 2003 (SARS-CoV), 2014 (Ebola virus), 2015 (MERS-CoV), 2016 (Zika virus), 2020 (SARS-CoV-2). t, t value of Student’s t-test; F, F value of analysis of variance;

H, H value of Kruskal-Wallis H test; SARS-CoV, Severe Acute Respiratory Syndrome coronavirus; EBOV, Ebola virus; MERS-CoV, Middle East Respiratory Syndrome coronavirus; ZIKV, Zika virus; SARS-CoV-2, severe acute respiratory syndrome coronavirus 2; CDC, Centers for Disease Control and Prevention of the United States; Chinese Center for Disease Control and Prevention.

(12)

of “Review and quality assurance” of RIGHT statement.

There may be two possible reasons for the above results.

First, since the RIGHT Statement was not published until 2017 (15), and all the viruses except for the SARS-CoV-2 emerged before 2017 (4), the guidelines therefore had no reporting standards to refer to. This can thus lead to lower reporting quality. Second, due to the nature of public health emergencies, guidance documents or recommendations need to be developed within a short period of time. The developers may not have time to organize an appropriate guideline development group include methodologists, or conduct systematic reviews of evidence and literature.

In 2017, WHO issued guidance for interim guidelines development of public health emergencies (9), and in the same year, Chen and colleagues developed the RIGHT statement to assist developers in reporting guidelines (15).

B u t t h e r e s u l t s o f o u r s t u d y s h o w e d t h a t b o t h methodological and reporting quality in the guidelines of SARS-CoV-2 remain poor, especially in the domains of

“Editorial independence” (AGREE-Ⅱ) and “Evidence”

(RIGHT). In addition, we also found that the quality in the guidelines for SARS-CoV were similar to the guidelines of SARS-CoV-2. In a subgroup analysis, we also found that the reporting quality differed across countries for SARS-CoV guidelines, and across developer organizations for SARS- CoV-2 guidelines. Zika virus guidelines, which tended to have higher scores on both methodological and reporting quality, were mostly developed or published by the WHO and the United States CDC (Center for Disease Control and Prevention). The majority of SARS-CoV and SARS- CoV-2 guidelines were developed or published by Chinese institutions, which is understandable as China was most

affected by these two outbreaks (4). These results suggest that Chinese guideline developers should strengthen the rigor, applicability and editorial independence in the development of such guidelines, to provide better guidelines to fight the ongoing SARS-CoV-2 epidemic and future public health emergencies in China.

China is experiencing increased health care use and expenditures, without sufficient controls to ensure quality and value, and cost-conscious and patient-centered guidelines based on the best available evidence could help establishing these quality and practice measures (16). So, it is important for China to improve quality of guideline.

In view of our results, we put forward the following suggestions for Chinese guideline developers. First, guideline development process should be reported more transparently and according to the RIGHT Statement.

Second, evidence-based methods should be applied in guideline development and evidence translation (16), and methodologists should be involved in the development process to support methodological and reporting quality.

Third, the process of peer review and quality control should be strengthened in guideline development. Finally, the important role of systematic reviews of existing evidence in the process of formulating the guidance should be emphasized.

There were also some limitations when using the AGREE-Ⅱ and RIGHT to assess the guidelines for these five viruses. The majority of the guidelines of public health emergencies are issued for the general public, meaning that the users and target populations are often the same. This problem is however not addressed in either AGREE-Ⅱ or RIGHT (13,14). Since there are still many Figure 5 Scatter plot for the correlation between AGREE-Ⅱ and RIGHT scores.

The Score of RIGHT (%)

The Score of AGREE II (%) 80

70 60 50 40 20 10 0

0 10 20 30 40 50 60 70 80

(13)

unknown factors related to SARS-CoV-2 and many studies about it are ongoing, the guidelines are being updated rapidly. The results and conclusion of our study regarding the SARS-CoV-2 guidelines therefore only reflect the early stages of the epidemic, and the quality of guidelines may improve later.

Conclusions

The methodological and reporting quality of guidelines on the five viruses (SARS-CoV, Ebola virus, MERS-CoV, Zika virus, and SARS-CoV-2) that have caused serious public health emergencies tends to be poor. Even in case of a rapidly developing infectious disease epidemic and lack of scientific data, the principles of evidence-based medicine and transparent reporting should be followed in guideline development as much as possible. When developing rapid advice guidelines and emergency guidelines in the case of a public health emergency, it is essential to follow WHO Handbook for Guideline Development (8) and the RIGHT statement to ensure the high quality and transparent, comprehensive and clear reporting.

Acknowledgments

Funding: This work was supported by 2020 Key R&D project of Gansu Province.

Footnote

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.

org/10.21037/atm.2020.03.130). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non- commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license).

See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Novel Coronavirus(2019-nCoV) Situation Report – 33[Internet]. World Health Organization; c2020 [cited 2020 Feb 22]. Available online: https://www.who.int/docs/

default-source/coronaviruse/situation-reports/20200222- sitrep-33-covid-19.pdf?sfvrsn=c9585c8f_2

2. World Health Organization. International health

regulations 2005. 2nd edition. Geneva: The Network, 2005.

3. Talisuna AO, Okiro EA, Yahaya AA, et al. Spatial and temporal distribution of infectious disease epidemics, disasters and other potential public health emergencies in the World Health Organisation Africa region, 2016-2018.

Globalization Health 2020;16:9.

4. Disease outbreak news [Internet]. World Health

Organization; c2020 [cited 2020 Feb 7]. Available online:

https://www.who.int/csr/don/archive/year/en/

5. World Health Organization. Emergency response framework. 2nd edition. Geneva: The Network, 2017.

6. PLoS Medicine Editors. How is WHO responding to global public health threats? PLoS Med 2007;4:e197.

7. Rose DA, Murthy S, Brooks J, et al. The Evolution of Public Health Emergency Management as a Field of Practice. Am J Public Health 2017;107:S126-33.

8. World Health Organization. WHO handbook for guideline development. 2nd edition. Geneva: The Network, 2014.

9. World Health Organization. Health emergency interim guidelines: a WHO guideline development framework and toolkit. 1st edition. Geneva: The Network, 2017.

10. Garritty CM, Norris SL, Moher D. Developing WHO rapid advice guidelines in the setting of a public health emergency. J Clin Epidemiol 2017;82:47-60.

11. Wang X, Chen Y, Yao L, et al. Reporting of declarations and conflicts of interest in WHO guidelines can be further improved. J Clin Epidemiol 2018;98:1-8.

12. Norris SL, Sawin VI, Ferri M, et al. An evaluation of emergency guidelines issued by the World Health Organization in response to four infectious disease outbreaks. PLoS One 2018;13:e0198125.

13. APPRAISAL OF GUIDELINES FOR RESEARCH &

EVALUATION Ⅱ[Internet]. The AGREE Collaboration;

c2020 [cited 2020 Feb 5]. Available online: https://www.

agreetrust.org/wp-content/uploads/2017/12/AGREE- II-Users-Manual-and-33-item-Instrument-2009- Update-2017.pdf

(14)

14. A Reporting Tool for Practice Guidelines in Health Care:

The RIGHT Statement. The RIGHT Group; c2020 [cited 2020 Feb 5]. Available online: http://www.right-statement.

org/right-statement/checklist

15. Chen Y, Yang K, Marusic A, et al. A Reporting Tool

for Practice Guidelines in Health Care: The RIGHT Statement. Ann Intern Med 2017;166:128-32.

16. Yang K, Chen Y, Li Y, et al. Editorial: can China master the guideline challenge? Health Res Policy Syst 2013;11:1.

Cite this article as: Zhao S, Cao J, Shi Q, Wang Z, Estill J, Lu S, Luo X, Zhao J, Zhang H, Wang J, Wang Q, Xun Y, Zhang J, Lv M, Liu Y, Nie X, Wang L, Zhang X, Li W, Liu E, Wang X, Chen Y; on behalf of COVID-19 evidence and recommendations working group. A quality evaluation of guidelines on five different viruses causing public health emergencies of international concern. Ann Transl Med 2020;8(7):500. doi: 10.21037/atm.2020.03.130

(15)

Supplementary Appendix 1 Search Strategy PubMed

#1 Coronavirus [Title/Abstract]

#2 "Middle East Respiratory Syndrome Coronavirus" [Title/Abstract]

#3 MERS [Title/Abstract]

#4 SARS [Title/Abstract]

#5 CoV [Title/Abstract]

#6 HCoV [Title/Abstract]

#7 "2019-nCoV" [Title/Abstract]

#8 "Wuhan-Cov" [Title/Abstract]

#9 Wuhan Coronaviru* [Title/Abstract]

#10 "Ebola virus" [Title/Abstract]

#11 Zika [Title/Abstract]

#12 OR#1-#11

#13 "Coronavirus" [Mesh]

#14 "Middle East Respiratory Syndrome Coronavirus"[Mesh]

#15 "Coronavirus Infections"[Mesh]

#16 "SARS Virus"[Mesh]

#17 "Ebola virus" [Mesh]

#18 "Zika Virus"[Mesh]

#19 "Zika Virus Infection" [Mesh]

#20 OR/#13-#19

#21 #12 OR #21

#22 "Practice guideline" [Publication Type]

#23 Guideline* [Title/Abstract]

#24 Guidance* [Title/Abstract]

#25 Recommendation* [Title/Abstract]

#26 Statement*[Title/Abstract]

#27 OR #23-#27

#28 #22 AND #28

Références

Documents relatifs

Chan School of Public Health, Emergency Preparedness, Research, Evaluation and Practice Program, Boston, MA, United States of America (USA) – Giorgia Argentini (project consultant

Human exposure to benzene has been associated with a range of acute and long-term adverse health effects and diseases, including cancer and haematological effects.. Exposure can occur

Osteomalacia (softening of the bones) and osteoporosis may occur in those exposed through living or working in cadmium-contaminated areas; for example, in an area of Japan where

The day-to-day oversight and management of major outbreaks and health emergencies, including WHO Grade 3 events, Public Health Emergencies of International Concern and

Identify chemicals that have a significant impact on public health and outline potential sources of exposure to these substances; B Assess the public health impacts and

• Continue human resource development across all core public health functions and fill critical gaps in Pacific regional resources in risk assessment, data management

• Reduce emissions of dioxins and dioxin-like substances as required under the Stockholm Convention on Persistent Organic Pollutants, 3 following guidance on best

1 WHO, Ebola virus disease epidemic in West Africa: Update and lessons learnt, Brazzaville, Congo, World Health Organization, Regional Office for Africa, 2014 Document AFR/RC64/9..