Evidence of survival benefit was often ambiguous in randomized trials of cancer treatments

(1)

Article

Reference

Evidence of survival benefit was often ambiguous in randomized trials of cancer treatments

PERNEGER, Thomas, et al.

Abstract

The objective of the study is to estimate the proportion of statistically significant survival improvements reported in randomized trials of cancer treatments that are also compatible with a clinically negligible benefit.

PERNEGER, Thomas, et al . Evidence of survival benefit was often ambiguous in randomized trials of cancer treatments. Journal of Clinical Epidemiology , 2020, vol. 127, p. 1-8

PMID : 32622900

DOI : 10.1016/j.jclinepi.2020.06.026

Available at:

http://archive-ouverte.unige.ch/unige:151658

Disclaimer: layout of this document may differ from the published version.

1 / 1

(2)

ORIGINAL ARTICLE

Evidence of survival benefit was often ambiguous in randomized trials of cancer treatments

Thomas V. Perneger*, Pauline Brindel, Christophe Combescure, Ang ele Gayet-Ageron

Division of Clinical Epidemiology, Geneva University Hospitals, Faculty of Medicine, University of Geneva, 1211 Geneva, Switzerland Accepted 23 June 2020; Published online 3 July 2020

Abstract

Objectives: The objective of the study is to estimate the proportion of statistically significant survival improvements reported in randomized trials of cancer treatments that are also compatible with a clinically negligible benefit.

Study Design and Setting: This is a cross-sectional study of reports of randomized clinical trials of cancer treatments that reported a statistically significant increase in overall survival, published in leading journals between 2009 and 2019. The main outcome variable was the hazard ratio (HR) for overall survival and its upper 95% confidence limit. An HR of 0.95 implies an absolute survival gain1.9%, and an HR of 0.90 implies an absolute survival gain3.8%; we reasoned that such survival gains can be considered clinically negligible, given the potential toxicity of oncologic treatments.

Results: Among 234 trial results, the mean point estimate of the HR was 0.664, and all HRs were below 0.90. The mean upper 95%

confidence limit for the HR was 0.897, but 37.6% of the values were0.95, and 59.0% were0.90. These proportions were lower when overall survival was the primary outcome of the trial (29.9%0.95 and 51.3%0.90).

Conclusions: Considering only point estimates of HRs, all trials reported clinically meaningful improvements in overall survival. How- ever, the upper confidence limits of a large proportion of HRs were also compatible with clinically negligible survival gains. Acknowl- edging the uncertainty regarding treatment benefits presents a challenge for the reporting of trial results and for clinical decision- making. Ó2020 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license (http://

creativecommons.org/licenses/by-nc-nd/4.0/).

Keywords:Estimation; Uncertainty; Confidence interval; Treatment effectiveness; Randomized trial

1. Introduction

A statistically significant clinical benefit shown in a randomized clinical trial (RCT) is an important consideration for the approval of new drugs. Although a simple rule based on statistical significance facilitates regulatory decisions, dichotomizing trial results as ‘‘statistically significant’’ or

‘‘nonsignificant’’ is increasingly seen as misguided [1e3]

because the evidence provided by a clinical trial is not inherently black or white. Any observed trial result is compatible with a range of possible ‘‘true’’ effects, of which some may be clinically important and others not.

This information is conveyed by the confidence interval

[4]. For example, if a trial yields a hazard ratio (HR) of 0.67, with a 95% confidence interval (95% CI) of 0.50 to 0.90, then the true HR may be as low as 0.50, a consider- able reduction in the mortality rate, but also as high as 0.90, which is less compelling. Even this statement does not capture the whole uncertainty, as the 95% CI procedure will include the true value of the HR in only 19 trials out of 20 on average (this is what ‘‘95%’’ means); in any given instance, the 95% CI may be off. The uncertainty about the magnitude of the treatment effect is often uncomfort- able for both doctors and patients [5e7], yet it is real.

Possibly, a key benefit of presenting the trial result as a

‘‘statistically significant’’ HR of 0.67 is that this allows the illusion that the magnitude of the effect is exactly known.

When the confidence interval of the HR includes only clinically important effects, such as a range from 0.40 to 0.60, the uncertainty has no impact on clinical or regulatory decisions. In other cases, however, a ‘‘statistically significant’’ result may be compatible with both clinically

Disclosure of interests: None of the authors has any competing interests related to this study.

* Corresponding author. Division of Clinical Epidemiology, Geneva University Hospitals, 6 rue Gabrielle-Perret-Gentil, 1211 Geneva, Switzerland. Tel.:þ41-22-372-9037; fax:þ41-22-373-9035.

E-mail address:thomas.perneger@hcuge.ch(T.V. Perneger).

https://doi.org/10.1016/j.jclinepi.2020.06.026

0895-4356/Ó2020 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/

licenses/by-nc-nd/4.0/).

e

(3)

What is new?

Key findings

Among 234 statistically significant hazard ratios representing survival benefits of new cancer treatments obtained in randomized trials, 37.6% had an upper confidence limit above 0.95 and 59.0%

had an upper confidence limit above 0.90.

The upper confidence limit of the hazard ratio was not commented upon in any of the articles.

What this study adds to what was known?

A ‘‘statistically significant’’ survival benefit is frequently compatible with a clinically negligible benefit, yet this remains unrecognized in primary publications.

What is the implication and what should change?

Regulatory and clinical decisions may be often based on ambiguous evidence of survival benefit.

Reports of clinical trials should include a thorough and open discussion of the uncertainty regarding the benefits of treatments, instead of relying on statistical significance.

Clinical trials should be designed so as to minimize the occurrence of ambiguous trial results.

important and clinically negligible benefits. A small improvement in survival may be considered clinically negligible when the treatment is toxic, costly, or both, which is the case of many cancer treatments. Although no consensus exists about what represents a clinically negligible survival gain, the European Society for Medical Oncology Magnitude of Clinical Benefit Scale (ESMO- MCBS) considers survival improvements !3% as representing a low level of clinical benefit [8]. An absolute difference in survival does not translate directly to an HR, but as we show in section2.3of Methods, HR values of 0.90 or 0.95 reflect survival gains that cannot exceed 3.8% or 1.9%.

In this study, we consider HR values of 0.90 or 0.95 as possible thresholds of negligible clinical benefit. How often ambiguous results occur among ‘‘statistically significant’’

trial results is not currently known.

In this study, we sought to estimate, among randomized trials of cancer treatments that demonstrated a statistically significant survival benefit, the proportion which was also compatible with a clinically negligible benefit, reflected by an upper 95% confidence limit on the HR exceeding 0.90 or 0.95. We limited this study to oncology trials that reported improved overall survival, to simplify the interpretation of the clinical importance of the observed effect.

2. Methods

2.1. Study population

In this observational study, we included RCTs in the field of oncology that reported a statistically significant improvement in overall survival for a new treatment (drug, drug combination, or radiotherapy), expressed as an HR, with a 95% CI. We identified RCTs by searching PubMed with the following syntax: ‘‘randomized trial’’ AND (cancer OR neoplasia) AND ‘‘overall survival’’ AND ‘‘journal’’[so]. We searched the New England Journal of Medicine, Lancet, Lancet Oncology, Journal of Clinical Oncology, JAMA, and JAMA Oncology because these journals represented the bulk of sources cited in the decisions of two regulatory agenciesdthe U.S. Food and Drug Admin- istration (FDA) and the European Medicines Agency (EMA) [9,10]. We screened all original articles published between January 1, 2009 and June 30, 2019. We retained one HR from each randomized comparison, or two when there were two prespecified ‘‘coprimary’’ analyses. In studies which reported survival results in several articles, over time, we retained the first article that claimed a statistically significant survival benefit. For separate research questions in the same trial (as in three-arm trials or factorial trials), we considered each key result as though it was a separate study. We did not include studies which claimed statistical significance based on a one-sided test with P O 0.025 (this would correspond to a two-sided PO0.05), significant differences that favored the comparison arm, significant differences that were observed in sub- groups only, prevention trials, noninferiority trials, combined analyses of several trials, and meta-analyses.

2.2. Study variables

For each trial, we retrieved the HR of the experimental treatment’s effect on overall mortality and the corresponding 95% CI, at the level of precision provided in the article, with the exception of upper HR confidence limits reported as 1.0 and aP-value strictly!0.05, which were entered as 0.999. We noted the year of publication, journal, sample size, comparator for the experimental treatment (open active control, placebo with or without other treatment, supportive care, or surveillance), whether it was a phase II or phase III trial, and whether overall survival was the primary efficacy outcome or a secondary outcome. We also searched the discussion section of each article for any acknowledgment of uncertainty about the magnitude of the treatment’s effect on overall mortality, specifically in reference to the confidence bounds on the HR.

The data were abstracted by one author (T.P.). Another author (A.G.A.) reabstracted 50 randomly selected articles, and because the intraclass correlation coefficients were between 0.98 and 1.00, the double entry was not performed on the remaining articles.

(4)

2.3. Determination of clinically negligible benefit

Absolute survival gains are easier to interpret than relative measures of risk [11,12] and are required for decision analysis, but this information is not always available. When mortality rates (or hazards) of the groups being compared are proportional (e.g., one is twice as high as the other at all times), the HR equals log(S₁)/log(S₀), where S₁is survival in the experimental group and S₀survival in the control group [13]. The absolute survival gain (DS) is the difference between survival proportions in the two groups, thus HR 5 ln (S₀þDS)/ln (S₀), and DS 5 S₀^HReS₀. We plottedDS as a function of S₀and HR (Fig. 1). The absolute survival gain cannot exceed 1.9% when the HR is 0.95, or 3.8% when the HR is 0.90. Herein we consider HR values of 0.95 and 0.90 as reasonable thresholds of clinically negligible benefit.

2.4. Statistical analysis

We aimed for a sample size of 200 HRs or more, to obtain 95% CIs on proportions (of clinically negligible

treatment effects) that will not exceed a width of60.07, using formula N5p (1-p)/se², where p is the proportion to be estimated (set between 0.3 and 0.5) and se its standard error (set to 0.035).

For HRs and their confidence limits, we reported the mean, standard deviation, quartiles, and range. We also described the distributions of the point estimates and upper confidence limits. We compared the proportions of upper 95% confidence limits above the two thresholds of clinically negligible benefit (0.90 and 0.95) across trial characteristics (year of publication, journal, sample size category, comparison treatment, phase of trial, primary vs. secondary outcome) using chi-square tests. We obtained a scatter plot of the upper HR confidence limit vs. the point estimate of the HR.

To express survival benefits on an absolute scale, we transformed each HR and confidence limit into an absolute survival gain, calculated for the situation where the control group reaches median survival (S₀ 5 50%; we use the same value of S₀ to allow comparisons, inde- pendently of duration of follow-up). We solved

Fig. 1. Absolute survival gain (in percent) in the experimental trial arm as a function of survival achieved in the control arm and as a function of the hazard ratio reflecting the efficacy of the experimental treatment under the assumption of proportional hazards.

(5)

DS 5 S₀^HReS₀ for S₀ 5 0.5. Again, we show the distributions of absolute survival gains and their confidence limits. Absolute survival gains (on a percent scale) were categorized as 0e0.99%, 1e1.99%, 2e3.99%, 4e7.99%, 8e15.99%, 16e31.99%, and 32%

and greater. Analyses were conducted using SPSS, version 24.

3. Results

The initial search retrieved 1,489 articles, among which 226 articles reported 234 statistically significant HRs demonstrating a survival benefit of the experimental cancer treatment (eight HRs were coprimary outcomes, or separate comparisons from three-arm trials or factorial trials).

Ninety-five percent confidence intervals were retrieved directly for 231 HRs, and computed for three others. One study [14] reported a 99.79% CI because of a partitioning strategy for type 1 error; we derived the 95% CI on the

log (HR) scale by changing the z statistic. Another study [15] reported only HRs and P-values for the two active treatments vs. the standard treatment in the main article, but CIs were provided for stratified HRs in an online appendix; we derived the 95% CIs of the overall HRs using fixed- effects meta-analysis.

The rate of publication of the trials was steady over time (Table 1). Almost a third of the HRs were published in the New England Journal of Medicine. Most HRs came from phase III trials. In half of the trials, overall survival was the primary outcome. The 226 trials tested treatments in patients who suffered from cancers of the blood (38 HRs, 16.8%), lung (31 HRs, 13.7%), breast (28 HRs, 12.3%), prostate (22 HRs, 9.7%), colon/rectum (19 HRs, 8.4%), stomach/esophagus (17 HRs, 7.5%), skin (16 HRs, 7.1%), pancreas (10 HRs, 4.4%), liver (10 HRs, 4.4%), uterus/

ovary (10 HRs, 4.4%), head/neck (7 HRs, 3.1%), urinary tract (7, 3.1%), brain (6 HRs, 2.7%), or from sarcoma (5, 2.2%). The mean sample size was 646 patients per trial (range 65 to 3,222).

Table 1.Characteristics of 234 hazard ratios (HRs) from randomized trials showing a statistically significant survival benefit of the experimental treatment, and proportions of upper 95% confidence limits of the HRs exceeding thresholds of 0.90 and 0.95

Characteristic of hazard ratio Frequency (%)

Upper HR 95% confidence limit

‡0.90 ‡0.95

Overall 234 (100) 59.0% 37.6%

Publication year: (P50.82) (P50.95)

2009e2011 55 (23.5) 60.0% 40.0%

2012e2014 62 (26.5) 62.9% 37.1%

2015e2017 71 (30.3) 54.9% 35.2%

2018e2019 46 (19.7) 58.7% 39.1%

Journal: (P50.013) (P50.019)

New England Journal of Medicine 76 (32.5) 43.4% 22.4%

Lancet 41 (17.5) 61.0% 43.9%

Lancet Oncology 50 (21.4) 66.0% 38.0%

Journal of Clinical Oncology 57 (24.4) 73.7% 50.9%

JAMA 5 (2.1) 40.0% 40.0%

JAMA Oncology 5 (2.1) 60.0% 60.0%

Sample size (P50.40) (P50.068)

65e399 78 (33.3) 55.1% 28.2%

400e799 90 (38.5) 64.4% 45.6%

800e3105 66 (28.2) 56.1% 37.9%

Comparison treatment (P50.88) (P50.98)

Open active treatment 179 (76.5) 58.1% 38.0%

Placebo (with other treatment or not) 44 (18.8) 61.4% 36.4%

Supportive care or observation 11 (4.7) 63.6% 36.4%

Drug development phase: (P50.27) (P50.76)

Phase II 20 (8.5) 45.0% 38.3%

Phase III 209 (89.3) 60.8% 30.0%

Unclassified 5 (2.1) 40.0% 40.0%

Overall survival outcome: (P50.017) (P50.015)

Primary (or coprimary) 117 (50.0) 51.3% 29.9%

Secondary 117 (50.0) 66.7% 45.3%

(6)

3.1. Hazard ratios

Means of HR point estimates and of the lower and upper confidence limits were 0.664, 0.500, and 0.897, respectively (Table 2). About two-thirds of the point estimates of the HRs were between 0.60 and 0.80, and none exceeded the thresholds of negligible clinical benefit of 0.90 or 0.95.

Most lower confidence limits were between 0.40 and 0.70, and none exceeded 0.80. As for upper confidence limits, 37.6% were at 0.95 or higher (95% CI: 31.6% to 44.0%), and 59.0% were at 0.90 or higher (95% CI: 52.6% to 65.1%). The proportions of upper confidence limits above the thresholds of negligible clinical benefit did not vary over years of publication, sample size, or comparator treatment, but were lower when the overall survival was a primary outcome and in phase III trials (Table 1). Studies published in the New England Journal of Medicine and JA- MA also had fewer upper HR limits above thresholds of negligible clinical benefit.

The scatter plot of upper HR confidence limits vs. HR point estimates showed the association between the two variables but also the size of the differences and the lower precision of secondary analyses of overall survival (Fig. 2).

3.2. Absolute survival gains

The means of the estimated absolute survival gains (at median survival in the control group) and of the lower and upper 95% confidence limits were 13.4%, 3.8%, and 21.0%, respectively (Table 3). All the point estimates corre- sponded to survival gains of 4% or more, and the majority (85.9%) were at 8% or more. Upper confidence limits were all at 8% or more. However, 51 (21.8%) of the lower confidence limits on the survival gains were less than 1%, and another 38 (16.2%) were between 1% and 2%. Thus for 38.0% of the studies, when the control group reached

median survival (50%), the experimental group might have a survival between 50% and 52%, if the lower confidence limits of the absolute survival gains were considered.

3.3. Acknowledgment of uncertainty

Of the 226 studies, 28 mentioned in the discussion one or more causes of uncertainty about the estimated treatment effect. Common recognized causes of uncertainty were patient crossover to the more effective treatment arm, survival data not being ‘‘mature’’, and limited sample size. Three discussions mentioned uncertainty related to estimation (emphasis added): ‘‘Given the small size of the trial and the large CIs around HRs observed, a larger trial would be needed to give more accurate estimates of the true benefit’’ [16], ‘‘The phase II nature of the trial, however, requires caution in interpretation of the results because the probability of unstable estimates of treatment effect and false-positive results increases with small sample size’’ [17], and ‘‘We note also that for overall survival, the number of events is small, and confidence intervals are wide’’ [18]. None of the discussion sections addressed the implications of the upper confidence bound of the HR.

The 28 studies that mentioned uncertainty were similar to the whole sample in terms of the upper 95% confidence limits on the HR: 17 (60.7%) of the upper limits exceeded 0.90, and 9 (32.1%) exceeded 0.95.

4. Discussion

This observational study confirmed that all point estimates of HRs capturing statistically significant survival benefits of new oncologic treatments represented clinically meaningful effects: all HRs were below 0.9, and a majority

Table 2.Distributions of observed hazard ratios, and of the lower and upper 95% confidence limits, for 234 treatment comparisons that showed a statistically significant survival benefit of the experimental treatment

Distribution statistic Hazard ratio: point estimate Lower 95% confidence limit Upper 95% confidence limit

Mean (SD) 0.664 (0.124) 0.500 (0.143) 0.897 (0.089)

Quartiles 0.62, 0.68, 0.76 0.44, 0.52, 0.60 0.85, 0.92, 0.97

Range 0.16 to 0.88 0.05 to 0.77 0.48 to 0.99

Categories, N (%)

Up to 0.199 1 (0.4) 5 (2.1) 0

0.20 to 0.299 2 (0.9) 21 (9.0) 0

0.30 to 0.399 3 (1.3) 22 (9.4) 0

0.40 to 0.499 20 (8.5) 49 (20.9) 1 (0.4)

0.50 to 0.599 26 (11.1) 73 (31.2) 2 (0.9)

0.60 to 0.699 81 (34.6) 50 (21.4) 4 (1.7)

0.70 to 0.799 75 (32.1) 14 (6.0) 24 (10.3)

0.80 to 0.899 26 (11.1) 0 65 (27.8)

0.90 to 0.949 0 0 50 (21.4)

0.95 to 1.00 0 0 88 (37.6)

(7)

ranged from 0.6 to 0.8. The mean value was 0.66, which cor- responds to a reduction in the mortality rate of one-third, or to an extension of mean survival by 50% (under the assumption of proportional hazards). This reassuring picture was contra- dicted by the analysis of the upper limits of 95% CIs of the HR, instead of the point estimates. This is an admittedly pessimistic view of the results, yet one that is compatible with the observationsdindeed it has been suggested that the confidence interval be renamed ‘‘compatibility interval’’

[19]. The upper limit of the HR was at or above 0.95 (which implies an absolute survival gain of!1.9%) for 37.6% of the estimates and at or above 0.90 (absolute survival gain ! 3.8%) for 59.0% of the estimates. This suggests that a sub- stantial proportion of randomized trials that are considered positive, in terms of statistical significance, are also compatible with a survival benefit that can be considered clinically negligible when rated against the risk and side effects of the treatment. The uncertainty of HR estimates also applies to statistically nonsignificant survival benefits, which may be compatible with large survival gains when the lower 95% confidence limit is considered, but nonsignificant HRs were not included in this analysis.

The uncertainty about the magnitude of the treatment benefit captured by the confidence interval is potentially

important for clinical and regulatory decision-making, and yet it is not discussed in reports of clinical trials. Only three of the 226 studies we included mentioned uncertain estimation or the width of the confidence interval, and none considered specific values of the confidence limits on the HR. This uncertainty is also often neglected when treatment benefits are presented to the patient [7]. The issue is not new, as uncertainty about the effectiveness of medical interventions is a core element of the Grading of Rec- ommendations Assessment, Development and Evaluation system for assessing evidence [20]. We suggest that authors of clinical trial reports not only comment on the clinical relevance of the point estimate of the HR, but also consider the implications of HRs as low and as high as the boundaries of the 95% CI. Similarly, clinicians and patients should at least examine if their treatment decision would change if treatment benefits were represented by either of these extreme values. Such practice would atten- uate the importance of the ‘‘statistically significant’’ point estimate, which is often treated as the definite answer. As a side note, we discuss here uncertainty about the average effect of the treatment. However, even if this parameter was precisely known, there would be no guarantee that an individual patient will benefit to the extent represented

Fig. 2. Scatter-plot of upper 95% confidence limits of hazard ratios, as a function of point estimates, in randomized trials that showed a statistically significant survival benefit of experimental cancer treatments, as primary (black dots) or secondary outcome (circles). Dotted line is the identity function.

(8)

by the trial average [21]. This additional layer of uncertainty poses a further challenge to clinical communication.

How to best communicate with patients about uncertainty and its implications deserves further research and development [6].

This study has several limitations. It is an exploratory study rather than a comprehensive assessment, as we have not included all trials in all areas of medicine and have considered only one outcome. This was done to facilitate interpretation, but limits generalizability. Furthermore, we have not considered the side effects of each drug, and there- fore did not determine what survival benefit might outweigh them in each instance. However, we believe that the thresholds of 0.90 and 0.95 for the HR are rather con- servative, notably in view of noninferiority margins used in recent oncologic trials, which should represent clinically negligible losses of effectiveness. Recent trials have used absolute noninferiority margins for overall survival of 5%

[22] or 10% [23], or HRs between 1.20 (inverse of 0.83) and 1.385 (inverse of 0.72) [24e28]. Another limitation concerns the conversion of HRs into absolute survival gains. The procedure we used relies on the proportionality of hazards, which may not apply to all situations. Finally, lower confidence limits for the HR in our sample often represented large reductions in mortality. We let them aside in this analysis because greater effectiveness is not likely to be problematic, but they are as well supported by the data as the upper limits.

Can the uncertainty that affects many trial results be reduced or eliminated? At this point we only suggest that the uncertainty should be acknowledged and addressed in clinical or policy decisions. Reducing the uncertainty by requiring a stricter definition of statistical significance [29] or by applying tests of non-null hypotheses [30] would lead to increases in sample size, which may hamper thera- peutic progress. A more promising long-term solution may

be Bayesian analysis, which would allow an integration of knowledge obtained in laboratory and preclinical studies with knowledge accrued through clinical trials [31].

In conclusion, we have observed that ambiguous evidence regarding the survival benefit of new anticancer treatments affects about half of positive randomized trial results (37.6% were compatible with an HR O0.95, 59.0% with an HRO0.90). This issue has implications for the design and reporting of clinical trials and perhaps more impor- tantly for the consideration of uncertainty in medical decision-making.

CRediT authorship contribution statement

Thomas V. Perneger:Conceptualization, Data curation, Formal analysis, Methodology, Project administration, Re- sources, Supervision, Writing - original draft. Pauline Brindel: Conceptualization, Data curation, Formal analysis, Writing - review & editing.Christophe Combescure:

Conceptualization, Formal analysis, Writing - review & editing. Angele Gayet-Ageron:Conceptualization, Data curation, Formal analysis, Project administration, Writing - review & editing.

Acknowledgments

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for- profit sectors.

No patients or members of the public were involved in the study.

Data availability: The data set is available as Excel file (attachment).

Ethics approval was not required.

Table 3.Distributions of absolute survival gain for the experimental group, in percent, when the control group reaches median survival (50%), and the associated lower and upper 95% confidence limits for 234 treatment comparisons that showed a statistically significant survival benefit of the experimental treatment

Distribution statistic Absolute survival gain: point estimate

Lower 95% confidence limit

Upper 95% confidence limit

Mean (SD) 13.4 (5.7) 3.8 (3.5) 21.0 (7.3)

Quartiles 9.0, 12.4, 15.2 1.0, 2.8, 5.5 15.8, 19.7, 23.8

Range 4.4 to 39.5 0 to 21.7 8.5 to 46.6

Categories, N (%)

Less than 1% 0 51 (21.8) 0

1% to 1.99% 0 38 (16.2) 0

2% to 3.99% 0 59 (25.2) 0

4% to 7.99% 33 (14.1) 62 (26.5) 0

8% to 15.99% 149 (63.7) 21 (9.0) 64 (27.4)

16% to 31.99% 49 (20.9) 3 (1.3) 146 (62.4)

32% and more 3 (1.3) 0 24 (10.3)

(9)

Appendix A

Supplementary data

Supplementary data to this article can be found online at https://doi.org/10.1016/j.jclinepi.2020.06.026.

References

[1] Greenland S, Senn SJ, Rothman KJ, Carlin JB, Poole C, Goodman SN, et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol 2016;

31(4):337e50.

[2] Rothman KJ. Disengaging from statistical significance. Eur J Epide- miol 2016;31(5):443e4.

[3] Amrhein V, Greenland S, McShane B. Scientists rise up against statistical significance. Nature 2019;567:305e7.

[4] Rothman KJ. A show of confidence. N Engl J Med 1978;299:

1362e3.

[5] Katz J. Why doctors don’t disclose uncertainty. Hastings Cen Rep 1984;14(1):35e44.

[6] Simpkin AL, Armstrong KA. Communicating uncertainty: a narrative review and framework for future research. J Gen Intern Med 2019;34:

2586e91.

[7] Politi MC, Han PKJ, Col NF. Communicating the uncertainty of harms and benefits of medical interventions. Med Decis Making 2007;27:681e95.

[8] Cherny NI, Sullivan R, Dafni U, Kerst JM, Sobrero A, Zielinski C, et al.

A standardised, generic, validated approach to stratify the magnitude of clinical benefit that can be anticipated from anti-cancer therapies: the European Society for Medical Oncology Magnitude of Clinical Benefit Scale (ESMO-MCBS). Ann Oncol 2015;26:1547e73.

[9] Tibau A, Molto C, Ocana A, Templeton AJ, Del Carpio LP, Del Paggio JC, et al. Magnitude of clinical benefit of cancer drugs approved by the US Food and Drug Administration. J Natl Cancer Inst 2018;110:486e92.

[10] Davis C, Naci H, Gurpinar E, Poplavska E, Pinto A, Aggrawal A.

Availability of evidence of benefits on overall survival and quality of life of cancer drugs approved by European Medicines Agency:

retrospective cohort study of drug approvals 2009-13. BMJ 2017;

359:j4530.

[11] Perneger TV, Agoritsas T. Doctors’ and patients’ susceptibility to framing bias: a randomized trial. J Gen Intern Med 2011;26:1411e7.

[12] Zipkin DA, Umscheid CA, Keating NL, Allen E, Aung K, Beyth R, et al. Evidence-based risk communication. A systematic review. Ann Intern Med 2014;161:270e80.

[13] Perneger TV. Estimating the relative hazard by the ratio of logarithms of event-free proportions. Contemp Clin Trials 2008;29:762e6.

[14] Robert C, Long GV, Brady B, Dutriaux C, Maio M, Mortier L, et al.

Nivolumab in previously untreated melanoma without BRAF muta- tion. N Engl J Med 2015;372:320e30.

[15] Slamon D, Eiermann W, Robert N, Pienkowski T, Martin M, Press M, et al. Adjuvant trastuzumab in HER2-positive breast cancer. N Engl J Med 2011;365:1273e83.

[16] Eeles RA, Morden JP, Gore M, Mansi J, Glees J, Wenczl M, et al. Adju- vant hormone therapy may improve survival in epithelial ovarian cancer:

results of the AHT randomized trial. J Clin Oncol 2015;33:4138e44.

[17] Garcia-del-Muro X, Lopez-Pousa A, Maurel J, Martin J, Martinez- Trufero J, Casado A, et al. Randomized phase II study comparing gemcitabine plus dacarbazine versus dacarbazine alone in patients with previously treated soft tissue sarcoma; the Spanish Group for Research on Sarcomas Study. J Clin Oncol 2011;29:2528e33.

[18] Heinemann V, Fischer von Weikersthal L, Decker T, Kiani A, Veh- ling-Kaiser U, Al-Batran S-E, et al. FOLFIRI plus cetuximab versus FOLFIRI plus bevacizumab as first-line treatment for patients with metastatic colorectal cancer (FIRE-3): a randomized, open-label, phase 3 trial. Lancet Oncol 2014;15(10):1065e75.

[19] Gelman A, Greenland S. Are confidence intervals better termed ‘‘uncertainty intervals’’? BMJ 2019;366:I5381.

[20] Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso- Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008;336:

924e6.

[21] Kravitz RL, Duan N, Braslow J. Evidence-based medicine, heteroge- neity of treatment effects, and the trouble with averages. Milbank Q 2004;82(4):662e87.

[22] Kim H-H, Han S-U, Kim M-C, Kim W, Lee H-J, Ryu SW, et al. Ef- fect of laparoscopic distal gastrectomy vs open distal gastrectomy on long-term survival among patients with stage I gastric cancer. The KLASS-01 Randomized Clinical Trial. JAMA Oncol 2019;5(4):

506e13.

[23] Mehanna H, Wong W-L, McConkey CC, Rahman JK, Robinson M, Hartley AGJ, et al. PET-CT surveillance versus neck dissection in advanced head and neck cancer. N Engl J Med 2016;374:1444e54.

[24] Mejean A, Ravaud A, Thezenas S, Colas S, Beauval J-B, Bensalah K, et al.

Sunitinib alone or after nephrectomy in metastatis renal cell carcinoma. N Engl J Med 2018;379:417e27.

[25] Lee SM, Falzon M, Blackhall F, Spicer J, Nicolson M, Chaudhuri A, et al. Randomized prospective biomarker trial of ERCC1 for comparing platinum and nonplatinum therapy in advanced non- small-cell lung cancer: ERCC1 Trial (ET). J Clin Oncol 2017;35:

402e11.

[26] Eisenberger M, Hardy-Bessard A-C, Kim CS, Geczi L, Ford D, Mourey L, et al. Phase III study comparing a reduced dose of caba- zitaxel (20 mg/m²) and the currently approved dose (25 mg/m²) in postdocetaxel patients with metastatic castration-resistant prostate cancer-PROSELICA. J Clin Oncol 2017;35:3198e206.

[27] Kitagawa R, Katsumata N, Shibata T, Kamura T, Kasamatsu T, Nakanishi T, et al. Paclitaxel plus carboplatin versus paclitaxel plus cisplatin in metastatic or recurrent cervivcal cancer : the open-label randomized phase III trial JCOG0505. J Clin Oncol 2015;33:

2129e35.

[28] Kayama T, Sato S, Sakurada K, Mizusawa J, Nishikawa R, Narita Y, et al. Effects of surgery with salvage stereotactic radiosurgery versus surgery with whole-brain radiation therapy in patients with one to four brain metastases (JCOG0504): a phase III, noninferiorityr- andomized controlled trial. J Clin Oncol 2018;36:3282e9.

[29] Farrington CP, Manning G. Test statistics and sample size formulae for comparative binomial trials with null hypothesis of non-zero risk difference or non-unity relative risk. Stat Med 1990;9:1447e54.

[30] Benjamin DJ, Berger JO, Johannesson M, Nosek BA, Wagenmakers E-J, Berk R, et al. Redefine statistical significance.

Nat Hum Behav 2018;2(1):6e10.

[31] Berry DA. Bayesian clinical trials. Nat Rev Drug Discov 2006;5:

27e36.