Equivalence trials for sepsis in newborns and young infants

(1)

a

Equivalence trials

for sepsis in newborns and young infants

Review of the statistical aspects

of trials claiming to show equivalence

of simplified treatment regimes

(2)

ISBN: 978-92-890-5571-0

© World Health Organization 2021

Some rights reserved. This work is available under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo).

Under the terms of this licence, you may copy, redistribute and adapt the work for noncommercial purposes, provided the work is appropriately cited, as indicated below. In any use of this work, there should be no suggestion that WHO endorses any specific organization, products or services. The use of the WHO logo is not permitted. If you adapt the work, then you must license your work under the same or equivalent Creative Commons licence. If you create a translation of this work, you should add the following disclaimer along with the suggested citation: “This translation was not created by the World Health Organization (WHO). WHO is not responsible for the content or accuracy of this translation. The original English edition shall be the binding and authentic edition: Equivalence trials for sepsis in newborns and young infants: review of the statistical aspects of trials claiming to show equivalence of simplified treatment regimes. Copenhagen: WHO Regional Office for Europe; 2021”.

Any mediation relating to disputes arising under the licence shall be conducted in accordance with the mediation rules of the World Intellectual Property Organization.

Suggested citation. Equivalence trials for sepsis in newborns and young infants: review of the statistical aspects of trials claiming to show equivalence of simplified treatment regimes. Copenhagen: WHO Regional Office for Europe; 2021. Licence: CC BY-NC-SA 3.0 IGO.

Cataloguing-in-Publication (CIP) data. CIP data are available at http://apps.who.int/iris.

Sales, rights and licensing. To purchase WHO publications, see http://apps.who.int/bookorders. To submit requests for commercial use and queries on rights and licensing, see http://www.who.int/about/licensing.

Third-party materials. If you wish to reuse material from this work that is attributed to a third party, such as tables, figures or images, it is your responsibility to determine whether permission is needed for that reuse and to obtain permission from the copyright holder. The risk of claims resulting from infringement of any third-party-owned component in the work rests solely with the user.

General disclaimers. The designations employed and the presentation of the material in this publication do not imply the expression of any opinion whatsoever on the part of WHO concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. Dotted and dashed lines on maps represent approximate border lines for which there may not yet be full agreement. The mention of specific companies or of certain manufacturers’

products does not imply that they are endorsed or recommended by WHO in preference to others of a similar nature that are not mentioned. Errors and omissions excepted, the names of proprietary products are distinguished by initial capital letters.

All reasonable precautions have been taken by WHO to verify the information contained in this publication. However, the published material is being distributed without warranty of any kind, either expressed or implied. The responsibility for the interpretation and use of the material lies with the reader. In no event shall WHO be liable for damages arising from its use.

In this report, four recently published randomized trials were analysed for their statistical power to demonstrate equivalence of simplified antibiotic regimens with the standard of care among children with bacterial infection. These trials were found to be underpowered in terms of both overall sample size and number of included children with bacterial infection. They cannot support the conclusion that the alternative antibiotic regimens studied are equally effective. Much larger sample sizes and more selective inclusion criteria are needed. The conclusion that simpler treatment regimens for the treatment of newborn sepsis are effective is not warranted, based on the published trials. The promotion of such regimens as an alternative to established treatments needs to be stopped.

Кeywords NEWBORN NEONATAL SEPSIS INFANT

SIGNS AND SYMPTOMS

Equivalence trials

for sepsis in newborns and young infants

Review of the statistical aspects of trials claiming to show equivalence

of simplified treatment regimes

(3)

for sepsis in newborns and young infants

Review of the statistical aspects of trials claiming to show equivalence

of simplified treatment regimes

(4)

(5)

v

Acknowledgements vi

Abbreviations vi

Introduction 1

Methods 3

Trials 3

Inclusion/exclusion criteria and case-finding 7

Prevalence of bacterial infection in infants eligible for recruitment 9

Sensitivity and specificity of recruitment criteria 11

Apparent treatment-failure rate among infants without bacterial infections 11

Statistical methods 11

Results 13

Corrected type I error rate, equivalence margin, and prevalence estimates 13

Sample size and recruitment criteria accuracy needed to show equivalence in neonates and young infants with bacterial infections 13

Visualization for power and sample size correction 14

Interactive tools for computing corrected measures 16

Discussion 17

Limitations 18

Conclusions 19

References 20

Annex 1 Details of statistical calculations 22

(6)

vi

of America, and Dr Pia Maier, Centre for Paediatric and Adolescent Medicine, University Hospital Heidelberg, Germany.

Statistical review of the paper was performed by Dr Denis Agniel, RAND Corporation, Santa Monica, United States of America.

The WHO Regional Office for Europe wishes to acknowledge the helpful discussions and critical review of the paper from: Professor John Carlin, Murdoch Children’s Research Institute and University of Melbourne, Australia;, Professor Ron Dagan, University of Beer Sheva, Israel;, Professor Trevor Duke, University of Melbourne, Australia;, and Professor Kim Mulholland, London School of Hygiene and Tropical Medicine, United Kingdom.

Abbreviations

AFRINEST African Neonatal Sepsis Trial

IMCI integrated management of childhood illness MLE maximum likelihood estimate

PPV positive predictive value

pSBI possible serious bacterial infection SATT Simplified Antibiotic Therapy Trial

(7)

Introduction

Serious bacterial infections are a major cause of neonatal deaths worldwide, including in the higher-mortality countries of the WHO European Region (1). Manifestations such as

pneumonia, meningitis and sepsis form a continuum that is difficult to distinguish in this age group, so all are summarized as neonatal sepsis (2).

Over the past 30 years, WHO has invested in several studies on the aetiology of neonatal sepsis and clinical signs that might possibly identify neonates and young infants with sepsis (3–5). The first compilation of such signs derived from the WHO Young Infants Study

undertaken in the early 1990s (3,4). The list comprised 14 signs, and this fact led to a high sensitivity but low specificity of the algorithm (2). This list of signs was integrated into the neonatal integrated management of childhood illness (IMCI) algorithm as signs indicating

“possible serious bacterial infection” (6).

Because of the large number of children being referred following this approach, and both parents and health workers being reluctant to refer children who had one sign but were not considered sick, another large multicentre study was undertaken (5). The analysis of this study showed that the list of predictors of possible sepsis could be reduced to seven signs, maintaining a reasonable balance of sensitivity and specificity (sensitivity of 85% and

specificity of 75% in this second study for neonates in the first week of life for indicating need for admission to hospital).

Both studies (2,5) were performed in ill infants who had been taken to a health facility. While the first study was interested mainly in predicting sepsis, the second was expanded to also identify infants with other conditions, such as birth asphyxia and jaundice. Even though this was a larger list of conditions, the term “possible serious bacterial infections” was maintained.

Looking at the yield of blood cultures in these studies, only around 10% of the preselected children had a positive culture, many of which had possible contaminants (4,7).

Signs of bacterial infection are often non-specific and subtle, and neonates might deteriorate quickly. Even where a qualified assessment is possible, the standard of care is considered to be admission to hospital for observation and initiation of antimicrobial treatment while waiting for test results, then monitoring the child’s progress in response to the initiated treatment.

Inpatient treatment usually consists of injectable antibiotics such as ampicillin and gentamicin. Oral treatment is not considered an option in this age group in developed countries (8).

Timely access to a competent health-care provider is often a problem where health systems are weak. Hospital treatment often is either not possible or not acceptable for families in resource-poor settings. They might not be convinced that the child is seriously ill, or the referral would put an undue financial or other burden on them. In addition, the poor reputation of many hospitals means they may not be seen as offering a good option for improving survival.

Without further work on clinical signs or diagnostic tests to identify children who really have a bacterial infection, a series of randomized controlled equivalence trials began to compare the standard of care with simplified antibiotic regimens administered in the community for young infants. These trials, called the African Neonatal Sepsis Trial (AFRINEST) and the Simplified Antibiotic Therapy Trial (SATT) (9–11), have been published within the last six years. They led to treatment recommendations for neonates and young infants in situations

(8)

“where referral is not possible” (12). The recommendations are often suggested as an alternative option for treatment of neonatal sepsis beyond established practice.

Some concerns have emerged about these trials. Equivalence trials of antibiotic treatments that include large numbers of neonates and young infants without bacterial infection are at high risk of erroneously concluding equivalence. This is the case even when there is a clinically meaningful difference between regimens for neonates and young infants with bacterial

infection. This statistical liability derives from the fact that following randomization, all trial arms will have similar and large proportions of neonates and young infants without bacterial infection.

These infants will not respond to antibiotic treatment of any kind, causing the spontaneous cure and failure rates to be roughly the same for all trial arms. As a result, the observed treatment difference between trial arms will be much smaller than the true treatment difference for infants with bacterial infections, who are the only infants who stand to benefit from antibiotic treatment. These observations are not new: other authors have discussed this effect and its consequences in detail for different bacterial infections, such as acute otitis media (13–18). The AFRINEST and SATT trials are particularly susceptible to this flaw since, as discussed, the inclusion criteria were based on clinical signs that have much less than perfect specificity for true bacterial infection.

This report shows that these trials of sepsis treatments in neonates and young infants were not able to demonstrate equivalence of antibiotic regimens as claimed. By adjusting key trial parameters to account for the likely proportion of randomized infants with true bacterial infections, it demonstrates that these trials were drastically underpowered to detect treatment equivalence. As a consequence, the true uncertainty in the estimated treatment difference is much higher than reported.

The report also discusses how the trial characteristics make the conclusion of treatment equivalence even less applicable to the general population of newborns and young infants for whom the recommendations “where referral is not possible” are made, even in high-mortality settings.

(9)

Methods

Trials

This section presents a summary of four randomized equivalence trials used for the

development of the guidelines for antibiotic treatment of neonates and young infants when referral is not possible (12). These trials, summarized below, aimed to compare antibiotic regimens for the treatment of clinical severe infection in outpatient settings.

1. The SATT trial in Pakistan, which compared three treatment regimens (9):

a. intramuscular procaine benzylpenicillin and gentamicin once per day for seven days;

b. oral amoxicillin twice per day and intramuscular gentamicin once per day for seven days; and

c. intramuscular procaine benzylpenicillin and intramuscular gentamicin both once per day for two days followed by oral amoxicillin twice per day for five days.

2. The AFRINEST trial at ﬁve sites in the Democratic Republic of the Congo, Kenya and Nigeria, which compared four treatment regimens (11):

a. intramuscular procaine benzylpenicillin and gentamicin once per day for seven days;

b. intramuscular gentamicin once per day and oral amoxicillin twice per day for seven days;

c. intramuscular procaine benzylpenicillin once per day for two days followed by oral amoxicillin twice per day for five days; and

d. intramuscular gentamicin once per day for two days followed by oral amoxicillin twice per day for seven days.

This part of the AFRINEST study included children with signs of infection other than fast breathing only.

3. The AFRINEST trial at ﬁve sites in the Democratic Republic of the Congo, Kenya and Nigeria, which compared two treatment regiments (10):

a. intramuscular procaine benzylpenicillin and gentamicin once per day for seven days; and

b. oral amoxicillin twice per day for seven days.

This part of the AFRINEST study included children with fast breathing only.

4. An equivalence trial in Bangladesh, which compared three treatment regimens (19): a. intramuscular procaine benzylpenicillin and gentamicin once per day for seven

days;

b. intramuscular gentamicin once per day and oral amoxicillin twice per day for seven days; and

c. intramuscular procaine benzylpenicillin and gentamicin once per day for two days followed by oral amoxicillin twice per day for five days.

Features and results of the trials are summarized in Fig. 1.

(10)

STUDY 2

Source: Tshefu et al. (11).

Amoxicillin and gentamicin

(7 days) 826 infants

B Procaine benzylpenicillin

and gentamicin (2 days), then amoxicillin (5 days)

862 infants

C Gentamicin and

amoxicillin (2 days), then amoxicillin (5 days)

848 infants

D A

Procaine benzylpenicillin and gentamicin

(7 days) 828 infants

Deaths by day 15 Deaths by day 8 PTF^a by day 8

10 = 1%

12 = 1%

67 = 8%

8 = 1%

10 = 1%

51 = 6%

17 = 2%

20 = 2%

65 = 8% :

10 = 1%

11 = 1%

46 = 5% : N = 3364 Source: Mir et al. (9).

Deaths before day 15 Deaths within 7 days

PTF^a within 7 days

11 = 1%

13 = 2%

90 = 12%

PTF^a within 7 days

7 = 1%

9 = 1%

76 = 10%

PTF^a within 7 days

12 = 2%

99 = 13% :

10 = 1%

N = 2251

Amoxicillin and gentamicin (7 days) 751 infants Procaine benzylpenicillin B

and gentamicin (7 days) 747 infants

A Procaine benzylpenicillin

and gentamicin (2 days), then amoxicillin (5 days)

753 infants

C STUDY 1

Fig. 1. Summary of equivalence trials on neonatal sepsis

SATT in Pakistan

AFRINEST in Democratic Republic of the Congo, Kenya and Nigeria

(11)

STUDY 3

Amoxicillin (7 days) 1135 infants Procaine benzylpenicillin B

A

Source: Tshefu et al. (10).

Deaths within 15 days Deaths by day 8 PTF^a by day 8

4 = <1%

234 = 22%

Deaths within 15 days Deaths by day 8 PTF^a by day 8

4 = <1%

221 = 19%

2 = <1%

N = 2196

STUDY 4

Gentamicin and amoxicillin (7 days) 782 infants Procaine benzylpenicillin B

A Procaine benzylpenicillin and

gentamicin (2 days), then amoxicillin (5 days) 790 infants

C

Source: Baqui et al. (19).

Deaths before day 15 Deaths before day 8 PTF^a by day 8

13 = 2%

14 = 2%

78 = 10%

Deaths before day 15 Deaths before day 8

PTF^a by day 8 65 = 8%

9 = 1%

12 = 2% Deaths before

day 15 Deaths before day 8 PTF^a by day 8

12 = 2%

6 = 1%

64 = 8% : N = 2367

a PTF: primary treatment failure.

Fig. 1 contd

Randomized, open-label equivalence trial in Bangladesh

AFRINEST in the Democratic Republic of the Congo, Kenya and Nigeria

(12)

Table 1. Trial parameters and additional assumptions

Parameter Study 1

Mir et al. (9)

Study 2

Tshefu et al. (11)

Study 3

Tshefu et al. (10)

Study 4

Baqui et al. (19) Specified by trial authors

Equivalence margin 5% 5% 5% 5%

Nominal power 90% 90% 80% 90%

Nominal type I error rate 2.5% 2.5% 2.5% 2.5%

Number randomized per trial

arm^a 753 862 1 135 795

Anticipated treatment failure rate among bacterial infections (used for sample size calculation)

10% 10% 10% 10%

Additional assumptions

Plausible range for prevalence of bacterial infections in neonates eligible for recruitment

1.5–7.5% 1.5–7.5% 1.5–7.5% 1.5–7.5%

Approximate sensitivity of

recruitment criteria 70% 70% 70% 70%

Plausible range for specificity of

recruitment criteria 60–80% 60–80% 60–80% 60–80%

Positive predictive value (% of children in trial with bacterial infection)^b

3–22% 3–22% 3–22% 3–22%

Apparent treatment failure rate among infants without bacterial infections

12% 7% 21% 9%

a Where sample sizes differed slightly across trial arms, the largest arm is shown.

b Calculated from the prevalence, sensitivity and specificity in the above rows.

Table 1 summarizes key parameters needed for the calculations presented in this report, including values specified by the trial authors (equivalence margin, power, type I error rate, sample size and the anticipated treatment failure rate used for the authors’ sample size calculations) and some important additional assumptions (prevalence of bacterial infection among neonates eligible for recruitment, sensitivity and specificity of the clinical signs used as recruitment criteria and the rate at which neonates without bacterial infection recover during the observation period).

Justifications for the assumptions in Table 1 are provided in detail below. The most important quantities that must be taken into account when computing the changes in trial design

parameters are: the prevalence of bacterial infection in the population of infants eligible for trial inclusion; and the specificity and sensitivity of the recruitment criteria for bacterial infection. Together, these parameters determine the expected proportion of young infants included in the trial who have a bacterial infection, also known as the positive predictive value (PPV). The recruitment criteria are therefore discussed first, along with case-finding methods that can also impact on prevalence, sensitivity and specificity, as outlined below.

(13)

Inclusion/exclusion criteria and case-finding

All four trials used a combination of clinical signs and symptoms to identify neonates and young infants with possible serious bacterial infection who were therefore eligible for

recruitment. These signs and symptoms were based on those established by the IMCI studies (3–5). Table 2 describes trial inclusion and exclusion criteria in more detail.

All trials used a combination of active and passive case-finding. Active case-finding involved community health workers or nurses systematically searching for cases of infection. The nurses typically visited children at home looking for clinical signs, and usually visited the same child on multiple occasions. The protocols also allowed passive case-finding: for example, children whose parents presented them to a clinic with concerns about severe infection were eligible for inclusion.

Table 2. Trial inclusion and exclusion parameters Inclusion criteria Exclusion criteria

Study 1

Mir et al. (9) - age: 0–59 days

- living in the catchment area - refusal by family to be admitted to

hospital

- one or more signs of clinical severe infection

- had one or more signs of critical illness:

• movement only when stimulated

• not feeding well on observation

• temperature ≥ 38 °C or < 35.5 °C

• severe chest indrawing (inward chest movement with every breath in one minute)

- if the family agreed to admission - weight at presentation was less

than 1 500 g

- major congenital malformations or suspected chromosomal abnormalities were present - surgical conditions that needed

hospital referral

- they had been admitted for illness in the past two weeks

- they had been included previously in the study

- they had one or more signs of critical illness:

• unconsciousness

• convulsions

• inability to feed

• apnoea

• inability to cry

• cyanosis

• bulging fontanelle

• major congenital malformations inhibiting oral antibiotic intake

• active bleeding needing transfusion

• persistent vomiting, defined as vomiting after three attempts to feed the infant within 30 minutes, with the infant vomiting after each attempt

(14)

Inclusion criteria Exclusion criteria

Study 2

Tshefu et al. (11) - age 0–59 days, any sign of clinical severe infection (stopped feeding well (defined as poor feeding on observation), movement only when stimulated, severe chest indrawing, and axillary temperature ≥ 38.0 °C or < 35.5 °C)

- parents did not accept or could not access referral-level care

- parents gave consent to participate in the study

- critically ill, characterized by the presence of any of the following signs:

• unconsciousness

• convulsions

• unable to feed at all

• apnoea

• unable to cry

• cyanosis

• dehydration

• major congenital malformations inhibiting oral antibiotic intake

• active bleeding requiring transfusion

• surgical conditions needing hospital referral

• persistent vomiting, defined as vomiting after three attempts to feed the baby within 30 minutes - very low weight (< 1 500 g at the

time of presentation)

- hospital admission for illness in the past two weeks

- previously enrolled in the study Study 3

Tshefu et al. (10) - age 0–59 days

- fast breathing (defined as

respiratory rate of ≥ 60 breaths per minute)

- parents did not accept or could not access referral-level care and - parents gave consent to participate

in the study

- signs of clinical severe infection defined as:

• poor feeding on observation

• movement only when stimulated

• severe chest indrawing

• axillary temperature ≥ 38.0 °C or

< 35.5 °C

- critical illness characterized by

• presence of unconsciousness

• convulsions

• inability to feed at all

• apnoea

• cyanosis

• dehydration

• major congenital malformations that inhibited oral antibiotic intake

• active bleeding that necessitated transfusion

• persistent vomiting, defined as vomiting after three attempts to feed the baby within 30 minutes

• very low weight (< 1 500 g at the time of presentation)

• hospital admission for illness in the past two weeks

• previous enrolment in the study

Table 2 contd

(15)

Inclusion criteria Exclusion criteria

Study 4

Bacqui et al. (19) - age 0–59 days

- residence within a predefined geographical area based on feasibility of follow-up visits - the presence of at least one of five

clinical signs of severe infection:

• severe lower chest wall indrawing

• axillary temperature 38.0 °C or more(≥ 100.4 °F) confirmed by a second reading

• axillary temperature of 35·5 °C or less (≤ 95.9 °F) confirmed by a second reading

• lethargy, defined as movement only on stimulation by the examining physician

• history of feeding problems, confirmed by poor suck on examination

- Infants with fast breathing alone (respiratory rate ≥ 60 breaths per minute) were excluded

- infants with signs of critical illnesses were excluded

- presence of any of the following signs:

• unconsciousness

• history or presence of convulsions at assessment

• inability to feed

• apnoea

• cyanosis

• major congenital malformations

• major bleeding

• persistent vomiting after three attempts to feed the baby within 30 minutes

• physician’s suspicion of meningitis - weight less than 1 500 g

- had been admitted to hospital for illness in the past two weeks - had previously been included in the

study

Table 2 contd

Prevalence of bacterial infection in infants eligible for recruitment

The prevalence of bacterial infection in the population of young infants eligible for recruitment into each trial has to be taken into account when analysing trial designs and computing the changes in trial design parameters. As discussed above, since the trials

reviewed here allowed both active and passive case-finding, this prevalence will be impacted both by true prevalence in the community and prevalence among ill infants presenting for care. Data that allow direct estimation of this prevalence are not available. Instead, a plausible range of values will be developed based on published data on the incidence of severe bacterial infection in neonates in the trial countries. This incidence has not previously been

systematically investigated.

A literature review aimed to enable estimation of the incidence of serious bacterial infections through two approaches: identifying blood culture-positive bacterial infections in south Asia and in individual countries (Bangladesh, India, Kenya, Nepal and Sri Lanka (Table 3)); and reviewing studies that calculated estimates on neonatal bacterial infection.

(16)

Table 3. Incidence of culture-positive neonatal sepsis based on literature review Incidence of culture-positive sepsis

per 1 000 live births Case fatality rates

South Asia 1.6

(Saha et al. (20)) NA

South Asia 15.7

(pooled incidence of 15 reports) (Chaurasia et al. (21))

34.4

(Chaurasia et al. (21))

Bangladesh 3

(Darmstadt et al. (22)) NA

Bangladesh NA 19.1

(Chaurasia et al. (21))

India 6.7

(Panigrahi et al. (23)) NA

India 16

(n = 14 studies) (Chaurasia et al. (21))

34.4

(Panigrahi et al. (23))

Kenya 5.5

(Chaurasia et al. (21)) NA

Nepal 11.6

(Chaurasia et al. (21)) 64.7

(Darmstadt et al. (22))

Sri Lanka 13.6

(Chaurasia et al. (21)) NA

Pakistan NA 30.9

(Panigrahi et al. (23)) NA: not available.

The reported incidence of culture-positive bacterial infection varied widely between

countries, but also between studies, ranging from 3–16 per 1000 live births. Two publications provided estimates of identified bacterial infections. In the study of Seale et al., a systematic literature review was undertaken to obtain estimates of the incidence of possible serious bacterial infection (pSBI). They estimated a pooled pSBI incidence of 7.6 per 100 live births (24). As this is not an incidence of bacterial infection, but only that of children with clinical signs, and the signs have limited specificity, this estimate does not reflect a bacteriological correlate and is therefore quite inflated. It therefore was not included in the calculations below.

Saha et al. estimated the cause-specific incidence per 1000 live births based on results of a partially latent class model. They calculated the mean incidence of bacterial infection to be 13.2 per 1000 live births in the overall population (20). As this is a population-based

incidence, it is probably higher than the point prevalence in children presenting with illness to a health facility and certainly higher than in those at any given time during a home visit.

Consequently, based on the reviews of culture-positive bacterial infections and estimates provided by Seale et al. and Saha et al., a rather wide and high plausible range for the point prevalence of bacterial infections of children eligible for recruitment in the trials of 1.5–7.5%, was chosen.

(17)

Sensitivity and specificity of recruitment criteria

A perfect diagnostic test would have 100% sensitivity and 100% specificity, but this is never possible in practice. Trade-offs between sensitivity and specificity must be made: algorithms can be adjusted to be more sensitive but less specific, or more specific but less sensitive. For clinical practice, not missing a case is considered more important, so the balance is often on the side of sensitivity. Clinical signs that are too unspecific, however, lead to overtreatment and over-referral, which presents a burden to patients and health systems. This was observed, as outlined in the introduction, with the first young-infants algorithm published in IMCI in 1998 (6).

The situation with treatment trials is radically different. Specificity is of the utmost

importance to demonstrate a difference between treatments. This was discussed in the design of vaccine trials, where having a specific outcome definition was considered important (25).

In contrast to the two clinical IMCI studies referred to above, from which estimates of

sensitivity and specificity of clinical signs were derived (sensitivity of 85% and specificity of 75%), all four trials reviewed here recruited neonates and young infants using a combination of active and passive case-finding. The active case-finding often involved repeat home visits to the same children and is therefore likely to have led to further reduced specificity compared to that estimated by the IMCI studies.

To the authors’ knowledge, there are no published studies reporting the sensitivity and specificity of clinical signs with a combination of active and passive case-finding. As a result, a wide range of possible values for the specificity of trial recruitment criteria, between 60% and 80%, was considered. For the sensitivity, which has less of an impact on power to detect a difference between treatments, a moderate value of 70% was chosen.

Apparent treatment-failure rate among infants without bacterial infections

When infants without bacterial infections are present in a trial, the rate at which their illness does not resolve during the observation period will affect the power of the trial. These infants will be recorded as treatment failures in each trial; this rate will therefore be referred to as the

“apparent treatment-failure rate” among infants without bacterial infections. This value was assumed to be approximately equal to the overall treatment-failure rate, averaged across all arms. This is a reasonable approximation, as the vast majority of infants recruited will not have bacterial infections as a result of the low underlying prevalence of bacterial infection and poor specificity of recruitment criteria.

Statistical methods

For each of the four trials described above, five statistical quantities corrected for the presence of neonates and young infants without bacterial infections in the study samples were

computed. These calculations are based on the following principle.

Suppose t is the true difference in the proportion of treatment failures for neonates and young infants receiving the standard antibiotic regimen compared to those receiving an experimental regimen. Also suppose that p is the probability that an infant recruited to the trial truly has a

(18)

bacterial infection. Assume that infants without bacterial infections will recover at the same rate when receiving either treatment. Then the difference in failure rates among all neonates and young infants included in the trial will be smaller than the true treatment difference among neonates and young infants with bacterial infections, making the treatment regimens appear closer to equivalent (13,15). Specifically, the expected treatment difference in the trial will be p*t, which is less than t since p < 1.

The following quantities were computed: detailed derivations for all calculations are provided in Annex 1. Here, these quantities and their interpretation will be described briefly. Readers interested in the statistical details should refer to Annex 1.

First, this report aims to demonstrate the impact of the presence of neonates and young infants without bacterial infections on key measures of statistical performance. For each trial, three corrected quantities were computed: type I error rate; power; and an estimate of the treatment difference, derived via maximum likelihood, and a 95% confidence interval (CI).

Secondly, changes in trial design parameters needed to demonstrate equivalence were

computed after correcting for the presence of infants without bacterial infections. Specifically, for each trial the following quantities were computed: the sample size needed to achieve the study’s nominal power (typically 90%) with current recruitment criteria; and the specificity needed to achieve the nominal power at the current sample size and with a sensitivity of 70%.

Specificity was emphasized over sensitivity because the specificity dominates the proportion of so-called false positives recruited (infants without bacterial infections). Power therefore is much more responsive to specificity than sensitivity.

The above calculations were based on assumptions about the prevalence of bacterial

infections in the study population, the sensitivity and specificity of recruitment criteria, and the spontaneous recovery rate for infants without bacterial infections (see Annex 1 for details). The results are sensitive to these assumptions. These assumptions were therefore varied within plausible ranges (see Table 1) and the corresponding range of results was computed. A web application was also developed to allow readers to repeat all calculations under different assumptions.¹

All analyses were performed in R Version 3.6.1 (The R Foundation, 2009).²

1 This can be accessed at: https://emgthomas.shinyapps.io/table3_calcs/

2 Code to reproduce all analyses is available at: https://github.com/emgthomas/statistical_aspects_equivalence_trials

(19)

Results

Corrected type I error rate, equivalence margin, and prevalence estimates

After correcting for the presence of neonates and young infants without bacterial infections, the type I error rate for all four trials considered was around 62% at minimum (Table 4, row 1).

Put another way, even when the true treatment difference among neonates and young infants with bacterial infections was clinically meaningful – at least as large as the stated equivalence margin – the risk of erroneously concluding equivalence was estimated to be 62% in the best- case scenario, and could be as high as 98%. These type I error rates are unacceptable by any standard: the nominal type I error rate for all trials was only 2.5%, and 5% is the norm (that is, a p-value of 0.05).

A simple approach to account for the presence of neonates and young infants without

bacterial infections in the statistical analysis of these trials is to adjust the equivalence margin accordingly (see Annex 1). However, all four trials were drastically underpowered to detect equivalence within the adjusted margin, with power in the range of 3–14% (Table 4, row 2).

An alternative approach is to correct the estimated treatment difference and its CI so that they represent estimates of the treatment difference in young infants with bacterial infections only.

This results in very wide CIs that include values well outside the trials’ stated equivalence margins (Table 4, row 3). For example, for the SATT trial (9), a conservative corrected 95% CI for the treatment difference is −21% to 5%, which includes the equivalence margin of 5%.

Under different assumptions about the likely fraction of infants in the trial with true bacterial infection, however, the CI for this trial could be as large as −94% to 75%. With a CI this wide, the conclusion of equivalence is untenable. Results for the other trials were similar.

Sample size and recruitment criteria accuracy needed to show equivalence in neonates and young infants with bacterial infections

When using current recruitment criteria, enormous sample sizes of at least 12 000 per trial arm, and possibly more than 1 million, would be needed to demonstrate treatment

equivalence in neonates and young infants with bacterial infection (Table 4, row 4). These sample sizes are not feasible. An alternative would be to use recruitment criteria with higher accuracy to detect bacterial infection while maintaining a more reasonable sample size.

Table 4 (row 5) shows that in a trial with 5000 infants per arm, recruitment criteria with a specificity of at least 96% for bacterial infection would be needed (assuming the sensitivity was 70%; note that lower sensitivity would necessitate even higher specificity).

(20)

Table 4. Trial features and results corrected for the presence of subjects with non-bacterial infections

Corrected

parameter Study 1

Mir et al. (9) Study 2

Tshefu et al. (11) Study 3

Tshefu et al. (10) Study 4 Baqui et al. (19)

1. Type I error (%) 64–83 86–98 62–82 74–92

2. Power (%) 3–10 3–14 3–10 3–12

3. Estimate and 95% CI for treatment difference^a

−0.09 (−0.21, 0.05) to

−0.14 (−0.94, 0.75)^b

−0.09 (−0.18, 0.02) to

−0.49 (−1.00, 0.40)^с

−0.12 (−0.27, 0.04) to

−0.62 (−1.00, 0.54)^d

−0.07 (−0.19, 0.06) to

−0.40 (−1.00, 0.55)^e 4. Required sample

size to achieve nominal power and type I error rate (to the nearest 1 000)

17 600–1 312 000 12 000–820 000 21 000–545 000 14 000–1 024 000

5. Specificity required to achieve nominal power and type I error rate for sample size per trial arm of 5 000 and sensitivity of 0.70

0.966–0.994 0.963–0.993 0.969–0.994 0.993–0.965

a Treatment difference: treatment failure rate in experimental arm minus treatment failure rate in standard arm.

If the trial had more than two arms, results are shown comparing one experimental arm only to standard treatment, as specified in footnotes.

b Standard treatment: intramuscular procaine benzylpenicillin and gentamicin for seven days; experimental treatment: oral amoxicillin and intramuscular gentamicin for seven days.

c Standard treatment: intramuscular procaine benzylpenicillin and gentamicin for seven days; experimental treatment: oral amoxicillin and intramuscular gentamicin for seven days.

d Standard treatment: intramuscular procaine benzylpenicillin and gentamicin for seven days; experimental treatment: oral amoxicillin for seven days.

e Standard treatment: intramuscular procaine benzylpenicillin and gentamicin for seven days; experimental treatment: intramuscular gentamicin and oral amoxicillin for seven days.

Visualization for power and sample size correction

Fig. 2 visualizes the impact of the percentage of neonates and young infants without bacterial infection on power and sample size required to establish treatment equivalence. It shows an example based on the Mir et al. (9) trial of sepsis in young infants in Pakistan. First, if the trial recruitment criteria had perfect specificity (100%) for bacterial infection – meaning the PPV is 1, so that all infants had culture-positive bacterial infections at recruitment – a sample size of 757 per trial arm would be required to detect treatment equivalence within a margin of 5%

and with 90% power.³ This is close to the required sample size of 750 computed by Mir et al. (9) and is illustrated on the plot by the lowest horizontal (green) line corresponding to n = 757, which meets the 90% power contour when PPV = 1.

3 Additional assumptions: type I error rate = 2.5%; treatment-failure rate among infants with bacterial infection = 10%; treatment-failure rate among infants without bacterial infection = 12%.

(21)

Fig. 2 also demonstrates what happens when the trial recruitment criteria have less than perfect accuracy for bacterial infection. Assume the sensitivity of these criteria is 70% and the specificity is between 60% and 80%. Based on a literature review, the prevalence of neonatal sepsis among Pakistani young infants screened for the trial could be as low as 1.5% or as high as 7.5% (Table 3). These assumptions imply a PPV of between 2.6% and 22.0%, represented on the plot by the two vertical orange lines (see Annex 1 for PPV formula). At the originally computed sample size of 757, this means that the true power of the study to detect

equivalence is between 3% and 10%, drastically lower than the assumed 90%. This is

illustrated by the power contours at the intersections of the horizontal green line (sample size n = 757) and the two vertical orange lines (PPV = 2.6% and PPV = 26.0%).

Finally, follow the vertical orange lines upwards until they meet the desired power contour of 90%. Moving horizontally across from here to the Y-axis shows that the sample size required to detect equivalence with 90% power is between 17 585 and 1 312 447 per trial arm, between 23 and 1700 times larger than the originally calculated n = 757.

Fig. 2. Impact of PPV on study power and required sample size

Note: the plot shows how power, shown by the contours, rapidly varies as a function of PPV (PPV: percentage of recruited infants with true bacterial infection, shown on the X-axis) and sample size per trial arm (Y-axis). Note that both axes are shown on a log scale. As an example, the assumptions for this plot mirror that of Study 1 ((9);

see Table 1): sensitivity = 70%; specificity = 60–80%; prevalence of bacterial infection = 1.5–7.5%; desired power = 90%; type I error rate = 2.5%; equivalence margin = 5%; treatment-failure rate among infants with bacterial infections = 10%; treatment-failure rate among infants without bacterial infections = 12% (Table 1).

An interactive version of Fig. 2 that allows the user to vary these assumptions is available at:

https://emgthomas.shinyapps.io/power_calc/.

(22)

Interactive tools for computing corrected measures

An online interface⁴ is available for repeating all calculations shown above. An interactive version⁵ of Fig. 2 is also available. These tools are intended to allow the reader to vary the assumptions shown in Table 1 to observe the impact on the results and to obtain results for other equivalence trials.

4 This can be accessed at: https://emgthomas.shinyapps.io/table3_calcs/

5 Available at: https://emgthomas.shinyapps.io/power_calc/

(23)

Discussion

This report shows that the conclusions of treatment equivalence made by four recently published trials of sepsis in neonates and young infants are not tenable (9–11,19). It became evident that these studies were critically underpowered to establish equivalence of antibiotic treatments among neonates and young infants with bacterial infection, the only infants for whom antibiotic treatments can be effective (Table 3). All four trials had an extremely high probability – 60% at minimum – of concluding treatment equivalence even if the experimental regimen was in reality worse than the standard of care by a clinically important margin of 5%.

Indeed, after accounting for the likely fraction of infants in the trials who did not have bacterial infections, the CIs for the treatment difference among neonates and young infants with bacterial infection may extend well beyond the 5% equivalence margin for all trials.

These corrected CIs may even include values that represent an extremely harmful effect of the experimental treatment relative to standard treatment, with treatment differences possibly higher than 20%. As such, the efficacy of the experimental antibiotic regimens studied requires urgent re-examination.

With current methods for trial recruitment, huge sample sizes of 12 000 per trial arm or more would be required to detect treatment equivalence for infants with bacterial sepsis (Table 4).

Such sample sizes probably are not feasible. The alternative for conducting an equivalence trial is to recruit infants using screening tools with a higher specificity for bacterial infection.

A way forward for this could be to include only neonates and young infants with positive blood culture in the study analysis. As the trials attempt to reflect a real-life situation, and results would only be available later, this would need to be done as a post hoc analysis of culture- positive cases.

Many neonates and young infants with a bacterial infection nevertheless will still have negative blood cultures, so demanding this might be too stringent. An alternative for

conducting a trial would be using a combination of tests such as white cell count, differential white cell count, procalcitonin and c-reactive protein, or serological markers of bacterial infection that would, independently or in combination, increase the likelihood that test- positive infants have a bacterial infection. With a sample size of 5000 per trial arm and sensitivity of 70%, a specificity of at least 96% is needed to detect equivalence with 90%

power. Although more specific screening tools and larger sample sizes will increase trial costs, the potential cost in terms of human lives put at risk by continuing to base sepsis treatment guidelines for neonates and young infants on questionable evidence is certainly much greater.

Another option may be to include a placebo arm in these trials. If the proportion of infants with bacterial infections in the trials is very low, as the analysis above suggests, then the apparent failure rate for young infants treated with antibiotics would likely appear to be very similar to placebo for all treatment arms. One recent trial in Pakistan included a placebo arm in a study of less severely ill young infants with fast breathing only, who were randomized to oral amoxicillin or placebo (26). The trial was stopped because there was a significant difference, with an approximately two-fold higher treatment failure in the placebo group (2.8% versus 5.9%). This indicates there were some infants with bacterial infection in the trial who did benefit from antibiotics. However, including a placebo arm still does not provide a means to estimate the treatment difference among children with bacterial infection, which, as demonstrated in this report, was most likely much larger than the 3% difference reported by this trial. The ethical dilemma of including a placebo arm in trials concerning treatment of serious bacterial infection in infants also emerges.

(24)

It has to be emphasized that the design of equivalence trials should reflect the prevalence of the disease as well as the sensitivity and specificity of the chosen inclusion and exclusion criteria. All four trials presented here were performed with a highly preselected study group.

Based on the chosen exclusion criteria, the infants who were randomized were likely to suffer from mild disease. This observation is consistent with the low mortality rates in all trials and study arms of 1–2%, which are much lower than previously reported mortality rates for neonates and young infants with severe infections ranging from 18.0–64.7% with a pooled case fatality rate of 34.4 (20,24). The preselection of such a study population, as pointed out in this report, is surprising, given that the intention of the trials was to evaluate treatment for neonates and young infants who are ill enough to require hospital admission under normal circumstances. One AFRINEST trial even excluded severely ill children as part of the study protocol.

While the authors of the studies explained the calculation of the sample size in detail, the number of participants in each trial arm was based on presumed prevalence of infection, and the power of the study and sample sizes were not revised or critically assessed once pre-trial assumptions were found to be not applicable.

This report reviewed four large trials cited in the guidelines on treatment of sepsis in young infants when referral is not possible (12). One additional study cited in the guidelines was not considered because it included only 150 infants per trial arm (27) compared to 750 or more for the four trials in this review. The smaller sample size of this study means the statistical limitations will be even greater compared to the trials reviewed. A systematic search was conducted to identify studies published after the guidelines were launched and which might add to the evidence base on the effectiveness of simplified antibiotic treatments administered in the community. This revealed two additional studies. The first compared oral amoxicillin to placebo arm and is discussed above (26). The second was a site-specific reanalysis of data from the multisite AFRINEST trial (28). While these studies were not reviewed in the same depth as in this report, they do not affect the conclusions drawn.

Limitations

This report made a number of assumptions about the possible range of true values for the underlying prevalence of bacterial infections, sensitivity and specificity of recruitment criteria, and apparent treatment-failure rates for infants without bacterial infection (Table 1). The results are sensitive to these assumptions. For prevalence, a conservative range was

considered which included values that are likely to be much higher than the true value. This would have led to underestimates of the uncertainty in the treatment differences estimated by the trials. The SATT trial took blood cultures for 2067 (84%) infants in the trial, of which 1712 were negative, 273 were contaminated and 81 were culture-positive (9). Excluding the

contaminated samples, this suggests about 4.5% of infants in the trial were culture-positive.

Another trial of young infants in Pakistan found a similar result, with pathogens isolated from 11 (5.0%) of 218 blood cultures (27). These are likely to be underestimates of the fraction of children in these trials with bacterial infection, since prior studies suggest almost 75% of blood cultures from children with sepsis fail to recover an organism (29,30). These results therefore are roughly consistent with the assumption shown in Table 1 that between 3% and 22% of children recruited to the trials reviewed in this report had a bacterial infection.

To the authors’ knowledge, currently there are no published estimates of the sensitivity and specificity of the combination of active case-finding and clinical signs used for recruitment to

(25)

the trials. A range of conservative values for the specificity and a moderate value for the sensitivity was therefore used in this report. An online interactive tool¹ is available to allow readers to vary all of the above assumptions.

Throughout this report, it is assumed that the apparent treatment-failure rate for neonates and young infants without bacterial infections is the same in all trial arms. This assumption may not hold if, for example, infants with viral infections recover faster when treated at home.

If so, this would make the trial results appear even more favourable to the experimental oral antibiotic regimens, further increasing the risk of type I error.

Finally, in this report the maximum likelihood estimator of the treatment difference among neonates and young infants with bacterial infections accounting for the presence of young infants without bacterial infection in the trials was presented. As discussed in Annex 1, this maximum likelihood estimator and corresponding CI are likely to be unreliable when the PPV of the trial recruitment criteria is very small. This reflects the fact that the trial data contain little information about the true treatment difference when few infants in the trial have bacterial infections.

Conclusions

This report shows that several published treatment trials for possible serious bacterial infections were unable to demonstrate equivalence of the experimental treatment options with the standard of care. Developing protocols and policies based on these results is therefore not admissible.

Bacterial infections in neonates and young infants remain a serious issue requiring action by the global community to reduce associated mortality. Unfortunately, the methods used by the trials reviewed led to poor-quality evidence and do not progress this endeavour. Other avenues need to be explored, some of which are discussed in the previous papers identifying clinical signs (2,5,6). Pretending to have a less demanding way of treating bacterial infections without knowing whether it is adequate, instead of investing in health systems, is ethically dubious. Efforts have to go in the direction of making sure newborns are born into, and live in, a protective environment that reduces their risk of infection and, when they have a possible bacterial infection, are treated with the best possible care in settings that improve their chances of survival.

(26)

References

1. Hug L, Alexander M, You D, Alkema L. National, regional, and global levels and trends in neonatal mortality between 1990 and 2017, with scenario-based projections to 2030: a systematic analysis. Lancet Glob Health 2019;7(6):e710–20.

2. Weber MW, Carlin JB, Gatchalian S, Lehmann D, Muhe L, Mulholland EK. Predictors of neonatal sepsis in developing countries. Pediatr Infect Dis J. 2003;22(8):711–7.

3. The WHO Young Infants Study Group. Serious infections in young infants in developing countries: rationale for a multicenter study. Pediatr Infect Dis J. 1999;18(10 Suppl.):S4–7.

4. The WHO Young Infants Study Group. Bacterial etiology of serious infections in young infants in developing countries: results of a multicenter study. Pediatr Infect Dis J.

1999;18(10 Suppl.):S17–22.

5. Young Infants Clinical Signs Study Group. Clinical signs that predict severe illness in children under age 2 months: a multicentre study. Lancet 2008;371(9607):135–42.

6. The integrated management of childhood illness (IMCI) chart booklet. Geneva: World Health Organization; 1998.

7. Hamer DH, Darmstadt GL, Carlin JB, Zaidi AK, Yeboah-Antwi K, Saha SK et al. Etiology of bacteremia in young infants in six countries. Pediatr Infect Dis J. 2015;34(1):e1–8.

8. Wilson CB, Nizet V, Maldonado Y, Remington JS, Klein JO. Remington and Klein’s

infectious diseases of the fetus and newborn infant, 8th edition. Philadelphia (PA): Saunders;

2015.

9. Mir F, Nisar I, Tikmani SS, Baloch B, Shakoor S, Jehan F et al. Simplified antibiotic regimens for treatment of clinical severe infection in the outpatient setting when referral is not possible for young infants in Pakistan (Simplified Antibiotic Therapy Trial (SATT)): a randomised, open-label, equivalence trial. Lancet Glob Health 2017;5(2):e177–85.

10. Tshefu A, Lokangaka A, Ngaima S, Engmann C, Esamai F, Gisore P et al. Oral amoxicillin compared with injectable procaine benzylpenicillin plus gentamicin for treatment of neonates and young infants with fast breathing when referral is not possible: a randomised, open-label, equivalence trial. Lancet 2015;385(9979):1758–66.

11. Tshefu A, Lokangaka A, Ngaima S, Engmann C, Esamai F, Gisore P et al. Simplified antibiotic regimens compared with injectable procaine benzylpenicillin plus gentamicin for treatment of neonates and young infants with clinical signs of possible serious bacterial infection when referral is not possible: a randomised, open-label, equivalence trial. Lancet 2015;385(9979):1767–76.

12. Guideline: managing possible serious bacterial infection in young infants when referral is not feasible. Geneva: World Health Organization; 2015 (https://apps.who.int/iris/

handle/10665/181426, accessed 23 June 2021).

13. Dagan R, McCracken GH Jr. Flaws in design and conduct of clinical trials in acute otitis media. Pediatr Infect Dis J. 2002;21(10):894–902.

14. Fox MP, Fox LM. Resolving design problems in equivalency trials. J Pediatr.

2006;149(1):12–6.

15. Marchant CD, Carlin SA, Johnson CE, Shurin PA. Measuring the comparative efficacy of antibacterial agents for acute otitis media: the “Pollyanna phenomenon”. J Pediatr.

1992;120(1):72–7.

(27)

16. Mulholland K, Carlin JB, Duke T, Weber M. The challenges of trials of antibiotics for pneumonia in low-income countries. Lancet Respir Med. 2014;2(12):952–4.

17. Powers JH. Interpreting the results of clinical trials on antimicrobial agents. In: Mandell G, Bennett JD, Dolin R, editors. Mandell, Douglas, and Bennett’s principles and practice of

infectious diseases. Sixth edition. Edinburgh: Elsevier/Churchill Livingstone; 2005.

18. Song JH. Introduction: the goals of antimicrobial therapy. Int J Infect Dis. 2003;7 (Suppl. 1):S1–4.

19. Baqui AH, Saha SK, Ahmed AS, Shahidullah M, Quasem I, Roth DE et al. Safety and efficacy of alternative antibiotic regimens compared with 7 day injectable procaine benzylpenicillin and gentamicin for outpatient treatment of neonates and young infants with clinical signs of severe infection when referral is not possible: a randomised, open-label, equivalence trial.

Lancet Glob Health 2015;3(5):e279–87.

20. Saha SK, Schrag SJ, El Arifeen S, Mullany LC, Shahidul Islam M, Shang N et al. Causes and incidence of community-acquired serious infections among young children in south Asia (ANISA): an observational cohort study. Lancet 2018;392(10142):145–59.

21. Chaurasia S, Sivanandan S, Agarwal R, Ellis S, Sharland M, Sankar MJ. Neonatal sepsis in South Asia: huge burden and spiralling antimicrobial resistance. BMJ 2019;364:k5314.

22. Darmstadt GL, Saha SK, Choi Y, El Arifeen S, Ahmed NU, Bari S et al. Population-based incidence and etiology of community-acquired neonatal bacteremia in Mirzapur, Bangladesh:

an observational study. J Infect Dis. 2009;200(6):906–15.

23. Panigrahi P, Chandel DS, Hansen NI, Sharma N, Kandefer S, Parida S et al. Neonatal sepsis in rural India: timing, microbiology and antibiotic resistance in a population-based

prospective study in the community setting. J Perinatol. 2017;37(8):911–21.

24. Seale AC, Blencowe H, Zaidi A, Ganatra H, Syed S, Engmann C et al. Neonatal severe bacterial infection impairment estimates in South Asia, sub-Saharan Africa, and Latin America for 2010. Pediatr Res. 2013;74(Suppl. 1):73–85.

25. Mulholland K, Hilton S, Adegbola R, Usen S, Oparaugo A, Omosigho C et al. Randomised trial of Haemophilus influenzae type-b tetanus protein conjugate vaccine [corrected] for prevention of pneumonia and meningitis in Gambian infants. Lancet 1997;349(9060):1191–7.

26. Tikmani SS, Muhammad AA, Shafiq Y, Shah S, Kumar N, Ahmed I et al. Ambulatory treatment of fast breathing in young infants aged < 60 days: a double-blind, randomized, placebo-controlled equivalence trial in low-income settlements of Karachi. Clin Infect Dis.

2017;64(2):184–9.

27. Zaidi AKM, Tikmani SS, Warraich HJ, Darmstadt GL, Bhutta ZA, Sultana S et al.

Community-based treatment of serious bacterial infections in newborns and young infants:

a randomized controlled trial assessing three antibiotic regimens. Pediatr Infect Dis J.

2012;31(7):667–72.

28. Lokangaka A, Bauserman M, Coppieters Y, Engmann C, Qazi S, Tshefu A et al. Simplified antibiotic regimens for treating neonates and young infants with severe infections in the Democratic Republic of Congo: a comparative efficacy trial. Matern Health Neonatol Perinatol.

2018;4(1):8.

29. Jacobs RF, Sowell MK, Moss MM, Fiser DH. Septic shock in children: bacterial etiologies and temporal relationships. Pediatr Infect Dis J. 1990;9(3):196–200.

30. Sáez-Llorens X, Vargas S, Guerra F, Coronado L. Application of new sepsis definitions to evaluate outcome of pediatric patients with severe systemic infections. Pediatr Infect Dis J.

1995;14(7):557–61.