ASSESSING THE VALUE OF MODELING AND SIMULATION IN HEALTH CARE: AN EXAMPLE BASED ON INCREASING ACCESS TO STROKE TREATMENT
Abstract
The use of modelling and simulation (M&S) to design, operate and troubleshoot care delivery processes has not been the subject of economic evaluation, so we undertake such an analysis of a modelling exercise to design better delivery: in this case, stroke care. First, the financial impact is assessed, followed by a cost-effectiveness analysis in which the clinical impact is also assessed. Because it is not usually possible to obtain all the costs of modelling, probabilistic sensitivity and threshold analyses are used to explore the uncertainties in the absence of complete information. Threshold analysis is then applied to calculate the upper bound cost and the level of service improvement that would be needed for M&S to represent good value for money.
Keywords
simulation; modelling; economic evaluation; health technology assessment; value assessment; impact analysis
1 Introductions
Several authors have undertaken systematic reviews of M&S in healthcare (Fone et al, 2003; Brailsford et al, 2009; Katsaliaki et al, 2010; Jahangirian et al, 2012) describing how modeling has been applied in healthcare and the main challenges faced. This corpus has been growing at 30 publications per day (Brailsford et al, 2009), but a clear trend is that only a small percentage (around 5-10%, depending upon definition) of papers relate to real-world scenarios (Brailsford et al, 2009; Jahangirian et al, 2012).
Moreover, Jahangirian et al (2012) compared the percentage of conceptual papers and applied papers in three sectors (health, manufacturing and the military) and showed that healthcare is significantly less mature in its adoption than either of the other two. There is also a literature around how to get the best out of healthcare M&S and how to undertake research or studies to maximum effect (Pidd, 2009;
Robinson, 2008; Brailsford et al, 2013).
A consistent message, therefore, is that healthcare modeling and simulation have had a limited impact upon service provision, while a second message is that healthcare modelling tends to be at a markedly different level of maturity in adoption when compared to other sectors. So how does the modelling community build the case for modelling in health? Again, the question is hardly new. Operational Researchers in other sectors have felt the need to justify their methods to host communities and Hollocks (2006), for instance, has done much to explore the value of modelling by identifying specific benefits.
Today, many sectors integrate modelling techniques into their design processes, but healthcare remains in a class of its own, with concepts of evidence that make extensive use of epidemiological tools, trials, meta-analysis and health economics. To explore the case that greater uptake of modeling methods in healthcare might be justified on the grounds of cost-effectiveness, we explore the case for modeling service provision by applying a health economic framework. The contribution of this paper, therefore, is to place the value of healthcare modelling within a framework of evidence that is increasingly familiar to service providers and certainly to policy-makers. In clinical practice this is a well-understood model, but applying it to intangibles such as the knowledge derived from statistical models is more complicated.
Chick and Gans (2009) have proposed a method systematically to address the question of value-for- money: our challenge is to present a practical way to start that journey. For the most part, the literature provides a narrative of benefit rather than an analysis of costs and benefits. Even simple comparisons of whether healthcare M&S represents better value-for-money than not modeling, or than using some other form of planning or design remains essentially unanswered.
Health economists use modelling extensively (Crown et al., 2017), often in the form of Markov models and decision-trees. However, the value of those models to the evaluation process is not itself subject to evaluation. The Cumberland Initiative, with its aim of promoting simulation and modelling of service design and provision, has identified the absence of formal repositories of strong evidence around the utility of modeling as a barrier to uptake. An event held in 2014 – the Festival of Evidence – to gather evidence around M&S methods, produced a report (Brailsford and Klein, 2015) that addresses many of the issues around the nature of evidence and reports many impact stories.
So what is the contribution of this paper? A complete cost-effectiveness analysis of healthcare modelling still lies some way ahead, so we started (Soorapanth and Young, 2015) with one of the few published narratives in which modelling was performed and implemented and the clinical outcomes reported (Monk, 2015) and reported a financial impact analysis of that scenario. In the present paper, we extend this to a full cost-utility analysis using QALYs but because not all the cost data for the modelling and especially for the implementation were available, we have employed sensitivity analysis to establish cost thresholds above which, at least, the modelling could not be considered cost-effective. Some of these thresholds are encouragingly high. Further, we identify the sort of data that will have to be collected in future to undertake such analysis and note the methodological development needed in order to assess the value of the modelling within a design phase or improvement cycle. In doing so, we take the theory a step forward, we identify the data that will need to be collected in future, we identify aspects of theory still to be developed and we present a method that others can adopt.
So then, starting with Soorapanth and Young (2015), summarized in Section 2, this paper goes on in Section 3 to presents our model implementation and key parameter values, while Section 4 presents the results of a case study, analyzed using the economic evaluation framework. We also explore parameter uncertainties and identify the thresholds where cost-effectiveness is likely or unlikely to result from re- designing a process based on modeling. Finally, in Section 5 we reflect on the findings and the extent to which they can be generalized or learned from.
2 Methodology
2.1 Case study in acute stroke care
We will illustrate the method using a case of acute care for Ischemic stroke. Ischemic strokes are caused by a blood clot in an artery leading to the brain. Thrombolysis, also referred to as ‘rt-PA’ or ‘alteplase’, is a clot-busting treatment to dissolve the clot and restore blood flow. It has been licensed for the treatment of acute Ischemic stroke worldwide and is the only licensed treatment for Ischemic stroke in the UK. If eligible for the treatment, patients treated with thrombolysis have significantly less risk of developing a
disability after stroke and better overall health outcomes (Hacke et al, 2004). The benefit of the treatment depends markedly on the time between stroke onset and the start of treatment: onset-to-treatment (OTT) (Lees et al, 2010). It is most effective if given within 3 hours of symptom onset. In UK, the treatment is licensed for up to 4.5 hours (Stroke Association, 2015).
The acute stroke care pathway determines the number of patients that can be reached within 3 hours and therefore the thrombolysis rate, and so several studies have been conducted to determine strategies to optimize this pathway and hence reduce the time to treatment (Monks et al, 2012; Penaloza-Ramos et al, 2014). Specifically, Monks et al (2012) used a discrete-event simulation model to model the emergency stroke care at a district general hospital in the UK. The model was used to explore alternative options to increase the thrombolysis rate and to reduce the time to treatment. Based on the simulation results, recommendations were made to change the care pathway. In the follow-up study (Monks et al, 2015), changes informed by the simulation were implemented in the real-world setting. We note that this degree of follow-up is rare in the literature. The study showed that the changes recommended by the simulation lead to an increase in the thrombolysis rate by 3.1% (95% CI 1.3%-4.7%). During the latter stage of their implementation and evaluation, the hospital achieved the highest thrombolysis rate of 14.5% overall. The study also reported a reduction in the average time from a patient’s arrival at hospital to treatment.
Here we analyze some of the benefits of Monks’ study in economic terms. We start with a straightforward budget impact analysis (Mauskopf et al, 2007) and cost-utility analysis (Drummond et al, 1997; Briggs et al, 2006) of administering thrombolysis to stroke patients. Although the original data in the Monks’ study is cast in terms of who received treatment within the golden hour, we have projected consequential health outcome in terms of quality-of-life gain/loss. This ensures that the model address the long term impact – the degree to which their final outcome is likely to be better because of the faster access to treatment.
2.2 Financial impact of M&S on healthcare administration’s budget
The general framework of the financial impact analysis has been published elsewhere (Soorapanth et al, 2015) and hence will be omitted here. In this section, we will present a summary of the framework and the information specific to the case study as necessary for understanding the sections that follow.
The analysis focuses on the financial impact and takes the perspective of administrators who need to plan and manage health-care budgets. To apply the framework to the case study of stroke care, we take the perspective of a National Health Service (NHS) funded hospital. We conduct the analysis for three time horizons; periods of one, two, and five-years.
Figure 1 presents the general financial impact model. The total population at risk includes all patients arriving at the hospital with acute stroke and the disease condition of interest is ischemic stroke. The size of the population with the disease condition is the total number of patients arriving at the hospital with stroke symptoms multiplied by the percentage of those patients having ischemic stroke. The population with ischemic stroke is then divided into two subgroups. A group of patients receiving proper diagnosis and treatment in a timely manner is called treated subgroup (i.e. receiving thrombolysis), and the remaining patients are considered the untreated subgroup. According to the results from Monks et al (2015), M&S leads to an increase in thrombolysis rate and potentially reduces the time to treatment. The impact of M&S then can be modeled as the increase in the proportion of patients receiving thrombolysis.
Using the thrombolysis efficacy data from Lees et al (2010), the number of patients with and without stroke-related disability (referred to as disabled and not-disabled survivors respectively) can be estimated among treated and untreated subgroups. Lastly, the unit costs of health-care services are estimated for disabled and not-disabled survivors over the time horizon of the analysis. The details about the unit costs of healthcare in the case study are provided in Appendix 1. The total costs of illness is calculated by multiplying the number of disabled and not-disabled survivors by the unit cost of healthcare associated with each type of survivors. The total costs of illness are computed twice, once for the current pathway
and once for the change pathway. The so-called financial impact of M&S is the difference in the total costs of illness between the two pathways (Figure 2).
2.3 Impact of M&S on health outcomes and long-term costs of care
As with the previous analysis, we compare two care pathways; the first represents the current care pathway in which a potential stroke patient arrives at the hospital, goes through diagnostic procedures and, if qualified, receives thrombolysis. The second, the changed pathway, is to repeat the pathway, this time using a redesigned process that relies on M&S to improve the speed of access to the stroke patient.
Figure 3 shows a decision tree model used to compute the cumulative health utility gain or loss and healthcare costs between the two pathways. In the case study, the M&S was used to improve patients’
access to treatment and hence to increase the thrombolysis rate (defined here as the percentage of stroke patients receiving thrombolysis within 4½ hours of the onset of the stroke). The thrombolysis rate is denoted as Ptreated. The Ptreated,current, and Ptreated,changed represent the thrombolysis rates in the current and changed pathways respectively. With or without a treatment, a stroke patient could be in one of four health states; healthy (state 1), minor stroke (state 2), major stroke (state 3), and death (state 4). A stroke patient is considered a not-disabled survivor if he/she is in either the healthy or minor stroke states, and a disabled survivor if in a major stroke state.
Figure 1: Financial impact model.
Figure 2: Computing the financial impact of modeling and simulation.
The probability of being in each health state depends upon whether the patient is treated with thrombolysis: we denote the conditional probability that a treated patient is in health state i (i= 1, 2, 3 and 4) as pitr, and the conditional probability that an untreated patient is in health state i as pintr. We use the quality-adjusted life year (QALY) as a measure of health outcome and the QALY associated with a health
Changed pathway Population
at risk
Current pathway Financial
Impact Model Total cost of illness under current pathway
Financial Impact Model
Total cost of illness under changed pathway
Financial Impact
state i is denotted as Qi. The discounted QALYs over the time horizon T is
∑
t=1
T Qi
(1+r)t−1 . The expected discounted QALYs can be computed as follows:
Figure 3: A decision tree model for computing the total quality-adjusted-life years (QALYs) and total long-term costs of care in the current and change pathways.
1-Ptreated, current Current
pathway
Ptreated, current Patient treated with thrombolysis within 3-4.5 hours of stroke onset
Patient not treated with thrombolysis
Minor stroke (P2tr) Major stroke (P3tr)
Discounted QALYs and long-term costs over
time horizon Healthy (P1tr)
Death (P4tr)
∑Q1, ∑C1
∑Q2, ∑C2
∑Q3, ∑C3
∑Q4, ∑C4
∑C3 Minor stroke (P2ntr)
Healthy (P1ntr)
Death (P4ntr) Major stroke (P3ntr)
∑Q1, ∑C1
∑Q2, ∑C2
∑Q3, ∑C3
∑Q4, ∑C4
Changed pathway
Ptreated, changed Patient treated with thrombolysis within 3-4.5 hours of stroke onset
Patient not treated with thrombolysis
Minor stroke (P2tr) Major stroke (P3tr) Healthy (P1tr)
Death (P4tr)
∑Q1, ∑C1
∑Q2, ∑C2
∑Q3, ∑C3
∑Q4, ∑C4
∑C3 Minor stroke (P2ntr)
Healthy (P1ntr)
Death (P4ntr) Major stroke (P3ntr)
∑Q1, ∑C1
∑Q2, ∑C2
∑Q3, ∑C3
∑Q4, ∑C4 1-Ptreated,
changed
E(discounted QALYs)
¿
(
Ptreated×∑
i=14 pitr∑
t=1T (1+rQi)t−1)
+( (1−Ptreated)
×∑
i=14 pintr∑
t=1T (1+Qri)t−1)
(1)
Where, r is the discount rate, typically 3%-5%. Let Ci be the long-term care costs of a stroke survivor in health state i. We can then calculate the expected discounted long-term care costs by replacing Qi with Ci
in the above equations.
2.4 Cost-utility and threshold analyses
Cost-utility analysis (CUA) is a standard approach for comparing and evaluating multiple health
interventions, based on common measures of effectiveness, such as QALY, and costs associated with the intervention. The results of a CUA would be useful for healthcare administrators in determining which interventions would be more effective and hence allow them the better to allocate limited resources. The details of CUA are available in the health economic literature (Drummond et al., 1997; Briggs et al., 2006) and are omitted here. To make comparisons, the incremental cost-effectiveness ratio (ICER) is calculated as follows:
ICER=∆ TC
∆ TQ (2)
Where in this case,
∆TC = Cost of M&S + E(total costs of illness in the changed pathway) - E(total costs of illness in the current pathway)
(3)
∆TQ=E(discounted QALYs in the changed pathway)
- E(discounted QALYs in the current pathway) (4)
The expected total costs of illness in each pathway includes the cost of thrombolysis treatment, the cost rehabilitation after treatment, and the expected discounted long-term care cost for disabled- and not- disabled survivors (the last cost term computed using Eq.1 with the replacement of Qi by Ci in the equation). The cost of M&S plus the difference in the expected total costs of illness between the changed and current pathways is denoted as ∆TC. The difference in the expected discounted QALYs is denoted as
∆TQ, where the expected discounted QALYs in each pathway is computed using Eq. 1. A healthcare intervention such as M&S is considered cost-saving if the savings in total costs of illness exceed the intervention costs (or ∆TC is negative). An intervention was considered cost-effective if the ICER is less than or equal to £20,000 or £30,000, the thresholds used by UK National Institute for Health and Clinical Excellence (NICE).
One of the key parameters for the evaluation, the cost of M&S, could vary significantly, so we performed a threshold analysis to determine the upper bound for the cost of M&S such that M&S is cost-effective.
By rearranging Eq. 2, the M&S cost threshold can be computed as follows:
M&S cost threshold= λ×∆TQ-
[
E(total costs of illness in changed pathway)-E(total costs of illness in current pathway)]
(5)
Where λ is the cost-effectiveness threshold such as £20,000.
2.5 Probabilistic sensitivity analysis
A precise assessment of the cost-effectiveness of a health intervention is not possible in this case because many model parameters are subject to uncertainty (e.g. lack of data, unrecorded or unpublished data, parameter estimation errors, even limited sample sizes) and so probabilistic sensitivity analysis (PSA) has been used. In the UK, NICE has updated its methods guidance for technology assessment to require the use of PSA (NICE, 2012). We used Monte Carlo sampling to explore the effect of parameter uncertainty on the cost and effectiveness results.
Since the goal of M&S is to improve access to treatment, the impact of M&S on a patient’s health outcome is linked closely to the efficacy of the treatment to improve a patient’s quality of life. In this case, the efficacy of thrombolysis is derived from the clinical trial data with a relatively small sample size. Hence, the outcomes could be quite different in the real-world. To account for such uncertainty, the proportions of patients being in health state i (i=1 to 4) conditional on the receipt of thrombolysis, is sampled from a multinomial distribution with the probability of outcomes shown in Table 4. More detail on how the probabilities of health states are derived is provided in Appendix 1.
Table 1: Model Parameters.
Base- Case
Sampling Distribution (range)
Reference
Number of patients arriving with acute stroke per year at a hospital
1000 - See Appendix 1
Among patients with acute stroke,
percentage of those having ischemic stroke
84% Uniform (67-90%)
Scottish Stroke Care Audit (2015)
Thrombolysis rate in the current pathway 3.8% Uniform (3.5-5%)
Monks et al (2015) Increase in the thrombolysis rate in the
changed pathway
3.1% Uniform
(1.3-4.7%)
Monks et al (2015) Among patients receiving thrombolysis,
percentage of those with OTT of less than 3 hours in both current and changed pathways
50% Uniform
(40-70%)
See Appendix 1
Cost of implementing thrombolysis treatment
(in 2000 GBP)
£1000 Uniform ( ± 50% range)
Sandercock et al.
(2004) Ambulatory rehabilitation (in 1996 GBP)
Not-Disabled Survivor
Disabled Survivor £38
£718
Uniform ( ± 50% range)
Chambers et al (2002); Sandercock et al (2004)
Long-term care cost per year (in 1996 GBP) Not-Disabled Survivor
Disabled Survivor £824
£10,632
Uniform (± 50% range)
Chambers et al (2002); Sandercock et al (2004)
Quality-adjusted life years Healthy
Minor stroke (MRS 2-3)
Major stroke (MRS 4-5)
Death (MRS 6)
1 0.55 0.25 0
-
Uniform (0.5-0.7) Uniform (0-0.3) -
Post et al (2001)
Consumer price index (overall):
year 2016 100.6 -
The Office for National Statistics
year 2000 year 1996
72.5 68.6
(2015) Annual discount rate for long-term costs and
QALYs
3.5% - NICE (2012)
Cost of M&S £30,000 Uniform
( ± 50% range)
See Appendix 1
Costs-Effectiveness Threshold £20,000 NICE (2012)
3 Model implementation and input parameter values
The model was developed in Microsoft Excel (and Visual Basic for Applications macros) with the aims of being transparent, user-friendly, and usable by a range of cross-disciplinary collaborators. Table 1 lists the parameter values and their sampling distributions used in Monte Carlo simulations. The key assumptions and more detail about the model parameters are provided in Appendix 1.
4 Results
4.1 Financial-impact analysis
The model was run for 1000 Monte Carlo simulation iterations. Table 2 presents the simulated estimates of the mean and standard error (SE). Negative values in the “Changed minus Current” column indicate cost-saving. For a hospital with 1,000 stroke cases per year, the changes recommended by the simulation study lead to an average cost-saving of £2,299 when the long-term cost of care is considered for one year.
Ironically, since more patients receive thrombolysis under the improved pathway, the total cost to the hospital of treatment increases. When considering longer time horizons, however, the saving becomes significant as the increased cost of thrombolysis is offset by savings in the long-term cost of care. It is estimated that the use of simulation leads to the saving in the total cost of care of, on average, £33,694, when the long-term cost of care is considered over a two-year period, and £121,653 for a five-year period.
Table 2: Financial impact results (based on 1000 Monte Carlo Simulation Iterations).
Current Pathway Changed Pathway Changed minus Current Mean (SE) total costs of illness
when long-term cost of care projected over 1 year
£3,908,013
(£37,253) £3,905,714
(£36,764)
-£2,299 (£921) Mean (SE) total costs of illness
when long-term cost of care projected over 2 years
£7,399,170 (£72,587)
£7,365,476 (£71,605)
-£33,694 (£1,710) Mean (SE) total costs of illness
when long-term cost of care projected over 5 years
£17,180,124 (£171,677)
£17,058,471
(£169,324) -£121,653
(£4,022)
4.2 Cost-utility analysis
Figure 4 shows scatter plots of ∆TC vs ∆TQ from the Monte Carlo simulations when the long-term costs of care are considered over periods of one, two and five years. For each time horizon, we conducted 1000 iterations. A pathway is said to ‘dominate’ if it leads to health gains (positive ∆TQ) at the same or lower total cost, or to lower total costs (negative ∆TC) for the same or better health gains. Based on the simulation results, the probabilities that the changed pathway dominates are estimated at 0.13, 0.39 and 0.64 for one, two, and five year horizons respectively.
When there is a trade-off between health gain and incremental total cost, the ICER is calculated and compare with the cost-effectiveness threshold λ. When the health gain and total cost both increase (∆TQ and ∆TC are both positive), the M&S is cost-effective if the ICER < λ, but when the health loss and total cost both decrease (∆TQ and ∆TC are both negative), the M&S is cost-effective if the ICER > λ.
According to the results, the probabilities that M&S is at least cost-effective (i.e. either cost-saving or cost-effective) are 0.32, 0.64, and 0.82 for one, two, and five year horizons accordingly.
Table 3 presents the estimated means and their standard errors of ∆TC and ∆TQ. The changed pathway results in the mean increase (SE) in the total QALYs of 0.51 (0.05) per 1000 stroke cases for a year’s time horizon. The mean increases (SE) in total QALYs are 1.75 (0.09) and 5.15 (0.23) for two and five year horizons respectively. The decreases in the total costs of illness are the same as the cost-saving results, with the addition of the cost of M&S (£30,000) in the changed pathway, reported in the financial impact analysis in the previous section.
Table 3: Means and standard errors of ∆TQ and ∆TC.
1 year 2 years 5 years
∆TQ ∆TC ∆TQ ∆TC ∆TQ ∆TC
Mean (SE)
0.51 (0.05)
£27,701 (£921)
1.75 (0.09)
-£3,694 (£1,710)
5.15 (0.23)
-£91,653 (£4,022)
Changed is cost-effective if ICER < λ Current
dominates
Current is cost-effective if ICER < λ
Changed dominates
Figure 4: Scatter plots of ∆TQ (X-axis) vs ∆TC (Y-axis) and the percentages of the results in each quadrant.
4.3 Threshold analysis of the M&S cost and performance benchmark
The results of the CUA depends on the estimate of the cost of the M&S, which may not be reported or may vary significantly from one healthcare problem to another. We therefore perform a threshold analysis
to find the upper cost bound, below which M&S is considered cost-effective. One interpretation of this bound is that it represents how much the M&S is worth, while a more pragmatic approach would seek to design and implement effective modelling below, or well below, that target.
Figure 5 shows the results of the analysis. Using λ of £20,000 and one-year time horizon, M&S must cost less than £13,550 to be cost-effective. If considering a longer time horizon, the upper bound costs increases to £70,783 for a two-year time horizon and £229,000 for a five-year time horizon. These upper bound costs assume that M&S increase thrombolysis rate by 3% (our base-case value).
The improvement of thrombolysis rate is another key unknown parameter that should be explored. From personal conversations with the case study’s lead, this rate could potentially increase to about 10% during one of their implementation phases. Hence, we varied this rate from 1%-10% in our threshold analysis. It is not surprising that when M&S leads to a higher thrombolysis rate increase, the upper bound cost of M&S would be higher. The higher thrombolysis rate results in higher health gain and more total costs saving, and hence increases the worth of M&S.
The results in Figure 5 could also be used to benchmark the performance measure of M&S. That is, we can use the plot to determine the minimum thrombolysis rate increase for a specific cost of M&S. For example, considering a two-year time horizon, if the M&S cost is estimated at £70,000, the thrombolysis rate increase needs to be at least 3% for the M&S to be considered cost-effective.
Figure 5: Threshold analysis for the cost of modeling and simulation, when the costs and benefits are considered over 1-, 2- and 5-year horizons.
5 Limitations of this research and future methodological development
The motivation behind this research has been to develop a method for value-for-money assessment using a framework based on health economics. The first step has been a post-hoc analysis of a piece of
modelling and the reported benefit, which serves as an example of what can be done and an exemplar of what might be possible. However, our study is limited on several fronts and we identify scope for methodological development. Nonetheless, it illustrates that it is possible to articulate, at least partially, what is currently not articulated at all, namely whether modelling represents value for money when designing services in healthcare. We observed the following:
(a) The cost of delivery per 1,000 patients has not been resolved into fixed and variable costs. Here, we are estimating costs and benefits, and many significant uncertainties remain. To undertake the next step, the fixed costs would have to be averaged across regions to account for variations, for instance, in hospital costs. The same applies to variable costs, which will vary regionally with protocols and efficiency in reaching stroke patients.
(b) Redesigning the process for reaching more patients for thrombolysis involves at least three activities:
the modelling, the decisions about what changes to implement, and the implementation of those decision in new processes. Most of those data are not available, so we have estimated the cost of the modelling which is just part of the redesign process. We then estimate the value of the entire redesign process and perform the analysis balancing the cost against benefit. We have estimated the healthcare cost saving (for 1,000 patients) over a 5 year horizon and compared this to the cost of the modelling.
This at least supports the argument that for 1,000 patients over a 5 year horizon, the cost of modelling is worth around 25% of the worth of the realized benefit. It is not a conclusion but it is a step on the way.
(c) Methodological development is needed to attribute the value of modelling in the improvement cycle to the total cost of the process from modelling to implementation. This step is not attempted in this paper and remains for future research. One observation would be that if other non-simulation methods were used to inform the decision to change the care delivery process, the cost of making the decision and implementing would tend to remain the same, so that this approach would still provide a
comparative value-for-money assessment between decision-informing methods (e.g. modelling versus running a design workshop or a Delphi cycle).
(d) We did not have information on the relative cost of running the service before and after – the operational or efficiency gains of the new services against the old. We have therefore ignored these factors which would have to be incorporated for completeness, which will require analysts to gather more data. However, this enables us to focus on two simplifying cases one of which may be adequate to inform decision makers in many cases. Where the primary goal is for improved outcomes and where the cost of running the service is assumed to be the same (as it is in the case we have analyzed), then the analysis presented is appropriate. However, there will also be cases where outcomes are expected to be similar, but the modelling is undertaken for operational reasons – throughput, efficiency, or cost savings. In the latter case, the complexity presented here may not be
needed, and something more akin to a return on investment analysis based on cash flows may suffice.
(e) Finally, this is a proof-of-concept analysis, and studies over many interventions are needed to assess how general these findings are. We plan in future to gather such information from a range of
modelling interventions.
The question in the minds of most decision-makers, however, is usually posed the other way around: how much should one spend on design in order to achieve a particular level of improvement? Again, the exemplar suggests routes forward when working out how to develop robust cases to assess the cost and benefit when spending money to design new services. A robust case must quantify benefit and cost reliably enough to make the return on investment clear for decision-makers but must be simple enough and use data that is readily available, so that it can be constructed quickly and not be contracted out to specialists over weeks or even months.
Clearly, there will always be a difference between the planning model and the final implementation:
planning models tend to be approximate, and most decision-makers recognize the implications of
approximations of all sorts, and a marginal prediction would be treated accordingly. In practice, planning is circumscribed by uncertainty and judgement: good planners working with good decision-makers will account for the main factors to produce models that are neither too simple nor too complicated. It is a matter of judgement and experience, but the case study presented here is a case in point: the key thing was to reach patients more quickly, and this was achieved.
6 Conclusions and discussions
We report an analysis to understand the value of modeling and simulation in economic terms and to start to put healthcare modeling on a similar footing to other health interventions. We believe this to be the first study of this type with supporting data. Although we focus on the modeling and simulation study, the framework presented here could be applied to other types of process improvement or non-simulation modeling as well.
The ultimate aim of our analysis is to be able to justify the use of modeling to clinical teams interested in practicing evidence base medicine. This case study is also an example of how economic analysis may be conducted using studies already in the public domain. The case is however used for demonstrating the method and, as noted, is not necessarily predictive of the value of the average modelling study. Thus, an important aspect of the study has been to take a modeling study reported in terms of short-term to intermediate-term outcomes, such as thrombolysis rate, and to wrap the economic evaluation around it.
The primary questions that emerge from this analysis concern the availability of data related to the modelling itself, the decision-making, and the implementation of the decisions, all together as a
healthcare intervention. Presumably, in this case, since the pathway was changed, the decision-making and the cost of designing and implementing a new process are already covered, so the only new feature would be the cost of the modelling. In this limited case study, we have demonstrated that the modelling shows a cost-benefit improvement. Since stroke morbidity carries significant costs over the years following the stroke, it is not surprising that the modelling demonstrates continuing better outcomes as the period of enquiry is lengthened.
We have also shown the analysis that could be used very early in the design process to search either for ways markedly to improve outcomes or markedly to improve efficiency. In such analysis, the data problem may be overcome by using the model to provide thresholds, such as, the new design will only be cost-effective if it can save an average of 5 minutes in reaching the patient, or, this new design will only be cost-effective if it takes £1,000 or more out of the service cost per patient. Such an approach would allow a targeted search for the savings of time or budget as part of the modelling exercise. It also supports the inverse type of discussion, namely, that if efficiency gains or outcome outcomes quantified as X can be achieved, it is worth spending up to £Y on the modelling. From the present example, it could make a big different to the willingness to pay for some modelling, and encourage stakeholders to be very clear in what was needed from the start.
Acknowledgements
We wish to thank Dr Thomas Monk for informal discussions around this work, and Dr Steffen Bayer for pointing us towards a source of stroke data in Scotland. Both are from the University of Southampton.
We are also grateful to Prof Thomas Archibald, the editor, for discussions around positioning this paper, to Prof Alec Morton for helpful suggestions, and to the anonymous referees who have helped to sharpen the focus of this paper.
Appendix 1: Model assumptions and the derivation of key parameters The Sizes of Population at Risk and Population with Disease Condition
In the case study, the size of the population at risk is the number of stroke cases at a hospital per year.
This number could vary greatly between hospitals. The higher the number the higher the financial impacts of M&S. According to the national statistics, stroke occurs approximately 152,000 times a year in the UK (Stroke Association 2015). There are approximately 150 NHS hospitals. Hence, we assume that the number of strokes at a hospital is approximately 1,000 cases per year. To estimate the size of the population with Ischemic strokes, we multiply the number of stroke per hospital per year and the
percentage of ischemic stroke among stroke patients. According the data from 2015 Scottish Stroke Care Audit report (Scottish Stroke Care Audit, 2015), the percentage of ischemic stroke averages 84% (67%- 90%).
Probability of Patient in Each Health State
The probability that a patient is in health state i, depends on whether or not he/she receives thrombolysis, referred to as the conditional probabilities as pitr, and pi ntr respectively. These probabilities are derived from the published randomized clinical trial data on the efficacy of thrombolysis.
Lees et al (2010) analyzed the efficacy of thrombolysis in treating patients with ischemic stroke and its relationship to onset-to-treatment time (OTT). The health outcomes were measured in terms of the disability levels up to 3 months after the stroke onset. The disability level was measured by a Modified Rankin Scale (MRS) score which ranges from 0 to 6. The score of 0 or 1 indicate no or not-significant disability, 2 indicates slight disability but able to look after own affairs without assistance, 3 to 5 indicate moderate to severe disability and nursing care required, 6 indicates death.
Based on the results from Lees et al (2010), the percentages of patients by MRS scores can be calculated based on the onset-to-treatment time (OTT) as presented in Table 2. Since the data on the OTT is not available in the Monk (2015) study, we assume that 50% of patients have OTT of less than 3 hours and
50% have OTT between 3-4.5 hours. We assume these percentages in both changed and current pathways.
Table 4: Probability of each health state approximated from the percentage of patients by modified rankin scale (MRS) score
Healthy (MRS 0-1)
Minor stroke (MRS 2-3)
Major Stroke (MRS 4-5)
Death (MRS 6) Patients receiving thrombolysis within
3 hours of the stroke onset (OTT = 0-3
hours) 42.11% 20.72% 19.41% 17.76%
Patients receiving thrombolysis within 3-4.5 hours of the stroke onset (OTT =
3-4.5 hours) 44.74% 23.68% 21.05% 10.53%
Patients not receiving thrombolysis
(placebo) 35.53% 26.32% 28.29% 9.87%
Unit Costs of Healthcare Services
Table 1 also shows the cost data (in their original price year) used in our analysis. All cost data are given in Great Britain Pounds (GBP) and are adjusted to a price year of 2016 using consumer price indices (The Office of National Statistics, 2016). Costs included in the model are the costs of implementing thrombolysis, the costs of rehabilitation, and the costs of long-term care for not-disabled or disabled survivors. The cost of acute care of stroke is not included in the model. Since we assume the same number of patients with stroke arriving at a hospital in the current and changed pathways, the total costs of acute care of stroke would be the same in both scenarios. The costs of long-term care for disabled and non-disabled patients are computed for 1-, 2- and 5- year time horizozns and are discounted by 3.5%
annual rate (as suggested by NICE guideline).The drug costs of thrombolysis were estimated at £480 (Sandercock et al, 2004). The additional resources required to deliver thrombolysis vary significantly across the UK (Sandercock et al, 2004) and a good estimate for the total cost of implementing thrombolysis is not available. The estimated total cost of £1000 from Sandercock et al (2004) is used in our analysis.
The cost of M&S
A robust analysis would require information on the cost of the modelling and then the cost of the decision-making and implementation that was informed by the modelling. In the case we have cited, there was an indication of the amount of time of the research fellow used (although no grade or salary band information). However, we have used that, together with publicly available university pay scales and estimates of the amount of supervision to estimate the cost of the modelling. We have also had to estimate the amount of clinical support and the salary of the stroke consultant (which we took from a – recent advert). This gives us an estimated figure. We use a broad +/-50% range in the sensitivity analysis.
These figures are given in Table 5.
Estimating the effort needed to make a decision on the back of a modelled scenario and then to undertake the service development to implement the improvement is much more difficult. Whether redesign is unique to modelling-based decisions, or will happen at some level, whatever reasons lead to the change, is also moot. If modelling is the only unique element of service change, then, of course, the other costs may well appear in both arms of the analysis and so present less of a problem.
Until robust data is available, it is probably better either to use a sensitivity analysis with a broad range of costs, or else to set the model up in such a way that the main output is a cost threshold below which the decision and redesign based on M&S becomes cost-effective.
Table 5 The cost of modeling and simulation (all costs are in 2016 GBP)
Job Title
Salary:
Mid- point (Range )
Annual Cost*
FTE Duration Total Cost Estimate
Mid-Range Research Fellow 38,183 76,366 0.40 0.5 15,273
Supervisor (Assoc. Prof) 57,674 115,348 0.15 0.5 8,651
Stroke Physician 102,500 205,000 0.02 0.5 2,050
License fee 1,000
Total £26,974
* Annual cost = (Mid-point of the salary range + 100% overhead)×12
REFERENCES
Brailsford S C, Harper P R, Patel B and Pitt M (2009). An analysis of the academic literature on simulation and modeling in healthcare. Journal of Simulation 3: 130-140.
Brailsford S C, Bolt T, Bucci G, Chaussalet T, Connell N A D, Harper P R, Klein J H, Pitt M and Taylor M A (2013). Overcoming the barriers: a qualitative study of simulation adoption in primary care trusts in the NHS. Journal of the Operational Research Society 64 (2): 157-168.
Brailsford S C and Klien J H (2015). The value of modeling and simulation in healthcare: a review of the evidence and some possible ways forward. Technical report. The Cumberland Initiative, February 2015. Accessed Mar 29, 2017. http://cumberland-initiative.org/2015/07/21/festival-of-evidence- reports-now-to-download/ date.
Briggs A H, Claxton K, and Sculpher M J (2006). Decision modelling for health economic evaluation.
Oxford University Press, USA.
Chambers M G, Koch P, and Hutton J (2002). Development of a decision-analytic model of stroke care in the United States and Europe. Value Health 5: 82-97.
Chick S E and Gans N (2009). Economic analysis of simulation selection problems. Management Science. 55: 421-437.
Crown, W., N. Buyukkaramikli, P. Thokala, A. Morton, M. Y. Sir, D. A. Marshall, J. Tosh, W. V. Padula, M. J. Ijzerman, P. K. Wong and K. S. Pasupathy (2017). Constrained Optimization Methods in Health Services Research-An Introduction: Report 1 of the ISPOR Optimization Methods Emerging Good Practices Task Force. Value in health : the journal of the International Society for Pharmacoeconomics and Outcomes Research 20(3): 310-319.
Drummond M F, O'Brien B, Stoddart G L, Torrance, G W (1997). Methods for the economic evaluation of health care programmes. 2nd edn, Oxford University Press, UK.
Fone D, Hollinghurst S, Temple M, Round A, Lester N, Weightman A, Roberts K, Coyle E, Bevan G, Palmer S (2003). Systematic review of the use and value of computer simulation modeling in population health and health care delivery. Journal of Public Health Medicine. 25 (4): 325-35.
Hacke W, Donnan G, Fieschi C, Kaste M, von Kummer R, Broderick J P, Brott T, Frankel M, Grotta J C, Haley E C Jr, Kwiatkowski T, Levine S R, Lewandowski C, Lu M, Lyden P, Marler J R, Patel S, Tilley B C, Albers G, Bluhmki E, Wilhelm M, Hamilton S; ATLANTIS Trials Investigators;
ECASS Trials Investigators; NINDS rt-PA Study Group Investigators (2004). Association of outcome with early stroke treatment: pooled analysis of ATLANTIS, ECASS, and NINDS rt-PA Stroke Trials. Lancet 363: 768-774.
Harper P R (2002). A framework for operational modeling of hospital resources. Health Care Management Science 5: 165-173.
Hollocks, B W (2006). Forty years of discrete-event simulation—a personal reflection. Journal of the Operational Research Society 57 (12): 1383–1399.
Jahangirian M, Naseer A, Lampros S, Young T, Eldabi T, Brailsford S, Patel B and Harper P (2012).
Simulation in health-care: lessons from other sectors. Operational Research 12: 45-55.
Jun J B, Jacobson S H, and Swisher J R (1999). Application of discrete-event simulation in health care clinics: a survey. Journal of the Operational Research Society 50 (2): 109-123.
Katsaliaki K and Mustafee (2010). Applications of simulation within the healthcare context. Journal of the Operational Research Society 62 (8): 1431-1451.
Lees K R, Bluhmki E, von Kummer R, Brott T G, Toni D, Grotta J C, Albers G W, Kaste M, Marler J R, Hamilton S A, Tilley B C, Davis S M, Donnan G A, Hacke W; ECASS, ATLANTIS, NINDS and EPITHET rt-PA Study Group Investigators (2010). Time to treatment with intravenous alteplase and outcome in stroke: an updated pooled analysis of ECASS, ATLANTIS, NINDS, and EPITHET Trials. Lancet. 375: 1695-1703.
Mauskopf J A, Sullivan S D, Annemans L, Caro J, Mullins C D, Nuijten M, Orlewska E, Watkins J, and Trueman P (2007). Principles of good practice for budget impact analysis: report of the ISPOR task force on good research practices--budget impact analysis. Value Health 10: 336-347.
Monks T, Pitt M, Stein K, and James M (2012). Maximizing the population benefit from thrombolysis in acute ischemic stroke: a modeling study of in-hospital delays. Stroke. 43: 2706-2711.
Monks T, Pearson M, Pitt M, Stein K, and James M A (2015). Evaluating the impact of a simulation study in emergency stroke care. Operations Research in Health Care. 6: 40-49.
National Institute for Health and Clinical Excellence (NICE) (2012). Methods for the Development of NICE Public Health Guidance. 3rd edn. Accessed May 5, 2015. www.nice.org.uk.
The Office for National Statistics (2015). Consumer price indices: 1988 to 2015. Accessed May 5, 2015.
www.ons.gov.uk.
Penaloza-Ramos M C, Sheppard J P, Jowett S, Barton P, Mant J, Quinn T, Mellor R M, Sims D, Sandler D, McManus R J; Birmingham and Black Country Collaborations for Leadership in Applied Health Research and Care Investigators (2014). Cost-effectiveness of optimizing acute stroke care services for thrombolysis. Stroke 45: 553-562.
Pidd M (2009). Tools for thinking. 3rd edn, John Wiley & Sons Ltd, Chichester, UK.
Post P N, Stiggelbout A M, Wakker P P (2001). The utility of health states after stroke: a systematic review of the literature. Stroke 32: 1425-1429.
Robinson S (2008). Conceptual modeling for simulation part I: definition and requirements. Journal of the Operational Research Society 59 (3): 278–290.
Sandercock P, Berge E, Dennis M, Forbes J, Hand P, Kwan J, Lewis S, Lindley R, Neilson A, and Wardlaw J (2004). Cost-effectiveness of thrombolysis with recombinant tissue plasminogen activator for acute ischemic stroke assessed by a model based on UK NHS costs. Stroke. 35:
1490-1497.
Soorapanth S and Young T (2015) Evaluating the financial impact of modeling and simulation in healthcare: proposed framework with a case study. In: Proceedings of the 2015 Winter Simulation Conference, pp. 1492-1502.
Scottish Stroke Care Audit (2015). 2015 National Report: Stroke Services in Scottish Hospitals. Accessed May 5, 2015. http://www.strokeaudit.scot.nhs.uk/Reports/Reports.html
Stroke Association (2015). State of the Nation: Stroke Statistics, January 2015. Accessed May 5, 2015.
www.stroke.org.uk.
