5. CASE STUDIES ON TRIAL APPLICATION OF THE KIND APPROACH
5.2. Generic case studies of hypothetical NESs
5.2.2. Comparison of five hypothetical NESs
This section presents an example of the comparative evaluation of five hypothetical NESs, demonstrating the comparative evaluation procedure in accordance with the KIND recommendations. This example illustrates the basic steps that need to be completed for the comparison of real systems: the formation of a performance table, the evaluation of weighting factors and other model parameters, the performance of evaluations and sensitivity analysis, and the interpretation of the evaluation results.
Within the study, for illustrative purposes, it is assumed that a set of 15 unnamed KIs and 15 secondary (additional) indicators are given, each of which can be evaluated either in scores or in natural (physical) units.
These performance indicators can be grouped into six evaluation areas and will be used to test the NES comparative evaluation procedure on hypothetical NESs. Figure 5.1 demonstrates the arrangement of the indicators in a hierarchical structure that is known as an objectives tree.
A performance table was formed at random and model parameters were selected in line with the recommendations made in the previous section:
(a) A set of 15 KIs and 15 SIs (all indicators need to be minimized) was used within the first and second tiers of the comparative evaluation procedure, respectively.
(b) Linearly decreasing single-attribute value functions were applied with the end points defined from the performance table.
(c) The indicators were evaluated using 5 and 10 point scoring scales within the first and second tiers of the comparative evaluation procedure, respectively.
5.2.2.1. Performance table
The values of all the KIs for the considered NESs are presented in the performance table (Table 5.1) for the problem. In accordance with the assumptions regarding the objectives to be achieved with each indicator (all indicators need to be minimized), an indicator value score of 1 is the best possible value, while an indicator value score of 5 is the worst one. The performance table is presented in the form of a value path in Fig. 5.2. This figure shows variations in the values of all KIs for the entire set of NESs and demonstrates how an improvement in a certain KI value reduces the other KI values due to the transition from one NES to another.
In the most evident form, the procedure for the comparative evaluation of NESs may be formulated by ranking NESs according to the closeness of each NES to the upper limit of a graph from a set of KIs. This closeness will depend on the metric type, which is a measure of the distance to the upper limit.
5.2.2.2. Weighting factors
The only assumption that was made to evaluate the weight values was that at each level of the objectives tree, the significance or importance of all the high level objectives, the areas of evaluation and the indicators are identical. This option assumes that the experts and the decision maker agree that high level objectives characterized by aggregated goals of cost, performance and acceptability are equally important.
Each evaluation area within the high level objectives (cost (economics), performance (waste management,
proliferation resistance, environment, country specifics) and acceptability (maturity of technology)) was assigned FIG. 5.1. The objectives tree.
an equal weighting factor. The sum of all the weighting factors for each of these areas was also required to be equal to unity.
At the final level (the level of the KIs), the weighting factors for each indicator included in the corresponding evaluation area were assumed to be equal. The final weighting factors calculated in accordance with the described assumption for each indicator are shown in Table 5.1.
5.2.2.3. Ranking results and interpretation of the results
The alternative ranking results are shown in Fig. 5.3 and the scores of the multi-attribute value functions are presented in Table 5.1. Of all the options considered, the NES-3 is the most preferred alternative. The alternatives NES-1 and NES-5 are close to each other according to the scores of the multi-attribute value functions and take the second and third places, respectively. The alternatives NES-4 and NES-2, despite the slight difference in the values of the multi-attribute value functions, may be considered as indistinguishable and will be evaluated further.
To interpret the ranking results, the multi-attribute value functions may be decomposed into individual components in accordance with the specified structure of the objectives tree. Depending on the depth and level of detail the interpretation requires, this decomposition may be performed at different levels in accordance with the structure of the objectives tree.
Table 5.2 shows the values for the high level objectives (cost, performance and acceptability), which determine the final scores. Based on this decomposition, the superior attractiveness of alternative 3 (NES-3) may be explained in the following manner: despite the mediocre scores for the high level objectives acceptability and cost, this alternative provides the best score for the high level objective performance. For this reason, this alternative takes the first place. The ranking results for the other alternatives can be interpreted in a similar way.
FIG. 5.1. The objectives tree.
TABLE 5.1. PERFORMANCE TABLE High level
objectives Evaluation areas KIs Weights NES performance
NES-1 NES-2 NES-3 NES-4 NES-5
Cost Economics KI-1 0.167 1 2 3 2 4
KI-2 0.167 2 4 2 1 2
Performance
Waste management KI-3 0.083 5 1 1 3 3
Proliferation resistance
KI-4 0.028 2 3 1 4 3
KI-5 0.028 5 5 3 3 4
KI-6 0.028 4 5 3 2 4
Environment KI-7 0.083 3 4 1 2 3
Country specifics
KI-8 0.017 4 3 4 3 4
KI-9 0.017 3 4 3 2 3
KI-10 0.017 3 4 2 3 4
KI-11 0.017 2 2 4 3 5
KI-12 0.017 2 4 2 4 2
Acceptability Maturity of technology
KI-13 0.111 4 2 4 4 1
KI-14 0.111 4 3 3 5 3
KI-15 0.111 3 4 3 5 4
FIG. 5.2. Value path.
TABLE 5.2. HIGH LEVEL AGGREGATED OBJECTIVE SCORES
Cost score Performance score Acceptability score Overall score
NES-1 0.278 0.106 0.167 0.55
NES-2 0.111 0.126 0.241 0.478
NES-3 0.167 0.288 0.222 0.677
NES-4 0.278 0.206 0 0.483
NES-5 0.111 0.127 0.278 0.516
FIG. 5.3. Ranking results.
TABLE 5.3. SECOND LEVEL SCORES
Economics Waste management Proliferation
resistance Environment Country specifics Maturity of technology
NES-1 0.278 0 0.028 0.028 0.05 0.167
NES-2 0.111 0.083 9.259 × 10−3 0 0.033 0.241
NES-3 0.167 0.083 0.074 0.083 0.047 0.222
NES-4 0.278 0.042 0.056 0.056 0.053 0
NES-5 0.111 0.042 0.032 0.028 0.025 0.278
In this case, if the decision maker and experts want a more detailed explanation of the reasons that led to the dominance of one alternative over another, the multi-attribute value function needs to be decomposed into components that correspond to each evaluation area. Table 5.3 shows the scores for alternatives for each of the evaluation areas. In particular, it is clear from these figures that NES-3, in almost all areas within the category of performance, surpasses all of its competitors, which ultimately leads to the conclusion that this alternative is the best one.
TABLE 5.1. PERFORMANCE TABLE High level
objectives Evaluation areas KIs Weights NES performance
NES-1 NES-2 NES-3 NES-4 NES-5
Cost Economics KI-1 0.167 1 2 3 2 4
KI-2 0.167 2 4 2 1 2
Performance
Waste management KI-3 0.083 5 1 1 3 3
Proliferation resistance
KI-4 0.028 2 3 1 4 3
KI-5 0.028 5 5 3 3 4
KI-6 0.028 4 5 3 2 4
Environment KI-7 0.083 3 4 1 2 3
Country specifics
KI-8 0.017 4 3 4 3 4
KI-9 0.017 3 4 3 2 3
KI-10 0.017 3 4 2 3 4
KI-11 0.017 2 2 4 3 5
KI-12 0.017 2 4 2 4 2
Acceptability Maturity of technology
KI-13 0.111 4 2 4 4 1
KI-14 0.111 4 3 3 5 3
KI-15 0.111 3 4 3 5 4
FIG. 5.2. Value path.
TABLE 5.2. HIGH LEVEL AGGREGATED OBJECTIVE SCORES
Cost score Performance score Acceptability score Overall score
NES-1 0.278 0.106 0.167 0.55
NES-2 0.111 0.126 0.241 0.478
NES-3 0.167 0.288 0.222 0.677
NES-4 0.278 0.206 0 0.483
NES-5 0.111 0.127 0.278 0.516
FIG. 5.3. Ranking results.
TABLE 5.3. SECOND LEVEL SCORES
Economics Waste management Proliferation
resistance Environment Country specifics Maturity of technology
NES-1 0.278 0 0.028 0.028 0.05 0.167
NES-2 0.111 0.083 9.259 × 10−3 0 0.033 0.241
NES-3 0.167 0.083 0.074 0.083 0.047 0.222
NES-4 0.278 0.042 0.056 0.056 0.053 0
NES-5 0.111 0.042 0.032 0.028 0.025 0.278
5.2.2.4. Sensitivity analysis
A sensitivity analysis has been performed for the weighting factors and shapes of the single-attribute value functions. In general, it should be noted that the selected model parameters, in accordance with the recommendations presented in the previous section, have shown the stability and robustness of the ranking results.
(a) Impact of single-attribute value function forms
Sensitivity analysis is a tool for evaluating the impact of assigned single-attribute value functions on the final results (ranks of alternatives). To demonstrate the sensitivity of the ranking results to the forms of the single-attribute value functions, a special statistical analysis was carried out by using randomly generated single-single-attribute value functions and building a statistical rank distribution for each alternative considered. The most likely ranks for each alternative and their statistical distributions are shown in Fig. 5.4. The analysis has shown that the most likely ranks of alternatives coincide with the ranks obtained using linear single-attribute value functions. The analysis confirms the reduced sensitivity of the ranking results to the form of the single-attribute value functions, which was possible through the selection of appropriate assumptions regarding the model parameters.
(b) Weight sensitivity
A weight sensitivity analysis provides an understanding of the influence of the assigned weights on the ranking of the alternatives. In the framework of the linear weight approach to the weight sensitivity analysis, experts have to select an indicator for the sensitivity analysis and analyse how the ranking alternatives will change with the weighting factor varying from 0 to 1 (other weights are automatically changed proportionally, keeping the weight sum equal to unity). Such analysis can be carried out for the weighting factors at each level of the objectives tree: the high level aggregated objectives, the evaluation area level and the individual indicator level. In the present study, this approach was applied for the high level of the objectives tree for illustration.
Figure 5.5 shows the overall scores (multi-attribute value functions) for each alternative as a function of the corresponding weighting factor value and the base case value for this weighting factor. Based on this information, the ranks of alternatives may be identified for different weighting factor values, while the weighting factor areas may also be obtained, providing the same ranking result. Based on this analysis, it may be concluded that the ranking results are not very sensitive to the weighting factor values and may be considered as stable, because they satisfy the following sensitivity test: a 10 per cent change in all weighting factors does not affect the ranks of the alternatives.
FIG. 5.4. Rank distributions.
(a) (b)
(c)
FIG. 5.5. Linear weights approach to weight sensitivity analysis. (a) Cost weighting factor; (b) performance weighting factor;
(c) acceptability weighting factor.
(c) Differentiation of undistinguished alternatives
This section illustrates the second stage in the implementation of the two tier comparative evaluation procedure, when alternatives that are indistinguishable using a set of KIs are differentiated using a set of SIs. It was shown above that NES-2 and NES-4 are indistinguishable alternatives. In this situation, further differentiation seems necessary and can be performed by using a set of SIs. A comparison of these two options was performed in accordance with the recommendations made in the previous section: the 15 SIs were involved in the procedure, with each indicator being evaluated on a 10 point scoring scale, and linearly decreasing functions defined on local domains were chosen as single-attribute value functions. A broader scoring scale was used to compare the two alternatives because it provides a subtler differentiation of the alternatives, reducing the likelihood of obtaining similar values for indicators, which would lead to the elimination of corresponding information from the evaluation.
Table 5.4 corresponds to the performance table for this problem.
The ranking results based on the above mentioned assumptions are presented in Table 5.5. Alternative NES-2 is the most attractive one, while alternative NES-4 is less attractive. The scores of the multi-attribute value functions are 0.522 (NES-2) and 0.367 (NES-4), which allows one to consider these alternatives as distinguishable ones. As FIG. 5.4. Rank distributions.
(a) (b)
(c)
FIG. 5.5. Linear weights approach to weight sensitivity analysis. (a) Cost weighting factor; (b) performance weighting factor;
(c) acceptability weighting factor.
shown in Table 5.5, the scores on the aggregated high level objective cost are identical for both systems. At the same time, the scores for NES-2 on the aggregated high level objectives performance and acceptability are superior to the corresponding scores for NES-4. This indicates that the former alternative is the most attractive option.
A comparison of the aggregated scores shown in Table 5.6 for all evaluation areas provides an opportunity to understand the contribution of these areas to the values for the high level objectives. Since the high level objectives cost and acceptability only include one evaluation area (‘economics’ and ‘maturity of technology’, respectively), the corresponding values of the aggregated scores for the high level objectives and evaluation area levels of the objectives tree are the same. Regarding the corresponding areas’ contribution to the high level objective performance, it should be noted that the first alternative is a less attractive option in terms of ‘proliferation resistance’, while this alternative prevails over its competitor in terms of ‘environment’ and ‘country specifics’. For the ‘waste management’ area, the scores are equal to zero for both alternatives because they have the same scores for all of the indicators included in this evaluation area.
A sensitivity analysis for the case of the two compared alternatives usually demonstrates the stability of the ranking results over a wide range of possible model parameters. The sensitivity analysis performed for the example considered confirms this thesis. In the case where two alternatives with domains for single-attribute value functions specified from the performance table are compared, the ranking results are not sensitive to the form of the single-attribute value functions.
The results for the application of the linear weights approach for high level objective weights are shown in Fig. 5.6. For all possible values of the weighting factors, the first alternative looks more attractive than the second one. Thus, it may be concluded that the ranking results are stable and robust to possible changes in weighting factor values.
The performed study showed that in cases where the MAVT method is correctly used (i.e. taking into account major requirements for the NES comparative evaluation procedure), it will provide an identification of the merits and demerits of the nuclear technological options being compared and their performance rankings based on the judgements and preferences of experts and decision makers. These case studies have also demonstrated the basic TABLE 5.4. PERFORMANCE TABLE
High level
objectives Evaluation areas SIs Weights NES performance
NES-2 NES-4
Cost Economics SI-1 0.167 3 9
SI-2 0.167 6 1
Waste management SI-3 0.083 6 6
Proliferation resistance
SI-4 0.028 9 7
SI-5 0.028 9 2
SI-6 0.028 4 4
Performance Environment SI-7 0.083 2 9
Country specifics
TABLE 5.5. HIGH LEVEL AGGREGATED OBJECTIVE SCORES
Cost score Performance score Acceptability score Overall score
NES-2 0.167 0.133 0.222 0.522
NES-4 0.167 0.089 0.111 0.367
TABLE 5.6. SECOND LEVEL SCORES
Economics Waste
management Proliferation
resistance Environment Country specifics Maturity of technology
NES-2 0.167 0 0 0.083 0.05 0.222
NES-4 0.167 0 0.056 0 0.033 0.111
(a) (b)
(c)
FIG. 5.6. Linear weights approach to weight sensitivity analysis. (a) Cost weighting factor; (b) performance weighting factor;
(c) acceptability weighting factor.
TABLE 5.4. PERFORMANCE TABLE High level
objectives Evaluation areas SIs Weights NES performance
NES-2 NES-4
Cost Economics SI-1 0.167 3 9
SI-2 0.167 6 1
Waste management SI-3 0.083 6 6
Proliferation resistance
SI-4 0.028 9 7
SI-5 0.028 9 2
SI-6 0.028 4 4
Performance Environment SI-7 0.083 2 9
Country specifics
SI-8 0.017 6 3
SI-9 0.017 1 8
SI-10 0.017 3 6
SI-11 0.017 2 8
SI-12 0.017 6 5
Acceptability Maturity of technology
SI-13 0.111 7 5
SI-14 0.111 6 7
SI-15 0.111 6 7
TABLE 5.5. HIGH LEVEL AGGREGATED OBJECTIVE SCORES
Cost score Performance score Acceptability score Overall score
NES-2 0.167 0.133 0.222 0.522
NES-4 0.167 0.089 0.111 0.367
TABLE 5.6. SECOND LEVEL SCORES
Economics Waste
management Proliferation
resistance Environment Country specifics Maturity of technology
NES-2 0.167 0 0 0.083 0.05 0.222
NES-4 0.167 0 0.056 0 0.033 0.111
(a) (b)
(c)
FIG. 5.6. Linear weights approach to weight sensitivity analysis. (a) Cost weighting factor; (b) performance weighting factor;
(c) acceptability weighting factor.
steps to be completed within the NES comparative evaluation procedure in the framework of the KIND project, which may be considered as a possible template for the representation and interpretation of calculation results that may be implemented in the comparison of real systems.