The INPRO collaborative project KIND has done the following:
— Developed guidance on the elaboration of KI sets for comparative evaluation of the performance and sustainability of different NES options;
— Adapted and elaborated advanced methods of expert judgement aggregation and sensitivity/uncertainty analysis to enable effective comparative evaluation of such options;
— Conducted case studies on trial application of the KIND approach;
— Explored the potential of the developed approach in application to both INESs and other NES options, including those of interest to technology users and newcomer countries.
The project has identified a number of specific considerations regarding the development of a set of KIs for comparative energy system evaluations involving NESs. When developing a KI set it is important to do the following:
— Identify the target audience for the evaluation and select KIs that will be understandable to that target audience.
— Determine the main areas to be evaluated and then assign KIs for those areas. The main areas could be based on what is important to different stakeholders in the evaluation and are to include all major perspectives.
— Use one or several KIs for each main area to be considered in the evaluation (e.g. safety, cost), as appropriate.
However, the total number of KIs must be limited, preferably to fewer than 20.
— Develop KIs with full consideration of which data are available for all of the options using the least detailed option as the basis. If using a quantitative KI, first verify that quantitative data are available for every option to be evaluated. If using a qualitative KI, verify that unbiased experts are available to provide the qualitative input.
— Not attempt to address every nuance of the system with a KI. It is to be remembered that these are ‘key’
indicators intended to address only the key elements of the system and the key issues to be considered in an evaluation. Acknowledge which less important aspects are being left out to keep the evaluation focused.
— Note that if large numbers of experts are involved in one area of evaluation, it may result in excessive number of KIs in that area, since they know their domain in detail, and will opt to have numerous KIs to measure every aspect. This does not necessarily improve the overall evaluation as the clarity of priorities can be lost in the ‘forest’ of KIs.
— Note that another approach to avoid an excessive number of KIs is to perform a multi-stage evaluation, where the first stage reduces the total number of options while the second stage goes into more detail in a couple of areas where it is difficult to separate the remaining options using the original KIs.
— Fully describe and document each KI, so that everyone involved can understand what is included, and how it is to be evaluated.
— Develop KIs that are as independent as possible from each other to avoid double counting the same aspects of the system.
— Put as much effort into developing the value/utility functions and weights as has gone into developing the KIs. These aspects of the KIND approach have just as much impact on evaluation results.
— Use sensitivity analyses to explore the robustness of the initial evaluation results. This acknowledges the uncertainty in the inputs and in the KIND approach and reveals where apparent differences between energy systems are tangible and not just within the margin of the uncertainty.
A key component of the KIND approach is the MCDA method, which is not a tool to suggest the ‘best option’ for all situations but a tool for finding a compromise or a trade-off among options for given circumstances (combining expert judgements on subject matter, as well as decision makers’ preferences).
The selection and justification of an adequate MCDA method and the most appropriate method parameters are important points that depend on a decision support problem. Another important issue to be carefully examined within sensitivity studies is the possible impact of uncertainty in input data on the alternative ranks.
The selection of the MCDA method that is most suitable for a given problem is a separate task that needs to be based on examination of the problem context and the information provided by the subject matter experts and a decision maker. Thus, depending on whether a detailed evaluation of all indicators without uncertainty will be performed, or instead the consideration will be limited to qualitative judgements regarding indicator values, different methods need to be used. For example, the MAVT, TOPSIS and PROMETHEE methods are more appropriate in the former case, whereas the AHP method is more suitable in the latter. If it is necessary to consider the uncertainty in the indicator values, the MAUT method is more suitable than, for instance, MAVT or others.
To structure a dialogue based on MCDA methods, it is necessary to describe all of the assumptions that have been made accurately to avoid misunderstandings in interpretation of the results. These assumptions then need to be analysed within a sensitivity analysis. A detailed explanation of the selected model assumptions in the case study report is an area of responsibility for experts carrying out MCDA based studies.
For KIs, it is only necessary to select those parameters or characteristics of an NES whose variation will result in tangible, well understood and interpreted changes of user oriented qualities. In this sense, comparative evaluation is complementary to the assessment, in which only full compliance with the defined set of criteria is required. Comparative evaluation is deemed to support the selection of an NES that is preferable for a particular user from a range of alternatives that otherwise meet (or are anticipated to meet) all regulatory and other assessment criteria.
Properly organized case studies based on the KIND approach represent a complex endeavour, not only operating formally with a set of analytical tools, but also helping to understand explicitly what could support the decision making process.
Experts are responsible for the quality of MCDA based studies conducted to support the establishment of a certain recommendation and inform a decision maker about the merits and demerits (costs, benefits and risks) of the options considered. The decision maker takes responsibility for the final decision, taking into account the results of MCDA based studies, and this decision needs to be accepted with a clear recognition and acceptance of the associated risks.
When correctly implemented, different theoretical frameworks applied in various MCDA methods yield similar ranking results. The KIND case studies demonstrated that, despite some differences in the ranks, the results are generally consistent and harmonized even when based on different methods. The numerical examples considered have identified that both elementary and more sophisticated judgement aggregation methods may be used effectively for the multi-criteria comparative evaluation of NESs in both technology and scenario based cases.
To avoid a black box effect, transparent and simplified MCDA methods (such as MAVT or simple additive weighting) are recommended for the objective of the KIND approach, given its application to less mature innovative technologies and proven nuclear and non-nuclear technologies, but this in no way restricts experts from applying other MCDA methods. Moreover, the scope of the MCDA methods does not need to be limited to those considered in this report. Experts may express their preference for other ranking/aggregation methods, but in this case it is essential to verify their applicability and results. It is also necessary to check whether under comparable conditions the results obtained using different methods are well coordinated and consistent.
If the evaluation is made based on only one method among all possible methods, questions may arise concerning the comprehensiveness of the study and the competence of the expert(s) who suggested that specific method. With this in mind, an approach that involves the application of several MCDA methods may be recommended, and this helps to ensure that the problem is recognized, understood and analysed more thoroughly.
It also helps decision makers and subject matter experts to achieve consistency in their judgements and evaluations and obtain a stable and robust ranking of the results. In numerical studies, it is rational to move from the application of the simplest methods to more complicated ones. This provides an opportunity to examine the application of more sophisticated MCDA methods in comparison with simpler ones and identify the most suitable option.
If two different groups of experts examine the same comparative evaluation problem, the ranks of alternatives may be different in the general case, even when the same method is used. This is caused by different points of view regarding weights and model assumptions (value/utility functions, etc.). A similar situation could be expected if two expert groups perform their study under the supervision of different experts who are skilled in using different methods. In this case, the probability that the same ranking will result, even if these groups have similar judgements about the problem, is even smaller. Moreover, when evaluating weights using different procedures (for example, swing and pairwise methods), the results may vary significantly, even when they are implemented within the same
MCDA method. Therefore, the representation of the significance of different indicators (weighting factors) is the most sensitive issue in the formal application of MCDA methods, and it requires accuracy and reasonableness.
Differences in ranks may occur due to different performance table representations when experts apply different MCDA methods. Unfortunately, there are no specific rules for conversion of the different variants of the performance table into a universal form that is suitable for use with different methods: the automatic conversion of natural values to scores may result in scores that differ from the scores that experts give directly in the scoring scales.
Performing a detailed uncertainty/sensitivity analysis helps to increase the confidence level for the results and conclusions significantly, as this backs the judgements with information on the stability of the results. One of the important areas for further development of the comparative evaluation approach for NESs is the extension of judgement aggregation methods to problems with uncertainties that are relevant to most of the real world problems.
While MCDA methods can be recommended for use in support of comparative evaluations of NESs, they are only a means to an end, and the identification (in a clear and understandable manner) of the features of NESs that result in different costs, risks and benefits (i.e. ‘key indicators’) is a more important output for supporting decision making.
The elaboration and trial applications of the KIND approach to date have indicated that it can not only be effective in practical comparative evaluations of less mature INESs, but also other energy options and, specifically, NES evolution scenarios and NES versus non-NES comparisons.
Trial applications have shown that the KIND approach provides sufficient flexibility and upgradability for users to choose, rank and sort different NES options at the technology and scenario levels. Such selection and ranking are achieved by searching for compromises among the conflicting indicators representing the performance and sustainability features of NESs and are based on both the judgements of subject matter experts and the preferences of decision makers. Several participants in the project have already examined the usefulness of the KIND approach in maintaining a dialogue with decision makers. What matters is not the final result of a comparative evaluation at the top aggregation level, but rather the option to go down to lower aggregation levels and, when necessary, to particular indicators, sensitivities and uncertainties. This kind of ‘down to the roots’ analysis and representation can make the dialogue with decision makers useful and productive.
Further steps in the elaboration of the KIND approach will be linked to applications of the developed toolkits to practical problems that address a variety of issues that interested Member States rate as important. This will be accomplished within the follow-up INPRO collaborative project on Comparative Evaluation of Nuclear Energy System Options (CENESO). The KIND toolkit will be used to streamline such systemic activities and formulate specific guidelines with respect to particular approaches aimed at improving the performance and sustainability of national, regional and global NESs.
Annex I
OVERVIEW OF PREVIOUS STUDIES
This annex reviews previous studies on the approach for the comparison of nuclear energy systems (NESs), including IAEA studies and projects [I–1 to I–8], several studies undertaken by the Nuclear Energy Agency of the Organisation for Economic Co-Operation and Development (OECD/NEA) on the evaluation of performance and the screening of advanced NESs [I–9 to I–15], Massachusetts Institute of Technology (MIT) studies on the future of nuclear power and nuclear fuel cycles (NFCs) [I–16 to I–18], and a related United States Department of Energy study [I–19]. A set of research articles give an overview of the experience gained in applying Multiple criteria decision making (MCDM) methods and tools in nuclear engineering [I–20 to I–27]. A preliminary analysis of the state of the art aggregation judgement measures applied to the performance comparison performed within the Roadmaps for a Transition to Globally Sustainable Nuclear Energy Systems (ROADMAPS) project is presented in Ref. [I–28].
I–1. IAEA RELATED STUDIES AND PROJECTS
I–1.1. INPRO methodology and GAINS framework for nuclear energy scenario assessment
I–1.1.1. The International Project on Innovative Nuclear Reactors and Fuel Cycles (INPRO) methodology
As part of INPRO, an NES performance assessment methodology was developed to meet national sustainability standards [I–1]. INPRO worked out a set of basic principles, user requirements, techniques and criteria to compare all NESs, including those with innovative components. The principles on which NES performance should be based to provide a sustainable energy supply were additionally proposed [I–2]. In order to become sustainable, each NES should meet all the basic principles in the areas of safety, economics, infrastructure, waste management, proliferation resistance, physical protection and environment.
Figures I–1 and I–2 show the INPRO hierarchy of requirements for Nuclear Energy System Assessment (NESA) that are applicable to the NES design. At the top level, 14 basic principles are established. The middle level (‘user requirements’) contains 52 user requirements specifying the conditions required to achieve compliance with the corresponding basic principle. Therefore, the user requirements contribute directly to the achievement of the top level goals. The third, bottom level presents a set of 125 criteria consisting of indicators and acceptance limits that help to make a judgement on the potential of the NES. These indicators may be based on a single parameter, an aggregated variable, or a status statement.
Two types of indicators, numerical and logical, are used to evaluate NES performance using the INPRO criteria. The numerical indicators reflect the measured or calculated values related to specific NES properties. An
FIG. I–1. INPRO hierarchy.
example could be the estimated probability of a major release of radionuclides to the containment obtained from a probabilistic safety analysis or the number of intact safety barriers maintained after a severe accident. The logical indicators usually refer to specific NES features, and they represent the answers to questions.
In the INPRO assessment areas, some indicators may be applicable to the overall NES, whereas others are only valid for specific components (e.g. reactors). They may relate to the functionality of a whole system or its components, or set out measures to implement analytical methods. Some indicators may utilize evaluation parameters (EPs). These parameters may help the INPRO user to determine whether or not the acceptance limit for an indicator has been met. For example, the evaluation parameters can be applied to a combination of design simplifications, improved materials, increased operating margins and increased redundancy for increasing the robustness of an NES component relative to an existing design, enhancing the defence in depth. The above classification is supplemented by another type of indicator, the so called key indicator (KI), which may be defined in each specific area or in all INPRO areas and depends on the preferences of the technology developers and the Member States.
The acceptance limit of an INPRO criterion represents a qualitative or quantitative target against which the value of an indicator can be compared. It provides a judgement of acceptability (pass/fail, good/bad, better/worse).
There are two types of acceptance limits — numerical (for quantitative targets) and logical (for qualitative targets).
Typically, a logical acceptance limit is a go/no go (i.e. positive (yes) or negative (no)), answer to a question that the indicator addresses.
In the past, INPRO methodology has been applied for comparative assessments of NES sustainability.
However, feedback from experts who attempted to compare different NES options with reference to sustainability using the full range of the INPRO methodology indicated that it is most suitable as an instrument for identifying gaps that prevent NES performance from being fully sustainable.
The INPRO methodology [I–2] has introduced the notions of KIs and ‘desired target values’ that can potentially be used as well. However, the number of KIs proposed in Ref. [I–2] is limited and significantly smaller than the overall number of indicators in the INPRO methodology. The extension of the concept of KIs with desired target values constitutes a ‘relative benefit index’ and a ‘relative risk index’ (to be defined for each KI). The risk may either include the uncertainty in the desired target value of the KI or reflect the required development effort and the level of concept maturity.
To elaborate the national and international recommendations for sustainable NES development, a structured and objective evaluation of the options is necessary. This process includes: (1) screening of one or more NESs selected by a Member State, which can be performed on a national, regional or global basis and aims at compatibility evaluation against objectives of sustainable energy development; (2) comparison of different NESs or their components (e.g. to find a preferred or optimum NES tailored to the needs of a Member State or to make a capability comparison on a global basis); (3) identification of necessary improvements that stimulate research and development to upgrade performance.
The above mentioned objectives can be achieved effectively through the thoughtful complementary application of the INPRO methodology (for sustainability assessment and gap analysis of each of the selected NES options) and the KIND approach (for comparative evaluation of NES options focused on benefits, risks and costs for a particular user). It should also be emphasized that both the INPRO methodology assessment and the comparative evaluation using the KIND approach are complementary; generically, they need to follow the assessments of safety and environmental impacts as defined according to national regulations and IAEA safety standards.
While performing an assessment based on the INPRO methodology [I–2], it is also indispensable to consider the development of nuclear energy in the future (i.e. to define the reference energy scenario (or scenarios) driven by a projected total time dependent demand for nuclear energy generation). This demand can be covered by different NESs; therefore, it is necessary to consider the deployment pace for each NES (power plants with related NFCs, NFC facilities) and the transition period from one NES type to the other. Comparative evaluation of different options for NES evolution scenarios can be analysed using the KIND approach (see Section 5.7).
I–1.1.2. The Global Architecture of Innovative Nuclear Energy Systems Based on Thermal and Fast Reactors Including a Closed Fuel Cycle (GAINS) analytical framework
The scenario oriented evaluation in Section 5.7.1 was based on the results from the INPRO collaborative project GAINS [I–3]. GAINS created an analytical framework for assessing transition scenarios from existing to future NESs.
FIG. I–2. The INPRO hierarchical structure for each area.
FIG. I–2. The INPRO hierarchical structure for each area.