1. BACKGROUND
On 27-28 April 2005, country N will be holding a combined national and international nuclear emergency response exercise. The exercise will involve the Alpha NPP, the Omega emergency measures organization (OEMO), the national nuclear emergency plan (NNEP), the International Atomic Energy Agency (IAEA), the Nuclear Energy Agency (NEA) and over 20 countries. The Hydro company and the regional authority are the main drivers for this exercise. The national and international dimensions are built upon the regional components.
At the national level, the aim of this exercise is to:
• verify the ability of the NNEP organization to respond to a nuclear emergency in country N; and
• verify the international coordination mechanisms for the exchange of information.
REX Inc. has been tasked to lead the evaluation of the NNEP component.
2. AIM
The aim of this guide is to provide instructions for the REX-led evaluation team.
3. SCOPE
The scope is limited to the national support centre groups within NNEP, i.e.:
(a) the National Coordinator;
(b) the Operations Section;
(c) the TSC;
(d) the National Coordination Officer and liaison team in the operations centre; and (e) the National Public Information Officer at the PIC.
4. ORGANIZATION 4.1. GENERAL
The NNEP national support centre evaluation team constitutes only part of the overall evaluation team and is shown in detail in Figure XIX-1.
Emergency Preparedness centre of country N provides the lead evaluator for the entire team.
His role is to coordinate the overall evaluation and consolidate the evaluation reports by the various evaluation team leaders into a single evaluation report.
Lead Evaluator (EPC)
Departmental evaluation team leader
Departmental evaluation team leader
Departmental evaluation team leader
NNEP evaluation team leader
Regional evaluator Operations
Regional evaluator Technical
Regional evaluator Public Information
Operations Section evaluator
TSC evaluator
PIC evaluator NNEP Evaluation Team
FIG.XIX-1. Evaluation team and overall national evaluation organization.
4.2. PERSONNEL
Personnel assigned to the evaluation team and primary locations are shown in Table XIX-1.
TABLE XIX-1. NNEP EVALUATION TEAM PERSONNEL
Position Name Primary location
NNEP evaluation team leader Operations Section evaluator TSC evaluator
PIC evaluator
Regional evaluator – Operations Regional evaluator – Technical Regional evaluator - Public Information
4.3. RESPONSIBILITIES
4.3.1. NNEP evaluation team leader
• Coordinate the NNEP evaluation team instructions and logistics.
• Evaluate the National Coordinator.
• Provide instructions to the NNEP evaluation team staff.
• Consolidate the evaluations and produce the NNEP evaluation report.
4.3.2. Operation Section, TSC and PIC evaluators
• Evaluate respective groups.
• Produce evaluation report for respective group in accordance with the evaluation objectives.
4.3.3. Regional evaluator – Operations
• Lead the regional evaluators.
• Evaluate the work of the National Coordination Officer and of the National Operations Liaison Officers.
• Evaluate the operational interface between the national team and the regional team at the operations centre.
4.3.4. Regional evaluator – Technical
• Evaluate the work of the National Technical Liaison Officer at the regional operations centre.
• Evaluate the technical interface between the national team and the regional team at the regional operations centre.
4.3.5. Regional evaluator – Public Information
• Evaluate the work of the National Public Information Officer at the PIC.
• Evaluate the interface between the national team and the regional team at the PIC.
4.4. EVALUATION INSTRUCTIONS
Exercise evaluations are often based on compliance with procedures or checklists. While this is appropriate for the evaluation of complex operating actions (e.g. power plant operations), it does not always provide an adequate assessment of groups involved in emergency management, assessment and decision-making. For this type of evaluation, it is better to measure performance, or the achievement of well defined goals.
A performance-based exercise evaluation is based on defined emergency response functional elements and objectives. This approach does not use evaluation checklists, and there are no procedures to “tick off” during the exercise. Instead, a performance-based exercise evaluation requires that the evaluators have a good knowledge of the evaluation objectives and evaluation criteria that apply to the group being evaluated, and that they keep detailed and accurate records of the pertinent group’s actions.
The actual evaluation takes place after the exercise, when all evaluators share their observations on specific emergency functions and objectives.
This evaluation is to be performance-based.
4.4.1. Guidance for evaluators
The role of evaluators is to observe the exercise and make note of their observations for later analysis and evaluation. They do not interact with the players and should route questions through a controller where possible. The evaluators must be able to recognise deficiencies and make recommendations; however, this does not mean evaluators should constantly be on the lookout for small errors. Only those deficiencies that affect the overall effectiveness need to be investigated thoroughly. Keep note of all deficiencies and quote examples to reinforce comments, to assist the Team Leader during the post-exercise evaluation and debriefing.
Evaluators should:
Prior to the exercise:
a. review the scenario timeline and response objectives applicable to their area; and b. become very familiar with the applicable response objectives and evaluation criteria.
During the exercise:
a. arrive at the assigned location at least 20 minutes prior to the beginning of the exercise;
b. clearly identify themselves as evaluators to the organization being evaluated at the start of the exercise;
c. position themselves to maximise their effectiveness in evaluating, and be passive observers (i.e. DO NOT interfere with exercise play);
d. record facts, not impressions. Identify specific occurrences with time, date, location and organization involved. Record occurrence of repetitive actions;
e. record the time of major scenario events and actions;
f. ensure that each item is relevant to the role of the person/organization being evaluated;
g. note strengths as well as weaknesses;
h. ask questions only for clarification, if absolutely necessary. Do not become involved in discussions with players or other evaluators or controllers;
i. listen to all communications that affect their particular area of evaluation;
j. observe but not correct; and
k. not change the scenario or provide data.
Following the exercise:
a. review their notes;
b. prepare a written report using the list of evaluation objectives and criteria as a guide;
c. attend the evaluation briefing and assist in the production of the consolidated evaluation report.
5. SCHEDULE
For planning purposes, the schedule for NNEP evaluation team members will be as shown in Table XIX-2. This schedule may change slightly as time draws nearer to the evaluation.
TABLE XIX-2. NNEP EVALUATION TEAM SCHEDULE
Event Time and location Comment NNEP exercise staff
briefing
21 April, 09:00 h, [Address]
All evaluation team members to attend.
Exercise scenario notes will be provided at this time.
Regional exercise staff briefing
26 April, 15:00 h,
Superior Room, [Address]
Meeting directed by the control staff. All regional evaluators to attend. This will be followed by a meet and greet.
Pre-exercise check 26 April, 17:00-18:00 All evaluators to report by telephone to the evaluation team leader, for last instructions and final questions.
Exercise 27-28 April. Evaluators report to the ______at 08:30 h.
Regional evaluators report to the ____at 06:45 h.
Regional evaluator at the PIC report to the PIC at 08:30 h.
Post-exercise evaluation meeting.
30 April, 09:00 h, [Address]
Evaluator written reports to be completed prior to this meeting.
6. LOGISTICS
All evaluators are requested to make their own travel arrangements. Hotel reservations have been booked as follows:
Evaluator Date Location
If alternate arrangements are required, evaluators are requested to notify the exercise organizer no later than 19 April, 17:00 h.
7. COMMUNICATIONS
Regional evaluators are requested to provide a cellular phone number to the exercise organizer for use during the exercise no later than 21 April. Reservations can be arranged provided that the exercise organizer is informed of the need for rental phones no later than 21 April. If rental phones are used, the exercise organizer is to be informed of the appropriate phone number immediately upon rental.
Communication with TSC and PIC evaluators will be done through the team leader.
TABLE XIX-3. TELEPHONE LIST
Evaluator Contact number