Assessment of candidate performance

To assess operator competencies during a simulator scenario, the assessors/examiners need to observe and note the facts, actions performed, verbal statements made between the candidates/MCR team and role players. Noting instruments observed, procedures being followed information requested from each other and role players. ANNEX VI demonstrates an example of personal evaluation for FSS.

The detailed candidate performance requirements should be described by the licensee or the regulatory agencies in such documents as:

 Job and Task Analyses (link from analysis to SEG objective and training implementation and examination, visible through adhering to and recording/documenting the SAT process during course design and development);

 Training aims and objectives defined in course material;

 Licensee documentation that specifies staff operating roles and responsibilities during normal day to day operation as well as during plant transients and emergencies.

6.2.1. MCR operator performance assessments

The NPP assessment criteria for the training and qualification of their operators will be defined and captured in the Facility Standards and Expectations documentation. The outline for this will be agreed by the facility (some Member States, this may require defining or approving by their regulator) and implemented by the NPP in their training and assessments, within these defined boundaries.

Evaluation of candidates during simulator assessments will be carried out to these defined assessment criteria. Documenting the criteria will help to ensure that standards of performance are applied to the competencies identified in a consistent manner throughout the Utilities NPPs.

The assessment criteria are designed to reflect the performance, behaviours and skills, supported by the knowledge required to perform the Operators day to day MCR position duties.

This includes the fundamentals of:

 Operators monitor plant operations and conditions closely;

 Operators control plant evolutions precisely;

 Operators establish a bias for a conservative approach to plant operations;

 Operators work effectively as a team;

 Operators have a solid understanding (knowledge) of plant design and system inter-relationships.

These Operator Fundamentals are relevant to all the MCR roles being evaluated.

Common skills, such as procedure compliance, panel monitoring, effective communication, diagnostic and decision-making skills, teamwork, etc.

The CRS and SM (Shift Manager) roles will include some higher levels of performance assessment, testing their experience and skills in supervisory, oversight, diagnostic and decision-making capabilities for their roles, e.g. emergency scheme management outside of daytime working hours.

Pre-defined and documented assessment/examination criteria, as agreed by the licensee and/or Regulatory Body, should be in place. Common practices that support consistent and improving performance levels, through continuous review of present practices and levels attained and comparison/benchmarking with other facilities and world best practices, include:

 Written observations of candidate performance, supported by pro-forma check sheets and/or procedural practices, e.g. SEGs;

 Visual and audio recording of candidate performance and key team support actions (some Member States recording performance is not agreed/permitted);

 Candidate’s log entries, procedural adherence tracking and written notes;

 Simulator recording of plant parameters, trends and alarms received and accepted;

 Simulator/computer records of candidate panel operations (switching and control selections) and simulator operator actions.

With this information, candidates can be assessed on the specific actions and behaviours required by the examination SEG scenarios aims and objectives, e.g.:

 Observed actions carried out correctly, such as selecting, following and adhering to the correct procedures during the scenario;

 Recognition of situations where equipment faults place the plant in a situation of decreasing safety margins or not covered by normal/fault operating procedures;

 Prioritizing actions when multiple faults occur;

 Correct diagnosis of faults and timely execution of required actions;

 Clear and concise communications; requesting procedural deviations where required;

 For supervisory staff; management and oversight of control room and plant resources.

The assessment criteria can be sub-divided to account for different levels of performance, requiring appropriate remedial actions to be in place to address each of these categories.

It is common to grade performance at many NPP as either Satisfactory or Unsatisfactory when benchmarked against the required standards and expectations.

Either side of this pass line there may be levels of performance which are then utilized during debrief to identify Strengths to be re-enforced by the training and assessment teams or AFI that may require addressing under the supervision of the line manager, not causing any impedance to continued line activities for the operator. Or some identified remedial action, which may require some additional training or mentoring and may place limitations online duties.

In addition to this there may be identified critical actions in the assessment scenario. The omission/failure to carry out these actions correctly constitutes a failure and remedial training and assessment prior to commencing MCR duties.

Critical errors are nominally errors that jeopardize plant safety in one form or another.

These critical errors are errors that the candidate makes by carrying out an incorrect action or a lack of any remedial action. They can be categorized in number of ways and will need defining in the NPP assessment standard and identified, periodically trained, assessed and recorded on the individual operators training program. A two-level critical assessment system is given below.

6.2.2. Simulator Critical Action

Simulator Critical Action is a significant operator or team action which must be completed correctly during the simulator assessment scenario. Unsatisfactory completion of critical tasks will result in automatic failure during assessments or significant remedial actions during non-assessed scenarios.

An error/improper action, or inaction, that has a serious immediate impact or is a potential threat to plant or public safety, e.g.:

 Non-compliance with Operating Rule/Technical Specification safety limits;

 Not maintaining control of reactor core cooling;

 Failing to take the required actions when it is a procedural requirement, e.g. failure to trip the reactor on a Technical Specification requirement to do so;

 Failure to take timely action on failure of the reactor safety system;

 Taking a deliberate action that results in a violation of the designed plant safety parameters, e.g. violates Technical Specification parameters for reactor power, fuel cooling, etc.;

 Action/inaction as specified in the SEG.

6.2.3. Simulator Safety Significant Action

Simulator Safety Significant Action is a significant operator or team action which must be completed correctly during the simulator training scenario. Un-satisfactory completion of safety significant tasks will require a follow up Shift Manager Action Plan to prevent re-occurrence. An action or inaction may have a potential impact on plant and/or public safety.

Depending on the circumstances and the NPP categorization of safety significant actions, two or more of these errors could warrant the automatic failure of a candidate.

Examples of Safety Significant Actions could include:

 Incorrect fault diagnosis leading to poor decision making;

 Incorrect Operation of controls or plant equipment;

 Inappropriate/lack of control room supervision and/or overview;

 In-correct prioritization of actions concerning Public, Personnel, Plant, Environment and Security, e.g. placing staff in a dangerous situation that could seriously affect health or safety;

 Action causing a reduction in effectiveness of a safety system, or not recognizing the reduction in effectiveness of a safety system;

 Not taking timely and proper action to prevent the degradation of plant safety system margins.

6.2.4. Grading – group results and programme deficiencies

An inappropriate action, taken by an operator, may occur with different candidates over several occasions, using the same scenario. On reviewing the assessment scenarios, procedures used, and previous training provided there may be a common mode of operator failure or inconsistency responsible for causing several candidates or shifts to make similar mistakes.

This could be the result of:

 Problems with the training provided on this topic/scenario recently or in the past;

 Problems with the scenario logic itself, leading operators to make the error, may require human factors investigations to resolve;

 Lack of or short comings in the process to deal with the situation being encountered or flaw in the methodology behind the coping strategy in place;

 Inaccuracies in procedures or administration, e.g. logging and PSCC. Once discovered this information should be provided as feedback to the plant operating organization and the procedures corrected via the appropriate process;

 In this case, the reason behind the common mode failure, if in fact that is what it is, will need to be fully explored and once resolved training may be desired to correct the problem.