GM1 ORO.FC.231(c) Evidence-based training

ED Decision 2021/002/R

TRAINING SYSTEM PERFORMANCE — FEEDBACK PROCESS — METRICS

(a)     Training metrics within the feedback process are a valuable source of data. Typical metrics may include but are not limited to:

(1)     differences in success rates between training topics;

(2)     the trainees’ feedback (e.g. surveys), which provides a different perspective as to the quality and effectiveness of the training;

(3)     instructor concordance assurance: this system is important to measure the effectiveness of the instructor calibration process. It is important to remind that the purpose of this system is not to spy on instructors or to pressure individuals to change their grading;

(4)     level 0 grading metrics (competent metrics): Metrics examples: distribution of pilots not competent after the SBT, distribution of pilots not competent in the EVAL and competent after the SBT;

(5)     level 1 grading metrics (competency metrics): Metrics examples:

(i)      distribution of level of performance within the range of competencies;

(ii)     differences in grades between aircraft types;

(6)     level 2 grading metrics (observable behaviour metrics): e.g. in specific example scenario elements. Metrics example: differences in displaying OBs between ranks of pilots;

(7)     level 3 grading metrics (other metrics such as data based on tasks): for instance, did the pilot calculate the landing distance? Or, did the pilots make a call-out in a specific manoeuvre? This level is usually linked to data collection of the SMS or EBT feedback loop (e.g. was the call-out of the TCAS manoeuvre correct? ‘TCAS I have control’). Metrics example: distribution of errors for various training scenarios and aircraft types.

(8)     during the simulator session, the operator may consider the level of grading metrics that the instructor needs to collect, taking into consideration the workload of the instructor.

(b)     Training metrics are an invaluable component in supporting an EBT programme, but they must be placed in the context of operational data because only the latter can justify the importance of specific training. For this purpose, data from the line evaluation of competence is important to measure the effectiveness of the EBT programme in operations. It may include data from the process for the monitoring of line operations.

(c)      Complex operators may, in the context of their safety management system, establish a safety action group dedicated to training: ‘training safety action group’. This may be a best practice to meet the implementing rule.