A Model-Based Measure to Assess Operator Adherence to Procedures

Abstract

Procedures play an important role in domains where humans interact with critical, complex systems. In such environments, the operator’s ability to correctly follow a given set of procedures can directly impact system safety. A quantitative measure of procedural adherence during training for complex system operation would be useful to assess trainee performance and evaluate a training program. This paper presents a novel model-based objective metric for quantifying procedural adherence in training. This metric is sensitive to both the number and nature of procedural deviations, and can be used with cluster analysis to classify trainee performance based on adherence. The metric was tested on an experimental data set gathered from volunteers using aircraft maintenance computer-based training (CBT). The properties of the metric are discussed, along with future possibilities.