Objective performance metrics for improved space telerobotics training

Abstract

NASA astronauts undergo many hours of formal training and self-study to gain proficiency in space teleoperation tasks. After each lesson, instructors score an astronaut's performance in several broad skill categories, including 'General Situational Awareness', 'Maneuvers/Task Performance', and 'Hand- Controller Techniques'. A plus, check, or minus indicates that the student is ahead of, at, or behind the expected skill level. The scoring of the final evaluation for a robotics training course is also largely subjective, with the instructor designating an integer score for the student between 1 (Unsatisfactory) and 5 (Strong) in the same skill categories. This thesis research project was designed to: (1) consider the variety of quantitative metrics that could be embedded into a space robotics training simulation, and (2) investigate at what point and by what means it is most constructive for performance assessment to be revealed to an operator-in-training. We reviewed the current largely qualitative space robotics performance metrics, as well as new quantitative kinematic metrics of manual control skills-including those explored thus far only in laboratory experiments-and additional measures of executive function and supervisory control performance. Kinematic metrics include quantitative measures such as rate of change of linear and rotational acceleration. Potential measures of executive function and supervisory control include camera selection and clearance monitoring. To instantiate our ideas, we chose a specific "fly-to" space telerobotics task taught in the early phases of NASA Generic Robotics Training (GRT) and developed a pilot training experiment (n=16) using our virtual robotics training workstation. Our goal was to evaluate potential performance metrics designed to encourage use of multi-axis control, and to compare real-time ("live") performance feedback alternatives (live visual vs. live aural vs. none). Movement time decreased and multi-axis and bimanual control use gradually increased across trials. All subjects had the opportunity to view post-trial performance feedback including these metrics. Although our subjects overwhelmingly preferred the live, visual feedback condition, no reliable additional effects of live feedback condition were found, except perhaps among the more experienced subjects. However, the experiment demonstrated that embedded performance metrics potentially could quantify and improve some important aspects of GRT evaluations.