Experiments and a model of pilot system failure detection during simulated Lunar landing

Abstract

Future complex systems, such as those found in piloted aircraft and spacecraft, will undoubtedly utilize significant automation to enhance pilot capabilities and enable novel mission scenarios. Off-nominal conditions may arise, such as failures of automation or other systems, and the ability of a pilot to quickly synthesize the presented state or system status information to detect failures might ultimately influence pilot safety and mission success. This thesis focuses on the performance of a single pilot performing a high workload, time critical, lunar landing task and aims to: 1) quantify the pilots' ability to detect different system failures in various control modes either with or without motion cues, 2) determine how pilots' visual attention changes in different control modes and just before detecting failures, and 3) create a failure detection model within a closed-loop human-vehicle model. In a first experiment, experienced aircraft pilots (n=14) were trained to fly a NASA lunar landing simulator equipped with a large amplitude motion system and to detect a thruster stuck firing, a noisy flight director, or a fuel leak. Each pilot's failure detection performance, mental workload, and situation awareness was then quantified experimentally. Surprisingly, the presence of motion did not affect the time needed for pilots to detect failures - even with failures that had a motion component - when compared to trials without motion. Detection time did, however, depend on both the type of failure and the number of flight axes being manually controlled. Mental workload and situation awareness monotonically degraded with the number of control axes, and further degraded when pilots had to detect failures. In a second experiment, pilots (n=12) flew a fixed-base lunar landing simulator with a similar scenario and system failures. An eye tracker was used to record the number of fixations on various display areas and their average duration. Visual dwell durations were found to increase on the display areas where failure cues appeared prior to the pilot's report that a failure had been detected. This suggests that failure detection is a two-step process: pilots first notice a conflict between actual and expected instrument indications, and then confirm the observation by closely monitoring those instruments before reporting a failure. In addition, it was shown that after pilots transitioned from autopilot to manual control, as expected, attention shifted from supervisory display elements to include elements relevant for manual control. However, increased attention on these instruments used for manual control did not result in faster detection of failure cues that appeared on them. A human failure detection model was developed within a closed-loop, human performance model. A vehicle model was added in order to represent the entire human-machine system. Failure detection was modeled as a two-step process as implied by the second experiment. A mental model was developed to calculate the pilot's internal estimate of system states and attention was simulated as a first-order Markov process. By modifying a threshold within each of the detector's two stages, failure detection simulated by the model was closely correlated (i.e. within two seconds in most cases) to the results of the first experiment. In addition, sensitivity analyses were performed on select model parameters to investigate their effect on failure detection. This research found that pilot's ability to detect failures quickly during simulated lunar landing depends on both the vehicle control mode and the failure cues. Small deviations in detection time during time-critical tasks can have a large impact on mission outcome. A human failure detection model was developed to model failure detection and, once further developed and validated, can be useful in predicting human behavior in future vehicles.