Effect of false alarm rate on pilot use and trust of automation under conditions of simulated high risk

Abstract

An experimental study was conducted to investigate the relationships between automation false alarm rate, human trust in automation, and human use of automation, specifically under conditions of simulated high risk. The experiment involved military pilots flying combat and non-combat missions while employing a missile warning sensor with two levels of automation. The experiment was performed in a high fidelity F- 16 simulator with U.S. Air Force F-16 pilots as test subjects. Three sensor false alarm rates were tested. The results showed that the level of missile warning sensor automation preferred by the pilots was strongly dependent on the false alarm rate of the automation. The data demonstrated that as false alarm rate increased, pilot trust of the missile warning sensor decreased, resulting in the pilot preferring a lower level of automation. However, the results also showed that even when pilots expressed little trust in the system, they preferred to use some level of automation rather than turn the system off. This result was attributed to the cost/benefit tradeoff of using the automation, as well as the extreme difficulty and hazard associated with the task. Analysis of the pilot mean reaction time in response to a missile warning sensor alert showed that reaction time increased as the false alarm rate increased, although this effect appeared to be less significant the higher the false alarm rate. Missile hit data indicated that as false alarm rate increased, the number of missiles that hit the pilot's aircraft also increased, demonstrating a degradation in pilot survivability. Missile hit results also indicated that using a higher level of automation improved pilot survivability for the scenarios tested.