ML Prediction Models to Identify Novel Beyond Visual Range Tactics and Error Analysis for DARPA AIR Agents

Abstract

This paper investigates the utility of using machine learning models to predict the outcome of simulated 2 vs. 2 Tactical Intercept engagements flown by autonomous agents in support of the DARPA Artificial Intelligence Reinforcements (AIR) program. We investigated the performance of four models: Feed Forward Neural Network, Random Forest, Extreme Gradient Boost, and Long Short Term Memory (LSTM). We examined their ability to successfully predict the outcomes of simulated engagements, tactical errors, and the execution of novel game plans by autonomous agents. The models were trained on 53 features pertaining to the agents including distance between aircraft, altitude, speed, missile availability, and other eventbased features from simulated runs. The LSTM model had the best performance towards the beginning of a run and was able to predict the correct winner with 87.8% accuracy only one minute into a run while the XGBoost model achieved the best overall performance with a 91.7% classification accuracy and an R² of 0.712. The XGB model was also able to correctly predict the winner of 84.7% of the runs after only seven minutes into the simulated engagement. These results demonstrate the utility and need for further investigation into other ML models potential to identify unique attributes and predictive analysis of more complex multi-agent scenarios that include additional criteria such as varying rules of engagement, incorporating acceptable levels of risk as well as other requirements fighter pilots must take into account during offensive and defensive operations needed to gain air superiority and support the objectives of the Joint Forces Commander.