Learning-based online monitoring for robust robotic perception

Abstract

Autonomous agents rely on accurate perception of the surrounding environment for robust operation. In some applications, perception errors due to model misspecifications or incorrect neural network predictions are inconsequential; however, in safety critical applications, like autonomous driving, one misprediction can have dire effects. There are several works that aim to characterize the robustness of perception offline, but there is a lack of tools to monitor the correctness of perception online during operation. In this thesis, we develop several algorithms to monitor the correctness of pedestrian detections and of 3D mesh models of the environment to enable autonomous agents to detect and react to inconsistencies in their world model. We start by developing a method to track humans in the context of a Simultaneous Localization and Mapping (SLAM) pipeline while monitoring the correctness of pedestrian localization via pose-graph optimization. We then move to the more fine-grained task of monitoring the pose and shape of detected humans from a single image. We develop several model-based approaches and a learning-based approach, the Adversarially Trained Online Monitor (ATOM). ATOM outperforms the model-based approaches and can be used to effectively flag perception errors for human shape and pose estimation. Finally, we investigate methods for monitoring 3D mesh models of the environment with face-level precision using several model-based methods and Face Error Network (FEN).