Gaze tracking : seeking critical information for autonomous excavation

Abstract

Automating excavation in mining and construction applications is crucial today as the supply of skilled operators cannot match market demand. To efficiently make control decisions for autonomous excavators without having to take in all visual inputs from a typical operator's field of view, gaze tracking is employed in solely extracting key visual information that skilled operators use in the field. Both a front facing camera depicting the world view of the subject and two eye facing cameras that track the subject's pupil movement are worn by a subject to identify regions and features that are of high interest to operators during a digging task. Key features, such as the interface between the soil and the bucket, are characterized using U-Net, a Convolutional Neural Network designed for image segmentation. Through this study, key regions, the inside of the bucket and the opening of the bucket, as well as key features, the soil-bucket interface, were identified to be of high interest to subjects. This information can serve to identify only the necessary visual inputs in the control decision process, thus shortening computation time.