A Koopman-Based Reduced-Order State Observer for Visual Localization of Robots

Abstract

A reduced-order observer using Koopman lifting linearization is developed for localization of a robot guided by a vision system. The Koopman operator is a powerful method for representing nonlinear robot dynamics as a linear model in a lifted space. Koopman faces two main challenges with robot localization. One is that the lifted linear system is not observable in general; standard Kalman filter and state observers cannot be applied to such non-observable systems. The other is that a large number of observables are required for accurate linearization. Here, we present 1) a new reduced-order state observer for a Koopman lifted linear model that satisfies the observability conditions, and 2) measurement of the multitude of Koopman observables by extracting many features from a camera image. These image features used as Koopman observables are directly measured in real-time and, thereby, make the observability matrix of the reduced-order state observer full rank. The method is developed for a robot crane system equipped with a vision system. We can estimate the endpoint of the robot using a reduced-order state observer of a lifted linear model where 20 observables are obtained from a visual image.