Low power time-of-flight imaging for augmented reality

Abstract

Mobile augmented reality (AR) technology has seen immense growth in recent years, such as the release of the Microsoft HoloLens2, which allows users to interact with virtual objects around them in the real world. 3D data is often used in AR applications to determine surface normals, allowing for the correct placement and orientation of virtual objects in the real world scene. Time-of-flight (ToF) cameras can be used to acquire this 3D data by emitting light and measuring its round trip time to obtain depth. However, continuous acquisition of high-quality depth maps requires the ToF camera to expend significant amounts of power emitting light, lowering the battery-life of the underlying device. To reduce power consumption, we can use 3D motion computed using the 2D pixel-wise motion of consecutive RGB images captured alongside the ToF camera to obtain a new depth map without illuminating the scene.

In this thesis, we propose depth map reconstruction to limit the ToF camera usage by estimating depth maps using previously captured ones. In our algorithm, we represent previously captured depth maps as a point cloud called the scene map. Each time the ToF camera is used, the captured depth data is added to the scene map to create a representation of the scene captured. The ToF camera is only used when the 3D motion cannot be obtained, or if the obtained depth map contains too many zero-depth (invalid) pixels. To evaluate our algorithm for use in AR applications, we evaluate the accuracy of surface normals and trajectory estimations on our depth maps, in addition to mean relative error (MRE). Using RGB-D datasets, we show that our algorithm reduces the usage of the ToF by up to 97% with negligible impact on surface normals and trajectory estimation, while obtaining depth maps with at least 70% valid pixels.

We further demonstrate the use of our algorithm with integration into an AR application. Finally, we explore the implementation of depth map reconstruction using a CPU-FPGA co-processing architecture to achieve real-time performance.